patx/pickledb
Rewrite PickleDB core, add full test suite, async optimizations, benchmarks, and project cleanup.
Commit edb62d3 · patx · 2025-12-10T14:09:36-05:00
Rewrite PickleDB core, add full test suite, async optimizations, benchmarks, and project cleanup. - Added dualmethod sync/async API and improved context manager behavior - Added/updated comprehensive pytest suite (sync, async, concurrency, atomic writes, 1M-key stress test) - Added benchmark.py for performance testing - Migrated project to pyproject.toml - Updated README + simplified website
Comments
No comments yet.
Diff
diff --git a/README.md b/README.md
index e4aca7d..24dc478 100644
--- a/README.md
+++ b/README.md
@@ -1,24 +1,505 @@
+[](https://patx.github.io/pickledb)
+
[](https://pepy.tech/projects/pickledb)
## **pickleDB: Your Lightweight, High-Speed Key-Value Store**
-- 💡 **Getting Started**: Check out pickleDB's [website](https://patx.github.io/pickledb) for installation instructions, a [user guide](https://patx.github.io/pickledb/guide) complete with advanced examples and the complete [API documentation](https://patx.github.io/pickledb/commands).
+`pickleDB` is a lightweight, in-memory key-value store designed for developers who want **simplicity, speed, and reliability** — without sacrificing modern capabilities. BSD 3-Clause License © Harrison Erd.
+
- 💫 **Blazing Speed**: Backed by the high-performance [orjson](https://pypi.org/project/orjson/) library, pickleDB handles millions of records with ease. Perfect for applications where every millisecond counts
- 😋 **Ridiculously Easy to Use**: With its minimalist API, pickleDB makes adding, retrieving, and managing your data as simple as writing a Python list. No steep learning curves. No unnecessary complexity.
- 🔒 **Rock-Solid Reliability**: Your data deserves to be safe. Atomic saves ensure your database remains consistent—even if something goes wrong.
-- 🐍 **Pythonic Flexibility**: Store strings, lists, dictionaries, and more—all with native Python operations. No need to learn special commands. If you know Python, you already know pickleDB.
-- ⚡ **Async Support** Use pickleDB's `AsyncPickleDB` class for async operations and saves made possible with aiofiles. Ready to go for use with web frameworks like Starlette, FastAPI, and [MicroPie](https://patx.github.io/micropie).
-- 💢 **Limitations**: The entire dataset is loaded into memory, which might be a constraint on systems with limited RAM for extremely large datasets. pickleDB is designed for simplicity, so it may not meet the needs of applications requiring advanced database features. For projects requiring more robust solutions, consider alternatives like [kenobiDB](Https://github.com/patx/kenobi), [Redis](http://redis.io/), [SQLite](https://www.sqlite.org/), or [MongoDB](https://www.mongodb.com/).
+- 🐍 **Simple Pythonic Flexibility**: Store strings, lists, dictionaries, and more—all with native Python operations. No need to learn special commands. If you know Python, you already know pickleDB.
- 🙋 **Community & Contributions**: We’re passionate about making pickleDB better every day. Got ideas, feedback, or an issue to report? Let’s connect on [GitHub Issues](https://github.com/patx/pickledb/issues)
+- 💾 **Portable**: Data is stored as standard JSON, human-readable and cross-language friendly.
+- 🕸️ **Async-Ready**: Non-blocking I/O with [aiofiles](https://pypi.org/project/aiofiles/). Works with web frameworks like Starlette, FastAPI, or [MicroPie](https://patx.github.io/micropie).
+- ⚡ **Unified API**: One class, one set of methods - works seamlessly in **both sync and async** environments.
+- 💢 **Limitations**: The entire dataset resides **in memory** while loaded which might be a constraint on systems with limited RAM for extremely large datasets. pickleDB is designed for simplicity, so it may not meet the needs of applications requiring advanced database features. For larger-scale or concurrent applications requiring a more robust, consider [DataSet](https://dataset.readthedocs.io/en/latest/), [Redis](https://redis.io/), [SQLite](https://www.sqlite.org/), or [MongoDB](https://www.mongodb.com/).
+- 📎 **Useful Links**: [GitHub](https://github.com/patx/pickledb) - [PyPI](https://pypi.org/project/pickleDB/) - [Report an Issue/Ask for Help](https://github.com/patx/pickledb/issues) - [Documentation](https://harrisonerd.com/pickledb)
+
+## Getting Started
+
+### Installation
+Install via pip:
+
+```bash
+pip install pickledb
+```
+
+### Synchronous Example
+
+```python
+from pickledb import PickleDB
+
+db = PickleDB("data.json")
+db.load()
+
+db.set("username", "alice")
+db.set("theme", {"color": "blue", "font": "sans-serif"})
+
+print(db.get("username")) # → "alice"
+
+db.save()
+```
+
+### Asynchronous Example
+
+```python
+import asyncio
+from pickledb import PickleDB
+
+async def main():
+ async with PickleDB("data.json") as db:
+ await db.set("score", 42)
+ value = await db.get("score")
+ print(value) # → 42
+
+asyncio.run(main())
+```
+
+
+## Core Methods
+
+| Method | Description |
+|--------|--------------|
+| `load()` | Loads the database from disk (async-aware). |
+| `save()` | Atomically saves the database to disk. |
+| `set(key, value)` | Sets or updates a key. |
+| `get(key, default=None)` | Returns the value for a key. |
+| `remove(key)` | Deletes a key if it exists. |
+| `all()` | Returns a list of all keys. |
+| `purge()` | Clears the entire database. |
+
+All of these methods can be used **synchronously or asynchronously** — just `await` them if inside an event loop.
-## **Performance Highlights**
+## Performance Highlights
+
pickleDB demonstrates strong performance for handling large-sized datasets:
| Entries | Memory Load Time | Retrieval Time | Save Time |
|--------------|------------------|----------------|-----------|
-| **1M** | 1.21 sec | 0.90 sec | 0.17 sec |
-| **10M** | 14.11 sec | 10.30 sec | 1.67 sec |
-| **50M** | 93.79 sec | 136.42 sec | 61.08 sec |
+| **1M** | 0.68 sec | 0.64 sec | 0.03 sec |
+| **10M** | 7.48 sec | 7.27 sec | 0.22 sec |
+| **50M** | 43.36 sec | 36.53 sec | 1.09 sec |
+
+Tests were performed on a Dell XPS 9350 running Ubuntu 24.04 using pickleDB's async mode.
+
+
+## User Guide and Examples
+
+### Add or Update Data
+
+You can add or update key-value pairs using the `set()` method:
+
+```python
+# Add a new key-value pair
+db.set('username', 'admin')
+
+# Or shorthand
+db['username'] = 'admin'
+
+# Update an existing key-value pair
+db.set('username', 'superadmin')
+print(db.get('username')) # Output: 'superadmin'
+```
+
+Keys are automatically converted to strings, and values can be any **JSON-serializable** object.
+
+### Retrieve Values
+
+You can retrieve a keys value using the `get()` method:
+
+```python
+# Get the value for a key
+print(db.get('username')) # Output: 'superadmin'
+
+# Using Python syntax sugar
+db['username'] # Output: 'superadmin'
+
+# Attempt to retrieve a non-existent key
+print(db.get('nonexistent')) # Output: None
+```
+
+### List All Keys
+
+You can get a list of all the keys currently in the database using the `all()` method:
+
+```python
+db.set('item1', 'value1')
+db.set('item2', 'value2')
+
+print(db.all()) # Output: ['username', 'item1', 'item2']
+```
+
+*Note:* This method shows all keys currently loaded, it does **not** guarantee they are persisted to the disk (yet).
+
+### Remove Keys
+
+To remove a key from the database use the `remove()` method:
+
+```python
+db.remove('item1')
+print(db.all()) # Output: ['username', 'item2']
+```
+
+### Purge the Database
+
+To remove all keys and their values from the database use the `purge()` method:
+
+```python
+db.purge()
+print(db.all()) # Output: []
+```
+
+### Saving Data
+
+**pickleDB does not auto-save by default** for performance reasons. To persist data, call `save()` manually or use a context manager:
+
+```python
+db.save() # Output: True
+
+# Context manager example
+with db:
+ db.set('foo', 'bar')
+ db.set('hello', 'world')
+# Automatically saves when exiting the context
+```
+
+*Note:* All the above methods work/display on the in-memory database. To persist any of the above methods actions use must call the `save()` method or use a context manager, as stated above.
+
+### Asynchronous Usage
+
+pickleDB 1.4 uses a **single unified class** for both synchronous and asynchronous contexts.
+
+```python
+import asyncio
+from pickledb import PickleDB
+
+async def main():
+ async with PickleDB('data.json') as db:
+ await db.set('score', 42)
+ print(await db.get('score')) # Output: 42
+
+asyncio.run(main())
+```
+
+Just `await` any method when inside an async function/event-loop.
+
+### Store and Retrieve Complex Data
+
+```python
+# Store a dictionary
+db.set('user', {'name': 'Alice', 'age': 30, 'city': 'Wonderland'})
+
+# Retrieve and modify
+user = db.get('user')
+user['age'] += 1
+db.set('user', user)
+
+print(db.get('user'))
+# Output: {'name': 'Alice', 'age': 31, 'city': 'Wonderland'}
+```
+
+### Use Lists for Dynamic Data
+
+```python
+db.set('tasks', ['Write code', 'Test app', 'Deploy'])
+
+tasks = db.get('tasks')
+tasks.append('Celebrate')
+db.set('tasks', tasks)
+
+print(db.get('tasks'))
+# Output: ['Write code', 'Test app', 'Deploy', 'Celebrate']
+```
+
+### Advanced Key Search
+
+You can filter keys dynamically using Python list comprehensions:
+
+```python
+def get_keys_with_match(db_instance, match):
+ return [key for key in db_instance.all() if match in key]
+
+db.set('apple', 1)
+db.set('apricot', 2)
+db.set('banana', 3)
+
+print(get_keys_with_match(db, 'ap'))
+# Output: ['apple', 'apricot']
+```
+
+### Namespaces
+
+Simulate namespaces using prefixes:
+
+```python
+db.set('user:1', {'name': 'Alice'})
+db.set('user:2', {'name': 'Bob'})
+
+def get_namespace_keys(db_instance, namespace):
+ return [key for key in db_instance.all() if key.startswith(f"{namespace}:")]
+
+print(get_namespace_keys(db, 'user'))
+# Output: ['user:1', 'user:2']
+```
+
+### Key Expiration (TTL)
+
+pickleDB doesn’t include TTL natively, but you can simulate it:
+
+```python
+import time
+
+def set_with_expiry(db, key, value, ttl):
+ db.set(key, {'value': value, 'expires_at': time.time() + ttl})
+
+def get_if_not_expired(db, key):
+ data = db.get(key)
+ if data and time.time() < data['expires_at']:
+ return data['value']
+ db.remove(key)
+ return None
+
+set_with_expiry(db, 'session', 'active', ttl=5)
+time.sleep(3)
+print(get_if_not_expired(db, 'session')) # 'active'
+time.sleep(3)
+print(get_if_not_expired(db, 'session')) # None
+```
+
+### Encrypted Storage
+
+```python
+from cryptography.fernet import Fernet
+
+key = Fernet.generate_key()
+cipher = Fernet(key)
+
+encrypted = cipher.encrypt(b"My secret data")
+db.set('secure', encrypted)
+
+decrypted = cipher.decrypt(db.get('secure'))
+print(decrypted.decode()) # Output: My secret data
+```
+
+### Batch Operations
+
+```python
+def batch_set(db, items):
+ for key, value in items.items():
+ db.set(key, value)
+
+batch_set(db, {'k1': 'v1', 'k2': 'v2', 'k3': 'v3'})
+print(db.all())
+
+def batch_delete(db, keys):
+ for key in keys:
+ db.remove(key)
+
+batch_delete(db, ['k1', 'k2'])
+print(db.all())
+```
+
+### Key Pattern Matching
+
+```python
+import re
+
+def get_keys_by_pattern(db, pattern):
+ regex = re.compile(pattern)
+ return [key for key in db.all() if regex.search(key)]
+
+db.set('user:1', {'name': 'Alice'})
+db.set('user:2', {'name': 'Bob'})
+db.set('admin:1', {'name': 'Charlie'})
+
+print(get_keys_by_pattern(db, r'user:\d'))
+# Output: ['user:1', 'user:2']
+```
+
+### Signal Handling for Graceful Shutdowns
+
+```python
+import signal, sys
+from pickledb import PickleDB
+
+db = PickleDB('data.json')
+
+signal.signal(signal.SIGINT, lambda s, f: (db.save(), sys.exit(0)))
+signal.signal(signal.SIGTERM, lambda s, f: (db.save(), sys.exit(0)))
+
+db.set('key1', 'value1')
+print("Running... Press Ctrl+C to save and exit.")
+while True:
+ pass
+```
+
+### Using pickleDB with Web Frameworks
+
+Example using [MicroPie](https://patx.github.io/micropie):
+
+```python
+from uuid import uuid4
+from micropie import App
+from pickledb import PickleDB
+
+db = PickleDB('pastes.json')
+
+class Root(App):
+ async def index(self, paste_content=None):
+ if self.request.method == "POST":
+ pid = str(uuid4())
+ await db.set(pid, paste)
+ await db.save()
+ return self._redirect(f'/paste/{pid}')
+ return await self._render_template('index.html')
+
+ async def paste(self, paste_id):
+ paste = await db.get(paste_id)
+ return await self._render_template('paste.html', paste_id=paste_id, paste_content=paste)
+
+app = Root()
+```
+
+## Core API Reference
+
+### Class: `PickleDB`
+
+```python
+class PickleDB(location: str)
+```
+
+A lightweight, JSON-backed key-value database. All data is kept in memory while loaded and written atomically to disk on `save()`.
+
+#### Parameters
+| Name | Type | Description |
+|------|-------|-------------|
+| `location` | `str` | Path to the JSON file backing the database. Tilde (`~`) is expanded. |
+
+
+### Context Manager Support
+
+#### Synchronous
+```python
+with PickleDB("data.json") as db:
+ db.set("foo", "bar")
+```
+
+#### Asynchronous
+```python
+async with PickleDB("data.json") as db:
+ await db.set("foo", "bar")
+```
+
+On successful exit, the DB is automatically saved.
+
+
+### Method Reference
+
+#### `load()`
+
+```python
+load() -> None
+await load() -> None
+```
+
+Loads the database into memory from disk if the file exists and contains valid JSON. Creates an empty database otherwise.
+
+
+#### `save()`
+
+```python
+save() -> bool
+await save() -> bool
+```
+
+Writes the in-memory database to disk.
+
+Returns `True` on success.
+
+Notes:
+- Uses a temporary file + `os.replace` for durability.
+- Automatically called on successful context-manager exit.
+
+#### `set()`
+
+```python
+set(key: str, value: Any) -> bool
+await set(key: str, value: Any) -> bool
+```
+
+Sets or updates a value for a key.
+`key` is coerced to `str`, and `value` must be JSON-serializable.
+
+Returns `True`.
+
+##### Syntax Sugar
+```python
+db["username"] = "alice"
+```
+
+#### `get()`
+
+```python
+get(key: str, default: Any = None) -> Any
+await get(key: str, default: Any = None) -> Any
+```
+
+Retrieves a value by key.
+
+Returns:
+- stored value
+- `default` if key does not exist
+
+##### Syntax Sugar
+```python
+value = db["username"]
+```
+
+
+#### `remove()`
+
+```python
+remove(key: str) -> bool
+await remove(key: str) -> bool
+```
+
+Removes a key.
+
+Returns:
+- `True` if removed
+- `False` if not found
+
+#### `all()`
+
+```python
+all() -> list[str]
+await all() -> list[str]
+```
+
+Returns a list of all keys currently in memory.
+
+#### `purge()`
+
+```python
+purge() -> bool
+await purge() -> bool
+```
+
+Clears the entire in-memory database.
+
+Returns:
+- `True`
+
+### Synchronous vs Asynchronous Behavior
+
+All public methods of `PickleDB` work in both runtimes:
+
+| Environment | Usage |
+|-------------|--------|
+| Synchronous | `db.load()` |
+| Asynchronous | `await db.load()` |
-Tests were performed on a StarLabs StarLite Mk IV (Quad-Core Intel® Pentium® Silver N5030 CPU @ 1.10GHz w/ 8GB memory) running elementary OS 7.1 Horus.
+Internally, the library detects whether it is inside an async event loop.
diff --git a/bench.py b/bench.py
new file mode 100644
index 0000000..10bc9ca
--- /dev/null
+++ b/bench.py
@@ -0,0 +1,128 @@
+#!/usr/bin/env python3
+import argparse
+import asyncio
+import os
+import time
+from pathlib import Path
+
+from pickledb import PickleDB
+
+
+def human_bytes(n):
+ """Return human-readable file size."""
+ for unit in ["B", "KB", "MB", "GB"]:
+ if n < 1024:
+ return f"{n:.2f} {unit}"
+ n /= 1024
+ return f"{n:.2f} TB"
+
+
+# ------------------------------------------------------------------------------
+# SYNC BENCHMARKS
+# ------------------------------------------------------------------------------
+
+def bench_sync(db_path: Path, count: int):
+ print(f"\n=== Sync Benchmark ({count:,} keys) ===")
+ db = PickleDB(str(db_path))
+
+ # SET
+ t0 = time.perf_counter()
+ for i in range(count):
+ db.set(f"key_{i}", i)
+ t1 = time.perf_counter()
+
+ # GET
+ for i in range(count):
+ db.get(f"key_{i}")
+ t2 = time.perf_counter()
+
+ # SAVE
+ db.save()
+ t3 = time.perf_counter()
+
+ set_rate = count / (t1 - t0)
+ get_rate = count / (t2 - t1)
+ save_time = t3 - t2
+ file_size = human_bytes(os.path.getsize(db_path))
+
+ print(f"SET: {set_rate:,.0f} ops/sec ({t1 - t0:.2f}s)")
+ print(f"GET: {get_rate:,.0f} ops/sec ({t2 - t1:.2f}s)")
+ print(f"SAVE: ({save_time:.2f}s)")
+ print(f"File size: {file_size}")
+
+
+# ------------------------------------------------------------------------------
+# ASYNC BENCHMARKS
+# ------------------------------------------------------------------------------
+
+async def bench_async(db_path: Path, count: int):
+ print(f"\n=== Async Benchmark ({count:,} keys) ===")
+ db = PickleDB(str(db_path))
+
+ # SET
+ t0 = time.perf_counter()
+ for i in range(count):
+ await db.set(f"key_{i}", i)
+ t1 = time.perf_counter()
+
+ # GET
+ for i in range(count):
+ await db.get(f"key_{i}")
+ t2 = time.perf_counter()
+
+ # SAVE
+ await db.save()
+ t3 = time.perf_counter()
+
+ set_rate = count / (t1 - t0)
+ get_rate = count / (t2 - t1)
+ save_time = t3 - t2
+ file_size = human_bytes(os.path.getsize(db_path))
+
+ print(f"SET: {set_rate:,.0f} ops/sec ({t1 - t0:.2f}s)")
+ print(f"GET: {get_rate:,.0f} ops/sec ({t2 - t1:.2f}s)")
+ print(f"SAVE: ({save_time:.2f}s)")
+ print(f"File size: {file_size}")
+
+
+# ------------------------------------------------------------------------------
+# CLI ENTRY
+# ------------------------------------------------------------------------------
+
+def main():
+ parser = argparse.ArgumentParser(description="PickleDB performance benchmark.")
+ parser.add_argument(
+ "-n", "--count",
+ type=int,
+ default=100_000,
+ help="Number of key-value pairs (default: 100k)"
+ )
+ parser.add_argument(
+ "--async",
+ action="store_true",
+ dest="async_mode",
+ help="Run async version instead of sync"
+ )
+ parser.add_argument(
+ "--db",
+ type=str,
+ default="benchmark_db.json",
+ help="Database filename (default: benchmark_db.json)"
+ )
+
+ args = parser.parse_args()
+ db_path = Path(args.db)
+
+ # Remove file if exists
+ if db_path.exists():
+ db_path.unlink()
+
+ if args.async_mode:
+ asyncio.run(bench_async(db_path, args.count))
+ else:
+ bench_sync(db_path, args.count)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..2293433
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,27 @@
+[](https://patx.github.io/pickledb)
+
+[pickleDB](https://patx.github.io/pickledb) is a fast, easy to use, in-memory Python
+key-value store with asynchronous support. It is built with the `orjson` module for
+extremely high performance and was originally inspired by Redis. It is licensed under
+the BSD three-clause license.
+
+### pickleDB is easy
+
+```python
+>>> from pickledb import PickleDB
+
+>>> db = PickleDB('example.json')
+
+>>> db.set('key', 'value')
+True
+
+>>> db.get('key')
+'value'
+```
+
+### Useful links
+
+- **Homepage**: [patx.github.io/pickledb](https://patx.github.io/pickledb)
+- **GitHub**: [github.com/patx/pickledb](https://github.com/patx/pickledb)
+
+
diff --git a/docs/commands.html b/docs/commands.html
deleted file mode 100644
index 9a312c6..0000000
--- a/docs/commands.html
+++ /dev/null
@@ -1,230 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
- <meta charset="UTF-8">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <link rel="shortcut icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
- <link rel="icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
- <link href="https://fonts.googleapis.com/css2?family=Poppins:wght@300;400;600&display=swap" rel="stylesheet">
- <title>pickleDB - Simple Key-Value Database</title>
- <style>
- body {
- font-family: 'Poppins', sans-serif;
- margin: 0;
- padding: 0;
-
- color: #333;
- }
- .container {
- max-width: 800px;
- margin: auto;
- margin-top: auto;
- margin-bottom: auto;
- background: #fff;
- padding: 40px;
- border-radius: 15px;
- text-align: center;
- }
- h1, h2 {
- font-weight: 600;
- }
- h2 {
- color: #008000;
- text-align: left;
- }
- p {
- font-size: 18px;
- line-height: 1.6;
- color: #555;
- text-align: left;
- }
- ul {
- text-align: left;
- }
- a {
- color: #008000;
- text-decoration: none;
- font-weight: 600;
- transition: color 0.3s ease;
- }
- a:hover {
- color: #008000;
- }
- pre {
- background: #f5f2f0;
- padding: 20px;
- border-radius: 10px;
- text-align: left;
- overflow-x: auto;
- font-size: 1.1em;
- }
- .logo img {
- max-width: 600px;
- }
- span.c2 {
- color: #8F5902;
- }
- span.c9 {
- color: #008000;
- }
- .github-banner img {
- position: fixed;
- top: 0;
- right: 0;
- border: 0;
- }
- .button-container {
- display: flex;
- justify-content: space-between;
- gap: 10px;
- align-items: center;
- }
- .button {
- display: flex; /* Use flexbox for proper vertical centering */
- align-items: center; /* Vertically center the text */
- justify-content: center; /* Horizontally center the text */
- background: #008000;
- color: #fff;
- padding: 0 30px; /* Only horizontal padding */
- height: 100px; /* Set a fixed height for both buttons */
- border-radius: 30px;
- font-size: 18px;
- font-weight: 600;
- text-align: center;
- width: 48%; /* Ensure both buttons take up equal width */
- transition: background 0.3s ease;
- }
- .button:hover {
- background: white;
- border: 2px solid #008000;
- color: #008000;
- }
- .left-button {
- margin-right: 10px;
- }
- .right-button {
- margin-left: 10px;
- }
- @media (max-width: 768px) {
- .container {
- width: 100%;
- padding: 20px;
- }
- .logo img {
- max-width: 100%;
- }
- .button {
- max-width: 100%;
- text-align: center;
- align-items: center
- }
- .github-banner {
- display: none;
- }
- }
- </style>
-</head>
-<body>
- <div class="container">
- <div class="logo">
- <a href="index.html"><img src="logo.png" alt="pickleDB logo"></a>
- </div>
- <h1>API Documentation</h1>
-
- <p>
- <h2>PickleDB()</h2>
- </p>
-
- <p><code><span class="c2">PickleDB</span>(<span class="c9">path</span>)</code> → Initialize a PickleDB instance with the specified path.</p>
- <ul>
- <li><span class="c9">path</span>: The path to the database file.</li>
- </ul>
-
- <p>
- <strong><code><span class="c2">PickleDB</span></code> Class Methods</strong>
- </p>
-
- <p><code><span class="c2">set</span>(<span class="c9">key</span>, <span class="c9">value</span>)</code> → Add or update a key-value pair in the database.</p>
- <ul>
- <li><span class="c9">key</span>: The key to set. Converted to string if not already.</li>
- <li><span class="c9">value</span>: The value to associate with the key. This can be any JSON serializable Python data type.</li>
- <li><span class="c9">Returns</span>: <em>True</em>.</li>
- </ul>
-
- <p><code><span class="c2">get</span>(<span class="c9">key</span>)</code> → Retrieve the value associated with a key.</p>
- <ul>
- <li><span class="c9">key</span>: The key to retrieve.</li>
- <li><span class="c9">Returns</span>: The value associated with the key, or <em>None</em> if the key does not exist.</li>
- </ul>
-
- <p><code><span class="c2">remove</span>(<span class="c9">key</span>)</code> → Delete a key and its value from the database.</p>
- <ul>
- <li><span class="c9">key</span>: The key to delete.</li>
- <li><span class="c9">Returns</span>: <em>True</em> if the key was deleted, or <em>False</em> if the key does not exist.</li>
- </ul>
-
- <p><code><span class="c2">all</span>()</code> → Retrieve a list of all keys in the database.</p>
- <ul>
- <li><span class="c9">Returns</span>: A list of keys.</li>
- </ul>
-
- <p><code><span class="c2">purge</span>()</code> → Clear all keys and values from the database.</p>
- <ul>
- <li><span class="c9">Returns</span>: <em>True</em>.</li>
- </ul>
-
- <p><code><span class="c2">save</span>(<span class="c9">option</span>)</code> → Save the current state of the database to the file.</p>
- <ul>
- <li><span class="c9">option</span>: OPTIONAL argument to pass `orjson.OPT_*` flags to configure serialization behavior.</li>
- <li><span class="c9">Returns</span>: <em>True</em> if the operation succeeds, or <em>False</em> otherwise.</li>
- </ul>
-
- <h2>AsyncPickleDB(PickleDB)</h2>
- <p><code><span class="c2">AsyncPickleDB</span>(<span class="c9">path</span>)</code> → Initialize an AsyncPickleDB instance with the specified path.</p>
- <ul>
- <li><span class="c9">path</span>: The path to the database file.</li>
- </ul>
-
- <p>
- <strong><code><span class="c2">AsyncPickleDB</span></code> Class Methods</strong>
- </p>
-
- <p><code><span class="c2">aset</span>(<span class="c9">key</span>, <span class="c9">value</span>)</code> → Asynchronously add or update a key-value pair in the database.</p>
- <ul>
- <li><span class="c9">key</span>: The key to set. Converted to string if not already.</li>
- <li><span class="c9">value</span>: The value to associate with the key. This can be any JSON serializable Python data type.</li>
- <li><span class="c9">Returns</span>: <em>True</em>.</li>
- </ul>
-
- <p><code><span class="c2">aget</span>(<span class="c9">key</span>)</code> → Asynchronously retrieve the value associated with a key.</p>
- <ul>
- <li><span class="c9">key</span>: The key to retrieve.</li>
- <li><span class="c9">Returns</span>: The value associated with the key, or <em>None</em> if the key does not exist.</li>
- </ul>
-
- <p><code><span class="c2">aremove</span>(<span class="c9">key</span>)</code> → Asynchronously delete a key and its value from the database.</p>
- <ul>
- <li><span class="c9">key</span>: The key to delete.</li>
- <li><span class="c9">Returns</span>: <em>True</em> if the key was deleted, or <em>False</em> if the key does not exist.</li>
- </ul>
-
- <p><code><span class="c2">aall</span>()</code> → Asynchronously retrieve a list of all keys in the database.</p>
- <ul>
- <li><span class="c9">Returns</span>: A list of keys.</li>
- </ul>
-
- <p><code><span class="c2">apurge</span>()</code> → Asynchronously clear all keys and values from the database.</p>
- <ul>
- <li><span class="c9">Returns</span>: <em>True</em>.</li>
- </ul>
-
- <p><code><span class="c2">asave</span>()</code> → Asynchronously save the current state of the database to the file.</p>
- <ul>
- <li><span class="c9">Returns</span>: <em>True</em> if the operation succeeds, or <em>False</em> otherwise.</li>
- </ul>
-
- <h1>Suggestions</h1>
- <p>If you would like to suggest an improvement or report an issue, please create an <a href="https://github.com/patx/pickledb/issues">issue on GitHub</a>.</p>
- </div>
-</body>
-</html>
diff --git a/docs/guide.html b/docs/guide.html
deleted file mode 100644
index 4e9abd8..0000000
--- a/docs/guide.html
+++ /dev/null
@@ -1,438 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
- <meta charset="UTF-8">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <link rel="shortcut icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
- <link rel="icon" href="https://pythonhosted.org/pickleDB/favicon.ico" type="image/x-icon">
- <link href="https://fonts.googleapis.com/css2?family=Poppins:wght@300;400;600&display=swap" rel="stylesheet">
- <!-- Highlight.js default style with explicit HTTPS URL -->
- <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/styles/default.min.css">
- <title>pickleDB - Simple Key-Value Database</title>
- <style>
- body {
- font-family: 'Poppins', sans-serif;
- margin: 0;
- padding: 0;
- color: #333;
- }
- .container {
- max-width: 800px;
- margin: auto;
- margin-top: auto;
- margin-bottom: auto;
- background: #fff;
- padding: 40px;
- border-radius: 15px;
- text-align: center;
- }
- h1, h2 {
- font-weight: 600;
- }
- h2 {
- color: #008000;
- text-align: left;
- }
- p {
- font-size: 18px;
- line-height: 1.6;
- color: #555;
- text-align: left;
- }
- a {
- color: #008000;
- text-decoration: none;
- font-weight: 600;
- transition: color 0.3s ease;
- }
- a:hover {
- color: #008000;
- }
- pre {
- background: #f5f2f0;
- padding: 20px;
- border-radius: 10px;
- text-align: left;
- overflow-x: auto;
- font-size: 1.1em;
- }
- .logo img {
- max-width: 600px;
- }
- span.c2 {
- color: #8F5902;
- }
- span.c9 {
- color: #008000;
- }
- .github-banner img {
- position: fixed;
- top: 0;
- right: 0;
- border: 0;
- }
- .button-container {
- display: flex;
- justify-content: space-between;
- gap: 10px;
- align-items: center;
- }
- .button {
- display: flex;
- align-items: center;
- justify-content: center;
- background: #008000;
- color: #fff;
- padding: 0 30px;
- height: 100px;
- border-radius: 30px;
- font-size: 18px;
- font-weight: 600;
- text-align: center;
- width: 48%;
- transition: background 0.3s ease;
- }
- .button:hover {
- background: white;
- border: 2px solid #008000;
- color: #008000;
- }
- .left-button {
- margin-right: 10px;
- }
- .right-button {
- margin-left: 10px;
- }
- @media (max-width: 768px) {
- .container {
- width: 100%;
- padding: 20px;
- }
- .logo img {
- max-width: 100%;
- }
- .button {
- max-width: 100%;
- text-align: center;
- align-items: center;
- }
- .github-banner {
- display: none;
- }
- }
- </style>
-</head>
-<body>
- <div class="container">
- <div class="logo">
- <a href="index.html"><img src="logo.png" alt="pickleDB logo"></a>
- </div>
- <h2>Installation</h2>
- <p>
- pickleDB is available on <a href="https://pypi.org/project/pickleDB/">PyPI</a>. Install it with <a href="https://pypi.org/project/pip/">pip</a> in just one command:
- </p>
- <pre><code class="language-bash">pip install pickledb</code></pre>
- <h2>Your first pickleDB</h2>
- <p>Open up your favorite terminal and start a Python shell. Enter the following commands and watch pickleDB go to work!</p>
- <pre><code class="language-python">from pickledb import PickleDB
-
-db = PickleDB('my_database.db') # Initialize the database
-
-db.set('greeting', 'Hello, world!') # Add a key-value pair
-
-db.get('greeting') # Retrieve the value. Output: 'Hello, world!'
-
-db.save() # Save the data to disk</code></pre>
- <p>You should now have a file called <code>my_database.py</code> in your working directory. In it you should see your key-value pair you just added.
- It’s that simple! In just a few lines, you have a fully functioning key-value store.</p>
- <h2>Add or update a key-value pair</h2>
- <p>We can add key-value pairs to our database with the <code>set(key, value)</code> method provided in the <code>PickleDB</code> class for synchronous operations:</p>
- <pre><code class="language-python"># Add a new key-value pair
-db.set('username', 'admin')
-
-# Or shorthand
-db['username'] = 'admin'
-
-# Update an existing key-value pair
-db.set('username', 'superadmin')
-db.get('username') # Output: 'superadmin'</code></pre>
- <p>Your <code>key</code> must be a string. If it is not, pickleDB will convert it into one for you using <code>str()</code>. Your <code>value</code> can be any JSON serializable object.</p>
- <h2>Retrieve the value associated with a key</h2>
- <p>In order to get the <code>value</code> of a <code>key</code> we can use <code>PickleDB</code>'s <code>get(key)</code> method. Remember your <code>key</code> must be a string.</p>
- <pre><code class="language-python"># Get the value for a key
-print(db.get('username')) # Output: 'superadmin'
-
-# Like the set() method, you can use Python syntax sugar here as well
-db['username'] # Output: 'superadmin'
-
-# Attempt to retrieve a non-existent key
-db.get('nonexistent_key') # Output: None</code></pre>
- <h2>Get a list of all keys</h2>
- <p>It is useful to get a list of all the keys in your database for many reasons. To do this we can use the <code>all()</code> method for this:</p>
- <pre><code class="language-python"># Add multiple keys
-db.set('item1', 'value1')
-db.set('item2', 'value2')
-
-# Retrieve all keys
-db.all() # Output: ['username', 'item1', 'item2']</code></pre>
- <h2>Delete a key and its value</h2>
- <p>To delete a key from the database you can use the <code>remove(key)</code> method. This will delete the key and its associated value.</p>
- <pre><code class="language-python"># Remove a key-value pair
-db.remove('item1')
-db.all() # Output: ['username', 'item2']</code></pre>
- <p>As you can see, when we call the <code>all()</code> method now, it no longer shows the key we removed.</p>
- <h2>Delete all keys</h2>
- <p>You can delete all data in the database using the <code>purge()</code> method in the <code>PickleDB</code> class.</p>
- <pre><code class="language-python">db.purge()
-
-db.all() # Output: []</code></pre>
- <h2>Persist the database to disk</h2>
- <p>In order for <strong>any</strong> of these other methods to be permanent you <strong>must</strong> call the <code>save()</code> method. Saving to disk only happens when this method is called. If you add a bunch of
- key-value pairs to your database and the program exits without this command being called all data will be lost! The explicit nature lets you tune the performance of pickleDB to your needs.</p>
- <pre><code class="language-python">db.save() # Output: True</code></pre>
- <p>You can also save the database using a context manager:</p>
- <pre><code class="language-python">with db:
- db.set('foo', 'bar')
- db.set('hello', 'world')
- db.set('candy', 'bad')
-
-# Database saved when the `with` statement exits.</code></pre>
- <h2>The <code>AsyncPickleDB</code> Class</h2>
- <p>While the <code>PickleDB</code> is great for simple and fast data storage, it is not thread safe or asynchronous. So when you are working with things like ASGI frameworks the standard <code>PickleDB</code> class is ill-suited.
- When you need async you need to use the <code>AsyncPickleDB</code> class. This class operates close to the same as we are used to. Let's take a look:</p>
- <pre><code class="language-python">import asyncio
-from pickledb import AsyncPickleDB
-
-async def main():
- # Initialize the AsyncPickleDB object
- db = AsyncPickleDB('my_database.db')
-
- # Add a key-value pair
- await db.aset('greeting', 'Hello, world!')
-
- # Retrieve the value
- greeting = await db.aget('greeting')
- print(greeting) # Output: 'Hello, world!'
-
- # Save the data to disk
- await db.asave()
-
-asyncio.run(main())</code></pre>
- <p>To use <code>AsyncPickleDB</code> you can simply add the <code>await</code> keyword and an "a" to the beginning of the synchronous commands you are used to:</p>
- <pre><code class="language-python">await db.aset('foo', 'bar')
-
-await db.aget('foo')
-
-await db.aremove('foo')
-
-await db.aall()
-
-await db.apurge()
-
-await db.asave()</code></pre>
- <h2>Store and Retrieve Complex Data</h2>
- <p>PickleDB works seamlessly with Python data structures. Example:</p>
- <pre><code class="language-python"># Store a dictionary
-db.set('user', {'name': 'Alice', 'age': 30, 'city': 'Wonderland'})
-
-# Retrieve and update it
-user = db.get('user')
-user['age'] += 1
-
-# Save the updated data
-db.set('user', user)
-print(db.get('user')) # Output: {'name': 'Alice', 'age': 31, 'city': 'Wonderland'}</code></pre>
- <h2>Use Lists for Dynamic Data</h2>
- <p>Handle lists with ease:</p>
- <pre><code class="language-python"># Add a list of items
-db.set('tasks', ['Write code', 'Test app', 'Deploy'])
-
-# Retrieve and modify
-tasks = db.get('tasks')
-tasks.append('Celebrate')
-db.set('tasks', tasks)
-
-print(db.get('tasks')) # Output: ['Write code', 'Test app', 'Deploy', 'Celebrate']</code></pre>
- <h2>Advanced Key Search</h2>
- <p>Search the database with the full power of Python:</p>
- <pre><code class="language-python"># Create simple helper methods based on what YOU need
-def get_keys_with_match_list(db_instance, match):
- return [key for key in db_instance.all() if match in key]
-
-def get_keys_with_match_dict(db_instance, match):
- return dict(filter(lambda item: match in item[0], db_instance.db.items()))
-
-# Create an instance of PickleDB
-db = PickleDB("example.json")
-
-# Add key-value pairs
-db.set('apple', 1)
-db.set('apricot', 2)
-db.set('banana', 3)
-
-# Use glob search to return a list
-matching_keys = get_keys_with_match_list(db, 'ap')
-print(matching_keys) # Output: ['apple', 'apricot']
-
-# Use glob search to return a dict
-matching_dict = get_keys_with_match_dict(db, 'ap')
-print(matching_dict) # Output: {"apple": 1, "apricot": 3}</code></pre>
- <h2>Namespace Support</h2>
- <p>If you use prefixes to simulate namespaces, you can manage groups of keys more efficiently:</p>
- <pre><code class="language-python"># Set multiple keys with a namespace
-db.set('user:1', {'name': 'Alice', 'age': 30})
-db.set('user:2', {'name': 'Bob', 'age': 25})
-
-# Get all keys in a namespace
-def get_namespace_keys(db_instance, namespace):
- return [key for key in db_instance.all() if key.startswith(f"{namespace}:")]
-
-user_keys = get_namespace_keys(db, 'user')
-print(user_keys) # Output: ['user:1', 'user:2']</code></pre>
- <h2>Expire Keys</h2>
- <p>Manually simulate a basic TTL (time-to-live) mechanism for expiring keys:</p>
- <pre><code class="language-python">import time
-
-# Set a key with an expiration time
-def set_with_expiry(db_instance, key, value, ttl):
- db_instance.set(key, {'value': value, 'expires_at': time.time() + ttl})
-
-# Get a key only if it hasn't expired
-def get_if_not_expired(db_instance, key):
- data = db_instance.get(key)
- if data and time.time() < data['expires_at']:
- return data['value']
- db_instance.remove(key) # Remove expired key
- return None
-
-# Example usage
-set_with_expiry(db, 'session_123', 'active', ttl=10)
-time.sleep(5)
-print(get_if_not_expired(db, 'session_123')) # Output: 'active'
-time.sleep(6)
-print(get_if_not_expired(db, 'session_123')) # Output: None</code></pre>
- <h2>Encrypted Storage</h2>
- <p>Use encryption for secure storage of sensitive data:</p>
- <pre><code class="language-python">from cryptography.fernet import Fernet
-
-# Initialize encryption
-key = Fernet.generate_key()
-cipher = Fernet(key)
-
-# Encrypt and save data
-encrypted_value = cipher.encrypt(b"My secret data")
-db.set('secure_key', encrypted_value)
-
-# Retrieve and decrypt data
-encrypted_value = db.get('secure_key')
-decrypted_value = cipher.decrypt(encrypted_value)
-print(decrypted_value.decode()) # Output: My secret data</code></pre>
- <h2>Batch Operations</h2>
- <p>Add multiple key-value pairs in a single operation:</p>
- <pre><code class="language-python">def batch_set(db_instance, items):
- for key, value in items.items():
- db_instance.set(key, value)
-
-# Add multiple keys
-batch_set(db, {'key1': 'value1', 'key2': 'value2', 'key3': 'value3'})
-print(db.all()) # Output: ['key1', 'key2', 'key3']
-
-# Delete multiple key-value pairs in a single operation:
-def batch_delete(db_instance, keys):
- for key in keys:
- db_instance.remove(key)
-
-# Example usage
-db.set('temp1', 'value1')
-db.set('temp2', 'value2')
-batch_delete(db, ['temp1', 'temp2'])
-print(db.all()) # Output: []</code></pre>
- <h2>Key Pattern Matching</h2>
- <p>Support complex matching patterns using regular expressions:</p>
- <pre><code class="language-python">import re
-
-# Get keys that match a regex pattern
-def get_keys_by_pattern(db_instance, pattern):
- regex = re.compile(pattern)
- return [key for key in db_instance.all() if regex.search(key)]
-
-# Example usage
-db.set('user:1', {'name': 'Alice'})
-db.set('user:2', {'name': 'Bob'})
-db.set('admin:1', {'name': 'Charlie'})
-matching_keys = get_keys_by_pattern(db, r'user:\d')
-print(matching_keys) # Output: ['user:1', 'user:2']</code></pre>
- <h2>Adding Custom Signal Handling</h2>
- <p>You can easily implement custom signal handling in your application to ensure graceful shutdowns and data persistence during unexpected terminations.
- Below is an example of how to integrate custom signal handling with pickleDB:</p>
- <pre><code class="language-python">import signal
-import sys
-from pickledb import PickleDB
-
-# Initialize the PickleDB instance
-db = PickleDB('my_database.db')
-
-# Register signal handlers for SIGINT (Ctrl+C) and SIGTERM (system termination)
-signal.signal(signal.SIGINT, lambda signum, frame: (db.save(), sys.exit(0)))
-signal.signal(signal.SIGTERM, lambda signum, frame: (db.save(), sys.exit(0)))
-
-# Example usage
-db.set('key1', 'value1')
-db.set('key2', 'value2')
-
-print("Database is running. Press Ctrl+C to save and exit.")
-
-# Keep the program running to allow signal handling
-try:
- while True:
- pass
-except KeyboardInterrupt:
- pass</code></pre>
- <h2>Async For Web Frameworks</h2>
- <p>For frameworks like FastAPI, Starlette, or MicroPie, use AsyncPickleDB to handle requests without blocking the server:</p>
- <pre><code class="language-python">from uuid import uuid4
-import asyncio
-from MicroPie import App
-from markupsafe import escape
-from pickledb import AsyncPickleDB
-
-db = AsyncPickleDB('pastes.db')
-
-class Root(App):
-
- async def index(self):
- if self.request.method == "POST":
- paste_content = self.request.body_params.get('paste_content', [''])[0]
- pid = str(uuid4())
- await db.aset(pid, escape(paste_content))
- await db.asave()
- return self._redirect(f'/paste/{pid}')
- return await self._render_template('index.html')
-
- async def paste(self, paste_id, delete=None):
- if delete == 'delete':
- await db.aremove(paste_id)
- await db.asave()
- return self._redirect('/')
- paste_content = await db.aget(paste_id)
- return await self._render_template(
- 'paste.html',
- paste_id=paste_id,
- paste_content=paste_content
- )
-
-app = Root()</code></pre>
- </div>
- <div class="github-banner">
- <a href="https://github.com/patx/pickledb">
- <img style="position: fixed; top: 0; right: 0; border: 0;" src="https://github.blog/wp-content/uploads/2008/12/forkme_right_green_007200.png" alt="Fork me on GitHub">
- </a>
- </div>
- <!-- Highlight.js Library with explicit HTTPS URL -->
- <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.7.0/highlight.min.js"></script>
- <script>
- // Initialize syntax highlighting on all code blocks
- hljs.highlightAll();
- </script>
-</body>
-</html>
diff --git a/docs/index.html b/docs/index.html
index e2a4bb3..b39cfc6 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -129,10 +129,12 @@
<div class="logo">
<img src="logo.png" alt="pickleDB logo">
</div>
- <p><strong>pickleDB is a lightweight, simple and fast Python key-value store with asynchronous support.</strong>
- It is built with the <a href="https://pypi.org/project/orjson/">orjson</a> module for extremely high performance and was inspired by <a href="http://redis.io/">Redis</a>. It is licensed under the BSD three-clause license.</p>
+ <p><strong>pickleDB is a fast, easy to use, in-memory Python key-value store with asynchronous support.</strong>
+ It is built with the <a href="https://pypi.org/project/orjson/">orjson</a> module for extremely high performance and
+ was originally inspired by <a href="http://redis.io/">Redis</a>. It is licensed under the BSD three-clause license. View the
+ source code, complete user guide, examples and API documentation on <a href="https://github.com/patx/pickledb">GitHub</a>.</p>
- <h2>pickleDB is Fun</h2>
+ <h2 style="margin-top:50px;">pickleDB is Fun</h2>
<pre><code>
<span class="c2">>>></span> <span class="c9">from</span> pickledb <span class="c9">import</span> PickleDB
@@ -145,15 +147,8 @@ True
'value'
</code></pre>
- <h2> And Easy to Install</h2>
+ <h2 style="margin-top:50px;"> And Easy to Install</h2>
<pre><code><span class="c2">$</span> pip install <span class="c9">pickledb</span></code></pre>
-
- <p><br><br>
- <div class="button-container">
- <a href="guide.html" class="button left-button">User Guide</a>
- <a href="commands.html" class="button middle-button">API Reference</a>
- <a href="https://github.com/patx/pickledb" class="button right-button">GitHub Repo</a>
- </div>
</p>
</div>
<div class="github-banner">
diff --git a/pickledb.py b/pickledb.py
index ff6daca..ea4c5e7 100644
--- a/pickledb.py
+++ b/pickledb.py
@@ -1,268 +1,115 @@
-"""
-Copyright Harrison Erd
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are
-met:
-
-1. Redistributions of source code must retain the above copyright
-notice, this list of conditions and the following disclaimer.
-
-2. Redistributions in binary form must reproduce the above copyright
-notice, this list of conditions and the following disclaimer in the
-documentation and/or other materials provided with the distribution.
-
-3. Neither the name of the copyright holder nor the names of its
-contributors may be used to endorse or promote products derived from
-this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
-IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
-TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
-PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-"""
import asyncio
import os
import aiofiles
import orjson
+from typing import Any
+
+
+def in_async():
+ """Check if running in an event loop."""
+ try:
+ asyncio.get_running_loop()
+ return True
+ except RuntimeError:
+ return False
+
+
+def dualmethod(func):
+ """Allows async methods to also be called synchronously."""
+ def wrapper(self, *args, **kwargs):
+ coro = func(self, *args, **kwargs)
+ if in_async():
+ # In an async context: return coroutine for 'await'
+ return coro
+ # In sync context: run it to completion and return result
+ return asyncio.run(coro)
+ return wrapper
class PickleDB:
"""
- A barebones orjson-based key-value store with essential methods:
- set, get, save, remove, purge, and all.
+ A unified async/sync key-value store using orjson + aiofiles.
"""
- def __init__(self, location):
- """
- Initialize the PickleDB object.
-
- Args:
- location (str): Path to the JSON file.
- """
+ def __init__(self, location: str):
self.location = os.path.expanduser(location)
- self._load()
+ self.db: dict[str, Any] = {}
+ self._lock = asyncio.Lock()
- def __setitem__(self, key, value):
- """
- Wraps the `set` method to allow `db[key] = value`. See `set`
- method for details.
- """
- return self.set(key, value)
+ async def _load(self):
+ """Pure async loader used by both sync and async entrypoints."""
+ if os.path.exists(self.location) and os.path.getsize(self.location) > 0:
+ async with aiofiles.open(self.location, "rb") as f:
+ data = await f.read()
+ self.db = orjson.loads(data)
+ else:
+ self.db = {}
- def __getitem__(self, key):
- """
- Wraps the `get` method to allow `value = db[key]`. See `get`
- method for details.
- """
- return self.get(key)
+ async def _save(self):
+ """Pure async saver used by both sync and async entrypoints."""
+ temp = f"{self.location}.tmp"
+ async with self._lock:
+ async with aiofiles.open(temp, "wb") as f:
+ await f.write(orjson.dumps(self.db))
+ await asyncio.to_thread(os.replace, temp, self.location)
+ return True
def __enter__(self):
- """
- Enter the context manager.
- Does nothing but return `self` for modifications.
- """
+ # Call the *internal* async method, not the dualmethod wrapper
+ asyncio.run(self._load())
return self
def __exit__(self, exc_type, exc_val, exc_tb):
- """
- Exit the context manager.
- Automatically saves changes if no exception occurred.
- """
if exc_type is None:
- self.save()
- return False
-
- def _load(self):
- """
- Load data from the JSON file if it exists, or initialize an empty
- database.
- """
- if (os.path.exists(self.location) and
- os.path.getsize(self.location) > 0):
- try:
- with open(self.location, "rb") as f:
- self.db = orjson.loads(f.read())
- except Exception as e:
- raise RuntimeError(f"{e}\nFailed to load database.")
- else:
- self.db = {}
-
- def save(self, option=0):
- """
- Save the database to the file using an atomic save.
-
- Args:
- options (int): `orjson.OPT_*` flags to configure
- serialization behavior.
-
- Behavior:
- - Writes to a temporary file and replaces the
- original file only after the write is successful,
- ensuring data integrity.
-
- Returns:
- bool: True if save was successful, False if not.
- """
- temp_location = f"{self.location}.tmp"
- try:
- with open(temp_location, "wb") as temp_file:
- temp_file.write(orjson.dumps(self.db, option=option))
- os.replace(temp_location, self.location)
- return True
- except Exception as e:
- print(f"Failed to save database: {e}")
- return False
-
- def set(self, key, value):
- """
- Add or update a key-value pair in the database.
-
- Args:
- key (any): The key to set. If the key is not a string, it
- will be converted to a string.
- value (any): The value to associate with the key.
-
- Behavior:
- - If the key already exists, its value will be updated.
- - If the key does not exist, it will be added to the
- database.
-
- Returns:
- bool: True if the operation succeeds.
- """
- key = str(key) if not isinstance(key, str) else key
- self.db[key] = value
- return True
-
- def remove(self, key):
- """
- Remove a key and its value from the database.
-
- Args:
- key (any): The key to delete. If the key is not a string,
- it will be converted to a string.
-
- Returns:
- bool: True if the key was deleted, False if the key does
- not exist.
- """
- key = str(key) if not isinstance(key, str) else key
- if key in self.db:
- del self.db[key]
- return True
- return False
-
- def purge(self):
- """
- Clear all keys from the database.
-
- Returns:
- bool: True if the operation succeeds.
- """
- self.db.clear()
- return True
-
- def get(self, key):
- """
- Get the value associated with a key.
-
- Args:
- key (any): The key to retrieve. If the key is not a
- string, it will be converted to a string.
-
- Returns:
- any: The value associated with the key, or None if the
- key does not exist.
- """
- key = str(key) if not isinstance(key, str) else key
- return self.db.get(key)
-
- def all(self):
- """
- Get a list of all keys in the database.
-
- Returns:
- list: A list of all keys.
- """
- return list(self.db.keys())
-
-
-class AsyncPickleDB:
- """
- A fully asynchronous orjson-based key-value store.
- Provides async load, save, and CRUD operations with file locking.
- """
-
- def __init__(self, location):
- self.location = os.path.expanduser(location)
- self._lock = asyncio.Lock()
- self.db = {}
+ asyncio.run(self._save())
async def __aenter__(self):
- await self.aload()
+ await self._load()
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
if exc_type is None:
- await self.asave()
- return False # Do not suppress exceptions
+ await self._save()
- async def aload(self):
- """
- Load data from the JSON file if it exists.
- """
- if os.path.exists(self.location) and os.path.getsize(self.location) > 0:
- try:
- async with aiofiles.open(self.location, "rb") as f:
- content = await f.read()
- self.db = orjson.loads(content)
- except Exception as e:
- raise RuntimeError(f"{e}\nFailed to load database.")
- else:
- self.db = {}
+ @dualmethod
+ async def load(self):
+ """Load JSON database from disk."""
+ await self._load()
- async def asave(self, option=0):
- """
- Save the database to file atomically.
- """
- temp_location = f"{self.location}.tmp"
- async with self._lock:
- try:
- async with aiofiles.open(temp_location, "wb") as temp_file:
- await temp_file.write(orjson.dumps(self.db, option=option))
- await asyncio.to_thread(os.replace, temp_location, self.location)
- return True
- except Exception as e:
- print(f"Failed to save database: {e}")
- return False
+ @dualmethod
+ async def save(self):
+ """Atomically save database to disk."""
+ return await self._save()
- async def aset(self, key, value):
+ @dualmethod
+ async def set(self, key, value):
+ """Set a key-value pair."""
async with self._lock:
self.db[str(key)] = value
- return True
+ return True
- async def aget(self, key):
+ @dualmethod
+ async def get(self, key, default=None):
+ """Get a key's value."""
async with self._lock:
- return self.db.get(str(key))
+ return self.db.get(str(key), default)
- async def aremove(self, key):
+ @dualmethod
+ async def remove(self, key):
+ """Remove a key-value pair."""
async with self._lock:
return self.db.pop(str(key), None) is not None
- async def aall(self):
+ @dualmethod
+ async def all(self):
+ """Return a list of all keys."""
async with self._lock:
return list(self.db.keys())
- async def apurge(self):
+ @dualmethod
+ async def purge(self):
+ """Remove all key-value pairs from database."""
async with self._lock:
self.db.clear()
- return True
+ return True
diff --git a/pickledb_dev.py b/pickledb_dev.py
deleted file mode 100644
index 8e843f9..0000000
--- a/pickledb_dev.py
+++ /dev/null
@@ -1,429 +0,0 @@
-"""
-Copyright Harrison Erd
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are
-met:
-
-1. Redistributions of source code must retain the above copyright
-notice, this list of conditions and the following disclaimer.
-
-2. Redistributions in binary form must reproduce the above copyright
-notice, this list of conditions and the following disclaimer in the
-documentation and/or other materials provided with the distribution.
-
-3. Neither the name of the copyright holder nor the names of its
-contributors may be used to endorse or promote products derived from
-this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
-IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
-TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
-PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-"""
-from __future__ import annotations
-
-import asyncio
-import math
-import os
-import threading
-from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple
-
-import aiofiles
-import orjson
-
-
-_VEC_KEY = "__pvindex__" # { key: [floats...] }
-
-
-def _cosine(a: Sequence[float], b: Sequence[float]) -> float:
- # Safe cosine similarity without numpy
- if not a or not b:
- return 0.0
- la = math.sqrt(sum(x * x for x in a))
- lb = math.sqrt(sum(x * x for x in b))
- if la == 0.0 or lb == 0.0:
- return 0.0
- # Use shortest length if vectors differ (graceful handling)
- n = min(len(a), len(b))
- dot = sum(a[i] * b[i] for i in range(n))
- return dot / (la * lb)
-
-
-def _normalize_key(key: Any) -> str:
- return str(key)
-
-
-class PickleDB:
- """
- A minimal orjson-based K/V store with optional autosave and a simple built-in vector index.
-
- Features
- - Thread-safe (RLock) for free-threaded Python (no GIL).
- - Atomic, durable save (temp file + fsync + replace).
- - Optional autosave on set/remove/purge/vector changes.
- - Lightweight vector index for approximate "semantic" lookups via cosine similarity.
- (You provide embeddings; we store and search them.)
-
- Notes
- - Vectors are persisted under the reserved key `_VEC_KEY`.
- - You can ignore vector APIs if you don't need them.
- """
-
- def __init__(self, location: str, *, autosave: bool = False):
- self.location = os.path.expanduser(location)
- self._lock = threading.RLock()
- self._autosave = autosave
- self.db: Dict[str, Any] = {}
- self._vecs: Dict[str, List[float]] = {}
- self._load()
-
- # Syntax sugar
- def __setitem__(self, key: Any, value: Any):
- self.set(key, value)
-
- def __getitem__(self, key: Any):
- return self.get(key)
-
- def __enter__(self):
- return self
-
- def __exit__(self, exc_type, exc_val, exc_tb):
- if exc_type is None:
- self.save()
- return False
-
- # IO
- def _load(self):
- with self._lock:
- if os.path.exists(self.location) and os.path.getsize(self.location) > 0:
- try:
- with open(self.location, "rb") as f:
- data = orjson.loads(f.read())
- except Exception as e:
- raise RuntimeError(f"{e}\nFailed to load database.")
- else:
- data = {}
-
- # Extract vectors & user data
- vecs = data.pop(_VEC_KEY, {})
- if not isinstance(vecs, dict):
- vecs = {}
- self.db = data
- # Ensure vectors are lists of floats
- cleaned: Dict[str, List[float]] = {}
- for k, v in vecs.items():
- if isinstance(v, (list, tuple)) and all(isinstance(x, (int, float)) for x in v):
- cleaned[k] = [float(x) for x in v]
- self._vecs = cleaned
-
- def save(self, option: int = 0) -> bool:
- """
- Persist atomically with fsync durability.
- """
- temp_location = f"{self.location}.tmp"
- with self._lock:
- try:
- # Compose payload including vectors
- payload = dict(self.db)
- if self._vecs:
- payload[_VEC_KEY] = self._vecs
-
- with open(temp_location, "wb") as tf:
- tf.write(orjson.dumps(payload, option=option))
- tf.flush()
- os.fsync(tf.fileno())
-
- os.replace(temp_location, self.location)
- dir_fd = os.open(os.path.dirname(self.location) or ".", os.O_DIRECTORY)
- try:
- os.fsync(dir_fd)
- finally:
- os.close(dir_fd)
- return True
- except Exception as e:
- print(f"Failed to save database: {e}")
- return False
-
- # Key/Value
- def set(self, key: Any, value: Any, *, autosave: Optional[bool] = None) -> bool:
- with self._lock:
- self.db[_normalize_key(key)] = value
- if (self._autosave if autosave is None else autosave):
- self.save()
- return True
-
- def get(self, key: Any) -> Any:
- with self._lock:
- return self.db.get(_normalize_key(key))
-
- def remove(self, key: Any, *, autosave: Optional[bool] = None) -> bool:
- k = _normalize_key(key)
- with self._lock:
- ok = False
- if k in self.db:
- del self.db[k]
- ok = True
- if k in self._vecs: # also remove vector, if any
- del self._vecs[k]
- ok = True
- if ok and (self._autosave if autosave is None else autosave):
- self.save()
- return ok
-
- def purge(self, *, autosave: Optional[bool] = None) -> bool:
- with self._lock:
- self.db.clear()
- self._vecs.clear()
- if (self._autosave if autosave is None else autosave):
- self.save()
- return True
-
- def all(self) -> List[str]:
- with self._lock:
- return list(self.db.keys())
-
- # Vectors
- def set_vector(self, key: Any, vector: Iterable[float], *, autosave: Optional[bool] = None) -> None:
- """
- Attach/update an embedding vector for `key`.
- """
- k = _normalize_key(key)
- vec = [float(x) for x in vector]
- with self._lock:
- self._vecs[k] = vec
- if (self._autosave if autosave is None else autosave):
- self.save()
-
- def get_vector(self, key: Any) -> Optional[List[float]]:
- with self._lock:
- return list(self._vecs.get(_normalize_key(key), [])) or None
-
- def remove_vector(self, key: Any, *, autosave: Optional[bool] = None) -> bool:
- k = _normalize_key(key)
- with self._lock:
- existed = self._vecs.pop(k, None) is not None
- if existed and (self._autosave if autosave is None else autosave):
- self.save()
- return existed
-
- def set_with_vector(
- self, key: Any, value: Any, vector: Iterable[float], *, autosave: Optional[bool] = None
- ) -> None:
- """
- Convenience: set value and vector for the same key in one shot.
- """
- k = _normalize_key(key)
- vec = [float(x) for x in vector]
- with self._lock:
- self.db[k] = value
- self._vecs[k] = vec
- if (self._autosave if autosave is None else autosave):
- self.save()
-
- def search_vector(
- self,
- query_vector: Sequence[float],
- *,
- top_k: int = 5,
- min_score: Optional[float] = None,
- filter_keys: Optional[Iterable[str]] = None,
- ) -> List[Tuple[str, float]]:
- """
- Return top_k (key, score) by cosine similarity.
- """
- q = [float(x) for x in query_vector]
- with self._lock:
- keys = set(filter_keys) if filter_keys else None
- scores: List[Tuple[str, float]] = []
- for k, v in self._vecs.items():
- if keys and k not in keys:
- continue
- s = _cosine(q, v)
- if min_score is None or s >= min_score:
- scores.append((k, s))
- scores.sort(key=lambda kv: kv[1], reverse=True)
- return scores[: max(0, top_k)]
-
-
-class AsyncPickleDB:
- """
- Fully-async variant with the same API shape.
- Safe for no-GIL as long as you confine an instance to a single event loop/thread.
-
- If you need cross-thread access to the SAME instance, add a separate `threading.RLock`
- around all public methods (or keep one instance per loop/thread).
- """
-
- def __init__(self, location: str, *, autosave: bool = False):
- self.location = os.path.expanduser(location)
- self._lock = asyncio.Lock()
- self._autosave = autosave
- self.db: Dict[str, Any] = {}
- self._vecs: Dict[str, List[float]] = {}
-
- async def __aenter__(self):
- await self.aload()
- return self
-
- async def __aexit__(self, exc_type, exc_val, exc_tb):
- if exc_type is None:
- await self.asave()
- return False
-
- # I/O
- async def aload(self):
- async with self._lock:
- if os.path.exists(self.location) and os.path.getsize(self.location) > 0:
- try:
- async with aiofiles.open(self.location, "rb") as f:
- content = await f.read()
- data = orjson.loads(content)
- except Exception as e:
- raise RuntimeError(f"{e}\nFailed to load database.")
- else:
- data = {}
-
- vecs = data.pop(_VEC_KEY, {})
- if not isinstance(vecs, dict):
- vecs = {}
- self.db = data
-
- cleaned: Dict[str, List[float]] = {}
- for k, v in vecs.items():
- if isinstance(v, (list, tuple)) and all(isinstance(x, (int, float)) for x in v):
- cleaned[k] = [float(x) for x in v]
- self._vecs = cleaned
-
- async def asave(self, option: int = 0) -> bool:
- temp_location = f"{self.location}.tmp"
- async with self._lock:
- try:
- payload = dict(self.db)
- if self._vecs:
- payload[_VEC_KEY] = self._vecs
-
- # Write temp file
- async with aiofiles.open(temp_location, "wb") as temp_file:
- await temp_file.write(orjson.dumps(payload, option=option))
- await temp_file.flush()
-
- # Replace & fsync directory on a thread to avoid blocking the event loop
- def _replace_and_sync():
- os.replace(temp_location, self.location)
- dir_fd = os.open(os.path.dirname(self.location) or ".", os.O_DIRECTORY)
- try:
- os.fsync(dir_fd)
- finally:
- os.close(dir_fd)
-
- await asyncio.to_thread(_replace_and_sync)
- return True
- except Exception as e:
- print(f"Failed to save database: {e}")
- return False
-
- # Key/Value
- async def aset(self, key: Any, value: Any, *, autosave: Optional[bool] = None) -> bool:
- async with self._lock:
- self.db[_normalize_key(key)] = value
- do_save = self._autosave if autosave is None else autosave
- if do_save:
- await self.asave()
- return True
-
- async def aget(self, key: Any) -> Any:
- async with self._lock:
- return self.db.get(_normalize_key(key))
-
- async def aremove(self, key: Any, *, autosave: Optional[bool] = None) -> bool:
- k = _normalize_key(key)
- ok = False
- async with self._lock:
- if k in self.db:
- del self.db[k]
- ok = True
- if k in self._vecs:
- del self._vecs[k]
- ok = True
- do_save = ok and (self._autosave if autosave is None else autosave)
- if do_save:
- await self.asave()
- return ok
-
- async def apurge(self, *, autosave: Optional[bool] = None) -> bool:
- async with self._lock:
- self.db.clear()
- self._vecs.clear()
- do_save = self._autosave if autosave is None else autosave
- if do_save:
- await self.asave()
- return True
-
- async def aall(self) -> List[str]:
- async with self._lock:
- return list(self.db.keys())
-
- # Vectors
- async def aset_vector(self, key: Any, vector: Iterable[float], *, autosave: Optional[bool] = None) -> None:
- k = _normalize_key(key)
- vec = [float(x) for x in vector]
- async with self._lock:
- self._vecs[k] = vec
- do_save = self._autosave if autosave is None else autosave
- if do_save:
- await self.asave()
-
- async def aget_vector(self, key: Any) -> Optional[List[float]]:
- async with self._lock:
- return list(self._vecs.get(_normalize_key(key), [])) or None
-
- async def aremove_vector(self, key: Any, *, autosave: Optional[bool] = None) -> bool:
- k = _normalize_key(key)
- async with self._lock:
- existed = self._vecs.pop(k, None) is not None
- do_save = existed and (self._autosave if autosave is None else autosave)
- if do_save:
- await self.asave()
- return existed
-
- async def aset_with_vector(
- self, key: Any, value: Any, vector: Iterable[float], *, autosave: Optional[bool] = None
- ) -> None:
- k = _normalize_key(key)
- vec = [float(x) for x in vector]
- async with self._lock:
- self.db[k] = value
- self._vecs[k] = vec
- do_save = self._autosave if autosave is None else autosave
- if do_save:
- await self.asave()
-
- async def asearch_vector(
- self,
- query_vector: Sequence[float],
- *,
- top_k: int = 5,
- min_score: Optional[float] = None,
- filter_keys: Optional[Iterable[str]] = None,
- ) -> List[Tuple[str, float]]:
- q = [float(x) for x in query_vector]
- async with self._lock:
- keys = set(filter_keys) if filter_keys else None
- scores: List[Tuple[str, float]] = []
- for k, v in self._vecs.items():
- if keys and k not in keys:
- continue
- s = _cosine(q, v)
- if min_score is None or s >= min_score:
- scores.append((k, s))
- scores.sort(key=lambda kv: kv[1], reverse=True)
- return scores[: max(0, top_k)]
-
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..9ae005b
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,28 @@
+[build-system]
+requires = ["flit_core>=3.9,<4"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "pickledb"
+version = "1.4"
+description = "An ultra micro ASGI web framework"
+keywords = ["pickle", "database", "json", "redis", "asyncio"]
+readme = "docs/README.md"
+authors = [{ name = "Harrison Erd", email = "[email protected]" }]
+license = {file = "LICENSE"}
+dependencies = ["orjson>=3.11.5", "aiofiles>=25.1.0"]
+requires-python = ">=3.10"
+classifiers = [
+ "Framework :: AsyncIO",
+ "Programming Language :: Python :: 3",
+ "Intended Audience :: Developers",
+ "Programming Language :: Python",
+ "License :: OSI Approved :: BSD License",
+ "Intended Audience :: Developers",
+ "Topic :: Database"
+]
+
+[project.urls]
+Homepage = "https://patx.github.io/pickledb"
+Repository = "https://github.com/patx/pickledb"
+
diff --git a/setup.py b/setup.py
deleted file mode 100644
index 2cbccb6..0000000
--- a/setup.py
+++ /dev/null
@@ -1,72 +0,0 @@
-"""
-pickleDB
---------
-
-pickleDB is lightweight, fast, and simple database based on the orjson module. And it's BSD licensed!
-
-
-pickleDB is Fun
-```````````````
-
-::
-
- >>> from pickledb import PickleDB
-
- >>> db = PickleDB('test.db')
-
- >>> db.set('key', 'value')
-
- >>> db.get('key')
- 'value'
-
- >>> db.save()
- True
-
-
-And Easy to Install
-```````````````````
-
-::
-
- $ pip install pickledb
-
-
-Links
-`````
-
-* `Website <https://patx.github.io/pickledb>`_
-* `Github Repo <https://github.com/patx/pickledb>`_
-
-
-Key Improvements in Version 1.0+
-````````````````````````````````
-
-* pickleDB 1.0 is a reimagined version designed for speed, simplicity, and reliability. This version is NOT backwards compatible. Key changes include:
-* Atomic Saves: Ensures data integrity during writes, eliminating potential corruption issues.
-* Faster Serialization: Switched to `orjson` for significantly improved speed.
-* Streamlined API: Removed legacy methods (e.g., `ladd`, `dmerge`) in favor of native Python operations.
-* Unified Handling of Data Types: Treats all Python-native types (lists, dicts, etc.) as first-class citizens.
-* Explicit Saves: The `auto_save` feature was removed to provide users greater control and optimize performance.
-* Added built in async class for use with event based applications.
-
-"""
-
-from distutils.core import setup
-
-setup(name="pickleDB",
- version="1.3.2",
- description="A lightweight and simple database using json.",
- long_description=__doc__,
- author="Harrison Erd",
- author_email="[email protected]",
- license="three-clause BSD",
- url="http://github.com/patx/pickledb",
- classifiers = [
- "Programming Language :: Python",
- "License :: OSI Approved :: BSD License",
- "Intended Audience :: Developers",
- "Topic :: Database" ],
- py_modules=['pickledb'],
- install_requires=['orjson', 'aiofiles'],
-)
-
diff --git a/test_pickledb.py b/test_pickledb.py
new file mode 100644
index 0000000..e086098
--- /dev/null
+++ b/test_pickledb.py
@@ -0,0 +1,343 @@
+# test_pickledb.py
+import os
+import json
+import asyncio
+import tempfile
+from pathlib import Path
+
+import pytest
+
+from pickledb import PickleDB # adjust this import if your module name is different
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def make_tmp_path(tmp_path, name="db.json") -> Path:
+ """Helper to get a db path inside pytest's tmp_path fixture."""
+ return tmp_path / name
+
+
+# ---------------------------------------------------------------------------
+# Basic sync usage
+# ---------------------------------------------------------------------------
+
+def test_sync_set_get_save_and_reload(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ # sync .set() & .save() use dualmethod (internally asyncio.run)
+ assert db.set("foo", {"bar": 1}) is True
+ assert db.set("num", 123) is True
+ assert db.save() is True
+
+ # New instance loads from same file
+ db2 = PickleDB(str(db_path))
+ db2.load()
+ assert db2.get("foo") == {"bar": 1}
+ assert db2.get("num") == 123
+
+
+def test_sync_get_default_and_missing_key(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ db.set("exists", "yep")
+ db.save()
+
+ db2 = PickleDB(str(db_path))
+ db2.load()
+
+ assert db2.get("exists", default="nope") == "yep"
+ assert db2.get("missing", default="nope") == "nope"
+ # default=None is implicit
+ assert db2.get("also_missing") is None
+
+
+def test_sync_remove_and_all(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ db.set("a", 1)
+ db.set("b", 2)
+ db.set("c", 3)
+
+ keys = db.all()
+ assert set(keys) == {"a", "b", "c"}
+
+ # remove existing key returns True
+ assert db.remove("b") is True
+ # removing again returns False
+ assert db.remove("b") is False
+
+ keys_after = db.all()
+ assert set(keys_after) == {"a", "c"}
+
+
+def test_sync_purge_clears_database(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ db.set("x", 1)
+ db.set("y", 2)
+ assert set(db.all()) == {"x", "y"}
+
+ assert db.purge() is True
+ assert db.all() == []
+
+ db.save()
+
+ # After reload, still empty
+ db2 = PickleDB(str(db_path))
+ db2.load()
+ assert db2.all() == []
+
+
+# ---------------------------------------------------------------------------
+# Basic async usage
+# ---------------------------------------------------------------------------
+
[email protected]
+async def test_async_set_get_save_and_reload(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ await db.set("foo", {"bar": 1})
+ await db.set("num", 123)
+ await db.save()
+
+ db2 = PickleDB(str(db_path))
+ await db2.load()
+ assert await db2.get("foo") == {"bar": 1}
+ assert await db2.get("num") == 123
+
+
[email protected]
+async def test_async_get_default(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ await db.set("exists", "yep")
+ await db.save()
+
+ db2 = PickleDB(str(db_path))
+ await db2.load()
+
+ assert await db2.get("exists", default="nope") == "yep"
+ assert await db2.get("missing", default="nope") == "nope"
+ assert await db2.get("also_missing") is None
+
+
[email protected]
+async def test_async_remove_all_purge(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ await db.set("a", 1)
+ await db.set("b", 2)
+ await db.set("c", 3)
+
+ keys = await db.all()
+ assert set(keys) == {"a", "b", "c"}
+
+ assert await db.remove("b") is True
+ assert await db.remove("b") is False
+ keys_after = await db.all()
+ assert set(keys_after) == {"a", "c"}
+
+ assert await db.purge() is True
+ assert await db.all() == []
+
+
+# ---------------------------------------------------------------------------
+# Context managers
+# ---------------------------------------------------------------------------
+
+def test_sync_context_manager_saves_on_success(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+
+ with PickleDB(str(db_path)) as db:
+ db.set("inside", "context")
+ # __exit__ should call .save() because no exception
+
+ # New instance: must see saved data
+ db2 = PickleDB(str(db_path))
+ db2.load()
+ assert db2.get("inside") == "context"
+
+
+def test_sync_context_manager_does_not_save_on_exception(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+
+ try:
+ with PickleDB(str(db_path)) as db:
+ db.set("inside", "context")
+ raise RuntimeError("boom")
+ except RuntimeError:
+ pass
+
+ # Because __exit__ sees an exception, it should NOT call .save()
+ # Therefore file may not exist, or be empty.
+ if db_path.exists():
+ content = db_path.read_bytes()
+ assert content == b"" # empty file
+ else:
+ # No file is also acceptable
+ assert not db_path.exists()
+
+
[email protected]
+async def test_async_context_manager_saves_on_success(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+
+ async with PickleDB(str(db_path)) as db:
+ await db.set("inside", "async_context")
+
+ db2 = PickleDB(str(db_path))
+ await db2.load()
+ assert await db2.get("inside") == "async_context"
+
+
[email protected]
+async def test_async_context_manager_does_not_save_on_exception(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+
+ with pytest.raises(RuntimeError):
+ async with PickleDB(str(db_path)) as db:
+ await db.set("inside", "async_context")
+ raise RuntimeError("boom")
+
+ if db_path.exists():
+ content = db_path.read_bytes()
+ assert content == b""
+ else:
+ assert not db_path.exists()
+
+
+# ---------------------------------------------------------------------------
+# Edge cases: missing file, empty file, atomic save
+# ---------------------------------------------------------------------------
+
+def test_load_on_missing_file_returns_empty_db(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ assert not db_path.exists()
+
+ db = PickleDB(str(db_path))
+ db.load()
+ assert db.all() == []
+ # After save, file should exist with {} or similar JSON
+ db.save()
+ assert db_path.exists()
+
+ raw = db_path.read_bytes()
+ data = json.loads(raw.decode("utf-8")) if raw else {}
+ assert data == {}
+
+
+def test_load_on_empty_file_returns_empty_db(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db_path.write_text("") # existing but empty file
+
+ db = PickleDB(str(db_path))
+ # Your load() treats 0-byte file as empty db
+ db.load()
+ assert db.all() == []
+
+
+def test_save_uses_temp_file_and_is_atomic(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ tmp_name = str(db_path) + ".tmp"
+
+ db = PickleDB(str(db_path))
+ db.set("a", 1)
+ db.save()
+
+ # temp file should not be left behind
+ assert not os.path.exists(tmp_name)
+
+ # file should be valid JSON
+ raw = db_path.read_bytes()
+ data = json.loads(raw.decode("utf-8"))
+ assert data == {"a": 1}
+
+
+# ---------------------------------------------------------------------------
+# Concurrency tests
+# ---------------------------------------------------------------------------
+
[email protected]
+async def test_async_concurrent_sets_are_all_present(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ async def writer(start, end):
+ for i in range(start, end):
+ await db.set(f"key_{i}", i)
+
+ # Kick off a few concurrent writers
+ tasks = [
+ asyncio.create_task(writer(0, 300)),
+ asyncio.create_task(writer(300, 600)),
+ asyncio.create_task(writer(600, 1000)),
+ ]
+ await asyncio.gather(*tasks)
+
+ keys = await db.all()
+ assert len(keys) == 1000
+ # spot check
+ assert await db.get("key_0") == 0
+ assert await db.get("key_999") == 999
+
+
[email protected]
+async def test_async_concurrent_gets_and_sets(tmp_path):
+ db_path = make_tmp_path(tmp_path)
+ db = PickleDB(str(db_path))
+
+ await db.set("counter", 0)
+
+ async def incrementer(n_times):
+ for _ in range(n_times):
+ # Not atomic on purpose, but we are testing that we don't crash
+ current = await db.get("counter", 0)
+ await db.set("counter", current + 1)
+
+ tasks = [asyncio.create_task(incrementer(100)) for _ in range(10)]
+ await asyncio.gather(*tasks)
+
+ # We *expect* the final value to be <= 1000 because increments are not atomic.
+ # The important part is no exceptions and a sensible final value.
+ final_value = await db.get("counter")
+ assert 0 < final_value <= 1000
+
+
+# ---------------------------------------------------------------------------
+# Stress test: 1,000,000 key-value pairs
+# ---------------------------------------------------------------------------
+
[email protected]
+async def test_stress_one_million_entries(tmp_path):
+ """
+ Stress test inserting and retrieving 1,000,000 key-value pairs.
+
+ NOTE:
+ - This test is intentionally heavy and may take a long time.
+ - Run it explicitly: `pytest -m stress`.
+ """
+ N = 1_000_000
+ db_path = make_tmp_path(tmp_path, "stress_db.json")
+ db = PickleDB(str(db_path))
+
+ # Insert 1M entries
+ for i in range(N):
+ await db.set(f"key_{i}", i)
+
+ # Retrieve 1M entries
+ for i in range(N):
+ value = await db.get(f"key_{i}")
+ assert value == i
+
+ await db.save()
+ assert db_path.exists()
+
diff --git a/tests.py b/tests.py
deleted file mode 100644
index 798e69a..0000000
--- a/tests.py
+++ /dev/null
@@ -1,213 +0,0 @@
-import unittest
-import os
-import time
-import signal
-import asyncio
-import aiofiles
-import orjson
-
-# Adjust the import path if needed. For example, if 'pickledb' is your own module,
-# ensure the relative or absolute path matches your project structure.
-from pickledb import PickleDB, AsyncPickleDB
-
-
-class TestPickleDB(unittest.TestCase):
- def setUp(self):
- """Set up a PickleDB instance with a real file."""
- self.test_file = "test_pickledb.json"
- self.db = PickleDB(self.test_file)
-
- def tearDown(self):
- """Clean up after tests."""
- if os.path.exists(self.test_file):
- os.remove(self.test_file)
-
- def _timeout_handler(self, signum, frame):
- """Handle timeouts for stress tests."""
- raise TimeoutError("Test exceeded the timeout duration")
-
- # Original Stress Test
- def test_stress_operation(self):
- """Stress test: Insert and retrieve a large number of key-value pairs, then dump."""
- timeout_duration = 600 # Timeout in seconds (10 minutes)
-
- # Set a signal-based timeout
- signal.signal(signal.SIGALRM, self._timeout_handler)
- signal.alarm(timeout_duration)
-
- try:
- num_docs = 1_000_000
-
- # Measure memory loading time
- start_time = time.time()
- for i in range(num_docs):
- self.db.set(f"key{i}", f"value{i}")
- mem_time = time.time()
- mem_duration = mem_time - start_time
- print(f"\n{num_docs} stored in memory in {mem_duration:.2f} seconds")
-
- # Measure retrieval performance before dumping
- start_time = time.time()
- retrieved_docs = [self.db.get(f"key{i}") for i in range(num_docs)]
- retrieval_time = time.time() - start_time
- print(f"Retrieved {num_docs} key-value pairs in {retrieval_time:.2f} seconds")
-
- # Measure dump performance
- start_time = time.time()
- self.db.save()
- dump_time = time.time() - start_time
- print(f"Dumped {num_docs} key-value pairs to disk in {dump_time:.2f} seconds")
-
- finally:
- signal.alarm(0) # Cancel the alarm after the test
-
- # Functional Tests
- def test_set_and_get(self):
- """Test setting and retrieving a key-value pair."""
- self.db.set("key1", "value1")
- self.assertEqual(self.db.get("key1"), "value1")
-
- def test_get_nonexistent_key(self):
- """Test retrieving a key that does not exist."""
- self.assertIsNone(self.db.get("nonexistent"))
-
- def test_remove_key(self):
- """Test removing a key-value pair."""
- self.db.set("key1", "value1")
- self.assertTrue(self.db.remove("key1"))
- self.assertIsNone(self.db.get("key1"))
-
- def test_remove_nonexistent_key(self):
- """Test removing a key that does not exist."""
- self.assertFalse(self.db.remove("nonexistent"))
-
- def test_purge(self):
- """Test purging all keys and values."""
- self.db.set("key1", "value1")
- self.db.set("key2", "value2")
- self.db.purge()
- self.assertEqual(self.db.all(), [])
-
- def test_all_keys(self):
- """Test retrieving all keys."""
- self.db.set("key1", "value1")
- self.db.set("key2", "value2")
- self.assertListEqual(sorted(self.db.all()), ["key1", "key2"])
-
- def test_dump_and_reload(self):
- """Test dumping the database to disk and reloading it."""
- self.db.set("key1", "value1")
- self.db.save()
- reloaded_db = PickleDB(self.test_file)
- self.assertEqual(reloaded_db.get("key1"), "value1")
-
- def test_invalid_file_loading(self):
- """Test initializing a database with a corrupt file."""
- with open(self.test_file, 'w') as f:
- f.write("corrupt data")
- with self.assertRaises(RuntimeError):
- PickleDB(self.test_file)
-
- def test_set_non_string_key(self):
- """Test setting a non-string key."""
- self.db.set(123, "value123")
- self.assertEqual(self.db.get("123"), "value123")
-
- def test_remove_non_string_key(self):
- """Test removing a key that was stored as a non-string key."""
- self.db.set(123, "value123")
- self.assertTrue(self.db.remove(123))
- self.assertIsNone(self.db.get("123"))
-
-
-class TestAsyncPickleDB(unittest.IsolatedAsyncioTestCase):
- async def asyncSetUp(self):
- """Set up an AsyncPickleDB instance with a real file."""
- self.test_file = "test_async_pickledb.json"
- if os.path.exists(self.test_file):
- os.remove(self.test_file)
- self.db = AsyncPickleDB(self.test_file)
-
- async def asyncTearDown(self):
- """Clean up after async tests by removing the test file."""
- if os.path.exists(self.test_file):
- os.remove(self.test_file)
-
- async def test_aset_and_aget(self):
- """Test setting and retrieving a key-value pair asynchronously."""
- await self.db.aset("key1", "async_value1")
- value = await self.db.aget("key1")
- self.assertEqual(value, "async_value1")
-
- async def test_aget_nonexistent_key(self):
- """Test retrieving a key that does not exist asynchronously."""
- value = await self.db.aget("nonexistent")
- self.assertIsNone(value)
-
- async def test_aremove_key(self):
- """Test removing a key-value pair asynchronously."""
- await self.db.aset("key1", "to_remove")
- removed = await self.db.aremove("key1")
- self.assertTrue(removed)
- value = await self.db.aget("key1")
- self.assertIsNone(value)
-
- async def test_aremove_nonexistent_key(self):
- """Test removing a key that does not exist asynchronously."""
- removed = await self.db.aremove("nonexistent")
- self.assertFalse(removed)
-
- async def test_apurge(self):
- """Test purging all keys asynchronously."""
- await self.db.aset("key1", "val1")
- await self.db.aset("key2", "val2")
- await self.db.apurge()
- keys = await self.db.aall()
- self.assertEqual(keys, [])
-
- async def test_aall_keys(self):
- """Test retrieving all keys asynchronously."""
- await self.db.aset("keyA", "valA")
- await self.db.aset("keyB", "valB")
- keys = await self.db.aall()
- self.assertListEqual(sorted(keys), ["keyA", "keyB"])
-
- async def test_asave_and_reload(self):
- """
- Test dumping (asave) the async database to disk and reloading it
- by creating a new AsyncPickleDB instance.
- """
- await self.db.aset("async_key", "async_val")
- await self.db.asave()
-
- # Create a new AsyncPickleDB instance to verify persistence,
- # then use its inherited synchronous `get` method.
- new_db = AsyncPickleDB(self.test_file)
- self.assertEqual(new_db.get("async_key"), "async_val")
-
- async def test_aset_non_string_key(self):
- """Test setting a non-string key asynchronously."""
- await self.db.aset(123, "val123")
- value = await self.db.aget("123")
- self.assertEqual(value, "val123")
-
- async def test_concurrent_access(self):
- """Test concurrent async set operations under the lock."""
- async def set_values(db, start, end):
- for i in range(start, end):
- await db.aset(f"key{i}", f"value{i}")
-
- # Run two coroutines concurrently
- await asyncio.gather(
- set_values(self.db, 0, 50),
- set_values(self.db, 50, 100)
- )
-
- # Verify data integrity
- for i in range(100):
- val = await self.db.aget(f"key{i}")
- self.assertEqual(val, f"value{i}")
-
-
-if __name__ == "__main__":
- unittest.main()