caterpillar.storage package¶
Module Contents¶
All communication with persistent storage devices is done via an implementation of Storage, which provides a key/value object interface to the underlying persistence device and transactions so we can enforce ACID.
Anyone can write an implementation of Storage, but it is envisaged for the most part that these implementations will usually be wrappers around other tools like SQLite or RockDB. No point re-inventing the wheel!
- exception caterpillar.storage.ContainerNotFoundError¶
Bases: caterpillar.storage.StorageError
No container by that name exists.
- exception caterpillar.storage.DuplicateContainerError¶
Bases: caterpillar.storage.StorageError
A container by the same name already exists.
- exception caterpillar.storage.DuplicateStorageError¶
Bases: caterpillar.storage.StorageError
A storage object already exists at the specified location.
- class caterpillar.storage.Storage(path, create=False, readonly=False)¶
Bases: object
Abstract class used to store key/value data on disk.
Implementations of this class handle the storage of data to disk. The view of persistent storage presented here is one of key/value pairs stored in any number of containers.
Implementers must provide primitives for implementing atomic transactions. That is, they must provide begin(), commit(), and meth:.rollback.
Storage implementations must also ensure that they provide reader/writer isolation. That is, if a storage instance is created and a transaction started, any write operations made in that transaction should not be visible to any existing or new storage instances. After the transaction is committed, the changes should not be visible to any existing instances that have called begin() but should be visible to any new or existing storage instances that are yet to call begin().
The __init__() method of a storage implementation should take care of the required bootstrap required to open existing storage OR create new storage (via a create flag). It also needs to support a readonly flag.
Finally, storage instances MUST be thread-safe.
- add_container(c_id)¶
Add a container with c_id (str) to this Storage.
Throws a DuplicateContainer error if there is already a container with this c_id.
- begin()¶
Begin a transaction.
- clear()¶
Clears this storage object of all data.
- clear_container(c_id)¶
Clear all values from container c_id (str).
- close()¶
Close this storage session and release all resources.
- commit()¶
Commit a transaction
- delete_container(c_id)¶
Delete a container with c_id (str) from this Storage.
Throws a ContainerNotFound error if the c_id doesn’t match a stored container.
- delete_container_item(c_id, key)¶
Delete the key/value pair from container c_id (str) identified by key (str).
- delete_container_items(c_id, keys)¶
Delete multiple key/value pairs from container c_id (str) using the iterable of keys.
- get_container_item(c_id, key)¶
Retrieve a single item with key from container with c_id.
- get_container_items(c_id, keys=None)¶
Generator across some or all the items stored in a container identified by c_id (str) depending on the value of keys (list).
If keys is None this method fetches all items otherwise this method fetches the items matching the keys in the keys list.
- get_container_keys(c_id)¶
Generator across all keys from container with c_id (str).
- get_container_len(c_id)¶
Retrieve count of items in container with c_id (str).
- rollback()¶
Rollback a transaction
- set_container_item(c_id, key, value)¶
Add a single key/value pair to container c_id (str). Both key and value will be coerced to strings by calling str().
- set_container_items(c_id, items)¶
Add numerous key value pairs to container c_id (str).
items should be an iterable of string key/value pairs.
- exception caterpillar.storage.StorageError¶
Bases: exceptions.Exception
Base for all Storage exceptions.
- exception caterpillar.storage.StorageNotFoundError¶
Bases: caterpillar.storage.StorageError
No storage object found at the specified location.
caterpillar.storage.sqlite module¶
An sqlite implementation of caterpillar.storage.Storage.
The only class is SqliteStorage which uses sqlite in WAL mode to achieve reader/writer isolation.
- class caterpillar.storage.sqlite.SqliteStorage(path, create=False, readonly=False)¶
Bases: caterpillar.storage.Storage
This class utilises SQLite to store data structures to disk.
Reader / writer isolation here is provided by using WAL mode. There are no changes to the default checkpoint behaviour of SQLite, which at the time of writing defaults to 1000 pages.
This storage type creates a new table for each container.
- add_container(c_id)¶
Add a data container identified by c_id (str).
Raises DuplicateContainerError if there is already a container called c_id (str).
- begin()¶
Begin a transaction.
- clear()¶
Clear all containers from storage.
- clear_container(c_id)¶
Clear all data in container c_id (str).
- close()¶
Close this storage object and all its resources, rendering it UNUSABLE.
- commit()¶
Commit a transaction.
- delete_container(c_id)¶
Delete container c_id (str).
- delete_container_item(c_id, key)¶
Delete item key (str) from container c_id (str).
- delete_container_items(c_id, keys)¶
Delete items keys (str) from container c_id (str).
- get_container_item(c_id, key)¶
Get item at key (str) from container c_id (str).
- get_container_items(c_id, keys=None)¶
Generator of items at keys (list) in container c_id (str).
If keys is None, iterates all items.
- get_container_keys(c_id)¶
Generator of keys from container c_id (str).
- get_container_len(c_id)¶
Get the number of rows in container c_id (str).
- rollback()¶
Rollback a transaction.
- set_container_item(c_id, key, value)¶
Add key/value pair to container c_id (str).
- set_container_items(c_id, items)¶
Add the dict of key/value tuples to container c_id (str).