caterpillar.storage package

Module Contents

All communication with persistent storage devices is done via an implementation of Storage, which provides a key/value object interface to the underlying persistence device and transactions so we can enforce ACID.

Anyone can write an implementation of Storage, but it is envisaged for the most part that these implementations will usually be wrappers around other tools like SQLite or RockDB. No point re-inventing the wheel!

exception caterpillar.storage.ContainerNotFoundError

Bases: caterpillar.storage.StorageError

No container by that name exists.

exception caterpillar.storage.DuplicateContainerError

Bases: caterpillar.storage.StorageError

A container by the same name already exists.

exception caterpillar.storage.DuplicateStorageError

Bases: caterpillar.storage.StorageError

A storage object already exists at the specified location.

class caterpillar.storage.Storage(path, create=False, readonly=False)

Bases: object

Abstract class used to store key/value data on disk.

Implementations of this class handle the storage of data to disk. The view of persistent storage presented here is one of key/value pairs stored in any number of containers.

Implementers must provide primitives for implementing atomic transactions. That is, they must provide begin(), commit(), and meth:.rollback.

Storage implementations must also ensure that they provide reader/writer isolation. That is, if a storage instance is created and a transaction started, any write operations made in that transaction should not be visible to any existing or new storage instances. After the transaction is committed, the changes should not be visible to any existing instances that have called begin() but should be visible to any new or existing storage instances that are yet to call begin().

The __init__() method of a storage implementation should take care of the required bootstrap required to open existing storage OR create new storage (via a create flag). It also needs to support a readonly flag.

Finally, storage instances MUST be thread-safe.

add_container(c_id)

Add a container with c_id (str) to this Storage.

Throws a DuplicateContainer error if there is already a container with this c_id.

begin()

Begin a transaction.

clear()

Clears this storage object of all data.

clear_container(c_id)

Clear all values from container c_id (str).

close()

Close this storage session and release all resources.

commit()

Commit a transaction

delete_container(c_id)

Delete a container with c_id (str) from this Storage.

Throws a ContainerNotFound error if the c_id doesn’t match a stored container.

delete_container_item(c_id, key)

Delete the key/value pair from container c_id (str) identified by key (str).

delete_container_items(c_id, keys)

Delete multiple key/value pairs from container c_id (str) using the iterable of keys.

get_container_item(c_id, key)

Retrieve a single item with key from container with c_id.

get_container_items(c_id, keys=None)

Generator across some or all the items stored in a container identified by c_id (str) depending on the value of keys (list).

If keys is None this method fetches all items otherwise this method fetches the items matching the keys in the keys list.

get_container_keys(c_id)

Generator across all keys from container with c_id (str).

get_container_len(c_id)

Retrieve count of items in container with c_id (str).

rollback()

Rollback a transaction

set_container_item(c_id, key, value)

Add a single key/value pair to container c_id (str). Both key and value will be coerced to strings by calling str().

set_container_items(c_id, items)

Add numerous key value pairs to container c_id (str).

items should be an iterable of string key/value pairs.

exception caterpillar.storage.StorageError

Bases: exceptions.Exception

Base for all Storage exceptions.

exception caterpillar.storage.StorageNotFoundError

Bases: caterpillar.storage.StorageError

No storage object found at the specified location.

caterpillar.storage.sqlite module

An sqlite implementation of caterpillar.storage.Storage.

The only class is SqliteStorage which uses sqlite in WAL mode to achieve reader/writer isolation.

class caterpillar.storage.sqlite.SqliteStorage(path, create=False, readonly=False)

Bases: caterpillar.storage.Storage

This class utilises SQLite to store data structures to disk.

Reader / writer isolation here is provided by using WAL mode. There are no changes to the default checkpoint behaviour of SQLite, which at the time of writing defaults to 1000 pages.

This storage type creates a new table for each container.

add_container(c_id)

Add a data container identified by c_id (str).

Raises DuplicateContainerError if there is already a container called c_id (str).

begin()

Begin a transaction.

clear()

Clear all containers from storage.

clear_container(c_id)

Clear all data in container c_id (str).

close()

Close this storage object and all its resources, rendering it UNUSABLE.

commit()

Commit a transaction.

delete_container(c_id)

Delete container c_id (str).

delete_container_item(c_id, key)

Delete item key (str) from container c_id (str).

delete_container_items(c_id, keys)

Delete items keys (str) from container c_id (str).

get_container_item(c_id, key)

Get item at key (str) from container c_id (str).

get_container_items(c_id, keys=None)

Generator of items at keys (list) in container c_id (str).

If keys is None, iterates all items.

get_container_keys(c_id)

Generator of keys from container c_id (str).

get_container_len(c_id)

Get the number of rows in container c_id (str).

rollback()

Rollback a transaction.

set_container_item(c_id, key, value)

Add key/value pair to container c_id (str).

set_container_items(c_id, items)

Add the dict of key/value tuples to container c_id (str).