PipelineIo

class toasty.pipeline.PipelineIo[source]

Bases: abc.ABC

An abstract base class for I/O relating to pipeline processing. An instance of this class might be used to fetch files from, and send them to, a cloud storage system like S3 or Azure Storage.

Methods Summary

check_exists(*path)

Test whether an item at the specified path exists.

get_item(*path[, dest])

Fetch a file-like item at the specified path, writing its contents into the specified file-like object dest.

list_items(*path)

List the items contained in the folder at the specified path.

load_from_config(path)

Create a new I/O backend from saved configuration.

put_item(*path[, source])

Put a file-like item at the specified path, reading its contents from the specified file-like object source.

save_config(path)

Save this object’s configuration to the specified filesystem path.

Methods Documentation

abstract check_exists(*path)[source]

Test whether an item at the specified path exists.

Parameters
*pathstrings

The path to the item, intepreted as components in a folder hierarchy.

Returns
A boolean indicating whether the item in question exists.
abstract get_item(*path, dest=None)[source]

Fetch a file-like item at the specified path, writing its contents into the specified file-like object dest.

Parameters
*pathstrings

The path to the item, intepreted as components in a folder hierarchy.

destwriteable file-like object

The object into which the item’s data will be written as bytes.

Returns
None.
abstract list_items(*path)[source]

List the items contained in the folder at the specified path.

Parameters
*pathstrings

The path to the item, intepreted as components in a folder hierarchy.

Returns
An iterable of (stem, is_folder), where stem is the “basename” of an
item contained within the specified folder and is_folder is a boolean
indicating whether this item appears to be a folder itself.
classmethod load_from_config(path)[source]

Create a new I/O backend from saved configuration.

Parameters
pathpath-like

The path where the configuration was saved.

Returns
A new instance implementing the PipelineIO abstract base class.
abstract put_item(*path, source=None)[source]

Put a file-like item at the specified path, reading its contents from the specified file-like object source.

Parameters
*pathstrings

The path to the item, intepreted as components in a folder hierarchy.

sourcereadable file-like object

The object from which the item’s data will be read, as bytes.

Returns
None.
save_config(path)[source]

Save this object’s configuration to the specified filesystem path.