dlt.destinations.impl.filesystem.factory

filesystem Objects

class filesystem(Destination[FilesystemDestinationClientConfiguration,
                             "FilesystemClient"])

View source on GitHub

__init__

def __init__(bucket_url: str = None,
             credentials: Union[FileSystemCredentials, Dict[str, Any],
                                Any] = None,
             layout: str = DEFAULT_FILE_LAYOUT,
             extra_placeholders: Optional[TExtraPlaceholders] = None,
             current_datetime: Optional[TCurrentDateTime] = None,
             always_refresh_views: bool = None,
             destination_name: str = None,
             environment: str = None,
             **kwargs: Any) -> None

View source on GitHub

Configure the filesystem destination to use in a pipeline and load data to local or remote filesystem.

All arguments provided here supersede other configuration sources such as environment variables and dlt config files.

The bucket_url determines the protocol to be used:

• Local folder: file:///path/to/directory
• AWS S3 (and S3 compatible storages): `s3://bucket-name
• Azure Blob Storage: `az://container-name
• Google Cloud Storage: `gs://bucket-name
• Memory fs: memory://m

Arguments:

• bucket_url str, optional - The fsspec compatible bucket url to use for the destination.
• credentials Union[FileSystemCredentials, Dict[str, Any], Any], optional - Credentials to connect to the filesystem. The type of credentials should correspond to the bucket protocol. For example, for AWS S3, the credentials should be an instance of AwsCredentials. A dictionary with the credentials parameters can also be provided.
• layout str, optional - A layout of the files holding table data in the destination bucket/filesystem. Uses a set of pre-defined and user-defined (extra) placeholders. Please refer to https://dlthub.com/docs/dlt-ecosystem/destinations/filesystem#files-layout
• extra_placeholders Optional[TExtraPlaceholders] - A dictionary of extra placeholder names that can be used in the layout parameter. Names are mapped to string values or to callables evaluated at runtime.
• current_datetime Optional[TCurrentDateTime] - Current datetime used by date/time related placeholders. If not provided, load package creation timestamp will be used.
• always_refresh_views bool, optional - Always refresh sql_client views by setting the newest table metadata or globbing table files
• destination_name str, optional - Name of the destination, can be used in config section to differentiate between multiple of the same type
• environment str, optional - Environment of the destination
• **kwargs Any - Additional arguments passed to the destination config