This is a Jitsu.Classic documentation. For the lastest version, please visit docs.jitsu.com. Read about differences here.
S3
Overview
The S3 source enables syncing of file-based tables with support for multiple files using glob-like pattern matching, and both Full Refresh and Incremental syncs, using the last_modified property of files to determine incremental batches. You can choose if this connector will read only the new/updated files, or all the matching files, every time a sync is run.The source is using Airbyte docker image (@airbyte/source-s3). Learn more how Airbyte-based sources work
How to connect
Please read setup guide for more information.Connection Parameters
| Parameter | Documentation |
|---|---|
dataset*string (required) | This source creates one table per connection, this field is the name of that table. This should include only letters, numbers, dash and underscores. Note that this may be altered according to destination. |
path_pattern*string (required) | Add at least 1 pattern here to match filepaths against. Use | to separate multiple patterns. Airbyte uses these patterns to determine which files to pick up from the provider storage. See wcmatch.glob to understand pattern syntax (GLOBSTAR and SPLIT flags are enabled). Use pattern ** to pick up all files. |
schemastring (not required) | Optionally provide a schema to enforce, as a valid JSON string. Ensure this is a mapping of { "column" : "type" }, where types are valid JSON Schema datatypes. Leave as {} to auto-infer the schema. |
formatobject (not required) | |
provider*object (required) |