Jitsu supports streaming of incoming event for data warehouse. In Stream mode all events are loaded to destinations in real-time one by one. That is the fastest way to get data to destinations, but it may not be as performant and cost-effective as Batch mode.
Jitsu uses intermediate queue that helps to survive temporary destination unavailability or slowdowns.
Currently, Jitsu supports only one streaming thread for destination. That may be a bottleneck in cases of high event number. In that case Batch mode is recommended.
- Apply multiplexing, put each multiplexed event to the destination queue. Queue items are persisted in
- Separate thread processes each queue. For each event:
table_name_templateexpression. If the result is an empty string - skip. If evaluation failed, the event is written to
- Apply LookupEnrichment step
- Apply Transformation and MappingStep (get BatchHeader)
- Get Table structure from memory (if memory cache is missing, get the schema from DWH)
- Do table patching (see above in batch step)
- If INSERT failed, refresh schema from DWH and repeat the step
- If failed, write the record to
- If success, write the event to