Forum Replies Created

Viewing 1 of 1 posts
  • novicewhistler

    Member
    February 10, 2022 at 8:41 pm

    This has now been answered in the slack channel. In case anyone is still curious, here’s a copy-paste of the answer.

    While it’s true that ingestion of data from Firehose to Redshift is supported, it’s not actually going to be in near-real-time because of how Firehose shepherds data to Redshift. Under the hood, Firehose has to load streaming data to Amazon S3 first, then from there, it issues a COPY command to move the data to Redshift. This method introduces latencies in the order of minutes as mentioned in this blog
    https://aws.amazon.com/blogs/big-data/integrate-etleap-with-amazon-redshift-streaming-ingestion-preview-to-make-data-available-in-seconds/

    The new way of streaming data to Redshift appears to be to leverage its streaming ingestion feature in conjunction with Kinesis Data Stream, which substantially lowers latency.

Viewing 1 of 1 posts