KDF + Opensearch + Kibana VS KDF + Redshift + Quicksight

[novicewhistler]

Hi!

I had a question about the use of these technologies to create near-real-time visualisations. One of the questions asked in the visualisation review section asks for a solution that provides near-real-time and time-sensitive visualisations (Screenshot of the question attached). Having loaded the data from Kinesis Data Firehose, wouldn’t Redshift Spectrum + Quicksight be more performant (Faster) than Opensearch + Kibana?

Also, in what situations would we prefer Kibana visualisations over Quicksight visualisations?

[novicewhistler]

This has now been answered in the slack channel. In case anyone is still curious, here’s a copy-paste of the answer.

While it’s true that ingestion of data from Firehose to Redshift is supported, it’s not actually going to be in near-real-time because of how Firehose shepherds data to Redshift. Under the hood, Firehose has to load streaming data to Amazon S3 first, then from there, it issues a COPY command to move the data to Redshift. This method introduces latencies in the order of minutes as mentioned in this blog
https://aws.amazon.com/blogs/big-data/integrate-etleap-with-amazon-redshift-streaming-ingestion-preview-to-make-data-available-in-seconds/

The new way of streaming data to Redshift appears to be to leverage its streaming ingestion feature in conjunction with Kinesis Data Stream, which substantially lowers latency.