It consumes 1-minute micro-batches from Kafka and then writes the data to S3 as a Delta Lake table. Downstream Spark consumers can use Spark structured streaming to stream-consume the Delta ...

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...

Data lakes stink. That's because lots of them turn to data swamps, and swamps stink. What's the difference between a data lake and a data swamp? A data lake is built on top of cost ...

Event-driven architectures are wonderful. But Kafka was never intended to be a database, and using it as a database won’t solve your problem. It’s a tale as old as time. An enterprise is struggling ...

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...

Streaming is hot. The demand for real-time data processing is rising, and streaming vendors are proliferating and competing. Apache Kafka is a key component in many data pipeline architectures, ...