Announcing General Availability of Zerobus Ingest, part of Lakeflow Connect

Summary

Zerobus Ingest has been announced as a General Availability service, providing a fully managed, serverless solution for streaming data directly into Delta tables, thus eliminating the need for traditional message buses like Kafka. This service supports high throughput and low latency, enabling organizations to streamline their data ingestion processes while reducing operational complexity and costs. By adopting a single-sink architecture, Zerobus Ingest allows data producers to bypass intermediate layers, leading to significant reductions in engineering overhead and improved performance. The service is designed to integrate seamlessly with existing data governance frameworks, ensuring compliance and lineage tracking from the moment data is ingested.

Key Learnings

• 1Zerobus Ingest simplifies data ingestion by removing the need for intermediate message buses, thus reducing costs and operational complexity.
• 2The service supports high-performance data streaming with sub-5-second latency and can handle thousands of concurrent connections.
• 3By utilizing a single-sink architecture, Zerobus Ingest eliminates the need for brokers and partitions, which are common in traditional streaming architectures.
• 4Integration is facilitated through gRPC and REST APIs, along with SDKs for multiple programming languages, enhancing flexibility for developers.
• 5The service ensures data governance and lineage tracking through Unity Catalog, providing a cohesive framework for data management.

Who Should Read This

Senior Data Engineers implementing real-time data ingestion solutions in cloud environments.

Test Your Knowledge

Topics