The open source technology will enable organizations to transform their existing messy data lakes into clean Delta Lakes with high quality data, says the company.
Databricks, the company founded by the original creators of Apache Spark, has announced a new open source project, called Delta Lake, to deliver reliability to data lakes.
According to the company, Delta Lake is the first production-ready open source technology to provide data lake reliability for both batch and streaming data.
This new open source project, Databricks said, will enable organizations to transform their existing messy data lakes into clean Delta Lakes with high quality data, thereby accelerating their data and machine learning initiatives.
Unreliable data in data lakes prevents organizations from deriving business insights quickly and significantly slows down strategic machine learning initiatives. Data reliability challenges derive from failed writes, schema mismatches and data inconsistencies when mixing batch and streaming data and supporting multiple writers and readers simultaneously.
“Today, nearly every company has a data lake they are trying to gain insights from, but data lakes have proven to lack data reliability. Delta Lake has eliminated these challenges for hundreds of enterprises. By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’,” said Ali Ghodsi, cofounder and CEO at Databricks.
How it helps improve data lake reliability?
Databricks explains –
- Delta Lake delivers reliability by managing transactions across streaming and batch data and across multiple simultaneous readers and writers.
- Delta Lakes can be easily plugged into any Apache Spark job as a data source, enabling organizations to gain data reliability with minimal change to their data architectures.
- With Delta Lake, organizations will no longer need to spend resources building complex and fragile data pipelines to move data across systems. Instead, developers can have hundreds of applications reliably upload and query data at scale.
- With Delta Lake, developers will be able to undertake local development and debugging on their laptops to quickly develop data pipelines.
- They will be able to access earlier versions of their data for audits, rollbacks or reproducing machine learning experiments.
- They will also be able to convert their existing Parquet, a commonly used data format to store large datasets, files to Delta Lakes in-place, thus avoiding the need for substantial reading and rewriting.
Delta Lake, which has long been a proprietary part of Databrick’s offering, is already deployed in production by companies like Viacom, Edmunds, Riot Games and McGraw Hill. The project can be found at delta.io and is under the permissive Apache 2.0 license.
“We’ve believed right from the onset that innovation happens in collaboration – not isolation. This belief led to the creation of the Spark project and MLflow. Delta Lake will foster a thriving community of developers collaborating to improve data lake reliability and accelerate machine learning initiatives,” added Ghodsi.
