By making Delta Lake open source, developers will be able to easily build reliable data lakes and turn them into ‘Delta Lakes’,” said Ali Ghodsi, co-founder and CEO at Databricks.

Delta Lake can be used as a drop-in replacement to access storage systems like HDFS. Data ingested into Spark through Delta Lake is stored in Parquet format in a cloud storage service of your choice.

The Delta Lake updates aim at helping data professionals create generative AI capabilities for their enterprise with foundation models from MosaicML and Hugging Face, among others.

Delta Lake 3.0 also includes Delta Kernel, an initiative that makes it easier to develop and maintain Delta connectors, and Liquid Clustering for cost-effective data clustering even as datasets ...