At its Knowledge + AI Summit right this moment, Databricks introduced that it’s open sourcing Unity Catalog, the metadata catalog that governs how customers and compute engines can entry knowledge. Coming off of final week’s information round Apache Iceberg, the transfer marks an essential shift for Databricks because it seeks to take care of momentum as clients more and more demand open lakehouse platforms.
Databricks unveiled Unity Catalog again in 2021 as a option to govern and safe entry to knowledge saved in Delta, the desk format that Databricks created in 2017 because the linchpin of its lakehouse technique. It has remained a proprietary product at Databricks since.
However lately, a competing desk format, Apache Iceberg, has gained momentum within the massive knowledge ecosystem. Databricks addressed Iceberg’s rise final week with the deliberate acquisition of Tabular, the lakehouse firm based by Iceberg’s creator. Databricks’ technique is to progressively transfer the Iceberg and Delta specs nearer collectively over time, thereby eliminating the variations between them.
That left the standard metadata catalog because the final piece standing between clients and their dream of a very open knowledge lakehouse. Databricks’ rival, Snowflake, addressed the potential lock-in of the metadata catalog final week with the launch of Polaris, which is predicated on Iceberg’s REST-based API. The corporate tells Datanami that it plans to donate the Polaris venture to open supply, seemingly the Apache Software program Basis, inside 90 days.
That left the still-proprietary Unity Catalog because the odd-man out on the metadata catalog layer, simply as a brand new period of open lakehouses abruptly arrives. To handle that strategic shift available in the market, Databricks determined to open supply Unity Catalog.
The transfer creates the “USB” for knowledge entry, Databricks CEO Ali Ghodsi mentioned throughout his keynote deal with at Databricks’ Knowledge + AI Summit in San Francisco.
“All of the silos that you just had earlier than, they’ll simply entry one copy of the info that’s in a standardized USB format underneath your possession,” Ghodsi mentioned. “It goes by one governance layer that’s simply standardized–that’s Unity Catalog–for your entire knowledge.”
Unity Catalog beforehand supported Delta and Iceberg, along with Apache Hudi, one other open desk format, through Databricks’ Delta Lake UniForm format. The truth is, Unity Catalog additionally helps Iceberg’s REST-based API, Ghodsi identified.
“We principally standardized the info layer and the safety layer so that you just personal your knowledge and the whole lot goes by these open interfaces,” he mentioned. “And I feel that’s going to be superior for the group, for everyone in right here. As a result of we simply have far more use instances. We’re going to have the ability to do way more innovation, and we’ll simply increase this marketplace for everyone concerned.”
Databricks CEO Ali Ghodsi introduced the open sourcing of Unity Catalog at Knowledge + AI Summit, June 12, 2024
Databricks clients applauded the transfer, together with AT&T and Nasdaq.
“With the announcement of Unity Catalog’s open sourcing, we’re inspired by Databricks’ step to make lakehouse governance and metadata administration attainable by open requirements,” mentioned Matt Dugan, AT&T’s vp for knowledge platforms. “The pliability to make the most of interoperable instruments with our knowledge and AI property, with constant governance, is core to the AT&T knowledge platform technique.”
“Databricks’ determination to open supply Unity Catalog offers an answer that helps eradicate knowledge silos and we stay up for additional scaling our platform, enhancing our governance, and modernizing our knowledge functions as we proceed to ship for our purchasers,” mentioned Lenny Rosenfeld, Nasdaq’s vp of capital entry platforms.
It’s not clear what open supply basis Databricks will select for Unity Catalog OSS, nor what the timeline will probably be. Beforehand, Databricks has chosen The Linux Basis to open supply numerous internally developed merchandise, together with Delta and MLFlow.
Unity Catalog will probably be posted to Github on Thursday throughout Databricks’ CTO Matei Zaharai keynote at Knowledge + AI Summit, the corporate mentioned.
Associated Gadgets:
All Eyes on Databricks as Knowledge + AI Summit Kicks Off
Databricks Nabs Iceberg-Maker Tabular to Spawn Desk Uniformity
Snowflake Embraces Open Knowledge with Polaris Catalog