Google BigQuery gets metadata service with Iceberg support

Google Cloud is adding a new metadata service that is compatible with Apache Iceberg to its managed data analytics service BigQuery in order to help enterprises cut down complexities around metadata management.

Named BigQuery metastore, the fully managed unified metadata service will provide processing engine interoperability while enabling governance, Google principal engineer Yuri Volobuev and senior product manager Vinod Ramachandran wrote in a blog post.

On the governance front, metastore will enable automated cataloging and universal search, business metadata, data profiling, data quality, fine-grained access controls, data masking, sharing, data lineage, and audit logging.

Unlike traditional metastores and other metadata management systems that are tightly coupled with data processing engines, metastore will work with multiple engines, such as BigQuery, Apache Spark, Apache Hive, Apache Flink, and the Iceberg table format.

Google said that if an enterprise is using multiple processing engines for analytics, they typically have to maintain multiple copies of their data and metadata in respective metastores and also recreate table definitions while switching query engines.

“You [enterprise users] also have to build pipelines to keep table definitions synchronized across different metastores. This fragmentation can result in stale metadata, lack of visibility into data lineage, security, and access challenges, and a subpar user experience,” the Google blog noted.

In contrast, BigQuery metastore will allow analytics engines to query one copy of the data with a single schema, whether the data is stored in BigQuery storage tables, BigQuery tables for Apache Iceberg, or BigLake external tables.

“The unification of metadata across engines makes it easier to discover and use data, supporting self-service BI and ML tools to drive innovation while maintaining data governance,” the Google employees wrote.

Additionally, BigQuery metastore is serverless with no setup or configuration required to scale workloads, the company said, adding that the no-operations environment also reduces the total cost of ownership for enterprises.

Last year, Google added its generative AI-based chatbot Gemini to BigQuery in order to ease several data-related tasks for enterprise professionals, it said Thursday.

Gemini inside BigQuery was expected to aid with code generation, code completion, code explanation (SQL, Python), help with data canvas, and provide partitioning and clustering recommendations.