Google Cloud adds Hive Metastore migration to Lakehouse

Google Cloud has expanded its Lakehouse runtime catalog to support migration from legacy Apache Hive Metastore environments. The service is designed to move existing Hive table metadata into a serverless catalog without copying the underlying data.

The update targets a long-standing part of big data infrastructure that has served as a central metadata registry for analytics systems built on Hadoop-era tools. Hive Metastore often sits behind Apache Spark, Presto and Hive deployments, helping those engines locate schemas, partitions and storage locations for Parquet and ORC data.

As data estates grow across multiple cloud analytics engines, including managed Spark services, BigQuery and Trino, older standalone metastore deployments can become harder to manage. In those setups, the metastore database can become a choke point when large Spark jobs request partition metadata at scale.

Migration path

The Lakehouse runtime catalog now supports both Iceberg REST Catalog and Hive Catalog, allowing organisations to register legacy Hive and Parquet tables alongside newer open table formats such as Apache Iceberg. It points table definitions to existing data stored in Google Cloud Storage, so users do not need to rewrite or duplicate the data itself.

The approach is aimed at companies that want to modernise metadata management without the cost and risk of moving petabytes of storage. Google said the migration process connects to an existing Hive Metastore, extracts external table definitions and partition maps, and registers them in the Lakehouse catalog.

Once registered, those tables can be discovered and queried across several engines that support the relevant interfaces, including Google Cloud Managed Service for Apache Spark, BigQuery and open-source engines that work with the Apache Iceberg REST specification.

Operational issues

Legacy Hive Metastore deployments typically rely on MySQL or PostgreSQL back ends to maintain schemas, partitions and file locations. As the number of tables and partitions grows, those relational databases can come under pressure from metadata-heavy tasks such as partition pruning and bulk listing.

In some production environments, complex Spark jobs can push metastore CPU utilisation to 100%, leading to query delays across clusters or out-of-memory failures, Google said. It added that maintaining high-availability databases, patching metastore daemons and tuning connection pools adds work for platform teams.

Governance is another issue. Older metastore designs emerged in perimeter-based Hadoop security models, while more recent cloud data estates often need fine-grained controls that apply across different compute engines. That can leave companies maintaining separate security policies in more than one control plane, according to Google.

Unified control

Google said the Lakehouse runtime catalog is integrated with Knowledge Catalog and Cloud IAM to apply table-level security and governance policies across supported engines. It also supports authorisation approaches including credential vending, allowing table access without direct access to files in the underlying Cloud Storage bucket.

The catalog is built on the Apache Iceberg REST catalog specification, which separates metadata discovery from the compute engine reading the data. That design reflects a wider industry shift towards open table formats and shared metadata layers that can be used by different query engines, rather than tying storage and governance to a single processing stack.

For customers still running self-managed Hive Metastore services on virtual machines backed by MySQL or PostgreSQL, the pitch is reduced infrastructure management. Google said the serverless model removes the need to operate dedicated metastore servers and is backed by its own infrastructure and Spanner.

Support for Cloud Storage dual-region and multi-region buckets is also included for failover scenarios. The offering fits into a broader effort by major cloud providers to persuade customers to replace self-managed Hadoop components with managed metadata, governance and analytics services.

The announcement also underlines the continued relevance of Hive Metastore in many large data estates despite the rise of newer catalog standards. Many organisations still depend on Hive-compatible tables and partition structures for production analytics, making migration tools that avoid rewriting data likely to draw close attention from data engineering teams.

Google said the objective is to help companies modernise those environments while keeping the same underlying files in place and making the tables available to a wider set of analytics engines and AI-driven query tools. It said the migration function "connects directly to your legacy Hive Metastore, extracts external table definitions and partition maps, and registers them cleanly into the serverless Lakehouse catalog", after which customers can use the data in Google Managed Spark, BigQuery and Conversational Analytics agents with Gemini.