To effectively govern data, you need a plan that connects where your data lives and is processed with where your business rules and definitions are managed. This is where the integration architecture between Databricks and Collibra comes in. Think of it as the blueprint for a bridge connecting two essential cities: Databricks, the hub for large-scale data engineering and AI, and Collibra, the capital for data governance and intelligence.

Databricks provides the powerful engine for processing and analysing data, while Collibra offers the framework for understanding, trusting, and controlling that data across the entire organisation. The architecture isn't just about making them talk; it's about making them collaborate intelligently, ensuring that technical data assets are enriched with business context and that governance policies are applied where the data is actually used.

A successful integration architecture relies on a few key components, each with a distinct role.

The Collibra Data Intelligence Cloud is the enterprise-wide system of record for governance. While Unity Catalog knows about the tables and jobs inside Databricks, Collibra understands the broader business context. It holds the business glossary, data policies, ownership details, and quality rules that apply across all company systems, not just Databricks.

The bridge between them is the Integration Layer. This is typically an automated connector or a set of APIs that handles the communication. Its job is to synchronise metadata between the two platforms, ensuring that both systems have a consistent and up-to-date view of the data landscape.

The interaction is a continuous, two-way synchronisation of metadata. It's a dialogue where each platform shares what it knows best, creating a comprehensive view of the organisation's data.

From Databricks to Collibra: Unity Catalog is the source of ground truth for technical assets. The integration layer regularly pulls information from it, such as:

• New or updated schemas, tables, and columns.
• Detailed column-level lineage generated from data processing jobs.
• Usage statistics and operational metadata.

This information populates Collibra, giving data stewards and business users a clear view of the physical data assets that exist and how they are being created and used.

From Collibra to Databricks: Collibra enriches the technical assets in Databricks with crucial business context. The integration pushes information back into Unity Catalog, such as:

• Business definitions and descriptions for tables and columns.
• Data classification tags (e.g., PII, Sensitive, Confidential).
• Information about data owners and stewards.

This enrichment means that a data scientist working in a Databricks notebook can see not just a table named cust_txn_q3, but also its official business definition, its owner, and a warning that it contains sensitive personal information, all without leaving their environment.

A robust integration architecture is built on a few core principles that ensure it remains scalable, maintainable, and effective.

Automation is Essential: Manual metadata updates are slow and prone to error. The architecture should be designed for automated, scheduled synchronisation. This ensures that the governance information in both systems is always current, reflecting the reality of your data landscape as it changes.

Bidirectional Synchronisation: The flow of information must go both ways. A one-way push from Databricks to Collibra only provides a technical catalogue. True data intelligence comes from pushing business context back into the tools where data professionals do their work.

Focus on Lineage: One of the most powerful features of the Unity Catalog is its ability to automatically capture column-level lineage. The architecture must prioritise syncing this lineage data to Collibra. This allows the business to trace data from its source all the way to a dashboard or report, which is critical for trust, impact analysis, and regulatory compliance.

Leverage Native Capabilities: A strong design leverages the native features of each platform. It uses Unity Catalog as the definitive governance point within Databricks and Collibra as the enterprise system of record. This avoids creating redundant or conflicting governance processes.

By following these principles, you can build an integration architecture that not only connects two platforms but also creates a foundation for a truly data-driven organisation.