Databricks Certified Data Engineer Associate Prep
Databricks Lakehouse Architecture
Beyond Warehouses and Lakes
For years, data teams faced a trade-off. They could use a data warehouse for reliable, structured data analytics, perfect for business intelligence. Or they could use a data lake for its cheap, flexible storage, ideal for raw, unstructured data and machine learning. Each had its strengths, but they created silos. Data had to be copied and moved between them, adding complexity and cost.
The data warehouse offered structure and performance, while the data lake offered scale and flexibility. But you couldn't get both in one system.
The Databricks Lakehouse Platform eliminates this trade-off. It's a single, unified system that combines the reliability and performance of a data warehouse with the low-cost, open storage of a data lake. It does this by adding a transactional storage layer on top of your existing cloud data lake.
This foundational layer is called an open-source technology that brings ACID transactions to data lakes. This means you can run concurrent read and write operations on your data without corruption. It also enforces data quality through schema validation and allows you to access previous versions of your data, a feature known as time travel.
The Databricks Architecture
Databricks operates on a unique architecture split into two main parts: the control plane and the data plane. Understanding this separation is key to understanding how Databricks provides both a managed service and data privacy.
Databricks serves as a cohesive data intelligence platform, seamlessly integrating with cloud storage and security within your cloud account.
The control plane is the part that Databricks manages in its own cloud account. It includes the web application, notebooks, job scheduling, and cluster management services. When you log into your Databricks Workspace, you are interacting with the control plane. It's the brains of the operation, coordinating all the tasks you want to perform.
The data plane is where your data lives and where the actual processing happens. This plane exists entirely within your cloud account (AWS, Azure, or GCP). Your data never leaves your environment. When you start a its resources are spun up inside your account, connect securely to the control plane, and process the data from your cloud storage. This separation ensures your data remains secure and under your control.
Navigating the Workspace
The Databricks Workspace is the web-based environment where you'll do all your work. It's designed to be collaborative and is tailored to different user roles, or "personas." You can switch between these personas to see a user interface optimized for your specific tasks.
| Persona | Primary User | Key Features |
|---|---|---|
| Data Science & Engineering | Data Engineers, Scientists | Notebooks, Workspace browser, Workflows, Compute management |
| SQL | Data Analysts, SQL Users | SQL editor, Query history, Dashboards, SQL Warehouses |
| Machine Learning | ML Engineers, Data Scientists | Experiments, Feature Store, Model Registry, Model Serving |
Regardless of the persona, the core components remain accessible. The Workspace allows you to organize assets like notebooks, libraries, and experiments into folders. You can manage data tables, configure compute resources (clusters and SQL warehouses), and schedule automated jobs through Workflows.
This unified environment is what truly enables the Lakehouse. A data analyst can query a table using pure SQL while a data scientist uses the exact same table to train a machine learning model in a notebook, with no data duplication. Both are working with the single source of truth stored in Delta Lake format in your data lake.
What fundamental problem did the traditional separation of data warehouses and data lakes create, which the Databricks Lakehouse Platform solves?
In the Databricks architecture, which part is managed by Databricks in its own cloud account and contains services like the web application, job scheduling, and cluster management?