No history yet

Databricks Environment Setup

The Databricks Workspace

Think of the Databricks workspace as your digital lab bench. It’s where you’ll find all your tools, projects, and data, neatly organized and ready to use. When you first log in, you'll see a navigation bar on the left side. This is your primary way to move around.

The key areas you'll use initially are:

  • Workspace: This is your file system. It’s where you create and organize notebooks and folders to keep your projects tidy.
  • Data: Here, you can browse databases, tables, and other data assets available to you.
  • Compute: This is where you manage the engine of your data operations: the clusters.

Computing Power with Clusters

Nothing happens in Databricks without a cluster. A cluster is essentially a group of computers that work together to run your code and process data. You can think of it as renting a supercomputer for a little while. Setting one up is your first step before you can run any analysis.

To create one, navigate to the Compute section and click Create Compute. You’ll see a few options, but here are the important ones to get started:

  1. Cluster name: Give your cluster a descriptive name, like dev-cluster-sara.
  2. Databricks Runtime version: This selects the software environment for your cluster. It includes Apache Spark and other libraries. Usually, sticking with a recent, stable version is a good idea.
  3. Worker type: This determines the power of the individual computers in your cluster. For learning, a general-purpose, smaller instance is fine.
  4. Workers: You can set a minimum and maximum number of worker machines. Databricks will automatically add or remove workers within this range based on your workload. This is called autoscaling.
  5. Terminate after...: This is a cost-saving feature. It automatically shuts down the cluster after a set period of inactivity, like 30 minutes. Always enable this.

Always set an auto-termination period for your clusters. An idle cluster still costs money, so shutting it down automatically is a critical best practice.

Once you've configured these settings, click Create Cluster. It will take a few minutes to spin up. You'll see its status change from 'Pending' to 'Running' when it's ready.

Managing Your Work

Your primary tool for working in Databricks is the notebook. A notebook is an interactive document where you can write and run code, add explanatory text, and display visualizations, all in one place. It's a powerful way to develop and share your data analysis.

Lesson image

To create a notebook, go to the Workspace section. You can create folders here to organize your projects. Inside a folder, click the + button and select Notebook. You'll be prompted to give it a name and choose a default language, like Python or SQL.

Inside the notebook, you'll see a block called a cell. You can write code in a cell and run it by pressing Shift + Enter. The output will appear directly below. You can add more cells to build out your analysis step-by-step. For example, in a Python notebook, you can run:

# This is a cell in a Databricks notebook
print("Hello, Databricks!")

Before you can run a cell, you must attach your notebook to a running cluster. You'll see a dropdown menu at the top of the notebook where you can select the cluster you just created. Once attached, you're ready to start processing data.

Quiz Questions 1/4

What is the primary function of the "Workspace" section in the Databricks navigation bar?

Quiz Questions 2/4

In Databricks, what must you do before you can run any code in a notebook?