YAML Pipeline Architecture

Structuring Your Pipeline

A well-structured YAML pipeline is easy to read, manage, and scale. Azure Pipelines uses a clear hierarchy to organize your workflow: stages, jobs, and steps. Think of it like a set of Russian nesting dolls.

Stages are the major divisions of your pipeline. They represent logical boundaries, like building the code, running tests, and deploying to production. Stages run sequentially by default.

Jobs are units of execution within a stage. Each job runs on an agent. You can run multiple jobs within a stage, either in sequence or in parallel.

Steps are the smallest building blocks. They are the individual tasks or scripts that run inside a job. A step can be a simple command-line script or a pre-built task from the Azure Pipelines marketplace.

# azure-pipelines.yml

stages:
- stage: Build
  jobs:
  - job: BuildJob
    steps:
    - script: echo 'Building the application...'
      displayName: 'Run build commands'

- stage: Test
  jobs:
  - job: TestJob
    steps:
    - script: echo 'Running tests...'
      displayName: 'Execute unit tests'

Proper indentation is critical in YAML. Use two spaces for each level of indentation. Incorrect spacing is the most common source of errors in pipeline files.

Controlling When Pipelines Run

You don't want every single commit to trigger a full build and deployment. Pipeline triggers give you precise control over when a pipeline should run. By default, a pipeline is triggered for every commit to any branch.

Pipeline as Code: Using YAML pipelines, your pipeline configuration is stored as code.

To be more specific, you can use a trigger clause to target certain branches. For example, you might only want to run the pipeline for changes to the main branch or any branch starting with features/.

trigger:
  branches:
    include:
    - main
    - features/*
    exclude:
    - features/experimental/*

Sometimes, changes to certain files, like documentation, don't require a new build. You can use paths to further refine your triggers. This tells the pipeline to run only if files in specific locations have changed.

# This trigger runs only when files in the 'src' directory change
trigger:
  branches:
    include:
    - main
  paths:
    include:
    - 'src/*'
    exclude:
    - 'docs/*'

Orchestrating the Workflow

Complex workflows require more than a simple sequence. You need to define dependencies to control the order of execution. For instance, you should never deploy code that hasn't passed its tests.

The dependsOn keyword lets you create these relationships between stages or jobs. A stage with a dependsOn clause will only start after the stages it depends on have completed successfully.

stages:
- stage: Build
  # ... build jobs and steps

- stage: Test
  dependsOn: Build # Test stage runs only after Build succeeds
  # ... test jobs and steps

- stage: DeployStaging
  dependsOn: Test # DeployStaging runs only after Test succeeds
  # ... deployment jobs and steps

You can also create more complex graphs. For example, you might run tests for different platforms in parallel and then deploy only after all of them have passed. To do this, you provide a list of dependencies to dependsOn.

- stage: Deploy
  dependsOn:
  - TestOnWindows
  - TestOnLinux
  jobs:
  # ... deployment job

For more fine-grained control, you can add a condition to a stage, job, or step. This allows you to run something only when specific criteria are met, like when a build is for a specific branch or when a previous stage fails.

Where Your Pipeline Runs

Your pipeline jobs don't run in a vacuum. They execute on machines called agents. Azure Pipelines provides a pool of Microsoft-hosted agents with pre-installed software for Windows, macOS, and Linux. You can also set up your own self-hosted agents for more control over the environment.

You specify which agent pool to use with the pool keyword. For Microsoft-hosted agents, you also specify the vmImage.

pool:
  vmImage: 'ubuntu-latest' # This applies to all jobs in the pipeline

stages:
- stage: Build
  jobs:
  - job: BuildLinux
    # This job inherits the pool from the pipeline level
    steps:
    - script: echo 'Building on Linux'

- stage: TestWindows
  jobs:
  - job: TestOnWindows
    pool:
      vmImage: 'windows-latest' # This overrides the pipeline-level pool
    steps:
    - script: echo 'Testing on Windows'

You can define a pool at the top level of your YAML file to set a default for all jobs, and then override it for specific stages or jobs as needed. This gives you the flexibility to build and test your application across multiple operating systems within a single pipeline.

Quiz Questions 1/6

What is the correct hierarchy of elements in an Azure Pipelines YAML file, from the largest division to the smallest building block?

Quiz Questions 2/6

You want to prevent your pipeline from running when changes are only made to documentation files (e.g., files in a /docs folder). Which YAML key should you use inside your trigger definition?

With these structural elements, you can design robust and efficient CI/CD workflows that are version-controlled right alongside your application code.

Mastering Azure DevOps CI Pipelines