Introduction to PyTorch

What is PyTorch?

PyTorch is an open-source machine learning library that helps developers build and train artificial intelligence models. Think of it as a workshop full of powerful tools and building blocks specifically designed for deep learning. It was developed by Meta AI and is now widely used in both academia and industry for projects ranging from computer vision to natural language processing.

PyTorch is an open-source deep learning framework designed to simplify the process of building neural networks and machine learning models.

At its heart, PyTorch is designed to be flexible, intuitive, and fast. It combines the ease of use of a scripting language like Python with the performance of low-level compiled languages. This makes it a popular choice for researchers who need to rapidly prototype new ideas and for engineers deploying large-scale models.

Getting Started

Before you can use PyTorch, you need to install it. The exact installation command depends on your operating system and whether you have a CUDA-enabled GPU. The best way to get the correct command is to visit the official PyTorch website and use its configuration tool.

For a standard CPU-only installation using pip, the command typically looks like this:

pip3 install torch torchvision torchaudio

Once installed, you can import it into your Python scripts just like any other library.

import torch

# Check the installed version
print(torch.__version__)

The Core: Tensors

The fundamental data structure in PyTorch is the tensor. If you've ever used NumPy arrays, tensors will feel very familiar. They are essentially multi-dimensional arrays that can run on GPUs for accelerated computing.

A tensor can be a number (a 0-dimensional scalar), a list of numbers (a 1-dimensional vector), a table of numbers (a 2-dimensional matrix), or a higher-dimensional structure.

You can create tensors in several ways. For example, you can create one directly from a Python list.

data = [[1, 2], [3, 4]]
x_data = torch.tensor(data)

print(x_data)

You can also create tensors with specific shapes and values, like a matrix of all zeros or one filled with random numbers.

# A 3x4 tensor of zeros
zeros_tensor = torch.zeros(3, 4)
print(zeros_tensor)

# A 2x3 tensor of random numbers between 0 and 1
rand_tensor = torch.rand(2, 3)
print(rand_tensor)

Tensors have attributes that describe their shape, data type, and the device (CPU or GPU) where they are stored.

tensor = torch.rand(3, 4)

print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")

Operations on tensors are straightforward and often have a syntax similar to NumPy. You can perform standard arithmetic, indexing, and slicing.

tensor = torch.ones(4, 4)

# Indexing and slicing
tensor[:,1] = 0
print(tensor)

# Joining tensors
t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)

# Element-wise multiplication
print(f"Tensor multiplied by itself: \n {tensor.mul(tensor)} \n")

# Matrix multiplication
print(f"Tensor matrix multiplied by its transpose: \n {tensor.matmul(tensor.T)}")

Automatic Differentiation

One of PyTorch's most powerful features is torch.autograd, its built-in automatic differentiation engine. When training a neural network, you need to calculate the gradient of the model's error with respect to its parameters. This process, called backpropagation, can be complex to implement manually. Autograd handles it for you.

Autograd tracks all operations on tensors. When you finish your computation, you can call .backward() to have all the gradients computed automatically.

To enable autograd to track a tensor's history, you set its requires_grad attribute to True.

x = torch.ones(2, 2, requires_grad=True)
print(x)

y = x + 2
print(y)

z = y * y * 3
out = z.mean()
print(z, out)

Let's break this down. We started with x. We then created y and z from it. Because x has requires_grad=True, PyTorch built a computational graph that remembers how out was created.

Mathematically, out can be expressed as a function of x. If the elements of x are all $x_i = 1$ , then $y_i = x_i + 2 = 3$ , and $z_i = 3y_i^2 = 27$ . Finally, $out = \frac{1}{4} \sum z_i = \frac{1}{4}(4 \times 27) = 27$ .

Now, let's calculate the gradient of out with respect to x. This is written as $\frac{\partial out}{\partial x}$ . We use the chain rule:

\frac{\partial out}{\partial x_i} = \frac{\partial out}{\partial z_i} \cdot \frac{\partial z_i}{\partial y_i} \cdot \frac{\partial y_i}{\partial x_i}

The individual derivatives are: $\frac{\partial out}{\partial z_i} = \frac{1}{4}$ $\frac{\partial z_i}{\partial y_i} = 6y_i = 6(3) = 18$ $\frac{\partial y_i}{\partial x_i} = 1$

Putting it all together, we get $\frac{\partial out}{\partial x_i} = \frac{1}{4} \cdot 18 \cdot 1 = 4.5$ . So, the gradient for every element in x should be 4.5.

Let's verify this with PyTorch by calling .backward() on out.

# Compute the gradients
out.backward()

# Print the gradient of out with respect to x
print(x.grad)

As expected, x.grad gives us a tensor where each element is 4.5. This automatic calculation is the backbone of training deep learning models. It allows the model to adjust its parameters to minimize error, step by step.

PyTorch Deep Learning Fundamentals