JAX Transformations

The Power of Transformation

JAX's real strength isn't just its NumPy-like API, but its ability to transform regular Python functions. These transformations allow you to speed up your code, calculate gradients, and vectorize operations without rewriting your logic from scratch. Let's look at the three most important ones: jit, grad, and vmap.

Just-in-Time Compilation

Python is flexible but can be slow because the interpreter executes code line by line. For numerical code that runs the same operations repeatedly, this overhead adds up. JAX solves this with Just-in-Time (JIT) compilation through its jax.jit transformation.

When you apply jax.jit to a function, JAX uses its XLA (Accelerated Linear Algebra) compiler to translate your Python code into highly optimized machine code that can run directly on a GPU or TPU. The function is compiled the first time it's called, and subsequent calls execute the fast, compiled version.

Let's see it in action. Here's a function that applies a few operations from jax.numpy.

import jax
import jax.numpy as jnp

def predict(W, b, inputs):
  return jnp.dot(inputs, W) + b

Now, let's create a JIT-compiled version of this function.

# Create a jit-compiled version of the function
fast_predict = jax.jit(predict)

While predict and fast_predict will return the same results, fast_predict will be significantly faster on subsequent runs, especially with large arrays. The first call has some overhead for compilation, but every call after that is pure speed. This is incredibly useful for speeding up the core loops in machine learning training.

Automatic Differentiation

Training machine learning models usually involves gradient descent, which requires calculating the derivative (or gradient) of a loss function. Doing this by hand is tedious and error-prone. JAX provides jax.grad to automate this process.

jax.grad takes a function and returns a new function that computes the gradient of the original. It's like asking JAX, "How does the output of this function change when I slightly change the input?"

Imagine we have a simple polynomial function:

f(x) = x^3 + 2x^2 - 3x + 1

The derivative, as you might remember from calculus, is:

f'(x) = 3x^2 + 4x - 3

Let's see how JAX can find this for us without any manual calculus. First, we define the function in code.

def f(x):
  return x**3 + 2*x**2 - 3*x + 1

Next, we use jax.grad to get the gradient function.

# Get a function that computes the gradient of f
df_dx = jax.grad(f)

# Calculate the gradient at x = 2.0
print(df_dx(2.0))

This will output 17.0. If we plug $x=2$ into our manually derived formula, we get $3(2)^2 + 4(2) - 3 = 12 + 8 - 3 = 17$ . It works perfectly! This is the magic that powers modern deep learning frameworks.

Effortless Vectorization

Often, you write a function that works on a single example but need to apply it to a whole batch of data. The typical way to do this is with a for loop, but loops are slow in Python. JAX's jax.vmap (for vectorizing map) lets you automatically convert a function designed for one data point into a function that can handle a batch of data points efficiently and in parallel.

Let's go back to our predict function, which processes a single input vector.

# Our original function
def predict(W, b, inputs):
  return jnp.dot(inputs, W) + b

# Let's create some dummy data
# A batch of 8 input vectors, each with 10 features
batched_inputs = jnp.ones((8, 10)) 

# Some dummy weights and bias
W = jnp.ones((10, 5))
b = jnp.ones(5)

If you tried to pass batched_inputs to predict, you'd get a dimension error because jnp.dot expects a single vector. With vmap, we can tell JAX to map the function over the first axis of our inputs.

# Tell vmap to map over the first axis of `inputs` (axis 0)
# W and b are not batched, so we mark them as None
batched_predict = jax.vmap(predict, in_axes=(None, None, 0))

# Now we can apply it to our whole batch
predictions = batched_predict(W, b, batched_inputs)

# The output will have a shape of (8, 5)
print(predictions.shape)

vmap automatically added a batch dimension to our operations, allowing us to process all eight inputs at once without writing a single loop.

Combining Transformations

The true power of JAX is that these transformations can be composed, or layered on top of each other. You can take a gradient of a vectorized function, or JIT-compile a function that computes gradients. This is how you build incredibly efficient and scalable models.

For example, let's say we want to compute the gradient of a loss function for an entire batch of data. We can first use vmap to batch our prediction and loss calculation, and then use grad to get the gradient of the total loss.

# 1. A function for the squared error loss on one example
def loss(W, b, inputs, targets):
  preds = predict(W, b, inputs)
  return jnp.sum((preds - targets)**2)

# 2. Vectorize the loss function to handle a batch
batched_loss = jax.vmap(loss, in_axes=(None, None, 0, 0))

# 3. Get the gradient of the *average* loss over the batch
@jax.jit
def update_step(W, b, inputs, targets):
  # Calculate the average loss over the batch
  avg_loss = lambda W, b: jnp.mean(batched_loss(W, b, inputs, targets))
  
  # Get the gradient with respect to W and b
  grads = jax.grad(avg_loss, argnums=(0, 1))(W, b)
  return grads

Here, we've combined vmap to handle batches, grad to calculate derivatives for training, and jit to compile the whole process into fast machine code. By composing these simple transformations, we've built a highly optimized training step for a machine learning model from basic Python functions.

Quiz Questions 1/5

What is the primary purpose of the jax.jit transformation?

Quiz Questions 2/5

You have a function predict(params, single_input) that works on a single data point. How would you use jax.vmap to make it process a batched_inputs array, where the batch dimension is the first axis?

Advanced JAX Techniques