I want to learn about GPUs and finetuning open source LLMs

Welcome! I am excited to help you dive into the world of training and customizing your own artificial intelligence models. Before we look at the hardware or the math, I would love to know a bit about your background so we can tailor this session perfectly to you.

To help me customize our journey, what is your current experience level with machine learning or GPUs? Also, do you already have specific hardware in mind, or are you looking at cloud solutions?

I have some ML experience and want to use my local GPU.

It is awesome that you have some machine learning experience and plan to use your local hardware! Local fine-tuning gives you total control over your data and saves you from paying ongoing cloud rental fees.

Your path

Hardware Requirements and Considerations

The Fine-Tuning Workflow

Optimization Techniques

The Real Value of Fine-Tuning

Before we get into the gears of your GPU, let's look at why developers choose to fine-tune open-source models rather than just sending API calls to proprietary giants. When you use a base model out of the box, it is a generalist. Fine-tuning is the process of taking that pre-trained model and training it further on a smaller, domain-specific dataset. This allows you to achieve three major advantages:

Data Privacy: Because you are training and running the model on your local GPU, your proprietary or sensitive data never leaves your machine. This is crucial for medical, legal, or proprietary business data.
Customization: You can teach a model a highly specific style, API formatting, or private terminology that it could never learn through simple prompting.
Lower Infrastructure Costs: By fine-tuning a smaller model (like a 7-billion parameter model) to do one specific job incredibly well, you can bypass the need for massive, expensive cloud clusters.

Continue to how GPUs process this data

Let's crack open the GPU to see exactly how it processes this massive amount of data. To understand why we need a GPU rather than a standard CPU, think of a as a brilliant, world-class mathematician. It can solve incredibly complex equations one after another, but it can only work on one or two calculations at a time. A GPU, on the other hand, is like an army of millions of middle-schoolers who only know basic addition. They can't solve advanced calculus, but if you give them a massive sheet of simple math problems to solve all at once, they will finish the entire page in a single second. Because deep learning is just billions of tiny, simple math equations, this massive parallel processing is exactly what we need.

Y = W \cdot X + B

When you fine-tune an LLM, the GPU has to hold the entire base model, your training data, and a lot of temporary math calculations in its ultra-fast onboard memory, known as . If your model is too big for your VRAM, the training process will instantly crash with an "Out of Memory" error. This is why researchers invented clever shortcuts like Parameter-Efficient Fine-Tuning, or PEFT. Instead of changing every single weight in the model during training, we only tweak a tiny fraction of them.

Why can't we just use regular system RAM instead of VRAM?

That is one of the most common questions when starting out, and the short answer comes down to a massive physical speed limit: bandwidth.

While your system RAM is reasonably fast for everyday computer tasks, its data pathway is a major bottleneck for AI. typically transfers data to the CPU at around 50 to 100 GB/s.

In contrast, modern sits directly on the graphics card right next to the GPU cores, allowing it to transfer data at an astronomical 1,000 GB/s or more.

A side-by-side technical layout comparing the direct high-bandwidth VRAM bus against a narrower system RAM bus connection.

Because fine-tuning requires constantly moving billions of model weights back and forth for every single training step, trying to use regular system RAM would make the training take weeks instead of hours. Since local VRAM is so precious and limited, researchers had to get creative. This brings us to Parameter-Efficient Fine-Tuning, or , which lets us train models without needing massive data-center GPUs.