Before a computer can understand and execute your code, it needs to learn how to read it. This is the job of a compiler, and its first step is lexical analysis. Think of it like reading a sentence. Your brain doesn't see a meaningless string of letters. Instead, it instantly groups letters into words and recognizes punctuation. A compiler does something similar with source code.

The lexical analyzer, often called a scanner or lexer, scans the raw source code as a stream of characters. Its primary goal is to break this stream down into a sequence of meaningful units called tokens. This process simplifies the next stages of compilation, which no longer have to deal with individual characters and can instead work with a structured list of these tokens.

Let's break down the core components of this process.

Consider this simple line of JavaScript code. To us, it's a variable declaration. To the lexical analyzer, it's just a sequence of characters to be processed.

The scanner reads this line from left to right, character by character. It groups them into lexemes and generates a corresponding stream of tokens. Notice how comments and whitespace are handled—they are typically identified and discarded because they aren't necessary for the program's logic.

As the scanner finds lexemes that represent identifiers—like variable names, function names, or class names—it needs a place to store information about them. This is the role of the symbol table.

The symbol table is a data structure that keeps a record of each identifier used in the source code. For each identifier, it stores important details, or attributes. During lexical analysis, the main attribute stored is simply the identifier's name (e.g., price). Later phases of compilation will add more information to the symbol table, such as the identifier's type (e.g., number, string), its scope (where it's accessible), and its memory location.

By creating this table early on, the compiler ensures that it has a single, consistent reference for every identifier in the program.

What happens if the scanner encounters a sequence of characters that doesn't match any known pattern? This is a lexical error. For example, a stray character like @ or $ might not be part of the language's defined syntax.

When the scanner sees the # in the variable name, it can't form a valid token. A good lexical analyzer won't just crash. It will report the error, usually with a line number and character position, to help the programmer find and fix it. It might then attempt to recover by skipping characters until it finds a point where it can resume tokenizing, allowing it to find other potential errors in the same file.

Let's check your understanding of these core concepts.

Lexical analysis is a fundamental first pass that transforms a chaotic stream of characters into an orderly sequence of tokens. This crucial step allows the rest of the compiler to focus on structure and meaning, rather than the raw text of the code.