Becoming a Forward Deployed Engineer at Snorkel AI
Data Centric AI Fundamentals
The Labeling Bottleneck
For a long time, the standard approach to building AI systems was model-centric. You'd get a big, fixed dataset and spend all your time tweaking the model architecture, tuning hyperparameters, and trying different algorithms to squeeze out better performance. The data was treated as the unchangeable starting point.
But in the real world, especially in large enterprises, the biggest bottleneck isn't the model—it's the data. Specifically, it's the process of getting high-quality labels for that data. Manually labeling thousands or millions of data points is slow, expensive, and often requires subject matter experts whose time is valuable. This manual process is the primary reason many AI projects stall.
Data-centric AI flips the script. Instead of holding the data fixed and iterating on the model, we hold the model fixed and iterate on improving the data. This shift often leads to more significant and faster gains in performance.
The key to this shift is finding a more efficient way to label data. Instead of labeling one data point at a time, what if we could label huge chunks of data at once by capturing the patterns and expertise of human labelers? This is the core idea behind weak supervision and programmatic labeling.
Programmatic Labeling
Instead of clicking "spam" or "not spam" on individual emails, imagine writing down the rules you use to make that decision. You might look for certain keywords, check the sender's domain, or notice unusual formatting. Each of these rules is a heuristic.
Programmatic labeling formalizes this process. We write Labeling Functions (LFs), which are simple functions that encode these heuristics. An LF takes a data point as input and either assigns a label or abstains if it's unsure. This allows us to apply domain expertise at scale, labeling millions of data points in the time it would take to manually label a few hundred.
/* Example LFs for classifying email as SPAM or NOT_SPAM */
// LF 1: Check for keywords
function contains_keywords(email) {
if (email.text contains "free money" or "act now") {
return SPAM;
} else {
return ABSTAIN; // Don't know
}
}
// LF 2: Check sender domain
function from_known_domain(email) {
if (email.sender ends with "@yourcompany.com") {
return NOT_SPAM;
} else {
return ABSTAIN;
}
}
// LF 3: Check for all caps subject
function all_caps_subject(email) {
if (email.subject is all uppercase) {
return SPAM;
} else {
return ABSTAIN;
}
}
These LFs are imperfect by design. They're noisy, they might have low accuracy, they'll overlap on some data points, and they'll often conflict. For example, an internal company email with an all-caps subject line like "URGENT ACTION REQUIRED" might be labeled SPAM by all_caps_subject but NOT_SPAM by from_known_domain. And for many emails, all three LFs might abstain.
This is the central challenge: how do we combine the outputs of these noisy, conflicting LFs into a single, high-quality training label for each data point?
The Generative Label Model
The magic happens in the generative label model. This is a statistical model that learns to intelligently combine the outputs from all the LFs. It doesn't need any ground-truth labeled data to do this. Instead, it learns by observing the patterns of agreement and disagreement among the LFs themselves.
The model figures out which LFs are more likely to be correct and how they might be correlated. For instance, if two different LFs that use keyword matching tend to agree with each other but disagree with an LF that checks the sender domain, the model can infer that the keyword LFs are correlated and might be less reliable than the domain-checking LF.
By modeling these relationships, the generative model can re-weight and combine the LF outputs to produce a single probabilistic label for every data point. This label represents the model's confidence that the data point belongs to a certain class (e.g., 95% SPAM, 5% NOT_SPAM). These probabilistic labels are then used to train a powerful, final machine learning model, like a deep learning classifier.
This data-centric loop—write LFs, generate probabilistic labels, train model, analyze errors, refine LFs—is a much faster and more scalable way to build AI systems than trying to perfect a model on a static, manually labeled dataset.
In the traditional model-centric approach to AI, what is considered the fixed, unchangeable starting point?
What is a 'Labeling Function' (LF) in the context of programmatic labeling?