This paper tackles a fundamental but often overlooked problem in modern AI pipelines: annotation errors and their cascading impact on model training, evaluation, and human-in-the-loop data cleaning systems.
We show that annotation error detection (AED) systems themselves break down under label noise, and that simple noise-aware training strategies can dramatically improve their effectiveness without changing model architectures or workflows.
Venue: HILDA @ SIGMOD 2025
Focus: Data quality, annotation error detection, noise-robust training
As AI systems scale, the assumption that “expert-labeled” datasets are clean no longer holds. Even widely used benchmarks contain substantial label noise, which undermines both training and evaluation reliability.
This work examines how label noise affects AED systems and identifies practical training interventions that restore their utility at scale.
Human-in-the-loop data cleaning pipelines typically follow this pattern:
However, this loop contains a hidden flaw:
If the model is trained on noisy labels, its ability to detect those very errors collapses.
This creates a paradox: AED systems rely on models that are most vulnerable to the noise they are supposed to detect.
This paper systematically investigates three questions:
Most modern AED systems share a common structure:
This study focuses on AED methods that depend on model training, which dominate current practice.
The experiments reveal a striking result:
Models trained with standard cross-entropy loss on noisy data suffer catastrophic AED performance collapse.
Across multiple NLP datasets:
In other words, training configurations that work well for prediction tasks fail completely for error detection.
Instead of proposing new AED algorithms, this paper takes a different approach; Fix the training dynamics. Two lightweight, widely applicable strategies are evaluated:
Robust Regularization
Constrains model outputs to prevent memorization of noisy labels by limiting extreme confidence.
Key effect: Reduces overfitting to incorrect annotations.
Robust Loss Design
Modifies the learning objective so that mislabeled examples exert less harmful influence during training.
Key effect: Stabilizes learning in the presence of conflicting or ambiguous labels.
🔹 Noise-Aware Training Works
Across datasets and AED methods:
🔹 Training Dynamics Matter More Than Architecture Traditional ML models sometimes outperform large neural models when:
This reinforces a core data-centric insight:
Better training beats bigger models in noisy environments.
🔹 Massive Reduction in Human Effort
With noise-aware training:
This work reframes annotation error detection as a training problem, not just a scoring problem:
This work demonstrates that annotation error detection fails silently when trained naively, but can be dramatically improved with simple, principled training adjustments. By aligning training strategies with the realities of noisy data, we can:
If you care about data quality, evaluation reliability, or human-in-the-loop AI, this work argues for one clear takeaway:
Clean models start with noise-aware training — not just better labels.
