The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining

Summary

The article presents a critical examination of Classifier-Based Quality Filtering (CQF) used in the pretraining of large language models (LLMs). It highlights the paradox that while CQF appears to enhance downstream task performance, it does not necessarily improve language modeling on high-quality datasets. The authors argue that CQF inadvertently filters out high-quality data, leading to unexpected results. By comparing CQF-trained models with those trained on synthetic data of varying quality, the study reveals significant differences in performance, challenging the conventional understanding of data quality in LLM training.

Key Learnings

• 1CQF can improve downstream task performance but may not enhance overall language modeling quality.
• 2The filtering process of CQF can inadvertently exclude high-quality data, impacting model training.
• 3Synthetic data experiments reveal stark differences in model behavior compared to CQF-trained models.
• 4The findings question the effectiveness of CQF as a reliable measure of data quality in LLM pretraining.

Who Should Read This

Senior Machine Learning Engineers evaluating data quality strategies for large language model training

Test Your Knowledge

Topics