AppleBeyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
Read Full ArticleSummary
The paper explores the limitations of using a single extractor for HTML-to-text conversion in the context of training large language models (LLMs). It highlights that relying on a fixed extractor can lead to suboptimal data coverage and utilization. The authors propose a method of combining multiple extractors to enhance token yield significantly, achieving up to a 71% increase in token yield while maintaining benchmark performance. Furthermore, they demonstrate that the choice of extractor can substantially impact the performance of downstream tasks, particularly for structured content like tables and code blocks, with notable performance variations observed in specific evaluations.
Key Learnings
- 1Using multiple extractors can significantly increase the token yield for LLM pretraining datasets.
- 2The choice of extractor affects the performance of models on structured content, with measurable differences in task outcomes.
- 3A single fixed extractor may not adequately capture the diversity of web content, leading to potential data loss.
- 4Combining extractors allows for a more comprehensive approach to data extraction, improving overall model performance.
Who Should Read This
Senior Machine Learning Engineers focusing on optimizing data preprocessing for large language models
Test Your Knowledge
What are the trade-offs between using a single extractor versus multiple extractors for HTML-to-text extraction?
How does the choice of extractor influence the performance of LLMs on specific downstream tasks?
What metrics can be used to evaluate the effectiveness of different extractors in this context?
In what scenarios might a single extractor still be preferable despite the potential benefits of multiple extractors?
How can the findings of this research inform future practices in dataset construction for LLMs?
Topics
More articles about Large Language Models
Explore Large Language Models engineering →LogSentinel: How Databricks uses Databricks for LLM-Powered PII Detection and Governance
The article presents LogSentinel, a sophisticated LLM-powered data classification system developed by Databricks for the automatic detection and classification of sensitive data, particularly...
From reactive to proactive: closing the phishing gap with LLMs
The article explores the transition from reactive to proactive email security measures through the integration of Large Language Models (LLMs). It highlights the limitations of traditional email...
How Cloudy translates complex security into human action
The article outlines how Cloudy, an LLM-powered explanation layer integrated into Cloudflare's security products, translates complex machine learning outputs into understandable guidance for security...
On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment
This paper addresses the critical issue of AI alignment in the context of large language models (LLMs), emphasizing the computational intractability of filtering mechanisms designed to prevent the...
Learning to Reason for Hallucination Span Detection
The paper presents a novel approach to hallucination span detection in large language models (LLMs) by incorporating explicit reasoning into the detection process. Traditional methods often treat...
More from Apple Engineering
View Apple engineering blogs →GenCtrl -- A Formal Controllability Toolkit for Generative Models
The article introduces GenCtrl, a formal controllability toolkit designed for generative models, addressing the critical need for fine-grained control in generative processes. It establishes a...
Flow Matching with Semidiscrete Couplings
The article presents a novel approach to flow matching using semidiscrete couplings, addressing limitations in traditional optimal transport methods. It highlights the inefficiencies of the OT flow...
Multi-Frequency Fusion for Robust Video Face Forgery Detection
The article presents a novel approach to video face forgery detection through a method termed Multi-Frequency Fusion. This technique utilizes a lightweight fusion of two handcrafted cues,...
On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment
This paper addresses the critical issue of AI alignment in the context of large language models (LLMs), emphasizing the computational intractability of filtering mechanisms designed to prevent the...
EMBridge: Enhancing Gesture Generalization from EMG Signals through Cross-Modal Representation Learning
The article presents EMBridge, a novel framework designed to enhance gesture generalization from electromyography (EMG) signals by leveraging cross-modal representation learning. By aligning EMG data...