Over-Searching in Search-Augmented Large Language Models

Summary

The article explores the phenomenon of over-searching in search-augmented large language models (LLMs), where unnecessary search tool invocations can degrade response quality and lead to computational inefficiencies. Through systematic evaluation, the authors identify key dimensions influencing over-searching, including query types and model categories. They introduce a new metric, Tokens Per Correctness (TPC), to quantify the performance-cost trade-off associated with search-augmented LLMs. The findings reveal that while search generally enhances accuracy for answerable queries, it can negatively impact abstention rates for unanswerable ones. The paper also discusses mitigation strategies and presents the OverSearchQA benchmark to advance research in this area.

Key Learnings

• 1Over-searching can lead to hallucinations and irrelevant context in LLM responses, highlighting the need for efficient retrieval mechanisms.
• 2The introduction of Tokens Per Correctness (TPC) provides a new way to evaluate the trade-offs between performance and computational cost in search-augmented LLMs.
• 3Mitigation strategies at both query and retrieval levels can significantly reduce the negative impacts of over-searching.
• 4The composition of retrieved evidence plays a crucial role in improving model performance, particularly in multi-turn conversations.
• 5Understanding the conditions under which search improves or harms model performance is essential for optimizing LLM applications.

Who Should Read This

Senior Machine Learning Engineers developing search-augmented large language models for knowledge-intensive applications.

Test Your Knowledge

Topics