Asynchronous Verified Semantic Caching for Tiered LLM Architectures

Summary

The article introduces 'Krites', an innovative asynchronous caching policy designed for large language models (LLMs) that enhances semantic caching efficiency without compromising critical path latency. Traditional caching methods often rely on a single embedding similarity threshold, leading to a tradeoff between conservative and aggressive thresholds that can either miss reuse opportunities or risk incorrect responses. Krites addresses this by asynchronously invoking an LLM judge to verify static responses that fall just below the threshold, promoting verified matches into a dynamic cache. This approach significantly increases the fraction of requests served with curated static answers, achieving up to 3.9 times more effective utilization of static responses in conversational and search workloads, while maintaining latency levels.

Key Learnings

• 1Krites enhances static cache coverage by leveraging LLM verification without altering serving decisions.
• 2The asynchronous nature of Krites allows for more effective reuse of static responses, reducing inference costs.
• 3The design of Krites mitigates the tradeoff inherent in static-dynamic caching architectures by dynamically validating responses.
• 4Performance improvements of Krites are demonstrated through trace-driven simulations, showcasing its practical benefits in real-world applications.

Who Should Read This

Senior Machine Learning Engineers implementing large language models in production environments seeking to optimize inference costs and latency.

Test Your Knowledge

Topics