The study introduces System 2 Attention (S2A) as a solution to the problem of incorporating irrelevant information in Transformer-based Large Language Models (LLMs). S2A improves performance on tasks involving opinion or irrelevant information by regenerating the input context to only include relevant portions before attending to it.
Result is increased factuality and objectivity and decreased sycophancy.
In Langroid (the agent-oriented LLM framework from ex-CMU/UW-Madison researchers), we call it Relevance Extraction — given a passage and a query, use the LLM to extract only the portions relevant to the query. In a RAG pipeline where you optimistically retrieve top k chunks (to improve recall), the chunks could be large and hence contain irrelevant/distracting text. We concurrently do relevance extraction from these k chunks: https://github.com/langroid/langroid/blob/main/langroid/agen...
One thing often missed in this is the un-necessary cost (latency and token-cost) of parroting out verbatim text from context. In Langroid we use a numbering trick to mitigate this: pre-annotate the passage sentences with numbers, and ask the LLM to simply specify the relevant sentence-numbers. We have an elegant implementation of this in our RelevanceExtractorAgent using tools/function-calling.
Here's a post I wrote about comparing Langroid's method with LangChain's naive equivalent of relevance extraction called `LLMChainExtractor.compress` , and no surprise Langroid's methos is far faster and cheaper:
https://www.reddit.com/r/LocalLLaMA/comments/17k39es/relevan...
2 comments
[ 1.6 ms ] story [ 16.1 ms ] threadOne thing often missed in this is the un-necessary cost (latency and token-cost) of parroting out verbatim text from context. In Langroid we use a numbering trick to mitigate this: pre-annotate the passage sentences with numbers, and ask the LLM to simply specify the relevant sentence-numbers. We have an elegant implementation of this in our RelevanceExtractorAgent using tools/function-calling.
Here's a post I wrote about comparing Langroid's method with LangChain's naive equivalent of relevance extraction called `LLMChainExtractor.compress` , and no surprise Langroid's methos is far faster and cheaper: https://www.reddit.com/r/LocalLLaMA/comments/17k39es/relevan...