#102: Lost In the middle - How language models use long contexts
Updated: Aug 23
Is more always better?
I’ve always assumed that if I want a chatbot to be more useful, giving it more relevant information would generally be better, especially if the context window (the maximum amount of text that an LLM can take in and reference while generating a response) is large enough to accommodate it.
“Lost in the Middle”
However, research has found that an LLM's ability to use relevant information could vary depending on where that information appeared within a long context.
Performance was often better when relevant information appeared near the beginning or end, and poorer when it was buried in the middle (Liu et al., 2024). In other words, information being within the context window does not guarantee that it will be used equally effectively.
More recent research has gone even further to say that LLM performance can deteriorate as context becomes longer, even when the model successfully retrieves the relevant information (Du et al., 2025).
What might this mean for a knowledge base?
Suppose I want students to use a chatbot to discuss a novel. My first thought would probably be to upload the whole book as my knowledge base. That seems sensible, since students can ask about any chapter, and the chatbot would have everything it needs to help them make connections across the text.
Knowing now that more context does not necessarily mean the LLM will make better use of that context, I might approach this a little differently.
If students are working on a particular section of the book, perhaps the chatbot only needs the relevant chapters and some context from earlier ones.
If I do want students to make connections across the whole book, I could curate the knowledge base differently (e.g. chapter summaries, key passages, character developments and major themes) rather than assume the full text will always be the best representation. I could also use AI to help summarise a particularly long resource first, review that summary, before including the refined version in the knowledge base.
Tl;dr More ≠ better
A bigger knowledge base isn't necessarily a better knowledge base. A smaller knowledge base isn’t inherently better as well; sometimes the whole source really is needed. What matters is whether the context provided is useful for the learning task.
When adding resources to a knowledge base, some questions to consider are:
What is the purpose? What do you want students to be able to ask or do with this chatbot?
What is relevant? Does the chatbot need every document, chapter or page to fulfil that purpose?
What can be removed? Is there duplicated, peripheral or irrelevant information that may not contribute to the task?
What can be condensed? Could a lengthy resource be represented through a carefully checked summary or selected extracts without losing its essence?
References
Bayram, C. (2025, November 7). Lost in the middle in LLMs. Medium. https://medium.com/@cenghanbayram35/lost-in-the-middle-in-llms-86e461dc7212
Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025). Context length alone hurts LLM performance despite perfect retrieval. Findings of the Association for Computational Linguistics: EMNLP 2025 (pp. 23281–23298). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.findings-emnlp.1264
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638
Comments