The term “agent” in the context of “LLM agent” refers to autonomous AI systems that leverage LLMs’ abilities beyond text generation. The agent is responsible for performing specific tasks by understanding the task, making decisions, and interacting with the external environment. Some of them are:

• Task executions: They are based on the given instructions, such as scheduling a meeting or booking a flight ticket.
• Decision-making: Decision-making involves analyzing data to determine the best course of action based on the available information.
• Task Management: Agents remember previous actions, ensuring they follow all the multi-step instructions without losing track.
• Interaction with external Systems: Agents can link with external tools and functions to update the records,  retrieve required information, perform calculations, and execute code.
• Adaptability: Agents can adapt to changes or new information by adjusting their behavior in real-time.

For example, if I upload a corporate expense policy document and ask a question, can I ensure that it uses only this document and not policy documents found on the web that it happened to be trained on?

No. While careful prompting and techniques like RAG can encourage an AI model to prioritize a set of provided documents, standard LLMs cannot be forced to use only that content. The model still has access to patterns and facts it learned during training and may blend that knowledge into its response — especially if the training data included similar content.

No. LLMs can fabricate (hallucinate) citations or use real sources in inaccurate or misleading ways. Some LLM systems include post-processing steps to verify citations, but these checks are not always reliable or comprehensive. Always verify that a cited source actually exists and that its content genuinely supports the information in the response.

Modern LLMs like GPT-4.1 and Gemini 2.5 offer million-token context windows — enough to hold entire books. This naturally raises the question: If we can fit everything in, why bother using a subset?

While these extended context windows are powerful, including all documents in the prompt isn’t always a good idea. There are several reasons why RAG still matters.

First, RAG isn’t just about keeping the prompt short. It’s about selecting the most relevant parts of the documents. Overloading the context with too much or irrelevant information can hurt performance, and keeping the context and prompt relevant, concise, and accurate often leads to better answers.

Second, even though LLMs can accept long contexts, they don’t process all parts equally well. Research has shown that AI models tend to focus more on the beginning and end of a prompt and may miss important information in the middle.

Finally, longer prompts mean more tokens, which increases API costs and slows down responses. This matters in real-world applications where cost and speed are important.

In short, long context windows are useful, but they don’t make retrieval obsolete. RAG remains an important tool, especially when you care about accuracy, efficiency, or cost. The option to use RAG should still be evaluated based on the needs of your specific application.