Ever asked ChatGPT or Claude something like "Book a flight to my hometown" — and it somehow knew you meant Austin because you mentioned it three conversations ago?
Feels like memory, right? It's not.
The Illusion of Memory
Large Language Models are stateless by design. Every time you start a conversation, the model has zero recollection of anything you've ever said. It processes text within a fixed context window — a limited amount of tokens it can "see" at once. Once that window is full or the conversation ends, everything is gone.
So how do AI assistants appear to remember things?
The answer lies in a technique called Retrieval-Augmented Generation (RAG), powered by vector stores and embeddings.
From Words to Numbers: How Embeddings Work
Before an AI can "search" through your past conversations, it needs to convert text into something it can mathematically compare. That's where embeddings come in.
An embedding is a numerical representation of text — a high-dimensional vector that captures the meaning of a sentence, not just the words. Think of it as a coordinate in a space where similar meanings cluster together.
For example:
- "I live in Austin, Texas" →
[0.23, -0.87, 0.45, ...] - "My hometown is Austin" →
[0.21, -0.85, 0.47, ...] - "I enjoy cooking pasta" →
[-0.56, 0.12, 0.89, ...]
The first two vectors are close together in this high-dimensional space because they carry similar meaning. The third one is far away. This proximity is measured using cosine similarity — a mathematical way to determine how "aligned" two vectors are.
Vector Stores: The Memory Layer
Once text is converted to embeddings, it needs to be stored somewhere that allows fast similarity search. That's what vector databases do.
Unlike traditional databases that match exact values (SQL WHERE clauses), vector stores find the most similar entries to a given query. When you ask "Book a flight to my hometown," the system:
- Converts your query into an embedding
- Searches the vector store for the closest matching vectors
- Finds the entry where you said "I live in Austin, Texas"
- Passes that context to the LLM along with your current question
- The LLM generates a response as if it "remembered"
The popular vector stores in the ecosystem right now:
| Vector Store | Type | Best For |
|---|---|---|
| FAISS | Open-source, local | Fast prototyping, research, on-device search |
| Pinecone | Cloud-managed | Production apps needing scale without ops overhead |
| Chroma | Open-source, lightweight | Quick RAG prototypes, developer-friendly API |
| Weaviate | Open-source, cloud-optional | Hybrid search (vector + keyword), multi-modal data |
| Qdrant | Open-source, Rust-based | High-performance filtering with vector search |
| Milvus | Open-source, distributed | Billion-scale vector search, enterprise workloads |
Each has trade-offs around scalability, latency, hosting model, and filtering capabilities. For most developers starting out, Chroma or FAISS are the easiest on-ramps. For production systems, Pinecone or Weaviate tend to be the go-to choices.
RAG: Putting It All Together
Retrieval-Augmented Generation is the architecture pattern that ties embeddings, vector stores, and LLMs together. The flow looks like this:
User Query ↓ Embed the query → Vector representation ↓ Search vector store → Find relevant context ↓ Combine context + query → Augmented prompt ↓ Send to LLM → Generate informed response
This is how most "smart" AI assistants work today. The LLM itself isn't remembering anything — it's being fed relevant context right before it generates a response. The intelligence is in the retrieval, not the generation.
Why This Matters for Developers
Understanding this architecture changes how you think about building AI features:
Context is everything. The quality of your AI's responses depends heavily on what context gets retrieved. Bad embeddings or a poorly chunked knowledge base will produce bad answers, regardless of how powerful the LLM is.
Chunking strategy matters. When you store documents in a vector store, you split them into chunks. Too large and you lose precision. Too small and you lose context. Finding the right chunk size for your use case is one of the most impactful optimizations you can make.
Embeddings aren't one-size-fits-all. Different embedding models capture different nuances. OpenAI's text-embedding-3-large, Cohere's embed-v3, and open-source options like BGE or E5 each have strengths depending on your domain and language.
Hybrid search wins. Pure vector search sometimes misses exact matches (like product IDs or specific names). Combining vector similarity with traditional keyword search — what Weaviate and others call "hybrid search" — often produces the best results.
The Honest Truth
AI doesn't truly understand or remember. It's mathematical similarity in high-dimensional space, dressed up with good UX. And honestly? That's fine. The results are genuinely useful.
But knowing how the trick works makes you a better builder. You stop treating AI as magic and start engineering systems that reliably deliver the right context at the right time.
That's the real skill in the age of AI — not prompt engineering, but context engineering.
Building something with RAG or vector stores? I'd love to hear about it. Reach out on LinkedIn or X.