> ## Content Index > Fetch the complete content index at: https://www.cognativ.com/llms.txt > Use this file to discover other available public pages before exploring further. # What Is a Vector Database? How It Works for LLMs and RAG - URL: https://www.cognativ.com/blog/what-is-a-vector-database/ - Published: 2026-11-04T08:46:19.000Z - Updated: 2026-11-04T08:46:48.000Z - Description: Vector databases find content by meaning. See how embeddings, HNSW indexes and hybrid search ground LLM answers in RAG, and what to weigh when picking one. - Author: Kevin Anderson - Tags: AI A vector database is a datastore built to store, index and search vector embeddings: long lists of numbers that represent the meaning of text, images, audio or code. Instead of matching exact keywords or values, a vector database answers the question "which stored items are most similar to this one?" That makes it the retrieval engine behind semantic search, recommendations and, above all, large language model (LLM) applications that use retrieval-augmented generation (RAG). LLMs know only what was in their training data, and they cannot see your contracts, tickets, manuals or product catalog. A vector database for LLM applications closes that gap. You convert your content into embeddings once, store them, and at question time retrieve the passages closest in meaning to the user's question so the model can answer from them. You can update the knowledge by updating the index, without retraining the model. **What this guide covers:** definitions, embeddings and similarity search, index internals, RAG, comparisons with other databases, types of vector stores, how to choose one, common mistakes and the state of the field in November 2026. ## What Is a Vector Database? A vector database manages high-dimensional vectors together with the metadata that describes them, such as a document ID, title, source, date, language or access permissions. Its core operation is nearest-neighbor search: given a query vector, return the k stored vectors that are closest to it under a chosen distance metric, optionally filtered by metadata. Traditional databases are optimized for exact lookups and joins, which do not help when the question is "find support articles that mean roughly the same thing as this complaint." Comparing a query against every stored vector works for small collections but becomes too slow as data grows, so vector databases rely on approximate nearest neighbor (ANN) indexes that check only a small, well-chosen fraction of the data while still returning most of the true closest matches. Production vector databases also add persistence, replication, backups, access control and SDKs. What sets them apart is that similarity, not equality, is the primary query. ## How Vector Embeddings and Similarity Search Work A vector is an ordered list of numbers. An embedding model, usually a transformer neural network, maps an input such as a sentence or an image into a fixed-length vector. Training places inputs with similar meaning close together. "Reset my password" and "I can't log in" end up as neighbors even though they share almost no words. ### Embedding dimensions Embedding models produce vectors with hundreds or thousands of dimensions. For example, the OpenAI embeddings guide lists 1,536 dimensions by default for text-embedding-3-small and 3,072 for text-embedding-3-large, with a parameter to shorten them. More dimensions can capture finer distinctions but cost more memory and compute. ### Similarity metrics The database ranks results with a distance or similarity function. Cosine similarity measures the angle between two vectors and ignores their length, which is why it is the common default for text. Dot product equals cosine when vectors are normalized to length 1, as OpenAI's embeddings are. Euclidean (L2) distance measures straight-line distance. Use the metric the embedding model was trained for, and build your index with the same metric. One rule matters more than any other: queries and documents must be embedded with the same model. Vectors from different models live in different spaces, and comparing them produces meaningless results. ![Raw text passes through an embedding model, becomes a vector of numbers, and lands near other content with a similar meaning.](https://storage.ghost.io/c/cc/dc/ccdca949-0fe4-4119-8990-70a44ad75060/content/images/2026/11/what-is-a-vector-database-1.png) An embedding model turns text into a vector, and similar meanings end up close together in vector space. ## How a Vector Database Works Under the Hood Most vector databases have three cooperating layers. The storage layer persists raw vectors and metadata, often with sharding and replication for scale. The indexing layer builds ANN structures so queries avoid scanning the whole collection. The query layer accepts a query vector, applies filters, runs the ANN search and returns the top-k matches with similarity scores. ### HNSW: graph-based indexing Hierarchical Navigable Small World (HNSW) is the most widely used index family today. Described by Malkov and Yashunin in [their HNSW paper](https://arxiv.org/abs/1603.09320?ref=cognativ.com), it links each vector to its near neighbors in a graph and stacks several graph layers. Sparse upper layers allow long jumps; dense lower layers allow fine-grained search. A query enters at the top and descends layer by layer toward the closest neighbors. HNSW offers fast, high-recall search, at the cost of more memory and slower index builds. ### IVF, quantization and other approaches Inverted file (IVF) indexes cluster vectors around centroids and search only the clusters nearest the query. Product quantization (PQ) and related scalar or binary quantization compress vectors into compact codes, cutting memory sharply with a modest loss of accuracy. These techniques are often combined, for example IVF with PQ. The Faiss library paper from Meta and co-authors describes these trade-offs in detail and notes that several vector database products build on Faiss or reimplement its algorithms. Every ANN index trades accuracy for speed. Recall measures how many true nearest neighbors the index returns, and index parameters let you move along that trade-off. Measure recall on your own data before tuning for speed. ![A query enters the sparse top layer of an HNSW graph, makes long jumps, descends to the dense lower layer and reaches its nearest neighbors.](https://storage.ghost.io/c/cc/dc/ccdca949-0fe4-4119-8990-70a44ad75060/content/images/2026/11/what-is-a-vector-database-2.png) HNSW searches start on a sparse upper layer for long jumps, then descend to denser layers to find the nearest neighbors. ## How a Vector Database Powers RAG Retrieval-augmented generation pairs a language model with an external, searchable memory. The idea was formalized by Lewis and colleagues in the 2020 paper [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401?ref=cognativ.com), which combined a generator with a dense vector index of Wikipedia and showed that swapping the index updates what the system knows. Today the vector database plays the role of that index. ### The RAG pipeline step by step Ingestion runs ahead of time. Documents are split into chunks, embedded and written to the vector database with metadata such as source and permissions. At query time, the user's question is embedded with the same model, the database returns the most similar chunks that pass the metadata filters, and those chunks are inserted into the prompt. The LLM then answers from the retrieved text and, ideally, cites it. This keeps answers grounded in your sources, lets you refresh knowledge by re-indexing instead of fine-tuning, and shows users where an answer came from. It reduces hallucinations but does not eliminate them: the model can still misread a passage or answer when retrieval returns nothing relevant. Teams building [custom LLM solutions](https://www.cognativ.com/ai/llm-solutions/) usually spend as much effort on chunking, filtering and evaluation as on the model itself. ### Hybrid search and reranking Pure vector search can miss exact terms such as product codes or names. Hybrid search runs a keyword method such as BM25 alongside vector search and merges the results, and a reranking model often rescores the top candidates before they reach the LLM. ![A user question is embedded and searched in the vector database, the most relevant chunks are retrieved, and the LLM writes an answer grounded in them.](https://storage.ghost.io/c/cc/dc/ccdca949-0fe4-4119-8990-70a44ad75060/content/images/2026/11/what-is-a-vector-database-3.png) At query time the vector database retrieves the closest chunks, and the LLM answers from that evidence. ## Vector Database vs. Traditional Databases A vector database does not replace your relational database or search engine; many systems use more than one. | Aspect | Relational database | Keyword search engine | Vector database | | ---------------- | ----------------------------- | ----------------------------------- | ---------------------------------------- | | Core query | Exact match, ranges, joins | Term match ranked by relevance | Nearest neighbors by meaning | | Data shape | Structured rows and columns | Text documents | Embeddings plus metadata | | Strength | Transactions and consistency | Precise terms and phrases | Synonyms, paraphrase, cross-modal search | | Weakness | No native similarity ranking | Misses meaning without shared words | Can miss exact codes and names | | Typical LLM role | System of record, permissions | Keyword half of hybrid search | Semantic retrieval for RAG | The lines are blurring: relational and search engines have added vector indexes, and vector databases have added keyword search. The real question is which engine handles your retrieval workload with the least operational overhead. ## Types of Vector Databases and Vector Stores In practice you will choose among four kinds of vector stores. ### Purpose-built vector databases Products such as Pinecone, Milvus, Qdrant, Weaviate and Chroma are designed around vector search, with managed or self-hosted deployment, metadata filtering and horizontal scaling. They suit large workloads where semantic retrieval is central. ### Vector extensions to existing databases PostgreSQL with the pgvector extension is the best-known example. Its documentation describes HNSW and IVFFlat indexes, half-precision, sparse and binary vector types, and, since version 0.8.0, iterative index scans that keep searching when filters remove too many results. The advantage is one system, one backup plan and transactional consistency between vectors and business data. ### Search engines with vector support Elasticsearch and OpenSearch combine mature keyword relevance with vector fields, which simplifies hybrid search if you already run them. ### Vector search libraries Libraries such as Faiss provide ANN indexes in code but no persistence, access control or replication. They fit research, batch jobs and embedded use. ![Four kinds of vector stores: purpose-built vector databases, vector extensions to existing databases, search engines with vector support, and vector search libraries.](https://storage.ghost.io/c/cc/dc/ccdca949-0fe4-4119-8990-70a44ad75060/content/images/2026/11/what-is-a-vector-database-4.png) Vector search is available as a dedicated database, an extension, a search engine feature or a code library. ## Vector Database Use Cases Semantic search is the foundation: people find documents by meaning rather than exact phrasing. Retrieval for LLMs builds on it, grounding assistants in policies and product data. Customer-facing assistants are among the most common deployments, and well-designed [AI chatbot development](https://www.cognativ.com/ai/agent-development/chatbots/) depends on retrieval that respects permissions and returns current content. Other use cases include recommendations, near-duplicate detection, image and multimodal search, code search, anomaly detection and long-term memory for AI agents. ## How to Choose a Vector Database Start from the workload, not the vendor list. ### Scale and growth Estimate how many vectors you will hold, at what dimension, and how fast they change. Modest collections often fit in a database you already run; very large ones push you toward sharding and quantization. ### Filtering and access control Enterprise RAG queries carry filters such as tenant, date and user permissions. Test filtered queries on realistic data, because heavy filtering can hurt recall in some ANN implementations. ### Hybrid search and reranking If users search for codes, names or exact phrases, you need keyword search alongside vectors. Check whether hybrid search is native or must be assembled from separate systems. ### Operations, cost and fit Weigh managed service against self-hosting, data residency, encryption and backups, and compare cost at your expected scale, including index memory and re-embedding when you change models. Confirm SDK and framework support, then run a proof of concept with your own documents and questions, measuring retrieval quality and latency together. ## Common Mistakes With Vector Databases - **Poor chunking.** Chunks that are too large dilute relevance; chunks that are too small lose context. Split on document structure and test sizes against real questions. - **Mixing embedding models.** Changing models without re-embedding the whole collection silently breaks search. - **Skipping metadata.** Without source and permission fields you cannot filter or cite. - **Vector search only.** Ignoring keyword retrieval misses exact identifiers that users type. - **No evaluation set.** Without a set of real questions and expected sources, you cannot tell whether a change improved retrieval or broke it. - **Stale indexes.** Deleted or outdated documents left in the index keep surfacing in answers. ## Vector Databases in 2026: Where Things Stand As of November 2026, vector search has become a standard feature rather than a separate category. Mainstream relational, document and search databases support vector types and ANN indexes, so many teams start with the database they already operate and adopt a dedicated vector database only when scale, latency or filtering needs justify it. Retrieval has matured too. Hybrid search, reranking and quantization are common defaults, and filtered search has improved, as pgvector's iterative scans illustrate. Longer LLM context windows have not removed the need for retrieval: sending whole document collections with every request is slow and expensive, and retrieval still decides which evidence the model sees. ## Frequently Asked Questions What is a vector database in simple terms? It is a database that stores data as lists of numbers representing meaning and finds the items most similar to a query. It lets software search by concept instead of exact words. Why do LLMs need a vector database? LLMs cannot see private or recent data. A vector database retrieves the most relevant passages from your content so the model can answer from them, which is the core of retrieval-augmented generation. Do you need a vector database for RAG? Not always. Small collections can work with an in-memory index, a vector extension such as pgvector, or keyword search. A dedicated vector database makes sense when scale, latency or filtering needs grow. What is the difference between a vector database and a relational database? A relational database answers exact queries on structured columns. A vector database ranks stored items by similarity to a query vector, which suits unstructured text, images and other content. Is PostgreSQL a vector database? Not by default, but with the pgvector extension PostgreSQL can store embeddings and run approximate nearest neighbor search with HNSW or IVFFlat indexes. What is HNSW in a vector database? HNSW (Hierarchical Navigable Small World) is a graph-based index that links vectors to their neighbors across several layers, so searches jump quickly toward the closest matches instead of scanning every vector. ## Conclusion: Getting Retrieval Right A vector database turns meaning into something software can search. Embeddings place similar content close together, ANN indexes such as HNSW find neighbors quickly, and in a RAG system the retrieved passages give an LLM the evidence it needs to answer about your own data. The database is only one part of good retrieval. Chunking, metadata, hybrid search, permissions and an honest evaluation set decide whether users get the right answer. Prove retrieval quality first, then choose the simplest store that meets your scale. If you are planning a RAG system, an internal knowledge assistant or a customer-facing chatbot and want retrieval designed in from the start, [talk to Cognativ about your AI project](https://www.cognativ.com/contact/).