Vector Search and Vector Databases

The demo searched fifty documents and felt like magic. Then the real corpus arrived: two million chunks, several teams with different access rights, and a user searching for error code 4021. The semantic search returned articles about "troubleshooting in general" and skipped the one page that contained the exact code.

Vector search is easy to demo and surprisingly easy to get wrong in production. It is a data-retrieval problem with its own trade-offs, not a feature you switch on.

What a vector store actually does

You store each chunk's embedding next to its text and metadata. At query time you embed the question and ask for the k nearest vectors. If you haven't met embeddings yet, start with tokens, embeddings and context windows.

Comparing the query to every vector is exact but slow once you have millions. So most stores use an approximate nearest neighbour index that trades a little recall for a lot of speed:

• Flat (brute force): exact, simple, and perfectly good for small collections.
• HNSW: a layered graph you walk towards the query. Fast with high recall, at the price of memory and slower index builds.
• IVF: clusters the vectors and searches only the nearest clusters. Lighter, with a bit more recall loss.
• Quantisation: compresses vectors to save memory, again at some cost to accuracy.

"Approximate" means the true nearest neighbour is sometimes missed. Measure recall on your own data instead of trusting defaults.

Filter first, then rank

Real corpora aren't one big pool. Chunks belong to tenants, teams and sensitivity levels. If you rank first and filter afterwards, you can end up with fewer than k results, or worse, leak something you filtered too late. Apply the filter as part of the search:

import math
                      
                      def cosine(a, b):
                          dot = sum(x * y for x, y in zip(a, b))
                          return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))
                      
                      CHUNKS = [
                          {"id": "hr-1",  "vec": [0.9, 0.1], "groups": {"hr"},          "text": "Salary bands for 2026"},
                          {"id": "kb-7",  "vec": [0.8, 0.3], "groups": {"all"},         "text": "How to request leave"},
                          {"id": "kb-9",  "vec": [0.1, 0.9], "groups": {"all", "eng"},  "text": "Deploy checklist"},
                      ]
                      
                      def search(query_vec, user_groups, k=2):
                          allowed = [c for c in CHUNKS if c["groups"] & user_groups]      # filter FIRST, then rank
                          ranked = sorted(allowed, key=lambda c: -cosine(query_vec, c["vec"]))
                          return [(c["id"], round(cosine(query_vec, c["vec"]), 3)) for c in ranked[:k]]
                      
                      print(search([0.9, 0.2], {"all"}))        # the HR chunk never appears, even though it is the closest match
                      print(search([0.9, 0.2], {"all", "hr"}))  # an HR user sees it
                      

The HR salary chunk is the closest match to the query, and a normal user never sees it. Most vector databases support filtered search natively. Test it with edge cases, because filter-heavy queries are where approximate indexes often behave worst.

Words still matter: hybrid search

Embeddings capture meaning and blur exact strings. Error codes, product names, ids and rare jargon are often better served by plain keyword search (BM25 and friends). Run both and merge the results. Reciprocal rank fusion is a simple, popular way to combine ranked lists without having to calibrate their scores:

def rrf(rankings, k=60):
                          """Reciprocal rank fusion: merge several ranked lists into one."""
                          scores = {}
                          for ranking in rankings:
                              for rank, doc in enumerate(ranking, start=1):
                                  scores[doc] = scores.get(doc, 0) + 1 / (k + rank)
                          return sorted(scores, key=scores.get, reverse=True)
                      
                      keyword_hits = ["err-4021-guide", "release-notes", "faq"]      # exact-term search finds the error code
                      vector_hits = ["troubleshooting", "faq", "err-4021-guide"]     # semantic search finds the concept
                      
                      print(rrf([keyword_hits, vector_hits]))
                      

The page containing the exact code comes first because both searches liked it. A cross-encoder reranker on the merged top results is the next common upgrade: slower per item, but far better at judging relevance for the few candidates you keep.

The solution architect's view

Treat retrieval as a data platform with owners and service levels, not a library call.

Permissions belong in the index

Store access metadata on every chunk and filter on it inside the query, using the identity of the person asking. A model can't be trusted to withhold what you've shown it. Decide how group changes propagate: if someone loses access, how quickly does the index reflect that?

Choose the store by requirements

A vector extension on a database you already run is often enough to start, with simpler operations, one backup story and transactional consistency with your other data. A dedicated vector service earns its place when scale, filtered-search performance, multi-tenancy or managed operations justify another system. Decide with a test on your data and filters, not with a feature table.

Plan the lifecycle

• Freshness: how fast do edits and deletions reach the index, and how do you prove a deleted document is gone?
• Embedding model changes: vectors from different models aren't compatible. Version them, and plan a re-embed with a dual index before cut-over.
• Capacity: index memory grows with vector count and dimensions. Estimate it before choosing a tier.

Measure retrieval separately

Keep labelled questions and track whether the right chunk lands in the top results. When the system gives a bad answer, this tells you whether to fix search or the prompt. The evaluation post shows how.

Questions for the design review

1. Where is access control enforced, and how fast do permission changes take effect?
2. What recall does the index deliver on our data, and how did we measure it?
3. Do we need keyword search alongside vectors for ids and exact terms?
4. How would we migrate to a different embedding model?
5. What are the memory, cost and latency at ten times today's data?