Blog · November 15, 2026

Does your app actually need a vector database?

Almost every tutorial on giving an AI feature access to your own content starts the same way: sign up for a vector database, embed everything, query the nearest neighbours, paste them into the prompt. It works, and it’s also a new vendor, a new bill, a new sync job and a new thing to keep consistent with your real data. For the amount of content most apps actually have, the database is the last piece you need, not the first. Here’s how to tell which side of the line you’re on.

What the database is really for

An embedding model turns a piece of text into a list of numbers, a vector, so that passages with similar meaning land close together. Retrieval is then a geometry problem: embed the user’s question, find the stored vectors nearest to it, and hand those passages to the model as context.

A vector database does two jobs on top of that. It stores the vectors, and it builds an approximate nearest-neighbour index (HNSW graphs are the common choice) so that a search over millions of vectors returns in milliseconds without comparing the query against every one of them. That index is the product. Storage alone is something any database or file already does. So the honest question isn’t “do I need retrieval?” but “do I have enough vectors that comparing against all of them is too slow?” For most apps, the answer is no by a wide margin.

Step one: count your corpus

Before embedding anything, measure what you’d be retrieving from. A common rule of thumb for English is about four characters, or three-quarters of a word, per token. Say your app’s help centre has 60 articles averaging 800 words. That’s 48,000 words, or roughly 64,000 tokens.

Current frontier models accept context windows in the hundreds of thousands of tokens. A corpus that size fits into the prompt whole, with room left over for the conversation. Then there’s nothing to retrieve and nothing to get wrong: the model sees every article, including the one a similarity search would have missed because the user phrased the question differently from how you wrote the answer.

The objection is cost, since you’re paying for 64,000 input tokens on every question. That’s exactly the case prompt caching was built for. Put the unchanging corpus at the start of the prompt, mark it cacheable, and repeat requests bill that prefix at a steep discount and process it faster. Check your provider’s current pricing and do the multiplication for your real traffic. For a help bot, a writing assistant grounded in a style guide, or anything working from a fixed reference document, “just send all of it” is often both the cheapest option and the most accurate.

Brute force is faster than you think

Suppose your content is too big for the prompt, or it’s per-user data like notes, journal entries or saved items, so you do need retrieval. You still may not need an index.

Do the arithmetic. Take 10,000 chunks embedded at 1,536 dimensions, which is the default output size of OpenAI’s text-embedding-3-small. Stored as 32-bit floats, that’s 10,000 × 1,536 × 4 bytes, or about 61 MB. Comparing a query against every chunk is about 15 million multiply-adds, which an ordinary server core gets through in a few milliseconds. A plain array in memory, a dot product in a loop and a sort of the top scores is exact nearest-neighbour search, with no approximation, no index to rebuild and no service to call.

For a single user’s data the numbers shrink further. Someone with 2,000 notes is a 3 MB matrix. You can keep it in the app’s local database as blobs, load it when the search screen opens, and search on the phone with no network at all. Many embedding models can also return shorter vectors, and storing them as 8-bit integers instead of floats cuts memory by a further four times, usually at a small cost in accuracy.

Where brute force stops working: somewhere in the hundreds of thousands to millions of vectors per search, or when one search has to cover every user’s data at once. That is the point an approximate index starts paying for itself, and not before.

Keyword search is often the better retriever

Embeddings are good at meaning and bad at exact strings. A user who types an error code, a product name, a SKU, an invoice number or a person’s name wants the record containing those exact characters, and semantic similarity can rank a passage about a similar error above the one about that error. App data is full of these tokens.

Plain full-text search handles them well and is probably already in your stack. SQLite ships FTS5 with a built-in bm25() ranking function, and Postgres has full-text search with tsvector. A sensible order is to start with keyword search, look at the queries it fails on, and add embeddings only for those, merging the two result lists (hybrid search). You often find the failures are a handful of synonyms that a small hand-written synonym table fixes for free.

When you genuinely do need one

Some apps really are on the far side of the line. You have millions of chunks in one shared corpus. Content changes constantly, and re-loading a big matrix on every write is wasteful. Searches need metadata filters (this workspace, this date range, these permissions) applied during the search, not after it. You need consistent low latency under heavy traffic.

Even then, look at what your existing infrastructure offers before adding a vendor. Postgres has the pgvector extension, with HNSW indexes, so the vectors sit next to the rows they describe and live inside the same transactions and backups. Firestore has built-in vector search for nearest-neighbour queries over documents. Cloudflare has Vectorize for Workers. A dedicated database makes sense once you’ve outgrown those, and you’ll know you have because you’ll have measurements showing it.

The parts that decide quality anyway

Whatever stores the vectors, three things decide whether the answers are any good, and none of them is the database.

Chunking. Split a help article mid-step and neither half answers the question. Chunk along the document’s own structure (a heading and its section, one FAQ entry, one record) and put the title into each chunk so it still makes sense on its own.

Evaluation. Write down 30 real questions and the passage that should answer each one. Run them after every change to chunking, model or ranking, and count how often the right passage lands in the top five. Without that list you’re tuning by vibes, the same trap covered in testing a feature that never gives the same answer twice.

The embedding model is a migration. Vectors from different embedding models aren’t comparable, so you can’t mix them. Switching models means re-embedding everything and swapping the whole set over at once. Store the model name next to every vector, and treat an upgrade the way you’d treat a schema change.

The quick decision: corpus fits in the context window → send all of it and cache the prefix · data is exact strings and names → full-text search first · under roughly a hundred thousand vectors per search → brute force in memory or on the device · millions, heavy filtering or constant writes → the vector support in the database you already run, then a dedicated one if measurements say so.

Start with the boring version

The order matters because each step tells you whether you need the next one. Sending the whole corpus shows you what perfect retrieval looks like. Keyword search shows you which questions need meaning rather than matching. Brute force shows you whether search speed is ever the bottleneck. Most apps stop at step one or two and ship a better feature for it, with one less system to keep in sync at 2 a.m.

Let AI read your screenshots → How prompt caching works