Embedding weaknesses: four conditions for accurate semantic search
Embeddings, in plain words, turn text into a set of numbers, so that phrases close in meaning sit close together. An employee's question and the right passage of a document find each other even with no words in common.
Embedding weaknesses are the places where that closeness fails: exact numbers, outdated revisions, shared access. The list is short and known in advance, so it can be checked point by point.
Why this matters
- Know in advance which queries document search will miss, and write that into the acceptance criteria
- Ask the vendor about access control before the database is built, not after
- Get a simple accuracy measurement that one employee can run in a day
What are embedding weaknesses in plain words
Picture a warehouse where boxes are arranged by what is inside them. Similar things sit together: fasteners with fasteners, lubricants with lubricants. The storekeeper searches by description rather than by number — and usually finds things faster than any catalogue.
That is how an embedding works: text becomes a set of numbers, and texts close in meaning end up close together. Search comes down to a single question — what lies nearest to the query.
Hence the weak spots. In a warehouse organised by meaning, it is hard to find one specific box by its inventory number: the number carries no meaning. Two revisions of the same contract sit side by side and cannot be told apart by description. And the shelf is shared: unless it is partitioned, everyone sees everything.
How it works and where it misses
- Documents are cut into chunks. Half a page is a typical chunk size. A table split down the middle loses its column headers, and the retrieved fragment stops making sense on its own.
- Each chunk is turned into numbers. Meaning survives, exact designations blur: part codes
АТ-4471-02andАТ-4471-12are almost the same thing to the model. - Chunks go into a vector database. Permissions on the source folders do not carry over. Access control is set through a separate field on each chunk and a filter on that field.
- The query is matched by closeness. Five to ten nearest chunks come back. Recency, version and document status have no effect on closeness — an old revision outranks the new one as often as the other way round.
- What is found goes to the model. From there RAG takes over: the answer is assembled from those chunks. A search miss turns into a confident wrong answer, because the model trusts whatever it was handed.
Shifts in tone are a story of their own. When a database is filled with dry procedural documents, support answers become drier and more formal, even as factual accuracy improves. OWASP describes this as a side effect of augmenting answers with retrieval, and it shows up on the support line long before it shows up in reports.
An example from practice
A service company with a 4,000-page knowledge base: procedures, equipment data sheets, correspondence with the manufacturer. Engineers ask by voice and in writing, and semantic search was connected in two weeks.
The figures below are a calculated benchmark for a company of this profile, not a report on a specific deployment. The measurement method is spelled out so it can be repeated on any company's own data.
The measurement is simple: 100 real engineer questions from one month, with the correct document marked by hand for each. The metric is the share of queries where the right document landed in the top five results. On pure semantic search, the benchmark is around 68 out of 100. The misses fall into three piles: exact part codes, outdated revisions, narrow manufacturer terminology.
Three things are then added. Exact matching on numbers and part codes running alongside semantic search. A filter on document date and status applied before the search. Reranking of the top twenty by a separate model. The same measurement on the same 100 questions gives around 84 out of 100.
The remaining 16 misses are more honestly left to people. Half of them are questions the database holds no answer to at all.
What the business gets out of it
- Acceptance gets a number. The top-five hit rate is measured before payment and after every revision. The vendor is given a threshold below which the work is not accepted.
- Leak risk is closed at the start. Permission fields are designed together with the database. Reworking access control on a finished database means reloading every document.
- Budget is spent precisely. The three piles of misses are fixed by different means at different prices: a date filter is a query change, a glossary of terms is a day's work, reranking is a separate model and a separate invoice.
- Answers stay in shape. Shifts in tone after the database grows are tracked on the same sample of conversations as accuracy.
When this can be skipped
With a base of a hundred documents, sorting into folders and ordinary keyword search cover the job entirely. The threshold where semantic search starts to pay off usually falls somewhere past a thousand documents, where questions are asked in the employee's words rather than the document's.
Nor is a study of these weaknesses needed where questions are strictly standard: ten request forms, fixed fields, ready-made answers. Such a process is cheaper to handle with a reference table and rules. A checklist helps gauge whether a given process is ready for automation.
What is worth checking
First: ask to see a measurement on the company's own documents and its own questions, not a demo on a prepared sample. A hundred queries and one day of labelling by a subject-matter specialist is enough.
Second: ask how access control works inside the database. An answer along the lines of "access is restricted at the application level" means that any route around the application opens the entire database. A working answer describes a field on every chunk and a filter on it before results are returned.
Third: find out who adds to the database and how. A single uploaded document with faulty data spreads through answers instantly, and it takes weeks to notice. A list of sources with an owner for each, plus a repeat measurement after every large addition, helps.
Frequently asked questions
What are embedding weaknesses in plain words?
They are the places where semantic search returns something similarly worded instead of the document actually needed. Exact numbers and part codes carry almost no meaning, an old revision of a document looks nearly identical to the new one, and a vector database does not enforce access control on its own. The list is short and known in advance, so it can be checked point by point.
What are embeddings in plain words?
They are a way of turning text into a set of numbers, where phrases close in meaning get similar numbers. After that conversion, search works by description rather than by exact wording: an employee's question and the right passage of a document find each other even with no words in common. The flip side is that exact numbers and part codes blur when converted into numbers.
How does semantic search differ from keyword search?
Keyword search finds a document only when the word matches literally. Semantic search finds by description: "won't start after a cold snap" leads to the procedure on starting current. But it confuses an account number or a part code with neighbouring ones. The working combination is both methods in a single query, plus filters by date and department.
Why does search return a similar document instead of the right one?
An embedding measures how close the wording is. Last year's supply contract and its new revision differ by two paragraphs, and to the model that is almost the same text. Filtering by date and version before the search helps: everything irrelevant is cut off by metadata first, and meaning is searched only within what remains.
What is hybrid search and why is it needed?
It combines three methods in a single query: search by meaning, search by exact match of a word or number, and a filter on fields such as date, department or document type. Exact matching catches part codes and account numbers; the embedding catches the meaning of the question. What is found is then re-sorted by a separate reranking model.
How can search quality be tested on a company's own documents?
Collect 100 real questions from employees or customers and mark the correct document for each one by hand. Then calculate the share of queries where the right document landed in the top five results. The measurement is repeated after every change and once a quarter: the database grows, and the hit rate shifts over time.
Who can see documents in a shared vector database?
By default, everyone with access to the search. Permissions on the source folders do not carry over during loading: the file becomes a set of numbers and text chunks. Access control is defined separately, through permission fields on each chunk and a filter applied before results are returned.
Let’s discuss your project?
Tell us about your process — we’ll suggest where AI pays off fastest.
Related articles
Embeddings explained simply: how machines search by meaning
Embeddings, explained simply, are a way of writing text down as numbers so that phrases with similar meanings end up with similar numbers.
What Is a Vector Database and Why Business Needs It
A vector database stores text and documents as numerical coordinates, so search runs on meaning rather than on matching words.