Retrieval at Depth

Choosing and tuning an embedding model

Leaderboards make a shortlist; your own data makes the choice

Scroll

Leaderboards make a shortlist; your own data makes the choice

Every retrieval system depends on its embedding model, and the public leaderboards are a reasonable place to build a shortlist. They are a poor place to make the final call. They average many tasks and domains, and a model that ranks well overall can do worse on your support tickets, legal clauses or internal code names. Run the shortlist against your own golden set, score recall@k, and let that decide.

Size is the next trade-off. Each vector costs its number of dimensions times the bytes per number: a million chunks at 1,024 dimensions in 32-bit floats take about 4 GB, and at 3,072 dimensions about 12 GB, with search cost growing to match. Some models are trained so that a vector can be cut to its first few hundred dimensions with modest loss, and storing the values in 8 bits or fewer shrinks it further. Check the quality loss on your own set before taking either saving. Switching models later means re-embedding the whole corpus, because query and document vectors have to come from the same model.

When a good general model still misses your domain, you can fine-tune it on pairs of queries and the passages that answer them. Training pulls each pair closer and pushes away passages that look similar but are wrong, which teaches the model your vocabulary. Those near misses, called hard negatives, matter most, so take them from the cases your current system gets wrong. A few thousand pairs from real queries is a common starting point. Measure before and after on a held-out part of the golden set, because a tuned model can gain on your domain and lose ground everywhere else.

Retrieval
THE SAME TICKETS, TWO EMBEDDERS general model tuned on your data billing shipping accounts vector storage for 1M chunks, 32-bit 768 dims 3.1 GB 1,024 dims 4.1 GB 3,072 dims 12.3 GB Choose on your own data. Every dimension is paid for in storage and search time.
The same tickets embedded by a general model and by one tuned on your data, and what each vector size costs to store.