Retrieval at Depth
Leaderboards make a shortlist; your own data makes the choice
Scroll
Leaderboards make a shortlist; your own data makes the choice
Every retrieval system depends on its embedding model, and the public leaderboards are a reasonable place to build a shortlist. They are a poor place to make the final call. They average many tasks and domains, and a model that ranks well overall can do worse on your support tickets, legal clauses or internal code names. Run the shortlist against your own golden set, score recall@k, and let that decide.
Size is the next trade-off. Each vector costs its number of dimensions times the bytes per number: a million chunks at 1,024 dimensions in 32-bit floats take about 4 GB, and at 3,072 dimensions about 12 GB, with search cost growing to match. Some models are trained so that a vector can be cut to its first few hundred dimensions with modest loss, and storing the values in 8 bits or fewer shrinks it further. Check the quality loss on your own set before taking either saving. Switching models later means re-embedding the whole corpus, because query and document vectors have to come from the same model.
When a good general model still misses your domain, you can fine-tune it on pairs of queries and the passages that answer them. Training pulls each pair closer and pushes away passages that look similar but are wrong, which teaches the model your vocabulary. Those near misses, called hard negatives, matter most, so take them from the cases your current system gets wrong. A few thousand pairs from real queries is a common starting point. Measure before and after on a held-out part of the golden set, because a tuned model can gain on your domain and lose ground everywhere else.
Retrieval