A client’s support bot spent three weeks failing to find one specific page. The page existed. It was in the index. It contained the exact product name the customer typed. The retriever kept handing the model four other pages instead, and the model, being helpful, made something up from those.
I did what you do. Chunked smaller. Chunked bigger. Swapped the embedding model twice. Added a reranker, which fixed it, and I moved on without really understanding why the first stage was so bad at a query that was almost literally a substring of the answer.
The reason, it turns out, is architectural, and Hugging Face just shipped a fix for it that is easier to try than it used to be. Sentence Transformers v6.0 added a fourth model type, MultiVectorEncoder, for ColBERT-style late interaction retrieval. I spent a weekend with it. Here’s what I think is true, including the parts that make it a worse idea than the benchmark tables suggest.
What a multi-vector model does differently
A normal embedding model reads your document and gives you one vector. Everything the model noticed about a 400-word support article has to survive being compressed into 768 numbers. Similarity is then one dot product between two summaries. That compression is why my exact product name kept losing: it’s one signal, averaged away among a few hundred others.
A multi-vector model keeps one vector per token and scores with MaxSim. For each query token, take its best match against any document token, then sum those maxima. Nothing gets averaged away, so a rare exact term stays a strong signal. The idea goes back to ColBERT, and the PyLate paper is the shortest readable account of how these models get trained if you want the theory.
In practice the API change is one word. This is the whole difference on the retrieval side:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Alibaba-NLP/gte-modernbert-base")
doc_embeddings = model.encode_document(["Paris is the capital of France."])
print(doc_embeddings.shape)
# (1, 768) -> one vector per document
Versus:
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("lightonai/mLateOn")
queries = ["What is the capital of France?"]
documents = [
"Paris is the capital of France.",
"Berlin is the capital and largest city of Germany, by both area and population.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape)
# (10, 128)
print(document_embeddings[0].shape, document_embeddings[1].shape)
# (10, 128) (19, 128)
Note what comes back: a list of 2D tensors, one per input, each shaped (num_tokens, embedding_dim). You can’t stack them into one rectangular array, because every document has its own token count. That single fact is responsible for most of the operational pain later in this post.
Scoring is a method call:
scores = model.similarity(query_embeddings, document_embeddings)
print(scores)
# tensor([[10.7942, 11.1104, 10.9743, 11.0811]])
Also worth knowing: encode_query and encode_document are not interchangeable. These models are asymmetric, they prepend different marker tokens for each side, and using the wrong one quietly gives you wrong embeddings rather than an error.
The honest benchmark is about one point
This is where I expected to get excited and mostly didn’t.
LightOn trained two models specifically so you could compare fairly: LateOn and DenseOn, same training data, same ModernBERT backbone, same 149M parameters, differing only in whether they keep one vector per token or pool down to one per document. That’s a real controlled comparison, which is rare enough that I want to credit it.
On NanoBEIR, LateOn scores a mean NDCG@10 of 0.6868 against DenseOn’s 0.6764. It wins on 9 of 13 datasets. It loses on four of them, including FiQA2018 by a fairly clear margin (0.5871 against 0.6491). On the full 15-dataset BEIR the pair scores 57.22 against 56.20, so the gap holds up outside the small benchmark.
One NDCG point. For short passages, that’s what the architecture buys you.
Then there’s this, and it changed my mind about where this technique belongs. On MLDR, a long-document retrieval benchmark, the multilingual siblings of the same pair score 77.92 against 51.59.
That is not one point. That is the dense model falling over. Which makes sense once you say it out loud: the longer the document, the more brutal the compression into a single vector, and the more there is to lose. If your corpus is tweets and FAQ entries, late interaction is a rounding error. If your corpus is contracts, manuals, or fifty-page PDFs, it’s a different class of result. My client’s support articles sat closer to the boring end, which is a useful thing to have learned before doing the migration.
I wrote a while back about how I decide between RAG and fine-tuning, and the same instinct applies here: measure your corpus before you pick your architecture, because the published averages are describing someone else’s documents.
The bill
Now the part the announcement posts underplay.
Encoding 4,874 Natural Questions passages with LateOn produced 608,414 token vectors. An average of 124.8 vectors per passage. In float32 that’s 311.5MB of index, against 7.5MB for the same passages under all-MiniLM-L6-v2 and 15.0MB under gte-modernbert-base.
Roughly 42x the storage. About 62 KiB per passage.
To Hugging Face’s credit, they immediately walk that back with real context. Compressed with PLAID, which stores a centroid id plus a quantized residual instead of the vector, the same 608,414 vectors take 92MB. And a 4096-dimensional dense model like Qwen3-Embedding-8B would need around 80MB for that same corpus. So the compressed multi-vector index sits in the same territory as dense indexes people already run happily. That’s a fair point and it defused most of my objection.
Token pooling helps further. HierarchicalTokenPooling clusters each document’s token vectors and replaces each cluster with its mean, keeping roughly 1 / pool_factor of them:
from sentence_transformers import MultiVectorEncoder
from sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling
model = MultiVectorEncoder("lightonai/LateOn")
pooling = HierarchicalTokenPooling(pool_factor=2)
document_embeddings = model.encode_document(documents, token_pooling=pooling)
At pool_factor=2 the index drops from 311.5MB to 156.4MB. At 3, 104.7MB. The original experiments measured the quality cost on BEIR at 100.6% of unpooled performance at factor 2, and 99.0% at factor 3. Halving your index for approximately nothing is the easiest yes in this entire post. Measure it on your own data before you commit to a factor, because that number is corpus-specific.
Where I’d actually put this
Not in front of your whole corpus.
Exhaustive model.similarity scoring is exact and scales linearly in total corpus tokens while keeping every vector in memory. The Hugging Face benchmark search takes about 120ms end to end over 4,874 documents on an RTX 3090. That’s fine. It does not extrapolate, and Sentence Transformers doesn’t ship a real late-interaction index, so past a few thousand documents you’re integrating Qdrant, Weaviate, Vespa, Milvus or fast-plaid yourself.
Qdrant’s own recommendation is to reserve late interaction for reranking a few hundred candidates rather than scanning a collection, and having read the rest of the numbers, I agree with them. The two-stage version looks like this:
from sentence_transformers import MultiVectorEncoder, SentenceTransformer
from sentence_transformers.util import semantic_search
retriever = SentenceTransformer("jinaai/jina-embeddings-v5-text-nano-retrieval")
reranker = MultiVectorEncoder("perplexity-ai/pplx-embed-v1-late-0.6b", trust_remote_code=True)
corpus_embeddings = retriever.encode_document(corpus, convert_to_tensor=True)
hits = semantic_search(retriever.encode_query([query], convert_to_tensor=True), corpus_embeddings, top_k=50)[0]
candidates = [corpus[hit["corpus_id"]] for hit in hits]
query_embeddings = reranker.encode_query([query])
document_embeddings = reranker.encode_document(candidates)
scores = reranker.similarity(query_embeddings, document_embeddings)[0]
You keep your existing cheap index. You pay the multi-vector cost on 50 documents instead of 500,000. You store nothing extra, because the reranker encodes candidates on the fly. This is the version I’d put in front of a paying client, and it’s most of the win.
The approximate index backends deserve a caution too. In Hugging Face’s own tests, Weaviate’s MUVERA made ingestion 3x faster and queries 1.8x faster, but the correct third passage didn’t appear even in its top 50. Vespa’s default second-phase window of 100 candidates left two of three correct passages unscored. Speed knobs on retrieval are not free, and the failure mode is silent.
Things that will waste an afternoon
A few sharp edges I hit or read about, listed so you don’t rediscover them.
MaxSim scores are not comparable across models. The score sums over query tokens, so its magnitude depends on how many query tokens a model’s recipe produces. LateOn scoring the same four documents gives numbers around 11; ColBERTv2 gives 12.8 to 27.2 for the same inputs. Any threshold you tuned is model-specific. There’s a meanmaxsim option that returns a bounded score if you need one.
Models with non-attend query expansion, which covers the Stanford checkpoints like colbert-ir/colbertv2.0 and answerdotai/answerai-colbert-small-v1, reject Flash Attention at load time. Use sdpa for those.
Saving is one-way. PyLate, Stanford ColBERT and colpali-engine checkpoints all load into MultiVectorEncoder, but save_pretrained output isn’t loadable by any of them. Decide before you fine-tune something you’ll need elsewhere.
And truncation is quiet. document_length caps how much of a document gets encoded, and a 662-token passage through LateOn’s cap of 300 comes back as 273 vectors with the rest simply gone. On a long-document corpus, the exact case where this architecture shines, that’s the setting most likely to be silently sabotaging you.
What I’d do this week
Skip the migration. Run one evaluation.
Take 200 real queries from your logs, the ones where users rephrased or gave up, and score your current retriever’s top 10 against a late-interaction reranker’s reordering of those same 10. No new index, no new infrastructure, about an hour of work. If the reranker pulls the right document up on queries your dense model buries, you have a business case. If it doesn’t, you’ve saved yourself a 42x index and a week.
That’s the same test I run before proposing any retrieval change on client projects, because the answer is corpus-specific often enough that guessing is expensive.
