A lot of the article focuses on problems induced by a 512-token input limit. For example, one needs a lot more chunks with such a small input, especially with overlap. I realize that some embedding models do have input contexts that small, but 8K and 32K are fairly widely supported and reduce chunking-related problems.
For languages like English, there's also usually a lot of redundancy within a text, so 512 tokens might not give a very clear indication of the context. Lots of documents have similar introductions (like "#include <foo.h>\n") that make short contexts and truncation particularly harmful.
Also, "Nothing in the document past that point can ever be retrieved, and nothing anywhere told you." This is user-hostile behavior, even if they didn't want to admit to users that the auto-embedding support was poor.
Finally, the paragraph later on about truncation being "what you already have" reads like Claude talking to the developer, not like a vendor talking to users. But sure, maybe this is a good default for a database searching page titles, chat logs and Xeets?
On the 512 tokens: that's the window of the model we benchmarked with (all-MiniLM-L6-v2), not a claim about embedding models in general. The article does mention text-embedding-3-small's 8,192 window, and the same setup works with 32K models like Qwen3-Embedding. If your documents fit the window, the advice stands: keep truncate.
A bigger window makes chunking easier, not irrelevant. max_tokens defaults to the model's own limit, so with an 8K model you get fewer, larger chunks and overlap matters a lot less.
Two reasons we still chunk even when the document would fit:
1. One vector per document is a summary of the whole thing, so a short, highly relevant section gets averaged away by everything around it. One vector per chunk turns the question into "does this document contain something close to the query?", with the doc scored by its best chunk. Your #include example is exactly that case: the first N tokens of every file look alike, and what distinguishes them is further down. That's the "deep content" split in the benchmark — truncate got 55% recall@5, recursive got 83%.
2. Cost. Transformer embedding time grows superlinearly with input length, so pushing a whole 8K or 32K document through a local model on CPU costs far more than embedding it as 512-token chunks. Remote APIs bill per token either way.
That said, you're right that our numbers only show the effect against a 512 window. We should rerun the same benchmark with an 8K and a 32K model. I'd expect the gap to shrink but not disappear, and that's worth measuring rather than assuming.
On "nothing anywhere told you": agreed, silent truncation is bad behavior. That line describes what Manticore used to do (and what most embedding pipelines still do by default), not a defense of it. truncate is still the default because multi-vector output needs a different column type, so it has to be opt-in.
On "what you already have": fair, that sentence reads badly. It means "the old default, unchanged", not "good enough for you". We'll reword it.
Just so you know, your comment was automatically hidden (“dead”) until I vouched for it now. Same for most of your recent submissions. Actually it’s probably because of your (exclusively self-promotional) submissions that your comments and submissions get hidden.
Bigger context windows help but they don't remove the need to chunk. Embedding 8K tokens into one vector smears everything, retrieval quality drops even though nothing got truncated.
For languages like English, there's also usually a lot of redundancy within a text, so 512 tokens might not give a very clear indication of the context. Lots of documents have similar introductions (like "#include <foo.h>\n") that make short contexts and truncation particularly harmful.
Also, "Nothing in the document past that point can ever be retrieved, and nothing anywhere told you." This is user-hostile behavior, even if they didn't want to admit to users that the auto-embedding support was poor.
Finally, the paragraph later on about truncation being "what you already have" reads like Claude talking to the developer, not like a vendor talking to users. But sure, maybe this is a good default for a database searching page titles, chat logs and Xeets?
On the 512 tokens: that's the window of the model we benchmarked with (all-MiniLM-L6-v2), not a claim about embedding models in general. The article does mention text-embedding-3-small's 8,192 window, and the same setup works with 32K models like Qwen3-Embedding. If your documents fit the window, the advice stands: keep truncate.
A bigger window makes chunking easier, not irrelevant. max_tokens defaults to the model's own limit, so with an 8K model you get fewer, larger chunks and overlap matters a lot less.
Two reasons we still chunk even when the document would fit:
1. One vector per document is a summary of the whole thing, so a short, highly relevant section gets averaged away by everything around it. One vector per chunk turns the question into "does this document contain something close to the query?", with the doc scored by its best chunk. Your #include example is exactly that case: the first N tokens of every file look alike, and what distinguishes them is further down. That's the "deep content" split in the benchmark — truncate got 55% recall@5, recursive got 83%.
2. Cost. Transformer embedding time grows superlinearly with input length, so pushing a whole 8K or 32K document through a local model on CPU costs far more than embedding it as 512-token chunks. Remote APIs bill per token either way.
That said, you're right that our numbers only show the effect against a 512 window. We should rerun the same benchmark with an 8K and a 32K model. I'd expect the gap to shrink but not disappear, and that's worth measuring rather than assuming.
On "nothing anywhere told you": agreed, silent truncation is bad behavior. That line describes what Manticore used to do (and what most embedding pipelines still do by default), not a defense of it. truncate is still the default because multi-vector output needs a different column type, so it has to be opt-in.
On "what you already have": fair, that sentence reads badly. It means "the old default, unchanged", not "good enough for you". We'll reword it.