Capacity limits¶
A bundle superposes items into one fixed-size vector, so it has a capacity: past some number of items it can no longer tell members from non-members. This page reports what was measured and where the measurement stops. It does not extrapolate.
What was measured¶
benchmark_scaling_results.json (generated by hms-scaling, 2026-06-19) stores N random items in one union (Bloom) bundle, then probes it with 50 members and 50 non-members per N. A positive gap means members still score above non-members on average. d' >= 2 means the separation between the two score distributions is at least two noise standard deviations. The table gives the last N in the sweep that meets each criterion, with the measured false-positive rate there.
| Dimension | Density | Active bits | Last N, positive gap | False positives there | Last N, d' >= 2 | False positives there | Largest N tested |
|---|---|---|---|---|---|---|---|
| 16,384 | 1/256 | 64 | 500 | 4% | 1,000 | 32% | 2,500 |
| 65,536 | 1/256 | 256 | 1,000 | 4% | 1,000 | 4% | 3,000 |
| 65,536 | 1/1024 | 64 | 2,800 | 14% | 4,000 | 26% | 10,000 |
| 131,072 | 1/256 | 512 | 1,000 | 6% | 1,500 | 26% | 3,000 |
| 131,072 | 1/1024 | 128 | 2,800 | 8% | 4,000 | 30% | 14,000 |
| 262,144 | 1/1024 | 256 | 4,000 | 12% | 4,000 | 12% | 14,000 |
| 262,144 | 1/4096 | 64 | 11,200 | 12% | 16,000 | 18% | 64,000 |
| 524,288 | 1/1024 | 512 | 4,000 | 2% | 4,000 | 2% | 14,000 |
| 524,288 | 1/4096 | 128 | 12,800 | 16% | 16,000 | 30% | 56,000 |
How to read it¶
- A "last N" is not a safe capacity. At D = 16,384 the false-positive rate is already 4% at N = 500 and 32% at N = 1,000. A deployment that needs a low false-positive rate has to stay well below these points, or verify candidates against the exact store.
- Earlier numbers were wrong. Previous versions of the README reported 2,478, 9,800 and 58,432 as a "hard wall". Those are the points where the bundle is saturated and the false-positive rate is 100%. They are not usable capacities and are no longer quoted.
- Sparser is roomier, up to a point. The 1/4096 density rows hold far more items than 1/256 at the same dimension, with fewer active bits per item (64 at D = 262,144, 128 at D = 524,288).
- Probes are small. 50 members and 50 non-members per point means a measured rate of 4% is two false positives. Treat single-digit percentages as coarse.
- The sweep grid differs per row. Each row's range is in the last column; nothing is claimed beyond it.
What this does not establish¶
- The test items are random vectors. Real text encodes to clustered, correlated vectors; the effect on capacity has not been measured here.
- It is a membership test on a union bundle. Per-fact retrieval of a value bound to a key from a superposition has its own, lower floor (the linked note reports roughly M/D of 0.1 to 0.3 for naive unbinding); see the superposition floor note.
- It does not describe the document store. Documents are stored one chunk at a time, not superposed, so their limits are memory and the resource limits in Production readiness.