[RFC] Bring your own type erasure#1068
Open
hildebrandmw wants to merge 4 commits into
Open
Conversation
added 2 commits
May 14, 2026 10:51
hildebrandmw
changed the title
added 2 commits
May 14, 2026 10:56
arkrishn94
approved these changes
May 14, 2026
Contributor
arkrishn94
left a comment
arkrishn94
left a comment
There was a problem hiding this comment.
Thanks Mark, this is super helpful.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an RFC documenting a “bring your own type erasure” pattern intended to reduce nested dispatch in distance-computation composition.
Changes:
- Adds a new RFC describing the motivation, example pattern, generated-code comparison, and trade-offs.
- Identifies potential application areas such as
DistanceProvider, spherical quantization kernels, and multi-vector backends.
Comments suppressed due to low confidence (3)
rfcs/01068-byo-type-erasure.md:31
- Grammar: the article and plural noun do not agree here.
The combination of unwrapping + delegation is used to create another trait object, leading to an unavoidable situations where we have at least two levels of dynamic dispatch.
rfcs/01068-byo-type-erasure.md:19
- Grammar: this compound modifier should be hyphenated before the noun.
Lower level APIs in our library use various flavors of type-erasure to enable polymorphism over metric, micro-architecture, and length specialization.
rfcs/01068-byo-type-erasure.md:44
- Grammar: this compound modifier should be hyphenated before the noun.
How can we redesign our lower level APIs to allow composition of distance computations?
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+29
to
+31
| Here, an inner distance computations (using one of the type-erasure approaches outlined above) need to be composed with a small unwrapping layer. | ||
| For `diskann-garnet` specifically, this unwrapping layer reifies the type of raw byte slices (translates from `&[u8]` to the type needed by the inner computer). | ||
| The combination of unwrapping + delegation is used to create another trait object, leading to an unavoidable situations where we have at least two levels of dynamic dispatch. |
| ## Proposal | ||
|
|
||
| The solution is relatively simple and is probably a variant of some visitor pattern. | ||
| For the purposes of this demonstration, assume we have two level of distance function factories. |
| Multiply, | ||
| } | ||
|
|
||
| // Instead of return a function pointer from level 1, we visit implementations of `Level1`. |
| The main trade-offs here are API complexity and compile times. | ||
| If the `level_1_factory` dispatches to many possible implementations, like the `DistanceProvider` API which dispatches across micro-architecture, metric, and length specialization, each higher level essentially redoes that work. | ||
|
|
||
| However, for distance functions that are called millions or billions of time in a hot loop, the extra complexity to minimize overhead is often worth it. |
| ## Summary | ||
|
|
||
| This RFC outlines a pattern for tackling composition of distance computers with only a single level of type erasure. | ||
| The goal is to streamline patterns like #1050 where trait object based distance computers are embedded in new-type wrappers to create yet another trait object. |
| mulss xmm0, xmm1 | ||
| ret | ||
| ``` | ||
| Everything is inlined! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1068 +/- ##
=======================================
Coverage 89.51% 89.51%
=======================================
Files 461 461
Lines 85920 85920
=======================================
Hits 76912 76912
Misses 9008 9008
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
JordanMaples
approved these changes
May 14, 2026
suri-kumkaran
approved these changes
May 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A small RFC describing a pattern we can use to reduce levels indirection in distance function composition. No true action is needed in the codebase, I'm just writing down what I've recommended to several people for an issue I'm seeing occur more frequently.
Rendered