Skip to content

add config examples to use PQ and SQ indexing and search for wiki-1M with Cohere embeddings #1047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
harsha-simhadri wants to merge 5 commits into
base: main
Choose a base branch
from
+112 −0
Open

add config examples to use PQ and SQ indexing and search for wiki-1M with Cohere embeddings #1047

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a623513
add config examples for wiki-1M with Cohere embeddings
harsha-simhadri May 10, 2026
bd3e73f
Potential fix for pull request finding
harsha-simhadri May 10, 2026
fdcbff1
fix: restore spherical-exhaustive.json and update wiki1M SQ config
harsha-simhadri May 11, 2026
10c4f6f
update sq
harsha-simhadri May 11, 2026
b32f1ad
update sq
harsha-simhadri May 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions diskann-benchmark/example/graph-index-product-quantization-wiki1M.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"search_directories": [
"../big-ann-benchmarks/data/wikipedia_cohere" ],
Comment thread
harsha-simhadri marked this conversation as resolved.
"jobs": [
{
"type": "graph-index-build-pq",
"content": {
"index_operation": {
"source":{
"index-source": "Build",
"data_type": "float32",
"data": "wikipedia_base.bin.crop_nb_1000000",
Copy link
Copy Markdown
Contributor

@hildebrandmw hildebrandmw May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice if we could run these through the dry run functionality to ensure they don't get out of date. The one problem would be file-path validation would fail without the existence of the data/query/groundtruth files.

Copy link
Copy Markdown
Contributor Author

@harsha-simhadri harsha-simhadri May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will look into dry run checks.

The goals of the new files is to illustrate the parameter choices to be used for larger datasets. The params in config files for test datasets are insufficient for larger datasets. If on the other hand you are oppposed to adding these entries here, please let me know and I can drop it, and not spend time on dry run checks

Copy link
Copy Markdown
Contributor

@magdalendobson magdalendobson May 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relatedly, it looks like there is no example config that builds a PQ index using the example datasets in this repo, although there is one for spherical. Is that something we should change?

Copy link
Copy Markdown
Contributor

@hildebrandmw hildebrandmw May 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding examples is great and would really help reduce friction for people trying out the library! My concern is around ensuring these examples are up-to-date, so in the future users don't run into the unfortunate situation where they try an example and it immediately fails for reasons other than the files not being in the correct place.

Apologies if my original comment didn't leave much direction. The main issue with the existing dry-run functionality is that file paths are also validated (at least for existence) as part of the dry-run check, which makes it harder to use it out-of-the-box.

The issue boils down to this check here. We want a way to turn it off when running a CI validation run, then we'd be good. The problem is how to turn it off cleanly in a way that is somewhat general. Maybe an additional "--no-environment" flag in addition to --dry-run to skip system dependent environment checks and making that available in the Checker struct? That would provide a nice place for generally disabling all kinds of system dependent checks for CI purposes.

"distance": "inner_product",
"start_point_strategy": "medoid",
"max_degree": 32,
"l_build": 100,
"alpha": 1.2,
"backedge_ratio": 1.0,
"num_threads": 32,
"multi_insert": {
"batch_size": 128,
"batch_parallelism": 32,
"intra_batch_candidates": "all"
}
},
"search_phase": {
"search-type": "topk",
"queries": "wikipedia_query.bin",
"groundtruth": "wikipedia-1M",
"reps": 1,
"num_threads": [
32
],
"runs": [
{
"search_n": 10,
"search_l": [
100,
150,
200,
300
],
"recall_k": 10
}
]
}
},
"num_pq_chunks": 192,
"seed": 13076402859301299683,
"max_fp_vecs_per_prune": 0,
"use_fp_for_search": false
}
}
]
}
57 changes: 57 additions & 0 deletions diskann-benchmark/example/graph-index-spherical-quantization-wiki1M.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"search_directories":[
"../big-ann-benchmarks/data/wikipedia_cohere"
],
"jobs": [
{
"type": "graph-index-build-spherical-quantization",
"content": {
"build": {
"data_type": "float32",
"data": "wikipedia_base.bin.crop_nb_1000000",
"distance": "inner_product",
"start_point_strategy": "medoid",
"max_degree": 32,
"l_build": 100,
"alpha": 1.2,
"backedge_ratio": 1.0,
"num_threads": 32,
"multi_insert": null
},
"search_phase": {
"search-type": "topk",
"queries": "wikipedia_query.bin",
"groundtruth": "wikipedia-1M",
"reps": 5,
"num_threads": [
32
],
"runs": [
{
"search_n": 10,
"search_l": [
50,
100,
150,
200,300
],
"recall_k": 10
}
]
},
"seed": 12648430,
"transform_kind": {
"padding_hadamard": "same"
},
"query_layouts": [
"same_as_data",
"full_precision"
],
"num_bits": 2,
"pre_scale": {
"some": 0.00390625
}
}
}
]
}
Loading