[diskann] Overhaul paged search

[hildebrandmw]

Paged search has been causing all kinds of issues for our code base and is actively getting in the way of simplifications in #1067 due to interactions with the PagedSearchState. But here's the thing: we've made the move to an async code base ... we can actually use that for once to make our lives easier.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 69.56522% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.48%. Comparing base (f4757a3) to head (9c773eb).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
diskann-providers/src/index/wrapped_async.rs	0.00%	28 Missing ⚠️
diskann/src/graph/index.rs	85.39%	13 Missing ⚠️
diskann-providers/src/index/diskann_async.rs	80.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (69.56%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1078      +/-   ##
==========================================
- Coverage   89.51%   89.48%   -0.04%     
==========================================
  Files         459      459              
  Lines       85646    85601      -45     
==========================================
- Hits        76663    76596      -67     
- Misses       8983     9005      +22     

Flag	Coverage Δ
miri	89.48% <69.56%> (-0.04%)	⬇️
unittests	89.10% <69.56%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann/src/graph/search/scratch.rs	98.21% <ø> (ø)
diskann/src/graph/test/cases/paged_search.rs	95.03% <100.00%> (-0.91%)	⬇️
diskann/src/provider.rs	95.14% <ø> (ø)
diskann-providers/src/index/diskann_async.rs	96.00% <80.00%> (-0.02%)	⬇️
diskann/src/graph/index.rs	95.68% <85.39%> (+0.23%)	⬆️
diskann-providers/src/index/wrapped_async.rs	46.42% <0.00%> (+0.54%)	⬆️

... and 27 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.