Update bdn 0.16 binarydata and async by DrewScoggins · Pull Request #5217 · dotnet/performance

DrewScoggins · 2026-05-07T20:25:20Z

This change enables runtime-async. It is not on by default, but must be activated by an experiment. The runs will be added in a separate PR.

This change also includes updating BDN to latest as it was required for runtime-async.

…on testing

Copilot

Pull request overview

This PR updates the performance harness and microbenchmark suite to work with newer BenchmarkDotNet (0.16 nightly), including BDN’s new async validator APIs and addressing a BinaryData naming collision, while also adding an experiment-gated runtime-async feature flag for net11+ runs.

Changes:

Update BenchmarkDotNet version and migrate custom validators to ValidateAsync returning IAsyncEnumerable<ValidationError>.
Rename microbenchmarks’ BinaryData payload type to avoid collision with System.BinaryData.
Adjust harness exporter implementation for BDN 0.16’s exporter API changes and add an experiment-gated runtime-async MSBuild feature flag.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/tests/harness/BenchmarkDotNet.Extensions.Tests/UniqueArgumentsValidatorTests.cs	Switch test to use async validator API.
src/harness/BenchmarkDotNet.Extensions/ValuesGenerator.cs	Add `notnull` constraint to dictionary key generation to align with modern TFMs.
src/harness/BenchmarkDotNet.Extensions/UniqueArgumentsValidator.cs	Migrate to `ValidateAsync` and return `IAsyncEnumerable`.
src/harness/BenchmarkDotNet.Extensions/TooManyTestCasesValidator.cs	Migrate to `ValidateAsync` and nullability updates.
src/harness/BenchmarkDotNet.Extensions/NoWasmValidator.cs	Migrate to `ValidateAsync`.
src/harness/BenchmarkDotNet.Extensions/MandatoryCategoryValidator.cs	Migrate to `ValidateAsync`.
src/harness/BenchmarkDotNet.Extensions/PerfLabExporter.cs	Rework exporter to implement `IExporter` and write JSON artifact asynchronously.
src/harness/BenchmarkDotNet.Extensions/DiffableDisassemblyExporter.cs	Add null-forgiving operators for reflection results.
src/harness/BenchmarkDotNet.Extensions/BenchmarkDotNet.Extensions.csproj	Retarget harness to net8.0 and adjust BDN source reference TFM.
src/Directory.Build.targets	Add experiment-gated `runtime-async` feature flag for net11+.
src/Directory.Build.props	Whitespace-only formatting adjustments.
src/benchmarks/micro/Serializers/DataGenerator.cs	Rename `BinaryData` to `BinaryDataPayload` and update generation/metadata.
src/benchmarks/micro/libraries/System.Text.Json/Serializer/WriteJson.cs	Update generic args to `BinaryDataPayload`.
src/benchmarks/micro/libraries/System.Text.Json/Serializer/ReadJson.cs	Update generic args to `BinaryDataPayload`.
src/benchmarks/micro/README.md	Update documented “available” TFMs to net8.0+.
src/benchmarks/micro/MicroBenchmarks.csproj	Remove net472-only package references block.
scripts/ci_setup.py	Add `runtimeasync` experiment env-var plumbing (`EnableRuntimeAsync=true`).
eng/Versions.props	Bump `BenchmarkDotNetVersion` to a newer nightly.
docs/benchmarkdotnet.md	Update runtime options list and remove outdated/private CLR-build guidance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

        /// GenerateValue is used to generate a random value in the appropriate range for both the key and value
        /// </summary>
-        public static Dictionary<TKey, TValue> Dictionary<TKey, TValue>(int count)
+        public static Dictionary<TKey, TValue> Dictionary<TKey, TValue>(int count) where TKey : notnull


@@ -29,10 +29,11 @@ public IEnumerable<ValidationError> Validate(ValidationParameters validationPara
                new ValidationError(
                    isCritical: true,
                    message: $"{group.Key.Descriptor.Type.Name}.{group.Key.Descriptor.WorkloadMethod.Name} has {group.Count()} test cases. It MUST NOT have more than {Limit} test cases. We don't have inifinite amount of time to run all the benchmarks!!",


@@ -361,18 +361,6 @@ dotnet run -c Release -f net9.0 --cli "C:\Projects\performance\.dotnet\dotnet.ex

 This is very useful when you want to compare different builds of .NET.



+                {
+                    File.Delete(filePath);
+                }
+                catch (IOException)


Copilot

Pull request overview

Copilot reviewed 20 out of 21 changed files in this pull request and generated 4 comments.

@@ -29,10 +29,11 @@ public IEnumerable<ValidationError> Validate(ValidationParameters validationPara
                new ValidationError(
                    isCritical: true,
                    message: $"{group.Key.Descriptor.Type.Name}.{group.Key.Descriptor.WorkloadMethod.Name} has {group.Count()} test cases. It MUST NOT have more than {Limit} test cases. We don't have inifinite amount of time to run all the benchmarks!!",


+    <!-- net8.0 is the lowest TFM exercised by the perf pipelines (see scripts/channel_map.py).
+         net472 (and other netfx) consumers are no longer supported; the harness uses BDN APIs
+         (e.g. IValidator.ValidateAsync returning IAsyncEnumerable<T>) that would otherwise require
+         polyfill packages on netstandard2.0. -->


 The `--runtimes` or just `-r` allows you to run the benchmarks for **multiple Runtimes**.

-Available options are: Mono, wasmnet70, CoreRT, net462, net47, net471, net472, netcoreapp3.1, net6.0, net7.0, net8.0, and net9.0.
+Available options are: Mono, wasmnet70, CoreRT, netcoreapp3.1, net6.0, net7.0, net8.0, and net9.0.


+                catch (IOException)
+                {
+                    string uniqueString = DateTime.Now.ToString("yyyyMMdd-HHmmss");
+                    string altPath = $"{Path.Combine(summary.ResultsDirectoryPath, GetFileName(summary))}-{FileCaption}-{uniqueString}.{FileExtension}";
+                    logger.WriteLineError($"Could not overwrite file {filePath}. Exporting to {altPath}");


LoopedBard3

Looks good if the test runs succeed 👍.

timcassell · 2026-05-07T22:22:52Z

-    public class PerfLabExporter : ExporterBase
+    // Implements IExporter directly (not ExporterBase) because BDN 0.16.0's new
+    // abstract ExporterBase.ExportAsync(Summary, CancelableStreamWriter, CancellationToken)
+    // takes a non-public CancelableStreamWriter, which we cannot satisfy from outside the


CancelableStreamWriter is public, not sure why copilot thinks it's internal. Not that it really matters if you're just using File.WriteAllTextAsync anyway, but the comment is incorrect.

BDN 0.16.0 ('Async Refactor', PR #2958) made breaking API changes that the harness still consumed: - ExporterBase.ExportToLog(Summary, ILogger) was removed; the new abstract ExportAsync uses an internal CancelableStreamWriter that subclasses can't satisfy. PerfLabExporter now implements IExporter directly. - IValidator.Validate returning IEnumerable<ValidationError> was replaced with ValidateAsync returning IAsyncEnumerable<ValidationError>. The four validators (UniqueArguments, TooManyTestCases, NoWasm, MandatoryCategory) use .ToAsyncEnumerable() (transitively from BDN's System.Linq.AsyncEnumerable dep), not async + yield return - the latter produces an AsyncIteratorMethodBuilder state machine that deadlocks with BDN's BenchmarkSynchronizationContext. Switched BenchmarkDotNet.Extensions from netstandard2.0 to net8.0. The netstandard2.0 target was only used to enable an opt-in net472 path, but PERFLAB_TARGET_FRAMEWORKS=net472 is no longer exercised in CI and the new BDN APIs would otherwise require polyfill packages. Removed the net472 package conditional from MicroBenchmarks.csproj and the net472/.NET Framework references from docs/benchmarkdotnet.md and the micro README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Switching BenchmarkDotNet.Extensions from netstandard2.0 to net8.0 brought in stricter nullable annotations from the BCL and BDN package, which combined with TreatWarningsAsErrors=true (set in src/Directory.Build.props) turned previously-hidden nullability warnings into build errors. - ValuesGenerator.Dictionary<TKey, TValue>: add 'where TKey : notnull' constraint required by Dictionary<,>. - UniqueArgumentsValidator.BenchmarkArgumentsComparer.Equals: match the IEqualityComparer<T>.Equals(T?, T?) interface signature; handle nulls. - TooManyTestCasesValidator.SkipValidation: parameter must be MemberInfo? since MemberInfo.DeclaringType returns Type?. - DiffableDisassemblyExporter: add null-forgiving (!) on reflection lookups whose return values are intentionally trusted by this type (it's a copy of internal BDN code that operates on known types). - PerfLabExporter: BuildJson can return null; use string?. Verified by running the exact CI commands locally (dotnet restore + build with --framework net11.0) using the SDK from global.json. Build succeeds with 0 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

BDN 0.16's sync entrypoints (BenchmarkSwitcher.Run / BenchmarkRunner.Run) install BenchmarkDotNetSynchronizationContext (a single-threaded message pump) before benchmark discovery. Discovery executes [ParamsSource] and [ArgumentsSource] callbacks; some perf-repo callbacks do sync-over-async (notably SslStreamTests.GetTls13Support, which calls HandshakeAsync(...) .GetAwaiter().GetResult()). Sync-over-async deadlocks on the single-threaded SyncCtx because the awaited continuation is queued back to a pump that the caller is blocking. Switch Program.Main to async Task<int> and await BenchmarkSwitcher.RunAsync. The async entrypoint never installs BenchmarkDotNetSynchronizationContext on the caller, so discovery runs on the default context and sync-over-async in source callbacks no longer deadlocks. Real benchmark execution still gets the SyncCtx semantics it needs because BDN installs it inside the per-benchmark execute path, not on the entrypoint thread. This is the supported BDN-recommended fix for the discovery-deadlock symptom; no BDN code change is required. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The serialization-test helper class `MicroBenchmarks.Serializers.BinaryData` shadowed the in-box `System.BinaryData` type (introduced in net6+ via `System.Memory.Data`, brought in transitively by Azure.Core). Inside `DataGenerator.cs` (namespace MicroBenchmarks.Serializers) the unqualified `typeof(BinaryData)` always bound to the local helper class. Inside `ReadJson.cs` / `WriteJson.cs` (namespace System.Text.Json.Serialization.Tests) C# enclosing-namespace lookup found `System.BinaryData` first, so the `[GenericTypeArguments(typeof(BinaryData))]` attribute resolved to a DIFFERENT type than the `Generate<T>()` switch was checking. The mismatch was latent on main (where System.Memory.Data wasn't compile-visible to ReadJson.cs) but surfaced as `System.NotImplementedException` in `DataGenerator.Generate<T>()` whenever the reference closure changed (e.g. retargeting BenchmarkDotNet.Extensions from netstandard2.0 to net8.0). Renaming the helper class to `BinaryDataPayload` removes the shadowing once and for all. No behavior change on main; future-proofs against reference-closure churn. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

PR #5195 enabled the runtime-async language feature unconditionally for net11.0+ via `<Features>`. To avoid affecting baseline measurement runs, gate it behind the existing experiment infrastructure so it only takes effect in the dedicated experiment lane. - src/Directory.Build.targets: only set `runtime-async=on` when the `EnableRuntimeAsync` MSBuild property is `true` (in addition to the existing TFM check). - scripts/ci_setup.py: when `--experiment-name=runtimeasync` is passed, emit `EnableRuntimeAsync=true` as an env var so MSBuild picks it up as a property (matches the pattern used by the `jitoptrepeat` experiment). Verified locally: BDN dry runs against the BinaryDataPayload tests succeed both with and without the env var (18/18 benchmarks each). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

agocke and others added 2 commits May 7, 2026 11:38

Enable runtime-async for net11.0+

c6ece0d

Update BenchmarkDotNet to 0.16.0-nightly.20260504.1025 for verificati…

a5782b1

…on testing

Copilot AI review requested due to automatic review settings May 7, 2026 20:25

Copilot started reviewing on behalf of DrewScoggins May 7, 2026 20:26 View session

DrewScoggins requested review from LoopedBard3 and caaavik-msft May 7, 2026 20:31

Copilot AI reviewed May 7, 2026

View reviewed changes

DrewScoggins force-pushed the update-bdn-0.16-binarydata-and-async branch from c2fb987 to 57d3cd2 Compare May 7, 2026 21:41

Copilot AI review requested due to automatic review settings May 7, 2026 22:02

DrewScoggins force-pushed the update-bdn-0.16-binarydata-and-async branch from 57d3cd2 to fbbc502 Compare May 7, 2026 22:02

Copilot started reviewing on behalf of DrewScoggins May 7, 2026 22:02 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

LoopedBard3 previously approved these changes May 7, 2026

View reviewed changes

timcassell reviewed May 7, 2026

View reviewed changes

DrewScoggins and others added 5 commits May 7, 2026 15:27

DrewScoggins dismissed LoopedBard3’s stale review via 2ee9131 May 7, 2026 22:27

DrewScoggins force-pushed the update-bdn-0.16-binarydata-and-async branch from fbbc502 to 2ee9131 Compare May 7, 2026 22:27

build-analysis Bot mentioned this pull request May 8, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update bdn 0.16 binarydata and async#5217

Update bdn 0.16 binarydata and async#5217
DrewScoggins wants to merge 7 commits intodotnet:mainfrom
DrewScoggins:update-bdn-0.16-binarydata-and-async

DrewScoggins commented May 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

LoopedBard3 left a comment

Uh oh!

timcassell May 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		@@ -361,18 +361,6 @@ dotnet run -c Release -f net9.0 --cli "C:\Projects\performance\.dotnet\dotnet.ex

		This is very useful when you want to compare different builds of .NET.

Conversation

DrewScoggins commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

LoopedBard3 left a comment

Choose a reason for hiding this comment

Uh oh!

timcassell May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DrewScoggins commented May 7, 2026 •

edited

Loading