Add stopping criterion sample count#341
Open
oleksandr-pavlyk wants to merge 3 commits intoNVIDIA:mainfrom
Open
Add stopping criterion sample count#341oleksandr-pavlyk wants to merge 3 commits intoNVIDIA:mainfrom
oleksandr-pavlyk wants to merge 3 commits intoNVIDIA:mainfrom
Conversation
CLI option --warmup-runs implemented and documented. The warm-up counts is enforced to always be positive. This is necessary to ensure that JIT-ting has occurred, and use of blocking kernel would not result in time-outs. Test is option parser is added.
Because warm-up runs are executed without use of blocking kernel, the blocking kernel was not jitted until actual measurements were collected. The module loading cost incurred during the first run shows as elevated CPU time noise value for the first measurement as noted in NVIDIA#339 This PR adds `this->block_stream(); this->unblock_stream();` prior to executing warm-up loop with use of blocking kernel disabled. This ensures that blocking kernel is instantiated during the warm-up, but it no other kernel is launched between its launch and stream sync thus avoiding deadlocking.
--stopping-criterion sample-count --target-samples 100 would stop once max(--min-samples, --target-samples) samples are collected
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR builds on top of #338, so it should be merged after #338 is merged.
This PR add stopping criterion sample-count with target-samples integral parameter, which allows users to collect deterministic number of samples.
Together with
--warmup-runs <count>this permits users to replicate their prior art of running fixed warm-up count and fixed timed runs, but still benefit from NVBench's benchmark reporting facilities, and more accurate GPU timing.The sample-count stopping criterion is essentially the "fixed" custom stopping criterion from
examples/custom_criterion.cu, but interaction with--min-sampleshas changed.Previously, stopping criterion was applied once
m_total_samples > m_min_samples, so running./build/bin/nvbench.example.cpp20.custom_criterion --min-samples 50would collect 51 samples.This PR replaced that condition with
m_total_samples >= m_min_samplesinstead.do_is_finished()method to ensurem_noise_tracker.back()is safe to call.The custom_criterion.cu example needs to be changed to implement an alternative stopping criterion in the future.