Fix performance of tracing rate limiter#1530
Open
sergeymatov wants to merge 2 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces configurable rate limiting for tracing output by adding a tracing-throttle-based layer to tracectl, wiring it into dataplane startup, and exposing configuration via a new CLI argument in args.
Changes:
- Add
TracingRateLimitConfig/TracingControl::init_with_rate_limit()and initialize tracing with optional rate limiting. - Add
--tracing-rate-limit BURST:REPLENISH_PER_SECONDparsing and propagate it into the launch configuration. - Add the
tracing-throttledependency to the workspace andtracectl.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tracectl/src/lib.rs |
Re-exports rate-limit config type from control. |
tracectl/src/control.rs |
Adds rate-limit config + subscriber initialization changes, including throttling logic and tests. |
tracectl/Cargo.toml |
Adds tracing-throttle dependency. |
dataplane/src/main.rs |
Initializes tracing with optional rate limit from CLI args; adjusts default log level behavior. |
Cargo.toml |
Adds tracing-throttle to workspace dependencies. |
Cargo.lock |
Locks new tracing-throttle transitive dependency. |
args/src/lib.rs |
Adds TracingRateLimit type, CLI parsing/help, and config propagation + tests. |
Comment on lines
+319
to
+337
| // `DEBUG` level remains unthrottled if we assume that it is the most verbose level | ||
| // that users would want to enable while digging into particular issues. | ||
| fn unthrottled_level_filter<S>() -> impl Filter<S> + Send + Sync | ||
| where | ||
| S: Subscriber + for<'span> LookupSpan<'span>, | ||
| { | ||
| filter_fn(|meta| matches!(*meta.level(), Level::DEBUG)) | ||
| } | ||
|
|
||
| fn throttled_level_filter<S>() -> impl Filter<S> + Send + Sync | ||
| where | ||
| S: Subscriber + for<'span> LookupSpan<'span>, | ||
| { | ||
| filter_fn(|meta| { | ||
| matches!( | ||
| *meta.level(), | ||
| Level::INFO | Level::TRACE | Level::ERROR | Level::WARN | ||
| ) | ||
| }) |
Comment on lines
+319
to
+320
| // `DEBUG` level remains unthrottled if we assume that it is the most verbose level | ||
| // that users would want to enable while digging into particular issues. |
| long, | ||
| value_name = "BURST:REPLENISH_PER_SECOND", | ||
| value_parser=TracingRateLimit::from_str, | ||
| help = "Optional rate limit for lower-severity tracing output. Syntax: BURST:REPLENISH_PER_SECOND. |
| tracing-error = { version = "0.2.1", default-features = false, features = [] } | ||
| tracing-subscriber = { version = "0.3.23", default-features = false, features = [] } | ||
| tracing-test = { version = "0.2.6", default-features = false, features = [] } | ||
| tracing-throttle = { version = "0.4.2", default-features = false, features = [] } |
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
7f3a6a7 to
0814455
Compare
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
0814455 to
0e90334
Compare
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
4 times, most recently
from
81f0207 to
65ae6b0
Compare
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
65ae6b0 to
510ff04
Compare
This reverts commit 75f964b. Signed-off-by: Sergey Matov <sergey.matov@githedgehog.com>
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
510ff04 to
c7c6cb5
Compare
sergeymatov
changed the title
Comment on lines
+420
to
+438
| // `DEBUG` level remains unthrottled if we assume that it is the most verbose level | ||
| // that users would want to enable while digging into particular issues. | ||
| fn unthrottled_level_filter<S>() -> impl Filter<S> + Send + Sync | ||
| where | ||
| S: Subscriber + for<'span> LookupSpan<'span>, | ||
| { | ||
| filter_fn(|meta| matches!(*meta.level(), Level::DEBUG)) | ||
| } | ||
|
|
||
| fn throttled_level_filter<S>() -> impl Filter<S> + Send + Sync | ||
| where | ||
| S: Subscriber + for<'span> LookupSpan<'span>, | ||
| { | ||
| filter_fn(|meta| { | ||
| matches!( | ||
| *meta.level(), | ||
| Level::INFO | Level::TRACE | Level::ERROR | Level::WARN | ||
| ) | ||
| }) |
Comment on lines
183
to
+188
| tonic = { version = "0.14.6", default-features = false, features = [] } | ||
| tracing = { version = "0.1.44", default-features = false, features = [] } | ||
| tracing = { version = "0.1.44", default-features = false, features = ["release_max_level_debug"] } | ||
| tracing-error = { version = "0.2.1", default-features = false, features = [] } | ||
| tracing-subscriber = { version = "0.3.23", default-features = false, features = [] } | ||
| tracing-test = { version = "0.2.6", default-features = false, features = [] } | ||
| tracing-throttle = { version = "0.4.2", default-features = false, features = [] } |
Comment on lines
+1321
to
+1322
| help = "Optional rate limit for lower-severity tracing output. Syntax: BURST:REPLENISH_PER_SECOND. | ||
| Example: --tracing-rate-limit 50:5 allows bursts up to 50 repeated messages and replenishes 5 messages per second. |
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
2 times, most recently
from
a2bc745 to
dc1c83c
Compare
Avoid using RwLock to reduce complexity for EnvFilter. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Sergey Matov <sergey.matov@githedgehog.com>
sergeymatov
force-pushed
the
pr/smatov/rate-limit-recover
branch
from
dc1c83c to
55a1289
Compare
mvachhar
reviewed
May 19, 2026
| layer::{Context, Filter, Layer}, | ||
| prelude::*, | ||
| registry::LookupSpan, | ||
| reload, |
Contributor
There was a problem hiding this comment.
This package implicityl has the RwLock being referenced, but it is not obvious that this is the case. Can you please expand the commit message to explain where the removed RwLock was and what was done to remove it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move to own atomic operations for per-level filter for callsites. Reloading of interest is done via ArcSwap rather then RwLock