Skip to content

[fix](be) Fix RuntimeFilter selectivity sampling_frequency lost during VExprContext recreation#62355

Merged
BiteTheDDDDt merged 5 commits intoapache:masterfrom
BiteTheDDDDt:fix-rf-selectivity-sampling-frequency
Apr 14, 2026
Merged

[fix](be) Fix RuntimeFilter selectivity sampling_frequency lost during VExprContext recreation#62355
BiteTheDDDDt merged 5 commits intoapache:masterfrom
BiteTheDDDDt:fix-rf-selectivity-sampling-frequency

Conversation

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor

@BiteTheDDDDt BiteTheDDDDt commented Apr 10, 2026

What problem does this PR solve?

introduced by #59832

Problem Summary:

RuntimeFilter selectivity tracking was completely non-functional because
sampling_frequency was lost during VExprContext recreation.

In RuntimeFilterConsumer::_get_push_exprs(), sampling_frequency=32 was set
on a temporary probe_ctx VExprContext. However, only VRuntimeFilterWrapper
expressions (VExpr) were returned, not the VExprContext. When
_append_rf_into_conjuncts() later created a new VExprContext via
VExprContext::create_shared(expr), the new context had default
_sampling_frequency=-1 (DISABLE_SAMPLING).

With _sampling_frequency=-1, the condition (_judge_counter++) >= -1
evaluated to 0 >= -1 → true on every call, causing reset_judge_selectivity()
to fire every time. This meant selectivity counters were perpetually reset
and never accumulated, making the runtime filter selectivity optimization
completely ineffective.

Fix: Store sampling_frequency in VRuntimeFilterWrapper (which survives
VExprContext recreation) and propagate it to VExprContext in
VRuntimeFilterWrapper::open(), which is called on both original and cloned
contexts.

Release note

Fixed a bug where RuntimeFilter selectivity tracking was non-functional due to
sampling_frequency being lost during VExprContext recreation, causing runtime
filters that should be skipped (due to low selectivity) to never be identified.

Check List (For Author)

  • Test: Unit Test
    • Added 2 regression tests to runtime_filter_selectivity_test.cpp
    • Added 3 new tests in vruntimefilter_wrapper_sampling_test.cpp
    • All 22 tests pass
  • Behavior changed: No (selectivity tracking was broken before, this makes it work as designed)
  • Does this need documentation: No

…g VExprContext recreation

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

RuntimeFilter selectivity tracking was completely non-functional because
`sampling_frequency` was lost during VExprContext recreation.

In `RuntimeFilterConsumer::_get_push_exprs()`, `sampling_frequency=32` was set
on a temporary `probe_ctx` VExprContext. However, only VRuntimeFilterWrapper
expressions (VExpr) were returned, not the VExprContext. When
`_append_rf_into_conjuncts()` later created a new VExprContext via
`VExprContext::create_shared(expr)`, the new context had default
`_sampling_frequency=-1` (DISABLE_SAMPLING).

With `_sampling_frequency=-1`, the condition `(_judge_counter++) >= -1`
evaluated to `0 >= -1 → true` on every call, causing `reset_judge_selectivity()`
to fire every time. This meant selectivity counters were perpetually reset
and never accumulated, making the runtime filter selectivity optimization
completely ineffective.

The fix stores `sampling_frequency` in VRuntimeFilterWrapper (which survives
VExprContext recreation) and propagates it to VExprContext in
`VRuntimeFilterWrapper::open()`, which is called on both original and cloned
contexts.

### Release note

Fixed a bug where RuntimeFilter selectivity tracking was non-functional due to
sampling_frequency being lost during VExprContext recreation, causing runtime
filters that should be skipped (due to low selectivity) to never be identified.

### Check List (For Author)

- Test: Unit Test
    - Added 2 regression tests to runtime_filter_selectivity_test.cpp
    - Added 3 new tests in vruntimefilter_wrapper_sampling_test.cpp
    - All 22 tests pass
- Behavior changed: No (selectivity tracking was broken before, this makes it work as designed)
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 10, 2026 11:35
@Thearas
Copy link
Copy Markdown
Contributor

Thearas commented Apr 10, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a BE runtime-filter selectivity regression where sampling_frequency was lost when runtime-filter expressions were re-wrapped into a new VExprContext, causing selectivity counters to reset continuously and making the “always true / can ignore” optimization ineffective.

Changes:

  • Persist sampling_frequency inside VRuntimeFilterWrapper and re-apply it to the active VExprContext during VRuntimeFilterWrapper::open().
  • Update RuntimeFilterConsumer::_get_push_exprs() to pass the computed sampling frequency into each created wrapper (instead of setting it on a temporary probe context).
  • Add regression/unit tests covering default sampling behavior and context recreation.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
be/src/exprs/vruntimefilter_wrapper.h Extend wrapper constructor to carry sampling_frequency and store it as a member.
be/src/exprs/vruntimefilter_wrapper.cpp Propagate wrapper-held sampling_frequency into VExprContext during open().
be/src/exec/runtime_filter/runtime_filter_consumer.cpp Compute sampling frequency once and pass it into wrapper creation paths.
be/test/exec/runtime_filter/runtime_filter_selectivity_test.cpp Add regression tests for disabled vs enabled sampling-frequency behavior.
be/test/exec/runtime_filter/vruntimefilter_wrapper_sampling_test.cpp New tests validating propagation across (re)created VExprContexts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread be/test/exec/runtime_filter/vruntimefilter_wrapper_sampling_test.cpp Outdated
Comment on lines +162 to +169
// Create a brand new VExprContext with the same VRuntimeFilterWrapper.
// The new context starts with default sampling_frequency = -1.
auto context2 = std::make_shared<VExprContext>(wrapper);
EXPECT_FALSE(context2->get_runtime_filter_selectivity().maybe_always_true_can_ignore());

// After open() on the new context, sampling_frequency should be propagated
ASSERT_TRUE(wrapper->open(_runtime_states[0].get(), context2.get(),
FunctionContext::THREAD_LOCAL)
BiteTheDDDDt and others added 2 commits April 10, 2026 20:30
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@BiteTheDDDDt
Copy link
Copy Markdown
Contributor Author

/review

Copy link
Copy Markdown
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 1 issue in the added regression coverage.

  1. be/test/exec/runtime_filter/vruntimefilter_wrapper_sampling_test.cpp: sampling_frequency_survives_context_recreation does not currently verify the post-condition it describes, so it can pass even if propagation to the recreated VExprContext regresses again.

Critical checkpoints

  • Goal of the task: The code change does fix the production bug by moving sampling_frequency onto VRuntimeFilterWrapper and restoring it in VRuntimeFilterWrapper::open(). The new tests partially cover this, but the recreated-context regression test is ineffective as written.
  • Change scope: Small and focused; the production fix is narrowly scoped to runtime-filter wrapper/context state propagation.
  • Concurrency: Applicable. The changed state remains per-VExprContext; I did not find a new shared-state or locking regression in the modified path.
  • Lifecycle / initialization: Applicable. The key lifecycle here is sharing one VExpr across recreated/cloned VExprContexts; the production code now restores state in open(), which is the right hook. The added test does not faithfully exercise that lifecycle.
  • Config changes: None. Existing runtime_filter_sampling_frequency semantics are preserved.
  • Compatibility: No FE/BE protocol, storage-format, or rolling-upgrade compatibility change found.
  • Parallel code paths: Checked the direct conjunct path, recreated-context path, clone path, and scan pushdown attachment path; the production fix covers the relevant runtime-wrapper openings.
  • Special conditions: The disable path (DISABLE_SAMPLING) remains explicit and understandable.
  • Test coverage: Incomplete for the recreated-context regression scenario because one new test lacks an assertion on the propagated state and uses a non-production call order.
  • Observability: Existing counters/logging remain sufficient; no additional observability appears required for this fix.
  • Transaction / persistence / data-write concerns: Not applicable.
  • FE-BE variable propagation: Not applicable.
  • Performance: No material performance concern in the production fix; it adds only a trivial per-open assignment.
  • Other issues: No additional correctness issues found in the modified production code beyond the ineffective regression test.

// Open the recreated context with FRAGMENT_LOCAL to match the real
// _append_rf_into_conjuncts path for a freshly created VExprContext.
ASSERT_TRUE(
wrapper->open(_runtime_states[0].get(), context2.get(), FunctionContext::FRAGMENT_LOCAL)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sampling_frequency_survives_context_recreation is not asserting the behavior it claims to protect. After context2->prepare()/open(), the test should exercise context2->get_runtime_filter_selectivity() and assert that low-selectivity updates can flip maybe_always_true_can_ignore() back to true; otherwise this still passes if propagation into the recreated context breaks again.

There is also a lifecycle mismatch here: _append_rf_into_conjuncts() drives the recreated context through conjunct->prepare(); conjunct->open();, but this test manually calls wrapper->open(..., context2, ...) first. That extra direct wrapper->open() can mask exactly the regression this test is supposed to catch.

@BiteTheDDDDt
Copy link
Copy Markdown
Contributor Author

run buildall

@BiteTheDDDDt BiteTheDDDDt force-pushed the fix-rf-selectivity-sampling-frequency branch from f5d4d1e to f9a109b Compare April 13, 2026 07:35
@BiteTheDDDDt
Copy link
Copy Markdown
Contributor Author

run buildall

…ing test

StubVExpr (leaf node with no children) defaulted to is_constant()=true
via VExpr::is_constant()'s all_of on empty children. On second open()
call (context2), _constant_col was already populated from context1,
causing DCHECK(column_wrapper != nullptr) to fail in get_const_col()
when called with nullptr from VExpr::open().

Fix: override is_constant() to return false, matching SLOT_REF semantics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@BiteTheDDDDt
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 84.62% (11/13) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.05% (20145/37975)
Line Coverage 36.59% (189412/517720)
Region Coverage 32.85% (147091/447728)
Branch Coverage 33.98% (64387/189460)

@hello-stephen
Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (13/13) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.68% (27401/37189)
Line Coverage 57.29% (295720/516136)
Region Coverage 54.41% (245843/451860)
Branch Coverage 56.14% (106696/190042)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 14, 2026
@github-actions
Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 191061a into apache:master Apr 14, 2026
32 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants