Skip to content

[feat][broker] PIP-475: synthetic-layout lookup for regular topics#25822

Open
merlimat wants to merge 1 commit into
apache:masterfrom
merlimat:pip-475-impl-1
Open

[feat][broker] PIP-475: synthetic-layout lookup for regular topics#25822
merlimat wants to merge 1 commit into
apache:masterfrom
merlimat:pip-475-impl-1

Conversation

@merlimat
Copy link
Copy Markdown
Contributor

Summary

First implementation PR for PIP-475: Regular-to-Scalable Topic Migration. Lays the broker-side foundation: the scalable-topic lookup session now accepts persistent:// and short-form topic names in addition to topic://, and synthesises a layout that wraps an existing regular (partitioned or non-partitioned) topic as one or more special segments. V5 clients can then operate against any topic through the scalable surface, even before it has been migrated — the SDK consumption of the synthetic layout will land in a follow-up PR.

Wire-format additions

  • SegmentInfoProto.underlying_topic_name — when set, the segment wraps the named persistent://... topic instead of having its own segment://... URI.
  • CommandScalableTopicUpdate.resolved_topic_name — the canonical topic://t/n/x identity, set on every successful response so a client that looked up via persistent://... or short-form learns the resolved name.

Java surface

  • SegmentInfo.underlyingTopicName + activeSpecial(...) factory + isSpecial() check.
  • TopicName.toScalableTopic() — derive the canonical scalable identity from a name in any input domain.

Broker behaviour

  • ServerCnx rejects non-persistent:// scalable-topic lookups early.
  • DagWatchSession.start() falls back to a synthetic layout for persistent:// input when no scalable metadata exists at the canonical path:
    • Non-partitioned → 1 special segment covering the full hash range.
    • Partitioned with N partitions → N special segments wrapping each persistent://...-partition-K with equal-width contiguous hash ranges.
  • The metadata-store watch remains active so a later migration that writes scalable metadata to the same path is observed and transparently replaces the synthetic layout.
  • pushUpdate emits the canonical topic://... resolvedTopicName, plus the new special-segment marker when set.

Test plan

  • DagWatchSessionTest covers synthetic layouts for both non-partitioned and partitioned regular topics, and verifies the wire response carries resolved_topic_name and underlying_topic_name (17/17 passing).
  • CommandsScalableTopicTest verifies resolved_topic_name round-trips through the new Commands.newScalableTopicUpdate signature (8/8 passing).
  • ScalableTopicControllerTest regression check (33/33 passing).
  • End-to-end V5 SDK test against synthetic layouts — lands with the SDK consumption PR.

Adds the broker-side foundation for PIP-475 (regular-to-scalable topic
migration). The scalable-topic lookup session now accepts persistent://
and short-form topic names in addition to topic://, and synthesises a
layout that wraps an existing regular (partitioned or non-partitioned)
topic as one or more special segments. V5 clients can then operate
against any topic through the scalable surface, even before it has been
migrated.

Schema additions:
 * SegmentInfoProto.underlying_topic_name — when set, the segment wraps
   the named persistent://... topic instead of having its own
   segment://... URI. Used by the synthetic-layout response.
 * CommandScalableTopicUpdate.resolved_topic_name — the canonical
   topic://t/n/x identity, set on every successful response so a
   client that looked up via persistent://... or short-form learns
   the resolved name.

Java surface mirrors the wire additions on SegmentInfo (with a new
activeSpecial(...) factory and isSpecial() check) and a new
TopicName.toScalableTopic() helper used by the broker to derive the
canonical identity regardless of input form.

Broker behaviour:
 * ServerCnx rejects non-persistent:// scalable-topic lookups early.
 * DagWatchSession.start() falls back to a synthetic layout for
   persistent:// input when no scalable metadata exists at the
   canonical path. Non-partitioned → 1 special segment covering the
   full hash range; partitioned with N partitions → N special segments
   wrapping each persistent://...-partition-K with equal-width
   contiguous hash ranges. The metadata-store watch remains active so
   a later migration that writes scalable metadata to the same path
   is observed and transparently replaces the synthetic layout.
 * pushUpdate emits the canonical topic://... resolvedTopicName, plus
   the new special-segment marker when set.

Tests:
 * DagWatchSessionTest covers synthetic layouts for both
   non-partitioned and partitioned regular topics, and verifies the
   wire response carries resolved_topic_name and underlying_topic_name.
 * CommandsScalableTopicTest verifies resolved_topic_name round-trips
   through the new Commands.newScalableTopicUpdate signature.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant