[feat][broker] PIP-475: synthetic-layout lookup for regular topics#25822
Open
merlimat wants to merge 1 commit into
Open
[feat][broker] PIP-475: synthetic-layout lookup for regular topics#25822merlimat wants to merge 1 commit into
merlimat wants to merge 1 commit into
Conversation
Adds the broker-side foundation for PIP-475 (regular-to-scalable topic migration). The scalable-topic lookup session now accepts persistent:// and short-form topic names in addition to topic://, and synthesises a layout that wraps an existing regular (partitioned or non-partitioned) topic as one or more special segments. V5 clients can then operate against any topic through the scalable surface, even before it has been migrated. Schema additions: * SegmentInfoProto.underlying_topic_name — when set, the segment wraps the named persistent://... topic instead of having its own segment://... URI. Used by the synthetic-layout response. * CommandScalableTopicUpdate.resolved_topic_name — the canonical topic://t/n/x identity, set on every successful response so a client that looked up via persistent://... or short-form learns the resolved name. Java surface mirrors the wire additions on SegmentInfo (with a new activeSpecial(...) factory and isSpecial() check) and a new TopicName.toScalableTopic() helper used by the broker to derive the canonical identity regardless of input form. Broker behaviour: * ServerCnx rejects non-persistent:// scalable-topic lookups early. * DagWatchSession.start() falls back to a synthetic layout for persistent:// input when no scalable metadata exists at the canonical path. Non-partitioned → 1 special segment covering the full hash range; partitioned with N partitions → N special segments wrapping each persistent://...-partition-K with equal-width contiguous hash ranges. The metadata-store watch remains active so a later migration that writes scalable metadata to the same path is observed and transparently replaces the synthetic layout. * pushUpdate emits the canonical topic://... resolvedTopicName, plus the new special-segment marker when set. Tests: * DagWatchSessionTest covers synthetic layouts for both non-partitioned and partitioned regular topics, and verifies the wire response carries resolved_topic_name and underlying_topic_name. * CommandsScalableTopicTest verifies resolved_topic_name round-trips through the new Commands.newScalableTopicUpdate signature.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First implementation PR for PIP-475: Regular-to-Scalable Topic Migration. Lays the broker-side foundation: the scalable-topic lookup session now accepts
persistent://and short-form topic names in addition totopic://, and synthesises a layout that wraps an existing regular (partitioned or non-partitioned) topic as one or more special segments. V5 clients can then operate against any topic through the scalable surface, even before it has been migrated — the SDK consumption of the synthetic layout will land in a follow-up PR.Wire-format additions
SegmentInfoProto.underlying_topic_name— when set, the segment wraps the namedpersistent://...topic instead of having its ownsegment://...URI.CommandScalableTopicUpdate.resolved_topic_name— the canonicaltopic://t/n/xidentity, set on every successful response so a client that looked up viapersistent://...or short-form learns the resolved name.Java surface
SegmentInfo.underlyingTopicName+activeSpecial(...)factory +isSpecial()check.TopicName.toScalableTopic()— derive the canonical scalable identity from a name in any input domain.Broker behaviour
ServerCnxrejectsnon-persistent://scalable-topic lookups early.DagWatchSession.start()falls back to a synthetic layout forpersistent://input when no scalable metadata exists at the canonical path:persistent://...-partition-Kwith equal-width contiguous hash ranges.pushUpdateemits the canonicaltopic://...resolvedTopicName, plus the new special-segment marker when set.Test plan
DagWatchSessionTestcovers synthetic layouts for both non-partitioned and partitioned regular topics, and verifies the wire response carriesresolved_topic_nameandunderlying_topic_name(17/17 passing).CommandsScalableTopicTestverifiesresolved_topic_nameround-trips through the newCommands.newScalableTopicUpdatesignature (8/8 passing).ScalableTopicControllerTestregression check (33/33 passing).