Expose Frame Packet Trailer (FrameMetadata) for video#286
Merged
Conversation
Surfaces the Rust FFI Frame Packet Trailer feature in the Unity SDK: attach FrameMetadata (user_timestamp, frame_id) to outgoing frames via RtcVideoSource.MetadataProvider, and read it off VideoFrame.Metadata on received frames. PacketTrailerFeatures on TrackPublishOptions is already proto-passthrough so no wrapper change is needed there. Adds a PlayMode E2E test that publishes with both packet-trailer features enabled, attaches known metadata to outgoing frames, and asserts the subscriber sees matching frame_id and user_timestamp. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Provides a Google.Protobuf-free entry point for enabling packet trailer features on TrackPublishOptions. Required because the LiveKit SDK ships Google.Protobuf.dll as an explicitly-referenced plugin (deliberate, to avoid conflicts with com.unity.ai.assistant), so a Unity project's default Assembly-CSharp cannot use the proto's repeated-field collection initializer or call .Add on PacketTrailerFeatures directly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Publishes the local camera with PTF_USER_TIMESTAMP and PTF_FRAME_ID packet trailers and attaches a monotonic frame_id plus UnixTimeUs user_timestamp to each outgoing frame via the new MetadataProvider hook. Logs received trailer values (throttled to ~1 second per stream) for both main and extra-video tiles, so the Unity console / on-screen ScrollingLog shows the metadata round-tripping from remote publishers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…oSource Single test helper with a pushFrames flag covers both modes: signaling-only publication propagation (default) and continuous media flow for round-trip tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Surfaces the Rust FFI Frame Packet Trailer feature on the Unity SDK so apps can attach
FrameMetadata(user_timestamp,frame_id) to outgoing video frames and read it off incoming ones — matching the Python SDK.RtcVideoSource.MetadataProviderpopulatesCaptureVideoFrameRequest.Metadataper outgoing frame;VideoFrame.Metadataexposes the received trailer onVideoStream.FrameReceived.TrackPublishOptionsExtensions.WithPacketTrailerFeatures(...)lets a Unity project's default Assembly-CSharp setPacketTrailerFeatureswithout referencingGoogle.Protobufdirectly (the plugin isisExplicitlyReferenced, deliberately).frame_id+UnixTimeUsuser_timestampto outgoing camera frames and logs received trailer values (throttled to ~1s per stream) in the console / on-screen ScrollingLog.VideoFrameMetadataTeststhat publishes with both trailer features enabled, pushes frames with known metadata, and asserts the subscriber sees matchingframe_id/user_timestamp.Test plan
dotnet buildof LiveKit Runtime, PlayModeTests, and Meet Assembly-CSharp — clean.livekit-server --dev.[Meet RX <peer>] frame_id=… user_ts=…~1×/sec per remote stream and the outgoingframe_idincreases monotonically.🤖 Generated with Claude Code