Skip to content

Feature/struct mapping #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
virtexalejandro merged 4 commits into from
May 14, 2026
Merged

Feature/struct mapping #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a3530be
feat(common): add StructMapping + GenericNativeLoader for native C++ …
virtexalejandro May 13, 2026
76f01af
refactor(common): migrate FB and Proto loaders onto GenericLoaderBase
virtexalejandro May 13, 2026
3981473
feat(common): ADL conversion hook for custom client types
virtexalejandro May 14, 2026
5d3f59b
fix: fix clang format
virtexalejandro May 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/SAMPLES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ samples/
generate_replay.cpp vtx_sample_generate arena simulator -> 3 data sources
advance_write.cpp vtx_sample_advance_write 3 data sources -> 3 .vtx replays (with post-processor)

arena_mappings.h JSON data model + JsonMapping<T> + ArenaToVtx::MapFrame()
arena_mappings.h JSON data model + JsonMapping<T> + StructMapping<T> + StructFrameBinding<ArenaFrame>
arena_generated.h autogenerated schema constants + typed Views + Mutators + ForEachX helpers
schemas/
arena_data.proto Protobuf game schema (namespace arena_pb)
Expand Down
28 changes: 18 additions & 10 deletions samples/advance_write.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@
// Mapping strategies demonstrated, one per format:
//
// JSON -> VTX::JsonMapping<T> specializations in arena_mappings.h are
// walked by VTX::UniversalDeserializer to build ArenaReplayJson,
// then ArenaToVtx::MapFrame() produces each VTX::Frame.
// walked by VTX::UniversalDeserializer to build ArenaReplayJson.
// Then VTX::StructMapping<T> + VTX::StructFrameBinding<ArenaFrame>
// drive VTX::GenericNativeLoader::LoadFrame() to produce each
// VTX::Frame. Field addresses come from PropertyAddressCache,
// same as the FBS path below.
//
// Proto -> VTX::ProtoBinding<T> specializations (below) are dispatched by
// VTX::GenericProtobufLoader::LoadFrame(). Field-name lookups go
Expand All @@ -43,6 +46,7 @@
#include "vtx/common/adapters/json/json_adapter.h"
#include "vtx/common/vtx_types_helpers.h"
#include "vtx/common/readers/frame_reader/flatbuffer_loader.h"
#include "vtx/common/readers/frame_reader/native_loader.h"
#include "vtx/common/readers/frame_reader/protobuff_loader.h"
#include "vtx/common/readers/frame_reader/universal_deserializer.h"
#include "vtx/common/readers/schema_reader/schema_registry.h"
Expand Down Expand Up @@ -164,8 +168,8 @@ namespace VTX {
[](const ::arena_pb::Projectile& projectile) { return projectile.unique_id(); });

if (src.has_match_state()) {
loader.AppendSingleActor(entity_bucket, ArenaSchema::MatchState::StructName, src.match_state(),
[](const ::arena_pb::MatchState&) { return std::string("match_001"); });
loader.AppendSingleEntity(entity_bucket, ArenaSchema::MatchState::StructName, src.match_state(),
[](const ::arena_pb::MatchState&) { return std::string("match_001"); });
}
}
};
Expand Down Expand Up @@ -276,13 +280,15 @@ namespace VTX {
// Data source 1 -- JSON
// ===================================================================
// Parses the whole JSON blob on Initialize(), then cursors through the
// deserialized ArenaReplayJson frame-by-frame. The manual MapFrame()
// bridge from arena_mappings.h handles the arena-types -> VTX conversion.
// deserialized ArenaReplayJson frame-by-frame. GenericNativeLoader walks
// StructMapping<> / StructFrameBinding<> from arena_mappings.h to produce
// each VTX::Frame -- same PropertyAddressCache path as the FBS source.

class ArenaJsonDataSource : public VTX::IFrameDataSource {
public:
explicit ArenaJsonDataSource(std::string filepath)
: filepath_(std::move(filepath)) {}
ArenaJsonDataSource(std::string filepath, const VTX::PropertyAddressCache& cache)
: filepath_(std::move(filepath))
, loader_(cache, false) {}

bool Initialize() override {
std::ifstream ifs(filepath_);
Expand Down Expand Up @@ -310,7 +316,8 @@ class ArenaJsonDataSource : public VTX::IFrameDataSource {
}

const ArenaFrame& af = replay_.frames[cursor_++];
out_frame = ArenaToVtx::MapFrame(af);
out_frame = VTX::Frame {};
loader_.LoadFrame<ArenaFrame>(af, out_frame, "ArenaFrame");

out_time = {af.game_time, std::nullopt, VTX::GameTime::EFilterType::OnlyIncreasing};

Expand All @@ -323,6 +330,7 @@ class ArenaJsonDataSource : public VTX::IFrameDataSource {
std::string filepath_;
nlohmann::json root_;
ArenaReplayJson replay_;
VTX::GenericNativeLoader loader_;
size_t total_ = 0;
size_t cursor_ = 0;
};
Expand Down Expand Up @@ -561,7 +569,7 @@ int main() {
VTX_INFO("on every recorded frame; the persisted .vtx contains the post-processed values.");

VTX_INFO("--- 1. JSON data source ---");
ArenaJsonDataSource json_ds(writer_dir + "/arena_replay_data.json");
ArenaJsonDataSource json_ds(writer_dir + "/arena_replay_data.json", arena_schema.GetPropertyCache());
RunPipeline(json_ds, reader_dir + "/arena_from_json_ds.vtx", schema, "arena-adv-json-0001",
VTX::CreateFlatBuffersWriterFacade);

Expand Down
209 changes: 102 additions & 107 deletions samples/arena_mappings.h
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,55 +1,47 @@
#pragma once
// arena_mappings.h -- Arena game data model + JSON mapping for the samples.
// arena_mappings.h -- Arena game data model + mappings (JSON ingest + VTX slot output).
//
// Two concerns live here:
// Two layers of compile-time reflection live here:
//
// 1. ArenaVec3 / ArenaQuat / ArenaPlayer / ArenaProjectile / ArenaMatchState /
// ArenaFrame / ArenaReplayJson
// C++ types that mirror the arena replay JSON structure.
// 1. VTX::JsonMapping<T>
// Drives UniversalDeserializer<>::Load<ArenaReplayJson>(JsonAdapter)
// in advance_write.cpp. Walks JSON -> C++ struct.
//
// 2. VTX::JsonMapping<T> specializations for each of those types.
// Same compile-time reflection pattern used by real integrations
// (see tools/integrations/sf/sf_mappings.h). Consumed by
// VTX::UniversalDeserializer<>::Load<ArenaReplayJson>(JsonAdapter)
// in advance_write.cpp.
// 2. VTX::StructMapping<T> + VTX::StructFrameBinding<ArenaFrame>
// Drives GenericNativeLoader::Load / LoadFrame in advance_write.cpp.
// Walks C++ struct -> VTX::PropertyContainer / VTX::Frame.
//
// 3. ArenaToVtx::MapFrame()
// Hand-written bridge from the arena data model to a VTX::Frame
// (PropertyContainer layout matches content/writer/arena/arena_schema.json).
// This is the "manual" counterpart to the schema-driven ProtoBinding
// and FlatBufferBinding specializations declared in advance_write.cpp.
//
// Property-vector indices assigned here MUST match the field order in
// arena_schema.json -- see the comments above each MapXxx() function.
// The C++ data model uses VTX::Vector / VTX::Quat directly so no per-field
// conversion is needed between the JSON ingest layer and the VTX output layer.
// (This is the equivalent of the manual ArenaToVtx::Map* helpers we deleted --
// the schema-driven StructMapping<> now does that work automatically.)

#include <cstdint>
#include <string>
#include <tuple>
#include <vector>

#include "vtx/common/adapters/json/json_policy.h"
#include "vtx/common/adapters/native/struct_mapping.h"
#include "vtx/common/readers/frame_reader/native_loader.h"
#include "vtx/common/readers/frame_reader/type_traits.h"
#include "vtx/common/vtx_types.h"

#include "arena_generated.h"

// ===================================================================
// Arena game data model (matches the JSON data source structure)
// ===================================================================

struct ArenaVec3 {
double x = 0, y = 0, z = 0;
};
struct ArenaQuat {
float x = 0, y = 0, z = 0, w = 1;
};

struct ArenaPlayer {
std::string unique_id;
std::string name;
int team = 0;
float health = 100.0f;
float armor = 50.0f;
ArenaVec3 position;
ArenaQuat rotation;
ArenaVec3 velocity;
VTX::Vector position;
VTX::Quat rotation;
VTX::Vector velocity;
bool is_alive = true;
int score = 0;
int deaths = 0;
Expand All @@ -58,13 +50,16 @@ struct ArenaPlayer {
struct ArenaProjectile {
std::string unique_id;
std::string owner_id;
ArenaVec3 position;
ArenaVec3 velocity;
VTX::Vector position;
VTX::Vector velocity;
float damage = 25.0f;
std::string type = "bullet";
};

struct ArenaMatchState {
// Default to the canonical id. The match-state JSON doesn't carry one;
// the FB/Proto bindings inject the same literal -- this keeps parity.
std::string unique_id = "match_001";
int score_team1 = 0;
int score_team2 = 0;
int round = 1;
Expand All @@ -90,25 +85,24 @@ struct ArenaReplayJson {
};

// ===================================================================
// JsonMapping<T> specializations (JSON key → C++ member)
//
// Same pattern as sf_mappings.h. Consumed by UniversalDeserializer
// or any code that inspects the mapping tuple at compile time.
// JsonMapping<T> specializations (JSON key -> C++ member)
// ===================================================================
// Two new mappings for the VTX math types so the JSON ingest layer can fill
// them directly. Same shape as the JSON file: {"x":..., "y":..., "z":...}.

template <>
struct VTX::JsonMapping<ArenaVec3> {
struct VTX::JsonMapping<VTX::Vector> {
static constexpr auto GetFields() {
return std::make_tuple(MakeField("x", &ArenaVec3::x), MakeField("y", &ArenaVec3::y),
MakeField("z", &ArenaVec3::z));
return std::make_tuple(MakeField("x", &VTX::Vector::x), MakeField("y", &VTX::Vector::y),
MakeField("z", &VTX::Vector::z));
}
};

template <>
struct VTX::JsonMapping<ArenaQuat> {
struct VTX::JsonMapping<VTX::Quat> {
static constexpr auto GetFields() {
return std::make_tuple(MakeField("x", &ArenaQuat::x), MakeField("y", &ArenaQuat::y),
MakeField("z", &ArenaQuat::z), MakeField("w", &ArenaQuat::w));
return std::make_tuple(MakeField("x", &VTX::Quat::x), MakeField("y", &VTX::Quat::y),
MakeField("z", &VTX::Quat::z), MakeField("w", &VTX::Quat::w));
}
};

Expand Down Expand Up @@ -137,6 +131,9 @@ struct VTX::JsonMapping<ArenaProjectile> {

template <>
struct VTX::JsonMapping<ArenaMatchState> {
// unique_id is intentionally NOT in this mapping -- it isn't in the JSON file.
// The C++ default ("match_001") survives because UniversalDeserializer
// skips missing keys.
static constexpr auto GetFields() {
return std::make_tuple(MakeField("score_team1", &ArenaMatchState::score_team1),
MakeField("score_team2", &ArenaMatchState::score_team2),
Expand Down Expand Up @@ -167,80 +164,78 @@ struct VTX::JsonMapping<ArenaReplayJson> {
};

// ===================================================================
// ArenaToVtx — arena game types → VTX PropertyContainer
//
// Property indices must match the field order in arena_schema.json.
// StructMapping<T> specializations (C++ member -> VTX slot name)
// ===================================================================
// Replaces the hand-written ArenaToVtx::MapPlayer / MapProjectile /
// MapMatchState helpers from the previous version. GenericNativeLoader
// walks these tuples automatically.

namespace ArenaToVtx {

inline VTX::Vector ToVtxVector(const ArenaVec3& v) {
return {v.x, v.y, v.z};
}
inline VTX::Quat ToVtxQuat(const ArenaQuat& q) {
return {q.x, q.y, q.z, q.w};
template <>
struct VTX::StructMapping<ArenaPlayer> {
static constexpr auto GetFields() {
return std::make_tuple(MakeStructField(ArenaSchema::Player::UniqueID, &ArenaPlayer::unique_id),
MakeStructField(ArenaSchema::Player::Name, &ArenaPlayer::name),
MakeStructField(ArenaSchema::Player::Team, &ArenaPlayer::team),
MakeStructField(ArenaSchema::Player::Health, &ArenaPlayer::health),
MakeStructField(ArenaSchema::Player::Armor, &ArenaPlayer::armor),
MakeStructField(ArenaSchema::Player::Position, &ArenaPlayer::position),
MakeStructField(ArenaSchema::Player::Rotation, &ArenaPlayer::rotation),
MakeStructField(ArenaSchema::Player::Velocity, &ArenaPlayer::velocity),
MakeStructField(ArenaSchema::Player::IsAlive, &ArenaPlayer::is_alive),
MakeStructField(ArenaSchema::Player::Score, &ArenaPlayer::score),
MakeStructField(ArenaSchema::Player::Deaths, &ArenaPlayer::deaths));
}
};

/// Player → entity_type_id 0
/// string[0]=UniqueID string[1]=Name
/// int32[0]=Team int32[1]=Score int32[2]=Deaths
/// float[0]=Health float[1]=Armor
/// vector[0]=Position vector[1]=Velocity
/// quat[0]=Rotation
/// bool[0]=IsAlive
inline VTX::PropertyContainer MapPlayer(const ArenaPlayer& p) {
VTX::PropertyContainer pc;
pc.entity_type_id = 0;
pc.string_properties = {p.unique_id, p.name};
pc.int32_properties = {p.team, p.score, p.deaths};
pc.float_properties = {p.health, p.armor};
pc.vector_properties = {ToVtxVector(p.position), ToVtxVector(p.velocity)};
pc.quat_properties = {ToVtxQuat(p.rotation)};
pc.bool_properties = {p.is_alive};
return pc;
template <>
struct VTX::StructMapping<ArenaProjectile> {
static constexpr auto GetFields() {
return std::make_tuple(MakeStructField(ArenaSchema::Projectile::UniqueID, &ArenaProjectile::unique_id),
MakeStructField(ArenaSchema::Projectile::OwnerID, &ArenaProjectile::owner_id),
MakeStructField(ArenaSchema::Projectile::Position, &ArenaProjectile::position),
MakeStructField(ArenaSchema::Projectile::Velocity, &ArenaProjectile::velocity),
MakeStructField(ArenaSchema::Projectile::Damage, &ArenaProjectile::damage),
MakeStructField(ArenaSchema::Projectile::Type, &ArenaProjectile::type));
}
};

/// Projectile → entity_type_id 1
/// string[0]=UniqueID string[1]=OwnerID string[2]=Type
/// vector[0]=Position vector[1]=Velocity
/// float[0]=Damage
inline VTX::PropertyContainer MapProjectile(const ArenaProjectile& pr) {
VTX::PropertyContainer pc;
pc.entity_type_id = 1;
pc.string_properties = {pr.unique_id, pr.owner_id, pr.type};
pc.vector_properties = {ToVtxVector(pr.position), ToVtxVector(pr.velocity)};
pc.float_properties = {pr.damage};
return pc;
template <>
struct VTX::StructMapping<ArenaMatchState> {
static constexpr auto GetFields() {
return std::make_tuple(
MakeStructField(ArenaSchema::MatchState::UniqueID, &ArenaMatchState::unique_id),
MakeStructField(ArenaSchema::MatchState::ScoreTeam1, &ArenaMatchState::score_team1),
MakeStructField(ArenaSchema::MatchState::ScoreTeam2, &ArenaMatchState::score_team2),
MakeStructField(ArenaSchema::MatchState::Round, &ArenaMatchState::round),
MakeStructField(ArenaSchema::MatchState::Phase, &ArenaMatchState::phase),
MakeStructField(ArenaSchema::MatchState::TimeRemaining, &ArenaMatchState::time_remaining));
}
};

/// MatchState → entity_type_id 2
/// string[0]=UniqueID string[1]=Phase
/// int32[0]=ScoreTeam1 int32[1]=ScoreTeam2 int32[2]=Round
/// float[0]=TimeRemaining
inline VTX::PropertyContainer MapMatchState(const ArenaMatchState& m) {
VTX::PropertyContainer pc;
pc.entity_type_id = 2;
pc.string_properties = {"match_001", m.phase};
pc.int32_properties = {m.score_team1, m.score_team2, m.round};
pc.float_properties = {m.time_remaining};
return pc;
}
// ===================================================================
// StructFrameBinding<ArenaFrame> (ArenaFrame -> VTX::Frame buckets)
// ===================================================================
// Same pattern as FlatBufferBinding<arena_fb::FrameData>::TransferToFrame
// and ProtoBinding<arena_pb::FrameData>::TransferToFrame in advance_write.cpp.
// The bucket/entity layout is game-specific; what gets pushed into each
// PropertyContainer is driven automatically by StructMapping<> above.

/// Full frame → single "entity" bucket.
inline VTX::Frame MapFrame(const ArenaFrame& af) {
VTX::Frame frame;
VTX::Bucket& bucket = frame.CreateBucket("entity");
for (const auto& p : af.players) {
bucket.unique_ids.push_back(p.unique_id);
bucket.entities.push_back(MapPlayer(p));
}
for (const auto& pr : af.projectiles) {
bucket.unique_ids.push_back(pr.unique_id);
bucket.entities.push_back(MapProjectile(pr));
}
bucket.unique_ids.push_back("match_001");
bucket.entities.push_back(MapMatchState(af.match_state));
return frame;
template <>
struct VTX::StructFrameBinding<ArenaFrame> {
static void TransferToFrame(const ArenaFrame& src, VTX::Frame& dest, VTX::GenericNativeLoader& loader,
const std::string& /*schema_name*/) {
dest = VTX::Frame {};
VTX::Bucket& bucket = dest.GetBucket("entity");
bucket.entities.clear();
bucket.unique_ids.clear();

loader.AppendActorList(bucket, VTX::ArenaSchema::Player::StructName, src.players,
[](const ArenaPlayer& p) { return p.unique_id; });

loader.AppendActorList(bucket, VTX::ArenaSchema::Projectile::StructName, src.projectiles,
[](const ArenaProjectile& p) { return p.unique_id; });

loader.AppendSingleEntity(bucket, VTX::ArenaSchema::MatchState::StructName, src.match_state,
[](const ArenaMatchState& m) { return m.unique_id; });
}

} // namespace ArenaToVtx
};
Binary file modified samples/content/reader/arena/arena_from_fbs_ds.vtx
View file
Open in desktop
Binary file not shown.
Binary file modified samples/content/reader/arena/arena_from_json_ds.vtx
View file
Open in desktop
Binary file not shown.
Binary file modified samples/content/reader/arena/arena_from_proto_ds.vtx
View file
Open in desktop
Binary file not shown.
Loading
Loading