PHOENIX-7787 Make CCF HAGroupStore ZK Updates backward compatible with existing ZK based client

[ritegarg]

What changes were proposed in this pull request?

Adds an opt-in sync from /phoenix/consistentHA/<G> to the legacy /phoenix/ha/<G> znode in
each region server's HAGroupStoreClient. Each client derives a combined ClusterRoleRecord
(local role + peer role) and writes it to its local /phoenix/ha via ZK stat-version CAS, so
pre-consistentHA ZK-registry clients keep reading a current view of the HA group.

New config keys (in QueryServices / QueryServicesOptions):

• phoenix.ha.legacy.crr.sync.enabled — master switch (default false).
• phoenix.ha.legacy.crr.reconciliation.interval.seconds — periodic reconciliation cadence
  (default 60; 0 disables only the periodic loop, the event-driven path still runs).

Convergence model:

• Event-driven on consistentHA CHILD_ADDED / CHILD_UPDATED (LOCAL or PEER).
• Periodic reconciler on a dedicated single-thread executor (0–30s jitter on first run).
• Curator NodeCache watcher on the legacy znode itself, so per-sync reads are in-memory
  rather than a ZK round-trip. The NodeCache is eventually consistent, so on apparent absence
  the sync path falls back to one authoritative getData() to disambiguate "absent" from
  "not-yet-cached" before deciding CREATE_NEW vs CAS_WITH_VERSION.
• On CAS BadVersion: log at INFO and bail. The NodeCache watch + next event/periodic cycle
  reconverges. No inline retry.

Code-level changes:

• ClusterRoleRecord: new explicit-RegistryType ctor; @JsonCreator now round-trips
  registryType (legacy znode is always written with RegistryType.ZK for backward
  compatibility, and must read back as ZK). New isLogicallyEqualIgnoringVersionAndRegistry
  helper used as the rewrite short-circuit.
• PhoenixHAAdmin:
  • Three new helpers on the legacy namespace:
    • getClusterRoleRecordAndStatInZooKeeper — atomic (data, stat) read via
      storingStatIn.
    • createOrUpdateClusterRoleRecordWithCAS(name, record, LegacyCrrWriteMode mode, int expectedStatVersion) — typed write API. LegacyCrrWriteMode is a public enum with
      values CREATE_NEW, FORCE_OVERWRITE, CAS_WITH_VERSION; the int argument is
      meaningful only for CAS_WITH_VERSION (validated >= 0 at entry). Both BadVersion
      (CAS lost) and NodeExists (create lost) surface as
      StaleClusterRoleRecordVersionException so callers retry uniformly.
  • Atomic-read fix on the existing getHAGroupStoreRecordInZooKeeper (replaces a
    pre-existing 2-RPC pattern that could return a stat version not matching its data
    bytes).
• HAGroupStoreClient: legacyHaAdmin + legacyCrrNodeCache + private legacyCrrSyncLock
  • dedicated periodic executor. Listener hooks for CHILD_ADDED / CHILD_UPDATED.
    CHILD_REMOVED is intentionally a no-op — the legacy /phoenix/ha znode is never deleted
    by this client.
• New StaleClusterRoleRecordVersionException (CRR-specific analog of
  StaleHAGroupStoreRecordVersionException).

Why are the changes needed?

The Consistent Cluster Failover work moved HA group state from /phoenix/ha (which held a
ClusterRoleRecord) to /phoenix/consistentHA (which holds an HAGroupStoreRecord and uses
an RPC-registry view of CRRs). Existing Phoenix clients built against the older ZK-registry
contract still read /phoenix/ha/<G> and expect:

• The znode to exist for any live HA group.
• The record's registryType to be ZK.
• The roles to reflect the latest state of both clusters.

Without this patch, after a CCF rollout the /phoenix/ha znode goes stale or absent, and old
ZK-registry clients break. This change keeps /phoenix/ha in sync with the new authoritative
/phoenix/consistentHA records on a per-RS basis, gated by an opt-in flag so it only runs
where the operator needs it.

Does this PR introduce any user-facing change?

Yes, but the new behavior is gated entirely behind the new master switch
(phoenix.ha.legacy.crr.sync.enabled), which defaults to false.

When the flag is enabled:

• Each HAGroupStoreClient writes to (and maintains) the legacy /phoenix/ha/<G> znode on
  its local ZK.
• Old pre-consistentHA ZK-registry clients pointed at /phoenix/ha will see fresh records
  again.

One incidental visible change independent of the flag: ClusterRoleRecord JSON
deserialization now respects the persisted registryType field. Previously it was hard-coded
to RPC regardless of JSON content. Existing JSON without a registryType field still
defaults to RPC, so deployments not exposed to legacy-sync writes see no behavior change.

How was this patch tested?

• 13 integration tests in HAGroupStoreClientIT covering: feature-off no-op, initial create,
  role-changing transition, state-only transition (no rewrite), peer-driven role flip,
  peer-absent → role2=UNKNOWN → recovery, pre-existing stale znode overwrite,
  CAS error mapping + write-mode dispatch (CREATE_NEW, FORCE_OVERWRITE,
  CAS_WITH_VERSION negative-version rejection), LOCAL/PEER CHILD_REMOVED
  (znode preserved), registryType stays ZK across multiple role cycles, reconciliation
  interval =0 (event-driven still works), and periodic-loop recovery after an external
  divergence.
• 4 new unit tests in ClusterRoleRecordTest covering JSON backward-compat: missing
  registryType defaults to RPC, explicit RPC round-trips, explicit ZK round-trips, and
  full ZK toJson/fromJson round-trip preserves registryType. Backed by three JSON
  fixtures under phoenix-core/src/test/resources/json/.
• Full HAGroupStoreClientIT: 39/39 pass.
• ClusterRoleRecordTest (unit): 16/16 pass.
• mvn spotless:check clean across the touched modules.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude (Anthropic).