/api/client/register causes PostgreSQL lock contention (Unleash OSS 7.4.1)

[davispalomino]

Describe the bug

We are observing sustained PostgreSQL lock contention (“blocked connections / lock waits”) caused by concurrent writes from the Unleash server to the client_applications table. During peak events, multiple sessions wait on wait_event_type='Lock' / wait_event='transactionid' for tens of seconds.

Using pg_blocking_pids(), the top blockers consistently show a batch INSERT into client_applications (app_name, seen_at, updated_at) (likely an upsert due to the PK on app_name). The problem becomes pronounced when 20+ pods register in parallel, for example during rollouts/restarts or autoscaling events.

Per Unleash docs, SDKs call POST /api/client/register on startup to register their existence (appName, strategies, SDK version, etc.).

Steps to reproduce the bug

1. Deploy Unleash OSS 7.4.1 pointing to PostgreSQL.
2. Run many client services/SDKs in Kubernetes with the same appName (or multiple appNames) and high replica counts (e.g., 10–20+ pods per app).
3. Trigger a high-concurrency event (e.g., rollout restart, mass redeploy, or HPA scale-up) so that 20+ pods register in parallel and hit POST /api/client/register around the same time.
4. Observe in PostgreSQL:
   • increased sessions with wait_event_type='Lock'
   • wait_event='transactionid'
   • blockers with query insert into "client_applications"...
5. (Optional) Confirm in Unleash UI Project → Applications that some applications report high “instances” counts.

Expected behavior

Under high instance count (many pods/SDKs), client registration should not cause sustained DB lock contention or long lock waits (tens of seconds). Ideally, the system should debounce/batch these writes and/or minimize row-level conflicts when many instances register concurrently.

Logs, error output, etc.

### PostgreSQL: constraint (PK) on `client_applications`

SELECT conname, contype, pg_get_constraintdef(oid) AS def
FROM pg_constraint
WHERE conrelid = 'client_applications'::regclass;


Expected output:
- `client_applications_pkey` PRIMARY KEY (`app_name`)

### PostgreSQL: lock summary (snapshot)

SELECT
  now() AS ts,
  count(*) AS waiting_sessions,
  max(now() - query_start) AS max_wait,
  avg(extract(epoch FROM (now() - query_start)))::int AS avg_wait_seconds
FROM pg_stat_activity
WHERE wait_event_type = 'Lock';


Example observed during incidents:
- `waiting_sessions`: ~15–17
- `max_wait`: ~25–47s
- `avg_wait_seconds`: ~8–12s

### PostgreSQL: top blockers (confirms blocking query)

SELECT
  now() AS ts,
  b.pid AS blocker_pid,
  b.client_addr,
  b.application_name,
  now() - b.xact_start AS blocker_xact_age,
  now() - b.query_start AS blocker_query_age,
  b.wait_event_type,
  b.wait_event,
  left(b.query, 200) AS blocker_query,
  count(*) AS blocked_sessions
FROM pg_stat_activity a
JOIN pg_stat_activity b
  ON b.pid = ANY (pg_blocking_pids(a.pid))
WHERE a.wait_event_type = 'Lock'
GROUP BY 1,2,3,4,5,6,7,8,9
ORDER BY blocked_sessions DESC, blocker_xact_age DESC;


Example (truncated):
- `insert into "client_applications" ("app_name","seen_at","updated_at") values (...),(...),...`
- `wait_event='transactionid'`
- one blocker can block ~10–12 sessions

### Note on `relation = n/a` in lock breakdowns
When analyzing locks via `pg_locks.relation`, `relation` may appear as `NULL` (“n/a”) because `transactionid`-based waits do not necessarily map to a relation OID in `pg_locks.relation`.

Screenshots

No response

Additional context

• Unleash version: 7.4.1 (Open Source)
• Hosting: self-hosted on Kubernetes
• Database: PostgreSQL (managed)
• Traffic pattern: multiple client services/SDKs; 20+ pods registering in parallel during rollouts/restarts/autoscaling.
• UI correlation: in Project → Applications, some apps show high “instances” counts (10–20+), correlating with increased registration concurrency.

Questions for maintainers

1. Is this pattern (transactionid lock waits caused by concurrent client_applications inserts/upserts) a known issue at scale?
2. Are there recommended settings/patterns for high-scale Kubernetes deployments to avoid sustained contention?
   • e.g., server-side debouncing/batching for registration writes, different upsert strategy, async persistence, etc.
3. Any recommended configuration for:
   • DB pool sizing (DATABASE_POOL_MIN/MAX)
   • rate limiting for /api/client/register
   • reducing registration write pressure without losing critical visibility?

What we can provide if needed

• Full (untruncated) blocker_query text
• Time series of waiting_sessions / max_wait
• Unleash replica count and DB pool configuration
• Ingress counts for POST /api/client/register during incident windows

Unleash version

7.4.1 (Open Source)

Subscription type

Open source

Hosting type

Self-hosted

SDK information (language and version)

Node.js unleash-client ^6.9.4

[gastonfournier]

Thanks for the detailed report and diagnostics.

From what you shared, this appears to be an enterprise-scale workload pattern (large SDK fleet plus synchronized registration bursts), which is beyond what OSS is typically designed for.

We do have Enterprise customers successfully operating at this scale, including with Unleash Enterprise Edge to offload significant traffic/work from core Unleash instances.

Given your scale and contention pattern, we recommend moving to Unleash Enterprise. If you’re open to that, we can coordinate an evaluation/migration path and connect you with our sales team so we can proceed with the right support track.

[davispalomino]

Hi @gastonfournier , thanks for the explanation and recommendation.

To move forward in parallel and help mitigate the contention pattern we observed, I’ve opened a PR with the proposed change for OSS:
• PR: #11394

Could you please review it and confirm whether it’s something you’d be willing to merge?
If you need benchmarks, load test results, or any adjustments to better align with your standards/approach, let me know and I’ll update it.

I understand that given the scale you recommend Unleash Enterprise (and Edge) and I’m open to evaluating that, but I’d also like to address this OSS point to reduce immediate impact and capture the learnings.

Thanks.

[gastonfournier]

Hi @davispalomino, thanks for the contribution, they're always valuable, and we'll look into it.

I was also looking at our current customer footprint: the average Unleash replicas/pods per customer is ~3, and the current observed maximum is 7 (with ~99% of customers at 4 or fewer). So while high-scale workloads are supported in Enterprise with the right setup, this reported topology is outside what we typically target for OSS defaults. Because of that, we want to be careful about merging behavior changes that could introduce other risks (e.g. consistency/side effects) for the broader OSS user base.

As a practical short-term solution, you can fork the OSS repo and tune this behavior for your environment, either by increasing the write/save interval beyond 10s or by applying the suggested patch in your fork and building your own image. That lets you validate impact in your workload without us introducing broader OSS behavior changes that may create other consistency or operational tradeoffs.

Regardless, I'll bring your proposal to the team and get back to you once we have something to share.

[gastonfournier]

Hi @davispalomino we came up with this proposal, which should help mitigate the issue (hopefully by half): #11400. We'll likely be cutting a release today or tomorrow due to a CVE with the docker image, and this will go out as part of that patch release.

[davispalomino]

Hi @gastonfournier , In our case, this topology is driven by Kubernetes autoscaling and parallel deployments, which are fairly common in modern cloud-native environments.

We’ll test patch #11400 as soon as it’s available and validate the impact under our real workload. In the meantime, we’ll also evaluate increasing the write/save interval to mitigate contention.

I appreciate you bringing the proposal to the team. From our perspective, this pattern may become more common in dynamically scaled environments, so any improvement that reduces lock contention under high concurrency would be very valuable for OSS as well.

Happy to share additional metrics if that helps with the evaluation.

Thanks again.

[gastonfournier]

Ok, the release 7.4.2 is out and includes #11400

In our case, this topology is driven by Kubernetes autoscaling and parallel deployments, which are fairly common in modern cloud-native environments.

We understand this as we also run our infra in k8s. But with years of improvements and optimizations, we can handle similar workloads during rolling deployments with a maximum of 10 pods.

We really look forward to unblocking your team with this release and starting a conversation about how to better support your needs in the future.

Please let us know if this improved the lock contention issue.

[davispalomino]

Hi @gastonfournier , Thanks for the update!
We’ve just deployed v7.4.2 (including #11400) to our environment and we’ll be monitoring/validating Unleash behavior, especially around lock contention during Kubernetes rollouts and autoscaling.
We’ll report back here once we have results, or sooner if we notice any anomalies.