ccp_nodemx_mem_request metric exposes values in kilobytes

[aeroyorch]

Overview

The ccp_nodemx_mem_request metric exposed by the OTEL-based sidecar reports values in kilobytes, but its description incorrectly states that it reports values in bytes.

Environment

Please provide the following details:

• Platform: Rancher
• Platform Version: v1.29.13+rke2r1
• PGO Image Tag: ubi8-16.8-0
• Postgres Version 16

Steps to Reproduce

REPRO

Provide steps to get to the error condition:

1. Activate OTEL metrics and resources in Postgres instance:

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  name: myinstance
spec:
  postgresVersion: 16
  instrumentation: {}
  image: "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:ubi8-16.8-0"
  instances:
  - name: instance1
    replicas: 3
    resources:
      requests:
        cpu: "4"
        memory: "32Gi"

1. Get metrics from Prometheus

2. Here the extraction from the collector sidecar config:

 - description: Memory request value in bytes
   metric_name: ccp_nodemx_mem_request
   static_attributes:
     server: localhost:5432
   value_column: request

EXPECTED

1. Metrics provided by ccp_nodemx_mem_request in bytes, as the ccp_nodemx_mem_limit does.

ACTUAL

1. Metrics are in KiB:

$ kubectl get pod PODNAME -o jsonpath='{.spec.containers[*].resources.requests.memory}'
32Gi

Prometheus output: 32768

[andrewlecuyer]

Hi @aeroyorch, thanks for submitting this! You are correct - ccp_nodemx_mem_request is not in bytes, even though the description says it is.

I'm therefore flagging this one as bug, and will be taking a look at next steps to get this corrected so that ccp_nodemx_mem_request returns in bytes. At a glance, it looks like we may want/need to remove the divisor that is being set for the mem_request Downward API volume, which is currently set to 1Mi:

postgres-operator/internal/postgres/reconcile.go

Line 140 in c7842e7

Divisor: oneMebibyte,

[aeroyorch]

Hi @andrewlecuyer ,

First of all, thank you for your quick response!

Just a quick comment regarding the divisor: I see that the same divisor is used in the code above for mem_limit, but in this case, the ccp_nodemx_mem_limit value is represented correctly:

{__name__=~"ccp_nodemx_mem_limit|ccp_nodemx_mem_request"} > 0

ccp_nodemx_mem_limit: 34359738368 # 32 GiB -> bytes
ccp_nodemx_mem_request: 32768 # 32 GiB -> MiB

So I'm not sure if the inconsistency could be coming from another part of the code.

[andrewlecuyer]

Diving deeper into the SQL behind ccp_nodemx_mem_request and ccp_nodemx_mem_limit, you can see that while the request comes from the Downward API, the limit is pulled from the cgroup v2 information:

postgres-operator/internal/collector/postgres_5s_metrics.yaml

Lines 336 to 345 in c7842e7

	- sql: >
	WITH d(key, val) as (SELECT key, val FROM monitor.cgroup_setof_kv('memory.stat'))
	SELECT
	monitor.kdapi_scalar_bigint('mem_request') AS request,
	CASE
	WHEN monitor.cgroup_mode() = 'legacy' THEN
	(CASE WHEN monitor.cgroup_scalar_bigint('memory.limit_in_bytes') = 9223372036854771712 THEN 0 ELSE monitor.cgroup_scalar_bigint('memory.limit_in_bytes') END)
	ELSE
	(CASE WHEN monitor.cgroup_scalar_bigint('memory.max') = 9223372036854775807 THEN 0 ELSE monitor.cgroup_scalar_bigint('memory.max') END)
	END AS limit,

So this accounts for the difference you noted. I.e., the request comes from the Downward API (which currently uses Mi), while the limit comes from cgroup v2 (which uses bytes).

As for a fix for this, please see the following PR: #4231. Any thoughts/feedback are welcome!

[benjaminjb]

Diving deeper into the SQL behind ccp_nodemx_mem_request and ccp_nodemx_mem_limit, you can see that while the request comes from the Downward API, the limit is pulled from the cgroup v2 information:

postgres-operator/internal/collector/postgres_5s_metrics.yaml

Lines 336 to 345 in c7842e7

• sql: >
  WITH d(key, val) as (SELECT key, val FROM monitor.cgroup_setof_kv('memory.stat'))
  SELECT
  monitor.kdapi_scalar_bigint('mem_request') AS request,
  CASE
  WHEN monitor.cgroup_mode() = 'legacy' THEN
  (CASE WHEN monitor.cgroup_scalar_bigint('memory.limit_in_bytes') = 9223372036854771712 THEN 0 ELSE monitor.cgroup_scalar_bigint('memory.limit_in_bytes') END)
  ELSE
  (CASE WHEN monitor.cgroup_scalar_bigint('memory.max') = 9223372036854775807 THEN 0 ELSE monitor.cgroup_scalar_bigint('memory.max') END)
  END AS limit,
  So this accounts for the difference you noted. I.e., the request comes from the Downward API (which currently uses Mi), while the limit comes from cgroup v2 (which uses bytes).

As for a fix for this, please see the following PR: #4231. Any thoughts/feedback are welcome!

Huh, I wonder if anything is using the Downward API mem_limit then?

[benjaminjb]

Hello, @aeroyorch, we recently released CPK 5.8.3, where we have removed the divisor for the Downward API. Thanks again for bringing this up, and hope this fix helps you.

I'm going to close this issue now, but please reopen it if there's something wrong with this still.

ETA. Here's the release notes with the specific fix called out.

[aeroyorch]

Hello @benjaminjb! Thanks a lot for the quick fix and for sharing the release notes! 🙌 I’ll test it out on my side, and reopen if anything else comes up.