Node agents `resampleStream()` creates Sox resampler even when input/output sample rates match, causing high native memory growth

[snickerdudle]

Describe the bug

Hi LiveKit team,

We’re seeing significant RSS/native memory growth in a Node.js LiveKit Agents service during active audio calls. After instrumenting the call path, the growth appears strongly correlated with AudioResampler / Sox FFI calls from resampleStream() even when the input audio sample rate already matches the requested output sample rate.

Environment

• Runtime: Node.js LiveKit Agents
• Packages:
  • @livekit/agents 1.2.7
  • @livekit/rtc-node 0.13.x
  • @livekit/agents-plugin-deepgram 1.2.7
• Audio path:
  • ParticipantAudioInputStream
  • AudioStream
  • resampleStream()
  • STT stream

Observed Behavior

During active calls, process RSS grows continuously. In one production observation, memory grew to just under 5 GB over roughly 90 minutes while calls were active, then plateaued after calls stopped.

We suspected LiveKit audio/native resources, so we added local instrumentation around:

• process.memoryUsage()
• FFI request/event counts
• FFI listener counts
• AudioFrame.fromOwnedInfo()
• AudioSource.captureFrame()
• AudioSource.close()
• AudioResampler.push()
• AudioResampler.flush()
• AudioResampler.close()

In a local single-call run, JS heap and tracked JS audio frames were not the main issue. After teardown:

• FFI listeners returned to 0
• AudioSource instances closed
• pending captureFrame() calls returned to 0
• live tracked audio frame bytes returned to ~0

However, FFI request volume was dominated by Sox resampling.

Before bypassing same-rate resampling, we saw output like:

topFfiRequests=pushSoxResampler:27420,captureAudioFrame:1374,...
topFfiEvents=audioStreamEvent:9263,...
ffiEventListeners=0
audioSourcesClosed=2
pendingCaptureFrameCalls=0

After adding a local monkey patch to bypass AudioResampler.push() when inputRate === outputRate, pushSoxResampler disappeared from the hot path. The bypass counter climbed exactly with input frames instead:

topFfiRequests=captureAudioFrame:875,...
sameRateResamplersBypassed=1
sameRateResamplerPushesBypassed=8206
ffiEventListeners=0
audioSourcesClosed=2
pendingCaptureFrameCalls=0

RSS also returned near baseline after teardown in the local reproduction with the bypass enabled.

Suspected Cause

@livekit/agents/src/utils.ts currently creates and uses an AudioResampler unconditionally inside resampleStream():

export function resampleStream({
  stream,
  outputRate,
}: {
  stream: ReadableStream<AudioFrame>;
  outputRate: number;
}): ReadableStream<AudioFrame> {
  let resampler: AudioResampler | null = null;
  const transformStream = new TransformStream<AudioFrame, AudioFrame>({
    transform(chunk, controller) {
      if (chunk.samplesPerChannel === 0) {
        controller.enqueue(chunk);
        return;
      }
      if (!resampler) {
        resampler = new AudioResampler(chunk.sampleRate, outputRate);
      }
      for (const frame of resampler.push(chunk)) {
        controller.enqueue(frame);
      }
    },
    flush(controller) {
      if (resampler) {
        for (const frame of resampler.flush()) {
          controller.enqueue(frame);
        }
        resampler.close();
      }
    },
  });
  return stream.pipeThrough(transformStream);
}

This means new AudioResampler(chunk.sampleRate, outputRate) and pushSoxResampler are used even when chunk.sampleRate === outputRate.

In our call path, ParticipantAudioInputStream creates an AudioStream with the target sample rate, then still wraps it in resampleStream({ outputRate: this.sampleRate }). That appears to cause a same-rate Sox resampling pass for every audio frame.

Suggested Fix

Short-circuit resampleStream() when the incoming frame sample rate already matches outputRate.

For example:

export function resampleStream({
  stream,
  outputRate,
}: {
  stream: ReadableStream<AudioFrame>;
  outputRate: number;
}): ReadableStream<AudioFrame> {
  let resampler: AudioResampler | null = null;

  const transformStream = new TransformStream<AudioFrame, AudioFrame>({
    transform(chunk, controller) {
      if (chunk.samplesPerChannel === 0 || chunk.sampleRate === outputRate) {
        controller.enqueue(chunk);
        return;
      }

      if (!resampler || resampler.inputRate !== chunk.sampleRate) {
        resampler?.close();
        resampler = new AudioResampler(chunk.sampleRate, outputRate);
      }

      for (const frame of resampler.push(chunk)) {
        controller.enqueue(frame);
      }
    },

    flush(controller) {
      if (!resampler) return;

      for (const frame of resampler.flush()) {
        controller.enqueue(frame);
      }
      resampler.close();
    },
  });

  return stream.pipeThrough(transformStream);
}

A simpler variant would also help:

if (chunk.sampleRate === outputRate) {
  controller.enqueue(chunk);
  return;
}

before constructing/pushing through the resampler.

Why This Matters

This avoids:

• unnecessary native Sox resampler allocation
• high-frequency pushSoxResampler FFI calls
• extra buffer copying through FfiClient.copyBuffer()
• native memory pressure during long-running/concurrent calls

In our service, this dramatically reduced FFI traffic and stopped the observed local RSS growth pattern.

Additional Notes

This may be especially visible in long-running production agents or concurrent-call workloads where audio frames continuously flow through ParticipantAudioInputStream.

Happy to provide more diagnostic output or test a patch if helpful.

Relevant log output

No response

Describe your environment

Environment

• Runtime: Node.js LiveKit Agents
• Packages:
  • @livekit/agents 1.2.7
  • @livekit/rtc-node 0.13.x
  • @livekit/agents-plugin-deepgram 1.2.7
• Audio path:
  • ParticipantAudioInputStream
  • AudioStream
  • resampleStream()
  • STT stream

Minimal reproducible example

No response

Additional information

No response