Skip to content

fix(llm): raise streaming timeout default #1464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rosetta-livekit-bot wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(llm): raise streaming timeout default #1464

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/slow-llamas-read.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@livekit/agents': patch
'@livekit/agents-plugin-baseten': patch
'@livekit/agents-plugin-google': patch
'@livekit/agents-plugin-mistralai': patch
'@livekit/agents-plugin-openai': patch
---

fix(llm): raise the default streaming request timeout to 30s while keeping it configurable through connection options
4 changes: 2 additions & 2 deletions agents/src/inference/llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import OpenAI from 'openai';
import { APIConnectionError, APIStatusError, APITimeoutError } from '../_exceptions.js';
import * as llm from '../llm/index.js';
import type { APIConnectOptions } from '../types.js';
import { DEFAULT_API_CONNECT_OPTIONS } from '../types.js';
import { DEFAULT_LLM_API_CONNECT_OPTIONS } from '../types.js';
import { type Expand, toError } from '../utils.js';
import {
type AnyString,
Expand Down Expand Up @@ -250,7 +250,7 @@ export class LLM extends llm.LLM {
chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
parallelToolCalls,
toolChoice,
inferenceClass,
Expand Down
7 changes: 6 additions & 1 deletion agents/src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@ export const DEFAULT_API_CONNECT_OPTIONS: APIConnectOptions = {
timeoutMs: 10000,
};

export const DEFAULT_LLM_API_CONNECT_OPTIONS: APIConnectOptions = {
...DEFAULT_API_CONNECT_OPTIONS,
timeoutMs: 30000,
};

/**
* Return the interval for the given number of retries.
* The first retry is immediate, and then uses specified retryIntervalMs.
Expand Down Expand Up @@ -65,7 +70,7 @@ export interface ResolvedSessionConnectOptions {

export const DEFAULT_SESSION_CONNECT_OPTIONS: ResolvedSessionConnectOptions = {
sttConnOptions: DEFAULT_API_CONNECT_OPTIONS,
llmConnOptions: DEFAULT_API_CONNECT_OPTIONS,
llmConnOptions: DEFAULT_LLM_API_CONNECT_OPTIONS,
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot May 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 AgentSession still uses 10s LLM timeout base, defeating the PR's 30s default

The PR's stated intent is to raise the default LLM streaming request timeout to 30s. While DEFAULT_SESSION_CONNECT_OPTIONS.llmConnOptions at agents/src/types.ts:73 was updated to use DEFAULT_LLM_API_CONNECT_OPTIONS (30s), the actual resolution of session connect options in AgentSession constructor at agents/src/voice/agent_session.ts:324 still uses DEFAULT_API_CONNECT_OPTIONS (10s) as the base:

llmConnOptions: { ...DEFAULT_API_CONNECT_OPTIONS, ...connOptions?.llmConnOptions },

In the voice pipeline, agent.ts:428 passes activity.agentSession.connOptions.llmConnOptions to llm.chat(), which overrides the plugin-level default. So the LLM timeout in the primary voice pipeline path remains 10s, not 30s. The plugin-level defaults (DEFAULT_LLM_API_CONNECT_OPTIONS) only take effect when llm.chat() is called directly without connOptions.

Prompt for agents 
In agents/src/voice/agent_session.ts line 324, the llmConnOptions base should use DEFAULT_LLM_API_CONNECT_OPTIONS instead of DEFAULT_API_CONNECT_OPTIONS. Import DEFAULT_LLM_API_CONNECT_OPTIONS from '../types.js' and change line 324 from:

  llmConnOptions: { ...DEFAULT_API_CONNECT_OPTIONS, ...connOptions?.llmConnOptions },

to:

  llmConnOptions: { ...DEFAULT_LLM_API_CONNECT_OPTIONS, ...connOptions?.llmConnOptions },

This ensures the voice pipeline (the primary code path for LLM calls) correctly uses the new 30s default timeout.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

ttsConnOptions: DEFAULT_API_CONNECT_OPTIONS,
maxUnrecoverableErrors: 3,
};
4 changes: 2 additions & 2 deletions agents/src/voice/testing/fake_llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import type { ChatContext } from '../../llm/chat_context.js';
import { FunctionCall } from '../../llm/chat_context.js';
import { LLMStream as BaseLLMStream, LLM, type LLMStream } from '../../llm/llm.js';
import type { ToolChoice, ToolContext } from '../../llm/tool_context.js';
import { type APIConnectOptions, DEFAULT_API_CONNECT_OPTIONS } from '../../types.js';
import { type APIConnectOptions, DEFAULT_LLM_API_CONNECT_OPTIONS } from '../../types.js';
import { delay } from '../../utils.js';

export interface FakeLLMResponse {
Expand Down Expand Up @@ -39,7 +39,7 @@ export class FakeLLM extends LLM {
chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
}: {
chatCtx: ChatContext;
toolCtx?: ToolContext;
Expand Down
4 changes: 2 additions & 2 deletions plugins/baseten/src/llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
* Configures the OpenAI plugin to work with Baseten's OpenAI-compatible API
*/
import type { APIConnectOptions } from '@livekit/agents';
import { DEFAULT_API_CONNECT_OPTIONS, inference, llm } from '@livekit/agents';
import { DEFAULT_LLM_API_CONNECT_OPTIONS, inference, llm } from '@livekit/agents';
import { OpenAI } from 'openai';
import type { BasetenLLMOptions } from './types.js';

Expand Down Expand Up @@ -73,7 +73,7 @@ export class OpenAILLM extends llm.LLM {
chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
parallelToolCalls,
toolChoice,
extraKwargs,
Expand Down
4 changes: 2 additions & 2 deletions plugins/google/src/llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import type { APIConnectOptions } from '@livekit/agents';
import {
APIConnectionError,
APIStatusError,
DEFAULT_API_CONNECT_OPTIONS,
DEFAULT_LLM_API_CONNECT_OPTIONS,
llm,
shortuuid,
} from '@livekit/agents';
Expand Down Expand Up @@ -186,7 +186,7 @@ export class LLM extends llm.LLM {
chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
toolChoice,
extraKwargs,
geminiTools,
Expand Down
4 changes: 2 additions & 2 deletions plugins/mistralai/src/llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import type { APIConnectOptions } from '@livekit/agents';
import {
APIConnectionError,
APIStatusError,
DEFAULT_API_CONNECT_OPTIONS,
DEFAULT_LLM_API_CONNECT_OPTIONS,
llm,
shortuuid,
} from '@livekit/agents';
Expand Down Expand Up @@ -118,7 +118,7 @@ export class LLM extends llm.LLM {
chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
toolChoice,
extraKwargs,
}: {
Expand Down
4 changes: 2 additions & 2 deletions plugins/openai/src/llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
//
// SPDX-License-Identifier: Apache-2.0
import type { APIConnectOptions } from '@livekit/agents';
import { DEFAULT_API_CONNECT_OPTIONS, inference, llm } from '@livekit/agents';
import { DEFAULT_LLM_API_CONNECT_OPTIONS, inference, llm } from '@livekit/agents';
import { AzureOpenAI, OpenAI } from 'openai';
import type {
CerebrasChatModels,
Expand Down Expand Up @@ -469,7 +469,7 @@ export class LLM extends llm.LLM {
chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
parallelToolCalls,
toolChoice,
extraKwargs,
Expand Down
6 changes: 3 additions & 3 deletions plugins/openai/src/responses/llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import {
APIConnectionError,
APIStatusError,
APITimeoutError,
DEFAULT_API_CONNECT_OPTIONS,
DEFAULT_LLM_API_CONNECT_OPTIONS,
llm,
log,
toError,
Expand Down Expand Up @@ -78,7 +78,7 @@ class ResponsesHttpLLM extends llm.LLM {
override chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
parallelToolCalls,
toolChoice,
extraKwargs,
Expand Down Expand Up @@ -411,7 +411,7 @@ export class LLM extends llm.LLM {
override chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
parallelToolCalls,
toolChoice,
extraKwargs,
Expand Down
4 changes: 2 additions & 2 deletions plugins/openai/src/ws/llm.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import {
APIStatusError,
APITimeoutError,
ConnectionPool,
DEFAULT_API_CONNECT_OPTIONS,
DEFAULT_LLM_API_CONNECT_OPTIONS,
llm,
stream,
toError,
Expand Down Expand Up @@ -232,7 +232,7 @@ export class WSLLM extends llm.LLM {
chat({
chatCtx,
toolCtx,
connOptions = DEFAULT_API_CONNECT_OPTIONS,
connOptions = DEFAULT_LLM_API_CONNECT_OPTIONS,
parallelToolCalls,
toolChoice,
extraKwargs,
Expand Down
Loading