Skip to content

Commit 4b6a6ea

Browse files
committed
feat(stepfun): add speech recognition, synthesis, and chat plugin
Add WebSocket and batch SSE speech recognition, streaming and HTTP SSE speech synthesis, and an optional OpenAI-compatible chat wrapper. Support regional endpoints, provider error handling, and conversational stream lifecycle management with regression coverage.
1 parent 4055234 commit 4b6a6ea

16 files changed

Lines changed: 5891 additions & 0 deletions

File tree

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# StepFun plugin for LiveKit Agents
2+
3+
Support for [StepFun](https://platform.stepfun.ai/docs) speech recognition,
4+
speech synthesis, and OpenAI-compatible chat models.
5+
6+
## Installation
7+
8+
```bash
9+
pip install livekit-plugins-stepfun
10+
# Include the OpenAI-compatible chat client:
11+
pip install 'livekit-plugins-stepfun[llm]'
12+
```
13+
14+
## Configuration
15+
16+
Set `STEPFUN_API_KEY`, or pass `api_key=` to the constructors. StepFun runs two
17+
regional clusters with separate endpoints and separate credentials:
18+
19+
| Cluster | Endpoint | Value |
20+
| --- | --- | --- |
21+
| Global (default) | `api.stepfun.ai` | `"global"` |
22+
| Mainland China | `api.stepfun.com` | `"china"` |
23+
24+
Select the cluster with `STEPFUN_CLUSTER` or `cluster=`. The two clusters use
25+
separate endpoints and separate credentials, so a key from one will not work on
26+
the other.
27+
28+
- `stepfun.STT()` streams recognition with server endpointing.
29+
`recognize()` uses the batch route; `use_streaming=False` selects batch
30+
recognition for sessions and requires local VAD through `stt.StreamAdapter`.
31+
Streaming input is intended for paced conversational audio, not file uploads;
32+
use `recognize()` for recorded audio. At end of input, server-VAD streams send
33+
endpointing silence and then a best-effort `input_audio_buffer.commit` (the
34+
gateway currently ignores it with server VAD, but it will finalize the last
35+
segment once commits are forwarded), and keep receiving: while a turn is pending this waits up
36+
to `conn_options.timeout` and fails on expiry, otherwise it ends after a short
37+
quiet period. The API has no acknowledgement that trailing silence was
38+
processed, so completion is best-effort: it succeeds only after a final was
39+
delivered, and otherwise reports that completion cannot be confirmed once a
40+
bounded interval elapses. A turn first reported later than the quiet period is
41+
not recovered. `aclose()` cancels without waiting.
42+
Failures after consuming audio are non-retryable because a fresh session
43+
cannot recover that input. `aclose()` cancels the stream without draining it.
44+
- `stepfun.TTS()` supports streaming and one-shot synthesis. Discover voice IDs
45+
using `await tts.list_voices()`. One-shot synthesis streams PCM over SSE, so
46+
audio starts arriving before the whole request is synthesized; inputs longer
47+
than 1,000 characters are split into sequential requests and emitted as one
48+
audio segment. Each request has an idle-read timeout plus a total deadline
49+
that scales with its text length.
50+
- `stepfun.LLM()` uses the optional OpenAI plugin for chat and tool calling.
51+
52+
Pass `language="zh"` for Chinese; the speech services default to English.
53+
Streaming recognition defaults to `stepaudio-2.5-asr-stream`, batch recognition
54+
to `stepaudio-3-asr-max`, synthesis to `stepaudio-3-tts`, and chat to
55+
`step-3.7-flash`. Custom model IDs are accepted.
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
"""LiveKit Agents plugin for StepFun (阶跃星辰) ASR, TTS and chat models."""
2+
3+
from __future__ import annotations
4+
5+
from typing import TYPE_CHECKING, Any
6+
7+
from livekit.agents import Plugin
8+
9+
from .log import logger
10+
from .models import (
11+
Cluster,
12+
Language,
13+
LLMModels,
14+
ReasoningEffort,
15+
STTBatchModels,
16+
STTModels,
17+
TTSModels,
18+
TTSVoices,
19+
)
20+
from .stt import STT, RecognizeStream, SpeechStream
21+
from .tts import TTS, ChunkedStream, SynthesizeStream
22+
from .version import __version__
23+
24+
# Register the optional OpenAI plugin on the importing thread, before jobs start.
25+
try:
26+
from .llm import LLM
27+
28+
_LLM_IMPORT_ERROR: ImportError | None = None
29+
except ImportError as exc:
30+
_LLM_IMPORT_ERROR = exc
31+
32+
if TYPE_CHECKING:
33+
from .llm import LLM
34+
35+
__all__ = [
36+
"LLM",
37+
"STT",
38+
"TTS",
39+
"ChunkedStream",
40+
"Cluster",
41+
"Language",
42+
"LLMModels",
43+
"ReasoningEffort",
44+
"RecognizeStream",
45+
"STTBatchModels",
46+
"STTModels",
47+
"SpeechStream",
48+
"SynthesizeStream",
49+
"TTSModels",
50+
"TTSVoices",
51+
"__version__",
52+
]
53+
54+
55+
def __getattr__(name: str) -> Any:
56+
"""Explain a missing `llm` extra at access time.
57+
58+
Only reached when the eager import above failed, since a successful one
59+
binds LLM as a module global and module __getattr__ is never consulted for
60+
names that already exist.
61+
"""
62+
if name == "LLM":
63+
assert _LLM_IMPORT_ERROR is not None
64+
raise _LLM_IMPORT_ERROR
65+
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
66+
67+
68+
class StepFunPlugin(Plugin):
69+
def __init__(self) -> None:
70+
super().__init__(__name__, __version__, __package__, logger)
71+
72+
73+
Plugin.register_plugin(StepFunPlugin())
74+
75+
# Hide symbols the doc generator would otherwise pick up
76+
_module = dir()
77+
NOT_IN_ALL = [m for m in _module if m not in __all__]
78+
79+
__pdoc__ = {}
80+
for _n in NOT_IN_ALL:
81+
__pdoc__[_n] = False

0 commit comments

Comments
 (0)