Skip to content

api.md

Latest commit

 

History

History
198 lines (154 loc) · 6.96 KB

api.md

File metadata and controls

198 lines (154 loc) · 6.96 KB
Raw

JoyAI-VL-Interaction

Chinese documentation: api.zh-CN.md

JoyAI-VL-Interaction is JoyAI's in-house real-time video stream interaction model. It supports video frame input, session state management, and output normalization.

API Information

Item Value
Base URL https://joyrouter.jd.com
Endpoint POST /v1/chat/completions
Session reset POST /v1/streaming/reset
Model name JoyAI-VL-Interaction
Protocol OpenAI Chat Completions compatible
Streaming Not supported (synchronous JSON only)
Authentication Authorization: Bearer $JOYAI_API_KEY

Key Features

  • Video frame input: Supports single-frame and batch-frame input. A time range can be specified for each frame.
  • Session state management: Uses the x-streaming-session header to isolate sessions and accumulate memory.
  • Output normalization: Normalizes model output to either </silence> (no response) or </response> one sentence (response).

Request Parameters

Headers

Header Required Description
Content-Type Yes application/json
Authorization Yes Bearer $JOYAI_API_KEY
x-streaming-session No Session ID. Providing one is strongly recommended; otherwise, requests share the default session. Only letters, digits, ._-, and a maximum of 120 characters are allowed.
x-system-prompt-key No Selects the system prompt: DEFAULT_SYSTEM_PROMPT_EN (with delegation) or DEFAULT_SYSTEM_PROMPT_NO_DELEGATION (without delegation).
x-frame-time-range No Time range for a single frame, for example, "10.0 seconds ~ 12.0 seconds".

Body

Field Type Required Description
messages array Yes Standard OpenAI messages array. The text in the last user message is used as the user's question.
model string No Backend model name. The default backend is used when omitted.
frame_time_ranges string[] No Array of time ranges for batch frames, in the same order as the images.
max_tokens int No Maximum number of tokens to generate.
temperature float No Sampling temperature.
top_p float No Nucleus sampling probability.

Image Input Format

Each image is an item in the content array. A single user message may contain multiple images for batch-frame input:

{ "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,/9j/4AAQ..." } }

Time Range Resolution Priority

  1. The frame_time_ranges array in the request body, mapped to images in order
  2. The x-frame-time-range header, used as a single value for the first frame
  3. A <X seconds ~ Y seconds> pattern extracted from the prompt text
  4. An automatically generated range based on FRAME_SECONDS (default: 1.0 second)

Examples

curl

curl -X POST https://joyrouter.jd.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JOYAI_API_KEY" \
  -H "x-streaming-session: session-001" \
  -d '{
    "model": "JoyAI-VL-Interaction",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Who is in the scene, and what are they doing?"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ..."}}
      ]
    }]
  }'

Python

import base64

import requests

URL = "https://joyrouter.jd.com/v1/chat/completions"
API_KEY = "your-api-key"


def image_to_data_url(path):
    with open(path, "rb") as f:
        b64 = base64.b64encode(f.read()).decode()
    return f"data:image/jpeg;base64,{b64}"


resp = requests.post(URL, headers={
    "Content-Type": "application/json",
    "Authorization": f"Bearer {API_KEY}",
    "x-streaming-session": "session-001"
}, json={
    "model": "JoyAI-VL-Interaction",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Who is in the scene, and what are they doing?"},
            {"type": "image_url", "image_url": {"url": image_to_data_url("frame.jpg")}}
        ]
    }]
})

print(resp.json()["choices"][0]["message"]["content"])

Batch Frames

curl -X POST https://joyrouter.jd.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JOYAI_API_KEY" \
  -H "x-streaming-session: session-001" \
  -d '{
    "model": "JoyAI-VL-Interaction",
    "frame_time_ranges": ["10.0 seconds ~ 11.0 seconds", "11.0 seconds ~ 12.0 seconds"],
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What changed between these two frames?"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
      ]
    }]
  }'

Response Format

{
  "id": "chatcmpl-a1b2c3d4...",
  "object": "chat.completion",
  "model": "streaming-infer-adapter",
  "choices": [{
    "index": 0,
    "message": { "role": "assistant", "content": "</response> A man is working on a laptop at a desk." },
    "finish_reason": "stop"
  }],
  "usage": { "prompt_tokens": 1200, "completion_tokens": 30, "total_tokens": 1230 },
  "streamingharness": {
    "main_model": "JoyAI-VL-Interaction",
    "raw_content": "</response> A man is working on a laptop at a desk.",
    "timing": { "adapter_total_ms": 150.3, "vllm_inference_ms": 115.0 }
  }
}

Output Normalization Rules

The content field is normalized to one of two possible results:

Output Meaning Trigger
</silence> The model chooses not to respond. There is no user question, the main model output contains </silence>, or the output is empty.
</response> <one sentence> The model chooses to respond. The main model output contains </response>; the first line following it is used as the response.

The raw main model output is available in streamingharness.raw_content.

Session Management

Session ID Resolution Priority

  1. The x-streaming-session header
  2. The x-session-id header
  3. The user field in the request body
  4. "default" as the fallback

Reset a Session

curl -X POST https://joyrouter.jd.com/v1/streaming/reset?model=JoyAI-VL-Interaction \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $JOYAI_API_KEY" \
  -H "x-streaming-session: session-001" \
  -d '{}'

Response: { "ok": true, "session_id": "session-001", "removed": true }

Notes

  • No streaming support: All endpoints return synchronous JSON and do not support SSE.
  • Session isolation: Providing x-streaming-session is strongly recommended. Otherwise, requests share the default session and state may leak between clients.
  • Text-only requests: Requests without images are forwarded directly to the main model. The response does not include the extended timing or memory fields.
  • Silence when no question is provided: If the request contains no user question text, the API returns </silence> immediately.