Inference CLI reference#

areno serve

Start an OpenAI-compatible HTTP server backed by the Areno inference engine. The server exposes /v1/chat/completions, accepts standard chat-completion tools fields, and keeps one rollout session open for the process lifetime so rollout state and CUDA graph state can be reused across requests.

areno serve \
  --model-path /path/to/hf/checkpoint \
  --tp-size 1 \
  --world-size 1 \
  --host 0.0.0.0 \
  --port 8000

areno serve#

Serve chat completions.

Options:

--model-path TEXT: Local checkpoint/tokenizer path or remote model repo ID. Required.
--model-hub [hf|modelscope]: Remote hub used when --model-path is not a local path. Use --model-hub hf for Hugging Face and --model-hub modelscope for ModelScope. Default: modelscope.
--tp-size INTEGER: Tensor parallel size. Default: 1.
--world-size INTEGER: Total number of local worker ranks. Default: 1.
--host TEXT: HTTP bind host. Default: 0.0.0.0.
--port INTEGER: HTTP bind port. Default: 8000.
--max-running-prompts INTEGER: Maximum concurrent rollout prompts per request chunk. Default: 16.
--default-max-tokens INTEGER: Default max generated tokens when requests omit a token budget. Default: 1024.
--decode-progress-interval-s FLOAT: Worker decode progress log interval. Default: 0.0.
--eager-decode: Disable decode CUDA graph and run rollout decode eagerly.
--attn-backend [flash|native]: Attention backend. Default: flash. Use native to run without flash-attn on the areno_accel native compatibility path. AReno automatically falls back to native on flash-attn-unsupported GPUs such as Tesla T4 and prints a warning. native is slower than flash on supported GPUs.
--disable-thinking: Pass enable_thinking=False to tokenizer chat templates when supported. Use this when serving a model whose chat template supports a thinking-mode switch and you want normal responses without reasoning spans. Tokenizers that do not accept enable_thinking automatically fall back to their normal chat-template call.

world-size must be divisible by tp-size.

Examples#

Single-rank server#

areno serve \
  --model-path /path/to/model \
  --tp-size 1 \
  --world-size 1 \
  --port 8000

TP4 server#

areno serve \
  --model-path /path/to/model \
  --tp-size 4 \
  --world-size 4 \
  --port 8000

Chat completion request#

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "areno",
    "messages": [
      {"role": "user", "content": "Solve 12 * 13."}
    ],
    "max_tokens": 128,
    "temperature": 0.0
  }'

Request fields#

POST /v1/chat/completions

Field	Type	Description
`model`	`str \| None`	Optional model name echoed by the client.
`messages`	`list[ChatMessage]`	Required chat messages.
`max_tokens`	`int \| None`	Generated token budget.
`max_completion_tokens`	`int \| None`	Alternative generated token budget.
`temperature`	`float`	Sampling temperature. Defaults to `1.0`; use `0.0` for greedy decoding.
`top_p`	`float`	Nucleus sampling threshold.
`top_k`	`int`	Top-k sampling threshold. Defaults to `-1`; non-positive values disable top-k filtering.
`n`	`int`	Number of completions per prompt.
`stream`	`bool`	Streaming flag. `true` is not supported.
`stop`	`str \| list[str] \| None`	Stop string or list of stop strings.
`seed`	`int \| None`	Deterministic sampling seed when sampling is enabled.
`tools`	`list[Tool] \| None`	OpenAI-compatible function tools. The same model-native tool-call parser used by agentic rollout converts generated tool-call text into `message.tool_calls` for supported model families.
`tool_choice`	`str \| dict \| None`	Optional tool-choice directive, including a forced function name.

ChatMessage fields:

role: Usually system, user, assistant, or tool.
content: Message content as str | list | None.

Multimodal content#

For a checkpoint with a compatible processor, content may be a list of text, image, audio, and video parts. AReno accepts OpenAI-style *_url parts and forwards normalized media references to the checkpoint processor:

{
  "role": "user",
  "content": [
    {"type": "video_url", "video_url": {"url": "/data/clip.mp4"}},
    {"type": "audio_url", "audio_url": {"url": "/data/clip.wav"}},
    {"type": "text", "text": "Describe the synchronized event."}
  ]
}

Supported part types are image_url, audio_url, video_url, and input_audio. The direct processor forms image, audio, and video are also accepted. input_audio contains base64 data and a format such as wav or mp3. Local paths are resolved on the server, not on the client, so the media files must be visible to the serving process. Data URLs can be used when the client and server do not share a filesystem.

Media support depends on the loaded model processor. See Multimodal inputs for serving and training guidance.

Continuous batching behavior#

The server runs inside a long-lived rollout session. Compatible requests can be admitted into an active worker decode loop through continuous batching; requests with different generation settings are scheduled separately. Requests are compatible when these fields match:

generated token budget
temperature
top-p
top-k
seed
stop token ids
EOS token id

Requests with different generation settings are scheduled separately.

Decode progress logs#

Set --decode-progress-interval-s to a positive value to print worker decode progress:

rollout decode progress: dp=0/4 active=32 cuda_graph=True tokens_per_second=2810.7

tokens_per_second is the scheduled decode throughput for that DP worker in the reporting window. It excludes prefill and is not the same as end-to-end request throughput. cuda_graph=True means the worker used CUDA graph replay for at least one decode step in that window; False means the window ran eagerly.

Tool calls#

areno serve supports the Chat Completions tool-call shape and reuses the same tool-call parser as agentic rollout:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="areno",
    messages=[{"role": "user", "content": "Choose a move: left or right."}],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "choose_move",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "direction": {"type": "string", "enum": ["left", "right"]},
                    },
                    "required": ["direction"],
                },
            },
        }
    ],
    tool_choice={"type": "function", "function": {"name": "choose_move"}},
)

print(response.choices[0].message.tool_calls)

Tool-call parsing is selected from the model/tokenizer family. Current parsers cover Qwen/Qwen3.5/MiniCPM-style <tool_call> blocks, Gemma4 tool-call blocks, and generic JSON tool-call output.

Help#

areno serve --help