tok/s missing with external llama.cpp api connection

[rozsazoltan]

Hi,

I noticed that Unsloth Studio shows tok/s when using the built-in llama.cpp integration, but not when I connect to my own llama.cpp / ik_llama.cpp server through the Connection API.

Streaming works, and the server is started with --metrics. The /metrics endpoint also works.

Is this expected? Does the built-in integration use extra llama.cpp timing/usage data that the external Connection API currently does not request or forward?

It would be nice if externally connected llama.cpp servers could show the same tok/s stats too.

Default	Connection API

[Icaro0310]

Most likely expected behavior. --metrics only enables the Prometheus /metrics endpoint, it doesn't put per-request timing into the response body. tok/s comes from the timings object llama.cpp returns on its native API, or from usage in the final SSE chunk on the OpenAI-compatible endpoint, and that last one only appears if the client sends stream_options: {"include_usage": true}.

My guess: the built-in integration reads timings directly, while the Connection API path goes through the OpenAI-compatible layer and Studio doesn't request or parse the usage chunk.

To confirm, stream a request and inspect the last chunk:

curl -N http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"hi"}],"stream":true,"stream_options":{"include_usage":true}}'

If there's no usage/timings in the output, Studio has nothing to compute tok/s from.

—
Related work of mine: github.com/Icaro0310 · Website: icaro0310.github.io

[rozsazoltan]

Your comments are really bizarre with these ads. Honestly, I couldn't care less about them, and I'm just adding them to my ad blocker.

[vectornguyen76]

Studio only shows tok/s when the external server sends back its usage chunk, and it only asks for that chunk on some connection types. On current releases the llama.cpp preset gets it and a Custom connection doesn't. Older builds didn't ask for it on llama.cpp connections either.

--metrics makes no difference because Studio never reads /metrics. The badge comes from the timings object llama-server attaches to its last SSE chunk. The frontend keeps those timings only when they arrive on the chunk that carries usage (chat-adapter.ts, final metadata). If there are no timings, it shows total stream time instead.

That chunk only exists when the request sets stream_options.include_usage, and the backend adds the field per provider type (external_provider.py):

_USAGE_STREAM_OPTION_PROVIDERS = frozenset(
    {"vllm", "llama_cpp", "ollama", "openrouter", "kimi", "lemonade", "qwen", "qwencloud"}
)

custom is deliberately left out because arbitrary OpenAI-compatible servers may reject the field. llama_cpp was added in v0.1.903-beta (Oct 6); in v0.1.902 and earlier the set was only {"vllm", "openrouter", "kimi", "lemonade"}. With include_usage, llama-server appends an empty-choices chunk with usage and puts timings on it (server-task.cpp).

To fix it:

1. Update Studio to v0.1.903-beta or newer.
2. Add a new connection and pick llama.cpp in "Choose a connection" (presets), using your server's /v1 URL. You can't change the type of an existing connection because the selector is disabled while editing.

I verified this with upstream llama.cpp, not ik_llama.cpp. To check your fork, run curl -N against /v1/chat/completions with "stream":true,"stream_options":{"include_usage":true} and look for timings next to usage in the last chunk.