JETINFERMeasurements & specs
Every number here was measured on the deployment in service, with the conditions that produced it stated next to it.
Throughput and latency
We run two profiles and never mix them, because what wins on latency loses on throughput. Everything below is the default cheapest profile.
| Measured at the real request shape | Result | Conditions |
|---|---|---|
| Total throughput | 615 tok/s | 29 concurrent, 200-token outputs |
| Per request, idle | 35.1 tok/s | one at a time |
| Per request, under full load | 24.7 tok/s | 29 concurrent |
| Time to first token | 561 ms | through the public endpoint, idle |
| Time to first token, p95 | 1.31 s | 29 concurrent, public endpoint |
| Context / max output | 262,144 / 65,536 | tokens |
Measured 2026-08-22 on the deployment in service. The first-token rows ran through
api.jetinfer.com, so they include the network exactly as you would see
it; the throughput rows ran against the gateway directly, so they don't. The edge
costs about 34 ms of first-token latency and 3% of throughput, measured.
How to read this against anyone else
We test at 6,500 input tokens, which is the production shape. Vendor marketing often quotes a 128-token prompt, where there is almost no prefill to do, and that roughly doubles the headline. Compare like for like.
Prefill runs at roughly 1,810 tokens a second, so your first-token time scales with your prompt unless the prefix is already cached.
| Your prompt | Cold first token | With 90% prefix cached |
|---|---|---|
| 6,500 tokens | 3.6 s | 0.4 s |
| 16,000 tokens | 8.8 s | 0.9 s |
| 32,000 tokens | 17.7 s | 1.8 s |
Pricing
$0.21 in / $2.00 out / $0.021 cached, per million tokens.
The same figures are served live by /v1/models, which is what bills;
if this page and the endpoint ever disagree, the endpoint is right. What bills
how does not move:
| What you send | How it bills |
|---|---|
| Repeated prefix, already cached | one tenth of the input rate |
| Writing a prefix into the cache | the input rate, no surcharge |
| A request we failed to serve | nothing |
Output is never cached, so it is new every time. The opt-in fastest profile is priced separately.
The endpoint
You can call this directly, with credit and a key from your account. Through a router instead, the integration is with them — their base URL, their key, JetInfer picked as the provider. Either way it is a plain OpenAI-compatible surface, streaming and logprobs included. Request shape, error codes and headers are on the docs page.
# Ordinary OpenAI SDK, no special client. client = OpenAI( base_url="https://api.jetinfer.com/v1", api_key="sk-...", ) response = client.chat.completions.create( model="qwen3.8-27b", messages=[{"role": "user", "content": "Summarize this changelog."}], stream=True, )
Supported
Streaming with usage totals in both modes, logprobs and top_logprobs, seed, stop sequences, temperature / top_p / top_k, frequency and presence penalties.
Tool use and structured output
Verified live on 21 August 2026, and re-run against the public endpoint on
every deploy. tool_choice:"auto" returns
finish_reason:"tool_calls" with a valid name and JSON arguments,
streaming included, and response_format returns an object matching
your schema.
Reasoning traces
Thinking output comes back in the reasoning field. It is never
merged into content, and never billed invisibly.
What you are depending on
Served in the EU, data residency verified from the deployment itself. The limits below are the ones that will bind you first.
Capacity
29 concurrent requests, and 600 requests per minute per account, raised on
request. Limits belong to the account, not the key: a second key is a
separate credential, not a second budget.
At saturation you get an immediate 429. Never a queue, never
billed for the wait.
int4 weights, declared
int4 weights (W4A16), declared as
"quantization": "int4" in /v1/models so you can
filter on it. int4 is what makes the price possible.
Context limit
262,144 tokens. Requests above the limit are rejected, never truncated to fit.
When it breaks
We health-check every 15 seconds with a real completion, and anything failing is dropped straight away. A restart is automatic, but four restarts in three hours stops it. That is broken, not unlucky.
No uptime SLA, no redundancy, no third-party monitor, and recovery is manual today. If you need a committed SLA, write to us before you build on this.
Data handling
Prompts and completions are never written to disk, logs, or analytics. We keep operational metadata for 13 months: request ID, key identifier, token counts, latency, status. Nothing is used for training, fine-tuning, or evaluation. It all stays in the EU. Full terms in the privacy policy; DPA on request.
Questions
About anything on this page: hello@jetinfer.com.