On October 20, DeepSeek published DeepInfer, the inference framework that serves DeepSeek V4.1 Flash. The repository had 5,691 stars when I checked, hours after the announcement. DeepSeek calls it "our in-house inference framework, now fully open source" (@deepseek_ai).
Then the README lands. This release is offline inference only. It runs on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs. The smallest efficient deployment in DeepSeek's own Q&A is 64 prefill GPUs plus 64 decode GPUs.
So you will not host V4.1 Flash on the card in your tower this weekend. That does not make the release dull. DeepSeek shipped V4.1-Flash three days ago. This is the serving half of that story: the kernels, the scheduler, and the cache tricks, with the boring parts left out.
TL;DR
- DeepInfer is the engine, not the service. No KV cache tier, no prefill/decode split, no load balancer.
- The code that exists is real and runnable, on SM100 or Ascend 950 hardware at cluster scale.
deepseek-recipeis the usable half: DeepSeek's own prompt rendering and tokenizer, MIT licensed.
What DeepSeek actually open-sourced
Two repositories shipped side by side, and they do different jobs.
| Repo | What it does | License | Who can use it |
|---|---|---|---|
| DeepInfer | Inference engine, kernels, scheduling, structured output | Apache-2.0 | Teams with SM100 or Ascend 950 |
| deepseek-recipe | API protocol to DeepSeek prompt conversion, tokenizer, streaming parser | MIT | Anyone building an API gateway |
DeepInfer is Python on top of C++ and TileLang kernels. The acknowledgments are blunt about its ancestry: it was built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. That matters for how you read the release. It is a heavily modified vLLM-lineage engine, and the interest is in the diff.
The headline number comes from DeepSeek's deployment write-up. One cluster serves 155 trillion tokens per day, at $38 of revenue per GPU-hour at off-peak pricing. Commenters on the Hacker News thread noted that revenue is not margin. Fair point, and DeepSeek never claims otherwise.
The hardware bill arrives before line one
The supported list is short and expensive.
| Requirement | Value |
|---|---|
| Accelerator | NVIDIA SM100 GPU, or Huawei Ascend 950 NPU |
| OS | Linux |
| Compiler | GCC 13 or newer |
| Python | 3.12 |
| CUDA | 13 (NVIDIA path) |
| CANN | 9.2 (Ascend path) |
The published benchmark ran on 4 SM100 GPUs for prefill and 64 for decode. The Q&A is more direct about deployment. Decode wants EP64 to EP128, and the prefill-to-decode ratio lands near 1:1. So the smallest efficient unit is 64 prefill GPUs plus 64 decode GPUs. A single prefill engine and a single decode engine will boot, but do not call that production.
If your inventory is eight H100s, this release is a reading assignment, not a deployment.
How to run it, in the order that works
These commands are from the repo. I have not executed them, because I do not own an SM100 node. Treat every timing claim below as DeepSeek's, not mine.
1. Build it. Start from an empty Python 3.12 environment. The docs stress "empty", and the build takes a while.
git clone https://github.com/deepseek-ai/DeepInfer
cd DeepInfer
export DEEPINFER_ACCELERATOR=nvidia # or: ascend
./build.sh
2. Run offline inference. The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017 on Hugging Face. It downloads on first run.
python examples/offline_inference.py --ep 8
python examples/offline_inference.py --ep 2 --dp 8
python examples/offline_inference.py --ep 2 --cp 8
Point DEEPINFER_HF_MODEL_DIR at a local copy to skip the download. Add --check_structured_output to test regex, JSON, and EBNF constraints after the sample outputs.
3. Benchmark your own hardware. examples/benchmark.py keeps prefill and decode separate so the two numbers do not contaminate each other.
export WORLD_SIZE=1 RANK=0
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
python examples/benchmark.py --cp 8 --prefill_batch_size 65536
Set PROFILE=1 to capture a trace. The docs warn that profiler overhead inflates step times. Use the trace to inspect kernel composition, not to claim tokens per second.
4. Accept that the last step is yours. There is no OpenAI-compatible server in this release. There is no router, no cache tier, and no autoscaler. DeepInfer gives you an engine plus examples. The HTTP layer is your problem.
The five settings worth understanding
Most of the repository is plumbing. These flags are where the performance lives.
| Variable | What it changes | Default |
|---|---|---|
DEEPINFER_ENABLE_MTP_VARLEN | Turns on DSpark variable-length speculative verification | off |
DEEPINFER_USE_NEXTN_DECODE | Maximum draft length per step | 5 |
DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE | Replays a bounded suffix instead of the full prompt | compact |
DEEPINFER_YOCO_PREFILL_KEEP_TOKENS | Size of that suffix | 140 |
DEEPINFER_BATCH_INVARIANT | Same output regardless of batch composition (NVIDIA only) | off |
DSpark is the clever one. A fixed draft length of 5 spends verification compute on tokens the model is unlikely to accept. DeepSeek trains a confidence head, sorts draft tokens by expected acceptance, and verifies only the prefix that pays for itself. The Q&A reports the number of verified draft tokens per step dropping from 5.00 to 3.2-4.4, while accepted length barely moves, 3.29 to 3.20. DeepSeek measures a 4-7% throughput gain at 60 concurrent sequences per rank, and 6% in production.
Reproducibility needs configuration. DEEPINFER_BATCH_INVARIANT=1 keeps results identical across batch compositions, and it is NVIDIA-only. For fully deterministic sampling, add DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed. If you publish benchmark numbers, that is the setup to document.
deepseek-recipe is the half you can use today
This is the repository I would install first.
python3 -m pip install deepseek-recipe
from deepseek_recipe import ChatCompletionRequest, ConversionOptions, DeepseekV41Encoding
request = ChatCompletionRequest({
"model": "DeepSeek-V4.1-Flash",
"messages": [{"role": "user", "content": "Hello"}],
})
converted = request.convert(ConversionOptions())
rendered = DeepseekV41Encoding().render_conversation(converted.conversation)
print(rendered.prompt)
The library converts Chat Completions, Responses, and Messages requests into one conversation format. It renders the prompt for V4 or V4.1, then parses model output back into each protocol, including streaming and tool calls. It covers thinking mode, images, response_format with JSON Schema or regex, and the Chat Completions prefix continuation field.
Two details show this is production code, not a demo. Special tokens in user text are encoded as ordinary text, so a prompt cannot inject a role marker. And tokenization runs per segment between special tokens, which cuts latency on long prompts.
The library is Rust with Python bindings, MIT licensed, and the PyPI release is 0.2.0. When a request omits max_tokens, recipe fills in 8,192 outside thinking mode, 65,536 inside it, and 131,072 with max effort. You still supply inference, tool execution, and transport. The repo ships Axum and FastAPI example servers with mock inference. Swap the mock for a real backend, and you have a gateway that behaves like DeepSeek's own endpoint.
If you are wiring an agent harness to a cheap model, the API-parity question is the same one from the open-weights pile-up. Recipe is the first time DeepSeek answered it in code.
What is missing, and why that is the story
The README lists three things a complete service needs and this release does not include.
| Missing piece | What DeepSeek runs internally |
|---|---|
| KV cache system | About 100 PiB of SSD on 3FS, with TTL of 3 days for agent traffic and 7 days for chatbot traffic |
| Vision encoder, prefill, decode disaggregation | Separate pools per stage, each with its own load balancer |
| Load balancing across replicas | Prefill, decode, vision encoder, and expert-parallel balancers |
The KV cache tier is the big one. DeepSeek keeps the block cache on a dedicated 3FS cluster instead of host memory, so any prefill machine can hit any prefix. Logical size is about 14.4 TB per GPU. The sliding-window cache is a separate thing. It stays in the memory of the machine that produced it, at 1,415,040 bytes per sequence for V4.1.
A commenter on Hacker News summed it up: "just enough to stir the pot but not enough to cook your own."
That is rektlessness, and it is the right read. DeepSeek says those parts are tightly coupled to internal infrastructure. Believe that or not, the effect is the same. The cheap-serving half is described in prose, not shipped as code.
Three ideas you can steal without the GPUs
The engine needs a cluster. The ideas do not.
Replay a suffix, not the prompt. SWA bounded replay recomputes only the last 128 tokens of a cached prompt, then truncates the attention window to that segment. For decoder layers, that replaces a 2,560-token replay with 128 tokens. DeepSeek says it nearly halves total prefill compute. If you run prefix caching anywhere, this is the trick to copy.
Compile grammars off the hot path. DeepInfer compiles JSON Schema, regex, or EBNF into an automaton on a separate machine, asynchronously. At decode time it advances the automaton and masks illegal tokens. Structured output stops being a latency tax.
Treat tokenizer boundaries as a security boundary. deepseek-recipe encodes special tokens by position and treats the same strings in user text as plain text. That kills a class of prompt-injection bugs that most gateways still have.
What I would do
If you use the DeepSeek API, change nothing. The API did not move, and reading the repository will not shrink your bill.
If you run a multi-provider gateway, install deepseek-recipe this week. MIT license, Python bindings included, and it gives your users DeepSeek-identical prompts and streaming. That is the highest-value artifact in the release.
If you own SM100 or Ascend 950 capacity, clone DeepInfer, run the benchmark, and budget the cache tier yourself. The 64 + 64 GPU figure is the honest floor for the published throughput. Anything smaller is a lab experiment, which is fine as long as you call it that.
Everyone else should read doc/inference_deployment_qa.md. It is free, specific, and better than most paid systems talks. Start with the Wide-EP section and the SWA cache section.
Watch the repository for the KV cache code. If that lands, the floor drops and the "open source but not deployable" complaint dies with it. Until then, DeepInfer is a blueprint with a working engine attached. That earns the stars, not a migration.
I keep the model-side comparisons in AI tools. Start with V4.1 Flash and the MiniCPM and OpenCode bundle wars.
Related reading
- DeepSeek V4.1 Flash, Stolen Keys, and the Harness Wars - DeepSeek V4.1 Flash undercuts frontier models on price, OpenCode fights a key-stealing distillation attack, an
- Claude vs ChatGPT vs Gemini: Real Developer Workflows Compared (2026 Edition) - Stop asking 'which is best.' Ask 'which is best for THIS task?' We tested Claude 3.5 Sonnet, GPT-4o, and Gemin
- Cursor IDE Deep Dive: The AI Code Editor That Actually Works (2026 Review) - VS Code is great, but Cursor is what happens when an IDE actually understands your code.
- Local LLMs for Privacy: Running Ollama, LM Studio, and llama.cpp (2026 Guide) - Stop sending your private data to OpenAI.
More in AI Tools.
Sources
- DeepSeek announcement thread — October 20, 2026
- deepseek-ai/DeepInfer — README, examples, Apache-2.0
- DeepInfer deployment Q&A — cluster, cache, DSpark, SWA numbers
- DeepSeek V4.1 Flash deployment write-up — Chinese, 155T tokens/day and $38/GPU-hour
- deepseek-ai/deepseek-recipe — Rust libraries and Python bindings, MIT
- deepseek-recipe on PyPI — version 0.2.0
- DeepSeek-V4.1-Flash-1017 on Hugging Face — MIT, 1M context
- Hacker News discussion — 498 points and 173 comments at the time of writing
- vLLM, SGLang, TensorRT-LLM — acknowledged lineage
Raja CRN
Tech enthusiast and developer.