Blog
DeepSeek open-sources DeepInfer, the software that runs its AI models
DeepSeek has released DeepInfer, the inference engine behind its production AI service, as open source under Apache 2.0. It supports DeepSeek-V4.1-Flash on Nvidia Blackwell and Huawei Ascend 950 chips. Key parts, including KV cache storage and load balancing, are not included, and DeepSeek's cost figures are its own.
Chinese AI company DeepSeek has released DeepInfer, the software it uses to run its own AI models for paying customers, as open source. The code went up on GitHub on 20 October 2026 under the Apache 2.0 licence, which allows commercial use. "We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable," the Hangzhou-based company said on X, as reported by Bloomberg.
What an inference engine does
Building an AI model is only half the job. Every time someone sends a question, the model has to be run on expensive chips to produce an answer. This step is called inference, and for popular services it is where much of the day-to-day cost sits. An inference engine is the software that decides how requests are grouped, how the model is split across many chips, and how earlier work is reused so it does not have to be repeated.
DeepSeek says DeepInfer is "the high-performance inference framework that powers DeepSeek's large-scale production services". According to its own documentation, a single DeepSeek cluster currently serves 155 trillion tokens a day (a token is a small piece of text) and earns about US$38 per GPU-hour at off-peak prices. These are the company's own figures and have not been independently checked.
What is in the release
According to the GitHub repository, the first release:
- Supports one model: DeepSeek-V4.1-Flash. DeepSeek says code for future models will be added to the same project.
- Runs on two kinds of hardware: Nvidia's newest Blackwell-generation (SM100) GPUs and Huawei's Ascend 950 chips. Support for a Chinese chip alongside Nvidia is notable, given US limits on exporting top Nvidia chips to China.
- Includes speed tricks such as splitting the model's "experts" across many chips, guessing several words ahead and then checking them (speculative decoding), and structured output that forces answers into a set format such as JSON.
- Offers repeatable results: an optional mode that gives identical answers to identical inputs, on Nvidia hardware only.
Code headers in the repository say parts of DeepInfer are "derived from vLLM", a widely used open-source inference project, and modified by DeepSeek.
DeepSeek also released deepseek-recipe, a component that converts requests between common AI API formats and the model's internal tokens. The company says it behaves exactly like the official DeepSeek API, so other hosts can offer the same behaviour.
What is missing
This is not a complete, ready-to-run service. DeepSeek states that three parts needed for a full deployment are not included: the system that stores and reuses past computations (the KV cache), the split of work between separate "prefill" and "decode" machines, and load balancing across copies of the model. The company says these are tightly tied to its internal systems, and it has described their design in a technical write-up instead.
The scale is also large. DeepSeek's deployment guide says the smallest efficient setup is about 64 GPUs for prefill plus 64 for decode. That puts the full approach out of reach for small teams, though cloud providers and large companies could use it.
Why it matters
DeepSeek's models are already known for low prices. Sharing the serving software could help other providers run DeepSeek models more cheaply and match the official API more closely. It also shows how one of China's leading AI labs gets high efficiency from its hardware, information most Western labs keep private.
The release comes in the same week the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products. That order covers US government systems; it does not stop businesses elsewhere from using DeepSeek's open code. How widely DeepInfer is adopted will depend on whether outside teams can match DeepSeek's reported numbers on their own hardware.