DeepSeek to Open Source its Inference Engine

Chinese AI lab DeepSeek on Monday announced its intention to open-source its inference engine. To achieve this, the company is “collaborating closely” with existing open-source projects and frameworks. 

Previously, when the company planned to open-source its inference engine, it identified challenges such as significant codebase divergence from the original framework, extensive infrastructure dependencies, and a limited capacity to maintain a large-scale public project.

The latest announcement further emphasises DeepSeek AI’s dedication to open-sourcing key components and libraries of its models. 

🙏 @deepseek_ai's highly performant inference engine is built on top of vLLM. Now they are open-sourcing the engine the right way: instead of a separate repo, they are bringing changes to the open source community so everyone can immediately benefit!https://t.co/0bEYh8p97W

Recently, during Open Source Week, the company released five high-performance AI infrastructure tools as open-source libraries. These enhance the scalability, deployment, and efficiency of training large language models. 

“It’s an honour to contribute to this thriving [open source] ecosystem and to see our models and code embraced by the community. Together, let’s push the boundaries of AGI and ensure its benefits serve all of humanity,” said DeepSeek in the announcement. 

Recently, the company, in collaboration with Tsinghua University, unveiled a new research study aimed at improving reward modelling in large language models by utilising more inference time compute. This research resulted in a model named DeepSeek-GRM, which the company asserts will be released as open source.

A few weeks ago, DeepSeek released an update for its DeepSeek-V3 model. The updated model, ‘DeepSeek V3-0324’, now ranks highest in benchmarks among all non-reasoning models. 

Artificial Analysis, a platform that benchmarks AI models, stated, “This is the first time an open weights model is the leading non-reasoning model, marking a milestone for open source.” The model scored the highest points among all non-reasoning models on the platform’s ‘Intelligence Index’. 

Recently, Reuters reported that DeepSeek plans to release R2 “as early as possible”. The company initially intended to launch it in early May but is now considering an earlier timeline.

The model is expected to produce “better coding” and can reason in languages beyond English.