AboutAI ToolsCursorClaude CodeLovableBolt.newAI tool reviewsTutorialsVibe codingAI agentsMobile apps with AIAll tutorialsBlogSponsorWork With MeJoin 285K+ Subscribers

5 November 2026 · 4 min read

Baidu open-sources ERNIE 5.5 Flash Base, a 280B-parameter MoE model for developers to fine-tune

On 5 November 2026 Baidu released ERNIE 5.5 Flash Base, a pre-training checkpoint with 280B total parameters and 14B active per token. Weights are on Hugging Face under a custom licence that restricts commercial use and geography.

On 5 November 2026 Baidu released the weights for ERNIE 5.5 Flash Base on Hugging Face [1][2][3]. This is a pure pre-training checkpoint — no instruction tuning, no RLHF, no mid-training [2]. A post-trained version will follow, but the company has not said when [1][2].

What is new in the architecture

The model is a text-only mixture-of-experts (MoE) system with 280 billion total parameters. Only 14 billion activate per token [2][4].

It stacks 43 Transformer layers with a hidden size of 4,096 and 64 attention heads [2]. Attention is split into two types:

  • 37 layers use Heavily Compressed Attention (HCA). They compress every 128-token block into a single cache entry and attend over that compressed sequence plus a short window of recent uncompressed tokens. This keeps the KV cache small [2][4].
  • 6 layers use Multi-head Latent Attention (MLA) combined with sparse attention. An indexer selects the top‑k historic positions for each query, so attention cost stays constant as context grows [2].

The HCA layers use wide 512‑dimension heads, while the compute‑heavy MLA layers use narrow 256‑dimension heads — a deliberate allocation of compute budgets [4]. A low‑rank gated attention mechanism cuts the gating projection’s parameter count by 75% [4].

Every layer except the first has a Multi‑Router Latent MoE block with 512 routed experts. Ten experts are activated per token, plus one shared expert that always fires [2][4]. The experts operate in a 2,048‑dimensional latent space rather than the model’s full width, roughly halving dispatch and combination traffic and expert FLOPs [2]. Multiple independently initialised sigmoid routers select experts, which raises the correlation between routing scores and actual utility [4].

Pre‑training used 33.5 trillion tokens with the Muon optimizer and FP8 mixed precision [2][3]. The context length was gradually extended to 128K tokens during training; fine‑tuning and inference support up to 1 million tokens [1][2]. The training data includes multilingual web text, code, and software development records (commits, pull requests, reviews, CI‑verified fixes) to support repository‑level reasoning and agentic coding tasks [2]. Baidu also added long‑document data and synthetic retrieval sequences for long‑context training [2].

Baidu’s FlashMask‑4 kernels, written in CuTe‑DSL, deliver up to 175% faster forward pass and 74% faster backward pass than FlashAttention‑4 on SM100 hardware [2][4]. The distributed Muon system fuses dense and MoE parameter synchronisation into one collective and spreads optimizer state more evenly across nodes [2].

Benchmark performance

Baidu compared the base checkpoint against three publicly released base models — DeepSeek‑V4‑Flash‑Base, Hy3‑preview‑Base and MiMo‑V2.5‑Base — using the same evaluation harness, prompts and LLM judge [2]. Selected results:

Benchmark ERNIE 5.5 Flash DeepSeek V4 Flash Hy3 preview MiMo V2.5
MMLU (5‑shot EM) 93.3 89.4 87.5 87.6
HumanEval+ (1‑shot Pass@1) 82.3 69.5 78.0 74.4
FullStackBench (1‑shot Pass@1) 54.6 43.3 56.7 54.1
InfiniteBench (0‑shot avg) 46.5 48.0 33.0 46.7

ERNIE leads on MMLU and HumanEval+, while Hy3 leads FullStackBench and DeepSeek leads InfiniteBench [2]. Coding benchmarks show a wider gap: MBPP+ is 81.5 vs 67.2 for DeepSeek, and LiveCodeBench is 40.1 vs 31.9 [4]. These numbers refer to base checkpoints, not post‑trained chat models, so direct comparisons with finished assistants like GPT‑5 or Claude are misleading [3].

Availability and pricing

Weights are available in FP8 and BF16 safetensors format on Hugging Face [1][2]. You can serve the model with SGLang or vLLM via the completions API [2][4]. NVIDIA has published a Megatron Bridge recipe for supervised fine‑tuning on Blackwell GPUs, validated on GB200 and GB300 systems; the components also support Hopper GPUs but without official validation [2].

The model uses a custom licence (ERNIE 5.5 Flash Base Model License Agreement v1.0) [2][3]. Non‑commercial use is free. Commercial use with annual related revenue under $50 million requires a one‑time notice and visible “Powered by ERNIE” attribution. Above that threshold, you need Baidu’s separate prior written authorisation [2][3]. The licence excludes the European Union, United Kingdom and South Korea entirely [2][3]. It also forbids using the model, derivative models or their outputs to improve an unrelated AI or machine‑learning system — a restriction that matters for distillation and synthetic‑data pipelines [2].

What it means for developers

ERNIE 5.5 Flash Base is a starting point, not a finished product. It gives teams a fixed checkpoint to run domain adaptation, instruction tuning or reinforcement‑learning experiments without repeating Baidu’s pre‑training run [2].

The architecture makes it cheap to run at inference. With only 14 billion active parameters per token and a compressed KV cache, the model is designed for agentic workloads that fire off thousands of calls rather than a single long chat session [3]. This focus on capability per dollar mirrors a broader trend among Chinese labs working under chip export restrictions [3].

However, the licence’s geographic exclusions and revenue triggers mean the model cannot be freely scaled worldwide or used commercially above $50 million in revenue without negotiation. Developers building commercial products should factor that in from the start. Pure research and below‑threshold deployments are unaffected [2].

In short, Baidu is betting that a lean, trainable base model with permissive non‑commercial terms will attract builders who care more about inference cost than outright benchmark records. The post‑trained version, when it arrives, will show whether that bet pays off for everyday use cases [3].

Sources

  1. Baidu releases ERNIE 5.5 Flash Base with 280 billion parameters as an open-source foundation model
  2. Baidu Releases ERNIE 5.5 Flash Base for Further Training
  3. Baidu open-sources ERNIE 5.5 Flash Base in a race to cheap inference - Startup Fortune
  4. Baidu Open-Sources ERNIE 5.5 Flash Base: 280-Billion-Parameter MoE Architecture With Only 14 Billion Activated Per Token — BigGo Finance
  • ernie
  • baidu
  • open-source
  • base-model
  • mixture-of-experts
  • llm

Researched and drafted with AI from the sources above.

Learn to build with AI

Hands-on tutorials on Cursor, Claude Code, AI agents and more.