GLM-5
A large open-weights GLM model that pairs sparse-attention and mixture-of-experts routing with an agentic-engineering training story.
GLM-5 matters because it is the first public GLM checkpoint framed around agentic engineering rather than casual code completion. The technical report positions the model as a sparse-attention, mixture-of-experts transformer with open weights, a long context window, and training choices aimed at sustained tool use and multi-step work.
At a glance
- Family
- glm
- Source type
- Open weights
- Released
- February 2026
- Authors
- GLM-5 Team
- Modalities
- Text
- Parameter count
- 744 billion total parameters
- Context length
- 202,752 tokens
- Precision
- bf16
What It Is
GLM-5 is an open-weights text model from Z.ai, released on 2026-02-17 in the GLM-5 technical report. It is a large sparse mixture-of-experts transformer advertised with 744 billion total parameters and about 40 billion active parameters per token.Inputs And Outputs
GLM-5 is a text-in, text-out model. The public release emphasizes long prompts, coding, and agentic workflows where the model must keep working across many tool calls instead of stopping after one short answer.Architecture
At a high level, GLM-5 keeps a decoder-only transformer skeleton but replaces dense attention and feed-forward paths with sparse attention and routed experts. The report highlights DSA-style sparse attention, mixture-of-experts blocks, multi-head latent attention for cache efficiency, RMSNorm, and RoPE as the named building blocks.Output
ProbabilitiesSoftmaxOutput
LinearTransformer architectureAttention sublayer containerSparse expert sublayer containerN×RMSNormSparse
Attention
(DSA)RMSNormMixture
Of ExpertsEmbeddingRoPEInput
EmbeddingInput
TokensSoftmax to Output
ProbabilitiesOutput
Linear to SoftmaxRMSNorm to Output
LinearMixture
Of Experts to RMSNormRMSNorm to Mixture
Of ExpertsRMSNorm to RMSNormSparse
Attention
(DSA) to RMSNormEmbedding to Sparse
Attention
(DSA)Embedding to RMSNormInput
Embedding to EmbeddingRoPE to EmbeddingInput
Tokens to Input
Embedding
Important Modules
These modules explain why GLM-5 is grouped with modern long-context MoE stacks rather than a plain dense transformer. Sparse attention and mixture-of-experts carry most of the scaling story, while latent attention, RMSNorm, and RoPE support stable training and long prompts.Training
The GLM-5 report frames training around large-scale pretraining followed by post-training aimed at agentic engineering: sustained coding, tool use, and multi-step task completion rather than one-shot chat. The public materials describe open-weight release in bf16 rather than a closed API-only surface.Training regimes
Linked papers
- GLM-5