Inference Engine

The serving runtime layer that takes model weights plus live requests and turns them into token outputs by coordinating kernels, scheduling, memory movement, and cache state on real hardware.

Opening summary

Model weights store what the model learned, but they do not decide how a real machine should serve requests. An inference engine is the runtime software that loads those weights, chooses kernels, schedules work, manages memory and key-value cache state, and turns live inputs into output tokens.

At a glance

Released

May 2020

System type

Runtime

Related modules

No related modules listed yet.

What It Is

An inference engine is the serving runtime around a trained model. It is the software layer that takes model weights plus live requests and actually executes the forward pass on hardware, instead of leaving those parameters as a passive checkpoint on disk.

Where It Sits

It sits between applications that send prompts and the accelerators or CPUs that run the math. The model weights define the learned parameters and architecture. The inference engine decides how those parameters are loaded, how token steps are scheduled, which kernels run, where data moves, and how reusable state such as the key-value cache stays available across prefill and decode.

How It Works

A request enters the runtime, the runtime loads or reuses model weights, dispatches kernels for the next layer operations, batches or interleaves work with other requests, moves activations and cache state through the available memory tiers, and emits output tokens. Kernels, scheduling, memory movement, and cache handling all live here because they are execution choices about how the same learned weights are served efficiently on real hardware.
Inference Engine System Flow
Request and weight flow
Cache writes and reuse
Cache reuse and reads
The model defines the computation. The inference engine decides how that computation is executed efficiently on actual hardware.

Practical Impact

This layer decides much of the real serving experience: latency, throughput, memory fit, and how gracefully the system handles long contexts or many simultaneous users. Quantization, cache policies, and batching strategies often look like separate serving topics, but in practice they become useful only when the inference engine coordinates them into one runtime plan.
textservedtokenlatencyapproxtextkernelwork+textschedulingdelay+textmemorymovement+textcacheoverhead\\text{served token latency} \\approx \\text{kernel work} + \\text{scheduling delay} + \\text{memory movement} + \\text{cache overhead}

Tags

References

  1. Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv, 2020.
  2. DeepSeek-AI. "DeepSeek-V4 Technical Report." 2026.

Speculative Decoding

Previous Page

On this page

What It IsWhere It SitsHow It WorksPractical ImpactRelated ToTagsReferences