LLM Inference Optimization and Deployment

Why inference needs its own engineering#

Training determines model parameters; inference determines how a trained model responds to requests under real constraints. A useful serving system has to balance response time, throughput, memory use, and cost. Improving one metric can worsen another, so measurements need a stated workload and a clear target.

Autoregressive generation has two different phases:

• Prefill processes the input prompt. Attention can be computed across prompt tokens in parallel, so this phase is often compute-bound.
• Decode produces output one token at a time. It repeatedly reads model weights and the attention state for prior tokens, and can become memory-bandwidth-bound.

Common measurements include time to first token (TTFT), inter-token latency, tokens per second, throughput, and end-to-end latency. A single number is not enough: batch size, prompt length, generated length, concurrency, and hardware all affect the result.

Memory: weights and the KV cache#

Weights are only part of inference memory. The key-value (KV) cache stores attention keys and values from earlier tokens so the model does not recompute the entire prefix for each generated token. Its size grows with sequence length, batch size, layers, and the number of stored attention heads. Long contexts and concurrent requests can therefore exhaust memory even when model weights fit.

The guide introduces several ways to reduce this pressure:

• Quantization stores weights at lower precision. Post-training quantization (PTQ) is applied after training; quantization-aware training (QAT) incorporates quantization effects during training. The source surveys bitsandbytes, GPTQ, and AWQ as examples. Memory savings and quality changes must be measured on the intended model and task.
• PagedAttention, used by vLLM in the source discussion, manages KV-cache blocks to reduce allocation waste and support dynamic batching.
• MQA and GQA reduce the number of key-value heads relative to ordinary multi-head attention. These are model architecture choices; they are not drop-in serving switches for a model that was trained with a different attention structure.

Faster attention and serving engines#

FlashAttention reorganizes attention computation to reduce high-bandwidth-memory traffic and avoid materializing the full attention matrix. It can improve speed and memory use when the hardware and implementation support it, but its effect depends on sequence lengths and kernels.

The Chinese guide walks through vLLM, TensorRT-LLM, and Hugging Face Text Generation Inference (TGI), then discusses tensor parallelism, pipeline parallelism, and request distribution. Those tools evolve quickly; the article’s comparisons are a snapshot of its writing period, not a current ranking. Before adopting an engine, check its supported models, hardware, precision formats, batching behavior, operational costs, and maintenance requirements.

A practical evaluation loop#

Optimization is easier to reason about when each change answers a specific question:

1. Define a representative request mix: prompt lengths, output lengths, concurrency, and latency constraints.
2. Record a baseline with the model, hardware, software versions, and measurement procedure.
3. Change one factor at a time, such as quantization, cache policy, batching, or parallelism.
4. Compare TTFT, inter-token latency, throughput, peak memory, cost, and task quality.
5. Keep the change only when it improves the intended trade-off without unacceptable regressions.

The source also covers containerized deployment, monitoring, autoscaling, request caching, and lower-cost compute options. These operational choices affect reliability and cost; they do not by themselves establish that a model has become more capable. For an RSI-oriented system, serving metrics should be considered alongside evaluation of the model’s behavior and the feedback used to improve it.

What the source covers#

The Chinese article contains examples for quantization, KV-cache handling, attention, serving engines, distributed inference, Docker and Kubernetes deployment, monitoring, and cost controls. Its numerical examples are instructional rather than results from a reproducible benchmark attached to the post. Use the original for the complete walkthrough, and independently benchmark any configuration before relying on its performance claims.