Accounting for Training Cost: Tokens, Compute, Time, and Peak Memory

Count what the denominator represents#

Two examples with 800 prompt and 200 answer tokens each contain 2,000 nonpadding input positions and 400 supervised targets. Padding each to 1,024 produces 2,048 tensor positions. In a hypothetical 0.5 seconds, throughput is respectively 4,000 input tokens/s, 800 targets/s, or 4,096 positions/s. Unscored context is not free computation.

Treat compute formulas as approximations#

For an appropriate dense full-training approximation, C≈6ND gives 6×10^15 FLOPs at N=10^8 and D=10^7. Kaplan et al., Section 2.1 explains the counting assumptions and omitted terms. State parameter conventions, padding, and attention limits. Trainable adapter parameter count cannot simply replace N, because frozen base layers still perform work.

Two 1,024-token sequences and one 2,048-token sequence have equal token totals but different sums of squared lengths: the latter is twice as large. This describes a dense-attention component, not necessarily twice the total runtime.

Itemize memory rather than quoting one constant#

Transformers 4.57.1 separates weights, gradients, optimizer state, activations, and temporary buffers. An explicitly assumed ledger of 2-byte weights, 2-byte gradients, 4-byte master weights, and two 4-byte Adam states totals 16 bytes per parameter: 1.6 billion bytes, about 1.49 GiB, for 100 million parameters, before activations and other overhead.

Implementations differ. Sharding, quantization, offload, master-weight choices, and gradient dtype change the ledger. Report per-device peaks with measurement scope, not just checkpoint size.

Measure the whole experiment#

If training occupies 70 minutes of a 100-minute job, doubling that part’s speed yields 65 minutes total, or about 1.54× overall speedup. Separate warmup and compilation from steady-state measurements, and account for asynchronous execution correctly.

Include data preparation, rollout generation, feedback, evaluation, storage, and failed runs. Ten two-hour searches cost twenty hours even if the selected final run took two. GPU-hours are not interchangeable across devices, and tokens are not currency without verified billing information.

The Chinese article recommends separate columns for estimates and measurements and comparison at matched quality or cost.