Understanding the Transformer Architecture

The basic building blocks#

A Transformer represents tokens as vectors, adds information about their positions, and processes the sequence with attention and feed-forward layers. In the encoder, a token can attend to other visible source tokens. In the decoder, a causal mask prevents a position from attending to future target tokens; cross-attention then lets the decoder use the encoder’s representation of the input.

The source article implements these components in PyTorch:

• Token embeddings map source and target token IDs into a shared model dimension.
• Positional encoding supplies order information that attention alone does not provide.
• Multi-head self-attention lets the model compute several attention patterns in parallel.
• Residual connections, LayerNorm, and dropout wrap attention and feed-forward sublayers.
• Cross-attention in the decoder uses the encoder output as keys and values while the decoder state provides queries.

For one attention head, the central operation is scaled dot-product attention:

where the mask $M$ excludes padding or, for autoregressive decoding, future positions. Multiple heads apply learned projections before their outputs are combined.

Encoder, decoder, and masks#

An encoder layer applies self-attention, adds the result back to its input, normalizes it, and then applies a position-wise feed-forward network with another residual connection and normalization. A decoder layer adds masked self-attention and cross-attention before its feed-forward sublayer.

The article’s full model stacks these layers, scales token embeddings, adds positional encodings, and projects decoder states to vocabulary logits. The masking code handles padding and causal visibility. This is the classic sequence-to-sequence setup shown in the tutorial; architectures used in current language models vary, and the example should not be mistaken for a description of every Transformer variant.

Training and generation examples#

The translation example shifts target tokens so the decoder predicts the next token from earlier ones. The article also shows two training techniques:

• Label smoothing spreads a small amount of target probability across non-target vocabulary entries rather than assigning all target mass to one token.
• Warmup learning-rate scheduling increases the learning rate during an initial warmup and then decays it according to the model dimension and step count.

For generation, the source demonstrates greedy decoding: repeatedly select the token with the highest next-token score until an end token or length limit is reached. This is a simple illustration of autoregressive generation, not a comparison of decoding methods.

Trade-offs and later directions#

Full self-attention compares positions across a sequence, giving a token access to broad context but making attention cost grow quadratically with sequence length. The article lists sparse and linear attention and relative positional representations as directions explored by later work. These names point to a design space; the source provides no controlled speed, memory, or quality measurements for them.

Connection to RSI and AGI#

The Transformer is an important model architecture, but an architecture alone does not define a system’s ability to evaluate or improve itself. For an RSI-oriented research program, architecture choices need to be connected to training objectives, data, feedback, and evaluations that can distinguish real capability gains from changes in cost or behavior.

For the complete Chinese walkthrough—including the PyTorch layers, mask construction, label smoothing, learning-rate schedule, and translation examples—see the original article.