The DeepSeek Sequence: A Technical Overview

Motivation & Overview

The authors got down to reply an essential query: Given a hard and fast
compute funds for pre-training, how will we select the size of the mannequin
and the way a lot coaching information to make use of? Prior research (e.g. Chinchilla vs.
GPT-3) differed on the ratio between these two elements. DeepSeek-LLM
addresses that by measuring scale differently. Earlier work
measured scale when it comes to what number of parameters had been within the mannequin,
DeepSeek-LLM as an alternative measured scale as non-embedding FLOPs/token They then discovered they may predict
computation with:

$$
C = M occasions D
$$

the place $C$ is the compute funds, $M$ is non-embedding
FLOPs/token, and $D$ is information dimension.

This extra granular illustration helps them predict how a 7B or 67B mannequin
may prepare on 2T tokens of bilingual information.

Coaching Instability

A central concern they grapple with is coaching instability (sudden irrecoverable
divergences within the coaching course of), which may usually manifest in
large-scale language fashions—particularly these with mixture-of-experts
or very lengthy contexts.

By fastidiously tuning studying charges, batch sizes, and different
hyperparameters , DeepSeek-LLM
demonstrates that steady large-scale coaching is achievable, nevertheless it
requires meticulous design of the structure of the transformer mannequin
along with the infrastructure of the Excessive Efficiency Computing (HPC)
information heart used to coach it. This interwoven design of each structure and
infrastructure is known as HPC Co-Design.

Knowledge High quality & Mannequin Scale

Some extent the authors make is about how information high quality shifts the optimum
ratio—i.e., higher-quality information can justify a much bigger mannequin for a similar
variety of tokens. You may intuit this by imagining two situations:

• State of affairs A: You have got a 100-billion-token corpus stuffed with duplicates,
  spammy textual content, or incomplete sentences. The mannequin may not glean a lot new
  information as a result of the information is partly redundant or low-value.
• State of affairs B: You have got a fastidiously curated 100-billion-token corpus with
  broad protection of code, math, multi-lingual dialogues, factual textual content, and many others. Every
  token is extra “information-rich,” so the mannequin can “afford” to make use of extra
  parameters with out hitting diminishing returns prematurely.

In different phrases, when information is denser in helpful data, scaling the
mannequin additional pays off as a result of every parameter can study from richer
alerts.

Key Takeaways

• Hyperparameter Scaling: They suggest easy power-law matches to select batch
  dimension and studying fee as compute $C$ grows.
• Bilingual Knowledge: They prepare two base sizes (7B, 67B) on 2T tokens
  masking English/Chinese language, then do Supervised Nice Tuning (SFT) and a less complicated
  preference-based alignment known as Direct Choice Optimization (DPO).
• Outcomes: The ensuing DeepSeek-LLM67B “Outperforms LLaMA-2 70B” on
  math/coding duties, illustrating how HPC co-designed approaches can maintain coaching
  steady whereas effectively pushing scale.

The seeds planted right here – scaling legal guidelines and infrastructure for very massive
coaching – will reappear in subsequent works.

Increasing the Mannequin Whereas Lowering Reminiscence

The place DeepSeek-LLM principally explored high-level scale tradeoffs,
DeepSeek-V2 dives into specifics of Transformer structure overhead. Two
large obstacles in massive LLMs are:

1. Consideration KV Cache: Storing Key/Worth vectors for 1000’s of tokens
   is memory-intensive.
2. Feed-Ahead Computation: Sometimes the most important consumption of FLOPs
   in a Transformer.

To tame each, they suggest:

1. Multi-Head Latent Consideration (MLA): compresses Key/Worth vectors to
   cut back reminiscence.
2. DeepSeekMoE: a sparse Combination-of-Consultants strategy that
   prompts a fraction of the feed-forward capability per token.

Multi-Head Latent Consideration (MLA)

In normal consideration, every token’s Q/Okay/V might be as massive as $d_{mannequin}$ 
occasions the variety of heads. MLA folds them into smaller “latent” vectors:

$$
quad
mathbf{c}_{t}^{KV} = W^{DKV}mathbf{h}_t,
quad
mathbf{okay}_{t}^{C} = W^{UK}mathbf{c}_t^{KV},
quad
mathbf{v}_{t}^{C} = W^{UV}mathbf{c}_t^{KV},
quad
$$

The place $c_{t}^{KV}$ is the compressed latent vector for keys and values.
$W^{DKV}$ is the down-projection matrix, and $W^{UK}, W^{UV}$ are the
up-projection matrices for keys and values, respectively. In easier phrases:

1. Replaces the usual QKV computation by utilizing low rank factorization to
   flip one matrix of dim (in, out) into two matrices of (in, rank) and (rank,
   out)
2. Undertaking the compressed KV latent vector for every head to get the total Okay
   and V head corresponding to every Q head
3. Cache the compressed KV latent vector as an alternative of every of the KV heads in
   full, and compute the KV heads on the fly from the latent vector.

DeepSeekMoE: Sparsely Activated FFNs

Subsequent, they undertake a Combination-of-Consultants (MoE) within the feed-forward blocks:

• Shared Consultants deal with common patterns for each token.
• Routed Consultants deal with specialised sub-problems, chosen dynamically through
  gating.
• Auxiliary Loss ensures balanced utilization so no skilled collapses (i.e. is
  by no means used).

They additional restrict cross-device routing with a “device-limited routing”
scheme – as an alternative of permitting any token to entry any skilled, DeepSeekMoE
selects a restricted variety of units ($M$) per token, and performs skilled
choice solely inside these units. The essential course of is as follows:

• Establish prime $M$ units that comprise specialists with the best affinity
  to the token
• Carry out prime $K_r$ skilled choice inside these $M$ units
• Assign the chosen specialists to course of the token

With out device-limited routing, MoE fashions can generate extreme
communication overhead which is incompatible with the {hardware} limitations
imposed on the DeepSeek group. As well as, MoE fashions usually danger uneven
skilled utilization, the place some specialists are overused whereas others stay
inactive. To stop this, DeepSeekMoE introduces three balancing loss
features:

• Knowledgeable-level Stability Loss ($L_{ExpBal}$):
• Ensures uniform distribution of tokens throughout specialists to forestall skilled
  collapse
• Makes use of a loss operate primarily based on softmax scores of token-expert
  affinity
• System-level Stability Loss ($L_{DevBal}$):
• Ensures workload is evenly distributed throughout units
• Communication Stability Loss ($L_{CommBal}$):
• Balances incoming and outgoing token routing to every system

Coaching & Outcomes

DeepSeek-V2, with ~236B complete params (21B activated), is pre-trained on
8.1T tokens. They do Supervised Nice Tuning (SFT) on 1.5M instruction samples,
then reinforcement studying (RL) for alignment. The tip outcome:

• Inference and coaching are each quicker and cheaper (MLA + sparse
  specialists)
• They continue to be steady at scale

This paper is absolutely when iteration good points on account of HPC Co-Design begin to
grow to be obvious. By designing the mannequin structure with the coaching
infrastructure in thoughts, and implementing a coaching regime that considers the
realities of the {hardware} (e.g. low interconnect speeds on H800s), the group
was capable of lay the inspiration for his or her most notable breakthrough.

Scaling MoE to 671B Whereas Preserving Effectivity

Constructing on V2, DeepSeek-V3 additional extends sparse fashions to 671B
parameters (37B activated), coaching on 14.8T tokens in below 2.8M H800
GPU hours. The authors credit score in depth HPC co-design:

Lastly, we emphasize once more the economical coaching prices of
DeepSeek-V3, summarized in Desk 1, achieved by way of our optimized
co-design of algorithms, frameworks, and {hardware}.

— DeepSeek-V3 Tech. Report, p.5

The foremost novelties are:

1. Refined MLA
2. Refined DeepSeekMoE
3. Co-Designed Coaching & Inference
   Frameworks

Refined MLA

Multi-Head Latent Consideration was launched in V2 to scale back KV cache overhead.
In V3, it’s additional refined with a number of new options:

• Dynamic Low-Rank Projection: As a substitute of a static compression dimension,
  MLA adjusts how strongly it compresses Key/Worth vectors relying on sequence
  size. For shorter sequences, much less compression preserves constancy; for
  extraordinarily lengthy sequences (32K–128K tokens), deeper compression manages reminiscence
  development.
• Adaptive Question Compression: The place V2 used a hard and fast $d_c$ dimension, V3
  employs an adaptive scaling of the question up/down at completely different layer depths.
  Early layers use higher-dimensional queries for expressiveness; deeper layers
  extra aggressively compress to save lots of activation reminiscence.
• Improved RoPE Dealing with: V2 solely partially decoupled keys, however V3 extends
  the idea for extra steady 128K context. They observe a “decoupled shared key”
  that reduces numerical drift in extraordinarily lengthy generations.
• Joint KV Storage: V2 saved compressed keys and values individually. V3
  merges them right into a shared compressed illustration to additional cut back reminiscence
  visitors throughout multi-node inference.
• Layer-Smart Adaptive Cache: As a substitute of caching all previous tokens for all
  layers, V3 prunes older KV entries at deeper layers. This helps maintain reminiscence
  utilization in examine when coping with 128K context home windows.

Collectively, these MLA refinements be certain that whereas DeepSeek-V3 can attend
throughout very lengthy sequences, the reminiscence overhead stays manageable.

Refined DeepSeekMoE: Auxiliary-Loss-Free, Increased Capability

On the MoE facet, DeepSeek-V3 drops the auxiliary-loss strategy from V2.
As a substitute of an express penalty time period, every skilled acquires a dynamic bias
$b_i$. If an skilled is overloaded at a step, $b_i$ decreases; if
underloaded, $b_i$ will increase. The gating determination then provides $b_i$ to the
token’s affinity:

$$
s’_{i,t} = s_{i,t} + b_i
$$

Key Enhancements:

• No Token Dropping: V2 often dropped tokens if sure specialists obtained
  overloaded, however the brand new bias-based technique retains every thing.
• Extra Activated Consultants: They elevate the variety of routed specialists from 6
  to eight per token, enhancing representational energy.
• Increased Stability: By eradicating auxiliary losses, they keep away from potential
  interference with the primary coaching goal, focusing purely on the
  intrinsico gating alerts plus bias changes.

Therefore, the ultimate feed-forward module is a mix of a small set of
shared specialists plus as much as 8 specialised specialists chosen adaptively.

Co-Designed Frameworks: FP8, DualPipe, and PTX Optimizations

Scaling an MoE mannequin to 671B demanded HPC-level options for coaching and
inference. The authors emphasize:

By way of the co-design of algorithms, frameworks, and {hardware}, we
overcome the communication bottleneck in cross-node MoE coaching, attaining
near-full computation- communication overlap.

— DeepSeek-V3 Tech. Report, p.5

FP8 Combined Precision

They undertake an FP8 information format for Normal Matrix Multiplications (GEMMs),
halving reminiscence. The danger is diminished numeric vary so that they offset it
with:

• Block-wise scaling (e.g., 1×128 or 128×128 tiles).
• Periodic “promotion” to FP32 after quick accumulation intervals to keep away from
  overflow/underflow.

DualPipe Parallelism

They
suggest DualPipe to overlap ahead/backward computation with the MoE
all-to-all dispatch. It rearranges pipeline levels to make sure that community
communication (notably throughout InfiniBand) is hidden behind native matrix
multiplications.

PTX-Stage & Warp Specialization

To completely exploit InifiniBand(IB) and NVLink:

• They tune warp-level directions in PTX (a degree decrease than CUDA),
  auto-tuning the chunk dimension for all-to-all dispatch.
• Dynamically partition Streaming Microcontrollers into communication vs.
  compute duties in order that token dispatch by no means stalls native GEMM.

Because of this, coaching prices had been minimize to 2.8M H800 GPU hours per run – low
for a 14.8T token corpus.

Outcomes

The ensuing DeepSeek-V3 excels at code, math, and a few multilingual
duties, outperforming different open-source LLMs of comparable scale. Deep HPC
co-design (FP8, DualPipe, PTX-level optimization) plus refined MLA/MoE
implementation obtain excessive scale with steady coaching.

Emergent Reasoning Behaviors By way of RL-Solely

All prior DeepSeek releases used SFT (plus occasional RL). Against this,
DeepSeek-R1-Zero tries an excessive: no supervised warmup, simply RL from
the bottom mannequin. They undertake Group Relative Coverage Optimization (GRPO),
which:

1. Samples a gaggle of old-policy outputs ${o_1, …, o_G}$
2. Scores every with a reward (on this case, rule-based)
3. Normalizes the benefit $A_i$ by group imply/stdev
4. Optimizes a clipped PPO-like goal

The reward operate for the R1 fashions is rule-based – a easy weighted
sum between 2 parts

• Accuracy Reward – if the duty has an goal right reply (e.g. a
  math drawback, coding process, and many others.), correctness is verified utilizing mathematical
  equation solvers for step-by-step proof checking, and code execution & take a look at
  circumstances for code correctness verification
• Format Reward – the mannequin is rewarded for following a structured
  reasoning course of utilizing express reasoning markers and
  

The relative benefit $A_i$ for a given output is calculated as:

$$
A_i = frac{r_i – imply({r_1, r_2, …, r_G})}{std({r_1, r_2, …, r_G})}
$$

the place $r_i$ is the reward calculated for the given output. The mannequin’s
coverage is up to date to favor responses with larger rewards whereas constraining
modifications utilizing a clipping operate which ensures that the brand new coverage stays
near the outdated.

In so many phrases: the authors created a testing/verification harness round
the mannequin which they exercised utilizing reinforcement studying, and gently guided
the mannequin utilizing easy Accuracy and Format rewards. In doing so, emergent
reasoning behaviors had been noticed:

• Self-verification the place the mannequin double-checks its personal solutions
• Prolonged chain-of-thought the place the mannequin learns to elucidate its
  reasoning extra completely
• Exploratory reasoning – the mannequin tries completely different approaches earlier than
  converging on a solution
• Reflection – the mannequin begins questioning its personal options and
  adjusting reasoning paths dynamically

R1-Zero might be probably the most attention-grabbing final result of the R1 paper for
researchers as a result of it realized advanced chain-of-thought patterns from uncooked reward
alerts alone. Nevertheless, the mannequin exhibited notable points:

• Readability Issues: As a result of it by no means noticed any human-curated language
  fashion, its outputs had been generally jumbled or combine a number of languages.
• Instability in Non-Reasoning Duties: Missing SFT information for normal
  dialog, R1-Zero would produce legitimate options for math or code however be
  awkward on easier Q&A or security prompts.
• Restricted Area: Rule-based rewards labored properly for verifiable duties
  (math/coding), however dealing with artistic/writing duties demanded broader
  protection.

Therefore, the authors concluded that whereas “pure RL” yields sturdy reasoning
in verifiable duties, the mannequin’s general user-friendliness was missing. This
led them to DeepSeek-R1: an alignment pipeline combining small cold-start
information, RL, rejection sampling, and extra RL, to “fill within the gaps” from R1-Zero’s
deficits.

Refined Reasoning By way of SFT + RL

DeepSeek-R1 addresses R1-Zero’s limitations by injecting a small quantity
of supervised information earlier than RL and weaving in further alignment steps.