Datagen Lab

running

◷train_gpt.py▸distributed_data_generator◎learner＄no GPU

Blocks

Quick summary

Python generator that reads FineWeb .bin shards, splits across 8 DDP ranks (each rank sees unique tokens), optionally…

Full explanation below the code →

fba-lab — train_gpt.py · distributed_data_generatorexecuting

// block: distributed_data_generator · lines 1420–1490$ study train_gpt.py --block distributed-data-generatorPython generator that reads FineWeb .bin shards, splits across 8 DDP ranks (each rank sees unique to… ✓

Explanation

Python generator that reads FineWeb .bin shards, splits across 8 DDP ranks (each rank sees unique tokens), optionally aligns to BOS boundaries, and yields 4 CUDA tensors per step: inputs, targets, cum_lengths, bigram_inputs.

Think about

Why must DDP ranks see different tokens rather than the same tokens shuffled differently?

// architecture

Live diagram

03 · train_gpt.py line 1420

distributed_data_generator

DDP-aware infinite data loader with BOS alignment

A Python generator that reads FineWeb .bin shards from disk, splits the data across 8 GPUs so each sees unique tokens, optionally aligns sequences to document starts (BOS token = 50256), and yields 4 CUDA tensors per training step.

The 4 tensors yielded per step

_inputs (int32) — input token IDs for this rank's slice
_targets (int64) — shifted-by-1 labels for next-token loss

_cum_lengths (int32) — cumulative doc boundaries (for varlen flash attention)
_bigram_inputs (int32) — hashed bigram indices from get_bigram_hash

DDP-aware infinite data loader with BOS alignment

The 4 tensors yielded per step

_inputs (int32) — input token IDs for this rank's slice
_targets (int64) — shifted-by-1 labels for next-token loss

_cum_lengths (int32) — cumulative doc boundaries (for varlen flash attention)
_bigram_inputs (int32) — hashed bigram indices from get_bigram_hash