Code · architecture · study mode

Step 13 of 28Skill: AI Systems Optimization
← Polar Lab
03 · train_gpt.py line 1420
distributed_data_generator

DDP-aware infinite data loader with BOS alignment

A Python generator that reads FineWeb .bin shards from disk, splits the data across 8 GPUs so each sees unique tokens, optionally aligns sequences to document starts (BOS token = 50256), and yields 4 CUDA tensors per training step.

FineWeb .bin shards 10B tokens BOSFinder scan for tok=50256 → BOS index array quickload: async thread DataPreloader background thread prefetches next shard next_batch() pick BOS-aligned spans split → 8 ranks yield 4 CUDA tensors _inputs (int32) _targets (int64) _cum_lengths (int32) _bigram_inputs (int32) 8 GPU workers rank 0 rank 1..6 rank 7 align_to_bos=True (training): sequences start at BOS boundaries. align_to_bos=False (validation): contiguous streaming.
The 4 tensors yielded per step
  • _inputs (int32) — input token IDs for this rank's slice
  • _targets (int64) — shifted-by-1 labels for next-token loss
  • _cum_lengths (int32) — cumulative doc boundaries (for varlen flash attention)
  • _bigram_inputs (int32) — hashed bigram indices from get_bigram_hash