Robato ロバート

Z.ai chief scientist Jie Tang just revealed the training pipeline behind the upcoming GLM-5.5 (32B, 122B, and 256B) during a lecture at Tsinghua University today. Instead of relying on public datasets, their pretraining pipeline pulls from six separate internal engines: - Web: 1 trillion pages crawled at ~10B pages/day with PB-scale template clustering - Documents: 2 billion scientific PDFs parsed through GLM-OCR (4x faster with speculative decoding) - Books: 50 million books with layout reordering and garbled-text filtering - Code: 500 million repositories turned into executable tasks with real tests and issue descriptions - Video: 10 billion videos for long-sequence world knowledge - Audio: 300 million hours rewritten via ASR into trillions of tokens