no finer feeling than the kiddos being so happy to see you after you come back from a long business trip
As fused compute-comms kernels become more common (especially because things like NVL72 make them easier to program than ever), actually testing bitwise equivalence on real sizes requires a full NVL72 supernode which... hahaha so expensive........
A wonderful section header, ruined by smelling LLMish. "Entangled knobs: experts per token, expert width, experts per layer"
To stop worrying about losing your job to an AI, consider picking up a higher religion like "we must rewrite all of humanities software to be vulnerability free or the apocalypse will happen"
> The second happens-before is what usually gets humans
I am so mad LLMs have totally ruined this turn of phrase for me
Been thinking a bit about what the future holds for coding in an agentic world and I have a pretty interesting specific example that is near and dear to my heart and which I want to write down: CUDA streams.
A virtue of startups is nimbleness, the ability to pivot when the winds change. Many of us at PyTorch are bigco infra people at heart, though: we believe that there exists valuable infra that requires time to build, if you wait and see, when it is needed, it will be too late.