Start learning GPU programming with one fully animated YouTube lecture. youtu.be/dEZP1qUNTOk This is the first video of my GPU programming series with Triton. Every idea in it is visual and animated, just like the GIF in this post, but for a full hour. You write c = a + b in PyTorch every day. This lecture shows what the GPU actually does with it. 𝐖𝐡𝐚𝐭 𝐭𝐡𝐞 𝐥𝐞𝐜𝐭𝐮𝐫𝐞 𝐜𝐨𝐯𝐞𝐫𝐬 - How a GPU sits inside your computer: PCIe, NVLink, servers and clusters - Inside an NVIDIA H100: SMs, FP32 lanes, the warp scheduler, tensor cores, shared memory - Threads, warps, blocks and grids, and how each thread finds its element - Memory: SRAM vs DRAM, DDR vs GDDR vs HBM, caches, latency vs bandwidth - Making memory fast: latency hiding, coalescing and kernel fusion - Why Triton exists, and how it lets you think in tiles instead of threads - Memory-bound vs compute-bound, arithmetic intensity and the roofline - Attention, the N x N matrix, and how FlashAttention fixes it No CUDA or hardware background needed. I want to make more long-form technical animated videos, something similar to what @neural_avb does. Taken inspiration from him 𝐖𝐡𝐚𝐭'𝐬 𝐧𝐞𝐱𝐭 - Lecture 2: writing our first Triton kernel, vector add, line by line, will post it in few days - More lectures on GPU programming and inference optimization - A planned series on deep learning for audio Subscribe to the channel so you don't miss them. Please also share the video with your ML and DL friends, or in any study or community groups you're part of. Watch the full lecture: youtu.be/dEZP1qUNTOk If you are into audio deep learning, inference, or Triton, follow @Mayank_022 me. I will be uploading more posts related to that.