David King

My take on MLX: (long rant/perspective ahead) I think there are several overlapping problems, each of them is holding local AI back on Apple devices. 1) Lack of real investment by Apple (as usual). Apple doesn’t even host the MLX repo under their own namespace. It is the primary method of running local inference on Mac devices but MLX has been essentially abandoned since the start of the year when Awni joined Anthropic, with no real hand off. It is frankly shameful Apple would let such an important project lie dormant, in maintenance mode with minor commits and no real innovation being worked on. When I brought MTP to Macs even before GGUF in April, there was someone else (AirRunner) who tried porting a simple version of MTP to MLX LM. TILL THIS DAY! It has not been merged to the repo. 3 months of work went into that branch and it has not been merged. Nvidia is not beloved because of the hardware. It is CUDA which is so good. On pure hardware maths, DeepSeek V4 flash should be able to run at 80+ TPS without D-Spark on an M3 Ultra. It is currently in the 30-40 range. Why? Because their aren’t mature kernels, MLX is the Wild West where people like me are left to do custom kernels (that never get merged) and become separate projects. Nvidia heavily contributed and incentivised the development of CUDA. Which is why two GARBAGE DGX sparks can run DeepSeek V4 flash faster than an M3 Ultra! The aggregate bandwidth of two sparks, no three sparks, NOT EVEN INCLUDING the latency of RDMA TP is less than an M3 ultra. It’s ridiculous. A parallel can be made with gaming, MacBooks could be absolute MONSTERS for gaming. Efficient ARM and a decent GPU. But Apple has done an abysmal job getting studios to port games to arm. When Nvidia made the spark laptop? He had Bond, Fortnite and several games ON LAUNCH. Because Jensen understands the importance of partnerships, which Apple has historically been terrible at and ignored. If Apple were to properly take over MLX, assign engineers to it, incentivise developers, accept and review contributions. MLX would be much better than it is now. It amazes me how Apple underestimates the importance of software and firmware nowadays. The BEST price/hardware for local AI inhibited only by poor developer relations and implementation. 2.) Hardware. The lack of floating point. Apple devices don’t hardware support low bit floating point precision, which is vastly superior for retaining accuracy at low quants. This has been available for years on Nvidia with FP8 and now NVFP4. This will change next year with the release of the M7. Right now developers are forced to hand tune individual layers from INT4, to INT8 or BF16 to retain quality and speed and avoid the emulation tax. But their are also people out there which release slop quants not caring about accuracy. I made this mistake by using an MLX community 4 bit flat quant with terrible KLD as the base of 3.6 Optimized Speed which has forever tarnished the reputation of MTPLX as “worse quality”. It’s a lack of resources too, it is not easy to learn these things. Nvidia publishes guides to NVFP4 and their own NVFP4 models while Apple does nothing. Final Notes: It will be very interesting to see where the local AI space goes in the coming years. Apple and Nvidia have separate weaknesses. Apple: amazing memory bandwidth and hardware for AI but poor firmware, software and kernels. Nvidia: great sheer compute, amazing firmware and kernels, stingy and cheap with hardware saving a bunch on the memory bus destroying bandwidth. Either Apple steps up their developer relations game, and invests in improvements to the ecosystem. Or Nvidia stops being cheap. 512 bit memory bus and/or newer unified memory. Whoever does one or the other wins local AI.