FlagOS Community

The gains are cumulative. From 47.57 tok/s, core FlagOS optimization reaches 74.34; adding a W8A8 output head reaches 82.30; using 4 Decode threads reaches 95.41. PP512 ends at 595.31 tok/s and TG128 at 21.91—showing that more Decode threads aren’t always better.

How it’s measured: PP512 = 512 ÷ (first-token time − dispatch time); TG128 = 127 ÷ (last-token time − first-token time); Total = 640 ÷ (last-token time − dispatch time). Each test is warmed up once, run 5 times, and reported by median.

The stack stays modular: vLLM schedules requests and manages KV/state; vLLM-plugin-FL handles model registration, quantization and Arm lifecycle; FlagGems supplies W8A8/GDN/Norm kernels; FlagTree-CPU compiles CPU Triton and runs TLE Raw paths.

Can a 1B multimodal model run efficiently on an Arm CPU—without a GPU❓ On the 8-core CIX P1, FlagOS deploys MiniCPM-V-4.7-1B with BF16 vision and W8A8 language layers, raising total throughput from 47.57 to 95.41 tok/s versus a minimally adapted baseline.

Useful compute is the real test. After cost, utilization and training losses compound, the paper estimates compute per dollar for large-scale training could be ~90× lower than on Earth at the optimum. This is not 90× slower GPUs; independent inference may fit orbit better.

Bigger is not always cheaper. Small satellites carry too much fixed overhead; megastructures add assembly and collision-avoidance costs. Under the paper’s assumptions, the TCO minimum is ~512 GPUs per satellite with modular servicing: about $28,000 per GPU-kW-year.

Orbit choice is a trade-off, not free power. Lower inclinations bring more eclipse time; sun-synchronous orbits improve solar duty cycle but can raise radiation exposure. Vacuum also removes convective cooling, so high-power GPUs need liquid loops and large radiators.

Capacity is the first constraint. With Starship-class launches and an optimistic 13 kg/kW full-system mass, one launch per day sustains ~34 GW in orbit. Using higher 2026 estimates of 34–59 kg/kW, the same launch cadence implies only ~8–13 GW.

Could AI data centers really move into orbit?❓ @elonmusk says space-based compute may cost less than infrastructure on Earth. At MICRO 2026, researchers tested that claim against launch capacity, power, cooling, servicing, networks and the economics of useful compute.

One PyTorch model runs on 8 AI chips with zero rewrites. Torch-FL gives one device entry, ~40 HAL functions light up each chip with a CPU fallback, and FlagGems / FlagTree / FlagCX pick per-kernel backends and move data. Run first, optimise later. #flagos #PyTorch #ai