A small team working on the Bittensor blockchain network just pulled off something that many leading researchers dismissed as impossible: training a 100-billion-parameter AI model across geographically distributed, consumer-grade hardware over the open internet — at a fraction of the cost of traditional data center approaches. The project, called Orion-100B, may be the most significant development in AI infrastructure this year that you haven’t heard about.
What Macrocosmos Actually Did
Macrocosmos, the team behind the IOTA distributed training system, published results on June 1st showing that their Orion-100B run achieved something unprecedented. Using 48 Nvidia A100 GPUs spread across five U.S. data centers — none of them co-located — they trained a 100-billion-parameter model based on the Llama-3.2 architecture, processing 1.1 billion tokens over approximately two days.
The headline number is the model FLOP utilization (MFU): 30.8% average, peaking at 38%. For context, that’s 65% of the training speed you’d get from equivalent hardware wired together in a single data center with high-speed interconnects. Previous distributed training efforts — projects like Covenant-72B or Intellect-1 — had achieved similar utilization rates, but only with far smaller models. Orion-100B did it at 10x the parameter count.
Perhaps more striking is the cost: individual peers in the IOTA network can participate for as little as $1.25 per hour, using a single A100 GPU. Compare that to the $50 per hour required to run a single “Covenant-class” peer (8 B200 GPUs) in competing decentralized training efforts. The total cost for an Orion replica — 16 non-co-located A100s — runs about $20 per hour.
Why This Is Technically Hard
Training a frontier-scale model is usually done in a “data center bubble”: hundreds or thousands of GPUs connected by ultra-fast InfiniBand or NVLink interconnects, housed in the same building, with latency measured in microseconds. Distributing that across the open internet, where latency is measured in milliseconds and bandwidth is variable, has long been considered either impossible or prohibitively inefficient.
The core problem is activation transfer. During training, each layer of a neural network produces intermediate outputs (activations) that must be passed to the next layer. In a 100-billion-parameter model, those activations are enormous — 140.6 MB per transfer between pipeline stages in Orion-100B’s configuration. Over a standard 1 Gbps internet connection, that would create a bottleneck that kills any speed advantage from distributed compute.
Macrocosmos solved this with ResBM (Residual Block Merging), a novel lossless compression technique the team developed and published as a separate paper. ResBM compresses activations by 64x — shrinking 140.6 MB down to just 2.2 MB per pipeline stage transfer. That compression ratio is what made the entire project viable.
The second major innovation is the fault-tolerant peer-to-peer networking protocol. In a data center, if one GPU fails, you can pause training and restart. In a distributed network, nodes drop in and out constantly. IOTA’s stochastic pathfinding algorithm continuously reroutes activations around failed nodes, maintaining training continuity without human intervention.
Flipping the Economics of AI
The implications for who can build AI are significant.
Today, training a genuinely frontier model costs tens to hundreds of millions of dollars. This has consolidated AI development into a handful of companies with the capital to reserve massive GPU clusters from cloud providers or build their own data centers. OpenAI, Google DeepMind, Anthropic, and Meta effectively have a hardware moat that prevents most organizations from entering the frontier.
Distributed training doesn’t eliminate cost — but it changes who bears it and how it scales. Rather than one organization fronting the entire GPU bill, compute can be aggregated from thousands of smaller contributors. Each contributor earns tokens in proportion to the compute they provide, a model Macrocosmos runs via Bittensor’s Subnet 9.
The team has already mapped out a clear roadmap for pushing this further:
Phase 1 (completed): Prove viability with co-located, enterprise-grade A100s at hundred-billion-parameter scale.
Phase 2 (in progress): Train across heterogeneous hardware — mixing different GPU generations to tap stranded or “obsolete” compute that data centers have already written off. Older A100s, V100s, even consumer 4090s could contribute.
Phase 3: Add interruptible spot-market instances, which typically run at 60-90% discount but can be reclaimed by cloud providers at short notice. IOTA’s fault tolerance is specifically designed to handle this.
Phase 4: Allow fully permissionless, untrusted contributions — the final step toward genuinely open, internet-scale AI training.
The Bear Case (and Why It Might Still Be Wrong)
It’s worth acknowledging the skepticism. When Macrocosmos presented an early version of this work at a San Francisco event last October, a leading AI researcher dismissed distributed pipeline-parallel training as “fighting gravity.” His point wasn’t wrong in principle: every added hop across the internet introduces latency, jitter, and failure modes that don’t exist inside a data center.
The counterargument Orion-100B provides is empirical rather than theoretical. The run completed. The MFU was high. The model showed stable learning dynamics across 28 full training cycles. The team notes that by increasing inner steps from 10 to 100 (still well within demonstrated stability ranges for DiLoCo-based training), overall compute utilization could rise to 97.8% — essentially matching a co-located setup.
There are real limitations to acknowledge: the current run used controlled, enterprise-grade nodes rather than truly heterogeneous or permissionless hardware. Training was stopped at 1.1 billion tokens for cost reasons, far short of what a useful frontier model would require. And the Bittensor token-incentive mechanism adds complexity that pure research groups don’t have to navigate.
But none of these are fundamental blockers. They’re engineering challenges on a proven foundation.
What Comes Next
The Macrocosmos team has signaled they intend to continue Project Orion with progressive rounds of constraint-relaxation — each one expanding the pool of potential contributors and further stress-testing the system. If the heterogeneous hardware and spot-instance phases succeed, the economics shift dramatically: AI training at frontier scale could become accessible to universities, smaller companies, and international research institutions that currently have no path to that compute.
For an industry that has spent the past three years watching compute concentration increase alongside model capability, Orion-100B represents a genuine technical counterweight. The question is no longer whether distributed training at hundred-billion-parameter scale is possible. Macrocosmos has answered that. The question now is how fast the efficiency gap with centralized infrastructure narrows — and what happens to the AI landscape if it closes.
The big labs are watching. They should be.
Sources: Orion-100B: Distributed pretraining arrives at hundred-billion-parameter scale (Macrocosmos, June 2026); Import AI 418: 100b distributed training run; Beyond Centralized Limits: Unpacking the First Globally Distributed 100B+ Parameter AI Model
