Mesh LLM — Distributed AI Computing on iroh

URL: https://iroh.computer/blog/mesh-llm Date: 2026-07-12 Source: Hacker News (#5, 275 points) Topic: AI LLMOps / Inference Infrastructure

Mesh LLM pools GPUs and memory across multiple machines and exposes them as a single OpenAI-compatible API. Built on the iroh P2P networking library, it allows running models three ways: (1) locally on a single node’s GPU, (2) routed to a peer that already has the model loaded, or (3) split across several machines (“Skippy” mode) by partitioning model layers across nodes for models too big for any single machine. Uses QUIC-based ALPN negotiation for gossip, routing, HTTP tunnels, and activation transport. Supports 40+ pretrained models from 0.5B to 235B parameters. No central server—every node boots an iroh endpoint.

Key Features

• OpenAI-compatible API at ${LOCALIP}:9337/v1
• Distributed model inference via iroh P2P mesh
• Model splitting (Skippy) for large models
• 40+ pretrained models supported
• Plugin architecture via manifests
• MCP, HTTP, inference, and mesh event protocol support

Links

• See also 2026-07-12-2002-plano-proxy — another AI proxy infrastructure project
• See also 2026-07-12-2002-otari-gateway — Mozilla’s LLM gateway
• Related: 2026-07-12-0808-agentgateway-proxy — Agentic proxy for AI agents