---

type: knowledge-digest title: ESP32-AI: Running 28.9M parameter LLM on $8 microcontroller tags: [edge-ai, llm, esp32, per-layer-embeddings, gemma, tiny-ml] source: https://github.com/slvDev/esp32-ai source_human: https://news.ycombinator.com/item?id=49050512

ESP32-AI: Edge AI Architecture Breakthrough

Core Idea

28.9M parameter language model running entirely on an $8 ESP32-S3 microcontroller using Google Gemma’s Per-Layer Embeddings to store 25M parameters in flash memory, achieving ~9.5 tok/s on-device inference.

Why It Matters

• 100× parameter increase over prior microcontrollers (260K → 28.9M)
• Architecture proof: large embedding tables can live in slow flash, reading only ~450 bytes per token
• Fully on-device, no network — true privacy and offline AI
• Potential for “IoT AI node networks” (multiple ESP32s in mesh)

Key Technical Details

• Per-Layer Embeddings (Google Gemma 3n/4): 25M param table in flash mmap XIP
• ~6 rows (~450 bytes) read per token from flash; 560K dense core in SRAM
• ESP32-S3: 512KB SRAM + 8MB PSRAM + 16MB Flash ($8 BOM)
• 4-bit quantization → 14.9MB model size
• Trained on TinyStories (Microsoft Research, arXiv:2305.07759)

Relevance to Hermes

1. Edge deployment path: Hermes agent on IoT hardware
2. Memory architecture inspiration: Per-Layer Embeddings = slow/fast memory hierarchy for agent knowledge
3. Monitor trajectory: next-gen ESP32-P4 could run 100M+ param models
4. Community signal: 1105⭐ in 3 days shows edge AI excitement