LLMs Are Complicated Now

Codex 核验（2026-10-10）：本次未取得 Andrej Karpathy 加入 Anthropic 的一手公告，该人员变动保留为未核实线索。架构取舍属于原作者观点，不应据此推断已发生的招聘事件或产业共识。

Source: HN | Points: 39 Author: Ian Barber URL: https://news.ycombinator.com/item?id=48605355 Digested: 2026-06-20 12:12 UTC

Summary

Modern LLMs have evolved from the clean Transformer stack into complex, heterogeneous architectures with multiple attention variants (grouped-query, compressed, sparse, sliding-window), MoE routing on everything from FFN layers to residual streams, and multi-GPU inference. This mirrors the complexity trajectory of recommendation systems.

Key Insights

1. Attention variants explosion: grouped-query, compressed, sparse, linear, sliding-window — no single ‘best’ attention
2. MoE routing extended beyond FFN to attention blocks and residual streams — ‘route everything’ trend
3. Multi-GPU inference adds comms ops creating new boundaries in the middle of models
4. Parallel to recsys trajectory: simple architecture → complexity from capability vs efficiency tension
5. AI agents can’t just ‘fix’ architecture complexity — you need composable, verifiable baselines before optimization
6. FlexAttention (PyTorch) exemplifies the right approach: design for composability up front with Triton templates
7. The claim that Karpathy joined Anthropic has not been confirmed here by a primary announcement; the architecture discussion remains the original author’s opinion

Relevance

Directly relevant to LLM architecture understanding for agent development. The recsys parallel is instructive — complexity is inevitable as capability demands grow. FlexAttention pattern (composable kernel generation) is a model for how to manage this complexity in Hermes tooling.

Takeaway

Design for composability up front. Don’t rely on agents to ‘fix’ architecture complexity — without a clean baseline, generated optimizations can’t be verified.

Tags

llm, architecture, transformer, agent, complexity, self-learning, web-surfer