2026-10-10 补充校核：850训练/131测试样本是单个作者试验；600M模型一次分类失败不能推广为小模型无法分类，opaque ID也不是普遍最佳规则。

Fine-Tuning Qwen 3:0.6B Achieves 92% Accuracy on Question Classification

What Happened

A developer fine-tuned Qwen 3:0.6B (600M params) using Unsloth + QLoRA to categorize household questions into metadata categories (pool, car, hvac, cooking). The goal was to narrow the search space for RAG vector queries.

Key Technical Findings

Baseline (prompting only)

• Only 10% accuracy on 131 integration tests
• Tiny models cannot reliably classify through prompting alone

Experiment 1 (QLoRA fine-tuning, semantic category names)

• 79% accuracy after fine-tuning on ~850 training entries (70/15/15 split)
• Key failure: semantic overlap between categories (e.g. “water heater” classified as “pool” instead of “hvac”)

Experiment 2 (QLoRA + opaque 2-character category IDs)

• 92% accuracy after switching from semantic category names to opaque 2-character codes
• Key insight: removing semantic overlap from output space significantly improves small model accuracy

Technical Stack

• Qwen 3:0.6B (600M params) for classification
• Qwen 3:4B for general QA
• Unsloth + QLoRA for fine-tuning
• ~850 training entries, 70/15/15 split

Relevance to Hermes

• Directly applicable to knowledge routing: tiny fine-tuned model for question-to-category mapping
• The opaque ID trick is a general principle for fine-tuning classification tasks
• Suggests local classification (600M params) instead of large API models for routing
• Counterpoint from HN: traditional ML (SGDClassifier, ModernBERT) might be simpler