Queue Infrastructure Evolution — Performance Benchmark

93%

Latency Reduction — 310ms → 23ms average

Benchmark Results

❌ v1 — Cloudflare KV

POST /jobs 350ms

GET /jobs/:id 280ms

GET /jobs (list) 450ms

GET /health 150ms

POST /receipts 320ms

Throughput 100 req/min

Rate limit 429 errors

✓ v3.1 — Local SQLite

POST /jobs 26ms

GET /jobs/:id 22ms

GET /jobs (list) 22ms

GET /health 22ms

POST /receipts 22ms

Throughput Unlimited

Rate limit None

Evolution Timeline

March 13

v1.0 — Cloudflare Worker + KV

Initial queue deployed. Cloud-based, global distribution. Immediate rate limit issues.

March 14

CRITICAL OUTAGE

Aggressive agent polling (10s × 3 agents) burned KV rate limits. 2+ hours downtime. Lesson: local-first for local networks.

March 15

v2.0 — Local SQLite Migration

Queue moved to Forge Mac Mini. Zero cloud dependency. Emergency auth tokens added.

March 15

Hub-and-Spoke Architecture

Mac Studio becomes central hub. All agents consolidated. 50x faster inter-agent communication.

March 16

MQTT + Bridge v3

Push-based messaging replaces polling. Bridge v3 with local inbox persistence. Zero message loss on restart.

March 17

Tailscale Mesh Network

All devices on tailnet. Cross-subnet communication solved. No port forwarding, no NAT issues.

March 18

v3.1 — Distributed AI Operating System

7-stage workflow, 3-layer verification, hash-chained receipts, independent verifier. Production ready.

Performance Factors

Improvement	Impact	Latency Gain
Local SQLite queue	Eliminated cloud round-trip	50-100x faster
Hub-and-spoke architecture	Single gateway, no routing	50x faster
MQTT push messaging	Zero polling overhead	Instant delivery
Tailscale mesh	P2P connections, no relay	Consistent latency
Central Memory (partial)	Local DB vs file reads	2-3x faster reads

Cost Analysis

Before (v1)

Cloudflare Worker: Free tier (limited)
KV reads: $0.50 per million
KV writes: $5.00 per million
Rate limit downtime: Expensive (unreliability)
Agent polling: 6 requests/minute × 3 agents = 18 req/min baseline

After (v3.1)

SQLite: $0 (included in Mac hardware)
Storage: 4.3MB for 6 jobs + 25 receipts
Throughput: Unlimited
Rate limits: None
Reliability: 99%+ (local-only dependency)

$0/mo

Queue operating cost — down from Cloudflare KV overages

Key Lessons

Local-first for local networks. Cloud adds latency and rate limits. If all agents are on same network, keep data local.
Do the math before deploying. 10s polling × 3 agents = 18 req/min baseline. Add job traffic, easily exceed 100 req/min.
Fix immediately, don't wait. KV rate limits reset "by morning" — but that's unacceptable for production systems.
Polling is expensive. Push-based (MQTT) eliminates wasted cycles and reduces latency to near-zero.
Architecture matters more than optimization. Hub-and-spoke + local SQLite gave 50x gains. No code optimization can match that.

Current State (March 18, 2026)

Metric	Value
Queue API	localhost:3334
Verifier API	localhost:3335
Database size	4.3MB
Jobs stored	6
Receipts stored	25
Average latency	23ms
Throughput limit	None (hardware bound)
Rate limits	None

Recommendations for Similar Deployments

Start local. Only add cloud when you need global distribution.
Measure before you scale. Know your request patterns before choosing infrastructure.
Push over poll. Always prefer push-based communication for real-time systems.
Mesh networks win. Tailscale solves NAT/subnet issues without manual configuration.
Independent verification. Separate verifier service prevents self-certification attacks.