Models ship faster than you can read. Build a process, not a habit.
Daily / weekly: input filters
- X / Twitter (curated): @karpathy, @simonw, @hwchase17, @swyx, @abacaj, @teortaxesTex, @rasbt, @philschmid.
- Hugging Face daily papers — a 10-minute scroll catches anything that matters.
- Arxiv Sanity — paper triage by relevance.
- Anthropic / OpenAI / Google blogs — model launches, deprecations.
- LangChain / LlamaIndex / Together AI / Modal blogs — applied patterns.
- HN / lobste.rs — for ecosystem and ops moves.
Weekly: leaderboards
- LMSYS Chatbot Arena — vibes, but useful as a directional signal.
- Open LLM Leaderboard (HuggingFace) — standardized open-weight comparisons.
- Vellum.ai model comparison — clean side-by-side cost/latency/context.
- Artificial Analysis — independent latency/cost/quality dashboards.
- MTEB (embeddings), MIRACL (multilingual retrieval), SWE-bench (coding agents).
Monthly: your model audit
A 1-hour ritual:
- Pick your top 3-5 production use cases.
- Run your eval suite against the current frontier (1-2 new candidates this month).
- Plot quality, latency, cost vs incumbent. Decide: stay, upgrade, or A/B test.
- Note deprecations with > 90 days runway and plan migration.
- Update a one-pager: "Current model choices and why."
Quarterly: deeper review
- Re-eval your prompts on new models — old prompts may be over-fit.
- Check fine-tune providers; sometimes the right move is to migrate fine-tune base.
- Audit cost: model pricing changes; you may be paying old rates by inertia.
The trap to avoid
Don't chase the new shiny model the day it ships. Wait for:
- Real-world benchmarks (community evals, not vendor charts).
- Tool / function-call reliability tested.
- Cost / rate limits stable.
- Your own eval suite agrees.
A model that wins LMSYS by 30 ELO points may still lose your eval. Trust your own data.