PM-AI Hiring Index
A web app that tracks, every day, where PMs are being hired and how much of that hiring asks for AI skills. I built it to replace opinions about the PM job market with data.
- Built with
- Claude Code
- Scope
- End to end
- Refreshes
- Daily
Product Manager
who builds with judgement.
I build platforms that other people build on. Today that is Box AI Studio, where enterprises build their own custom AI agents. Before that, developer and ML platforms at Niantic and Uber.
Independent work
This is where I follow my curiosity. I use AI to build and to teach me, then share what I discover with my community.
Each answer raised the next question
- Where it started Is AI really changing how PMs get hired? PM-AI Hiring Index ↓
- Building it, I kept running out of tokens. Which model should I actually use? Builder-Bench ↓
- Readers asked how I wrote the tests. How do you grade an AI agent’s work? Agent eval series ↓
- Then: did I need the expensive model at all? Can better tools make a cheaper model reliable enough? Scout ↓
Builder-Bench
A benchmark of how well coding agents turn real PM requests into working deliverables. I kept reaching for the flagship model out of habit, with no idea whether a cheaper one would do. I was just guessing, so I built a way to find out.
- Real PM tasks
- 23
- Categories
- 4
- Models tested
- 6 Claude models
Agent eval series
Agent eval for Claude Code on PM work, built from scratch and explained through one real example. An eval, in one sentence: for a given input, how reliably does the agent produce an accurate output?
Scout
A research agent that answers questions about Product Manager job postings, built to test whether better tools can make a cheaper model reliable enough for the job. They can, up to a point: tools cannot fix judgment. About half of Haiku’s failures were judgment errors that no tool change fixed.
- Sonnet 5.5 with better tools, against Opus 5 with basic tools
- 93% vs 77%
- Of the cost per attempt
- 28%
- Haiku 4.5 on tasks it was not tuned against
- About 30%
Career
Most of my career has been platform work: building the layer that other teams, developers and enterprises build on. Each role gave me a different shape. Put together in the chart below, they cover the full range of platform product work.
Select a role to see its shape.
Self-assessed depth on six skills platform teams hire for, from light exposure at the center to deep ownership at the edge.
Writing
I write down what I learn, mostly so I only have to learn it once.
- Can better tools make a cheaper AI model reliable enough for the job?Scout · Oct 2026
- Agent Eval, Part 3: The JudgeEval series · Aug 2026
- Agent Eval, Part 2: The GradingEval series · Aug 2026
- Agent Eval, Part 1: The QuestionsEval series · Jul 2026
- Which Model Should I Use? So I Built a Benchmark to Find OutBuilder-Bench · Jun 2026
- From Spec to Shipped: Going End to End with Claude Code as a Product ManagerHiring Index · May 2026
- What I Learned Running Evals for AI AgentsAt Box · May 2026
- What Enterprise AI Agents Actually RequireAt Box · Apr 2026
Contact
If you are building or evaluating agents too, I would love to compare notes.