Build or Replaced
Today: Devtools must be open source | Andy Pavlo joins ClickHouse to establish ClickHouse Labs | 200 Milliseconds Episode date: 2026-08-04.
Download MP3 →JOSH: It's Tuesday, August 4. This is Build or Be Replaced — powered by ScanBrief.dev. I'm Josh, here with Erik Anderson. ERIK: Expertise is back in style, which is annoying for everyone who thought prompting meant guessing better words. JOSH: Stick around — Erik's got an AI pro tip at the end about making agents prove their work. [pause] JOSH: ScanBrief scored 82 items across 56 sources today, and the first signal is devtools. The argument is simple: devtools must be open source. Erik, is that idealism, or is that just how serious teams work now? ERIK: That's how serious teams work. Closed devtools are fine until you need to fix the thing blocking prod at 2 AM. Then you either patch it, extend it, or sit there refreshing a vendor ticket like a sad little dashboard. [beat] JOSH: Second headline: LLMs reward expertise. That sounds obvious, but it keeps coming up. ERIK: It matters because the model doesn't replace your judgment. It multiplies it. A strong engineer with Claude gets real work done. A weak prompt from someone who doesn't know the domain gets confident soup. [beat] JOSH: Third one: people are running giant open models on tiny machines. Qwen 80B in 4.3 gigs of RAM on a Mac, and a 35B model on an iPhone. Wait, really? ERIK: Really. Some of it is clever compression, some of it is tradeoffs, and some of it is demo energy. But the direction is real. Smaller local models are going to eat a lot of boring automation work. [pause] JOSH: Start with devtools. Why do you care so much whether they're open source? ERIK: Because the tool is part of the system. Not next to it. Part of it. If my automation pipeline depends on a black box, then my incident response has a blind spot. That's not engineering. That's hoping. [beat] JOSH: But plenty of companies buy closed tools and do fine. ERIK: Fine until they don't. Network teams know this pain. You build around Cisco NSO, Terraform, Kubernetes, Selenium, whatever. The minute your tool can't express the weird thing your business actually needs, you're stuck. Open source gives you an escape hatch. JOSH: Is this about control? ERIK: Control, speed, and survival. PrimeBus is basically my answer to that problem. Everything emits events. Agents subscribe. Tests fail, telemetry lands, Gandalf checks the result, and the auto-merger either lets it through or blocks it. JOSH: And today the numbers were pretty wild. ERIK: PrimeBus auto-merger has run 2295 attempts since June 5th. 1507 merged, 788 blocked by Gandalf, 0 escalated to me. That's the point. The blocked count is the win. Guardrails caught the bad changes before they became my problem. [beat] JOSH: That's the part people miss. They hear blocked and think failure. ERIK: Exactly. Blocked means the system had standards. If your AI agent merges everything, you didn't build automation. You built a slot machine with Git access. JOSH: Dry, but fair. ERIK: Devtools have to be inspectable for the same reason. If a tool formats code, touches infra, opens PRs, reads logs, or ships changes, I need to know what it did and why. And when it's wrong, I need a path to fix it. JOSH: So where does this land for builders listening today? ERIK: Pick tools you can bend. OpenTelemetry, NATS, Postgres, GitHub Actions, local CLIs, open plugins. Build on boring parts you can inspect. Then wrap AI around that. Don't build a production system where the only debugging method is asking a chatbot nicely what happened. [pause] JOSH: The second story connects pretty cleanly: LLMs reward expertise. Why is that still surprising people? ERIK: Because the marketing sold magic. The reality is better, but less lazy. Claude can write the code. GPT can reason through an API. Gemini can chew through a giant document. But the operator still matters. [beat] JOSH: Operator meaning the human. ERIK: The human, the prompts, the test harness, the repo context, the acceptance criteria, the rollback plan. All of it. Expertise shows up in what you ask for, what you notice, and what you reject. JOSH: Give me the practical version. ERIK: A junior person says, "Build me a dashboard." A senior builder says, "Use the existing auth middleware, read from this telemetry table, keep the polling under five seconds, don't add a new state library, and write the Playwright check for empty data." Same model. Totally different result. JOSH: That's the whole show in one example. ERIK: Yeah. The prompt isn't magic words. It's compressed experience. [beat] JOSH: How does that show up in your systems? ERIK: ScanBrief is a good example. It doesn't just grab feeds and summarize headlines. It scores items, ranks them, dedupes them, and gives me enough context to decide what belongs in the show. Today it scored 82 items across 56 sources. That's not a replacement for editorial judgment. It's a fast intake system. JOSH: So the machine narrows the field. ERIK: Right. Then I bring the filter. Is this useful? Is it noise? Does it change how builders should act today? The system can surface "LLMs reward expertise." I decide that the real story is engineers becoming more valuable when they know what good looks like. JOSH: That feels opposite of the replacement panic. ERIK: It is and it isn't. Weak task execution gets replaced. Strong judgment gets amplified. If your whole job is moving text from one box to another, rough week. If you understand the business, the infra, the failure modes, and the cost of being wrong, you're suddenly driving a much bigger machine. [pause] JOSH: Let's talk about the local model story. Running an 80B Qwen in 4.3 gigs of RAM sounds like science fiction or a benchmark trap. ERIK: Both can be true. Demos usually hide the ugly parts. Latency, quality loss, context limits, weird quantization behavior. But the signal is still huge. Local inference keeps getting cheaper and more practical. JOSH: What does that change for automation? ERIK: It changes where intelligence can live. Right now a lot of agentic work calls cloud APIs. That's fine for high-value reasoning. But tons of tasks don't need the biggest model. Log triage. Config classification. First-pass runbook matching. Ticket labeling. "Is this alert real or garbage?" That can run close to the machine. [beat] JOSH: On the server itself? ERIK: Sometimes. On the laptop. On a small box in the lab. On an edge device. PrimeSentinel doesn't need a poetry degree to notice a stuck job. It needs logs, timing, thresholds, and a decent classifier. Save the expensive model for the fix plan. JOSH: So local model first, cloud model when it matters? ERIK: That's the pattern I like. Cheap local model watches. Strong cloud model reasons. Rules and tests decide whether anything ships. PrimeDash shows me 144 services running on the production server right now. I don't want all 144 yelling at a frontier model every time a log line looks spicy. JOSH: That would get expensive fast. ERIK: Fast and dumb. You route by severity. A local model can say, "This looks like a retry loop." A rule can say, "It has happened six times in ten minutes." PrimeBus can publish the event. Then Claude gets called only when the system has enough evidence. [beat] JOSH: Where do people mess this up? ERIK: They pick the model first. Wrong order. Start with the workflow. What decision has to be made? What data proves it? What is the cost of a bad answer? Then choose the model. Some jobs need Opus-level reasoning. Some jobs need a regex and a nap. JOSH: That's very technical and also emotionally true. ERIK: Networking taught me that. In NSO, you don't throw genius at every config diff. You validate intent, render config, check state, commit, and roll back if needed. AI should fit into that discipline. Agents are workers. They are not the change-control board. [pause] JOSH: There's also that "200 milliseconds" headline in the list today. Not one of the big AI stories, but it feels related. ERIK: Latency is the tax nobody wants to discuss. If your tool takes two seconds for every small action, users feel it. If an agent waits ten seconds between steps, your automation starts moving like a committee. JOSH: So 200 milliseconds is a product rule? ERIK: It's an experience rule. Fast feedback makes people trust systems. Slow feedback makes them open another tab. In automation, slow loops also hide bugs. PrimeBus works because events move quickly and the system reacts while context is still fresh. JOSH: How should a builder apply that? ERIK: Measure the loop that matters. Not the vanity benchmark. For a devtool, time from save to useful feedback. For an AI agent, time from event to proposed action. For a dashboard, time from service failure to visible state. PrimeDash isn't there to look pretty. It's there so I can see what broke before the morning gets stupid. [beat] JOSH: And that ties back to open devtools again. ERIK: Yep. If you can inspect the tool, you can measure the loop. If you can measure it, you can improve the system. If you can't measure it, congratulations, you bought a mystery. [pause] ERIK: This episode is sponsored by Prime Automation Solutions. If you're still doing it manually, we automate it. Also, special on a website — $250. primeautomationsolutions.com [pause] JOSH: Alright, what's the AI pro tip today? ERIK: Make your agent produce receipts before it touches anything important. Not a summary. Receipts. [beat] JOSH: What does that mean in practice? ERIK: Require three fields before action: files inspected, evidence found, and exact command or change proposed. If the agent says a test failed, make it show the test name. If it says a service is unhealthy, make it show the log line or metric. If it wants to edit code, make it state the acceptance check first. JOSH: So no vibes-based commits. ERIK: Correct. Vibes are for playlists, not production. Add a gate that rejects outputs without evidence. This works with Claude, GPT, local models, whatever. The model can be smart and still skip the boring proof. Don't let it. [beat] ERIK: That's your tip. Use it. [pause] JOSH: We also drop daily market picks and automation tips on YouTube — search Build or Be Replaced. [pause] JOSH: One more thing — we started a Discord for builders. If you're shipping AI, automation, or anything that makes a human obsolete — come hang out. Link at buildorbereplaced.dev. ERIK: Post what you built. We'll post what we're building. Real wins, real builds, no fluff. [pause] ERIK: Build or be replaced. JOSH: If you want these signals in your inbox every morning, scanbrief.dev. See you tomorrow.