Mon–Fri · 6 AM ET
← All Episodes
EP  • 00:10:00

elementalsouls/Claude-BugHunter: A Claude Code skill bundle for bug… | Build or Be Replaced

Today: elementalsouls/Claude-BugHunter: A Claude Code skill bundle for bug hunting and external red-team work - 82 skills, 15 slash commands, 681 disclosed-report patterns curated across 24 core vulnerability classes, plus enterprise identity + infrastructure attack matrices. | NVIDIA/SkillEvaluator: Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. | Qwen4-Max Episode date: 2026-11-16.

Download MP3 →

Transcript

JOSH: It's Monday, November 16. This is Build or Be Replaced — powered by ScanBrief.dev. I'm Josh, here with Erik Anderson.
ERIK: Give an AI permission to fix your website, and you'd better know whether that includes changing the rules for publishing it.
JOSH: Stick around — Erik's got an AI pro tip at the end about keeping an agent's retry from becoming your duplicate invoice.
JOSH: First, the headlines. Old Bitly links being hijacked: what's the builder's takeaway?
ERIK: Go check the links your customers still use. An old email doesn't expire because you forgot you sent it. Neither does the trust somebody puts in a link carrying your business name.
JOSH: Where would you start?
ERIK: Customer onboarding emails. Documentation. Anything that sends somebody toward a login or a payment. Follow the redirect and check the destination. That's a small maintenance job with a very good reason to exist.
JOSH: Next headline: Protomaps says self-hosting map tiles has gotten cheap and easy. Interesting?
ERIK: Absolutely, if maps are part of your product. But price the maintenance before you celebrate the hosting bill. Owning a dependency means owning the update process and knowing what happens when it breaks.
JOSH: And Bharat Semi lands a $24 million seed round for AI sensor technology. Your reaction?
ERIK: Funding buys room to build. My interest is the job the sensor can do. When you're choosing technology, start with the thing you need to measure and the decision that measurement supports. A funding headline doesn't answer either question.
JOSH: Back to that opening warning. Where does an ordinary request turn into something dangerous?
ERIK: At the handoff between understanding a request and authorizing an action. Claude can read an email and figure out what somebody wants. That doesn't mean the email gets to decide which files Claude can change or whether a change goes live.
JOSH: Put that in terms of something you actually run.
ERIK: email-reactor handles client website change requests through email. That's a useful place to draw the boundary. The message describes the work. The application controls what work is permitted. You want those decisions separated before an agent starts touching anything.
JOSH: A client says, “Change our hours.” What should happen?
ERIK: Identify the client, identify their website, locate the hours, and prepare the change. The permission should be narrow enough that changing hours doesn't quietly become changing account access. Same email address. Completely different authority.
JOSH: But the client owns the website. Why wouldn't their instructions be enough?
ERIK: Because a request can contain mistakes, forwarded material, or instructions meant for somebody else. Authentication tells you who submitted something. You still need to decide whether the requested action belongs within the access you've granted.
ERIK: Think about a forwarded email chain. The current message might say, “Use these hours.” Farther down, somebody wrote, “Delete the old page.” You don't want the agent treating every sentence in the chain as a fresh order.
JOSH: Wouldn't a strong prompt catch that?
ERIK: A strong prompt helps the model interpret the message. It doesn't replace access controls. Tell Claude to edit the right website, then give the editing tool access to that website. Don't hand it access to every client and hope the wording holds.
JOSH: That sounds like network engineering with a chat window.
ERIK: Exactly. Coming from network automation, this part feels familiar. You define the intended change, check the scope, inspect the result. An agent makes the interface more flexible. It doesn't make unrestricted credentials a better idea.
JOSH: Where does PrimeBus fit into that?
ERIK: PrimeBus carries telemetry into my auto-review and auto-merge pipeline. Events help the system decide when work needs attention. But the text inside an event shouldn't be allowed to rewrite the policy for handling that event.
ERIK: Imagine a failing test includes a response from an outside service. That response lands in a log. Claude reads the log while diagnosing the failure. Useful evidence. But any command written inside that response is still outside content.
JOSH: Even after your own system has copied it?
ERIK: Especially then. Copying something into a trusted system doesn't make its author trusted. Put an origin on it. Keep it identifiable as data. Let the worker's permissions come from configuration the incoming message can't edit.
JOSH: Could you solve that at the NATS layer?
ERIK: Use NATS permissions to control who can publish and subscribe where. That's part of the design. Then validate the event and check the requested action at the worker. Transport access and permission to modify a repository answer different questions.
JOSH: What would you test first?
ERIK: Send a harmless request containing an instruction outside its scope. Ask for an hours update, then include a sentence requesting a deployment-policy change. The expected result is the authorized edit and a refusal of the extra instruction.
ERIK: Run that in a lab with a disposable website. Inspect the actual files and tool calls. A response saying “I respected the boundary” isn't evidence that the boundary held. The change itself is the evidence.
JOSH: And if the agent refuses everything?
ERIK: Then you haven't finished. A useful boundary lets legitimate work through. Test the ordinary request too. Otherwise you've built a very expensive mailbox that says no.
JOSH: That gets us to the next problem. How do you tell whether an agent instruction actually makes the system better?
ERIK: Give it work where you can judge the result. A skill file can sound excellent and still make an agent worse. My position is simple: instructions earn their place by changing behavior in a useful way.
JOSH: What's a fair test?
ERIK: Start with a task you understand. For code review, use a change containing a defect you can demonstrate. Ask the agent to review it with the skill available. Then review the same change without that skill, keeping the rest of the setup comparable.
JOSH: What counts as a win?
ERIK: Finding the defect and explaining how it affects the code. Preferably with a check you can run. Saying “consider better error handling” doesn't count if the actual bug is a duplicate write after a timeout.
JOSH: Can an agent pass by being suspicious of everything?
ERIK: Absolutely. That's why you also need working code in the evaluation. A reviewer that blocks every change can look wonderfully cautious. Nothing ships. The dashboard stays peaceful. Customers may have a different opinion.
JOSH: How would you apply that to PrimeBus?
ERIK: For an auto-review pipeline, I'd evaluate the decision separately from the explanation. Did the review permit the safe change? Did it block the unsafe one? Then look at whether its explanation identifies the real reason.
ERIK: An agent can make the right decision for the wrong reason. That matters because the next change might look similar while behaving differently. You want evidence tied to the code, not a convincing paragraph that happened to land on the right answer.
JOSH: What about overlapping instructions?
ERIK: That's worth testing directly. Suppose a general instruction says to preserve public interfaces, while a task-specific instruction encourages broad cleanup. Give the agent a small fix near an interface. See whether it keeps the change focused or starts renovating the neighborhood.
JOSH: Sounds familiar.
ERIK: Every engineer has met the tiny bug fix that arrived carrying a new framework. Agents can do that too. The review should ask whether each changed file helps satisfy the request. Unrelated improvements belong in separate work.
JOSH: How much evaluation does somebody need before trying this?
ERIK: Start with cases that represent the decision you're handing over. An ordinary success. A request outside scope. A failure that needs human attention. Those are test cases, not a claim that you've covered everything.
ERIK: Save the input, the tool calls, and the resulting changes. When you revise the instructions, run those cases again. Now you can spot a regression instead of arguing about which version sounds more professional.
JOSH: Where do people usually make this harder than it needs to be?
ERIK: Changing everything together. New model, new instructions, new tools, new permissions. Then something improves and nobody knows why. Hold the environment steady while you test the instruction change. Give yourself a result you can explain.
JOSH: What would you ship first?
ERIK: A narrow task with a clear pass condition and a recoverable result. Preparing a website edit for review is a useful starting point. Once the evidence supports more authority, expand it deliberately. Don't confuse a fluent answer with permission to touch production.
JOSH: Before the tip, a word from our sponsor.
ERIK: This episode is sponsored by Prime Automation Solutions. If you're still doing it manually, we automate it. Also, special on a website — $250. primeautomationsolutions.com
JOSH: Alright, what's the AI pro tip today?
ERIK: Put a stable operation ID on any agent action that can create something you don't want duplicated. An invoice. A ticket. A client notification. Generate that ID in your application before the tool call, and reuse it if the call needs another attempt.
JOSH: Why does the retry need the same ID?
ERIK: Because a timeout doesn't tell you whether the action happened. The service might have created the invoice and lost the response. If the agent tries again with a fresh identity, it can create another invoice.
ERIK: Where the destination supports idempotency keys, pass that stable ID through its supported mechanism. For your own service, enforce uniqueness and track the operation's state. A completed retry should return the existing result. An uncertain result needs reconciliation before another write.
JOSH: What's the test someone can run today?
ERIK: In a sandbox, let the write succeed, then deliberately drop the response. Retry with the same operation ID and inspect the destination. You should find the original result without a duplicate. Ask Claude to help build that test before you give the workflow real credentials. That's your tip. Use it.
ERIK: That tip is straight out of The Autonomous Engineer — my book on building systems that run themselves. Grab it on Amazon.
JOSH: One more thing — we started a Discord for builders. If you're shipping AI, automation, or anything that makes a human obsolete — come hang out. Link at buildorbereplaced.dev.
ERIK: Post what you built. We'll post what we're building. Real wins, real builds, no fluff.
ERIK: Build or be replaced.
JOSH: If you want these signals in your inbox every morning, scanbrief.dev. See you tomorrow.