From the Backlog to the Build Log

Verifying AI Output, Even When It's Video: October 23 - October 29, 2026

What I Worked On

This week's lesson in Intelligent Engineering was about verifying AI output, and it came from my AI video studio instead of a .NET app. If I can get verification in place for something as subjective as a video, making sure it works in code is easy by comparison.

It was a light week for DevNitro itself. My water heater went out and one of my circuits lost power, so a good part of the week went to the house. I made no commits to the DevNitro repo, and my DevNitro product time came to under four hours of research and tests. Most of my hours went to outreach prep and to the studio I wrote about last week.

Even when I wasn't at my desk, I made sure the computers in my house kept working. Two of the biggest studio jobs ran overnight, and all 27 studio commits came out of DevNitro sessions. Opus wrote the code and its own tests as it built each video with the tools I gave it, so the studio is one more real test of DevNitro. I also do this kind of exercise because it may give me ideas for improving the way I build software with DevNitro.

What the AI Did Well

A lot of what goes wrong in a video is a matter of taste. Even so, the agent found the parts it could measure and wrote checks for them. In a lyric video with a 3D character who plays the accordion, one scan poses every frame without rendering any video and reports by frame number where his body pops or a hand ends up inside the accordion. On Tuesday the agent added rules for a world that is physically right, such as "nothing passes into anything" and "nothing flickers," plus tools that find overlapping solids and frame-to-frame flicker.

For the anime cooking videos I started Wednesday night, the agent set up overnight batches. Each batch renders several dishes, runs a local vision model over the panels and clips, and writes a report of what it flagged, so my morning starts with a list of what to look at.

What Needed Correction

The checks didn't catch everything, and that's the subjective part. Monday night I gave the agent seven and a half hours on its own with a different lyric video and told it to keep going until there were no known issues. It stopped with known issues, and the lyrics lacked context in many scenes. The studio had no check for that yet. Tuesday morning the agent wrote a brief for a fresh agent to measure how the reference videos tie their lyrics into each scene, then rewrite the studio's rules from those numbers. Two of the new rules became checks with unit tests: a sung word has to stay lit for 0.8 seconds, and the planner refuses a storyboard line that only says where the words sit.

What Shipped

I didn't deploy DevNitro this week. On Saturday I released "A Certificate Authority's Private Key Was the Number 4" on HowdyChad. It opens with part of The Key Was Four, the music video I put on HowdyChad Creates on October 22, and then I talk through the security news behind it. On Wednesday I published a video, "AI Copy-Pastes the Same Bug Into Every App. When Do You Stop?" with animation from the studio.

Getting the Outreach Ready

Go-to-market took the most time, about 22 hours. The plan was to send the outreach Monday, but most of the week went to getting the right companies and people into the lists. The GTM app an autonomous run built in September now searches a company's people in one pass, then makes a second pass for managers.

Two Security Checks

Saturday I asked DevNitro whether any of its tool calls had sent personal data in request headers. It wrote a read-only SQL script that lists every web request made through a shell command, flags any custom headers, cookies, auth, or email addresses, and pairs each call with its output. It also confirmed that web_fetch sets its headers in code, so the agent can't add any.

That afternoon I saw an advisory about a compromised LiteLLM release. Other AI tools pull it in, so I had the agent check the whole machine: every Python install and its startup files, WSL, my repos, and my Docker images, which it inspected without starting them. Nothing held a bad version, and PyPI had already pulled the release. The agent also named the limit. The advisory's indicators weren't public yet, so the check rests on versions and files.

DevNitro Ideas, Parked for Now

Thursday night I wrote down what I want in DevNitro when the timing is right: workers that run on other subscriptions under a main agent, main agents that talk to each other, remote machines, scheduled issues, and a handoff file before compaction. Most of the list comes from one note: if something is working, get it out of my face until it needs me. It is taking everything within me not to continue to create features in DevNitro at the moment. GTM is what is important, and that starts Monday.

The Numbers

Category Hours
Business 22h 7m
Content 19h 16m
Product / Features 2h 52m
Testing 0h 52m
Total 45h 7m

DevNitro product, test, and deploy time came to 3h 44m, down from 11h 58m last week. Go-to-market took 22h 7m, up from 13h 2m, and 8h 42m of the content time went to the studio.

Where This Fits in Intelligent Engineering

Code is the easier case. A test passes or fails, and a build compiles or it doesn't. Video has some of those same facts in it, like a foot below the floor or a word that goes dark too soon, and the agent turned each one it found into a check that runs. What a check can't judge yet, like whether a lyric fits its scene, still comes to me, and each new rule moves a little more of that to the checks. Intelligent Engineering asks the same of a .NET app. You decide what done means, write it down as a check that runs, and let the check prove the outcome, instead of reading every line the AI writes.

Each week in the studio teaches me something about verifying AI output that I can bring back to DevNitro. If you want to see DevNitro in action, grab a spot on my calendar and I'll give you the tour. If you want to see how your team can be more efficient using AI (even using your existing tools), I can help with that as well. Back at it next week.