<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[working theory]]></title><description><![CDATA[Perspectives from engineers on software engineering, AI, infrastructure, security, and the problems we’re working through.]]></description><link>https://newsletter.depot.dev</link><image><url>https://substackcdn.com/image/fetch/$s_!hswk!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F075047b8-c4c5-44f7-a61f-0b325d00153d_512x512.png</url><title>working theory</title><link>https://newsletter.depot.dev</link></image><generator>Substack</generator><lastBuildDate>Tue, 22 Sep 2026 15:15:26 GMT</lastBuildDate><atom:link href="https://newsletter.depot.dev/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Depot Technologies Inc.]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[workingtheoryeng@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[workingtheoryeng@substack.com]]></itunes:email><itunes:name><![CDATA[working theory]]></itunes:name></itunes:owner><itunes:author><![CDATA[working theory]]></itunes:author><googleplay:owner><![CDATA[workingtheoryeng@substack.com]]></googleplay:owner><googleplay:email><![CDATA[workingtheoryeng@substack.com]]></googleplay:email><googleplay:author><![CDATA[working theory]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[How I delegate the work, not my judgment]]></title><description><![CDATA[A diff tells you what changed, not why. How to hand agents the work while keeping the intuition to challenge what they produce.]]></description><link>https://newsletter.depot.dev/p/how-i-delegate-the-work-not-my-judgment</link><guid isPermaLink="false">https://newsletter.depot.dev/p/how-i-delegate-the-work-not-my-judgment</guid><dc:creator><![CDATA[Iris Scholten]]></dc:creator><pubDate>Wed, 19 Aug 2026 21:16:02 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/9e238535-2252-47bc-8865-b82a015cb7be_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I&#8217;m doing so much more with agentic workflows. Agents do the tedious work and are way better at multitasking than my own brain. But how do I maintain my knowledge of the whole system?</p><p>If I&#8217;m using agents to write code, investigate issues, and fix weird bugs, it becomes much harder to commit that work to memory. I can read a final diff and understand what it does, but that&#8217;s not always the same thing as understanding <em>why</em> the system works the way it does or having the intuition to know when a proposed change is off.</p><p>Even if an agent writes the change, I maintain and own the system (at least for now).</p><h2>The old way I built context</h2><p>Before I relied much on agents, I built context through the whole process of solving a problem. I&#8217;d investigate an issue, read logs, and figure out what was happening. Then I would write a ticket capturing the problem and relevant context so others (and especially future me) would know what was going on and why we were doing this. When I started working on it, I would need to find where the behavior lived in the codebase, understand the existing implementation, and decide how to change it. Finally, I would test and validate the result.</p><p>That process was not always fast, but a useful side effect was that by the end, I had a mental model of that part of the system and pretty deep context about tradeoffs and why changes were made. I knew where that behavior lived, what had caused the issue, what alternatives I had considered, and what other pieces of the system could be affected. Even when I forgot the finer details, I still had higher-level intuition about generally how that part of the system worked.</p><h2>What agents changed</h2><p>Agents can now move through that process incredibly quickly. But I have noticed a tradeoff. If the entire workflow happens through agents, I can end up mostly looking at the final answer. I know the bug was fixed. I can see the before and after repro. I can see the diff and the passing tests. But I may not have built the context that I used to get by working through each step myself.</p><p>That can slowly take away my intuition about the system, the architecture, and the codebase. The question I keep coming back to is &#8220;If I am mostly reading the final answer, when am I building the intuition to challenge it?&#8221;</p><h2>I&#8217;m responsible for judgment</h2><p>If agents can also debug the next problem and implement the next change, why do I still need to understand the system? I do not need to retain every implementation detail the way I did in the before agent times, but I do need a strong enough mental model to direct the work, evaluate the result, and make decisions that shape the system over time.</p><p>I am happy to delegate time-consuming work. But delegation does not transfer ownership. I&#8217;m still responsible for how the system behaves in production, whether a proposed change fits its architecture, and what tradeoffs we make as it evolves.</p><p>That understanding is also what allows me to use agents effectively. Agents can produce answers that are plausible, technically valid, and still wrong for the system as a whole. Without enough context of my own, I cannot reliably challenge those answers or recognize when a local fix may create a larger problem. Choosing between several reasonable approaches also becomes harder if I first have to relearn the system or depend on an agent to surface every relevant constraint.</p><p>I want to delegate the work without delegating away my judgment. I still want to be able to explain why we made a change and remember that later when investigating issues or making further changes.</p><p>So far, I&#8217;ve made a few changes to my personal workflows that help me keep building that intuition.</p><h2>Stay involved in planning</h2><p>Agents can create a reasonable plan quickly and it&#8217;s tempting to accept it and move on. But planning is also where requirements, priorities, scope, and architectural tradeoffs get decided. Those decisions can disappear into an agent&#8217;s implementation if I am not paying close enough attention, especially while multitasking agents across different tasks (more on this later).</p><p>My general rule of thumb is that if something needs a decision, I want it to reach me and not be an invisible choice made by the agent that I only notice after the code is written.</p><p>One tool I find useful is Compound Engineering&#8217;s brainstorm skill. I start by giving it the context of what I want, and it follows up with questions that make me clarify the problem before we settle on a plan. That gets me past a hand-wavy feature outline and makes the decisions behind the work more explicit. For details I have not already specified, it makes me weigh tradeoffs I may not have considered yet and clarify the scope. Some of those boundaries might seem obvious to me, but making them explicit helps both me and the next agent working on the code.</p><p>Another tool I&#8217;ve been trying out is the <code>grill-with-docs</code> skill. It starts by exploring the relevant code and existing domain documentation and then asks high-leverage questions one at a time about the plan or domain concept. It can challenge vague terminology, point out when my assumptions conflict with the codebase, and help surface edge cases I might otherwise overlook.</p><p>The value is not that these tools make design decisions for me. They make it harder for me to handwave a decision that needs to be made before implementation begins. Being more deliberate about this part of the design-to-shipping process has helped me retain context.</p><h2>Review agent work more pedantically</h2><p>Agent output can easily land in a place where it technically works, the tests pass, and the overall approach seems reasonable. That does not automatically mean it&#8217;s written the way it should be. So I treat review as more than a final correctness check; it&#8217;s also the point where I make sure the code reflects the decisions, patterns, and level of care I want to maintain.</p><p>This means going deep on things that can look minor at first glance, such as where code lives, whether names make the intent obvious, whether an abstraction is at the right level and provides the intended value, and whether comments are adding useful context or just adding more text to read.</p><p>I also use review to make sure I understand why each meaningful change exists. The question is not only &#8220;does this work&#8221; but also &#8220;would I have made these same choices&#8221; and &#8220;is this how I want the codebase to evolve.&#8221; Sometimes that means keeping the approach; sometimes it means changing it to something simpler or easier to read.</p><h2>Cap multitasking</h2><p>One place where my workflow is wildly different from when I started using agents in my day-to-day is the amount of multitasking I do. Agents make it easy to kick off a lot of work at once, which is especially powerful when doing initial investigations. But investigation is often only the beginning.</p><p>My involvement usually increases as a task moves into planning, implementation, review, and validation. Not every task demands the same amount of attention, either. Fixing a small bug with clear expected behavior in a familiar part of the codebase is very different from building a new feature that requires decisions about scope, architecture, and how the system should behave. Some changes need very little handholding, while others need me closely involved.</p><p>My agents can do a lot of work, but I still have only one singular brain. I might be able to ask agents to pick up several unrelated problems at once, but I cannot deeply plan and design five large unrelated features and fixes at once. If I try, I end up with shallow context on all of them.</p><p>Because of that, my limit isn&#8217;t a fixed number of tasks. I try to judge how much involvement each one will need and how that will change as the work progresses. I may have more tasks running while agents are gathering information, then narrow my focus once those tasks begin requiring decisions and careful review. The question is no longer &#8220;How much can I kick off at once?&#8221; It is &#8220;How much can I reasonably stay engaged with at once?&#8221; Being disciplined about that helps me benefit from the amount of parallel work agents can do without creating an information firehose that I cannot meaningfully process.</p><h2>Staying in the loop does not mean being involved in everything</h2><p>I don&#8217;t want to be personally involved in every little thing my agents do. A small wording change or light refactor does not need the same level of thought as building out a new feature or changing how a system behaves. The goal is to spend more time where the work establishes new patterns, changes existing behavior, or interacts with a complex system.</p><p>Agents are making it much faster to produce code. For me, that makes it more important to deliberately maintain context as someone who has to keep owning and maintaining the systems behind that code. I am still figuring out the right balance. But I want to make sure agents help me move through the codebase faster, not leave me behind.</p><h2></h2>]]></content:encoded></item><item><title><![CDATA[GitHub is the wrong shape for this new world]]></title><description><![CDATA[The branch-and-pull-request model was built for humans typing slowly. That's not how code gets written anymore.]]></description><link>https://newsletter.depot.dev/p/github-is-the-wrong-shape-for-this</link><guid isPermaLink="false">https://newsletter.depot.dev/p/github-is-the-wrong-shape-for-this</guid><dc:creator><![CDATA[Kyle Galbraith]]></dc:creator><pubDate>Wed, 29 Jul 2026 21:28:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/125fbc93-276b-4daa-a9df-e4a06f3d9793_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>A lot of attention is given to the overall performance and reliability of GitHub. Rightfully so. But I think the more interesting thing to question is the paradigm that we use with GitHub. It&#8217;s the wrong shape for how we build software today, and we need better tools, workflows, and infrastructure primitives that meet the new demand.</p><h2>Software engineering has changed</h2><p>I realize this statement is a lot like saying &#8220;water is wet.&#8221;</p><p>We live in a fundamentally different world for developing software today. Are there familiar things? Definitely. Are there things from the past that we can use in this new world? Absolutely. Code is still code after all. As a mentor used to preach to me, it&#8217;s bytes in and bytes out.</p><p>The bottlenecks we underinvested in are still around, from source control to CI/CD to code review and even deployments. Only now they are killing the newfound velocity we have.</p><p>The pain of these bottlenecks is compounding because everyone can now contribute code. Code has expanded outside of an engineering team and into other teams like sales, support, and marketing. Meaning, a 10-minute build is felt by literally everyone in a company, not just a small subset.</p><p>This has profound implications, as it calls into question the assumptions and things we have accepted as normal. It raises questions that I think are still unanswered:</p><ul><li><p>Who owns the code? Or, put another way, who is responsible for the code that gets deployed?</p></li><li><p>How do we trust the code?</p></li><li><p>How do we review all of this new code being generated?</p></li><li><p>How do we connect code from one place to work with code in another place?</p></li><li><p>How do we weigh the costs of token spend with the output of quality features?</p></li></ul><p>Ask these five questions of five different people, and you&#8217;ll most likely get five different answers.</p><p>This is what I mean when I say that software engineering has changed. It&#8217;s not just that LLMs and agents can generate an entire feature off a five-sentence prompt. That&#8217;s a tool we are using.</p><p>It&#8217;s that the entire paradigm around software engineering is now drastically different and moving at a velocity that we simply can&#8217;t keep up with. Assumptions we have made are crumbling with each passing release.</p><p>I believe we must rethink our paradigm and the tools we are using. We are bending our existing tools and human-centric paradigms into this new world. This is the wrong approach.</p><h2>Collaboration vs infrastructure</h2><p>Really, this blog post should be titled something like &#8220;GitHub, GitLab, Bitbucket, and others are the wrong shape for this new world.&#8221; But that&#8217;s a mouthful, so when I say GitHub, read it as all of them.</p><p>I think of GitHub as a collaboration tool. Google Docs is a place where I can go to collaborate on this blog post with our technical writing team. GitHub is a place I can go to collaborate with other engineers on the code that makes up Depot.</p><p>I grew up with GitHub. In fact, I wanted to work at GitHub when I first became a software engineer. My entire muscle memory is built around the paradigm that GitHub pushed me to use.</p><ul><li><p>Write code in a branch.</p></li><li><p>Open a pull request when I&#8217;m ready for my changes to be reviewed.</p></li><li><p>Wait for CI and checks to pass to validate that everything works.</p></li><li><p>Review comments from my colleagues and go back and forth on ideas in the PR.</p></li><li><p>Once everything is green and my colleagues have signed off, merge my changes.</p></li></ul><p>There are derivatives of this workflow. There are wildly differing opinions on the effectiveness of this paradigm. But ask any engineer, and they will know it like the back of their hand.</p><p>When LLMs first started to come online, it was natural to bolt them onto our existing paradigm. They could write code in a branch, open a pull request, get a code review from a human (or another agent), use the PR to collaborate via comments, run our tests via our existing CI pipelines, and eventually merge the code.</p><p>It was logical. The systems existed, and the agents at that time still largely moved at human speed.</p><p>Then everything changed. Agents with the latest generation of models got incredibly good. Freakishly good. We went from models where we often had to nudge the Jenga blocks back into place to get good code out, to models that produce consistently correct code given enough context.</p><p>What&#8217;s the net result of better models? More code, more branches, more parallel work, and more strain on our existing clunky human-powered paradigm.</p><p>Our collaboration pattern, and the underlying systems that back it, are now the primary bottleneck. Every engineering team is feeling bottlenecks in GitHub itself, CI, code review, pull requests, security scanning, and even deployments.</p><p>There is now an impedance mismatch between our tools and how we want to build. Our existing collaboration paradigm, largely built around GitHub, is now in conflict with how we actually build software today.</p><p>We must rethink our paradigm for software delivery. Not through the lens of human-centered collaboration, but through high-throughput infrastructure primitives.</p><h2>High throughput software delivery</h2><p>Building a new software delivery paradigm requires taking a step back from the human-centric process that we have used over the last decade. Rethinking software delivery, not as a sequence of human actions, but as a continuous automated process driven by systems.</p><p>Once you start imagining that world, the problem of software delivery starts to look less like a collaboration tool and more like infrastructure. Where infrastructure, in this context, means a small number of primitives on which everything else can be built. We can look to how we have adopted cloud-native technologies for inspiration:</p><ul><li><p>Compute as a primitive gave us VMs</p></li><li><p>Storage gave us object stores</p></li><li><p>Networking gave us load balancers</p></li></ul><p>We should be thinking about the fundamental infrastructure primitives we need for software delivery:</p><ul><li><p><strong>Source control</strong>: durable, immutable history of how code evolved</p></li><li><p><strong>Execution</strong>: isolated, performant compute for building, testing, and validating changes</p></li><li><p><strong>Artifacts</strong>: reproducible outputs that can be moved across systems</p></li><li><p><strong>Caching</strong>: reuse of deterministic work</p></li><li><p><strong>Identity</strong>: proving who or what produced code</p></li><li><p><strong>Policy</strong>: machine-enforceable rules for quality, security, and compliance</p></li></ul><p>These aren&#8217;t features of a developer tool. They&#8217;re infrastructure primitives. Just as we stopped thinking about provisioning individual servers and started thinking about compute, storage, and networking as composable building blocks, software delivery needs the same shift.</p><p>The winners of the next decade won&#8217;t build a better pull request. They&#8217;ll build the infrastructure that makes software generation, validation, and deployment operate at machine scale.</p>]]></content:encoded></item><item><title><![CDATA[Improving system safety with Temporal Logic of Actions (TLA+)]]></title><description><![CDATA[Agents write the TLA+ spec from your code. On the rebuilt Registry GC it caught a live-data-deleting race.]]></description><link>https://newsletter.depot.dev/p/improving-system-safety-with-temporal</link><guid isPermaLink="false">https://newsletter.depot.dev/p/improving-system-safety-with-temporal</guid><dc:creator><![CDATA[Wito Delnat]]></dc:creator><pubDate>Mon, 20 Jul 2026 21:01:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/0c1d6632-05db-46b7-88bb-8151918803da_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>The hardest distributed systems bugs to find aren&#8217;t in any single operation. Two processes each do the right thing, but in an order nobody thought of, and data disappears. Tests don&#8217;t catch these bugs because tests run the interleavings you imagined, while the bug is in the one you didn&#8217;t.</p><p>At the scale of Depot Registry, this is no longer hypothetical. Once you reach a certain request volume, a one-in-a-million interleaving becomes a reality and happens on a schedule you don&#8217;t control.</p><p>So when we rebuilt the garbage collector for <a href="https://depot.dev/blog/now-available-depot-registry-v2">Depot Registry</a>, we model checked it with TLA+. The model checker found a real bug that our tests and reviews had missed. It also forced us to be precise about a design decision that sounds absurd on first read: our registry stores immutable, content-addressed blobs (that by definition never change), and it depends on S3 bucket versioning anyway.</p><h2>What TLA+ is</h2><p><strong>TLA+</strong> is a language for describing a system as states and transitions. <strong>TLC</strong> is a model checker that explores every reachable state and every possible interleaving, then tells you whether your invariants hold in all of them. You don&#8217;t write the implementation in TLA+. You write a simplified model of it that&#8217;s small enough for the checker to explore in full, but close enough to the real thing that a bug in the model points at a bug in your system.</p><p>Here&#8217;s a trivial model: two clients withdrawing from a shared wallet, each doing a read-then-write without locking.</p><pre><code><code>---------------- MODULE Wallet ----------------
EXTENDS Integers

VARIABLES balance, read

Init == balance = 10 /\ read = [c \in {"a", "b"} |-&gt; -1]

Check(c) == read[c] = -1
            /\ read' = [read EXCEPT ![c] = balance]
            /\ UNCHANGED balance

Withdraw(c) == read[c] &gt;= 8
               /\ balance' = balance - 8
               /\ read' = [read EXCEPT ![c] = -2]

Next == \E c \in {"a", "b"}: Check(c) \/ Withdraw(c)

NoOverdraft == balance &gt;= 0
================================================</code></code></pre><p>Each definition is a transition: <code>Check(c)</code> reads the balance, <code>Withdraw(c)</code> subtracts 8 if the client sees enough money. <code>/\</code> means &#8220;and,&#8221; while <code>\/</code> means &#8220;or.&#8221; The primed variable <code>balance'</code> is the value in the next state, and <code>NoOverdraft</code> is the invariant we want to hold everywhere.</p><p>TLC breaks it in four steps: client <code>a</code> checks and sees 10, client <code>b</code> checks and sees 10, both withdraw, and the balance is -6. That&#8217;s the classic check-then-act race, found mechanically, with a step-by-step trace showing exactly how to reproduce it. No test run was unlucky. The checker simply tried every ordering.</p><p>The wallet is a toy example. Here&#8217;s the same idea in one of the real invariants from our registry GC model:</p><pre><code><code>ManifestNeedsData ==
    \A p \in PusherIDs: manifestExists[p] =&gt; s3Versions /= {}</code></code></pre><p>Read it from left to right:</p><ul><li><p><code>ManifestNeedsData</code> is the name, and <code>==</code> means &#8220;is defined as.&#8221;</p></li><li><p><code>\A</code> means &#8220;for all.&#8221;</p></li><li><p><code>p \in PusherIDs</code> means <code>p</code> ranges over every pusher in the model.</p></li><li><p><code>manifestExists[p]</code> asks whether pusher <code>p</code> has a committed manifest.</p></li><li><p><code>=&gt;</code> means &#8220;if the left side is true, the right side must be true.&#8221;</p></li><li><p><code>s3Versions /= {}</code> means the set of S3 versions is not empty.</p></li></ul><p>Put together, the invariant is roughly this pseudocode:</p><pre><code><code>for each p in PusherIDs:
    if manifestExists[p]:
        assert s3Versions is not empty</code></code></pre><p>The last clause might look too weak: it says <em>some</em> S3 version exists, not that the <em>right</em> blob exists. That&#8217;s intentional. This model has a single blob digest, because the race we care about is whether GC can delete that blob while a manifest still needs it. In a multi-digest model, the invariant would need to be indexed by digest: <code>s3Versions[manifestDigest[p]] /= {}</code>. Reviewing a model means asking whether choices like this preserve the question you wanted to answer.</p><p>Model the moving parts (uploads, database transactions, GC workers), state what must always be true, and let the checker do to your design what production traffic eventually will.</p><h2>Why we can afford TLA+ now</h2><p>TLA+ has been around for decades, and it has a reputation problem: everyone agrees it&#8217;s powerful, but almost nobody budgets the weeks it takes to write and maintain a faithful model next to a moving implementation. That was our position too. Before this year, a spec for our GC would have lost the prioritization fight every single time.</p><p>What changed is that we don&#8217;t write the model by hand anymore. An agent reads the implementation, the Go transactions and SQL and S3 calls, and translates it into a spec. We review the rest: do the invariants say what we mean, and does the model abstract away the right things? Writing TLA+ was the expensive part. Deciding what must always be true was always the cheap part, and it&#8217;s the part that stays human.</p><p>The result is a spec of the registry&#8217;s three-tier garbage collector that models concurrent pushers, two GC domains, a counter reconciler, and injected counter drift, all interleaved. It&#8217;s rooted in the real implementation, transaction by transaction. TLC explores 14,290,224 distinct states in about 21 minutes and proves 10 safety invariants and 2 liveness properties. The most important invariant is the first one: a committed manifest never loses its blob data.</p><p>We now use AI to ship faster like everyone else. But we&#8217;re also using it to build systems that are more correct than what we had the time to verify before.</p><h2>The race: why our immutable blobs use S3 versioning</h2><p>OCI registries are content-addressable. Within Depot Registry, blobs live at <code>blobs/sha256/&lt;digest&gt;</code>, and the digest is the hash of the content. Upload the same blob twice and you get byte-identical data at the same key. Nothing ever changes in place. Under that model, S3 bucket versioning looks pointless: every version of an object would be identical.</p><p>But the problem is the deletes.</p><p>Garbage collection has to remove blobs that nothing references anymore. The GC worker marks a blob with zero references, waits out a grace period, re-verifies, and deletes. But the references live in MySQL and the bytes live in S3, and there is no transaction that spans both systems. Which opens a gap:</p><ol><li><p>GC verifies the blob has zero references and decides to delete it.</p></li><li><p>Concurrently, a client pushes an image containing that exact blob. Same digest, same key. The upload writes to S3 and commits a new reference.</p></li><li><p>GC&#8217;s delete lands and removes the object that a just-committed manifest now points to.</p></li></ol><p>Every individual step is correct. Yet the interleaving deletes live data. And &#8220;a client re-pushes a blob right as it becomes garbage&#8221; is not exotic: it&#8217;s what happens when a popular base image cycles out of use and back in.</p><p>You could try to fix this with locks or with ever-more-careful re-checking, but you can&#8217;t re-check S3 and delete in one atomic step. So instead we made the delete itself precise. The bucket has <strong>versioning enabled</strong>: a re-upload of the same key becomes a new version instead of overwriting. When GC marks a blob, it records the specific S3 version ID it saw. When it deletes, it deletes only that version:</p><ol><li><p>GC marks the blob and captures version <code>v1</code>.</p></li><li><p>The concurrent push writes the same bytes as version <code>v2</code> and commits its reference.</p></li><li><p>GC deletes <code>v1</code>, and only <code>v1</code>. <code>v2</code>, the version the new manifest was built on, is untouched.</p></li></ol><p>We don&#8217;t use versioning to keep history. Every version of a blob is byte-for-byte identical, so there&#8217;s no history to keep. We use it as a delete fence: it turns &#8220;delete this key&#8221; into &#8220;delete exactly the bytes I inspected,&#8221; which makes a delete safe to race against a write. That&#8217;s the answer to the puzzle in the title of this section, and it&#8217;s a good pattern for any content-addressable store that garbage collects: versioning, not immutable content, makes deletion safe.</p><h2>Turning a doubt into a line of TLA+</h2><p>One concern in the registry design was how its reference counters behave under failure. To reduce contention on hot blobs, we use a lightweight saga: increment a reference counter, do the work, and compensate with a decrement on failure. A reconciler repairs counters when a crash prevents that compensation from completing. The important detail is that a wrong counter is not symmetric. Overcounting delays GC, while undercounting can make GC treat a referenced blob as garbage and delete live data.</p><p>That asymmetry was our doubt, so we told the agent to verify exactly that. It came back with an invariant:</p><pre><code><code>ManifestCountNeverUndercounts ==
    blobActive =&gt; blobManifestCount &gt;= TrueGlobalManifestCount</code></code></pre><p>Whenever the blob&#8217;s row is active, the <em>stored</em> counter must be at least the <em>true</em> count derived from the physical link rows. The <code>&gt;=</code>, rather than <code>=</code>, captures the asymmetry directly: overcounts are tolerated, undercounts are a violation.</p><p>For that invariant to tell us anything useful, the model also has to include incorrect counters. We added a dedicated drift process that changes them in the same directions as failed compensations and historical desynchronization:</p><pre><code><code>process DriftInjector = "drift"
begin
    Drift:
        await driftBudget &gt; 0;
        either
            await blobActive /\ blobLinkCount &gt; 0;
            blobLinkCount := blobLinkCount - 1;   \* undercount
        or
            await blobActive /\ blobLinkCount &lt; N + DRIFT;
            blobLinkCount := blobLinkCount + 1;   \* overcount
        end either;
        driftBudget := driftBudget - 1;
        goto Drift;
end process;</code></code></pre><p>This part of the spec is written in PlusCal, a front-end syntax that compiles down to TLA+, which is why it reads like pseudocode. <code>await</code> blocks the step until its condition holds, and <code>either/or</code> is nondeterministic choice: TLC explores both branches wherever this process could run, interleaved with the pushers, GC workers, and reconciler. Rather than representing one specific bug, the process represents the broader condition that a counter may be wrong in either direction when another operation runs. The checker can then verify that destructive steps recount the physical rows instead of relying on a stale counter.</p><p>The model also has invariants for its own internal bookkeeping:</p><pre><code><code>S3HeadOK ==
    /\ (s3Versions = {}) &lt;=&gt; (s3Current = 0)
    /\ (s3Versions /= {}) =&gt;
        /\ s3Current \in s3Versions
        /\ s3Current = MaxVersion(s3Versions)</code></code></pre><p><code>S3HeadOK</code> says the &#8220;current version&#8221; pointer is empty exactly when the version set is empty, and otherwise points at the newest version. It does not express a product guarantee; it checks that the model&#8217;s S3 abstraction remains internally consistent. These checks help distinguish a failure in the system being modeled from a mistake in the model itself.</p><p>Tests and model checking cover different ground. Tests exercise concrete implementations, while the model explores interleavings that would be difficult to reproduce deliberately.</p><h2>Tips and tricks on getting started yourself</h2><p>Formal verification used to be a luxury reserved for teams with time to burn. That constraint is gone. The tedious part, faithfully translating an implementation into a spec, is now something you can delegate to an agent, while you make the judgment calls. Here&#8217;s what we wish we knew six months ago.</p><p><strong>Pick your battles:</strong> Don&#8217;t throw TLA+ at everything. Good candidates are competing processes, tricky transactional boundaries, autonomous workers racing on timing, or work spanning multiple systems with no shared transactions. For example, a CRUD endpoint doesn&#8217;t need a model checker.</p><p><strong>Have a bias for action:</strong> While you&#8217;re still learning TLA+, the goal is not a correctness proof. That comes later, if ever. When people hear &#8220;TLA+,&#8221; the first objection is always &#8220;but what if the model doesn&#8217;t match reality?&#8221; That&#8217;s not the point: testing and formal verification both exist to increase trust, and a model checker is one more trust dial. So don&#8217;t wait until you understand every line of a generated spec. Run it and see what falls out. Worst case you lose an afternoon; best case you&#8217;ve found a thread to pull. If the cost of a mistake is high, that&#8217;s when you invest real time hunting for flaws in the spec itself.</p><p><strong>Steal this workflow:</strong> Use cheaper models to generate sequence diagrams of the interleaving procedures, either at design time or from an existing implementation. Review them, simplify, and abstract away steps that don&#8217;t matter. I like Codex for this exploration: it renders the diagrams nicely, and side-chats make it easy to pull on a topic without derailing the main thread. Once the diagrams say what you mean, hand them to a frontier model (Fable at extra high effort, in our case) to generate the TLA+ spec. That first spec will be a black box: you can&#8217;t read TLA+ yet, so all you can judge is what goes in and what comes out. That&#8217;s fine. Start there.</p><p><strong>Refine the workflow:</strong> Now make the black box transparent, one piece at a time. The entry point is the invariants, because they&#8217;re the readable part: short statements about what must always be true. There are usually 1&#8211;3 obvious ones. Use agents to propose more; then cut ruthlessly. After a few iterations the rest of the spec stops being opaque too: you start recognizing the transitions, then questioning them. Is this really how our retry behaves? Does the model even allow two workers here? Now you&#8217;re reading the spec whitebox, finding problems in the model itself instead of just trusting its output.</p><p><strong>Turn doubts into invariants:</strong> Describe what you&#8217;re <em>not confident</em> about, since that&#8217;s exactly the confidence a checker can buy you. For us it was an asymmetry in our GC counters &#8212; overcounting is always safe, undercounting never is &#8212; and the <a href="https://depot.dev/blog/tla-verification#turning-a-doubt-into-a-line-of-tla">previous section</a> explained what the agent did with that doubt. Your doubts are the spec&#8217;s best requirements.</p><p><strong>Treat findings as leads, not verdicts:</strong> The model might not match reality, so a violated invariant is a starting point, a thread to keep pulling. Take the counterexample trace, turn it into a sequence diagram, and zoom in until you either fully understand the race and fix it, or find the mismatch between model and implementation. Either outcome is progress: you&#8217;ve gone from unknown unknown to something you can point at.</p><p>You now have another tool in your toolbox: tests check the interleavings you thought of; TLA+ explores the ones you didn&#8217;t.</p><p></p>]]></content:encoded></item><item><title><![CDATA[Why your team processes suck]]></title><description><![CDATA[A team process without a principle underneath it is a real problem. Protect the "why" as you scale.]]></description><link>https://newsletter.depot.dev/p/why-your-team-processes-suck</link><guid isPermaLink="false">https://newsletter.depot.dev/p/why-your-team-processes-suck</guid><pubDate>Mon, 13 Jul 2026 17:07:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/08bb8533-8de1-467d-92ed-0f8f1c003f76_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I recently took an engineering management role, which means I guess this is who I am now: a person with opinions about meetings. My ancestors would be proud.</p><p>Engineers are funny about this kind of stuff. In technical work, we tend to care deeply about what things do and why they work. But the second <em>process</em> comes up, everyone suddenly gets very comfortable saying, &#8220;Well, this is what we did at my last company.&#8221;</p><p>Or worse: &#8220;This is just how we do things.&#8221;</p><p>These are terrible answers.</p><p>It makes sense why this happens. Computer systems are easy in one specific way: they are mostly deterministic and mostly under our direct control. Human systems are much messier. There is no USB-C port I can plug into a coworker&#8217;s brain and upload <code>stop-being-a-dick.exe</code>.</p><p>Tragic, honestly.</p><p>People systems are stochastic (at best, we have <em>influence</em>, not control). A lot of engineers write them off as the mushy, manager-brained stuff that gets in the way of the &#8220;real&#8221; work. The irony is that ignoring the people system is a great way to make sure the real work doesn&#8217;t get done. Or worse, that the wrong work gets done extremely efficiently.</p><p>Process isn&#8217;t inherently shitty. Teams need it to move context, make decisions, and notice friction. But process becomes theater when we confuse the <strong>format</strong> with the <strong>principle</strong>.</p><h2>The standup test</h2><p>The <em>format</em> of a process is the visible part: a daily meeting, a retro, a Slack post every Friday. The <em>principle</em> is the reason the thing exists in the first place.</p><p>Take standup. The classic format is: &amp;#x2A;what did I do yesterday, what am I doing today, any blockers.* If the process stops there people eventually mentally check out.</p><p><em>&#8220;Yesterday I worked on making the logs page faster.&#8221;</em></p><p>Cool. What does that mean? Is it going well? How is it impacting what others are working on? Is it about to burst into flames?</p><p>The shape of standup survived, but the reason for it did not. I&#8217;ve seen the exact same format produce completely different outcomes depending on the principle underneath it. If a manager&#8217;s principle is <em>control</em>, a standup turns into a daily surveillance ritual to prove everyone is earning their paycheck. The room feels like a tribunal.</p><p>If the principle is <em>connection</em>, it becomes the one moment in a distributed day where everyone is actually together. We pick up on the weird little human signals that don&#8217;t survive the trip into Slack. We notice when someone is circling the same problem for the third day in a row.</p><p>Same format. Different principles. Completely different outcomes.</p><h2>Scaling process and culture</h2><p>When the principle is useful and understood, the format can change without losing utility. When the principle is missing, all that&#8217;s left is a ritual. And rituals are sticky.</p><p>We hit this wall recently with project updates. Our leadership was staying informed through Linear project updates. When there were three projects, that was fine. When it grew to fifteen, it broke. The updates were scattered across different days, with completely different altitudes of context.</p><p>A template could only fix so much. The deeper issue was the principle: <em>visibility at the right altitude</em>.</p><p>Project-level updates were too granular for leadership and too scattered for the ICs to see how the work fit together. So, we killed the ritual and changed the process. We introduced <strong>initiatives</strong>: a collection of related projects tied to a larger business objective, managed by one owner in one channel.</p><p>Upward communication got better because leadership had one place to look at the right altitude. Lateral communication got better because ICs finally had a forcing function to see how their work connected to the broader system.</p><p>The old process wasn&#8217;t stupid. It worked until it didn&#8217;t.</p><h2>Keep the why alive</h2><p>I don&#8217;t want process for the sake of process. I want exactly enough of it to make the team work. Not so much that everyone spends their lives feeding status machines, and not so little that context disappears into DMs and engineers become isolated code islands.</p><p>That balance changes as a company scales.</p><p>If I don&#8217;t know why a process exists, I don&#8217;t know what to keep, what to change, or what to kill. I am just copying shapes. Process doesn&#8217;t suck. Process without principles really does.</p><h2></h2>]]></content:encoded></item><item><title><![CDATA[Staying in control of your codebase in the AI era]]></title><description><![CDATA[AI writes code faster than you can review it, but you're on call when it breaks. A Go workflow to stay in control.]]></description><link>https://newsletter.depot.dev/p/staying-in-control-of-your-codebase</link><guid isPermaLink="false">https://newsletter.depot.dev/p/staying-in-control-of-your-codebase</guid><dc:creator><![CDATA[Peter Hasko-Nagy]]></dc:creator><pubDate>Tue, 30 Jun 2026 16:52:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/99831ab1-da63-4ea7-86fc-b5749cf82145_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I love coding, and I love my job. Like many others in the industry, I found it hard to cope with the changes of the last couple of years. At first, I had an existential crisis. Then I started experimenting and watching how people actually work with AI, which greatly reshaped my view. (I really recommend this <a href="https://www.youtube.com/watch?v=vmKvw73V394">video from Jon Gjengset</a>, where he shows how he works with AI, what it&#8217;s good for, and how it&#8217;s ultimately just another tool in your toolbox.)</p><p>Embracing AI and steering it to produce maintainable code at an acceptable cost is the name of the game. There are countless smart people out there working on different harnesses and techniques to crack the problem. The current craze is loops, but aside from being too expensive, the approach just doesn&#8217;t click with me.</p><p>There&#8217;s a difference in effective AI use between working alone and being part of a team on the same project. In the latter case, I have obligations to my employer and my team. Some employment contracts even contain a clause like this: &#8220;the Employee shall perform their duties to the best of their ability and skill.&#8221; I tend to take that line rather seriously. Ridiculous as it may sound, this is my creed. I take great pride in my work, and I plan to keep it that way. I invest heavily in keeping my code maintainable and reviewable, to make life easy for my peers and my future self. Let me show you the AI assisted development workflow I&#8217;m currently happy with.</p><h2>What works for me</h2><p>The approach that works today is constraint. Selectively adding guardrails to get more consistent, predictable output. I work with Go the most in my day-to-day, so my examples will use it, but I think these techniques apply to almost any other language. That said, Go has the advantage that different developers tend to produce fairly similar code, thanks to the language&#8217;s rigid structure.</p><h3>Linter</h3><p>I find that linting is the alpha and omega of steering agents to produce acceptable code. As expected, I&#8217;m using the standard <code>golangci-lint</code> as my linter. These are the rules I mostly rely on to influence the shape the code takes:</p><pre><code><code>  settings:
    cyclop:
      max-complexity: 10
    funlen:
      lines: 90
      statements: 50
    revive:
      rules:
        - name: argument-limit
          arguments:
            - 8
        - name: file-length-limit
          severity: error
          arguments:
            - max: 1000
              skipComments: true
              skipBlankLines: true
</code></code></pre><p>Your peers will thank you later for the effort you put into constraining the code&#8217;s dimensions. Limiting the lines of code in a single file and capping cyclomatic complexity forces the agent to break code down into smaller logical units across multiple files. Function length and the number of allowed function arguments complement these rules, making the code far more readable to humans.</p><h3>Testing</h3><p>Agents are useful for generating (unit) tests to save you from the grunt work. Proper test coverage has always been important, but it matters more than ever in the era of full project rewrites. Take the agent by the hand and don&#8217;t let go until it can produce sane test cases.</p><p>I usually start by writing a few very good test cases manually. I almost always write table-driven tests, so that&#8217;s a good starting point. Agents can easily generate mocks, but I usually give them a framework to do so. Invest early in coming up with sane interfaces, and force the agent to use those to generate mock structs with deterministic code generators such as moq. Agents like to solve problems in creative ways. Without guardrails, they usually produce working but hard-to-read assertions. GPT 5.4+ likes to write tests that basically &#8220;grep&#8221; a line of business logic, checking character by character for the existence of a function and calling it a unit test. Weird.</p><p>I usually instruct the agent to use the testify package to keep test cases standardized and easy to read. Once the preparations are done, you can just ask the agent to look at your reference test implementation and write future tests based on it.</p><p>Take a look at this simple go project:</p><pre><code><code>package main

import "errors"

type calculator struct{}

func (c calculator) add(a int, b int) (int, error) {
&#9;if a == 0 || b == 0 {
&#9;&#9;return 0, errors.New("values cannot be 0")
&#9;}
&#9;return a + b, nil
}

func (c calculator) subtract(a int, b int) (int, error) {
&#9;return a - b, nil
}

func addWrapper(a int, b int, c calculator) (int, error) {
&#9;return c.add(a, b)
}

func subtractWrapper(a int, b int, c calculator) (int, error) {
&#9;return c.subtract(a, b)
}

func main() {
&#9;addWrapper(10, 10, calculator{})
}</code></code></pre><p>I prompted <code>Claude 4.8 (1M context) High-thinking</code> with the following text:</p><pre><code><code>Please take a look at blogpost/main.go. I would like you to write unit tests for addWrapper function. The test should have at least 3 different inputs. Write idiomatic go.</code></code></pre><p>The agent actually produced table-driver tests without being asked, and the tests themselves are fine, but we could make them more readable. I&#8217;m not happy with how the table-driven tests look, and those assertions could be much easier on the eyes. I&#8217;m also not happy with the direct use of <code>calculator</code>, which could change at any time, for example by introducing dependencies.</p><p>See the agent&#8217;s first attempt</p><pre><code><code>  package main

  import (
  &#9;"testing"
  )

  func TestAddWrapper(t *testing.T) {
  &#9;tests := []struct {
  &#9;&#9;name    string
  &#9;&#9;a       int
  &#9;&#9;b       int
  &#9;&#9;want    int
  &#9;&#9;wantErr bool
  &#9;}{
  &#9;&#9;{
  &#9;&#9;&#9;name: "two positive values",
  &#9;&#9;&#9;a:    10,
  &#9;&#9;&#9;b:    10,
  &#9;&#9;&#9;want: 20,
  &#9;&#9;},
  &#9;&#9;{
  &#9;&#9;&#9;name: "positive and negative values",
  &#9;&#9;&#9;a:    5,
  &#9;&#9;&#9;b:    -3,
  &#9;&#9;&#9;want: 2,
  &#9;&#9;},
  &#9;&#9;{
  &#9;&#9;&#9;name:    "zero operand returns error",
  &#9;&#9;&#9;a:       0,
  &#9;&#9;&#9;b:       7,
  &#9;&#9;&#9;wantErr: true,
  &#9;&#9;},
  &#9;}

  &#9;for _, tt := range tests {
  &#9;&#9;t.Run(tt.name, func(t *testing.T) {
  &#9;&#9;&#9;got, err := addWrapper(tt.a, tt.b, calculator{})

  &#9;&#9;&#9;if tt.wantErr &amp;&amp; err == nil {
  &#9;&#9;&#9;&#9;t.Fatalf("addWrapper(%d, %d) error = %v, want %v", tt.a, tt.b, err, tt.wantErr)
  &#9;&#9;&#9;}
  &#9;&#9;&#9;if got != tt.want {
  &#9;&#9;&#9;&#9;t.Errorf("addWrapper(%d, %d) = %d, want %d", tt.a, tt.b, got, tt.want)
  &#9;&#9;&#9;}
  &#9;&#9;})
  &#9;}
  }</code></code></pre><p>Lets make some changes to the main file, and create a reference unit test implementation:</p><pre><code><code>package main

import "errors"

var errZeroValues = errors.New("values cannot be 0")

//go:generate moq -stub -out calculator_mocks.go . Calculator
type Calculator interface {
&#9;add(a int, b int) (int, error)
&#9;subtract(a int, b int) (int, error)
}

type calculator struct{}

func (c calculator) add(a int, b int) (int, error) {
&#9;if a == 0 || b == 0 {
&#9;&#9;return 0, errZeroValues
&#9;}
&#9;return a + b, nil
}

func (c calculator) subtract(a int, b int) (int, error) {
&#9;return a - b, nil
}

func addWrapper(a int, b int, c Calculator) (int, error) {
&#9;return c.add(a, b)
}

func subtractWrapper(a int, b int, c Calculator) (int, error) {
&#9;return c.subtract(a, b)
}
</code></code></pre><p>I introduced a sentinel error called <code>errZeroValues</code>. I also added a Calculator interface, along with a generator that writes a mock to <code>calculator_mocks.go</code>. Now let&#8217;s write our reference test implementation:</p><pre><code><code>package main

import (
&#9;"testing"

&#9;"github.com/stretchr/testify/require"
)

var mockCalculator = &amp;CalculatorMock{
&#9;subtractFunc: func(a, b int) (int, error) {
&#9;&#9;return a - b, nil
&#9;},
}

func TestSubtractWrapper(t *testing.T) {
&#9;tcs := map[string]struct {
&#9;&#9;a       int
&#9;&#9;b       int
&#9;&#9;want    int
&#9;&#9;wantErr error
&#9;}{
&#9;&#9;"positive and positive values": {
&#9;&#9;&#9;a:    10,
&#9;&#9;&#9;b:    10,
&#9;&#9;&#9;want: 0,
&#9;&#9;},
&#9;&#9;"positive and negative values": {
&#9;&#9;&#9;a:    10,
&#9;&#9;&#9;b:    -10,
&#9;&#9;&#9;want: 20,
&#9;&#9;},
&#9;&#9;"negative and negative values": {
&#9;&#9;&#9;a:    -10,
&#9;&#9;&#9;b:    -10,
&#9;&#9;&#9;want: 0,
&#9;&#9;},
&#9;&#9;"zero operands": {
&#9;&#9;&#9;a:    0,
&#9;&#9;&#9;b:    0,
&#9;&#9;&#9;want: 0,
&#9;&#9;},
&#9;}

&#9;for name, tc := range tcs {
&#9;&#9;t.Run(name, func(t *testing.T) {
&#9;&#9;&#9;got, err := subtractWrapper(tc.a, tc.b, mockCalculator)
&#9;&#9;&#9;require.NoError(t, err)
&#9;&#9;&#9;require.Equal(t, tc.want, got)
&#9;&#9;})
&#9;}
}</code></code></pre><p>Lets see what the next prompt produces:</p><pre><code><code>Please re-read blogpost/main.go, and the reference test implementation in blogpost/main_test.go. I would like you to follow the practices seen in the TestSubtractWrapper reference unit test, and write TestAddtWrapper.</code></code></pre><p>The agent extended the mock and used require, just as asked. (Please ignore the fact that the mock and the real implementation are the same; this is just a small example.) This shows that agents are capable of producing sane unit tests, given some investment in crafting good examples.</p><p>See the agent&#8217;s revised attempt</p><pre><code><code>  package main

  import (
  &#9;"testing"

  &#9;"github.com/stretchr/testify/require"
  )

  var mockCalculator = &amp;CalculatorMock{
  &#9;addFunc: func(a, b int) (int, error) {
  &#9;&#9;if a == 0 || b == 0 {
  &#9;&#9;&#9;return 0, errZeroValues
  &#9;&#9;}
  &#9;&#9;return a + b, nil
  &#9;},
  &#9;subtractFunc: func(a, b int) (int, error) {
  &#9;&#9;return a - b, nil
  &#9;},
  }

  func TestSubtractWrapper(t *testing.T) {
  &#9;tcs := map[string]struct {
  &#9;&#9;a       int
  &#9;&#9;b       int
  &#9;&#9;want    int
  &#9;&#9;wantErr error
  &#9;}{
  &#9;&#9;"positive and positive values": {
  &#9;&#9;&#9;a:    10,
  &#9;&#9;&#9;b:    10,
  &#9;&#9;&#9;want: 0,
  &#9;&#9;},
  &#9;&#9;"positive and negative values": {
  &#9;&#9;&#9;a:    10,
  &#9;&#9;&#9;b:    -10,
  &#9;&#9;&#9;want: 20,
  &#9;&#9;},
  &#9;&#9;"negative and negative values": {
  &#9;&#9;&#9;a:    -10,
  &#9;&#9;&#9;b:    -10,
  &#9;&#9;&#9;want: 0,
  &#9;&#9;},
  &#9;&#9;"zero operands": {
  &#9;&#9;&#9;a:    0,
  &#9;&#9;&#9;b:    0,
  &#9;&#9;&#9;want: 0,
  &#9;&#9;},
  &#9;}

  &#9;for name, tc := range tcs {
  &#9;&#9;t.Run(name, func(t *testing.T) {
  &#9;&#9;&#9;got, err := subtractWrapper(tc.a, tc.b, mockCalculator)
  &#9;&#9;&#9;require.NoError(t, err)
  &#9;&#9;&#9;require.Equal(t, tc.want, got)
  &#9;&#9;})
  &#9;}
  }

  func TestAddWrapper(t *testing.T) {
  &#9;tcs := map[string]struct {
  &#9;&#9;a       int
  &#9;&#9;b       int
  &#9;&#9;want    int
  &#9;&#9;wantErr error
  &#9;}{
  &#9;&#9;"positive and positive values": {
  &#9;&#9;&#9;a:    10,
  &#9;&#9;&#9;b:    10,
  &#9;&#9;&#9;want: 20,
  &#9;&#9;},
  &#9;&#9;"positive and negative values": {
  &#9;&#9;&#9;a:    10,
  &#9;&#9;&#9;b:    -10,
  &#9;&#9;&#9;want: 0,
  &#9;&#9;},
  &#9;&#9;"negative and negative values": {
  &#9;&#9;&#9;a:    -10,
  &#9;&#9;&#9;b:    -10,
  &#9;&#9;&#9;want: -20,
  &#9;&#9;},
  &#9;&#9;"zero operands": {
  &#9;&#9;&#9;a:       0,
  &#9;&#9;&#9;b:       0,
  &#9;&#9;&#9;want:    0,
  &#9;&#9;&#9;wantErr: errZeroValues,
  &#9;&#9;},
  &#9;}

  &#9;for name, tc := range tcs {
  &#9;&#9;t.Run(name, func(t *testing.T) {
  &#9;&#9;&#9;got, err := addWrapper(tc.a, tc.b, mockCalculator)
  &#9;&#9;&#9;if tc.wantErr != nil {
  &#9;&#9;&#9;&#9;require.ErrorIs(t, err, tc.wantErr)
  &#9;&#9;&#9;&#9;return
  &#9;&#9;&#9;}
  &#9;&#9;&#9;require.NoError(t, err)
  &#9;&#9;&#9;require.Equal(t, tc.want, got)
  &#9;&#9;})
  &#9;}
  }</code></code></pre><h3>Agents.md and Claude.md</h3><p>I often think my Markdown files might look naive to agentic power users, but I like to keep things simple, and it seems to work for me. This is the basic structure I usually start with:</p><pre><code><code># AGENTS

&lt;Short description what the project is about&gt;

# Main components
&lt;A list of the main components and their path within the project with a very short description&gt;

# Most important go packages
&lt;A list of important go packages and what they should be used for&gt;

# Development

`make build` cross compiles to linux
`make generate` generates the gRPC files
`make test` executes the unit-tests
`make lint` executes the linter check (needs docker)
`make lint-proto` lints the proto files

## Coding style
- The code should be idiomatic go
- You should listen to the linter all the time
- Code should be self-explanatory. prefer code readability over comments
- Comments should be short, to the point
- Comments shouldn't include ticket numbers
- Comments shouldn't include made up worlds and phrases not in the english dictionary
- New files shouldn't extend over 1000 lines of code (excluding comments and whitespaces)
- Think twice before adding code to files where the number of lines in a file already surpass 700
- Unit tests are nice, but don't write tests just for the sake of writing tests
- When adding new unit-tests, keep the style from the existing tests
- PRs should be small, encapsulating a single feature or fix.
- If you need to stuff multiple features or related changes into a PR, oranize them into separate commits

# Deployment
&lt;How to deploy to STAGING&gt;. # deploying to production should follow process</code></code></pre><p>That&#8217;s all. It overlaps somewhat with what we covered earlier on linter rules and unit tests, but I find that the more we surround the agent with walls, the more deterministic and higher-quality its output becomes.</p><p>The remarks about comments are probably the interesting part. I&#8217;ve found that Opus 4.7+ and GPT 5.4+ models produce complete garbage in their comments when not regulated. Let&#8217;s take a look at the following comment:</p><pre><code><code>exitCode, exitErr := command.GetExitState()
if exitErr != nil {
&#9;// Synthetic "exit status N" goes on the legacy line rail only; it
&#9;// would corrupt callers that concatenate StderrRaw to reconstruct
&#9;// the process stderr (DEP-5505).
&#9;if err := stream.Send(&amp;somepackage.ExecuteResponse{
&#9;&#9;ExitCode: exitCode,
&#9;&#9;Stderr:   exitErr.Error(),
&#9;}); err != nil {
&#9;&#9;return fmt.Errorf("send exit error to stream: %w", err)
&#9;}
}</code></code></pre><ol><li><p>It includes a ticket number for no good reason.</p></li><li><p>It contains a made-up expression like legacy line rail.</p></li><li><p>It refers to code it just changed as &#8220;legacy.&#8221;</p></li><li><p>It takes real mental effort just to understand what the comment is even about. What does &#8220;synthetic&#8221; even mean in the context of exit codes?</p></li></ol><p>Furthermore, I&#8217;ve observed that after slightly changing a function, GPT 5.4+ likes to write something like: &#8220;Previously myfunction was working like this &#8230; (and then it details how the function worked before the change), but since TICKET-NUMBER, the function now does this &#8230; .&#8221; I&#8217;m not sure why the models write comments like this, but I&#8217;m pretty sure this is what Git is for. To add to that, in Rust code GPT 5.4+ will happily write comments 50 to 100 lines long. It would be insane to burden my peers and my future self with that much noise, so I like to guide the agent on how to write sane comments, and I think you probably should too. As a bonus, you might want to set up Cursor Bugbot or another reviewer agent to check comments against a rubric on open PRs.</p><h2>Processes</h2><p>So far, we&#8217;ve worked to produce sane agentic code that resembles something the team would expect from other engineers. To pass the ceremony of validating and merging code into the codebase, we need to take a few extra steps.</p><h3>PR descriptions</h3><p>When I&#8217;m not working across a lot of parallel lanes, I still like to write PR descriptions. I aim to keep them short and to the point, and I occasionally include screenshots of the product changes, which help peers understand them. When I have the agent write them, I hold it to the same rules I described earlier for comments.</p><h3>Making PRs reviewable</h3><p>As a rule of thumb, I try to keep changes small, say under 200 changed lines overall. Of course, sometimes that isn&#8217;t possible: there are large refactors and new features. When the changeset is sufficiently large, I try to decompose the changes into separately reviewable commits. You can split the changes into commits by logical boundaries, modules, packages, and so on. Whatever works best for your team. Agents are insanely good at this; you can just tell them to break your change into small commits right before opening a PR. Another promising approach is GitHub <a href="https://github.github.com/gh-stack/">stacked PRs</a>. The feature is in preview, but it makes reviewing code so much easier.</p><h2>The missing cherries on top</h2><p>I&#8217;m quite happy with my current workflow, since these simple tricks can greatly improve the quality of the code produced, but there&#8217;s always room for improvement. There are a few things on my wishlist that would make life easier:</p><h3>Organization level Agents.md</h3><p>It would be so nice if GitHub (and other forges) supported defining organization-level AGENTS.md directives that would merge with repo-level Markdown files. This way, I could define global defaults (<code>comment styling</code>), and the repo-specific directives (<code>deploy</code>, <code>build</code>, etc.) would stay in the repositories.</p><h3>Closing the feedback loop</h3><p>Since Opus 4.7 and GPT 5.4, agents have started using some REALLY weird tools. For example, I often see them run Perl scripts. Other times, instead of editing a file, they make a copy, then move the new file in to replace the original. Sometimes they offer to run a shell script so exotic that I&#8217;d rather ask them to do something else. An obvious solution is to run the agents in (I&#8217;m not super happy with the local isolation levels agents provide). Running them in a remote sandbox solves the isolation problem, but takes away the instant feedback I get locally. I&#8217;d like to run my integration suite after each remote modification to my codebase, so it mirrors my local workflow. It&#8217;s absolutely doable, but involved, and as far as I know no provider supports it natively.</p><h2>Haven&#8217;t you heard that coding is solved?</h2><p>You might be thinking, &#8220;But Peter, so many words wasted on code quality, when coding is already solved. Nobody needs to read code anymore!&#8221;</p><p>I respectfully beg to differ. LLMs make a lot of mistakes even today, and the quality of the code they produce is far from deterministic or uniform. Furthermore, I&#8217;m responsible for the work I release, or that goes out under my supervision. After all, I&#8217;m the one on call, not Claude. There are many cases where I don&#8217;t care about code quality at all, such as quick experiments, small scripts, visualizations, and so on. But until the day LLMs stop making mistakes (that day might come soon), and you can&#8217;t tell human code from agent code, expect me to show up and read all the code. To the best of my ability and skill, as the contract says.</p>]]></content:encoded></item><item><title><![CDATA[Paranoia is baseline now: Security in the AI era]]></title><description><![CDATA[AI made finding obscure Linux bugs fast and cheap. Kernel hardening sorted by what it costs you.]]></description><link>https://newsletter.depot.dev/p/paranoia-is-baseline-now</link><guid isPermaLink="false">https://newsletter.depot.dev/p/paranoia-is-baseline-now</guid><dc:creator><![CDATA[Héja Péter (Vau)]]></dc:creator><pubDate>Thu, 11 Jun 2026 22:56:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/3aa9b5e5-9d9f-4280-a4a9-5b39661f395e_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2>Paranoia used to be optional</h2><p>There&#8217;s a certain attitude people have toward the extremes of hardening. Disable Hyper-Threading? Sure, if performance doesn&#8217;t matter! Lock kernel module loading after boot? Cool, until it breaks. I&#8217;ve had that attitude myself, and for the longest time, it was the right one.</p><p>The kernel had earned a lot of trust. Bugs got reported, embargoes held (mostly...), and the really nasty bugs usually got patched before exploitation in the wild. So I never even really thought about obscure kernel features - if it has a CVE against it <em>right now</em>, it&#8217;s usually already fixed. Many hardening steps felt like a mix between an insurance policy that won&#8217;t ever pay out and security theater with performative hardening as the main star.</p><p>Then, this spring, something genuinely changed. AI increased productivity for all - even attackers, who could look for vulnerabilities faster and in more places than ever before. And once such a bombardment of vulnerabilities starts, you stop asking &#8220;is this feature vulnerable today&#8221; and start asking &#8220;isn&#8217;t it reckless to let the kernel auto-load whatever&#8221;. Unused kernel surface is no longer just debt.</p><h2>The wave</h2><p>Five Linux local privilege escalation (LPE) vulnerabilities got disclosed and immediately abused in a couple of weeks. Copy Fail started the party. It was the result of an hour-long AI scan of the kernel&#8217;s <code>crypto/</code> subsystem. The published exploit for it was a short Python script that worked on almost everybody&#8217;s favorite Linux distributions. The vulnerability had been sitting in <code>algif_aead</code>, introduced by an optimization that everybody had in their kernel, and nobody really needed.</p><p>Here&#8217;s the run:</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!bDn8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!bDn8!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 424w, https://substackcdn.com/image/fetch/$s_!bDn8!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 848w, https://substackcdn.com/image/fetch/$s_!bDn8!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 1272w, https://substackcdn.com/image/fetch/$s_!bDn8!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!bDn8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png" width="728" height="361" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:722,&quot;width&quot;:1456,&quot;resizeWidth&quot;:728,&quot;bytes&quot;:232656,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:&quot;https://workingtheoryeng.substack.com/i/210825200?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!bDn8!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 424w, https://substackcdn.com/image/fetch/$s_!bDn8!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 848w, https://substackcdn.com/image/fetch/$s_!bDn8!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 1272w, https://substackcdn.com/image/fetch/$s_!bDn8!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F74ed88e7-0968-49dc-b805-278ff84e0245_2631x1304.png 1456w" sizes="100vw" fetchpriority="high"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>CVE references from the table above:</p><ul><li><p><strong>Apr 29 &#8212; Copy Fail:</strong> <a href="https://www.cve.org/CVERecord?id=CVE-2026-31431">CVE-2026-31431</a> &#8212; page-cache write &#8594; root (AF_ALG + <code>splice()</code>, <code>algif_aead</code> module)</p></li><li><p><strong>May 7 &#8212; Dirty Frag:</strong> <a href="https://www.cve.org/CVERecord?id=CVE-2026-43284">CVE-2026-43284</a> + <a href="https://www.cve.org/CVERecord?id=CVE-2026-43500">CVE-2026-43500</a> &#8212; page-cache write &#8594; root (IPsec ESP, <code>esp4</code>/<code>esp6</code>, <code>rxrpc</code> modules)</p></li><li><p><strong>May 13 &#8212; Fragnesia:</strong> <a href="https://www.cve.org/CVERecord?id=CVE-2026-46300">CVE-2026-46300</a> &#8212; page-cache write &#8594; root (XFRM ESP-in-TCP coalescing, <code>esp4</code>/<code>esp6</code>, <code>rxrpc</code> modules)</p></li><li><p><strong>May 20 &#8212; ssh-keysign-pwn:</strong> <a href="https://www.cve.org/CVERecord?id=CVE-2026-46333">CVE-2026-46333</a> &#8212; FD theft &#8594; secrets/root (<code>ptrace</code> exit race + <code>pidfd_getfd</code>)</p></li><li><p><strong>May 21 &#8212; PinTheft:</strong> <a href="https://www.cve.org/CVERecord?id=CVE-2026-43494">CVE-2026-43494</a> &#8212; page-cache write &#8594; root (<code>io_uring</code> + RDS zerocopy double-free, <code>rds_tcp</code>, <code>rds</code> modules)</p></li></ul><p>Four of the five are almost the same bug: some optimized crypto or zerocopy path writes into a buffer the kernel doesn&#8217;t actually own, and you end up with a controlled write into the page cache of a Set User ID (SUID) binary you&#8217;re allowed to read. They don&#8217;t actually use the network, and none of them are extremely privileged. (I&#8217;m not a kernel hacker, so I rely on the write-ups, reading the patches and a bit of guesswork, but the pattern is real.)</p><p>These kinds of things have happened before. We had embargoes violated or not-quite-responsible disclosures regarding vulnerabilities. We always had some people looking at recent Linux patches, trying to reverse engineer vulnerabilities from patches - we also had hasty fixes introducing new bugs. But we never had this scale or speed.</p><p>Patches are still a must - but that&#8217;s reactive. The patching is now a race against something that can read 100x more code than you, even the boring bits. But we can give them less attack surface and a harder time.</p><h2>Paranoia keeps becoming the default</h2><p>Just to preface this: nothing listed here is really new. These hardenings started their lifetime as paranoid measures, then moved into normalcy, best practice or even defaults.</p><p>For example, OpenBSD has always been quick and an early adopter of these ideas and has often been called paranoid due to its choices. When OpenBSD devs separated the privileges in OpenSSH, to separate the unprivileged process talking on the network (risky), it felt like over-engineering, but today it&#8217;s common. Their W^X (memory is writeable or executable, not both) and a stack-smashing protector (ProPolice) were adopted in OpenBSD system-wide in 2003, but it took Linuxes one more year for NX and multiple years to turn stack protector on (Ubuntu and RHEL ~2006, Debian ~2013).</p><p>I have to also tip my hat to PaX and grsecurity. They have maintained a huge set of out-of-tree patches to harden the Linux kernel, with innovative and early adoption of hardening ideas. Just listing what they had patches for early on would often sound like an advertisement for their paid products.</p><p>Sadly, these paranoid hardenings often came with a cost (performance, ease of use, etc.) up front, but very abstract protection against a bug (or bug class) that might not exist yet, so they were usually adopted after the fact.</p><h2>Sorting hardening by cost</h2><p>Hardening is often a balancing game, and it&#8217;s not just one thing or one idea. Each choice can have wildly different trade-offs, and they all depend on what you are trying to achieve. Some of them are trivial choices, some of them may make a service 30% more expensive to run. To make choices easier, I feel it&#8217;s easier to think about categories:</p><ul><li><p><strong>A - Free.</strong> Simple housekeeping, won&#8217;t harm anything.</p></li><li><p><strong>B - Nearly free.</strong> Turning off genuinely useful things, but not things we use.</p></li><li><p><strong>C - Costs you visibility.</strong> Makes operations harder (debug, monitor), but closes doors.</p></li><li><p><strong>D - Costs you some performance.</strong> Can have a negative effect on performance, but very tiny. Might hurt on high node count.</p></li><li><p><strong>E - Costs you real money.</strong> You lose throughput, you have to run more nodes or more expensive hardware. Danger zone.</p></li></ul><p>This is neither a definitive guide for hardening, nor a todo list! The scope here is mostly about the Linux kernel, but these ideas can be applied to other pieces of software and infrastructure. If you don&#8217;t &#8220;check all the boxes&#8221; and apply all hardening, you are not necessarily reckless. If you are conscious about your trade-offs, then you will be able to decide what is worth it and what is not. Also, do note that in this list, only the cost of running servers and services is considered - things like spending engineering hours for security, maintaining customizations, and making decisions, are not part of it.</p><p>It&#8217;s important to note that I&#8217;m writing about generic advice and I&#8217;m not defining a concrete threat model here - you will need to consider your own requirements and threat model to weigh any advice in this post. I also want to highlight that the goal of the hardening measures and ideas I&#8217;m talking about is to minimize the chance an attack is successful or useful for information extraction, denial of service or similar malicious goals. Exploitation often involves multiple vulnerabilities, where attackers might have to orient themselves and might have to pivot from lesser to privileged access.</p><h3>A - Free: the housekeeping</h3><p>The Linux kernel&#8217;s support for protocols, hardware and others is extensive, but this also means a bigger attack surface. While maintaining your own custom kernel build per use-case might be good, the kernel distributed with common distributions can also be locked down. For example, you may blocklist rarely used kernel modules, which <a href="https://documentation.ubuntu.com/security/security-features/kernel-protections/#denylist-rare-protocols">Ubuntu already does</a> for you. This meant that PinTheft (<a href="https://www.cve.org/CVERecord?id=CVE-2026-43494">CVE-2026-43494</a>) didn&#8217;t affect stock Ubuntu installations. The good thing about this is that you may also blocklist and unload kernel modules on running machines, like the workarounds for these vulnerabilities, which reduces the chance of being affected by faulty hotfixes, such as Fragnesia (<a href="https://www.cve.org/CVERecord?id=CVE-2026-46300">CVE-2026-46300</a>).</p><p>Some hardening can also just make attackers&#8217; lives harder, like reducing information leaks on the system. For example, don&#8217;t let any user read <code>dmesg</code> (sysctl: <code>kernel.dmesg_restrict=1</code>), which can contain sensitive information that regular users really shouldn&#8217;t read. Another example is to not let <code>setuid</code> / <code>setgid</code> programs coredump (sysctl: <code>fs.suid_dumpable=0</code>), which will hold sensitive information in case of, say, <code>sudo</code>. Two more concrete hardenings catching real bugs are making NULL pointer dereference exploitation much harder by making low memory unmappable (sysctl: <code>vm.mmap_min_addr=65536</code>), and enforcing stricter symlink/hardlink checks (sysctl: <code>fs.protected_symlinks=1</code>, <code>fs.protected_hardlinks=1</code>), so TOCTOU (time of check, time of use) symlink attacks are much harder.</p><p>Another one to consider is to reduce <code>setuid</code> binaries to a minimum, and make them unreadable by anybody outside of root - which could block attacks relying on poisoning the page cache of these <code>setuid</code> files. Be careful with persistence here, though.</p><p>These are pretty easy to implement, free in terms of performance, not too intrusive and won&#8217;t reduce observability for operators. Many of these hardenings became baseline for some popular Linux distributions.</p><h3>B - Nearly free: purge unused modules</h3><p>The basic hygiene step could be to enforce signature checking on kernel modules (set <code>/sys/module/module/parameters/sig_enforce</code> to 1), to block loading unsigned modules. While <code>lockdown=integrity</code> (<code>/sys/kernel/security/lockdown</code> set to <code>integrity</code>) also allows this, that value also blocks other functionality, such as <code>kexec</code> and some debugging opportunities - more on that later. This might be enabled based on your distribution (Ubuntu enabled it by default in 20.04 LTS) and your UEFI secure boot setting.</p><p>A standard Debian or Ubuntu image for example ships multiple thousands of kernel modules. What used to be paranoid but I think is a good practice now is to prune your kernel modules and block kernel module loading after boot (via <code>/proc/sys/kernel/modules_disabled</code>), which also prevents an attacker from loading or unloading any kernel modules. (While kernel module signing exists, vulnerable kernel modules are also signed.) Doing this on fixed-purpose nodes can be pretty easy (like hypervisors, simple nginx web servers), but it can be really painful for generic hosts (like kubernetes nodes) or generic images.</p><h3>C - Costs visibility: ptrace, eBPF</h3><p>I think hardening measures impacting observability are pretty divisive: observability helps you, but it can also help attackers. I remember calling someone paranoid for disabling eBPF altogether, but I do now agree that limiting it at least is a minimum (sysctl: <code>kernel.unprivileged_bpf_disabled=1</code> - 1 is disabled until reboot, 2 is disabled, but reversible), so only <code>root</code> can load BPF programs.</p><p>There is also Yama, which can help harden certain parts of the kernel. For example, you can enforce that only <code>CAP_SYS_PTRACE</code> privileged processes can do ptrace (sysctl: <code>kernel.yama.ptrace_scope=2</code>), but it&#8217;s also sensible to turn this off completely (sysctl: <code>kernel.yama.ptrace_scope=3</code>) until a reboot. The ssh-keysign-pwn (<a href="https://www.cve.org/CVERecord?id=CVE-2026-46333">CVE-2026-46333</a>) vulnerability could&#8217;ve been prevented with this. However, completely disabling ptrace means no <code>gdb -p</code> or <code>strace -p</code> - so don&#8217;t do this on your dev machine, but it&#8217;s fine on a server.</p><p>You can also reduce visibility into the kernel by hiding kernel pointers from userspace (sysctl: <code>kernel.kptr_restrict=2</code>), which helps prevent information leaks.</p><h3>D - Costs performance: pinning, libc</h3><p>In this category, there are a couple of hardening options, which will likely reduce performance, but can be worth it. One example is if you run workloads for multiple tenants, and want to provide better isolation on your hypervisor node. In this case, it will be necessary to do CPU core pinning at least, so no two tenants are on the same CPU core at the same time, so fewer resources are shared - more on this in the last point. Depending on your business, you might even have to separate tenants between NUMA nodes, and might have to consult your CPU documentation.</p><p>One thing that is pretty divisive is using glibc or musl libc. On one hand, musl libc&#8217;s code is incredibly easy to understand, and small enough to reason about its correctness, giving a &#8220;safer feeling&#8221; codebase. On the other hand, glibc has had more eyeballs on it for years, and the performance is also on its side.</p><p>The math here is pretty interesting: 3% performance may not seem like much, but these loses stack up, and a couple of percentages can silently bleed a lot of money.</p><h3>E - Costs real money: turn off SMT</h3><p>Let&#8217;s get back to OpenBSD and paranoia a bit. 20 years ago, CPUs were considered reliable, then came the hardware bugs (Spectre and its siblings), which were fixable by microcode updates to some degree, but might require compiler patches, kernel support, and it can cause performance loss. The choice is clear: be vulnerable, or be slower by some percentages.</p><p>However, there is also the case of SMT (Simultaneous multithreading, or Hyper-Threading in the case of Intel), which essentially lets 2 threads independently execute on a single CPU core, albeit slower. The upside is that instead of 1 physical CPU core&#8217;s performance you get 2 times roughly 65% performance, resulting in up to 30% overall performance increase, depending on workload. The problem is that this means that 2 threads are executing in the same CPU, and they share a physical, classically indivisible unit of computing. Good performance, potential problems.</p><p>As a reaction to CPU hardware bugs, OpenBSD disabled SMT altogether in June 2018 (sysctl <code>hw.smt</code> default off), calling everyone else to follow suit. Then, still in 2018 and later in 2019, multiple SMT hardware bugs were disclosed, such as L1TF (e.g. <a href="https://www.cve.org/CVERecord?id=CVE-2018-3615">CVE-2018-3615</a>, <a href="https://www.cve.org/CVERecord?id=CVE-2018-3620">CVE-2018-3620</a>, <a href="https://www.cve.org/CVERecord?id=CVE-2018-3646">CVE-2018-3646</a>), MDS (e.g. <a href="https://www.cve.org/CVERecord?id=CVE-2018-12126">CVE-2018-12126</a>, <a href="https://www.cve.org/CVERecord?id=CVE-2018-12127">CVE-2018-12127</a>, <a href="https://www.cve.org/CVERecord?id=CVE-2018-12130">CVE-2018-12130</a>, <a href="https://www.cve.org/CVERecord?id=CVE-2019-11091">CVE-2019-11091</a>) and TAA (<a href="https://www.cve.org/CVERecord?id=CVE-2019-11135">CVE-2019-11135</a>), where these vulnerabilities were either caused by SMT or were amplified by SMT.</p><p>So, OpenBSD was right. But Linux wasn&#8217;t wrong either, because 30% of performance loss is extreme - leaving that on the table means you have to pay for much more resources, and the business is hurt. So, while OpenBSD disabled it by default, Linux decided to keep it enabled by default, because in a plethora of server use-cases the potential vulnerabilities don&#8217;t matter (big data clusters, storages, etc.) or can be mitigated (core isolation between tenants, hardened databases, etc.), and if you keep your personal machine up-to-date, keeping up to 30% of performance gain is not that reckless.</p><p>There is also io_uring, which for example Google decided to get rid of on production servers and ChromeOS and blocked for Android, because in the kCTF program (where Google pays for kernel vulnerability reports) <a href="https://security.googleblog.com/2023/06/learnings-from-kctf-vrps-42-linux.html#:~:text=in%2E-,Learnings%20and%20Statistics">60% of their successful vulnerability submissions were io_uring</a>, meaning it has a poor track record as a Linux feature, amassing more vulnerability reports in this program that year, than the rest of the Linux kernel. So, it is straightforward to disable it (sysctl: <code>kernel.io_uring_disabled=2</code>). However, io_uring can provide dramatic performance increases in certain IO-heavy workloads - some PostgreSQL synthetic benchmarks claim even 2-3x increases, but real-world numbers can easily be in the 10-30% range, depending on many factors. I would argue that in a number of use-cases (e.g. big data clusters, etc.), it makes a lot of sense to enable io_uring if it makes your workload faster or more efficient, so you can reduce the required number of hosts. The PinTheft (<a href="https://www.cve.org/CVERecord?id=CVE-2026-43494">CVE-2026-43494</a>) vulnerability also required io_uring.</p><h2>Doing this without overdoing it</h2><p>The hardest part of hardening is finding balance in everything - performance, ease of use, and also, very importantly, the culture. It can easily become a <em>Paranoiafest</em>, because there is always something that can be locked down further. This can fatigue people, to the point that they just start ignoring the warnings after the hundredth &#8220;this is extremely important&#8221; unimportant tunable that catches nothing.</p><p>I recommend documenting hardening steps, especially:</p><ul><li><p>why you do it (threat model, historical bug count, or if it&#8217;s an immediate fix for a vulnerability)</p></li><li><p>what are the costs (resources, operational, complexity)</p></li><li><p>operational notes (why it was made, or how it changes operations), if necessary</p></li></ul><p>This documentation makes it easier for everyone to digest decisions and review them later.</p><p>While paranoia in hardening used to be optional, in a race with AI-assisted bug hunting it is reckless to leave unnecessary attack surface. It is also very important to remind ourselves that hardening is supposed to help ensure things are <em>working</em> safely - a turned-off computer buried in the forest is extremely secure, but won&#8217;t serve customers.</p><p></p>]]></content:encoded></item><item><title><![CDATA[What we need from CI for agentic engineering]]></title><description><![CDATA[Agents push code faster than CI can run. What CI has to become when nobody's at the keyboard.]]></description><link>https://newsletter.depot.dev/p/what-we-need-from-ci-for-agentic</link><guid isPermaLink="false">https://newsletter.depot.dev/p/what-we-need-from-ci-for-agentic</guid><dc:creator><![CDATA[Kyle Galbraith]]></dc:creator><pubDate>Tue, 31 Mar 2026 20:20:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/9b766644-ba98-4c74-b48e-64dab7f8394c_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>It&#8217;s not a controversial statement to say that software engineering has dramatically changed over the past 18 months. The way I write code today is hardly ever by hand. Instead, I can now prompt an agent, or even fleets of agents, to take my wild ideas and make them a reality.</p><p>Just in the past three months we went from having to be intimately in the loop with our agents to now throwing them all kinds of things and letting them cook. We&#8217;re all collectively moving at a velocity never before seen.</p><p>But, we&#8217;re all hamstrung by the elephant in the room: existing CI platforms and paradigms simply can&#8217;t keep up.</p><p>CI was designed for humans.</p><h2>The impedance mismatch</h2><p>The CI systems we have today are built around specific assumptions: a developer writes some code, pushes it to a branch, opens a pull request, and waits. Most engineers context switch to something else while waiting for CI to finish. The traditional processes measure feedback loops in minutes or tens of minutes. It wasn&#8217;t great but it was fine enough.</p><p>With agents writing code, that assumption collapses. An agent can write code in seconds, commit it, monitor CI, read the results, watch for regressions in the logs, detect issues, fix them, and push again &#8211; in a loop &#8211; much faster than any CI system was designed to support. The agent isn&#8217;t doing something else while CI runs. It&#8217;s blocked.</p><p>Today CI systems force humans back into the loop. To help the agent context switch to something else while waiting for CI. To paste errors from CI back into the agent and say &#8220;fix it&#8221;. To help the agent get logs about what regressed.</p><p>Every time an agent has to push code and wait for CI or have a human help it understand its results, you&#8217;ve added friction to a process that&#8217;s supposed to be autonomous.</p><p>CI is now in the critical path in a way it never quite was before.</p><h2>What CI needs to be for agents</h2><p>If you think about what makes CI painful specifically for agentic workflows, although engineers often talk about wanting these capabilities as well, a few things stand out:</p><h3>Targeted reruns, not full pipeline reruns</h3><p>Nothing is more annoying than trying to debug one step inside of a fifteen step CI workflow. That drives engineers nuts. But it actively blocks agents from finishing their work. When an agent is finishing a feature, waiting for a 15-minute pipeline to finish while validating a 3-line fix inside of a fifteen step workflow is genuinely wasteful in terms of time and tokens. Agents need to be able to rerun a single job inside of a workflow &#8211; not restart from scratch every time.</p><h3>Run real CI on local patches</h3><p>Agents and engineers today have to commit and pray to learn if their debugging hypothesis is correct. That&#8217;s the worst possible inner loop: write a change, commit it, push it up, wait 5 minutes, get a result, shit it&#8217;s broken, revert, try again. CI should be able to be invoked with local file changes from the agent writing the code.</p><h3>All context behind an API</h3><p>Most CI systems were designed for humans clicking through dashboards. Their API is a bolted on concept, not something designed to be the primary interface. Agents don&#8217;t click through dashboards. They need to trigger runs, poll status, retrieve logs, and make decisions programmatically. If the API isn&#8217;t there to give agents context, you&#8217;re forcing them into hacky workarounds.</p><h3>Speed and orchestration at scale</h3><p>A single engineer can be operating tens of agents simultaneously. Several agents, several branches, all needing CI at the same time. The latency, queueing, and run time of your CI pipelines and their backing providers matter a lot more when you have 20 agents all trying to validate changes at once. Not to mention that you need pricing that is clear without any hidden gotchas like one minute minimums.</p><h2>What this looks like with Depot CI</h2><p>We built Depot CI and simultaneously used it as the CI system for our own agentic engineering workflows. We made design decisions that make Depot CI a fundamentally different system that engineers and agents can use.</p><p>First, it&#8217;s intended to support arbitrary syntaxes or code that represent a CI pipeline. GitHub Actions is the first syntax we&#8217;re supporting, but we&#8217;re working on our own SDK if you want to use code. We&#8217;re also planning to make the syntax system open source so that you can define your own.</p><p>You or your agent can convert your GitHub Actions workflows to Depot CI with a single command:</p><pre><code><code>depot ci migrate</code></code></pre><p>The <code>depot ci migrate</code> command discovers your existing workflows, applies compatibility fixes, and copies the result to your <code>.depot</code> directory. Your <code>.github</code> folder stays intact &#8212; you can run GitHub Actions and Depot CI in parallel while you validate, then cut over when you&#8217;re ready.</p><h3>Targeted reruns</h3><p>Agents can now kick off a CI workflow without having to trigger any other kind of event or push code to your repository. You can do that with:</p><pre><code><code>depot ci run --workflow .depot/workflows/ci.yml</code></code></pre><p>Or if you only want to run a specific job, you can do that with:</p><pre><code><code>depot ci run --workflow .depot/workflows/ci.yml --job smoke_container</code></code></pre><p>Or if you want to have your agent <a href="https://depot.dev/docs/ci/how-to-guides/debug-with-ssh">debug a specific step in a job</a>, you can do that with:</p><pre><code><code>depot ci run --workflow .depot/workflows/ci.yml --job smoke_container --ssh-after-step 3</code></code></pre><h3>Run real CI on local patches</h3><p>What if your agent has local changes it hasn&#8217;t committed yet? It can run CI against them directly &#8212; no push required. Every <code>depot ci run</code> command picks up local file changes automatically. The agent writes a change, validates it, and only commits once it knows it works. No more pushing broken commits just to find out they&#8217;re broken.</p><h3>All context behind an API</h3><p>Depot CI provides a full API for agents to interact with. Today, you or your agents can access the API through the Depot CLI. You can trigger workflows as described above. But you can also poll status, retrieve logs, and make decisions programmatically.</p><p>Check what workflows are queued or running:</p><pre><code><code>depot ci run list</code></code></pre><p>Check the status of a specific workflow run:</p><pre><code><code>depot ci status &lt;run-id&gt;</code></code></pre><p>Retrieve logs for a specific job attempt:</p><pre><code><code>depot ci logs &lt;job-attempt-id&gt;</code></code></pre><h3>Speed and orchestration at scale</h3><p>The previous three are all about helping agents rapidly iterate and validate their changes with real CI.</p><p>But for that to work, the speed and reliability of the CI system must be able to scale to the needs of the agentic workflows. The volume at which agents can validate and test their changes is going to be massive.</p><p>A single engineer today might be running five, ten, or twenty agents simultaneously &#8212; each on its own branch, each needing CI to validate its work. That&#8217;s not a hypothetical, that&#8217;s just how people are working right now. The queueing behavior, cold start latency, and per-job runtime of your CI system matter a lot more in that world than they did when a team of ten humans was pushing thirty commits a day.</p><p>Depot CI is a completely new category of CI system built with performance at every step. Plumbing that moves a commit to a running job with minimal queue time. Runners optimized for speed and security with our single-tenant architecture. Flexibility to make CI workflows as fast as you need them to be.</p><p>It&#8217;s all there from the start. Not an afterthought. Not a bolt on. Not a checkbox.</p><p>We charge for Depot CI by the second. No one minute minimums. Your agent can validate its changes in seconds instead of minutes, and you only pay for the seconds it takes to run.</p><h2>The bigger picture</h2><p>Targeted reruns, local patch execution, programmatic API access for logs/triggering jobs/checking status, observability into what&#8217;s happening in your job, fast orchestration, low queue times, and reliable execution. Engineers have been asking for these things for years. But no CI system was built around them from the start.</p><p>We built a programmable CI system from the ground up with all of these concepts baked in. Depot CI is built for engineers and agents. It&#8217;s the system engineers have been asking for. And it&#8217;s also the system we need to support the velocity of agentic engineering.</p>]]></content:encoded></item><item><title><![CDATA[The differences between QEMU microvm and Cloud Hypervisor]]></title><description><![CDATA[Choosing a microVM for CI: API, GPU passthrough, snapshotting, live migration, and hotplug, side by side.]]></description><link>https://newsletter.depot.dev/p/the-differences-between-qemu-microvm</link><guid isPermaLink="false">https://newsletter.depot.dev/p/the-differences-between-qemu-microvm</guid><dc:creator><![CDATA[Peter Hasko-Nagy]]></dc:creator><pubDate>Wed, 18 Feb 2026 21:45:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/4a570114-7c24-45d7-a376-b275cd0bae87_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2>Prologue</h2><p>I bet you&#8217;ve heard the term microVM recently&#8212;it goes hand in hand with CI and sandboxes, both hot topics&#8482; of 2026. Say you want to build a cool CI project with super fast-booting VMs. How do you choose from the myriad of available technologies? We&#8217;re not here to tell you which one to choose, but we can at least try to explain some differences between QEMU microvm and cloud-hypervisor. As the fundamentals and goals are quite similar, we&#8217;ll concentrate on the most notable differences that might be interesting from the perspective of building a platform.</p><h2>What is a microVM?</h2><p>(If you&#8217;re already familiar with microVMs, feel free to skip to the next section.)</p><p>microVMs are lightweight virtual machines with strong isolation guarantees, a reduced attack surface, and usually pretty fast boot times. In Linux-land, the most notable players are all KVM-based. The VMs are typically managed by a VMM (Virtual Machine Monitor) such as Firecracker or cloud-hypervisor. There are a few performance tricks that make microVMs snappier compared to full virtualization:</p><ul><li><p><strong>Paravirtualization</strong>, where the guest OS is aware of being virtualized and communicates with the hypervisor via optimized API calls instead of emulation.</p></li><li><p><strong>Minimalist kernels</strong>, compiled only with the necessary components so they don&#8217;t need to load extra modules at boot.</p></li><li><p><strong>Lightweight init</strong>, employing a specialized, low-footprint init script instead of something like systemd.</p></li></ul><p>Due to the lower resource overhead, you can achieve higher density by packing more microVMs on a single host compared to traditional VMs, especially with CPU overcommit. These properties make microVMs a good fit for short-lived, ephemeral workloads.</p><h2>Comparison</h2><h3>Background</h3><p>cloud-hypervisor is a modern VMM written in Rust. It&#8217;s inspired by Firecracker and even shares some of its components. As a relatively new project focused on cloud workloads, it has no business supporting legacy CPU architectures and devices. cloud-hypervisor runs on KVM (Linux) and Microsoft Hypervisor, but has no macOS support.</p><p>The QEMU microvm machine type is also relatively new and also inspired by Firecracker. It&#8217;s powered by KVM and runs exclusively on Linux. Notably, it doesn&#8217;t support PCI-only devices, preferring virtio-mmio as its transport instead. More on the drawbacks later.</p><h3>API support</h3><p>cloud-hypervisor has a rich, user-facing <a href="https://raw.githubusercontent.com/cloud-hypervisor/cloud-hypervisor/master/vmm/src/api/openapi/cloud-hypervisor.yaml">REST HTTP API</a>. In fact, this is the primary way to manage VMs and interact with the VMM. When a cloud-hypervisor instance is started, the API is made available via either a UNIX socket or a file descriptor. As you may have noticed, there isn&#8217;t a higher-level abstraction or control plane that lets you list running VMs, observe their statuses, etc. Each invoked binary corresponds to a single cloud-hypervisor process.</p><p>See the <a href="https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/api.md#rest-api-endpoints">cloud-hypervisor API docs</a> for the full endpoint list.</p><p>The QEMU microvm machine type doesn&#8217;t expose a REST API, but it does offer several other ways to interact with the VMM. One notable mention is <a href="https://wiki.qemu.org/Documentation/QMP">QMP</a>, a JSON-based protocol over a UNIX socket. As an alternative, you might use libvirt, an open-source VM management tool and API that supports various hypervisors, including QEMU. The bad news is that microvm compatibility varies greatly depending on the Linux distribution you&#8217;re using, so you&#8217;ll probably end up compiling your own libvirt version. Unfortunately, libvirt uses its own RPC-based binary format instead of REST and relies heavily on XML for configuration, which might discourage some.</p><p>For more information, see the <a href="https://libvirt.org/html/index.html">API reference for libvirt</a>.</p><h3>GPU passthrough</h3><p>GPU passthrough is a way to make a physical GPU attached to the host available inside the VM. The VM should see the device as if it were native.</p><p>cloud-hypervisor uses the VFIO (Virtual Function I/O) framework, implementing the vfio-pci device to allow attaching compatible devices to the VM. (It doesn&#8217;t have to be a GPU: it could be an SD card or a network card as well.) TL;DR: cloud-hypervisor supports GPU passthrough.</p><p>2026 is all about GPUs so all VMMs must support GPU passthrough, right? Well, not quite. As mentioned earlier, QEMU microvm doesn&#8217;t support PCI devices, which means direct GPU passthrough isn&#8217;t possible out of the box. (However, since QEMU 5.2, PCIe is configurable for microVMs as well.)</p><h3>Snapshotting</h3><p>The essence of snapshotting is capturing a VM&#8217;s state and dumping it to disk for later use. The saved data typically includes CPU register state, disk contents, device states, and dirty (in-use) memory pages. This state can then be restored at a later point.</p><p>cloud-hypervisor supports snapshotting via its API. First, the VM needs to be paused, then the snapshot can be taken. When restoring a VM from a snapshot, it needs to be explicitly resumed. The only caveat is that VFIO device states are not included in the snapshot. cloud-hypervisor allows snapshots to be taken regardless of the disk format, including raw disks. Watch out though, snapshotting isn&#8217;t guaranteed to work across different versions.</p><p>Snapshotting should work with QEMU microvms as well, using the <code>savevm</code> and <code>loadvm</code> commands. These are exposed to the user via the QEMU monitor or QMP. At least one writable qcow2 disk is needed, as the entire VM state is dumped into the disk image. Alternatively, QEMU&#8217;s <code>migrate</code> command can dump the full VM state (CPU, RAM, devices) to an external file, which works regardless of disk format, though the disk image must be managed separately in that case.</p><h3>Live migration</h3><p>Live migration is the act of moving a running VM from one process to another on the same host, or from one host to another entirely. All processes and connections within the VM should remain intact, and the source VM should gracefully stop once the migration completes. For this to work, both environments need to be identical. The migration is typically performed by iteratively copying dirty memory pages to the destination and switching over once the delta is small enough. Live migration is useful when a host needs to be decommissioned but the workloads must keep running (like for mission-critical services).</p><p>cloud-hypervisor supports live migration. As usual, the functionality is exposed via its API.</p><p>Nothing in the documentation explicitly says that QEMU microvm doesn&#8217;t support live migration, so there&#8217;s reason to believe it does. However, live migration with ongoing external IO isn&#8217;t supported for the microvm machine type.</p><p>For both cloud-hypervisor and QEMU, live migration isn&#8217;t guaranteed to work across different versions, which is a small price to pay for such a useful feature.</p><h3>Hotplug</h3><p>Hotplugging is the act of adding, removing, or modifying VM resources and devices without requiring the VM to be stopped or rebooted. Just as plugging in a USB device makes it instantly appear on your machine, VMs should be no different, right?</p><p>Well, cloud-hypervisor supports hotplugging x86 vCPUs, PCI devices, and memory resizing. When adding extra vCPUs to a running VM, they need to be &#8220;onlined&#8221; on the guest side. Multiple memory hotplugging methods are supported, some of them semi-manual (see the <a href="https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/hotplug.md#memory-hot-plug">documentation</a> for details). Adding or removing PCI devices (including GPUs) is handled by the guest with no manual work required.</p><p>Unfortunately, QEMU microvm doesn&#8217;t support hotplugging at all.</p><h2>Conclusion</h2><p>To make everything we&#8217;ve covered more digestible, let me summarize it in a table.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!Hyub!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Hyub!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 424w, https://substackcdn.com/image/fetch/$s_!Hyub!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 848w, https://substackcdn.com/image/fetch/$s_!Hyub!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 1272w, https://substackcdn.com/image/fetch/$s_!Hyub!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Hyub!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png" width="1456" height="719" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:719,&quot;width&quot;:1456,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:188941,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/png&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:&quot;https://workingtheoryeng.substack.com/i/211227682?img=https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png&quot;,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="" srcset="https://substackcdn.com/image/fetch/$s_!Hyub!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 424w, https://substackcdn.com/image/fetch/$s_!Hyub!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 848w, https://substackcdn.com/image/fetch/$s_!Hyub!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 1272w, https://substackcdn.com/image/fetch/$s_!Hyub!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F835458e7-8285-4e07-bacc-fafff9ec3d4d_1907x942.png 1456w" sizes="100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container restack-image buttonBase-GK1x3M"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button tabindex="0" type="button" class="pencraft pc-reset pencraft icon-container view-image buttonBase-GK1x3M"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a></figure></div><p>There&#8217;s no intention to declare a winner here. You might have a very good reason to choose either of these amazing technologies. For example, if you run virtualization across multiple platforms and guest OSes, and you don&#8217;t need GPU passthrough or hotplugging, it might make sense to go with QEMU, which has the same API and shenanigans as the rest of your stack. As with everything in life, collect information, capture your requirements, and make an educated decision.</p><p>If you&#8217;re interested in which VMM we chose, stay tuned! Join our <a href="https://discord.gg/MMPqYSgDCg">Discord Community</a> to get all the latest.</p>]]></content:encoded></item><item><title><![CDATA[My intuition doesn't work anymore]]></title><description><![CDATA[An LLM refactored a 1500-line mess into clean code. Months later, only agents worked in it.]]></description><link>https://newsletter.depot.dev/p/my-intuition-doesnt-work-anymore</link><guid isPermaLink="false">https://newsletter.depot.dev/p/my-intuition-doesnt-work-anymore</guid><pubDate>Thu, 12 Feb 2026 17:26:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/43d94e5a-c158-4445-868d-537d169af655_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2>John Henry&#8217;s hammer</h2><p>John Henry is an American folk legend. He&#8217;s a steel-driving man. Driving rail spikes through plains and mountain tunnels alike. He raced a steam-powered drill to prove a human could outwork a machine. He won. Then he dropped dead, hammer still in his hand.</p><p>I keep thinking about what that must have felt like. Standing there, watching the machine drive spikes into the earth with speed, consistency, and precision. Wondering if the skills you&#8217;ve spent your life getting good at were about to stop mattering.</p><p>Because so many of my intuitions as an engineer&#8212;intuitions I&#8217;ve honed over a decade about readability, maintainability, and &#8220;doing it right&#8221;&#8212;feel wrong now. Part of how I judged myself as an engineer was based on the empathy I showed in my code. How much easier I made it for the next engineer to read and maintain. But I&#8217;m barreling into a world where that &#8220;next engineer&#8221; is just a bundle of statistics finding the next token.</p><p>I don&#8217;t know what to do with that. Let me show you what I mean.</p><h2>The moment I noticed</h2><p>I recently found myself staring at a 1500-line file in our codebase. It was a dumping ground. It had no structure, terrible naming conventions, and logic that meandered like me when I&#8217;m with my wife at Target: seemingly aimless, just hoping I could get the heck out of there.</p><p>My engineer instincts kicked in. I felt that familiar itch, the urge to refactor. I fired up an LLM and told it to clean the file up. It did a beautiful job. It organized the logic, extracted duplications into helper functions, and turned that dumping ground into something I could more easily parse with my eyeballs.</p><p>But then I stopped and looked at the pristine new file. It was better. It was cleaner. But did it actually matter?</p><p>We write clean code so that other humans can read it. It&#8217;s empathy and insurance for the next human that has to touch it. Over the following months, most every change to that file was made by an agent. Not once did I need to open it and squint at the logic. The agent never cared that I&#8217;d organized it into neat little sections. It would have been just as happy chewing through the dumping ground. I was refactoring for me, and I used &#8220;empathy for my fellow engineers&#8221; as justification.</p><h2>40 years of empathy for the human brain</h2><p>Look at these two loops.</p><p>Example A:</p><pre><code><code>MOV CX, 10
L1:
ADD AX, 1
DEC CX
JNZ L1</code></code></pre><p>Example B:</p><pre><code><code>for i in range(10):
    count += 1</code></code></pre><p>Which one is easier to understand? For 99.999% of us it&#8217;s B. If you picked A, congratulations, I award you one thousand nerd points.</p><p>Why do we prefer B? It&#8217;s not because Python is faster. It&#8217;s way slower. We prefer it because it&#8217;s faster for our brains to parse what the heck is going on.</p><p>For the last 40 years, many of the advances in software have been about empathy for the human brain. We built abstractions, invented high-level languages, and wrote books like &#8220;Clean Code&#8221; because human cognitive capacity is limited. Descriptive variable names and small functions are useful because, unlike computers, I forget what I read a few lines ago.</p><p>Code became optimized for the reader, not the compiler.</p><p>But what happens when the reader is no longer human?</p><h2>Code as artifact</h2><p>Here&#8217;s the thing I can&#8217;t unthink: the code is not the point. It&#8217;s an instructional artifact to produce something useful for a user. No user downloads an app and says &#8220;Man, I hope they preferred composition over inheritance.&#8221;</p><p>I tried to talk myself out of this. &#8220;Sure, the AI can write it, but can you fix it when it breaks at 3 AM?&#8221;</p><p>Being on-call used to mean a customer messaged me with some obscure error, and I&#8217;d spend the next thirty minutes manually traversing Cloudwatch and Axiom logs. Hoping to find a needle in a haystack. Now, I copy-paste the error, hit the Cloudwatch and Axiom MCPs, and boom: 9 out of 10 times I find the issue. What used to be a log safari is now ninety seconds and a few tool calls. Multiply that across a team, across a week, and I can&#8217;t really tell myself that clean code is what&#8217;s saving time.</p><p>Then I told myself it was about structure. Fine, variable names don&#8217;t matter, but structure matters. And maybe it does. But when I&#8217;m honest with myself, I can&#8217;t point to evidence that an LLM writes worse code in a messy repo. And that bothers me. I&#8217;ve watched agents make correct changes in files I&#8217;d be afraid to touch. The machine doesn&#8217;t need the guardrails that I need.</p><p>The last thing I clung to was depth. AI sucks at fixing deep architectural rot that rapidly fills the context window. And that&#8217;s true.</p><p>For how much longer?</p><h2>Where the tunnel needs to go</h2><p>It was never about the hammer though. It&#8217;s knowing where the tunnel needed to go. The judgement, the systems thinking, the ability to communicate and tease the <em>real</em> product requirements. That&#8217;s the real work. It always was.</p><p>And this is where John Henry&#8217;s story diverges from ours. Or so I like to tell myself. He was pure strength and execution. We&#8217;re the ones who decide where the tunnel goes and why it&#8217;s being built at all. The machine could match his hammer but it can&#8217;t match our judgement. Right?</p><p>Because what really worries me is that the same pattern that is making clean code less relevant is climbing the ladder we&#8217;re all standing on. Each rung AI climbs, I tell myself <em>this</em> is the one that requires human judgement. And each time, the horizon gets a little closer.</p><p>So, what happens when the machine catches up? When it knows where the tunnel should go?</p><p>I don&#8217;t have an answer. But I think the honest version of this essay isn&#8217;t &#8220;relax, the important stuff is safe.&#8221; It&#8217;s that the boundary between where humans excel and machines excel is blurrier than I&#8217;d like to admit. And it&#8217;s still moving.</p><p>And you know what? I still pick up the hammer some days. I miss how it feels in my hands.</p>]]></content:encoded></item><item><title><![CDATA[The bottleneck has shifted from writing code to integrating it]]></title><description><![CDATA[Writing code stopped being the slow part. Now it's integration, review, and CI, and workflows haven't caught up.]]></description><link>https://newsletter.depot.dev/p/the-bottleneck-has-shifted-from-writing</link><guid isPermaLink="false">https://newsletter.depot.dev/p/the-bottleneck-has-shifted-from-writing</guid><dc:creator><![CDATA[Kyle Galbraith]]></dc:creator><pubDate>Mon, 02 Feb 2026 21:14:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/86d912cb-1f25-4414-9850-89b411a67571_1200x630.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>It&#8217;s officially 2026, and teams are settling back into their routines after the holidays. We&#8217;re seeing new teams adopt Depot and existing teams ramp up usage.</p><p>What&#8217;s fueling this? It&#8217;s AI and LLMs. We&#8217;re experiencing a fundamental shift in software development.</p><p>For decades, the biggest bottleneck to innovation was writing code. But today, that&#8217;s no longer true. Code and ideas can now iterate as fast as you can articulate them to an AI agent with a powerful model like Opus 4.5 or GPT-5.2 Codex.</p><p>Iteration can be done in seconds instead of hours. And a new generation of tools, services, and products are being imagined faster than ever.</p><p>The bottleneck has shifted from writing code to integrating it.</p><h2>How did we get here?</h2><p>When LLMs first became available to the public, the developer ecosystem was quick to experiment with them. We saw folks generate code from prompts. It kind of worked and it was neat. But it was also a bit of a mess. The code was often buggy and the quality was often questionable. But it was a start.</p><p>We started seeing their potential, but we didn&#8217;t really trust them.</p><p>So we figured out ways to integrate them into our existing workflows. The idea of spinning up agents via things like GitHub Actions or CircleCI became a thing. We quickly started seeing AI agents being &#8220;bolted on&#8221; to our existing workflows for code reviews, test generation, and more.</p><p>In essence, we took this new experimental capability and bolted it on to our existing human-centric workflows.</p><p>Then things changed. It felt like overnight, but it was steady improvements in model capabilities every couple of weeks. Models like Opus 4.5 and GPT-5.2 Codex are better at understanding code, better at generating code, better at understanding context, better at understanding codebases.</p><p>We shifted from a world where the agents and models were experimental and hard to trust to one where they are reliable, productive, and better than humans at many tasks.</p><h2>The new traffic jam that has been created</h2><p>With that shift, a new traffic jam formed in our human-centric workflows.</p><p>Engineering teams operating with agents are now seeing the downstream effects of their newfound productivity. Pull requests pile up faster than humans can review them. CI queues grow because builds and tests are taking too long to run. Merge conflicts multiply as more changes flow simultaneously.</p><p>Repurposing our existing workflow that focused on engineers in the middle of everything is now at the root of the bottleneck. Bolting AI into our existing workflows worked okay initially. This worked when writing code was the slow part. When a developer spent days on a feature, 20 minutes of CI wasn&#8217;t the constraint. <strong>When you can generate a feature in 20 minutes, a 20-minute CI pipeline is unacceptable.</strong></p><p>I&#8217;m seeing this with Depot customers right now. They&#8217;re coming to us with more and more requests to hyper-optimize very specific parts of their CI workflows.</p><p>Can you help make our tests faster? Could you support caching this thing as well? Could you get us a runner in milliseconds for these CI checks that take 30 seconds to complete?</p><p>All these questions stem from teams that want to move faster, and they&#8217;re seeing the bottlenecks that have always been there but now are more pronounced.</p><p>We&#8217;re producing exponentially more code and iterating faster than ever before. But we can&#8217;t keep the momentum of this newfound power if we can&#8217;t move it through review, integration, testing, and deployment faster.</p><h2>What got us here won&#8217;t get us there</h2><p>The human-centric workflows we built assume developers work at a measured, deliberate pace. Code review happens asynchronously. CI runs are expensive. Integration happens infrequently because changes are large and risky.</p><p>None of these hold anymore. AI agents produce working code in minutes, attempt dozens of approaches simultaneously, and generate code 24/7. But they&#8217;re forced into collaboration patterns designed for humans working business hours on carefully crafted changes.</p><h2>The real-time feedback loop is the key</h2><p>Here&#8217;s what I believe: <strong>real-time feedback loops are what empower software engineering at scale.</strong></p><p>We need to rethink our workflows to support this new reality. We need tests to run on every commit, merge conflicts to self-resolve automatically, context about how the code was developed should live right next to the code, and builds should be near instant. All of this should be seamless for both humans and agents.</p><p>It&#8217;s not about a human in the loop anymore. It&#8217;s about humans orchestrating work being done at scale, with engineers deciding what good vs. bad is, what to iterate on, what ideas to explore, and what to ship next.</p><p>We&#8217;ve seen this at Depot with build performance. Teams with 60-second builds make fundamentally different decisions than teams with 40-minute builds. They try more things. They validate assumptions faster. They catch issues earlier. <strong>Speed doesn&#8217;t trade off with quality. Speed enables quality.</strong></p><h2>We need a new paradigm</h2><p>The solution isn&#8217;t faster pull requests or cheaper CI pipelines. Those are incremental improvements to a fundamentally outdated model.</p><p>We need workflows that integrate continuously, validate automatically, provide real-time context, and work for both humans and agents.</p><p>Think about Google Docs. Multiple people work on the same document simultaneously. Changes sync in real-time. Comments live in the document. Spelling and grammar checks run automatically as you type. <strong>Software engineering collaboration should work the same way.</strong></p><p>Every team adopting AI coding tools will hit this bottleneck. Some already have. But we are likely only at 1-2% adoption of AI coding tools. As adoption increases, this bottleneck becomes the most pronounced barrier to break down. Teams that figure out how to bust through it will have a massive competitive advantage. They&#8217;ll operate with a new paradigm, where agents have everything they need to go from engineers&#8217; ideas to running in production as quickly and autonomously as possible. They will run circles around their competitors who don&#8217;t have this new paradigm. They&#8217;ll operate above their weight class.</p>]]></content:encoded></item></channel></rss>