"End of task guardrails" are routinely skipped as context fills up

[Ovid]

No matter how many times I try to adjust the constitution/spec/plan/tasks, I cannot get Spec Kit to reliably ensure all tests are passing when it finishes a task. Much of this appears to be the context rot and the LLM "forgetting" about end-of-task guardrails. I've filed an issue with vscode copilot chat to help with this, but the core problem is that the Spec Kit workflow doesn't offer a clear way to deal with this.

I can type "fix the following broken tests, ensuring that you're respecting the constitution/spec/plan/tasks. Ensure that you're understanding if it's the code that is broken or the tests that are broken and fix appropriately. At the end of this, YOU MUST RERUN THE ENTIRE TEST SUITE, fixing any new broken tests."

But that's a pain, so I added a /fixtests command that I can easily run in a new session. However, there are other guardrails, this won't fix, such as "halt after each chunk of work and let the user commit." It works at first, but as context rot sets in, it doesn't and Spec Kit happily commits the changes for me.

I don't have a clear solution for this, but "end of task" guardrails that must never be skipped are routinely skipped. It would be nice if there was some way of addressing this. I keep trying to make tasks as small as possible, but some problems, even when broken into small chunks, turn out to be hard problems.

[foertel]

Unfortunately this is very much given in working with LLMs. Everything spec-kit would do, would add even more to the context.

Most of us use deterministic solutions to this problem, mostly called "harness". You can do it with claude code hooks (running tests before every attempt to commit stuff to git) or small frameworks like pocketFlow (defining a series of steps that will be run, no matter what the LLMs "decide").

[harikrishnan83]

I use claude code and In my experience assigning subagents to handle a set of tasks an help address the issue of context build up. Here is a detailed writeup I had put together on this. Maybe in your case you can move the details in your /fixtests command to a Subagent and in the Constitution you can say any test failures have to be handled specifically by that subagent. Hope this helps.

[foertel]

I like the idea of sub-agents, but their observebility is really crappy. If your sub-agent "fixes" something, it will only return a summary of the result to the main agent and then all context will be lost. You can not iterate on that. Most of the time, you will not know what your sub-agent actually did.

Most people I know use sub-agents only for summarized context retrieval. Like "go through this 80 kb of log file and give me a short explanation of patterns you see". So the logfile will not end up in your context.