Thoughtworks November Tech Radar Notes
Agentic Mutation Testing
Process of evaluating test suite quality by automatically introducing small, artificial bugs to application code & verifying at least one fails.
With AI generated code and tests increasing rapidly, it’s becoming easier to inflate and harder to catch superficial tests that lack meaningful validation.
Agentic Mutation Testing can give a team the guardrail to ensure that generated tests aren’t just being generated to pass without being able to catch actual logic changes.
This can be done autonomously, as part of the agentic execution, without directly modifying code outside of the sandboxed environment that the agent is working on, PRE COMMIT.
Path to Production Team Exercise
Increased code throughput can expose and/or increase bottlenecks in the path to production.
It is becoming increasingly important for teams to be optimising the end-to-end flow rather than time to write code.
This activity is best done in a workshop, possibly on-site, where the teams that are responsible for designing, developing and releasing the software can collaborate on defining the path to production with the least resistance. Capturing pain points, current process, desired process, steps to get there and known limitations / blockers.
Regular Threat Modeling
Threat Modeling is an exercise to identify potential threats in owned systems. It’s a core practice in software engineering but has become even more important in the era of agentic engineering.
Threat Modeling should be a regular, human led part of the SDLC, not an afterthought or one off task.
Effective Agentic Code Knowledge Index
Having agents build up your readme and supporting documentation can be extremely effective, but unlike buildable code & software, it doesn’t inherit safety measures. The knowledge index that agents are creating needs to be treated like a first-class citizen and teams MUST be reviewing it, pruning it and ensuring that as part of the agentic workflow it is being maintained.
I would go as far as saying that documentation that is being created by an agent should be treated with the MOST amount of scrutiny, while undoubtedly providing immense efficiency gains.
Feedback Flywheel
The Feedback Flywheel concept is based around continuously improving the agentic workflow, possibly after every single session an engineer has. There will be additional cost associated with this technique, but if it is providing continuous improvement to the workflow that future sessions rely on, that seems invaluable.
I feel fairly sceptical about this given agents can already be fairly question led, I wouldn’t want this to be a loop-trap so there probably ought to be some guidance in the prompting around how many questions you want it to ask.
Mixed with the following topic though, it could be quite powerful.
Agent Suggestion Box
The concept for this is fairly simple, an area whereby agent sessions can submit suggestions to. It is imperative that this is reviewed and prioritised by humans, I wouldn’t ever subscribe an agentic workflow that takes an agent’s suggestion and then delivers it without a human at least reviewing and signing off the suggestion as something that we want to deliver.
We would want another agent to possibly try to de-dupe as if this is an engineering-wide initiative it wouldn’t scale very well with a large number of engineers and would be a nightmare of a backlog to maintain.
Overnight Agentic Build Workflow
Thoughtworks have teams experimenting with the idea of spending more time preparing agents to implement overnight agents at a feature and/or epic level, the teams will dedicate capacity to building an extensive documentset that enables the agents the context to deliver and then for it to be completed overnight by an agent/s for the team to then review it in the morning.
I have been experimenting with this myself both at inside and outside of work and honestly, the results are pretty incredible, I have built parallel streams of work outside of my team and run a full agentic workflow on them and the results have been mixed, sometimes they have been really impressive and gone straight through peer review and QA with no additional input, other times have been picked apart by engineers in code-review and QA against missed requirements and broken functionality, arguably, the engineer responsible for orchestration would usually pick up from the agent, but my particular experiment was to see what the current frontier models were capable of with no touch engineering.
I think overnight agentic work has a place in the engineering lifecycle, but it heavily relies on good requirements being captured, great context being provided to the agent of the current system, desired output both technically and from a product perspective and a strong engineering SDLC to catch anything that should not make it past a local environment.
Tests as data >? Tests as code
Tests as data is not a brand new concept being created for agents, it’s already been done when doing BDD and Gherkin format. Experimenting is being done at Thoughtworks around having data driven test data alongside a strong test harness that executes the underlying code against the data provided. I thought this was really interesting, it’s not something i’ve expiremented with personally, but wanted to document it in my notes to drive discussion with colleagues.