Coding Agents: Revolutionizing Manual Software Testing

The New Era of Coding Agents

Coding agents stand out due to their unique ability to not only generate code but also execute it immediately. This feature makes them infinitely more practical than large language models (LLMs), which merely produce code without any verification of its functionality. Indeed, it is crucial never to assume that code generated by an LLM is operational until it has been tested in real-world conditions. Coding agents, on the other hand, can confirm that the code they produce works as intended or iterate until it achieves that goal.

The Crucial Importance of Unit Testing

Encouraging agents to write unit tests, particularly by adopting the test-driven development (TDD) method, is a powerful approach to ensure that the code has been properly tested. However, this method is not the only path to follow. Successfully passing tests does not necessarily guarantee that the code functions as expected. Experienced developers know that even when all automated tests pass, the code can still fail spectacularly, for example, by causing a server crash on startup or omitting a crucial element from the user interface. Thus, automated tests cannot replace manual testing. Seeing a feature in action with one's own eyes before integrating it into a release remains a valuable step. I have often found that manual tests conducted by agents reveal issues that automated tests did not detect.

Strategies for Manual Testing by Agents

How an agent should "manually" test a piece of code largely depends on the nature of that code. For Python libraries, an effective method is to use python -c "... code ...". This technique allows you to pass a string of Python code directly to the interpreter, including code that imports other modules. Coding agents are generally familiar with this trick and sometimes use it spontaneously. However, reminding them to test using python -c can often be beneficial:

• Test this new function on edge cases using python -c

For other languages, similar mechanisms may exist, and in their absence, an agent can quickly write a demonstration file, compile it, and execute it. I sometimes encourage the use of the /tmp directory to prevent these temporary files from being accidentally added to the main repository.

• Write code in /tmp to test edge cases of this function, then compile and execute it

Many projects involve creating web applications using JSON APIs. In these cases, I recommend that agents test them using curl:

• Start a development server and explore this new JSON API with curl

Asking an agent to "explore" often encourages testing various aspects of a new API, quickly covering a wide range of scenarios. If an agent discovers a malfunction during their manual tests, I advise them to fix it using the red/green TDD approach, ensuring that the new case is covered by permanent automated tests.

Browser Automation for Testing Web Interfaces

Setting up a manual testing procedure becomes even more crucial when a project involves an interactive web interface. Historically, these interfaces were difficult to test through code, but the last decade has seen significant improvements in web browser automation systems. Running a real browser like Chrome, Firefox, or Safari against an application can reveal a multitude of issues in a realistic setting. Coding agents are well-versed in these tools. The most powerful of them today is Playwright, an open-source library developed by Microsoft. Playwright offers a comprehensive API with bindings in several popular programming languages and can automate any popular browser engine.

Simply asking your agent to "test this with Playwright" may suffice. The agent can then choose the most appropriate language binding or use Playwright's playwright-cli tool. Coding agents work very well with dedicated command-line interfaces (CLI). Vercel's agent-browser is a complete CLI wrapper around Playwright, specifically designed for use by coding agents.

My own project, Rodney, pursues a similar goal, although it uses the Chrome DevTools Protocol to directly control an instance of Chrome. Here’s an example of a prompt I use to test features with Rodney:

• Start a development server, then use uvx rodney --help to test the new homepage, look at the screenshots to confirm that the menu is in the right place

This prompt incorporates several tips:

• Saying "use uvx rodney --help" encourages the agent to run rodney --help via the package management tool uvx, which automatically installs Rodney the first time it is called.

• The command rodney --help is specifically designed to provide agents with all the necessary information to understand and use the tool.

• Saying "look at the screenshots" suggests to the agent to use Rodney's screenshot command and reminds them that they can use their own visual capabilities on the resulting image files to assess the visual appearance of the page.

This prompt integrates a large number of manual tests into a concise instruction!