OpenAI and Playwright: Transforming LLMs into Autonomous Browsers

Applications of Extended Language Models (LLM)

Integrating an LLM agent with a web browser represents a significant advancement in enhancing automated workflows. By using the OpenAI Agents SDK and Playwright MCP, we can create an agent capable of autonomously navigating and interacting with web pages. This article explores how this technology works and presents a practical case study to illustrate its application.

1. Understanding the Mental Model

An LLM agent that uses a browser essentially operates in a continuous interaction loop. This loop begins with a given task and the current state of the browser. The agent evaluates this state, chooses an action to take, and transmits that action to the browser. The browser responds, altering its state, which then becomes the new input for the agent. This process repeats until the agent deems the task complete.

For this loop to function, two types of connections are necessary between the agent and the browser:

• An observation channel, which allows the agent to receive the current state of the browser.
• An action channel, which enables the agent to interact with the browser.

Observation channels can include screenshots or structured information about the page, such as the Document Object Model (DOM) or the accessibility tree. For the action channel, the agent can use mouse and keyboard commands, target specific elements on the page, or issue higher-level browser commands.

While the choices for observation and action are generally independent, two common associations are observed:

• Screenshots with mouse and keyboard actions based on coordinates.
• Structured page state with actions targeted at elements.

In this article, we focus on using structured page observations with browser actions targeted at elements.

2. Case Study: Resolving a Customer Support Request

To illustrate the practical application of this technology, we developed an agent using a browser to resolve a customer request via a web support console.

2.1 Setting Up the Support Console

We designed a simple customer support console, built with HTML, CSS, and JavaScript. This static web application does not require a backend or database, as all data resides in the browser. The console can be served locally using Python's built-in HTTP server:

python -m http.server 8000 --bind 127.0.0.1

This makes the console accessible at http://127.0.0.1:8000, which we will pass to the agent under the name APP_URL.

The console is divided into two parts: a support inbox on the left, and a section displaying the associated command, customer context, and resolution policies on the right. The agent must examine an incoming support request and resolve it entirely through this interface.

2.2 Configuring the Agent Using a Browser

To configure the agent, we use the OpenAI Agents SDK for execution and Playwright MCP for connecting to the browser. Here’s what our configured agent looks like:

# pip install openai-agents
from agents import Agent, ModelSettings
from openai.types.shared import Reasoning

name="Support Console Browser Agent",
model="[gpt](/glossaire/gpt)-5.4",
model_settings=ModelSettings(
reasoning=Reasoning(effort="medium"),
instructions=AGENT_INSTRUCTIONS,
mcp_servers=[playwright_server],

Three elements are essential here: the LLM client, the agent's instruction, and the browser tools.

First, we connect the Agents SDK to Azure OpenAI:

from openai import AsyncAzureOpenAI
from agents import (
set_default_openai_api,
set_default_openai_client,
azure_client = AsyncAzureOpenAI(
api_key=os.environ["OPENAI_API_KEY"],
api_version=os.environ["OPENAI_API_VERSION"],
azure_endpoint=os.environ["OPENAI_API_BASE"],
set_default_openai_client(azure_client)
set_default_openai_api("responses")

We register the client with the Agents SDK and configure it to use the Responses API.

Next, we define the agent's instruction minimally:

AGENT_INSTRUCTIONS = """
You are an agent that can interact with a web browser.
"""

We have only defined the agent's role. The actual task will be specified in the prompt sent to the agent.

We then need to configure Playwright MCP. Playwright is a browser automation library that controls a real browser to perform actions like clicks and inputs. MCP (Model Context Protocol) exposes these capabilities as tools that the agent can use.

Playwright MCP uses Node.js. To install it on Windows:

winget install OpenJS.NodeJS.LTS
brew install node
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.5/install.sh | bash
\. "$HOME/.nvm/nvm.sh"
nvm install --lts

Next, verify the installation. We configure the Agents SDK to start the MCP server via the following npx command:

from agents.mcp import MCPServerStdio
playwright_server = MCPServerStdio(
name="Playwright MCP",
"command": "npx",
"@playwright/mcp@latest",

Here, npx retrieves and executes the latest Playwright MCP package. The -y flag automatically accepts the confirmation prompt from npx, while --browser chrome tells Playwright which browser to launch. Chrome is the default browser for Playwright MCP, and if it is already installed, no separate installation is required.

MCPServerStdio configures the Agents SDK to launch Playwright MCP as a local process. When the agent first calls a browser tool, Playwright MCP opens a visible Chrome window and executes the requested browser action.

2.3 Running the Agent

We can now give the agent a concrete task:

APP_URL = "http://127.0.0.1:8000"
Open {APP_URL} and resolve the support case for order ORD-1042.
The customer says they received the wrong item. Use the information available in the
application to determine and apply the appropriate resolution. Add a concise internal
note and [make](/outil/make) sure the resolution was successfully recorded.
Report what you did when the task is complete.

In the task prompt, we described how to access the application and the desired outcome.

We then run the agent with:

from agents import Runner
async with playwright_server:
result = await Runner.run(
print(result.final_output)

The async with block starts the Playwright MCP process and keeps it connected while the agent runs. We use max_turns to set an upper limit on the number of turns the agent can take.

Once started, Chrome would open, and we could observe the agent working through the support console.

The final response accurately summarizes the outcome:

Resolved CASE-4107 for order ORD-1042 with Replacement.

The case now shows as Resolved, the recorded action is Replacement, and the audit log contains the corresponding resolution entry.

If desired, you can also inspect the browser tool calls and their outputs in this way:

for item in result.new_items:
print(type(item).__name__, item)

During my execution, the agent found the case associated with ORD-1042 and inspected the order, the customer's request, the inventory status, and the relevant resolution policy. It then concluded that a replacement was appropriate, added an internal note, and submitted the resolution. Finally, it checked the updated case and the audit log to confirm that the action had been recorded.

This is exactly the agentic behavior we aim for.

3. From Browser Use to General Computer Use

What we have built in this case study is an agent using a browser. However, the underlying model, that is, the observe, decide, act, and repeat loop, naturally extends to general computer use.

What changes are the observation and action channels.

In our case, Playwright MCP provides the agent with structured information about the page and allows it to target individual web elements. A more general computer use agent could instead operate from screenshots and control the mouse and keyboard by coordinates.

You can find the repository for our case study here: https://github.com/ShuaiGuo16/llm-browser-agent/tree/main