Architecture and Capabilities of Browser

Use: An AI-Powered Web Automation

Framework
Limitation Acknowledgment: During the research phase, direct access to the raw content of
certain key files within the browser-use GitHub repository (e.g., browser_use/agent/service.py,
browser_use/browser/browser.py, browser_use/controller/service.py, and the browser_use/llm/
directory) was reported as "inaccessible" by the research tool. Despite this limitation, significant
architectural details have been inferred and extracted from available information, including the
browser_use/__init__.py file , detailed analyses of specific services like DomService and
Controller , usage examples , and the comprehensive README.md. This report synthesizes
these available data points to provide the most concrete architectural overview possible.

Executive Summary
Browser Use is an innovative open-source Python library designed to facilitate AI-driven web
automation. Its core capability lies in empowering Large Language Models (LLMs) to interact
with web browsers in a manner akin to human users. The framework is characterized by a
modular architecture, a fundamental reliance on the Playwright automation library, and broad
compatibility with a diverse range of LLMs. The project has garnered substantial attention within
the developer community, evidenced by over 60,000 GitHub stars and significant financial
backing, including support from Y Combinator and over $17 million in seed funding. However, a
critical security vulnerability identified in its companion Web UI necessitates careful
consideration for deployment and operational security.

1. Introduction to Browser Use

1.1. Purpose and Core Capabilities
Browser Use is an open-source initiative engineered to enable AI-powered agents to interact
seamlessly with web browsers. Its fundamental objective is to render websites accessible to AI
agents, allowing them to programmatically control and interact with browser functionalities. The
primary operational flow involves extracting interactive elements from web pages, subsequently
enabling AI to navigate, populate forms, activate buttons, and execute intricate web workflows.
The overarching vision articulated by the project is to empower users to "Tell your computer
what to do, and it gets it done".
This framework addresses a significant challenge in the domain of web automation. Traditional
approaches, often relying on tools like Playwright or Selenium directly, necessitate explicit,
line-by-line scripting for every interaction. This creates a rigid and often brittle automation
pipeline, highly susceptible to breakage with minor changes in web page structure.
Concurrently, Large Language Models have demonstrated exceptional capabilities in
comprehending natural language and formulating complex plans. Browser Use strategically
bridges these two domains: LLMs provide the high-level intent and strategic decision-making,
which Browser Use then translates into precise, executable browser actions. This integration
effectively closes the gap between the advanced reasoning capabilities of AI and the dynamic,
often unpredictable nature of the modern web. The implication of this design is profound:
Browser Use emerges as a pivotal technology for developing truly autonomous web agents. It
signifies a shift from mere web scraping to intelligent, adaptive interaction, potentially paving the
way for future user interfaces where complex web tasks are accomplished through natural
language commands rather than explicit graphical user interface manipulations.

1.2. Key Features and Advantages

Browser Use incorporates several design elements that contribute to its efficacy and growing
adoption:
●​ AI-Powered Decision Making: The system integrates AI-driven decision-making
processes, which allows it to adapt effectively to dynamic web page structures. This
capability is a result of combining advanced AI techniques with robust browser automation
primitives.
●​ Simplified Automation Workflow: A core advantage is its ability to eliminate the need
for extensive, complex scripting, thereby making web automation more intuitive and
powerful for developers. This simplification is a direct benefit for rapid development and
deployment of automation tasks.
●​ Multilingual Versatility: While the core library is predominantly implemented in Python
(accounting for 95.1% of the codebase, with JavaScript making up 3.7%) , Browser Use
supports integration with multiple programming languages, including Python, JavaScript
(Node.js), TypeScript, Go, and Rust. This broad language support enhances its flexibility
for diverse development environments.
●​ Enhanced Robustness: The framework features a "self-healing mechanism" and a
built-in error handling and automatic recovery system. This is a crucial aspect for
real-world applications where web pages can change unexpectedly. When a traditional
automation script encounters a change in an element's identifier or position, it typically
fails. Browser Use's self-healing capability implies that its AI agent does not rely solely on
static selectors. Instead, it can re-evaluate the current state of the page, infer the new
location or characteristic of the desired element, and adapt its subsequent actions. This
resilience is paramount for maintaining continuous operation in dynamic production
environments where web user interfaces are frequently updated. The ability to
automatically recover from unforeseen errors significantly reduces the maintenance
overhead typically associated with web automation scripts, making Browser Use highly
valuable for business-critical applications such as quality assurance testing, data
extraction, and complex workflow automation where the stability of the UI cannot be
guaranteed.
●​ Advanced Operational Capabilities: Beyond basic interactions, Browser Use offers
sophisticated features such including the fusion of vision-based input with HTML
extraction, multi-tab management, precise element tracking, and compatibility with a wide
array of Large Language Models. Furthermore, it supports the definition and execution of
custom actions, such as saving data to files, writing to databases, or sending notifications.
●​ Market Traction and Financial Endorsement: The project has rapidly gained significant
traction within the IT community, accumulating over 60,000 GitHub stars within a few
months. This widespread adoption is further underscored by substantial financial backing,
including support from Y Combinator and over $17 million in seed funding.
2. Overall Project Structure and Ecosystem
2.1. Repository Layout and Key Directories
The browser-use GitHub repository is meticulously structured to support the development and
deployment of AI agents capable of controlling web browsers. The organization of its top-level
directories provides a clear indication of its functional modularity and development priorities.
●​ .cursor/rules: This directory likely contains specific rules or configurations tailored for the
Cursor Integrated Development Environment (IDE), suggesting adherence to particular
coding conventions or linting standards within the project.
●​ .github: As a standard GitHub directory, this typically houses Continuous
Integration/Continuous Deployment (CI/CD) workflows, such as GitHub Actions, along
with issue templates and other repository-level configurations. Its presence signifies a
commitment to automated testing, continuous integration, and streamlined development
practices.
●​ bin: This directory is conventionally used for executable scripts or binaries, which might
include utility scripts for development or deployment.
●​ browser_use: This is the central source code directory for the browser-use Python
library. It encapsulates the core implementations of the AI agent logic, mechanisms for
browser interaction, DOM extraction functionalities, and the integration points for Large
Language Models.
●​ docker: Containing Docker-related files, including Dockerfile and Dockerfile.fast, this
directory indicates that the project supports containerization. This enables users to run the
browser-use agent in a consistent and isolated environment, simplifying deployment and
ensuring reproducibility across different systems.
●​ docs: This directory is specifically allocated for project documentation. The README.md
explicitly mentions that contributions to this folder are welcomed, highlighting the project's
emphasis on comprehensive and accessible documentation.
●​ eval: This directory likely contains code or scripts dedicated to evaluating the
performance and robustness of the AI agents and their browser interactions. This
suggests a data-driven approach to improving agent reliability.
●​ examples: This crucial folder provides various demonstrations and practical use cases for
the browser-use library, illustrating how it can be applied to different automation tasks.
These examples serve as valuable starting points for new users.
●​ static: Typically, this directory holds static assets such as images, CSS files, or
JavaScript files, which might be utilized by the project's Web UI or other auxiliary
components.
●​ tests: This directory is dedicated to housing unit tests, integration tests, and other
testing-related files. The README.md specifically highlights the tests/agent_tasks/
subdirectory for adding tasks for CI validation, underscoring a strong focus on ensuring
the robustness and reliability of the agent's operations.
Key files at the repository root further define the project's configuration and metadata:
●​ .env.example: This file provides a template for the .env file structure, detailing the
environment variables required for configuring API keys for various LLM providers. This
streamlines the setup process for users.
●​ pyproject.toml: As a modern standard for Python project configuration, this file manages
 dependencies (e.g., playwright, dotenv, openai, anthropic), defines the build system, and
contains project metadata. It serves a similar function to requirements.txt in older Python
projects, centralizing dependency management.
●​ LICENSE: This file specifies that the project is released under the MIT License, defining
the terms of its open-source distribution.
●​ README.md: This serves as the primary introduction to the project, offering a quick start
guide, showcasing demos, outlining the project's vision and roadmap, and providing
guidelines for contributions.
The project's structured approach, particularly the presence of Docker files and detailed local
setup instructions , indicates a deliberate focus on reproducibility and a positive developer
experience. The adoption of pyproject.toml for dependency management aligns with
contemporary Python ecosystem best practices. Furthermore, the provision of an .env.example
file simplifies the often cumbersome process of API key configuration for AI-driven projects. This
comprehensive emphasis on clear setup procedures, robust dependency management, and
containerization suggests that the project prioritizes lowering the barrier to entry for new users
and fostering broader community engagement, which is vital for the sustained growth and
adoption of an open-source framework.
Table 1: Key Project Directories and Their Functions
Directory Function
.cursor/rules Configuration for Cursor IDE, enforcing coding
standards.
.github CI/CD workflows (GitHub Actions), issue
templates, and repository configurations.
bin Contains executable scripts or binaries.
browser_use Core source code for the browser-use library,
including AI agent logic, browser interaction,
DOM extraction, and LLM integration.
docker Dockerfiles for containerization, ensuring
consistent and isolated environments.
docs Project documentation.
eval Code and scripts for evaluating agent
performance and robustness.
examples Demonstrations and practical use cases of the
library.
static Static assets for Web UI or other components.
tests Unit and integration tests, including agent task
validation for CI.
2.2. Core Dependencies and Environment Setup
The operational foundation of Browser Use is built upon several key dependencies and requires
a specific environment configuration to function effectively.
●​ Python: The project mandates Python version 3.11 or higher for its execution. This
requirement ensures compatibility with modern Python features and libraries.
●​ Playwright: Browser Use's capability for browser automation is fundamentally reliant on
Playwright. Installation typically involves pip install playwright followed by playwright install
chromium to set up the necessary browser binaries. Playwright is a robust framework
known for its ability to control various browsers (Chromium, Firefox, WebKit) and its
 modern API for interacting with web elements. The strategic decision to utilize Playwright
is a critical architectural choice that underpins the project's stated goals of robustness and
adaptability. Unlike older automation tools, Playwright's architecture is well-suited for
handling complex, single-page applications (SPAs) and dynamic web content, which are
prevalent across the modern internet. Its reliability in element selection and interaction
directly contributes to Browser Use's "self-healing" capabilities and its ability to operate
effectively on intricate websites. This choice enables Browser Use to deliver on its
promise of adaptable web automation, making it significantly more resilient to common
web development patterns that often cause simpler automation tools to fail.
●​ LLM Client Libraries: Integration with various Large Language Models is a cornerstone
of Browser Use. This necessitates the installation of specific client libraries, such as
langchain_openai or langchain_ollama, depending on the chosen LLM provider.
●​ dotenv: For secure and convenient management of sensitive information like API keys,
Browser Use utilizes the dotenv library. This allows users to store API keys in a .env file,
which is then loaded into the application's environment variables. This is a standard
security practice to prevent hardcoding credentials directly into the codebase.
●​ uv: The project explicitly recommends uv for Python environment management and
efficient dependency installation. uv is a modern tool designed for speed and reliability in
managing Python packages.
●​ lxml_html_clean: A notable dependency issue has been documented where the
lxml.html.clean module, a component for HTML cleaning and parsing, was refactored into
a separate project. This required an explicit installation of lxml_html_clean to resolve
runtime errors. This scenario exemplifies the inherent challenges in managing
dependencies within rapidly evolving software ecosystems, where changes in upstream
libraries can necessitate immediate adjustments in downstream projects.

2.3. Development Practices (CI/CD, Testing)

The development lifecycle of Browser Use is characterized by a strong emphasis on automated
quality assurance and continuous integration, reflected in its repository structure and
documented practices.
The presence of the .github directory signifies the implementation of Continuous
Integration/Continuous Deployment (CI/CD) workflows, likely powered by GitHub Actions. This
indicates that code changes are automatically built, tested, and potentially deployed upon
commit, ensuring a rapid feedback loop for developers.
A particularly strong focus is placed on automated testing, as evidenced by the dedicated tests
directory and specific mentions of tests/agent_tasks/. The project encourages users to
contribute their specific automation tasks as YAML files within this directory. These user-defined
tasks are then automatically executed and evaluated against predefined criteria whenever
updates are pushed to the main repository. This proactive quality assurance mechanism directly
addresses a fundamental challenge in AI agent development: maintaining reliability in dynamic
environments. AI agents, especially those interacting with external systems like web browsers,
are inherently susceptible to subtle failures caused by changes in LLM behavior, evolving web
UI designs, or updates to underlying libraries. By integrating user-contributed tasks into the CI
pipeline, the project shifts the burden of continuous validation from individual users to the core
development team. This leverages a diverse set of real-world scenarios to collectively improve
the overall robustness of the agent. This practice fosters a more stable and trustworthy library,
as potential regressions or breaking changes are identified and rectified early in the
development cycle, benefiting the entire user base. It also serves to encourage community
contribution by providing a clear and impactful pathway for users to ensure their specific
automation needs remain functional and reliable.
The existence of an eval directory further suggests the use of dedicated evaluation frameworks
or scripts to systematically measure the performance and robustness of the AI agents. This
commitment to quantitative evaluation underscores a data-driven approach to improving the
system's capabilities. Additionally, the use of .pre-commit-config.yaml indicates the
implementation of pre-commit hooks, which automate code quality checks (e.g., linting,
formatting) before commits are finalized. This practice enforces code consistency and reduces
the likelihood of introducing common errors, contributing to overall code health and
maintainability.

3. Core Architectural Components: A Deep Dive

The browser-use library is architected as a collection of interconnected Python modules, each
responsible for a distinct aspect of the AI-driven web automation process. The primary
components work in concert to translate high-level natural language instructions into concrete
browser actions.
Table 2: Core Architectural Components and Their Interdependencies
Component Primary Function Key Interdependencies
Agent Orchestrates the overall LLM, BrowserContext,
AI-driven automation workflow, Controller, DomService
interprets tasks, and plans
actions.
Browser Manages Playwright browser Playwright, BrowserConfig,
instances and contexts. BrowserContextConfig
BrowserContext Provides an isolated browsing Browser,
environment within a Browser BrowserContextConfig
instance.
DomService Extracts and processes the Playwright Page,
Document Object Model (DOM) buildDomTree.js, DOMState,
to identify interactive elements SelectorMap
and represent page state.
Controller Registers and executes specific BrowserContext, ActionModel,
browser actions based on ActionResult
agent's commands.
LLM (e.g., ChatOpenAI) Provides the large language External LLM APIs,
model capabilities for SystemPrompt
understanding tasks, planning,
and generating actions.
3.1. The Agent Module: Orchestrating AI-Driven Automation
The Agent module serves as the central orchestrator of the entire AI-driven automation workflow
within Browser Use. It is the primary interface through which users define tasks and initiate
automated browser interactions. The Agent class is exposed directly through the browser_use
package.
The Agent's core responsibilities include:
 ●​ Task Interpretation: Receiving and understanding the user's high-level task, expressed
in natural language.
●​ Action Planning: Interacting with the Large Language Model (LLM) to generate a
sequence of browser actions required to fulfill the given task. This involves translating the
LLM's output into structured commands that the Controller can execute.
●​ State Management: Maintaining a representation of the browser's current state, including
the Document Object Model (DOM) and potentially a history of past actions, to inform
subsequent decisions. The AgentHistoryList is imported, suggesting a mechanism for
tracking the agent's historical interactions.
●​ Workflow Execution: Iteratively interacting with the browser through the BrowserContext
and Controller to perform the planned actions. The Agent typically runs asynchronously,
as indicated by the asyncio usage in examples.
The interaction flow of the Agent with other components is critical:
1.​ Initialization: An Agent instance is initialized with a specific task and an llm instance
(e.g., ChatOpenAI). It can also be configured with an existing BrowserContext for more
granular control or parallelization.
2.​ LLM Interaction: The Agent sends the current browser state (obtained via DomService)
and the task to the configured LLM. The SystemPrompt is imported, indicating that the
Agent likely uses a predefined system prompt to guide the LLM's behavior and instruct it
on available tools/actions.
3.​ Action Generation: The LLM, based on the prompt and current state, proposes the next
action (represented by ActionModel ) to achieve the task.
4.​ Action Execution: The Agent delegates the execution of this action to the Controller.
5.​ State Update: After an action is performed, the Agent receives an ActionResult , updates
its internal state, and potentially requests a new DOM state from the DomService to
prepare for the next action.
The roadmap for the Agent module includes significant enhancements such as improving agent
memory to handle over 100 steps, enhancing planning capabilities by loading website-specific
context, and reducing token consumption by optimizing the system prompt and DOM state
representation. These planned improvements highlight the project's commitment to making the
AI agents more efficient, capable, and cost-effective in complex, multi-step web interactions.
The ability to handle longer sequences of actions and leverage contextual information more
effectively will directly contribute to the framework's capacity for truly autonomous and
sophisticated web automation.

3.2. Browser Interaction Layer: Leveraging Playwright

The browser interaction layer is primarily managed by the Browser and BrowserContext
components, which abstract the complexities of direct browser manipulation through the
Playwright framework. The Browser and BrowserConfig classes are exposed at the top level of
the browser_use package.
●​ Browser Control and Configuration (Browser and BrowserConfig): The Browser
class is responsible for launching and managing browser instances (e.g., Chromium,
Firefox, WebKit) via Playwright. It can be initialized with a BrowserConfig object, which
allows for detailed configuration of the browser's behavior. Key configuration options
include:
○​ headless: A boolean flag determining whether the browser runs in headless mode
(without a visible UI) or with a graphical interface. Running in headless mode is
 common for server-side automation, while a visible UI can be useful for debugging
or human-in-the-loop scenarios.
○​ disable_security: A boolean flag to disable browser security features. This can be
particularly useful when dealing with cross-origin requests or iframes that might
otherwise be blocked by browser security policies. However, disabling security
features introduces significant risks, as discussed in the security considerations
section.
○​ extra_chromium_args: Allows passing additional command-line arguments directly
to the Chromium browser instance.
○​ wss_url / cdp_url: Enables connecting to an existing browser instance via
WebSocket or Chrome DevTools Protocol (CDP).
○​ keep_open: Determines whether the browser instance remains open after the script
finishes execution.
○​ cookies_file: Specifies a path to a cookies file for persistence across sessions,
useful for maintaining login states.
●​ Context Management and Parallelization (BrowserContext and
BrowserContextConfig): The BrowserContext represents an isolated browsing
environment within a Browser instance. Each context has its own cookies, storage, and
cache, providing a clean slate for each automation task or agent. This isolation is crucial
for running multiple agents or tasks concurrently without interference. The
BrowserContextConfig allows for further customization of the browsing environment,
including:
○​ wait_for_network_idle_page_load_time: Time to wait for network requests to finish
before considering the page loaded.
○​ browser_window_size: Defines the viewport dimensions for the browser window.
○​ locale: Sets the browser's locale (e.g., 'en-US').
○​ highlight_elements: A debugging feature to visually highlight elements during
automation.
○​ viewport_expansion: Adjusts the viewport size for capturing more content.
The design choice to leverage Playwright is a strategic architectural decision. Playwright's
asynchronous API necessitates the use of Python's asyncio library for managing concurrent
operations. This asynchronous nature allows Browser Use to efficiently manage multiple
browser interactions without blocking the main execution thread, which is vital for performance
in complex automation scenarios. Furthermore, the explicit support for multiple BrowserContext
instances per Browser instance facilitates parallelization. This means that instead of running
multiple browser instances (which can be resource-intensive), Browser Use can create multiple
isolated contexts within a single browser, allowing for the parallel execution of similar tasks,
such as finding contact information for numerous companies concurrently. This capability
significantly enhances the scalability and efficiency of the framework for high-volume
automation.

3.3. DOM Extraction and Understanding: Bridging Web Content to AI

The DomService module is responsible for extracting and processing the Document Object
Model (DOM) of a web page, translating it into a structured representation that Large Language
Models can comprehend and act upon. The DomService and its associated view models
(DOMBaseNode, DOMElementNode, DOMState, DOMTextNode, SelectorMap) are exposed by
the browser_use package.
 ●​ Mechanism for Element Identification (DomService): The DomService class is
initialized with a Playwright Page object, which serves as its interface to the live web
page. Its primary method, get_clickable_elements, is designed to identify interactive
elements on the page. This method can optionally highlight these elements visually in the
browser and focus on a specific element.A critical architectural detail is that the actual
DOM tree building logic is offloaded to a separate JavaScript file named buildDomTree.js.
This JavaScript code is executed within the browser's context by Playwright. This design
choice represents a clear separation of concerns and an optimization for performance.
Python orchestrates the high-level agent logic and interacts with LLMs, while JavaScript
handles the low-level, performance-critical task of extracting real-time DOM state directly
within the browser's environment. JavaScript, running natively in the browser, has direct
and highly optimized access to the browser's DOM APIs. This is significantly more
efficient for complex, real-time DOM traversal and state extraction compared to purely
remote Python commands, which would involve a bridge to the browser for every
interaction. Therefore, by delegating DOM tree construction to JavaScript, Browser Use
ensures that the most efficient tool is utilized for the most performance-sensitive part of
web interaction. This hybrid approach allows Browser Use to be highly performant and
adaptable to dynamic web pages, as the in-browser JavaScript can capture complex UI
states more effectively.
●​ State Representation for LLM Comprehension: The JavaScript component traverses
the live DOM, identifies elements based on criteria (e.g., links, buttons, input fields,
elements with event handlers), extracts relevant attributes (tag name, text content,
visibility, position, size), and constructs a structured representation of the DOM. This
structured data is then returned to the Python DomService. The dom/views.py file defines
the data structures used for this representation, including DOMBaseNode,
DOMElementNode, DOMTextNode, and DOMState. The DOMState object encapsulates
the entire DOM state, including the element tree and a SelectorMap, which likely provides
mappings of selectors (e.g., CSS selectors, XPath expressions) to specific DOM elements
for subsequent interactions. The DomService also maintains an xpath_cache, suggesting
that it may generate or utilize XPath expressions for element location and cache them for
improved performance.The project's roadmap includes plans to enhance DOM extraction
capabilities by enabling detection for all possible UI elements and improving the state
representation to ensure that all LLMs can better understand the page content. These
improvements are crucial for the LLM's ability to make accurate decisions and generate
precise actions, especially on visually complex or non-standard web interfaces.

3.4. Action Orchestration and Customization: The Controller

The Controller module is responsible for orchestrating and executing the specific browser
actions dictated by the AI agent. It acts as an intermediary, translating the LLM's high-level
action commands into concrete Playwright operations. The Controller class is exposed at the
top level of the browser_use package.
●​ Registry of Actions: The Controller maintains an internal registry of supported browser
actions. Upon initialization, it registers a set of default browser actions. These default
actions cover common web interactions, including:
○​ ClickElementAction: Clicking on a specific web element.
○​ DoneAction: Signaling the completion of a task.
○​ ExtractPageContentAction: Extracting textual content from the current page.
 ○​ GoToUrlAction: Navigating to a specified URL.
○​ InputTextAction: Typing text into an input field.
○​ OpenTabAction: Opening a new browser tab.
○​ ScrollAction: Scrolling the page.
○​ SearchGoogleAction: Performing a Google search.
○​ SendKeysAction: Sending keyboard inputs.
○​ SwitchTabAction: Switching between open browser tabs.
A key feature of the Controller is its extensibility: users can register custom actions that
the AI agent can then leverage. This is achieved by decorating Python functions (either
synchronous or asynchronous) with @controller.action(). Custom actions can accept
parameters, optionally defined using Pydantic models for structured input, and can even
gain access to the Browser instance if requires_browser=True is specified. This
mechanism allows developers to extend the agent's capabilities beyond predefined
browser interactions, enabling it to perform application-specific logic, interact with external
APIs, or integrate with other systems (e.g., saving job details to a database, as shown in
an example ). This extensibility is vital for adapting Browser Use to a wide range of
specialized automation tasks, making it a highly flexible framework.
●​ Execution Flow of AI-Generated Commands: When the Agent determines the next
action, it passes an ActionModel (which represents the LLM's chosen action and its
parameters) to the Controller. The Controller then looks up the corresponding registered
function and executes it, leveraging the BrowserContext to interact with the web page.
The result of the action is encapsulated in an ActionResult. The Controller also
incorporates utility functions like time_execution_async and time_execution_sync,
suggesting performance monitoring capabilities for actions. This modular design ensures
a clear separation between the AI's planning and the actual execution of browser
operations, contributing to the system's maintainability and debuggability.

3.5. Large Language Model (LLM) Integration: The Brain of the Agent
The LLM integration layer is fundamental to Browser Use, as it provides the "intelligence" that
enables the agent to understand tasks, plan actions, and adapt to web environments. The llm
subdirectory within browser_use is dedicated to housing the implementations for various LLM
integrations. While direct access to the raw content of browser_use/llm/ and
browser_use/llm/openai.py was not available, their usage is extensively documented through
examples and other snippets.
●​ Supported LLM Providers and API Key Management: Browser Use offers broad
compatibility with a variety of leading LLM providers, allowing users to choose the model
best suited for their needs and budget. Supported providers include:
○​ OpenAI (e.g., gpt-4o)
○​ Anthropic (e.g., Claude 3.5 Sonnet)
○​ Google (e.g., Gemini 2.0)
○​ DeepSeek (e.g., DeepSeek-V3, deepseek-r1)
○​ Grok
○​ Novita
○​ Ollama (for local models like qwen2.5)
○​ Qianfan (for Ernie 4.0)
API keys for these providers are managed through environment variables, typically loaded from
a .env file using the dotenv library. This is a standard and recommended practice for handling
sensitive credentials in development and deployment environments.
Table 3: Supported LLM Providers and Configuration
LLM Provider Example Models / Libraries Configuration Method
OpenAI gpt-4o (via OPENAI_API_KEY in .env
langchain_openai.ChatOpenAI)
Anthropic Claude 3.5 Sonnet (via ANTHROPIC_API_KEY in .env
ChatAnthropic)
Google Gemini 2.0 GOOGLE_API_KEY in .env
DeepSeek DeepSeek-V3, deepseek-r1 DEEPSEEK_API_KEY in .env
Grok (Specific model not specified) GROK_API_KEY in .env
Novita (Specific model not specified) NOVITA_API_KEY in .env
Ollama qwen2.5 (via OLLAMA_API_BASE in .env
langchain_ollama.ChatOllama)
Qianfan ernie-4.0-turbo-128k (via QIANFAN_API_KEY,
ChatQianfan) QIANFAN_API_BASE in .env
●​ Prompting Strategies and Vision Integration: The Agent module utilizes a
SystemPrompt to guide the LLM's behavior, instructing it on how to interpret the web page
state and what actions are available. This prompt engineering is crucial for enabling the
LLM to effectively reason about the web environment. Browser Use also supports vision
capabilities, combining visual input with HTML extraction to provide a richer
understanding of the web page to the LLM. This vision integration is particularly valuable
for complex layouts or when visual cues are more important than raw HTML structure for
decision-making. The Web UI, for instance, allows users to disable vision if not needed, or
to configure models like DeepSeek with specific settings for local execution (e.g.,
unchecking "Use Vision" for Ollama-hosted models). The roadmap indicates a continuous
effort to enhance planning capabilities by loading website-specific context and reducing
token consumption by optimizing the system prompt and DOM state representation. This
focus on prompt efficiency and contextual awareness is vital for improving both the
performance and cost-effectiveness of LLM interactions.

4. Usage Patterns and Advanced Capabilities

Browser Use is designed for ease of use, offering multiple interaction patterns and advanced
features to cater to various automation needs.

4.1. Quick Start and Basic Automation Examples

Getting started with Browser Use is straightforward. After installing the Python package via pip
(requires Python >= 3.11) and setting up Playwright browsers, users can quickly spin up an
agent. A basic Python example demonstrates the core workflow: initializing an Agent with a task
and an llm, then calling its run() method. For instance, automating a login involves initializing the
agent, opening a web page, typing username and password into input fields, and clicking a
submit button. The framework provides numerous examples in its examples directory,
showcasing diverse use cases such as adding grocery items to a cart, adding LinkedIn followers
to Salesforce leads, finding and applying for machine learning jobs, writing letters in Google
Docs, and searching for models on Hugging Face. These examples serve as practical templates
for new users to adapt for their specific automation requirements.

4.2. Interactive CLI and Web UI for Agent Interaction

Beyond programmatic scripting, Browser Use offers alternative interfaces for interacting with the
AI agent:
●​ Interactive CLI: Users can install a command-line interface (CLI) version of Browser Use
(pip install "browser-use[cli]") which provides an interactive environment similar to other AI
code assistants. This allows for direct, conversational interaction with the agent.
●​ Web UI and Desktop App: A companion project, browser-use/web-ui, provides a
graphical user interface built on Gradio. This Web UI enables users to interact with their
browser-use agent conversationally directly from their browser. Key features of the Web
UI include the ability to provide human intervention when necessary (e.g., solving
CAPTCHAs, guiding complex decisions, or handling unexpected situations), facilitating
seamless human-agent collaboration. It also supports deep research capabilities,
collaborative agents, indexed information sources, and offers enhanced features like
video display and real-time page display when the browser is running in headless mode.
The Web UI supports a wide range of LLMs and allows for custom browser usage,
including persistent browser sessions to avoid re-logging into sites. A desktop application
is also available for testing.

4.3. Custom Actions and Workflow Definition

Browser Use is designed to be highly extensible. As detailed in the Controller section, users can
define and register custom actions that extend the agent's capabilities beyond standard browser
interactions. This allows for integration with external systems, execution of custom logic, or
interaction with specific application features not covered by default actions. The ability to pass
structured parameters to these custom actions using Pydantic models further enhances their
utility and maintainability.
The project's roadmap also outlines plans to allow users to record workflows, which can then be
re-run by Browser Use as a fallback mechanism, even if the underlying web pages change. This
feature aims to make automation more robust against UI changes and simplify the creation of
complex, multi-step tasks. The vision includes creating various templates for common workflows
such as tutorial execution, job applications, QA testing, and social media automation, which
users can readily copy and adapt. This emphasis on workflow reusability and templating
significantly reduces the effort required to set up and maintain complex automation scenarios.

5. Design Principles and Future Roadmap

5.1. Underlying Design Philosophies
While Browser Use does not explicitly list its own "design principles" in the provided snippets
(unlike general web design principles found in ), its architecture and stated goals reflect several
implicit philosophies:
●​ AI-Centricity: The core design revolves around empowering AI agents, specifically LLMs,
to interact with the web. This means prioritizing the LLM's understanding of the web
environment and its ability to make decisions.
 ●​ Modularity and Separation of Concerns: The clear division into Agent, Browser,
DomService, Controller, and LLM modules demonstrates a commitment to modular
design. This allows for independent development, testing, and maintenance of each
component, improving overall system robustness and flexibility.
●​ Robustness and Adaptability: Features like the "self-healing mechanism" and built-in
error recovery highlight a design philosophy focused on creating resilient automation that
can handle the dynamic and often unpredictable nature of the web. The choice of
Playwright as the underlying browser automation tool further supports this, given its
modern capabilities for handling complex web applications.
●​ Developer Experience: The provision of quick start guides, comprehensive examples,
clear dependency management (pyproject.toml, uv), and containerization support
(docker) indicates a strong emphasis on making the library easy to set up, use, and
contribute to.
●​ Extensibility: The ability to register custom actions and integrate with various LLMs
showcases a design that encourages users to extend the framework's capabilities for
specific use cases.

5.2. Planned Enhancements and Strategic Vision

The project's roadmap outlines ambitious plans for future development, focusing on improving
the core capabilities of the AI agent and enhancing the user experience :
●​ Agent Improvements:
○​ Memory Enhancement: Plans include improving agent memory to handle
workflows exceeding 100 steps. This will involve techniques such as
summarization, compression, and Retrieval-Augmented Generation (RAG) to
manage long interaction histories efficiently.
○​ Enhanced Planning: Future work aims to improve the agent's planning capabilities
by enabling it to load website-specific context, allowing for more informed and
strategic decision-making.
○​ Token Consumption Reduction: Efforts will focus on optimizing the system
prompt and DOM state representation to reduce token consumption by LLMs,
thereby lowering operational costs and improving efficiency.
●​ DOM Extraction Refinements:
○​ Broader UI Element Detection: The project intends to enable detection for all
possible UI elements, including complex ones like date pickers and dropdowns.
○​ Improved State Representation: Work will continue on improving the
representation of UI elements to ensure that all LLMs can accurately understand
what is present on the page.
●​ Workflow Capabilities:
○​ Record and Rerun Workflows: A significant planned feature is the ability for users
to record workflows, which can then be re-run by Browser Use even if the
underlying web pages change. This will provide a robust fallback mechanism for
automation.
○​ Workflow Templates: The creation of various templates for common tasks (e.g.,
tutorials, job applications, QA testing, social media) will allow users to quickly copy
and adapt pre-defined automation flows.
●​ User Experience (UX) Enhancements:
○​ Improved Documentation: A continuous effort to improve documentation is
 planned.
○​ Speed Optimization: General improvements to the speed of the framework are a
priority.
○​ Human-in-the-Loop Execution: Enhancing the ability for human intervention
during agent execution is also on the roadmap.
●​ Parallelization: The project recognizes that the true power of a browser agent lies in its
ability to parallelize similar tasks. The vision includes enabling parallel execution of tasks
(e.g., finding contact information for 100 companies concurrently), with results being
processed by a main agent that can then kick off further parallel subtasks. This capability
is crucial for scaling automation to large datasets or high-volume operations.
●​ Datasets and Benchmarking: Plans include creating datasets for complex tasks,
benchmarking various models against each other, and fine-tuning models for specific
automation scenarios. This indicates a commitment to empirical validation and continuous
improvement of the AI agents' performance.
These roadmap items collectively demonstrate a strategic vision for Browser Use to evolve into
a more intelligent, robust, efficient, and user-friendly platform for autonomous web agents. The
focus on memory, planning, and parallelization suggests a trajectory towards handling
increasingly complex and high-volume automation challenges.

6. Security Considerations
While Browser Use offers powerful automation capabilities, it is imperative to address certain
security considerations, particularly concerning its companion Web UI and default browser
configurations.

6.1. Identified Vulnerabilities

A critical security vulnerability has been identified in the browser-use/web-ui project, which
serves as a web application for running browser-use agents. This vulnerability stems from the
web-ui's use of Python's pickle module for serializing and deserializing settings. Python's pickle
module is inherently insecure when deserializing data from untrusted sources, as it can execute
arbitrary code during the deserialization process. This means that an attacker could craft a
malicious pickle file, and if a user or system loads this file through the web-ui's configuration tab,
arbitrary code could be executed on the server hosting the web-ui. This constitutes a Remote
Code Execution (RCE) vulnerability.
While this vulnerability is present in the web-ui companion project rather than the core
browser-use library itself, its impact on browser-use deployments is significant. The web-ui is a
primary interface for many users to interact with browser-use agents. A successful RCE exploit
could lead to the compromise of the host server, potentially exposing sensitive information such
as LLM API keys stored in the .env file, or allowing the attacker to gain control over the system
running the agent. This risk is exacerbated by the fact that dozens of internet-facing web-ui
instances have been identified, making them publicly reachable targets for attackers. Even for
privately run instances, if an attacker could somehow upload a malicious pickle file, the
consequences would be severe.

6.2. Default Browser Security Configuration

Another important security aspect is the default browser configuration used by Browser Use.
The framework, when launching Chromium via Playwright, by default includes the
--disable-web-security command-line flag. This flag significantly weakens the browser's native
security protections. Specifically, it disables critical security features such as the Same-Origin
Policy (SOP), which normally prevents web pages from interacting with content from different
origins. Disabling web security can make the browser instance, and by extension the
browser-use agent, more vulnerable to various web-based attacks, including Cross-Site
Scripting (XSS), Cross-Origin Resource Sharing (CORS) bypasses, and other malicious script
injections if the agent navigates to a compromised or malicious website.
This default setting highlights a tension between functionality and security. Disabling web
security can be advantageous for certain automation scenarios, particularly when dealing with
cross-origin content or complex interactions that might otherwise be blocked. However, it
introduces a heightened risk profile. The combination of a known RCE vulnerability in the web-ui
and the default disabling of browser security features creates a critical security posture that
demands careful attention from users. An attacker exploiting the RCE could potentially leverage
the less secure browser environment for further malicious activities, or conversely, if the agent is
directed to a malicious site, the disabled security features could facilitate a more severe
compromise.

6.3. Best Practices and Mitigation

Given these considerations, users deploying Browser Use, especially with the web-ui
component, should adhere to robust security practices:
●​ Avoid Public Exposure of Web UI: Publicly accessible web-ui instances should be
avoided unless absolutely necessary and secured with strong authentication and network
access controls.
●​ Input Validation and Sanitization: If custom configurations or settings are loaded from
external files, ensure rigorous validation and sanitization of inputs to prevent
deserialization vulnerabilities. Ideally, avoid pickle for untrusted data.
●​ Isolated Environments: Deploy browser-use agents and the web-ui in isolated
environments, such as Docker containers or virtual machines, with minimal necessary
permissions. This limits the blast radius in case of a compromise.
●​ Review Browser Security Settings: Carefully evaluate the necessity of
disable_security=True in BrowserConfig. If not strictly required for the automation task,
this setting should be set to False to maintain the browser's native security protections.
●​ Regular Updates: Keep browser-use and its web-ui component updated to the latest
versions to benefit from security patches and bug fixes.
●​ Least Privilege: Configure API keys and other credentials with the principle of least
privilege, granting only the necessary permissions for the agent to perform its tasks.

7. Conclusion
Browser Use represents a significant advancement in AI-driven web automation, providing a
robust and flexible framework for empowering Large Language Models to interact with dynamic
web environments. Its modular architecture, built upon the reliable Playwright library, facilitates
the translation of high-level AI intent into precise browser actions. The project's commitment to
developer experience, evidenced by comprehensive documentation, clear dependency
management, and containerization support, contributes to its growing adoption and community
engagement.
The strategic decision to offload complex DOM traversal to in-browser JavaScript, coupled with
the "self-healing" mechanism, positions Browser Use as a resilient solution for automating tasks
on ever-changing websites. The extensive support for various LLM providers further enhances
its versatility, allowing users to integrate with a wide array of AI models.
However, the identified security vulnerability in the companion web-ui project, stemming from
insecure pickle deserialization, along with the default disabling of browser security features,
underscores the critical importance of secure deployment practices. Users must be diligent in
securing their environments and configuring browser settings appropriately to mitigate potential
risks.
The ambitious roadmap, focusing on enhancing agent memory, planning, parallelization, and
user experience, indicates a clear vision for Browser Use to evolve into an even more
sophisticated and efficient platform for autonomous web agents. As AI capabilities continue to
advance, frameworks like Browser Use will be instrumental in bridging the gap between
intelligent systems and the vast, dynamic landscape of the internet.

Works cited

1. browser_use/__init__.py at ... - Cloud Native Build (CNB),

https://cnb.cool/8888/github.com/browser-use/browser-use/-/blob/1ec222849498b707c8e11a23
29a9e871dd91471d/browser_use/__init__.py 2. browser_use/dom/service.py at ... - Cloud
Native Build (CNB),
https://cnb.cool/8888/github.com/browser-use/browser-use/-/blob/3fd3beb4d52c8078b9a4b094
5173d104777e2843/browser_use/dom/service.py 3. browser_use/dom/views.py at
3fd3beb4d52c8078b9a4b0945173d104777e2843 · 8888/github.com/browser-use/browser-use -
Cloud Native Build (CNB),
https://cnb.cool/8888/github.com/browser-use/browser-use/-/blob/3fd3beb4d52c8078b9a4b094
5173d104777e2843/browser_use/dom/views.py 4. browser_use/controller/service.py at
3fd3beb4d52c8078b9a4b0945173d104777e2843 · 8888/github.com/browser-use/browser-use -
Cloud Native Build (CNB),
https://cnb.cool/8888/github.com/browser-use/browser-use/-/blob/3fd3beb4d52c8078b9a4b094
5173d104777e2843/browser_use/controller/service.py 5. browser-use 0.1.17 - PyPI,
https://pypi.org/project/browser-use/0.1.17/ 6. Browser-use with OpenAI + Langchain for
Automating Web Browsing | by Sumit Soman,
https://medium.com/@sumit.somanchd/browser-use-with-openai-langchain-for-automating-web-
browsing-ba6db7439566 7. MilesGordenker/browser-use-wrapper: Make websites accessible
for AI agents - GitHub, https://github.com/MilesGordenker/browser-use-wrapper 8.
browser-use/examples/models/qwen.py at main - GitHub,
https://github.com/browser-use/browser-use/blob/main/examples/models/qwen.py 9.
kekee000/browser-use-play - GitHub, https://github.com/kekee000/browser-use-play 10.
browser-use/browser-use: Make websites accessible for AI agents. Automate tasks online with
ease. - GitHub, https://github.com/browser-use/browser-use 11. Exploring Browser Use Agent:
The Future of AI-Powered Web Automation - DEV Community,
https://dev.to/rajnishjaisankar/exploring-browser-use-agent-the-future-of-ai-powered-web-autom
ation-3gkd 12. Build AI Agents with browser-use and Scraping Browser - Bright Data,
https://brightdata.com/blog/ai/browser-use-with-scraping-browser 13. Getting RCE on
browser-use/web-ui AI Agent Instances - Kudelski Security Research,
https://research.kudelskisecurity.com/2025/04/23/getting-rce-on-browser-use-web-ui-ai-agent-in

15. browser-use/web-ui: 🖥️
stances/ 14. Local Setup - Browser Use, https://docs.browser-use.com/development/local-setup
Run AI Agent in your browser. - GitHub,
https://github.com/browser-use/web-ui 16. Releases · browser-use/web-ui - GitHub,
https://github.com/browser-use/web-ui/releases 17. Runtime issue with sample code in the
README. #496 - GitHub, https://github.com/browser-use/browser-use/issues/496 18.
browser.py:23: SyntaxWarning: invalid escape sequence '\ ' · Issue #478 - GitHub,
https://github.com/browser-use/browser-use/issues/478 19. The best library for LLM to use a
browser. Browser use tutorial in python - YouTube,
https://www.youtube.com/watch?v=gtglgiG2iwo 20. Design Principles - Mozilla Protocol,
https://protocol.mozilla.org/docs/fundamentals/principles 21. 9 Principles of Good Web Design -
read our guidelines to consider - Feelingpeaky,
https://www.feelingpeaky.com/9-principles-of-good-web-design/