Usage

Vision mode (default)

Run the agent with screenshot-based visual input. This is the default and requires a GPT-4V capable model.

conda activate webvoyager

nohup python -u run.py \
    --test_file ./data/tasks_test.jsonl \
    --api_key YOUR_OPENAI_API_KEY \
    --headless \
    --max_iter 15 \
    --max_attached_imgs 3 \
    --temperature 1 \
    --fix_box_color \
    --seed 42 > test_tasks.log &

This runs in the background. Follow progress with tail -f test_tasks.log. Results (screenshots + interaction logs) are written to the results/ directory.

Text-only mode

Uses the accessibility tree instead of visual screenshots. Cheaper and faster; works with GPT-4 Turbo.

nohup python -u run.py \
    --test_file ./data/tasks_test.jsonl \
    --api_key YOUR_OPENAI_API_KEY \
    --headless \
    --max_iter 15 \
    --max_attached_imgs 1 \
    --temperature 1 \
    --text_only \
    --api_model gpt-4-1106-preview \
    --seed 42 > test_tasks_text_only.log &

Key parameters

Flag	Default	Description
--test_file	required	Path to .jsonl task file
--api_key	required	OpenAI API key
--headless	off	Run Chrome without a visible window
--max_iter	15	Max browser interaction steps per task
--max_attached_imgs	3	Number of recent screenshots sent to the model
--temperature	1	Model sampling temperature
--text_only	off	Use accessibility tree instead of screenshots
--api_model	gpt-4-vision-preview	OpenAI model name
--fix_box_color	off	Normalize bounding-box highlight color
--seed	42	Random seed for reproducibility

Task file format

Each line in the .jsonl task file is a JSON object with at least a web (start URL) and ques (natural-language instruction) field. Example:

{"web": "https://www.amazon.com", "ques": "Find the best-selling book in the Mystery category and add it to the cart."}

Evaluation

After a run completes, score the results with the automated GPT-4V evaluator:

cd evaluation
bash run_eval.sh

run_eval.sh calls auto_eval.py and writes scores to evaluation.log.