Thesis: Can AI Agents Replace the Tester? Comparing LLM Browser Agents and Scripted End-to-End Tests

High Level Description

End-to-end (E2E) tests verify that a web application works the way a user experiences it. They are valuable but expensive:

scripted tests (e.g., Playwright, Cypress) take time to write, break when the user interface changes, and only find the bugs

someone thought of in advance. A new alternative is LLM-based agents that control a real browser. They read the screen, click,

type and judge whether the application behaves correctly, much like a human exploratory tester. This thesis will compare AI-driven

exploratory testing with traditional scripted E2E tests on open-source web applications with deliberately planted bugs, to find

out which approach finds more bugs, how reliable, and at what cost.

Cost is a central question: AI agents are cheap to set up but every test run consumes paid model usage, while scripted tests are

expensive to write and maintain but cheap to run.

Project Description

The work will involve setting up one or two open-source web applications as test subjects, preferably from a regulated domain

where workflows, user roles and data integrity matter.

To measure how well each testing approach finds bugs, the bugs must be known in advance. A catalogue of 30–50 realistic faults

will therefore be built and injected into the applications in a controlled, repeatable way. The faults will cover:

The AI agents will be built by connecting large language models (LLMs) to a real web browser, so they can navigate, click, type

and read the screen like a human tester. At least two different LLMs will be compared, including both a large, more capable model

and a smaller, cheaper one, to see how model choice affects bug detection and cost.

Cost will be tracked for every approach throughout the work. For the agents this includes model usage (tokens and API cost), run

time and the number of repeated runs needed for reliable results. For scripted tests it includes the time to write the tests, the

time to update them after UI changes, and the compute time to run them. Human time spent reviewing agent reports and sorting out

false alarms will also be counted.

Purpose

In this thesis investigate these questions:

• False alarms – How often do agents report issues that are not real bugs, and how much human effort does it take to sort them

out?

• Stability & reproducibility – How much do results vary across repeated runs, and can a reported bug be reliably reproduced from

the agent's report?

• Resilience to change – After realistic UI changes (renamed buttons, moved menus, new layouts), which approaches still work, and

what does maintenance cost?

• Cost per run & per bug – What does a single test run cost for each approach, and what is the cost per bug found? How much can

be saved with smaller models, shorter sessions or limiting exploration, and how much bug detection is lost?

• Total cost over time – Compare the total cost of each approach over a realistic period, including setup, running, maintenance

and human review. When does each approach break even, and how does the cost change if tests run on every code change rather

than nightly or before each release?

An Exciting Journey with Knightec Group

Semcon and Knightec have joined forces as Knightec Group. Together, we are Northern Europe’s leading strategic partner in

product and digital service development. With a unique combination of cross-functional expertise and a holistic business

understanding, we help our clients realize their strategies – from idea to complete solution.

Practical Information

This is a thesis position, located at our office in Sundsvall. Start date January or March 2027.

Please submit your application as soon as possible, but no later than 2026-11-30. If you have any questions, you are welcome to

contact johanna.edstrom@knightec.se. Note that due to GDPR, we only accept applications through our careers page.