Native testers checked whether an AI assistant could really finish tasks for users in several markets.
With native testers in each market running real everyday tasks, and evidence for every result. For an AI assistant, Translia's testers in Japan, Vietnam, Korea, Brazil, Thailand and other markets worked across web, mobile and desktop. Each result had a screenshot and a session ID, so failures could be sorted by cause and fixes assigned.
An international consumer AI app needed to know whether its AI assistant could actually complete tasks for users in different markets. The hard part: the assistant often said “done” when the task was not done.
Testers in Japan, Vietnam, Korea, Brazil, Thailand and other markets ran everyday tasks in their own language, such as booking home services or booking a flight.
Each task was run on web, mobile and desktop clients.
Each result came with screenshots and the session ID and was checked one by one: where the task failed, where a human had to take over, and what the model did wrong.
Because every result had evidence behind it, failures could be sorted by cause: model capability, the target website blocking the assistant, or a bug in the product. The client could assign fixes by category instead of debating anecdotes.
We run native-speaker testing in each market and hand back a report your client's team can act on.
How we work with partners →