The Stranger in the Women's App

It is past midnight and I am watching a stranger use my app.

She is not real. She is a piece of software I have asked to behave like a particular kind of person: a woman travelling on her own for the first time, nervous about the basics, the sort who reads everything before she taps anything. She is signed into an ordinary account on the live app, the same one anyone would get from the App Store. She taps, she waits, she takes a screenshot, she decides what someone like her would do next. I am not steering. I am reading what she writes.

The app is called Sola, and it is built only for women who travel solo. That is the whole bet. A woman lands somewhere she has never been, alone, and she opens this instead of a forum or a group chat, because it is supposed to know the things she actually needs and the women she might actually want to meet. The screen I care about most is the one where she can find other women who happen to be in the same city and arrange to see them, for a coffee, a walk, a shared taxi to the airport at five in the morning. Right above the list of people there is a line of text that promises, in so many words, a community of women. Everything below that line is supposed to be true to it.

She reads the promise. Then she stops.

She writes that she is confused. Directly above the line about a community of women there is a profile. The name on it is Mathieu. The photo is empty, a grey rectangle where a face should be. She does not know it is wrong because she read the code, because she has never seen the code and never will. She knows it is wrong the way any woman opening that screen would know it is wrong. There is a man here, in the one place there are not supposed to be men.

I have looked at that screen more times than I can count.

“

I built that screen. I never saw him.

Here is the thing about a promise like that. It does not live in any one place in the code that I could point to and guard. It is spread across a sign-up flow, a database, a sort order, a query that runs when the screen loads, three or four functions written on different days by a version of me in a different mood. Every one of those pieces can be individually correct and the promise can still break, quietly, with nobody lying and nothing throwing an error. The tests pass. The type checker is clean. A second engineer reads the change and approves it. All of it can be green while a man sits on the screen meant only for women. That is the stuff that hides in plain sight from the person who made it, because he knows what it is supposed to say, so he reads what he meant instead of what is there.

For a long time the only thing that could stand where she stands was an actual woman, recruited, scheduled, paid, watched over a screen-share while I held my breath. That is slow and it is precious, and you do not get to run it forty times a week. So most of what ships, ships on faith, on the quiet assumption that because the parts are correct the whole keeps its word. I had been running on that faith for months.

The fix came from the margin of an email about something else.

I was getting ready to talk to one of my mentors, and I had asked Clémence, my cofounder at the time, to help me write the questions I wanted to put to him. She sent the questions back. Then, almost as a footnote underneath them, sitting in the margin of an email that had nothing to do with the mentor, she floated a different idea entirely: what if we used AI agents to run our user testing? She had even sketched four of them. She left it there, sideways, the way you mention a thing you are not sure is worth mentioning.

I tried it that same week, months ago, and it did not really work. The models of that moment would hold a character for a few taps and then quietly forget who they were, so the anxious first-timer would drift into talking like the impatient veteran without noticing. Worse, they would describe screens they had not actually looked at. The model would narrate a perfectly plausible app, fluent and confident, that was nothing like the app on my screen. I could not separate what had happened from what the machine had imagined, which is the one thing user testing has to give you. A tester that invents its own findings is worse than no tester, so I shelved it.

I came back to it last week with the same idea and a more capable model. I want to be careful here, because the easy version of this is a story about one clever model, and that misses what actually happened. The model I used that night has already been retired. The one I use now does the same job. What changed is that capability crossed a line: software got good enough to stay in character through a long, messy session, and good enough to look before it spoke. When the skeptic said a screen confused her, I could open the exact numbered screenshot and see what she saw. The narration and the reality finally matched. Once that line is behind you, which particular model you reach for matters far less than what you build around it.

What I built around it is the part worth explaining. The technique has a name people use to dismiss it, synthetic users, which just means AI standing in for the people you would otherwise recruit and interview. The fair criticism, the one I had agreed with for months, is that they make things up. Ask one to role-play a customer and give you an opinion and it will hand you a fluent, agreeable paragraph about a screen it never saw. So I changed one rule. I stopped asking them what they thought.

Instead I put each persona in front of the real, running app. Not a mockup, not a description, the actual product on a simulator, which is just an iPhone running in a window on my laptop, signed into an ordinary test account on the same backend that serves real users. Then one instruction did most of the work. Judge only from what is on the screen. Never read the code. And if a screen leaves you confused, do not push past it, because that confusion is itself the finding. The agent taps based on what it can see, waits, takes a numbered screenshot, decides what its persona would do next. The screenshots are the evidence, so a claim the pictures do not support gets caught the moment you look. The guardrails on a live backend were boring and absolute: never post anything public, never sign out, never press a button that destroys something.

My laptop has 8GB of memory and will run only two of these simulated phones at once, so the four personas went in two waves, late into the early morning. By the end I had a stack of findings, and I do not trust a single agent's word about any of them, so each serious one went to a separate set of agents whose only job was to look at the numbered screenshots and either back the claim or throw it out.

Mathieu was the one that landed in my chest, and two of the testers found him without my prompting. But the others were real damage. The tool that turns the places a woman has saved into an actual trip could not add a single place, not even ones she had saved minutes earlier, which is to say the entire point of that feature was dead. A profile rendered the literal word "undefined" where a date should have been. A filter control came up as a row of giant vertical bars. None of this is exotic. It is the ordinary, brutal kind of broken that the person who made it stops being able to see.

It was not all wounds, and the praise I had not fished for is the only kind worth anything. The testers each landed, on their own, on the same things they liked: the local knowledge written for one specific street after dark rather than a whole city at large, and reaching real content in the first minute with no quiz and no wall asking for money first. The skeptic, the one built to look for the catch, said she would cite the paragraphs. I sat with that one for a while.

The part that actually convinced me came four days later. I fixed Mathieu. Then I ran the testers again, just to confirm the fix had held.

Mathieu was still there. And now there were two of them, a second man, John, standing in the same place.

My fix had been a quiet bit of re-ranking, a sort meant to push men down and out of view. It sorted on the field that records whether a profile belongs to a woman, and that field was empty for every profile in the database. It had never been filled in, not once. So the sort had nothing to act on and changed precisely nothing, and the room still had men in it.

Sit with what had already passed by then.

Every instrument I owned was answering a different question than the one I cared about, because a test checks whether the code does what the code says, and all of mine said yes. The only thing in the building that caught it was a synthetic stranger looking at the actual screen and seeing that the promise was still broken.

That is the whole point, and it is smaller and more useful than the grand claims people tend to make about AI. Tests confirm that the code keeps its word to itself. Whether the product keeps its word to a person is a different question, and a harder one. That second word lives on the screen, in the second a woman reads "a community of women" and finds a man's name above the line, and no test I could write would ever look there. To catch it you need something that behaves like a person and judges from the outside, with no access to your intentions and no sympathy for them. The judgment, the part that makes any of this work, is all in the constraints: look only at the screen, treat your own confusion as data, and never trust one witness without a second. The model only has to be good enough to follow them, which is why I do not lose sleep over the one I used that night being retired.

None of this replaces the women. A synthetic stranger cannot tell me whether Sola feels respectful or hollow, whether it earns the space it takes up on her phone or gets deleted on the train home, whether she would tell a friend. That is the conversation I care about most, and no simulator is having it for me. What the agents do is the work nobody should be doing at one in the morning across hundreds of taps, the work that makes sure that by the time a real woman opens the app, the man is gone, the trip builder adds the place, and the date is a date.

The idea for all of it was Clémence's, written sideways in the margin of an email about something else. I almost scrolled past it.