Own Words

Ця сторінка англійською.

Блог · 2026-09-25

We tested drafts, not reputations

Every week a new model is "the best ever" at something. As a translator you have no way to check whether it's the best at your books. Neither did we. So we measured.

The question

Not "which model is smartest" or "which tops a leaderboard". Which one writes a first draft closest to what a real literary translator actually wrote: closest to her, on a novel she was translating.

The test

More than a dozen models, from the big names to the small, cheap ones. The same instructions for all of them: the ones Own Words actually sends, with her past paragraphs as examples, the run-up, and the notes on how she translates.

Her own paragraphs as the answer key, from work she had done before the tool existed, so no model had a head start from her having edited its earlier draft. Dialogue and description, short and long.

No peeking. For every draft we checked that her translation of that paragraph, and everything after it, was not in what the model saw.

The score: how much of the draft her text changes. Lower is closer. It's a blunt measure, since a legitimate alternative counts the same as a mistake, but it's the same for every model.

Chance first. The same model, drafting the same paragraph twice, already varies noticeably. Any "win" smaller than that is the model rolling dice, and we didn't count it.

The result

No model wrote measurably closer to her than the modest one we use by default. The two that came nearest were within chance, and cost many times more per draft. Several were measurably further from her.

We also tried two even cheaper models. They were measurably worse, and one made errors of meaning, not taste: "stubble" became a stair, "awkward" became deft. That is the kind of mistake a translator has to catch and can't unsee. Saving a fraction of a cent isn't worth one of those.

Two things we didn't expect

You keep what lands in front of you. When we replayed a real evening of her work, the model that had drafted those paragraphs at the time "won" by a mile. Not because it was better, but because her final text was an edit of its draft. That's why the fair test is work done without the tool.

The near-miss from a neighbouring language. Every translator knows the form that is nearly right and simply wrong in her language. We went looking for it across about fifteen hundred drafts and found a handful. None came from the default.

What we did with it

The default stays. You can switch models any time. And because a word count can't measure literary judgement, the final word goes to you: on real work, Own Words sometimes shows two drafts side by side, unnamed, in random order, and a change only wins when translators clearly prefer it.

We use large language models. We'd just rather know which one, and why, than take a leaderboard's word for it.


The whole picture: How Own Words works. Or see what it drafts on your book: the first ten pages are free.