Four AI Translators, One Editor and a Blind Test

Blog post #93

Four translation drafts converging into one edited manuscript beside anonymous voting tokens.

AI-generated illustration.

The neighbor’s grandfather wrote a novel in 1930, Der Marsch ins Chaos, about Austrian reservists marching into the First World War. It is full of Viennese and Bohemian soldier dialect. This week the librarian who helps the family decided that the dialect is too old to carry over: the book goes straight from German to Swedish, and the dialogue becomes ordinary spoken Swedish. That answer made the job clearer. It also made it harder, because now everything depends on the translation itself being very good.

So instead of trusting one model, I set up a process where four of them compete and the people who know the book get to judge them without knowing who is who.

What shipped

  • A shared brief. A one-page style guide (faithful first, keep the author’s long piling-up sentences and his shifts between past and present tense, no dialect, Swedish quotation marks) and a glossary for military terms and Austrian words: Landsturm becomes landstorm, Gefreiter becomes vicekorpral, melde gehorsamst becomes anmäler lydigast, the same phrase the Swedish translation of The Good Soldier Švejk uses.
  • Four independent translations of chapters I and II, each made from exactly the same prompt: Claude, ChatGPT, Grok and Gemini. The text is cut into numbered sections of 40 to 140 words, so every version can be compared section by section.
  • An AI editor. A fifth pass reads the German and all four Swedish versions and writes one final version. It gets the four without names, and the letters are reshuffled in every section, so it can’t recognise and favour its own model’s text.
  • A blind test page. Every section shows five versions, A to E, and the letters are shuffled per section. Readers can open the German original, click “Best”, and comment on each version. The key that maps letters to models lives only on my machine. Nobody sees it until the votes are in.
  • A scheduled email to the family and the librarian at 08:00, asking them to read, vote and comment.

What’s working

  • The four are close. All of them are good. The differences are in the details, which is exactly what a blind human reading is for: Koffer turning into “bag” instead of “trunk”, how military the drill sergeant sounds, how broken a Czech carpenter’s German should sound in Swedish.
  • The editor earns its place. Its notes for chapter I were the most useful thing in the whole run. It caught that Unsinn means “nonsense”, not “madness” (three of the four got that wrong), that da war ich schon froh means “I was actually glad” and not “I was already glad”, and it kept one word for the same gate through the whole chapter where the translators switched back and forth. It also listed what it could not decide and who should decide it.
  • Subscriptions, not API credits. All four run from the terminal on plans I already pay for: ChatGPT through codex exec, Grok through xAI’s Grok Build CLI (grok -p) on SuperGrok, Gemini through Google’s Antigravity CLI (agy -p). Claude runs the whole thing and translates as one of the four.

What’s unclear or broken

  • Gemini CLI is gone for personal accounts. Google replaced it with Antigravity CLI in June 2026. Many guides still describe the old login.
  • The Claude desktop app is an MSIX package on Windows. When Claude installed agy, Windows quietly redirected the write in AppData into the app’s private copy, so the binary existed for Claude and not for my own terminal. The fix was to move it to a normal folder in my user profile and add that to PATH. Good to know for every installer from now on.
  • Gemini 3.1 Pro disappeared after one chapter. I’m signed in with my company Workspace account, which seems to get only Antigravity’s starter quota. For chapter II only Gemini 3.8 Flash was available. So “Gemini” is a different model in the two chapters, and I have written that down next to the key.
  • No result yet. The votes start coming in today. I don’t know which model wins, or whether the editor beats all of them.

Decisions made

  • The original decides, not the majority. If three translators agree and the fourth is right, the fourth wins.
  • Votes plus comments. A vote says which version won. A comment says why, and that is what improves the style guide for the next 18 chapters.
  • The winner becomes the base translator, the others become reviewers. Whether the editor step stays depends on how often E wins.
  • No full text on the open web. The source edition is under copyright. The test page is behind the project’s access code.

Tooling & process

  • Every chapter runs the same pipeline: extract the German from Word, cut it into sections, build the prompt, run four translators in parallel, run the editor, build the blind page, deploy.
  • Next: count the votes, reveal the key, and write down what the readers’ comments teach us before chapter III.

— Stefan