One sentence, two languages: building voice AI for English and Swahili

Listen to how people actually talk in Nairobi and you'll hear this constantly: "Niko busy right now, I'll call you baadaye." One sentence, two languages, no warning. Sometimes a third — Sheng, the urban mix that's neither textbook English nor textbook Swahili.

Now look at how voice systems are built. ASR is set to a language. TTS picks a voice for a language. The LLM is prompted in a language. The whole pipeline assumes a caller picks one and stays there. They don't. And the moment they don't, things start to break in ways that are easy to miss from a desk in a city where everyone speaks the same one.

This is the part of building for a Kenyan audience that nobody warns you about. Here's where it bites and how I approach it.

ASR breaks first

Speech-to-text is where the cracks show up earliest. Most ASR models are trained and tuned per language. Point one at "English" and the Swahili words in a mixed sentence come back as garbage — best-guess English that sounds nothing like what was said. Point it at "Swahili" and the English half suffers.

Utterance-level language detection doesn't save you either, because the two languages aren't in separate utterances — they're in the same breath. There's no clean point to switch.

On top of that, Swahili is lower-resource than English. Less training data means worse accuracy, full stop. And "Swahili" isn't one thing: textbook Kiswahili sanifu, coastal Swahili, and the Kenyan colloquial people actually speak on the phone are not the same, and a model trained on the formal version will stumble on the casual one. Add names, places, and Sheng and the transcript quality drops right where your callers are most natural.

If you only test on clean, single-language, textbook sentences, you'll think it works. Real calls will tell you otherwise.

The model speaks textbook Swahili, not street

The LLM side has a subtler problem. These models are far more fluent in English than in Swahili — there's just more English in the world. So Swahili output tends to come back stiff and over-formal, the kind of Kiswahili sanifu you'd read in an exam, not how anyone talks to you on a call. It also drifts: ask for Swahili and the model keeps leaking back into English, especially on technical or financial terms.

So you can't just translate your English prompt and call it bilingual. You have to be explicit about register — that the bot should sound like a normal person on the phone, mirror the caller's mix, and not lecture them in formal Swahili when they spoke to it in half-English. Getting that tone right is more prompt work than the English version ever needs.

TTS: fewer voices, awkward in the seams

Speech synthesis has its own gaps. There are far fewer good Swahili voices than English ones, and the ones that exist can mangle the seams — a sentence that switches languages mid-way, an English brand name dropped into Swahili, a Kenyan name the voice has never seen. Numbers are their own trap: money (Ksh), phone numbers, and dates all get spoken differently in mixed speech, and a voice that reads them the "English" way sounds wrong.

None of this is fatal, but it's the difference between a bot that sounds local and one that sounds like it was built somewhere else and shipped here.

How I'd build it

A few principles I'd hold to:

Don't make the caller pick a language. Any flow that starts with "Press 1 for English, 2 for Kiswahili" has already lost, because the honest answer is "both, mixed." Design for mixed input from the first turn.

Mirror the caller. Whatever blend they use, answer in roughly the same blend. People relax when the bot talks the way they do. A system that insists on pure Swahili to a code-switching caller feels as off as one that insists on pure English.

Keep a domain glossary in both languages. The handful of terms that matter — loan, balance, payment, due date, the product names — should be pinned down in both, so the model and the TTS handle them consistently instead of improvising.

Test on real speech, not textbook sentences. This is the big one. The gap between how Swahili is written in training data and how it's spoken on a Kenyan phone call is where systems fail. You only find that gap by listening to real calls.

Respect register and politeness. Swahili carries respect markers that English doesn't — how you'd address an elder, the weight of a greeting. On a collections or service call especially, getting the register wrong reads as rude in a way an English-only builder wouldn't notice.

The point

Building voice AI for Kenya isn't English-with-a-Swahili-option. It's accepting that your callers will mix languages in a single sentence, that your ASR, LLM, and TTS were mostly built for one of those languages, and that the work is closing that gap on purpose. Speaking both languages yourself helps — you can hear what's wrong. But you still have to design for the mix from the start, because the tools won't do it for you.

---

Neville James Achieng builds LLM and voice systems in Nairobi. github.com/Neville777.