How a voice agent follows a caller between languages mid-call

A customer picks up a reminder call in English and answers in Tamil. Two minutes later they read out their account number in English, because that is how they always say numbers. A good human agent handles this without thinking: they move to Tamil, and they do not move back because a few digits came out in English.

A voice agent has to make the same judgement on every turn, in real time. This post explains how a Graine agent decides when to change language, what it deliberately ignores, and how to set up each language so the change sounds like the same person carrying on.

Eighteen languages, eleven that follow the caller

A Graine agent can speak 18 languages, and within a single call it can follow the caller between 11 of them:

• Follows the caller mid-call: English (en), Hindi (hi), Tamil (ta), Telugu (te), Kannada (kn), Malayalam (ml), Marathi (mr), Bengali (bn), Gujarati (gu), Punjabi (pa) and Urdu (ur).
• Also spoken, best chosen per call: Assamese (as), Odia (od), French (fr), Spanish (es), Indonesian (id), Malay (ms) and Dutch (nl).

You choose which of them each agent speaks, and an agent only ever switches between its own languages.

What makes the agent switch

Two things move a call into another of the agent's languages.

• A full sentence in that language. Not a word, not a phrase: a complete thought. A caller who says a whole sentence in Telugu has told the agent which language they are comfortable in.
• An explicit ask. The caller names the language ("can you speak Tamil?"), asks for it in their own words, or answers the agent's own question about which language they would prefer.

When the call switches, the agent says its switching line in that language's voice and carries on from where the conversation was. Only the language changes.

What deliberately does not

Most of the skill is in what the agent ignores:

• A borrowed word. "Okay", "EMI", "account", "booking" inside a Hindi sentence are part of everyday Hindi. They are not a request to speak English.
• A greeting. A caller may open with "vanakkam" or "namaste" out of habit and then carry on in English. A greeting sets the tone, not the language.
• Numbers and identifiers. Account numbers, amounts, dates, PIN codes and one-time passwords are very often said in English whatever language the conversation is in. Reading out "four, seven, two, nine" never moves the call.
• A fragment. "Haan", "theek hai", "sari", a short "yes" or "no": too little to decide anything on.

Languages that sound alike also need more evidence before the agent moves. Hindi and Urdu share most of their everyday spoken vocabulary, and Hindi shares many words with Marathi and Punjabi. Between close pairs like these, the agent waits for a clearer signal, so a few shared words never flip the call.

Here is how that plays out on a call, as an illustration:

Agent (en)  Hi Priya, this is Asha about your loan.
            Is now a good time?
Caller      Haan, okay.
            → a fragment: stays in English
Caller      Is my EMI still due on the fifth?
            → English with a borrowed word: stays
Caller      (a full sentence in Tamil)
            → a full sentence: switches to Tamil
Agent (ta)  (its Tamil switching line, then the answer,
            in the Tamil voice)
Caller      (in Tamil) My account number is 4729…
            → digits in English: stays in Tamil

When to switch only on request

Every agent with more than one language has a setting for when it may change language:

• When the caller speaks one of them (the default): a full sentence in another of its languages, or a request, changes the language.
• Only when the caller asks: it changes language only when asked, by name, with a question like "can you speak Tamil?", or in answer to its own question about language.

Choose the second when the language of the call is part of what was agreed, such as a disclosure that must be read in the language on file, or when your callers mix two languages in every sentence and you would rather the agent held steady. In the dashboard it is one choice under the agent's languages; through the API it is one field:

{ "multilingual": { "language_switch_trigger": "caller_requested" } }

requested_or_auto_detected puts the default back.

Set up each language as its own

An agent that switches well does not translate itself on the fly. Each language is set up as its own:

• Voice. Each language has its own voice, chosen for how natural it sounds in that language.
• Speech recognition. Each language listens with recognition suited to it, so a Tamil caller is heard as Tamil.
• Prompt. Each language has its own prompt. Write it the way a native speaker would brief a colleague, not as a word-for-word translation of the English one.
• Name. The agent can introduce itself with a different name in each language; it fills {agent_name} in that language's switching line.
• Switching line. One short sentence, said in the new language's voice the moment the call moves into it. It may use {agent_name} and {language}.

Start the call in the right language

Following the caller is the safety net; opening in the right language is better. When your records say which language a customer prefers, send agent_language with one of the agent's language codes among the call's variables:

curl -X POST "https://api.graine.ai/v2/calls" \
  -H "Authorization: Bearer $GRAINE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"agent_id": "AGENT_ID", "to_number": "+919800000001",
       "variables": {"first_name": "Priya", "agent_language": "ta"}}'

In a campaign, the same value is a column named agent_language in your list. A few rules keep this predictable:

• Left out or empty, the call opens in the agent's default language.
• A code the agent does not speak is refused where you send it: a single call answers with an error that lists the agent's languages, and an uploaded list leaves that contact out, says why in its skipped rows, and dials the rest.
• On an agent with only one language, agent_language is an ordinary variable.

Best practices

1. Open in the customer's language when you know it, and let mid-call following handle the rest. An agent with memory on remembers the language a caller chose last time.
2. Keep each agent to the languages your callers use. Every extra language is one more voice, prompt and switching line to review.
3. Write each prompt natively, including how amounts and dates are read back, and have a native speaker read it aloud.
4. Keep the switching line to one sentence. The caller wants their answer, not an announcement.
5. Test with real speakers before going live: greetings, mixed sentences, read-out numbers and a clear request in each language.

Build it with us

If your customers speak more than one language, and most do, we would like to hear what your calls need. Write to us at hello@graine.ai, or create an account and set up your first multilingual agent today.