Articles / Local ASR server for code-switched call centre audio

Sep 27, 2026 · Sanatel Consulting

What this is about

We built speech analytics for call centres that runs entirely on one server inside the customer's perimeter: no audio and no transcript ever leaves it. The calls are code-switched — Russian mixed with Kazakh or Uzbek — and the recordings come off a PBX at telephony quality.

Nothing off the shelf covers that combination. Whisper and local LLMs are libraries, not services: they process one file per request. Turning them into a system meant building a layer on top — a job queue, an ensemble of language passes, hallucination filtering, and a scoring function that picks the best result.

This article is about that layer. Every number and every snippet comes from a running system, including the places where the first obvious solution turned out to be wrong.

The problem: why this is not a standard ASR task

The input is a stereo recording from Asterisk: the agent on one channel, the customer on the other. That is the one thing working in our favour — speaker roles are known exactly, no diarisation needed.

Everything else works against us:

Code-switching. A single call mixes Russian with Kazakh or Uzbek. Usually the switch happens once, right after the greeting, but it also happens mid-call.
Colloquial vocabulary. Customers speak in everyday language with loanwords. Models fine-tuned on literary and news corpora degrade noticeably on it.
Telephony audio. 8 kHz narrowband, 64 kbps joint-stereo MP3, agents speaking into a handset rather than a headset. This is about the worst input on which Whisper is still usable.

Cloud providers handle Russian acceptably and Kazakh and Uzbek considerably worse — quite apart from the question of sending recordings outside at all.

Why on-premise only

Three reasons. The legal one is the hardest, though not the one that comes up most often in sales conversations.

Law. In Kazakhstan, Law No. 94-V on personal data and its protection requires personal data to be stored in a database located on the territory of the republic (art. 12 (2)); cross-border transfer is governed separately and needs its own grounds. For banks there is a second layer — banking secrecy, under which client information is disclosed to third parties only with the client's written consent.

In Uzbekistan the picture changed in 2026. Law No. ZRU-1125 of 26 March 2026 restated art. 27¹ of Law No. ZRU-547: mandatory localisation now covers a closed list — biometric data, genetic data, and data on subscribers of telecom operators. Other categories may be processed abroad subject to statutory conditions, but the implementing regulations for those conditions had not been adopted at the time of writing.

So in Uzbekistan the blunt argument that cloud processing is illegal no longer holds for everyone. A different one does: the conditions for cross-border transfer are not yet spelled out, and a local deployment simply removes the company from that uncertainty.

Trust. Independently of the law, mid-size and large buyers state the requirement directly: recordings must not leave the perimeter. The standard pre-sales ask is a dedicated server on company premises, or at least a rack of their own in someone else's data centre. Shared cloud is usually off the table entirely.

Cost of ownership. Cloud ASR bills per minute. At thousands of calls a day, one GPU pays for itself quickly and makes the budget predictable.

Architecture: a layer on top of two engines

Three processes live on the server: Python/FastAPI with faster-whisper, Ollama with Qwen2.5, and our own ASR Service in TypeScript (Node.js, Fastify). The last one is the system; the first two are libraries without it.

What it does:

Exposes a single HTTP API with token auth and JSON-schema validation. Several internal systems call it, not one script.
Keeps an in-memory job queue and moves every job through a fixed state machine.
Prepares audio: checks whether original mono WAV files are available or if stereo MP3 must be used. ffmpeg splits stereo into two mono channels and resamples to 16 kHz; files are validated for size and duration.
Assembles the result, picks the winner, writes logs, and serves a status page: jobs per state, recognition time, GPU temperature, variant scores, speech coverage.
Cleans up after itself: channel files are deleted right after processing, jobs expire after 24 hours.

Job states:

draft → new → wave_ready → winner → refine → done → error

Transitions are driven by three independent switcher loops — one each for audio preparation, recognition and LLM calls. Each runs on its own interval and is guarded against re-entry:


let gIntervalRunning = false;

async function asrStateSwitcher(): Promise<void> {
  if (gIntervalRunning) return;   // only one instance at a time
  gIntervalRunning = true;

  // refine -> done, winner -> refine, wave_ready -> winner
  // each block takes the oldest job in its state

  gIntervalRunning = false;
}

That is the entire concurrency story in single-threaded Node: while the ASR loop waits on a network call to Python, the ffmpeg loop prepares channels for the next call. No workers, no broker — at thousands of calls a day that would be overkill.

The recognition calls themselves are strictly sequential: the Python service is synchronous and returns 429 when busy. Within a job the passes run one after another; parallelising them on a single GPU buys nothing.

Three passes instead of one

In auto-detect mode Whisper picks one language for the whole segment. On code-switched speech it misfires, sometimes spectacularly: on Uzbek content our logs show the detector returning English on one call and Bashkir on another. Forcing the language removes that error — but only where the segment really is monolingual.

So every channel is transcribed three times: one automatic pass over the language pair and two forced passes. The pair is a job parameter; the pipeline itself does not depend on it:


/** 'uz,ru' -> ['uz,ru', 'uz', 'ru'];  'kk,ru' -> ['kk,ru', 'kk', 'ru'] */
export function getLanguageCandidateSettings(languages: string): string[] {
  const langs = languages.split(',').map(s => s.trim()).filter(Boolean);
  return [languages, ...langs];
}

The first element is the whole string, which on the Python side means "do not force a language". The rest are forced passes. Uzbekistan gets uz,ru → auto + uz + ru; Kazakhstan gets kk,ru → auto + kk + ru.

That is six ASR calls per conversation: three passes across two channels. A deliberate trade — on code-switched audio the quality gain outweighs the time.

One side effect worth knowing: with a forced language Whisper does not run detection at all. The language field in the response equals whatever you passed in, and language_probability is always 1.0. Meaningful detection exists only on the auto pass, so you cannot build winner selection on language confidence.

None of this is state of the art. The research path for code-switching is architectural: an encoder refiner for intra-sentential switching, language-aware decoder adapters. Ours is an engineering compromise — acceptable quality with no fine-tuning and no research team.

Model selection: one base, one specialist

The base model is Whisper large-v3 via faster-whisper. It handles the auto pass and Russian, and on Russian we are satisfied with it. On Uzbek we are not.

Selection criteria for the specialist were simple: Whisper architecture or convertible to CTranslate2 (otherwise it will not load into our engine), and quality on our own recordings rather than on public benchmarks.

We ran three models over the same real calls, forced Uzbek pass. Here is the one where large-v3 essentially collapsed:

Metric / large-v3 / Model A / Model B
Segments: 8 / 4 / 5
Speech coverage, s: 13.0 / 34.7 / 110.7
Coverage / call duration: 0.095 / 0.255 / 0.814
Low-confidence ratio: 0.875 / 0.000 / 0.000
Weighted confidence: 57.3 / 71.9 / 94.3

Model B recovered 81 % of the call as coherent speech where the base model managed 9.5 %. The text is readable throughout: a ticket refund, a booking number, an explanation about a third-party travel agency.

Weighted confidence across three calls:

Call / large-v3 / Model A / Model B
#1 (138.5 s): 69.5 / 78.9 / 90.3
#2 (136.0 s): 57.3 / 71.9 / 94.3
#3 (239.9 s): 66.8 / 77.2 / 93.8

Model A is an Uzbek fine-tune of large-v3-turbo; model B is a fine-tuned Whisper-medium converted to CTranslate2. The gap is consistent — 12 to 23 points on every call. What settled it: on all three recordings our hallucination filters rejected zero segments from model B, while rejecting segments from the other two routinely.

One detail worth flagging: a coverage ratio above 1.0 is not a bug. Coverage is summed across both channels and divided by the call duration; when both parties talk a lot, the sum exceeds the length of the recording.

Both models will not fit in 16 GB of VRAM alongside the LLM, so the specialist does not run as a second server — it is hot-swapped inside the same process between requests. Measured swap time on a warm cache is about 0.9 seconds, roughly 5 % of the processing time for one call.

Hallucinations: six classes we had to catch by hand

On silence, noise and speech fragments Whisper produces plausible text that was never spoken. Without a cleanup layer it lands in the dialogue, inflates the metrics and breaks winner selection. All six classes below were found on real recordings, not taken from documentation.

1. Prompt echo. The model repeats the initial prompt verbatim as if it were a line of speech. A compound prompt on the auto pass echoes in fragments, so we also compare against its individual sentences.

2. Known artefact phrases. Traces of subtitle contamination in the training data: "Subtitles by…", "To be continued", "Subscribe to the channel", "Thank you for watching", the Turkish "Altyazı". Match on the stable part, not the whole phrase — a variant with a different verb slips through otherwise.

3. Foreign writing system. On noise the decoder wanders into a random script. Cyrillic and Latin cover all our languages; if other characters exceed 30 %, the segment is dropped. The filter is project-wide rather than per-language: Cyrillic alone does not distinguish Russian from Kazakh or Uzbek.

4. Cross-segment loops. The most common class: one phrase repeated 13–15 times in a row. Detectable only across the whole channel, so filtering takes the full segment array rather than one segment at a time. Thresholds differ for long and short phrases — an agent may legitimately repeat a question twice, and "yes", "ok", "thanks" repeat constantly.

5. In-segment repetition. Junk like qaqqaqqaqqaq… is caught by compressibility. Whisper computes the same ratio internally, but it is wired into a temperature retry loop that does not run when temperature is a single value. So we duplicate it independently:


const COMPRESSION_RATIO_THRESHOLD = 2.4;

function isCompressionAnomaly(text: string): boolean {
  const buf = Buffer.from(text, 'utf-8');
  if (buf.length === 0) return false;
  return buf.length / deflateSync(buf).length > COMPRESSION_RATIO_THRESHOLD;
}

6. Implausible duration. Not a filter but a clamp. Whisper regularly assigns tens of seconds to a short line — in one case a six-word sentence spanned 114 seconds. The text may well be real and only the duration fake, so we keep the segment but compute metrics from a plausible speaking time at 12 characters per second.

A principle worth keeping: the raw recognition output is never modified. Filtering affects only the assembled dialogue and the metrics, so any disputed case can be audited against the original data.

One class we do not catch yet, and we know it: a grammatically correct, non-repeating filler phrase in another language ("thank you so much for joining us today" over silence). Not a loop, not on the blocklist, ordinary Latin script. The cheap fix suggests itself — it carried 40-plus characters in 0.18 seconds, so the text could not physically have been spoken. A symmetric check for "duration too short for this much text" closes the class without touching anything else.

Scoring: how the winner is chosen

This is where we spent the most time, and where the first obvious answer is wrong.

The obvious answer is to take the variant with the highest mean confidence. It fails for two reasons. First, avg_logprob on hallucinations is regularly higher than on real speech — the model decodes invented text more confidently than a barely audible line. Second, a variant that transcribed a quarter of the call and lost the rest looks excellent by mean confidence.

We had a call where mean-confidence selection preferred a variant that had lost roughly 40 % of the content. After that the formula took its current shape:

score = weightedConfidence × (coverage / coverage_max) × (segments / segments_max)

Term by term:

Weighted confidence. Each segment's avg_logprob is mapped onto a 0–100 scale and averaged weighted by speech duration, not by segment count. Otherwise a dozen short junk lines outweigh one long real utterance.
Relative speech coverage. How many seconds of speech the variant recovered, normalised against the best variant of the same call. This is the protection against losing content.
Relative segment count. A correction for granularity: a variant that lumped the call into a few long blocks gains no advantage over a detailed one.


function computeScores(variants: AsrAnalyzedVariant[]): Record<string, number> {
  const maxCoverage = Math.max(...variants.map(v => v.coverageSeconds), 1e-9);
  const maxSegmentCount = Math.max(...variants.map(v => v.segmentCount), 1);

  const scores: Record<string, number> = {};
  for (const v of variants) {
    const coverageRatio = v.coverageSeconds / maxCoverage;
    const segmentCountRatio = v.segmentCount / maxSegmentCount;
    scores[v.languageSetting] = Number(
      (v.weightedConfidence * coverageRatio * segmentCountRatio).toFixed(4),
    );
  }
  return scores;
}

And the limitation to understand from the outset: this is a ranking function, not an absolute quality measure. Both relative terms are normalised against the best variant of the same job, so for the winner they are almost always 1.0 and the score collapses back into mean confidence, with all of its problems. You cannot use the score as a "did this transcribe or not" threshold. That needs a separate metric.

Dialogue assembly and the rejection threshold

The winning variant becomes a dialogue: cleaned segments from both channels are merged onto one timeline and sorted by start time. The role comes from the channel — which one is the agent is stated in the job. No diarisation involved.

Timestamps are emitted as they came. The duration clamp from filter 6 affects metrics only; adjusting the timestamps too would misalign the dialogue.

The dialogue is assembled for the winner alone. Losing variants keep their metrics — useful material for auditing disputed calls and for tuning the formula later.

The external quality gate is separate and simple:


if (job.analysis.winnerSpeechCoverageRatio >= 0.28) {
  result.state = 'done';
  result.dialogue = job.analysis.winnerDialogue;
} else {
  result.state = 'error';
  result.comment = 'job bad: low Coverage Ratio!';
}

If the winner covered less than 28 % of the recording with speech, the job is marked as not transcribed. That call never reaches semantic analysis and never reaches reporting. Returning an honest "could not transcribe" beats handing over fragments that someone will use to judge an agent's performance.

Semantic analysis on a local LLM

A transcript on its own answers no business question. The substantive output comes from Qwen2.5 via Ollama on the same server: a summary and outcome of the call, a checklist review (greeting, needs discovery, objection handling, agreed next step), customer needs and reasons for refusal, and an agent assessment against consistent criteria.

Qwen was chosen for its handling of Russian, Kazakh and Uzbek. The 7B version is in production; 14B is the alternative when more VRAM is available.

The response is requested against a JSON schema through Ollama's format parameter, so the result loads into analytics without regex parsing.

One decision we made and will not revisit: we do not run LLM post-correction over transcripts. The idea sounds reasonable — ask the model to fix obvious ASR typos. In practice a local model's context will not hold a full dialogue as JSON, and on chunks it starts rewriting phrasing and collapsing turns. Validating "array length before and after" catches half of those responses, but the risk of a silent meaning change remains. We invested in transcription quality instead of cosmetics on top of it.

A second practical decision: only dialogues longer than twenty turns go to semantic analysis. Short ones — misdials, transfers, "call me tomorrow" — yield nothing substantive while consuming the same GPU time as everything else.

Hardware and throughput

The current rig: NVIDIA RTX 5060 Ti 16 GB, Ubuntu 24.04 under KVM. 16 GB of VRAM is the practical minimum, because the recognition model and the analysis LLM sit on the card at the same time. With less you get a choice: a smaller LLM with weaker output, or unloading one model before loading the other, paying the latency on every call.

Now the throughput, and here is the trap that is easy to fall into.

It is tempting to assume that since ASR and the LLM are separate processes, they run in parallel and daily throughput follows the larger of the two times. They do not. Without MPS or MIG, two CUDA contexts on one card are not executed simultaneously — the driver time-slices between them. Both processes are alive and working their queues, but the compute is divided, not added.

So the arithmetic is additive:

calls per day = 86400 / (t_ASR + t_LLM)

Our measured figure: about 30 seconds for a 2–3 minute call, covering all three passes across both channels plus the engine swap. LLM time depends on checklist size and generated length; at 30 seconds per dialogue the estimate lands at roughly 2,500 - 3,000 calls per day under round-the-clock load.

Two things improve that. First, the twenty-turn threshold: some calls never reach the LLM at all, leaving only ASR time in the denominator. Second, the queue: calls arrive unevenly and are processed evenly, including overnight. For a customer who does not need real-time transcription, that keeps the card near full utilisation.

For higher volume the configuration scales to a second card or a second server, with the queue distributed across them and no architectural change.

What's next, and what we deliberately skip

Near-term work, in descending order of return on effort:

Original WAV instead of MP3. Right now we get whatever the PBX emits by default. Obtaining the original files is a configuration question on the Asterisk side. The cheapest accuracy gain available (already implemented at the time of writing).
A customer glossary in the initial prompt. Product names, models, common personal names. Days of work, no training involved.
A specialised model for Kazakh. The approach transfers unchanged — only the engine on the forced pass differs. Same selection criteria.
Symmetric duration check for the hallucination class described above.

Domain fine-tuning deserves its own note, because small teams tend to write it off as out of reach. It is not. A noticeable in-domain gain takes roughly 10–30 hours of labelled audio — 250 to 750 calls of 2–3 minutes. The labelling is not from scratch: run your own pipeline and post-edit, about 5–10× real time. That is a person-month of work, not a research project.

Why we have not done it: it only pays off for a specific customer with steady volume, on their data and inside their perimeter. Not for a pilot. This is "not yet and not for everyone", not "beyond us".

What we will not do: LLM post-correction of transcripts (reasons above), and chasing academic state of the art in code-switching. A multi-pass ensemble with forced languages is a pragmatic engineering compromise, and we describe it as exactly that.