Two industry-standard measures across every model and every audio-quality condition. Numbers come from the full evaluation set (thousands of clips), not just the 20 example sentences.
Percentage of words the model got wrong. Lower is better. A WER of 20% means roughly 1 in every 5 words is incorrect.
| Model | Clean Studio | Ambient Room Noise | Harsh Environment | Distant / Low-Volume Mic | Wideband Phone (VoLTE) | Randomized Phone Channel | Narrowband 2G + Packet Loss | Average |
|---|---|---|---|---|---|---|---|---|
| Ethio-ASR Amharic (600M) | 36.5% | 38.7% | 42.1% | 36.6% | 45.1% | 46.7% | 74.7% | 45.8% |
| Ethio-ASR Multi (300M) | 62.8% | 68.6% | 77.1% | 62.9% | 85.6% | 88.3% | 128.4% | 81.9% |
| Ethio-ASR Multi (600M) | 48.2% | 51.6% | 56.0% | 48.5% | 60.6% | 61.9% | 89.6% | 59.5% |
| Ethio-ASR Multi (94M) | 59.2% | 62.9% | 68.6% | 59.2% | 73.8% | 75.8% | 97.0% | 70.9% |
| Meta MMS-1B | 100.6% | 100.4% | 100.4% | 100.7% | 100.8% | 101.0% | 104.9% | 101.2% |
| OmniASR-CTC (1B) | 43.7% | 61.8% | 64.0% | 43.2% | 65.3% | 66.7% | 91.9% | 62.4% |
| OmniASR-CTC (300M) | 54.1% | 65.6% | 68.3% | 54.3% | 72.1% | 74.0% | 95.7% | 69.2% |
| OmniASR-CTC (3B) | 33.5% | 56.9% | 56.9% | 33.3% | 57.9% | 59.9% | 78.5% | 53.8% |
| OmniASR-LLM (1B) | 33.5% | 76.4% | 74.2% | 33.4% | 78.9% | 78.4% | 93.4% | 66.9% |
| OmniASR-LLM (300M) | 42.4% | 82.7% | 85.2% | 47.0% | 81.6% | 81.6% | 95.0% | 73.6% |
| OmniASR-LLM (3B) | 27.3% | 74.7% | 69.5% | 27.3% | 61.4% | 61.9% | 91.1% | 59.0% |
| OpenAI Whisper-small | 161.8% | 163.9% | 169.9% | 160.8% | 155.1% | 158.4% | 250.2% | 174.3% |
| Average across models | 58.6% | 75.4% | 77.7% | 58.9% | 78.2% | 79.5% | 107.5% | 76.5% |
Percentage of individual characters wrong. Lower is better. CER is usually smaller than WER because a wrong word often shares some characters with the correct one.
| Model | Clean Studio | Ambient Room Noise | Harsh Environment | Distant / Low-Volume Mic | Wideband Phone (VoLTE) | Randomized Phone Channel | Narrowband 2G + Packet Loss | Average |
|---|---|---|---|---|---|---|---|---|
| Ethio-ASR Amharic (600M) | 13.4% | 14.7% | 17.0% | 13.4% | 19.2% | 20.8% | 50.2% | 21.2% |
| Ethio-ASR Multi (300M) | 27.5% | 32.2% | 40.5% | 27.5% | 47.7% | 50.5% | 107.7% | 47.7% |
| Ethio-ASR Multi (600M) | 21.7% | 24.3% | 27.8% | 21.9% | 32.6% | 34.2% | 68.6% | 33.0% |
| Ethio-ASR Multi (94M) | 25.1% | 27.6% | 31.8% | 25.1% | 36.5% | 38.1% | 71.3% | 36.5% |
| Meta MMS-1B | 81.1% | 81.8% | 82.4% | 81.1% | 83.0% | 83.1% | 89.2% | 83.1% |
| OmniASR-CTC (1B) | 14.2% | 16.2% | 19.4% | 14.4% | 22.4% | 24.1% | 78.0% | 27.0% |
| OmniASR-CTC (300M) | 18.9% | 20.4% | 24.0% | 19.0% | 28.8% | 31.3% | 76.8% | 31.3% |
| OmniASR-CTC (3B) | 11.5% | 13.6% | 15.4% | 11.3% | 15.9% | 17.8% | 61.0% | 20.9% |
| OmniASR-LLM (1B) | 11.3% | 16.1% | 17.7% | 11.4% | 20.3% | 21.2% | 60.4% | 22.6% |
| OmniASR-LLM (300M) | 14.2% | 18.9% | 21.5% | 14.7% | 23.7% | 24.9% | 63.0% | 25.9% |
| OmniASR-LLM (3B) | 10.2% | 16.0% | 17.4% | 10.3% | 18.8% | 21.0% | 64.6% | 22.6% |
| OpenAI Whisper-small | 151.7% | 152.4% | 160.0% | 150.3% | 145.9% | 149.0% | 226.1% | 162.2% |
| Average across models | 33.4% | 36.2% | 39.6% | 33.4% | 41.2% | 43.0% | 84.7% | 44.5% |
WER (Word Error Rate): out of every 100 spoken words, how many did the model get wrong? Wrong words, missing words, or extra words all count as one error each.
CER (Character Error Rate): same idea but at the character level. Useful for languages like Amharic where a small character slip doesn't necessarily change meaning.
Lower is always better. 0% would mean a perfect transcription.