Word Error Rate broken down by speaker gender. If the two bars are similar height, the model is fair; if one is much taller, that group is being under-served.
Across all models & audio conditions, average WER is 72.7% for male speakers and 79.0% for female speakers — a gap of 6.2pp (higher (worse) for female).
| Model | Clean Studio | Ambient Room Noise | Harsh Environment | Distant / Low-Volume Mic | Wideband Phone (VoLTE) | Randomized Phone Channel | Narrowband 2G + Packet Loss | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| M | F | Δ | M | F | Δ | M | F | Δ | M | F | Δ | M | F | Δ | M | F | Δ | M | F | Δ | |
| Ethio-ASR Amharic (600M) | 35.2% | 37.3% | +2.1 | 37.0% | 39.8% | +2.9 | 39.5% | 43.7% | +4.2 | 35.4% | 37.4% | +2.0 | 40.0% | 48.4% | +8.4 | 41.2% | 50.1% | +8.9 | 67.1% | 79.6% | +12.5 |
| Ethio-ASR Multi (300M) | 58.8% | 65.3% | +6.5 | 62.8% | 72.2% | +9.4 | 69.8% | 81.6% | +11.8 | 58.9% | 65.4% | +6.4 | 74.0% | 92.9% | +18.9 | 76.9% | 95.4% | +18.5 | 123.1% | 131.7% | +8.6 |
| Ethio-ASR Multi (600M) | 46.6% | 49.2% | +2.6 | 48.9% | 53.4% | +4.4 | 52.3% | 58.4% | +6.0 | 46.8% | 49.5% | +2.7 | 53.3% | 65.2% | +11.9 | 54.7% | 66.5% | +11.7 | 83.4% | 93.5% | +10.1 |
| Ethio-ASR Multi (94M) | 55.8% | 61.2% | +5.4 | 58.8% | 65.5% | +6.7 | 63.5% | 71.9% | +8.3 | 55.9% | 61.2% | +5.4 | 66.1% | 78.7% | +12.6 | 68.2% | 80.6% | +12.4 | 93.3% | 99.3% | +6.0 |
| Meta MMS-1B | 100.5% | 100.7% | +0.2 | 100.1% | 100.6% | +0.4 | 100.1% | 100.6% | +0.5 | 100.5% | 100.7% | +0.2 | 100.4% | 101.0% | +0.6 | 100.4% | 101.3% | +0.9 | 103.2% | 105.9% | +2.8 |
| OmniASR-CTC (1B) | 41.7% | 44.9% | +3.2 | 61.3% | 62.2% | +0.9 | 62.3% | 65.1% | +2.8 | 41.5% | 44.3% | +2.8 | 62.2% | 67.2% | +5.0 | 63.1% | 68.9% | +5.8 | 88.6% | 94.0% | +5.4 |
| OmniASR-CTC (300M) | 52.0% | 55.5% | +3.4 | 63.8% | 66.7% | +2.9 | 65.6% | 70.0% | +4.4 | 52.2% | 55.6% | +3.3 | 67.0% | 75.3% | +8.3 | 68.7% | 77.3% | +8.6 | 93.5% | 97.1% | +3.6 |
| OmniASR-CTC (3B) | 31.9% | 34.5% | +2.6 | 56.8% | 56.9% | +0.1 | 56.2% | 57.4% | +1.3 | 31.7% | 34.3% | +2.6 | 57.6% | 58.2% | +0.6 | 58.6% | 60.7% | +2.2 | 69.4% | 84.2% | +14.8 |
| OmniASR-LLM (1B) | 32.4% | 34.2% | +1.8 | 78.0% | 75.4% | -2.6 | 78.0% | 71.9% | -6.2 | 32.5% | 33.9% | +1.4 | 79.5% | 78.4% | -1.1 | 79.4% | 77.7% | -1.7 | 82.9% | 100.0% | +17.1 |
| OmniASR-LLM (300M) | 42.4% | 42.3% | -0.1 | 81.8% | 83.3% | +1.5 | 83.6% | 86.1% | +2.5 | 46.5% | 47.3% | +0.8 | 81.3% | 81.7% | +0.4 | 81.4% | 81.8% | +0.4 | 88.2% | 99.2% | +11.0 |
| OmniASR-LLM (3B) | 26.6% | 27.8% | +1.2 | 78.2% | 72.5% | -5.7 | 76.4% | 65.1% | -11.3 | 26.3% | 27.9% | +1.6 | 67.7% | 57.4% | -10.3 | 65.2% | 59.7% | -5.5 | 82.4% | 96.7% | +14.3 |
| OpenAI Whisper-small | 145.2% | 172.4% | +27.2 | 148.8% | 173.4% | +24.6 | 157.2% | 177.9% | +20.8 | 149.0% | 168.3% | +19.4 | 138.3% | 165.8% | +27.5 | 148.0% | 165.0% | +17.1 | 215.1% | 272.5% | +57.4 |