Optimizing Streaming Speech Recognition on Arm® Edge Devices with ExecuTorch

When you’re speaking to a device, you expect text to appear as you talk. That makes streaming automatic speech recognition (ASR) a different challenge from batch transcription, where the model has the complete recording before it starts. Strong batch accuracy does not necessarily carry over into streaming. One cross-model ASR study illustrates this gap. Qwen3-ASR-1.7B achieved the best batch accuracy among the models tested, but its word error rate (WER) rose by 77% when evaluated with 2.4-second audio chunks. In that study’s cross-model streaming comparison (Table 5), NVIDIA® Nemotron Speech Streaming 0.6B had the lowest average streaming WER when operating over 560 ms audio chunks. The comparison makes a critical point: accuracy must be measured at the chunk cadence used in deployment. This case study takes the newer, multilingual Nemotron 3.5 from PyTorch to ExecuTorch and profiles its on-device execution on a Google Pixel 11. We ask how precision and chunk size change streaming quality, how much processing time the audio requires with one inference thread, where the workload spends its time, and which native kernels run. The goal is a practical way to evaluate accuracy and latency together. In this blog post, you will learn: How ExecuTorch connects PyTorch models to optimized Arm CPU kernels. Why streaming ASR needs its own quality evaluation, and how longer chunks improve final-transcript accuracy at the cost of higher latency. How Arm Scalable Matrix Extension 2 (SME2) reduces total ASR processing time with FP16 and INT8 quantization using per-channel weights (QC8W). How to combine model operator timings and native kernel samples to understand performance and identify remaining bottlenecks. How to choose chunk size and precision together to meet your accuracy and latency targets. From PyTorch model to CPU instructions ExecuTorch is PyTorch’s edge inference framework, with export tools and a lightweight runtime. The export process prepares the model for inference and produces a .pte file that the device runtime can load and execute. Supported operations are delegated to XNNPACK, a CPU inference library. On Android, XNNPACK can select microkernels from Arm® KleidiAI software: small, optimized routines that perform the inner matrix calculations. Operators not delegated to XNNPACK run using ExecuTorch’s portable CPU kernels. Figure 1. From model export to CPU instructions. SME2 extends Arm’s instruction set to accelerate matrix calculations, including general matrix multiplication (GEMM). In the Arm C1 CPU cluster implementation with SME2, the matrix unit is shared by cores in the CPU cluster, and KleidiAI provides microkernels that use it for supported operations. Enabling SME2 in a build makes those paths available. However, native profiling is needed to show which kernels actually ran. What one streaming call contains Nemotron 3.5 is a multilingual streaming ASR model built around an RNN-T architecture with three parts: The encoder uses FastConformer attention and convolution layers to represent the audio. The predictor uses an LSTM to represent the history of emitted tokens. The joiner uses a small feedforward layer to combine the encoder’s audio representation with the predictor’s token-history representation and predict the next token. It can also produce a blank, which advances decoding to the next audio position. The frontend converts audio into log-mel feature vectors, producing one vector for every 10 ms. The encoder reduces the number of time positions by a factor of 8, so its output has one position for every 80 ms of audio. This creates a more compact sequence representing the same audio duration. Figure 2. From waveform windows to streaming encoder positions. Lookahead is the future context available within a chunk; chunk cadence is how often fresh audio is collected for a call. At lookahead 13, a call collects 1,120 ms of audio and produces 14 encoder positions. The predictor and joiner processes all 14 valid positions, accumulating their tokens into the transcript. Lookaheads 0, 3, and 6 give 80, 320, and 560 ms chunks. Longer chunks increase the wait for the first text and subsequent updates; preprocessing and inference add time after audio collection. Choose chunk size and precision together We measured transcription accuracy on the host using the ExecuTorch runtime with its XNNPACK CPU backend. We created a local streaming adapter that uses the Hugging Face Open ASR Leaderboard evaluation harness for dataset preparation, transcript normalization, and WER scoring. We evaluated FP32, FP16, and INT8 at four chunk sizes. The INT8 specification is QC8W which uses dynamic INT8 activations and per-channel INT8 weights for eligible linear operations, with FP32 outputs. Figure 3. Streaming WER versus chunk size across four corpora. Each point on the above figure covers 14,019 utterances from LibriSpeech Other (LS Other), AMI Cleaned, Earnings22, and VoxPopuli Cleaned AA datasets. WER divides total word errors by total reference words across all four corpora. Longer chunks improve final-transcript accuracy. From 80 to 1,120 ms, FP16 WER falls from 15.2% to 11.5%, and QC8W from 15.5% to 11.7%. FP16 essentially matches FP32, differing by less than 0.03 percentage points at every chunk size, so we exclude FP32 from further performance analysis. QC8W preserves strong accuracy, adding 0.21–0.31 percentage points of WER relative to FP32. These host WER results inform precision and lookahead choices. However, it does not measure how long it takes for text to appear on the device. Test the real-time budget on an edge device We processed a 24.9-second recording on a Google Pixel 11 smartphone device using 1,120 ms chunks and one inference thread pinned to a mid-core CPU. We chose the mid-core CPU as a representative target for a sustained streaming workload on the system. We tested two builds of ExecuTorch 1.3.1, toggling SME2 support on and off. Each build retained XNNPACK and KleidiAI, and used identical model artifacts. Each configuration had two warm-ups and one measured tracing run. Totals cover processing the loaded recording from audio preprocessing through final detokenization. Figure 4. Whole-recording processing time with SME2 disabled and enabled. SME2 substantially reduces processing time: FP16 falls from 9.79 to 4.61 seconds (2.12x faster), and QC8W from 5.20 to 3.24 seconds (1.61x faster). With and without SME2, both process the recording in much less time than its audio duration. Dividing the processing time by the audio duration we calculate real-time factors of 0.19 for FP16 and 0.13 for QC8W (with SME2 on). This provides substantial processing headroom with only a single inference thread. Audio still has to arrive before it can be decoded, so this headroom does not remove the chunk-collection delay. But it leaves capacity for other CPU work between chunks and can help manage thermal load during sustained transcription. Find the bottleneck with ETDump ETDump records ExecuTorch operator events and timings. To understand where the 2.12X FP16 and 1.61X QC8W speedups come from, we grouped machine learning (ML) operators across the encoder, predictor, and joiner into categories: GEMM (blue): Matrix multiplication, mainly the encoder’s dense layers. Data movement (orange): Copies, concatenation, repeats, and embedding lookup. Elementwise (red): Activations, comparisons, and other per-value work. Convolution and normalization (green and purple): Convolution filters and normalization operators; both are relatively small here. Other (violet): Small remaining timed operations, such as tensor creation. Overhead (gray): Full-run time outside the grouped operator events; this can include runner and framework overhead. Figure 5. Operator costs and overhead behind the whole-run speedup. ETDump times are summed across the encoder, predictor, and joiner networks. Adding residual overhead gives the full processing time shown in Figure 4. GEMM drives most of the speedup. Enabling SME2 reduces GEMM time from 8.2 to 3.1 seconds for FP16 and 4.1 to 2.0 seconds for QC8W. GEMM remains the largest part of the run, though its share of total processing time falls from 84% to 67% for FP16 and 78% to 62% QC8W. Data movement takes roughly the same time for both builds, so its relative share grows as GEMM accelerates: from 8% to 17% for FP16 and 9% to 17% for QC8W. This remaining work limits the overall speedup and offers a target for further optimization. Which kernels ran? ETDump locates the expensive operator families, but does not name the native microkernels that execute them. Arm Streamline allows us to sample native call paths and measure hardware counters. On the Pixel 11, we measured 38.4% SME utilization for FP16 and 53.7% for QC8W. This indicates how much of each capture the matrix hardware was allocated; it is not a percentage of peak arithmetic throughput. We also sampled GEMM functions to identify the microkernels selected by each build. The function names reveal the sampled paths. The _sme2_mopa suffix denotes a matrix outer-product accumulation routine. NEON and I8MM are the vector instruction families used by the control paths. Precision SME2 Sampled GEMM paths and functions FP16 True KleidiAI FP16 SME2 kai_kernel_matmul_clamp_f16_f16p2vlx2_f16p2vlx2_2vlx2vl_sme2_mopa XNNPACK FP16 NEON xnn_f16_gemm_minmax_ukernel_6x16__asm_aarch64_neonfp16arith_cortex_a75 FP16 False XNNPACK FP16 NEON xnn_f16_gemm_minmax_ukernel_6x16__asm_aarch64_neonfp16arith_cortex_a75 QC8W True KleidiAI INT8 SME2, FP32 output kai_kernel_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa KleidiAI FP32 SME2 kai_kernel_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa QC8W False KleidiAI INT8 NEON I8MM, FP32 output kai_run_matmul_clamp_f32_qai8dxp4x8_qsi8cxp4x8_16x4_neon_i8mm These captures confirm that SME2-targeted KleidiAI GEMM kernels ra