Bringing NAM A2 to Embedded Hardware

The Daisy Seed board has the following specs:

• ARM Cortex M7 running at 480 MHz
• 1 MB of SRAM (of which 128 kB are tightly coupled RAM for data)
• 8 MB of Flash memory (for firmware/stored models)
• 96 kHz stereo codec (we used 48 kHz mono)

The Daisy Seed has optionally an extra 64 MB of SDRAM, but that RAM is much slower than the onboard SRAM. People usually use that RAM for loopers and delays, not anything that requires super intense realtime math like a neural model… so we ignored it in these experiments.

To make testing and benchmarking easier, we plugged the Daisy Seed into a Daisy Pod, which is basically a powered breakout board for the Daisy that includes stereo audio I/O, a micro SD card slot, and a couple of knobs. Most important for us was the SD card, as we could load a bunch of NAM models into it and have our benchmarking software run all of them sequentially (and write the logs to the SD card as well).

Challenges

We already knew from the beginning that memory limitations were strict (and that’s one of the reasons for Slimmable NAM). With 1 MB for both the model weights and any buffers, we are restricted in the size of model we can use. We decided to try a smaller model than A1, known in the community as A1-nano, with a little but important tweak: replacing the tanh activation with a ReLU. Tanh is an expensive operation that requires too much computation, which led the deep learning community to look for cheaper options. ReLU fits the bill here as one of the simplest activation functions that is currently used in lots of AI models.

While the code in NeuralAmpModelerCore has been battle-tested in the NAM plugin, it was not designed with embedded applications in mind. When we first tried to run it on the Daisy Seed, even with an A1-nano-ReLU model, it would take over 5 seconds of compute time to process 2 seconds of audio... which we should understand would not work for a pedal that is supposed to be doing audio processing in real time.

Optimization process

In order to tackle that issue, we first instrumented the code with profiling to capture how long the code spent on each operation, so we could identify easy targets for optimization. The actual main culprit for the slowdown was Eigen, the library we use for linear algebra, more specifically in how it does matrix multiplications with small matrices and how it allocates temporary memory in embedded devices without an operating system. We added specialized operations for specific sizes of matrices that are used in A1-nano-ReLU, as well as other unrelated optimizations.

Once we identified all the bottlenecks in the code for the A1-nano-ReLU model, we came up with different optimization tactics. I would love to say that I first implemented microbenchmarks to see which ones really mattered and then I added them to the code, but the real process was actually in reverse: I started by optimizing the actual Core code, then implemented microbenchmarks to better understand what was going on clown-face.

The microbenchmarks are all available in this repository for reference. These can be used as a reference for your own hardware, if you are implementing an embedded device that runs NAM.

Results

Micro-benchmark Results: Inline Ops vs Eigen

These benchmarks measure individual operations in isolation to guide optimization decisions for the NeuralAmpModelerCore WaveNet inference engine. Each section presents results for both platforms side-by-side.

Platforms

1. Inline GEMM vs Eigen GEMM

The 1x1 convolution (Conv1x1) is the single hottest operation in WaveNet inference -- it runs once per layer per frame-block to mix channels. It boils down to a matrix-matrix multiply: (channels_out x channels_in) * (channels_in x frames). Eigen's general GEMM path allocates temporary panel buffers via malloc on every call, which is fine on desktop but catastrophically slow on Cortex-M7 with newlib. The "inline generic" variant replaces Eigen with a plain triple-loop; the "unrolled" variant fully unrolls the loop at compile time for known dimension pairs (e.g., 4x8), keeping weights in registers.

Matrix sizes matching typical WaveNet 1x1 convolution dimensions (channels x channels, applied across frames). Each size is measured once: Eigen is benchmarked first, then the inline variant(s) run against that same baseline with the same input data.

Desktop (48 frames)

Desktop (2048 frames)

Daisy (48 frames)

Desktop: Generic inline triple-loops lose badly to Eigen (0.10--0.38x) because Eigen leverages NEON SIMD. Fully unrolled compile-time specializations beat Eigen at all sizes and frame counts (1.15--3.06x), with the largest wins at small frame counts where Eigen's setup overhead is proportionally larger.

Daisy: Inline GEMM beats Eigen at every size because Eigen's general GEMM path calls malloc/free for panel buffer allocation on every invocation. The generic triple-loop wins by 1.04--1.47x. The fully unrolled 4x8 specialization (weights loaded into registers) wins by 3.10x -- the biggest single optimization in the entire WaveNet pipeline.

1b. DTCM Memory Placement Effect on GEMM (Daisy only)

The STM32H750 has 128 KB of DTCM (Data Tightly-Coupled Memory) that provides single-cycle, deterministic access -- no cache misses, no wait states. Production code copies model weights into DTCM to avoid cache eviction during the full WaveNet pipeline. This benchmark isolates the question: does DTCM placement speed up the GEMM kernel itself, or is the benefit only about avoiding interference from other data?

Generic Inline GEMM

Unrolled 4x8 GEMM

Daisy: DTCM placement has no measurable benefit for GEMM operands. In some cases it is marginally slower (up to 5%). The Cortex-M7's D-cache achieves near-single-cycle latency for hot data that fits in cache, and these small weight matrices (64--256 bytes) are always cache-hot after warmup. DTCM's advantage is deterministic latency (no cold-start misses), not throughput for steady-state computation. The production DTCM weight copy remains justified for avoiding cache eviction by other data in the full WaveNet pipeline, but the GEMM kernel itself sees no benefit in isolation. This benchmark is not super representative of real-world usage, as we'd often be juggling multiple operations and invalidating the cache.

2. Effect of __restrict__ on Inline GEMM

The __restrict__ qualifier tells the compiler that pointers don't alias, theoretically enabling more aggressive load/store reordering and vectorization in tight loops. Since our inline GEMM reads from two input matrices and writes to a third, aliasing analysis could matter. Both versions use noinline to prevent the compiler from deducing non-aliasing from calling context, isolating the effect of the qualifier itself.

Desktop (48 frames)

Desktop (2048 frames)

Daisy (48 frames)

Desktop: __restrict__ has negligible effect in most cases. The compiler already deduces non-aliasing. The one outlier (4x4 at 2048 frames, 1.19x) is likely noise or a minor alignment effect.

Daisy: Zero measurable effect, same as desktop. The compiler already deduces non-aliasing regardless of platform.

3. std::memcpy vs Eigen Block Assignment

WaveNet layers frequently copy intermediate activation buffers -- for example, saving the pre-activation state for the residual path, or copying gated outputs into the next layer's input. These are contiguous matrix copies (all elements in one flat buffer). The question is whether a plain std::memcpy or Eigen's block assignment (.leftCols() = ...) generates faster code, given that memcpy quality varies dramatically across C libraries.

Desktop (48 frames)

Desktop (2048 frames)

Daisy (48 frames)

Desktop: std::memcpy consistently beats Eigen block assignment, with the gap widening at larger sizes (up to 2.15x). The platform's highly optimized memcpy implementation (NEON/AVX) outperforms Eigen's generated stores.

Daisy: The opposite of desktop -- Eigen block assignment is 4.3--4.6x faster than std::memcpy. Newlib's memcpy implementation is not optimized for the Cortex-M7's AXI bus and D-cache; Eigen's generated code likely uses more efficient register-width stores.

4. Element-wise Operations: Unrolled Loop vs Eigen

Element-wise addition and accumulation appear throughout WaveNet: skip connections sum layer outputs (z = a + b), and residual connections accumulate into a running buffer (dst += src). Eigen uses expression templates that defer evaluation and fuse operations, but for simple element-wise ops the template machinery may add overhead compared to a plain loop that the compiler can auto-vectorize directly.

Desktop -- Addition (z = a + b)

Desktop -- Accumulation (dst += src)

Daisy -- Addition (z = a + b, 48 frames)

Desktop: Unrolled loops are 1.0--1.5x faster for element-wise ops, with bigger wins at small/medium buffer sizes typical of real-time audio. Accumulation shows larger gains at small frame counts.

Daisy: Unrolled loops are modestly faster (1.08x), consistent with desktop results at 48 frames. The savings per call are small (~100--120 cycles) but accumulate over many WaveNet layers.

5. Bias Broadcast: Unrolled vs Eigen colwise()

Every convolution layer adds a per-channel bias vector after the matrix multiply: each element of the bias is broadcast across all frames in that channel's row. Eigen provides colwise() for this pattern; the alternative is a hand-written loop that iterates over channels and uses pointer arithmetic to stride across frames.

Desktop

Daisy (48 frames)

Desktop: Unrolled bias broadcast is 1.05--1.66x faster than Eigen's colwise(), with larger wins at higher frame counts.

Daisy: Eigen wins by 2--5%, reversing the desktop result. Eigen's colwise() likely generates tighter scalar code on ARM than the hand-written loop. Keep Eigen for bias broadcast on Daisy.

6. Hardswish: Branchy vs Branchless

Hardswish is a piecewise activation function with three regions: 0 for x <= -3, x for x >= 3, and x * (x + 3) / 6 in between. The naive implementation uses if/else branches, which cause pipeline stalls from misprediction when inputs span all three regions (as they do in typical WaveNet activations). The branchless version uses fmin/fmax to compute the same result without conditional jumps. Input values are uniformly distributed in [-5, 5] to trigger worst-case branch misprediction.

Desktop

Daisy

Desktop: Branchless hardswish is 3--7x faster than the branchy version. The simple branchless scalar loop actually beats the manually unrolled variant -- the compiler auto-vectorizes the scalar version more effectively.

Daisy: Branchless hardswish is 1.20--1.21x faster, a much smaller win than desktop. The Cortex-M7 branch predictor handles the 3-region pattern better than expected, and without SIMD the branchless version can't exploit vectorization. Still worth using since it's a consistent 20% improvement.

7. Activation Loop Unrolling (1-wide vs 4-wide)

Activation functions are applied element-wise to every value in the activation buffer after each convolution. Manual 4-wide unrolling processes four elements per loop iteration, giving the compiler explicit instruction-level parallelism (ILP) and aligning with SIMD register widths (4 floats = 1 SSE/NEON register). The benefit depends on whether the activation is compute-bound (cheap ops like ReLU) or latency-bound (expensive ops like expf in Sigmoid/SiLU).

Desktop

Daisy

Desktop: Manual 4-wide unrolling helps ReLU (up to 1.44x) but has negligible effect on transcendental-heavy activations (Sigmoid, SiLU) where expf dominates the cost. Softsign is already fast enough that unrolling provides no meaningful benefit.

Daisy: A modest 9% win for ReLU, consistent with desktop at n=384. Sigmoid is dominated by expf cost (~124 cycles/element), making unrolling irrelevant.

8. LUT Activation vs Computed (expf)

Sigmoid, SiLU, and Tanh all depend on expf(), which is the single most expensive scalar operation in WaveNet inference (~124 cycles per element on Cortex-M7). A lookup table (LUT) pre-computes the activation over a fixed input range, then uses linear interpolation between table entries at runtime -- replacing the expf call with an array index, a multiply, and an add. The tradeoff is a small approximation error that depends on table size. The computed baseline is measured once per activation/size and shared across all LUT sizes.

Desktop (n = 192)

Desktop (n = 384)

Desktop (n = 16384)

Daisy (n = 384)

Desktop: LUT + linear interpolation is 1.7--2.6x faster than computed expf. A 1024-point table offers a good speed/accuracy tradeoff (~5e-06 max error). Tanh benefits the most since it evaluates two expf calls per element.

Daisy: LUT + linear interpolation is 2.81x faster than computed SiLU, a larger win than desktop (1.87x for SiLU at n=384). The bigger gap reflects the higher relative cost of expf on Cortex-M7 (~124 cyc/element computed vs ~44 cyc/element LUT).

9. Strided Sub-matrix Copy (Desktop only)

WaveNet's gated activation splits a 2N-channel activation into two N-channel halves (for the sigmoid gate and tanh path). When the source matrix has 2N rows but the destination has N rows, source and destination have different column strides in memory, so a simple memcpy won't work. Eigen's .topRows().leftCols() handles this via expression templates; the manual version uses explicit pointer arithmetic to copy row-by-row with the correct strides.

Desktop

Desktop: Manual strided copy is 2.9--4.5x faster than Eigen's .topRows().leftCols(). Eigen's expression template overhead dominates for non-contiguous copies.

10. Depthwise Conv: Inline vs Eigen Diagonal

Some WaveNet variants use depthwise (channel-wise) 1x1 convolutions, where each channel is scaled independently -- equivalent to multiplying each row of the activation matrix by a scalar weight. Eigen expresses this as weight.asDiagonal() * input, which constructs a diagonal matrix view and dispatches to a general matrix multiply. The inline version skips the abstraction and directly multiplies each element by the corresponding channel weight.

Desktop

Daisy (48 frames)

Desktop: Inline element-wise depthwise conv is 1.8--4.5x faster than Eigen's asDiagonal() multiply across all tested sizes.

Daisy: Inline wins at 4 channels (1.47x) but ties at 8 channels. The 8-channel case may be hitting register pressure limits where Eigen's code generation produces equivalent scalar code.

11. FiLM Layer: Inline vs Eigen Array Expressions

Feature-wise Linear Modulation (FiLM) applies a learned per-channel affine transform -- output = scale * input + shift -- to condition one network's activations on another's predictions. In WaveNet, FiLM layers modulate hidden activations using conditioning signals. Eigen expresses this with .array() broadcasting; the inline version uses a nested loop over channels and frames. "Scale only" omits the shift term, reducing the operation to a per-channel multiply.

Desktop -- Scale + Shift

Desktop -- Scale Only

Daisy (48 frames)

Desktop: Inline FiLM is 1.2--2.3x faster in most cases. Scale-only is uniformly faster. The 8-channel x 2048-frame scale+shift case shows a regression (0.36x), likely a cache/vectorization edge case.

Daisy: Inline FiLM is 1.08--1.11x faster, a modest but consistent win across both channel counts.

12. Ring Buffer Write: Eigen Block vs Nested Loop

WaveNet uses dilated causal convolutions, which require access to past input frames. A ring buffer stores these frames in a fixed-size circular matrix, and each new block of frames must be written at the current write position. This is a contiguous multi-channel write into the middle of a larger matrix. Eigen's .middleCols() compiles to an optimized block copy; the nested-loop version writes element-by-element with explicit index arithmetic.

Desktop

Daisy (48 frames)

Desktop: Eigen's block assignment wins by up to 4.25x for small channel counts. At 8 channels and large frame counts, the nested loop catches up. Eigen compiles .middleCols() = ... to vectorized contiguous writes that scalar nested loops can't match.

Daisy: Eigen is 9.4--9.9x faster -- an even larger win than desktop. The scalar nested loop generates extremely poor code on ARM, while Eigen compiles to an optimized contiguous copy. This is the strongest argument for keeping Eigen in the codebase.

Summary: Daisy vs Desktop

JSON - when processing text becomes a problem

NeuralAmpModelerCore uses nlohmann::json to parse .nam files, but it requires a lot of memory to parse a model and memory is not exactly something we have plenty of... E.g. an A1 NAM model stored as a JSON file takes almost 400KB of memory. Remember when we said the Cortex M7 has between 1-2MB of RAM? Yeah, that is going to be a problem.

JSON is cool and very accessible for users, as you can just open it in a text file or parse it without any special dependencies in most programming languages. And parsing JSON is definitely not an issue on desktop (your browser does it thousands of times every day without breaking a sweat)... so we did not want to replace the JSON format.

Our approach was to refactor the factory methods in NeuralAmpModelerCore so they expect some sort of model configuration object instead of a JSON object, and offload parsing either JSON or another file-based representation of models to a parser. This way, both JSON and other formats can use the same "entry point" to build a model from a file, they would just use different parsers.

We then came up with the namb format, a compact binary format for models to be used instead of the regular nam format in embedded devices. This format is meant to be used as an exchange format between a controller application (say, an app to load models that runs on your phone) and the embedded device. The app converts the nam format that is retrieved from the T3K website into a compact format that is more suited to your pedal, then transmits it over Bluetooth or an USB cable. There is no model conversion here: think of namb just as a more "compressed" format. Learn more at this repo.

Contributions

All of the optimizations that were helpful for us to run NeuralAmpModelerCore on the Daisy were merged, meaning these are all readily available for anyone who wants to try something similar. They currently require compiling with CXXFLAGS+=USE_NAM_INLINE_GEMM.

In order to enable using our binary loader, model construction in core was decoupled from JSON parsing, which was also merged. The code for nam-binary-loader is available as a separate library and tool in this repo.

Finally, we are publishing example code for the Daisy Seed board, to be used as a blueprint for porting NeuralAmpModelerCore to embedded targets. This was our target during optimization: we worked until we were able to run an A1 nano ReLU model in real time on the Daisy Seed. The code and instructions for how to build and run it on the Daisy are in their own repo.

We hope this helps anyone who is interested in experimenting with supporting NAM in their hardware with trying things out.