Details on model weight compression enclaves, supported precision profiles, and HVX/HMX accelerator layouts.

Edge deployments require optimization of model weights to fit strict memory layouts. Aura SDK implements state-of-the-art quantization techniques to reduce the footprint of LLMs while minimizing perplexity degradation.

Supported Precision Profiles

The Snapdragon Hexagon tensor engine is optimized for integer math. We support the following quantization configurations:

Profile	Weights Precision	Activations Precision	Hardware Acceleration
w4a16	4-bit INT	16-bit dynamic FP	Hardware-accelerated (Optimal)
w8a8	8-bit INT	8-bit INT	Hardware-accelerated
w16a16	16-bit FP	16-bit FP	Hardware-accelerated
fp32	32-bit FP	32-bit FP	CPU Fallback (No NPU acceleration)

w4a16 Quantization

The w4a16 format represents the best trade-off between speed and model accuracy for generative models. The weights are stored as 4-bit integers with a per-channel scale, and activations are dynamically upscaled to 16-bit floating point inside the tensor processors during operations.

Quantization Toolchain Example

To convert an existing HuggingFace transformers model to Aura’s w4a16 format:

python -m aura_quantizer --model meta-llama/Llama-3-8B --output ./llama3-w4a16.aura --precision w4a16 --calibration-dataset wiki-text

Accuracy Calibration

Quantization can introduce mathematical rounding errors. Using calibration datasets ensures that high-weight tensor paths are allocated high-precision scales, preserving complex reasoning capabilities of the target neural networks.