Exa boosts embedding throughput on AWS AI chips through custom optimizations

Executive Briefing
“Search company Exa improved document embedding throughput and cost efficiency by 47...”
Key Takeaways
- 1.A migration to re-embed a priority corpus of 30 billion documents temporarily doubled the workload, triggering a re-evaluation of hardware infrastructure and optimization choices.
- 2.The engineering team applied optimizations including weight layout alignment, compiler autocasting, and custom kernel development using the Neuron Kernel Interface on AWS Neuron hardware.
- 3.These enhancements delivered roughly a 47 percent improvement in performance and cost per one million embedded documents against the baseline while maintaining a 0.999 cosine similarity.
Search company Exa improved document embedding throughput and cost efficiency by 47 percent. The engineering team achieved this using AWS Inferentia2 and Trainium2 hardware paired with an agentic development workflow.
Product Features & Key Highlights
A migration to re-embed a priority corpus of 30 billion documents temporarily doubled the workload, triggering a re-evaluation of hardware infrastructure and optimization choices.
The engineering team applied optimizations including weight layout alignment, compiler autocasting, and custom kernel development using the Neuron Kernel Interface on AWS Neuron hardware.
These enhancements delivered roughly a 47 percent improvement in performance and cost per one million embedded documents against the baseline while maintaining a 0.999 cosine similarity.
Practical Assessment & Evaluation
Designing custom kernels tailored to specific model shapes helps eliminate high-bandwidth memory bottlenecks and removes redundant calculations, allowing systems to fully exploit specialized accelerator hardware.
Pricing, Availability & Next Steps
The exact global cost efficiency and latency impacts when scaling across multiple AWS regions outside the Oregon test environment remain unquantified.







