Perplexity builds custom serving infrastructure to make AI search faster and cheaper

Running an AI search engine that fields roughly 400 million queries a month is expensive. Perplexity AI has been quietly building the infrastructure to make it less so, publishing research on custom serving technology that spans everything from compact embedding models to a proprietary engine capable of handling trillion-parameter models.

The embedding play

At the core of Perplexity’s retrieval improvements is the pplx-embed family of models, which includes pplx-embed-v1 and pplx-embed-context-v1. These come in two sizes: a 0.6 billion parameter variant and a larger 4 billion parameter version. Both were released in February 2026.

Through quantization techniques, which essentially reduce the numerical precision of model weights without destroying accuracy, Perplexity claims up to 32x reductions in storage requirements. Both models were optimized specifically for low-storage deployment, a design choice that reflects the practical reality of web-scale retrieval.

ROSE and custom networking

The more ambitious piece of infrastructure is what Perplexity calls ROSE, short for Runtime-Optimized Serving Engine. Introduced around February 2025, ROSE is designed to serve a wide range of model architectures, including the increasingly popular Mixture-of-Experts (MoE) format that powers many of today’s largest language models.

Perplexity addressed MoE serving with custom RDMA networking optimizations that achieve up to 97.1% bandwidth efficiency. RDMA, or Remote Direct Memory Access, lets GPUs communicate without going through the CPU. The engine runs across AWS Elastic Fabric Adapter environments. In November 2025, Perplexity published its first paper on arXiv detailing these optimizations and open-sourced a library called fabric-lib containing custom MoE kernels that deliver what the company describes as state-of-the-art latencies for trillion-parameter model serving.

Hybrid inference and the on-device push

In June 2026, Perplexity unveiled a hybrid local-cloud inference orchestrator. The system automatically routes tasks between on-device models running on local hardware and more powerful cloud-based models, depending on the complexity of the query. Perplexity’s orchestrator supports multiple local silicon options, and keeping certain queries on-device means they never hit a server.

What this means for the inference arms race

The company runs on NVIDIA infrastructure and has strategic partnerships with both NVIDIA and CoreWeave, the GPU cloud provider that has become a go-to for AI workloads. At 400 million monthly queries, the 97.1% bandwidth efficiency figure for ROSE and the 32x storage reduction from quantized embeddings translate directly into infrastructure cost savings.

Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.