LANNS A Web-scale Approximate Nearest Neighbor Lookup System

Doshi Ishita, Das Dhritiman, Bhutani Ashish, Kumar Rajeev, Bhatt Rushi, Balasubramanian Niranjan. Arxiv 2020

[Paper]    
ARXIV

Nearest neighbor search (NNS) has a wide range of applications in information retrieval, computer vision, machine learning, databases, and other areas. Existing state-of-the-art algorithm for nearest neighbor search, Hierarchical Navigable Small World Networks(HNSW), is unable to scale to large datasets of 100M records in high dimensions. In this paper, we propose LANNS, an end-to-end platform for Approximate Nearest Neighbor Search, which scales for web-scale datasets. Library for Large Scale Approximate Nearest Neighbor Search (LANNS) is deployed in multiple production systems for identifying topK ($100\le topK\le 200$) approximate nearest neighbors with a latency of a few milliseconds per query, high throughput of 2.5k Queries Per Second (QPS) on a single node, on large ($\sim$180M data points) high dimensional (50-2048 dimensional) datasets.

Similar Work

• Approximate Nearest Neighbor Search For Low Dimensional Queries
• EFANNA An Extremely Fast Approximate Nearest Neighbor Search Algorithm Based On Knn Graph
• Subspace Approximation For Approximate Nearest Neighbor Search In NLP
• Efficient Approximate Nearest Neighbor Search For Multiple Weighted l_pleq2 Distance Functions