Binary Code Based Hash Embedding For Web-scale Applications

Yan Bencheng, Wang Pengjie, Liu Jinquan, Lin Wei, Lee Kuang-chih, Xu Jian, Zheng Bo. Arxiv 2021

[Paper]    
ARXIV Deep Learning

Nowadays, deep learning models are widely adopted in web-scale applications such as recommender systems, and online advertising. In these applications, embedding learning of categorical features is crucial to the success of deep learning models. In these models, a standard method is that each categorical feature value is assigned a unique embedding vector which can be learned and optimized. Although this method can well capture the characteristics of the categorical features and promise good performance, it can incur a huge memory cost to store the embedding table, especially for those web-scale applications. Such a huge memory cost significantly holds back the effectiveness and usability of EDRMs. In this paper, we propose a binary code based hash embedding method which allows the size of the embedding table to be reduced in arbitrary scale without compromising too much performance. Experimental evaluation results show that one can still achieve 99\% performance even if the embedding table size is reduced 1000$\times$ smaller than the original one with our proposed method.

Similar Work

• Learning To Embed Categorical Features Without Embedding Tables For Recommendation
• Mem-rec Memory Efficient Recommendation System Using Alternative Representation
• Compositional Embeddings Using Complementary Partitions For Memory-efficient Recommendation Systems
• Fine-grained Embedding Dimension Optimization During Training For Recommender Systems