Scaling Conceptual, Semantic Search to Terabytes


Semantic search is tremendously powerful, capable of extending searches to find close matches phrased differently than the search input. However, scaling semantic search, taking it from a snail’s pace to lightning speed is challenging. I will cover various methods reach enterprise performance levels, going from a few thousand comparisons per second to 500M on commodity hardware. We will look at semantic search in the context of an alerting platform where as textual data is ingested, it is compared to pre-defined alerts (vectors) to surface important data to the analyst.