Semantic Search (Vector Search) - When your term finds similar terms

Simply put: A semantic search allows you to find synonyms and related terms or concepts along with a single search term.

Scheme of Semantic Search and Vector Search

Encoding

Long before you execute your query, the system, or rather the encoding model, was trained using a process called contrastive learning. Billions of texts and text fragments were used to adjust the model's weights so that it calculates suitable vectors for its intended purpose. Contrastive learning trains the model so that vectors representing similar meanings are located close together, while those with different meanings are far apart. This model then forms the basis of the embedding model for vector search.

Typically, the search is performed across a defined corpus, such as a document or reference collection.

Embedding

1. Chunking and Tokenization of the Collection

To make the collection searchable, the texts within it are split into chunks, such as sentences, words, or word fragments. The smallest unit is called a token. Each token has its own numerical ID. The word "heart," for example, could have the token ID 4821.

The token IDs of a sentence thus form a vector that can be mapped in space.

These vectors of the collection are stored in databases.

2. Chunking and Tokenization of the Query

Your query is also translated into tokens and vectors and (temporarily) stored in databases.

Retrieval

3. Matching Vectors

A retrieval module then statistically calculates which vectors from the query best match the vectors of the documents or references in the corpus (which vectors are closest in space).

4. Calculating Relevance (optional)

Afterward, a re-ranking (sometimes) takes place. It is calculated which documents are most relevant to the query. These documents or references are returned in a relevance order.