Hybrid Search With The OpenAI Assistants API

Enhancing The OpenAI Assistants API with Metadata Filtering

Vector databases have transformed how applications leverage large datasets to perform intelligent searches based on semantic similarity. Yet, even with the powerful capabilities of vector searches, pinpointing precise content among vast datasets can be challenging. By introducing robust metadata filtering, Ragwalla provides developers with precise control over their search scope, significantly improving the relevance of query results.

How Metadata Indexing Works

When creating a vector store, developers can define up to 10 custom metadata indexes. When defining metadata developers can determine whether to use manual or LLM auto mode.

• Manual Mode: Developers explicitly set keywords.

• LLM Auto Mode: An LLM (Large Language Model) generates keywords automatically for the specified metadata indexes.

This flexible system ensures that metadata precisely matches your application needs, significantly enhancing search accuracy.

Filtering Operations Supported

Our vector store supports robust metadata filtering options that give developers granular control over their search queries:

• Equality ($eq): Select records matching exactly the metadata value.

• Inequality ($ne): Exclude records with a specific metadata value.

• Inclusion ($in): Select records whose metadata values fall within a specified set.

• Exclusion ($nin): Exclude records whose metadata values are within a specified set.

• Range Queries ($lt, $lte, $gt, $gte): Select records based on numerical ranges.

Example Usage

Consider a vector store containing support documents. Developers could index metadata such as document type (docType), creation date (created_at), and author name (author).

A sample metadata schema might look like:

{
  "author": String,
  "created_at": Number,
  "topic": String
}

To find vectors matching specific criteria, such as documents created after January 1, 2024, and authored by "John Doe", a developer would issue a query like:

{
  "metadata": {
    "author": { "$eq": "John Doe" },
    "created_at": { "$gte": 1740582937 }
  }
}

This query narrows down results by applying filters directly on the metadata before the similarity search is executed, ensuring only relevant vectors are considered.

Illustrations of Vector Searches

Vector Search Without Filtering

Query: "How to reset password?"
│
└─── Semantic Similarity Search
     │
     ├── "Password reset instructions"
     ├── "Resetting account password"
     ├── "Troubleshooting login issues"
     └── "Creating a new account"

Vector Search With Keyword Metadata Filtering

Metadata Filter: docType = 'support'

Vector Search
  │
  ├── Metadata Filter (docType = 'support')
  │   ├── "Resetting your password" (docType: support)
  │   ├── "Troubleshooting login issues" (docType: support)
  │   └── "Updating user profile settings" (docType = 'support')
  └── Semantic Similarity Match ──▶ Final Relevant Results

Vector Search With Numerical Range Metadata Filtering

Metadata Filter: created_at ≥ 1740582937

Vector Search
  │
  ├── Metadata Filter (created_at ≥ 1740582937)
  │   ├── "Document A (1740582938)"
  │   ├── "Document B (1740582999)"
  │   └── "Document Older Than Specified Range (1740582936)" (filtered out)
  │
  └── Filtered Dataset ──▶ Semantic Similarity Search ──▶ Final Results

Examples of Metadata Filters

Equality Example:

{"author": {"$eq": "Alice"}}

Inclusion Example:

{"topic": {"$in": ["AI", "Machine Learning"]}}

Range Query Example:

{"created_at": {"$gte": 1740582937, "$lte": 1740582937}}

Benefits of Metadata Filtering

• Accuracy: Improves precision by limiting searches to the most relevant subset of your vector database.

• Performance: Significantly boosts query performance by reducing the dataset processed.

• Customization: Offers flexibility by enabling both developer-defined and automated LLM-generated metadata.

Benefits Visualized

• Without Filtering:

All Documents
  │
  └──▶ Semantic Search ──▶ Broad Results

• With Filtering:

All Documents
  │
  └──▶ Metadata Filtering ──▶ Reduced Subset ──▶ Semantic Search ──▶ Precise Results

Conclusion

Hybrid search allows developers to build smarter, faster, and more precise data retrieval systems. Whether manually setting keywords or automating the process with LLM Auto mode, this approach enhances the effectiveness and efficiency of vector-based applications.