Blaze

Overview

Blaze is a Go engine that provides fast, full-text search capabilities through an inverted index implementation. It's designed for applications that need to search through text documents efficiently without relying on external search engines.

Installation

go get github.com/wizenheimer/blaze

Quick Start

package main

import (
    "fmt"
    "github.com/wizenheimer/blaze"
)

func main() {
    // Create a new inverted index
    idx := blaze.NewInvertedIndex()

    // Index some documents
    idx.Index(1, "The quick brown fox jumps over the lazy dog")
    idx.Index(2, "A quick brown dog runs fast")
    idx.Index(3, "The lazy cat sleeps all day")

    // Search with BM25 ranking
    matches := idx.RankBM25("quick brown", 10)

    // Print results
    for _, match := range matches {
        fmt.Printf("Document %d (score: %.2f)\n", match.DocID, match.Score)
    }
}

Query Builder API

The Query Builder provides a type-safe, fluent API for constructing complex boolean queries with roaring bitmaps. No string parsing, no syntax errors - just clean, composable code.

Basic Queries

Single Term

// Find all documents containing "machine"
results := blaze.NewQueryBuilder(idx).
    Term("machine").
    Execute()

AND Query

// Find documents with BOTH "machine" AND "learning"
results := blaze.NewQueryBuilder(idx).
    Term("machine").
    And().
    Term("learning").
    Execute()

OR Query

// Find documents with "python" OR "javascript"
results := blaze.NewQueryBuilder(idx).
    Term("python").
    Or().
    Term("javascript").
    Execute()

NOT Query

// Find documents with "python" but NOT "snake"
results := blaze.NewQueryBuilder(idx).
    Term("python").
    And().Not().
    Term("snake").
    Execute()

Complex Grouped Query

// (machine OR deep) AND learning
results := blaze.NewQueryBuilder(idx).
    Group(func(q *blaze.QueryBuilder) {
        q.Term("machine").Or().Term("deep")
    }).
    And().
    Term("learning").
    Execute()

BM25 Ranked Results

// Get top 10 results ranked by relevance
matches := blaze.NewQueryBuilder(idx).
    Term("machine").
    And().
    Term("learning").
    ExecuteWithBM25(10)

for _, match := range matches {
    fmt.Printf("Doc %d: score=%.2f\n", match.DocID, match.Score)
}

Convenience Functions

// AllOf: Documents containing ALL terms (AND operation)
results := blaze.AllOf(idx, "machine", "learning", "python")

// AnyOf: Documents containing ANY term (OR operation)
results := blaze.AnyOf(idx, "cat", "dog", "bird")

// TermExcluding: Term with exclusion (AND NOT operation)
results := blaze.TermExcluding(idx, "python", "snake")

Architecture

Query Processor Architecture

The query processor uses a hybrid storage approach combining roaring bitmaps for document-level operations and skip lists for position-level operations.

Hybrid Storage Model

Blaze uses a sophisticated hybrid storage model that combines the speed of roaring bitmaps with the precision of skip lists.

Query Execution Flow

Here's how a complex query executes step-by-step, showcasing the power of the hybrid storage approach.

Roaring Bitmap Internals

Roaring bitmaps provide exceptional compression and performance through adaptive container selection.

Memory Layout

Understanding the memory layout helps optimize for large-scale deployments.

Performance

Benchmarks

Operation Costs

Search Operations

Phrase Search

Find exact sequences of words. Blaze uses skip lists to efficiently verify consecutive positions.

// Find exact phrase "quick brown fox"
matches := idx.FindAllPhrases("quick brown fox", blaze.BOFDocument)

for _, match := range matches {
    start, end := match[0], match[1]
    fmt.Printf("Found in Doc %d from Pos %d to %d\n",
        int(start.DocumentID), int(start.Offset), int(end.Offset))
}

Proximity Search

Find documents containing all terms (not necessarily consecutive). Score documents by how close the terms appear together.

// Rank documents by proximity
matches := idx.RankProximity("quick brown", 5)

for _, match := range matches {
    fmt.Printf("Doc %d: Score %.2f\n",
        int(match.Offsets[0].DocumentID),
        match.Score)
}

Skip Lists Explained

Skip lists are the core data structure for position tracking. They provide O(log n) operations while being simpler than balanced trees.

Text Analysis Pipeline

Blaze transforms raw text into searchable tokens through a multi-stage pipeline.

BM25 Algorithm

BM25 (Best Matching 25) is the industry-standard ranking function used by Elasticsearch, Solr, and Lucene. It considers term frequency, document length, and term rarity.

// Search and rank using BM25
results := idx.RankBM25("machine learning", 10)

for _, match := range results {
    fmt.Printf("Doc %d: Score %.2f\n",
        match.DocID,
        match.Score)
}

Real-World Examples

E-commerce Search

func SearchProducts(idx *blaze.InvertedIndex, query string, 
                    category string, excludeOutOfStock bool) []blaze.Match {
    qb := blaze.NewQueryBuilder(idx).Term(query)
    
    // Add category filter
    if category != "" {
        qb.And().Term(category)
    }
    
    // Exclude out of stock items
    if excludeOutOfStock {
        qb.And().Not().Term("outofstock")
    }
    
    return qb.ExecuteWithBM25(20)
}

Multi-Category Search

func SearchInCategories(idx *blaze.InvertedIndex, 
                        query string, categories []string) []blaze.Match {
    qb := blaze.NewQueryBuilder(idx).Term(query)
    
    if len(categories) > 0 {
        qb.And().Group(func(q *blaze.QueryBuilder) {
            q.Term(categories[0])
            for i := 1; i < len(categories); i++ {
                q.Or().Term(categories[i])
            }
        })
    }
    
    return qb.ExecuteWithBM25(50)
}

Advanced Boolean Search

// Find documents about (machine learning OR deep learning) 
// AND (python OR tensorflow)
results := blaze.NewQueryBuilder(idx).
    Group(func(q *blaze.QueryBuilder) {
        q.Phrase("machine learning").Or().Phrase("deep learning")
    }).
    And().
    Group(func(q *blaze.QueryBuilder) {
        q.Term("python").Or().Term("tensorflow")
    }).
    ExecuteWithBM25(20)

Use Cases

• Document Search Systems: Build full-text search for documentation, articles, or knowledge bases
• Log Analysis: Search through application logs with boolean queries and ranking
• Code Search: Index and search source code repositories
• E-commerce: Product catalog search with filters and facets
• Email Search: Fast, local email search without external services
• Content Management: Search articles, posts, and media metadata

Key Features

Search Capabilities

• Term Search: Find documents containing specific terms
• Phrase Search: Exact multi-word matching
• Boolean Queries: Type-safe AND, OR, NOT operations
• BM25 Ranking: Industry-standard relevance scoring
• Proximity Ranking: Score by term proximity
• Position Tracking: Track exact word positions

Text Processing

• Tokenization: Unicode-aware text splitting
• Stemming: Snowball (Porter2) stemmer
• Stopword Filtering: Remove common words
• Case Normalization: Case-insensitive search
• Configurable Pipeline: Customize analysis behavior

Data Structures

• Roaring Bitmaps: 400x compression for document sets
• Skip Lists: O(log n) position lookups
• Inverted Index: Efficient term-to-position mapping
• Binary Serialization: Compact storage format