docs(adr): ADR-0021: Search Index Format Evolution

ADR-0021: Search Index Format Evolution

Status

Accepted

Context

mdbook-htmx generates a search index (search-index.json) that enables full-text search. As the project evolves, the search index format will need to change to support new features:

• Additional metadata (headings, code blocks, tags)
• Scope-aware filtering
• Multi-language support
• Performance optimizations
• Integration with different search backends (lunr.js, Meilisearch, Algolia)

Changes must be backward-compatible or provide clear migration paths.

Decision Drivers

1. Backward Compatibility - Older readers should gracefully handle new formats
2. Forward Compatibility - Newer readers should handle older formats
3. Performance - Index size and generation time matter
4. Flexibility - Support multiple search backends
5. Clarity - Format should be self-documenting

Decision

Use semantic versioning with a version field, optional fields, and a migration strategy.

Version Schema

{
  "$schema": "https://schemas.arusty.dev/mdbook-htmx/search-index/1.0.0.json",
  "version": "1.0.0",
  "generated": "2026-01-04T12:00:00Z",
  "generator": "mdbook-htmx 0.1.0",
  "config": {
    "language": "en",
    "minChars": 2,
    "stopWords": true
  },
  "pages": [...]
}

Version Numbering

MAJOR.MINOR.PATCH

MAJOR: Breaking changes (field removed, meaning changed)
MINOR: New optional fields, new page fields
PATCH: Bug fixes, documentation

Compatibility Rules

Reader Version	Index Version	Behavior
1.x	1.x	Full compatibility
1.x	2.x	Error or degraded mode
2.x	1.x	Migration applied

Version Detection

function loadSearchIndex(index) {
  const version = index.version || "0.0.0";
  const [major] = version.split(".").map(Number);

  switch (major) {
    case 0:
      return migrateV0(index);
    case 1:
      return index; // Current version
    case 2:
      throw new Error(`Search index v${version} requires mdbook-htmx upgrade`);
    default:
      throw new Error(`Unknown search index version: ${version}`);
  }
}

Format Evolution Plan

Version 1.0.0 (Current)

Base format with core fields:

{
  "version": "1.0.0",
  "generated": "2026-01-04T12:00:00Z",
  "config": {
    "language": "en"
  },
  "pages": [
    {
      "path": "/guide/intro.html",
      "title": "Introduction",
      "content": "Full text content for search...",
      "headings": ["Getting Started", "Prerequisites"]
    }
  ]
}

Version 1.1.0 (Planned)

Add scope and tags (optional fields):

{
  "version": "1.1.0",
  "pages": [
    {
      "path": "/internal/roadmap.html",
      "title": "Product Roadmap",
      "content": "...",
      "headings": ["Q1 Goals", "Q2 Goals"],
      "scope": "internal",       // NEW
      "tags": ["planning"]       // NEW
    }
  ]
}

Version 1.2.0 (Planned)

Add code block indexing:

{
  "version": "1.2.0",
  "pages": [
    {
      "path": "/api/auth.html",
      "title": "Authentication API",
      "content": "...",
      "headings": [...],
      "codeBlocks": [           // NEW
        {
          "language": "rust",
          "content": "fn authenticate(...)"
        }
      ]
    }
  ]
}

Version 1.3.0 (Planned)

Add heading anchors for deep linking:

{
  "version": "1.3.0",
  "pages": [
    {
      "path": "/guide/config.html",
      "title": "Configuration",
      "content": "...",
      "sections": [             // NEW (replaces headings)
        {
          "id": "basic-setup",
          "title": "Basic Setup",
          "level": 2,
          "content": "..."
        }
      ]
    }
  ]
}

Version 2.0.0 (Future)

Breaking changes if needed:

{
  "version": "2.0.0",
  "format": "segmented",        // NEW format type
  "segments": {
    "pages": [...],
    "headings": [...],          // Separate index
    "code": [...]               // Separate index
  }
}

Migration Strategies

V0 to V1 Migration

function migrateV0(index) {
  // V0 had no version field
  return {
    version: "1.0.0",
    generated: new Date().toISOString(),
    config: { language: "en" },
    pages: (index.docs || index.pages || []).map(doc => ({
      path: doc.url || doc.path,
      title: doc.title,
      content: doc.body || doc.content,
      headings: doc.breadcrumbs || []
    }))
  };
}

Field Presence Handling

function getSearchableContent(page) {
  const parts = [page.title, page.content];

  // Handle optional fields gracefully
  if (page.headings) {
    parts.push(...page.headings);
  }

  if (page.sections) {
    parts.push(...page.sections.map(s => s.title));
    parts.push(...page.sections.map(s => s.content));
  }

  if (page.codeBlocks) {
    parts.push(...page.codeBlocks.map(b => b.content));
  }

  if (page.tags) {
    parts.push(...page.tags);
  }

  return parts.filter(Boolean).join(' ');
}

Backend-Specific Formats

Lunr.js (Client-Side)

{
  "version": "1.0.0",
  "backend": "lunr",
  "index": {
    "version": "2.3.9",
    "fields": ["title", "content", "headings"],
    "fieldVectors": [...],
    "invertedIndex": [...]
  },
  "store": {
    "/guide/intro.html": {
      "title": "Introduction",
      "headings": ["Getting Started"]
    }
  }
}

Meilisearch

{
  "version": "1.0.0",
  "backend": "meilisearch",
  "documents": [
    {
      "id": "guide-intro",
      "path": "/guide/intro.html",
      "title": "Introduction",
      "content": "...",
      "headings": ["Getting Started"],
      "_tags": ["guide"]
    }
  ],
  "settings": {
    "searchableAttributes": ["title", "headings", "content"],
    "filterableAttributes": ["_tags", "scope"]
  }
}

Algolia

{
  "version": "1.0.0",
  "backend": "algolia",
  "records": [
    {
      "objectID": "guide-intro",
      "path": "/guide/intro.html",
      "title": "Introduction",
      "content": "...",
      "hierarchy": {
        "lvl0": "Guide",
        "lvl1": "Introduction",
        "lvl2": "Getting Started"
      }
    }
  ]
}

Schema Validation

JSON Schema for validation:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://schemas.arusty.dev/mdbook-htmx/search-index/1.0.0.json",
  "title": "mdbook-htmx Search Index",
  "type": "object",
  "required": ["version", "pages"],
  "properties": {
    "version": {
      "type": "string",
      "pattern": "^[0-9]+\\.[0-9]+\\.[0-9]+$"
    },
    "generated": {
      "type": "string",
      "format": "date-time"
    },
    "generator": {
      "type": "string"
    },
    "config": {
      "$ref": "#/$defs/config"
    },
    "pages": {
      "type": "array",
      "items": { "$ref": "#/$defs/page" }
    }
  },
  "$defs": {
    "config": {
      "type": "object",
      "properties": {
        "language": { "type": "string", "default": "en" },
        "minChars": { "type": "integer", "default": 2 },
        "stopWords": { "type": "boolean", "default": true }
      }
    },
    "page": {
      "type": "object",
      "required": ["path", "title", "content"],
      "properties": {
        "path": { "type": "string" },
        "title": { "type": "string" },
        "content": { "type": "string" },
        "headings": {
          "type": "array",
          "items": { "type": "string" }
        },
        "scope": { "type": "string" },
        "tags": {
          "type": "array",
          "items": { "type": "string" }
        },
        "sections": {
          "type": "array",
          "items": { "$ref": "#/$defs/section" }
        },
        "codeBlocks": {
          "type": "array",
          "items": { "$ref": "#/$defs/codeBlock" }
        }
      }
    },
    "section": {
      "type": "object",
      "required": ["id", "title", "level"],
      "properties": {
        "id": { "type": "string" },
        "title": { "type": "string" },
        "level": { "type": "integer", "minimum": 1, "maximum": 6 },
        "content": { "type": "string" }
      }
    },
    "codeBlock": {
      "type": "object",
      "required": ["content"],
      "properties": {
        "language": { "type": "string" },
        "content": { "type": "string" },
        "filename": { "type": "string" }
      }
    }
  }
}

Implementation

Rust Index Generator

// src/search/index.rs
use serde::{Serialize, Deserialize};
use semver::Version;

pub const CURRENT_VERSION: &str = "1.0.0";

#[derive(Serialize, Deserialize)]
pub struct SearchIndex {
    pub version: String,
    pub generated: String,
    pub generator: String,
    pub config: SearchConfig,
    pub pages: Vec<SearchPage>,
}

impl SearchIndex {
    pub fn new(config: SearchConfig) -> Self {
        Self {
            version: CURRENT_VERSION.to_string(),
            generated: chrono::Utc::now().to_rfc3339(),
            generator: format!("mdbook-htmx {}", env!("CARGO_PKG_VERSION")),
            config,
            pages: vec![],
        }
    }

    pub fn is_compatible(&self, reader_version: &str) -> bool {
        let index_ver = Version::parse(&self.version).unwrap();
        let reader_ver = Version::parse(reader_version).unwrap();

        // Same major version is compatible
        index_ver.major == reader_ver.major
    }
}

#[derive(Serialize, Deserialize)]
pub struct SearchPage {
    pub path: String,
    pub title: String,
    pub content: String,
    #[serde(skip_serializing_if = "Vec::is_empty")]
    pub headings: Vec<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub scope: Option<String>,
    #[serde(skip_serializing_if = "Vec::is_empty")]
    pub tags: Vec<String>,
}

JavaScript Loader

// search-loader.js
const SUPPORTED_VERSIONS = {
  min: "1.0.0",
  max: "1.99.99"
};

export async function loadSearchIndex(url) {
  const response = await fetch(url);
  const index = await response.json();

  // Version check
  const version = index.version || "0.0.0";

  if (compareVersions(version, SUPPORTED_VERSIONS.min) < 0) {
    console.warn(`Migrating old search index v${version}`);
    return migrate(index);
  }

  if (compareVersions(version, SUPPORTED_VERSIONS.max) > 0) {
    throw new Error(
      `Search index v${version} is too new. ` +
      `Max supported: ${SUPPORTED_VERSIONS.max}`
    );
  }

  return index;
}

function compareVersions(a, b) {
  const [aMajor, aMinor, aPatch] = a.split('.').map(Number);
  const [bMajor, bMinor, bPatch] = b.split('.').map(Number);

  if (aMajor !== bMajor) return aMajor - bMajor;
  if (aMinor !== bMinor) return aMinor - bMinor;
  return aPatch - bPatch;
}

Configuration

[output.htmx.search]
enabled = true
index-format = "1.0"      # Major.minor
backend = "lunr"          # lunr | meilisearch | algolia | custom
include-code = false      # Index code blocks
include-headings = true   # Index headings
min-chars = 2             # Minimum query length
stop-words = true         # Filter common words

Size Optimization

Content Truncation

impl SearchPage {
    pub fn from_chapter(chapter: &Chapter, config: &SearchConfig) -> Self {
        let content = if config.truncate {
            truncate_content(&chapter.content, config.max_length)
        } else {
            chapter.content.clone()
        };

        Self {
            path: chapter.path.clone(),
            title: chapter.name.clone(),
            content,
            ..Default::default()
        }
    }
}

fn truncate_content(content: &str, max_length: usize) -> String {
    if content.len() <= max_length {
        return content.to_string();
    }

    // Truncate at word boundary
    let truncated = &content[..max_length];
    if let Some(last_space) = truncated.rfind(' ') {
        truncated[..last_space].to_string()
    } else {
        truncated.to_string()
    }
}

Compression

// For large indexes, use compression
async function loadCompressedIndex(url) {
  const response = await fetch(url);
  const blob = await response.blob();

  // Decompress if gzipped
  if (url.endsWith('.gz')) {
    const ds = new DecompressionStream('gzip');
    const decompressed = blob.stream().pipeThrough(ds);
    const text = await new Response(decompressed).text();
    return JSON.parse(text);
  }

  return response.json();
}

Consequences

Positive

• Clear upgrade path with versioning
• Backward compatible minor changes
• Self-documenting format
• Multiple backend support

Negative

• Version checking adds complexity
• Migration code must be maintained
• Larger index with more fields

Mitigation

• Clear version constants
• Migration functions are isolated
• Optional fields reduce size

Alternatives Considered

No Versioning

Just change the format as needed.

Rejected because:

• No way to detect incompatibility
• Breaks existing deployments
• No migration path

Breaking Changes Only

Only use major versions, always break.

Rejected because:

• Forces unnecessary upgrades
• Loses backward compatibility
• Bad user experience

Separate Index Files

Different file per backend.

Rejected because:

• Complicates build process
• Multiple files to manage
• Harder to switch backends

References

• Semantic Versioning
• JSON Schema
• Lunr.js Serialization
• Meilisearch Documents
• Algolia Record Format