Initial commit: QMD - Quick Markdown Search

A CLI tool for searching markdown knowledge bases using hybrid retrieval:
- BM25 full-text search via SQLite FTS5
- Vector semantic search via sqlite-vec + Ollama embeddings
- LLM re-ranking with qwen3-reranker (logprobs-based scoring)
- Reciprocal Rank Fusion with weighted queries and position-aware blending

Features:
- `qmd add .` - Index markdown files in current directory
- `qmd embed` - Generate vector embeddings
- `qmd search` - BM25 full-text search
- `qmd vsearch` - Vector similarity search
- `qmd query` - Hybrid search with query expansion + reranking

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Tobi Lutke 2025-12-07 19:16:16 -05:00
commit 39193ea252
No known key found for this signature in database
8 changed files with 1840 additions and 0 deletions

9
.gitignore vendored Normal file
View File

@ -0,0 +1,9 @@
node_modules/
*.sqlite
.DS_Store
archive/
texts/
.cursor/
*.md
!README.md
!CLAUDE.md

111
CLAUDE.md Normal file
View File

@ -0,0 +1,111 @@
---
description: Use Bun instead of Node.js, npm, pnpm, or vite.
globs: "*.ts, *.tsx, *.html, *.css, *.js, *.jsx, package.json"
alwaysApply: false
---
Default to using Bun instead of Node.js.
- Use `bun <file>` instead of `node <file>` or `ts-node <file>`
- Use `bun test` instead of `jest` or `vitest`
- Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild`
- Use `bun install` instead of `npm install` or `yarn install` or `pnpm install`
- Use `bun run <script>` instead of `npm run <script>` or `yarn run <script>` or `pnpm run <script>`
- Bun automatically loads .env, so don't use dotenv.
## APIs
- `Bun.serve()` supports WebSockets, HTTPS, and routes. Don't use `express`.
- `bun:sqlite` for SQLite. Don't use `better-sqlite3`.
- `Bun.redis` for Redis. Don't use `ioredis`.
- `Bun.sql` for Postgres. Don't use `pg` or `postgres.js`.
- `WebSocket` is built-in. Don't use `ws`.
- Prefer `Bun.file` over `node:fs`'s readFile/writeFile
- Bun.$`ls` instead of execa.
## Testing
Use `bun test` to run tests.
```ts#index.test.ts
import { test, expect } from "bun:test";
test("hello world", () => {
expect(1).toBe(1);
});
```
## Frontend
Use HTML imports with `Bun.serve()`. Don't use `vite`. HTML imports fully support React, CSS, Tailwind.
Server:
```ts#index.ts
import index from "./index.html"
Bun.serve({
routes: {
"/": index,
"/api/users/:id": {
GET: (req) => {
return new Response(JSON.stringify({ id: req.params.id }));
},
},
},
// optional websocket support
websocket: {
open: (ws) => {
ws.send("Hello, world!");
},
message: (ws, message) => {
ws.send(message);
},
close: (ws) => {
// handle close
}
},
development: {
hmr: true,
console: true,
}
})
```
HTML files can import .tsx, .jsx or .js files directly and Bun's bundler will transpile & bundle automatically. `<link>` tags can point to stylesheets and Bun's CSS bundler will bundle.
```html#index.html
<html>
<body>
<h1>Hello, world!</h1>
<script type="module" src="./frontend.tsx"></script>
</body>
</html>
```
With the following `frontend.tsx`:
```tsx#frontend.tsx
import React from "react";
// import .css files directly and it works
import './index.css';
import { createRoot } from "react-dom/client";
const root = createRoot(document.body);
export default function Frontend() {
return <h1>Hello, world!</h1>;
}
root.render(<Frontend />);
```
Then, run index.ts
```sh
bun --hot ./index.ts
```
For more information, read the Bun API docs in `node_modules/bun-types/docs/**.md`.

333
README.md Normal file
View File

@ -0,0 +1,333 @@
# QMD - Quick Markdown Search
A CLI tool for searching markdown knowledge bases using hybrid retrieval: combining BM25 full-text search, vector semantic search, and LLM re-ranking.
## Architecture
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ QMD Search Pipeline │
└─────────────────────────────────────────────────────────────────────────────┘
┌─────────────────┐
│ User Query │
└────────┬────────┘
┌──────────────┴──────────────┐
▼ ▼
┌────────────────┐ ┌────────────────┐
│ Query Expansion│ │ Direct Query │
│ (qwen3:0.6b) │ │ (×2 weight) │
└───────┬────────┘ └───────┬────────┘
│ │
│ 1 alternative query │
└──────────────┬──────────────┘
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ FTS Search │ │ FTS Search │ │ FTS Search │
│ (BM25) │ │ (BM25) │ │ (BM25) │
└───────┬────────┘ └───────┬────────┘ └───────┬────────┘
│ │ │
┌───────┴────────┐ ┌───────┴────────┐ ┌───────┴────────┐
│ Vector Search │ │ Vector Search │ │ Vector Search │
│(embeddinggemma)│ │(embeddinggemma)│ │(embeddinggemma)│
└───────┬────────┘ └───────┬────────┘ └───────┬────────┘
│ │ │
└──────────────────┼──────────────────┘
┌───────────────────────┐
│ RRF Fusion + Bonus │
│ (Top-rank preserved) │
│ Top 30 Kept │
└───────────┬───────────┘
┌───────────────────────┐
│ LLM Re-ranking │
│ (qwen3-reranker) │
│ Yes/No + logprobs │
└───────────┬───────────┘
┌───────────────────────┐
│ Position-Aware Blend │
│ (RRF + Reranker) │
└───────────────────────┘
```
## Score Normalization & Fusion
### Search Backends
| Backend | Raw Score | Conversion | Range |
|---------|-----------|------------|-------|
| **FTS (BM25)** | SQLite FTS5 BM25 | `Math.abs(score)` | 0 to ~25+ |
| **Vector** | Cosine distance | `1 / (1 + distance)` | 0.0 to 1.0 |
| **Reranker** | LLM 0-10 rating | `score / 10` | 0.0 to 1.0 |
### Fusion Strategy
The `query` command uses **Reciprocal Rank Fusion (RRF)** with position-aware blending:
1. **Query Expansion**: Original query (×2 for weighting) + 1 LLM variation
2. **Parallel Retrieval**: Each query searches both FTS and vector indexes
3. **RRF Fusion**: Combine all result lists using `score = Σ(1/(k+rank+1))` where k=60
4. **Top-Rank Bonus**: Documents ranking #1 in any list get +0.05, #2-3 get +0.02
5. **Top-K Selection**: Take top 30 candidates for reranking
6. **Re-ranking**: LLM scores each document (yes/no with logprobs confidence)
7. **Position-Aware Blending**:
- RRF rank 1-3: 75% retrieval, 25% reranker (preserves exact matches)
- RRF rank 4-10: 60% retrieval, 40% reranker
- RRF rank 11+: 40% retrieval, 60% reranker (trust reranker more)
**Why this approach**: Pure RRF can dilute exact matches when expanded queries don't match. The top-rank bonus preserves documents that score #1 for the original query. Position-aware blending prevents the reranker from destroying high-confidence retrieval results.
### Score Interpretation
| Score | Meaning |
|-------|---------|
| 0.8 - 1.0 | Highly relevant |
| 0.5 - 0.8 | Moderately relevant |
| 0.2 - 0.5 | Somewhat relevant |
| 0.0 - 0.2 | Low relevance |
## Requirements
### System Requirements
- **Bun** >= 1.0.0
- **macOS**: Homebrew SQLite (for extension support)
```sh
brew install sqlite
```
- **Ollama** running locally (default: `http://localhost:11434`)
### Ollama Models
QMD uses three models (auto-pulled if missing):
| Model | Purpose | Size |
|-------|---------|------|
| `embeddinggemma` | Vector embeddings | ~1.6GB |
| `ExpedientFalcon/qwen3-reranker:0.6b-q8_0` | Re-ranking (trained) | ~640MB |
| `qwen3:0.6b` | Query expansion | ~400MB |
```sh
# Pre-pull models (optional)
ollama pull embeddinggemma
ollama pull ExpedientFalcon/qwen3-reranker:0.6b-q8_0
ollama pull qwen3:0.6b
```
## Installation
```sh
bun install
```
## Usage
### Index Markdown Files
```sh
# Index all .md files in current directory
qmd index
# Index with custom glob pattern
qmd index "**/*.md"
# Index specific directory
qmd index "docs/**/*.md"
```
### Generate Vector Embeddings
```sh
# Embed all indexed documents
qmd embed
```
### Search Commands
```
┌──────────────────────────────────────────────────────────────────┐
│ Search Modes │
├──────────┬───────────────────────────────────────────────────────┤
│ search │ BM25 full-text search only │
│ vsearch │ Vector semantic search only │
│ query │ Hybrid: FTS + Vector + Query Expansion + Re-ranking │
└──────────┴───────────────────────────────────────────────────────┘
```
```sh
# Full-text search (fast, keyword-based)
qmd search "authentication flow"
# Vector search (semantic similarity)
qmd vsearch "how to login"
# Hybrid search with re-ranking (best quality)
qmd query "user authentication"
```
### Options
```sh
-n <num> # Number of results (default: 5)
--min-score <num> # Minimum score threshold (default: 0)
--full # Show full document content
-csv # CSV output (for piping/scripting)
-md # Output as markdown
-xml # Output as XML
--index <name> # Use named index
```
### Output Format
Default output is colorized CLI format (respects `NO_COLOR` env):
```
93% docs/guide.md:42
│ This section covers the **craftsmanship** of building
│ quality software with attention to detail.
│ See also: engineering principles
67% notes/meeting.md:15
│ Discussion about code quality and craftsmanship
│ in the development process.
```
- **Score**: Color-coded (green >70%, yellow >40%, dim otherwise)
- **Path**: Shortened relative to current directory
- **Line**: Line number where match was found (omitted for vector-only results)
- **Snippet**: Context around match with query terms highlighted
### Examples
```sh
# Get 10 results with minimum score 0.3
qmd query -n 10 --min-score 0.3 "API design patterns"
# Output as markdown for LLM context
qmd search -md --full "error handling"
# Use separate index for different knowledge base
qmd --index work search "quarterly reports"
```
### Manage Collections
```sh
# List all indexed collections
qmd list
# Show database statistics
qmd stats
# Forget a collection
qmd forget
```
## Data Storage
Index stored in: `~/.cache/qmd/index.sqlite`
### Schema
```sql
collections -- Indexed directories and glob patterns
documents -- Markdown content with metadata
documents_fts -- FTS5 full-text index
content_vectors -- Embedding cache (by content hash)
vectors_vec -- sqlite-vec vector index
```
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `OLLAMA_URL` | `http://localhost:11434` | Ollama API endpoint |
| `XDG_CACHE_HOME` | `~/.cache` | Cache directory location |
## How It Works
### Indexing Flow
```
Markdown Files ──► Parse Title ──► Hash Content ──► Store in SQLite
│ │
└─► FTS5 Index ◄─────────────────────┘
```
### Embedding Flow
```
Document ──► Format for EmbeddingGemma ──► Ollama API ──► Store Vector
"title: X | text: Y" /api/embed
```
### Query Flow (Hybrid)
```
Query ──► Expand (3 variations) ──► FTS + Vector (per variation)
Merge (max score)
Top 25 candidates
LLM Re-rank (0-10)
Final ranked results
```
## Model Configuration
Models are configured as constants in `qmd.ts`:
```typescript
const DEFAULT_EMBED_MODEL = "embeddinggemma";
const DEFAULT_RERANK_MODEL = "ExpedientFalcon/qwen3-reranker:0.6b-q8_0";
const DEFAULT_QUERY_MODEL = "qwen3:0.6b";
```
### EmbeddingGemma Prompt Format
```
// For queries
"task: search result | query: {query}"
// For documents
"title: {title} | text: {content}"
```
### Qwen3-Reranker
A dedicated reranker model trained on relevance classification:
```
System: Judge whether the Document meets the requirements based on the Query
and the Instruct provided. Note that the answer can only be "yes" or "no".
User: <Instruct>: Given a search query, determine if the document is relevant...
<Query>: {query}
<Document>: {doc}
```
- Uses `logprobs: true` to extract token probabilities
- Outputs yes/no with confidence score (0.0 - 1.0)
- `num_predict: 1` - Only need the yes/no token
### Qwen3 (Query Expansion)
- `num_predict: 150` - For generating query variations
## License
MIT

47
bun.lock Normal file
View File

@ -0,0 +1,47 @@
{
"lockfileVersion": 1,
"configVersion": 1,
"workspaces": {
"": {
"name": "2025-12-07-bm25-q",
"dependencies": {
"sqlite-vec": "^0.1.7-alpha.2",
},
"devDependencies": {
"@types/bun": "latest",
},
"optionalDependencies": {
"sqlite-vec-darwin-arm64": "^0.1.7-alpha.2",
"sqlite-vec-darwin-x64": "^0.1.7-alpha.2",
"sqlite-vec-linux-x64": "^0.1.7-alpha.2",
"sqlite-vec-win32-x64": "^0.1.7-alpha.2",
},
"peerDependencies": {
"typescript": "^5",
},
},
},
"packages": {
"@types/bun": ["@types/bun@1.3.3", "", { "dependencies": { "bun-types": "1.3.3" } }, "sha512-ogrKbJ2X5N0kWLLFKeytG0eHDleBYtngtlbu9cyBKFtNL3cnpDZkNdQj8flVf6WTZUX5ulI9AY1oa7ljhSrp+g=="],
"@types/node": ["@types/node@24.10.1", "", { "dependencies": { "undici-types": "~7.16.0" } }, "sha512-GNWcUTRBgIRJD5zj+Tq0fKOJ5XZajIiBroOF0yvj2bSU1WvNdYS/dn9UxwsujGW4JX06dnHyjV2y9rRaybH0iQ=="],
"bun-types": ["bun-types@1.3.3", "", { "dependencies": { "@types/node": "*" } }, "sha512-z3Xwlg7j2l9JY27x5Qn3Wlyos8YAp0kKRlrePAOjgjMGS5IG6E7Jnlx736vH9UVI4wUICwwhC9anYL++XeOgTQ=="],
"sqlite-vec": ["sqlite-vec@0.1.7-alpha.2", "", { "optionalDependencies": { "sqlite-vec-darwin-arm64": "0.1.7-alpha.2", "sqlite-vec-darwin-x64": "0.1.7-alpha.2", "sqlite-vec-linux-arm64": "0.1.7-alpha.2", "sqlite-vec-linux-x64": "0.1.7-alpha.2", "sqlite-vec-windows-x64": "0.1.7-alpha.2" } }, "sha512-rNgRCv+4V4Ed3yc33Qr+nNmjhtrMnnHzXfLVPeGb28Dx5mmDL3Ngw/Wk8vhCGjj76+oC6gnkmMG8y73BZWGBwQ=="],
"sqlite-vec-darwin-arm64": ["sqlite-vec-darwin-arm64@0.1.7-alpha.2", "", { "os": "darwin", "cpu": "arm64" }, "sha512-raIATOqFYkeCHhb/t3r7W7Cf2lVYdf4J3ogJ6GFc8PQEgHCPEsi+bYnm2JT84MzLfTlSTIdxr4/NKv+zF7oLPw=="],
"sqlite-vec-darwin-x64": ["sqlite-vec-darwin-x64@0.1.7-alpha.2", "", { "os": "darwin", "cpu": "x64" }, "sha512-jeZEELsQjjRsVojsvU5iKxOvkaVuE+JYC8Y4Ma8U45aAERrDYmqZoHvgSG7cg1PXL3bMlumFTAmHynf1y4pOzA=="],
"sqlite-vec-linux-arm64": ["sqlite-vec-linux-arm64@0.1.7-alpha.2", "", { "os": "linux", "cpu": "arm64" }, "sha512-6Spj4Nfi7tG13jsUG+W7jnT0bCTWbyPImu2M8nWp20fNrd1SZ4g3CSlDAK8GBdavX7wRlbBHCZ+BDa++rbDewA=="],
"sqlite-vec-linux-x64": ["sqlite-vec-linux-x64@0.1.7-alpha.2", "", { "os": "linux", "cpu": "x64" }, "sha512-IcgrbHaDccTVhXDf8Orwdc2+hgDLAFORl6OBUhcvlmwswwBP1hqBTSEhovClG4NItwTOBNgpwOoQ7Qp3VDPWLg=="],
"sqlite-vec-windows-x64": ["sqlite-vec-windows-x64@0.1.7-alpha.2", "", { "os": "win32", "cpu": "x64" }, "sha512-TRP6hTjAcwvQ6xpCZvjP00pdlda8J38ArFy1lMYhtQWXiIBmWnhMaMbq4kaeCYwvTTddfidatRS+TJrwIKB/oQ=="],
"typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="],
"undici-types": ["undici-types@7.16.0", "", {}, "sha512-Zz+aZWSj8LE6zoxD+xrjh4VfkIG8Ya6LvYkZqtUQGJPZjYl53ypCaUwWqo7eI0x66KBGeRo+mlBEkMSeSZ38Nw=="],
}
}

47
package.json Normal file
View File

@ -0,0 +1,47 @@
{
"name": "qmd",
"version": "1.0.0",
"description": "Quick Markdown Search - Full-text and vector search for markdown files",
"type": "module",
"bin": {
"qmd": "./qmd"
},
"scripts": {
"qmd": "bun qmd.ts",
"index": "bun qmd.ts index",
"vector": "bun qmd.ts vector",
"search": "bun qmd.ts search",
"vsearch": "bun qmd.ts vsearch",
"rerank": "bun qmd.ts rerank",
"link": "bun link"
},
"dependencies": {
"sqlite-vec": "^0.1.7-alpha.2"
},
"optionalDependencies": {
"sqlite-vec-darwin-arm64": "^0.1.7-alpha.2",
"sqlite-vec-darwin-x64": "^0.1.7-alpha.2",
"sqlite-vec-linux-x64": "^0.1.7-alpha.2",
"sqlite-vec-win32-x64": "^0.1.7-alpha.2"
},
"devDependencies": {
"@types/bun": "latest"
},
"peerDependencies": {
"typescript": "^5"
},
"engines": {
"bun": ">=1.0.0"
},
"keywords": [
"markdown",
"search",
"fts",
"vector",
"sqlite",
"bm25",
"embeddings",
"ollama"
],
"license": "MIT"
}

14
qmd Executable file
View File

@ -0,0 +1,14 @@
#!/bin/bash
# qmd - Quick Markdown Search
# Run with: ./qmd or symlink to PATH
# Resolve symlinks to find actual script location
SOURCE="${BASH_SOURCE[0]}"
while [ -L "$SOURCE" ]; do
DIR="$(cd -P "$(dirname "$SOURCE")" && pwd)"
SOURCE="$(readlink "$SOURCE")"
[[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
done
SCRIPT_DIR="$(cd -P "$(dirname "$SOURCE")" && pwd)"
exec bun "$SCRIPT_DIR/qmd.ts" "$@"

1250
qmd.ts Executable file

File diff suppressed because it is too large Load Diff

29
tsconfig.json Normal file
View File

@ -0,0 +1,29 @@
{
"compilerOptions": {
// Environment setup & latest features
"lib": ["ESNext"],
"target": "ESNext",
"module": "Preserve",
"moduleDetection": "force",
"jsx": "react-jsx",
"allowJs": true,
// Bundler mode
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"verbatimModuleSyntax": true,
"noEmit": true,
// Best practices
"strict": true,
"skipLibCheck": true,
"noFallthroughCasesInSwitch": true,
"noUncheckedIndexedAccess": true,
"noImplicitOverride": true,
// Some stricter flags (disabled by default)
"noUnusedLocals": false,
"noUnusedParameters": false,
"noPropertyAccessFromIndexSignature": false
}
}