docs: rewrite SDK section for 2.0, fix MCP tool names, add changelog
- Expand SDK documentation from ~70 lines to comprehensive coverage: store creation modes, unified search(), retrieval, collections, context, indexing, types, and lifecycle - Fix MCP tools section: old names (qmd_search, qmd_deep_search) replaced with actual registered names (query, get, multi_get, status) - Write 2.0.0 changelog under [Unreleased]
This commit is contained in:
Tobi Lutke
parent
b252219add
commit
a444c86382
23
CHANGELOG.md
23
CHANGELOG.md
@ -2,6 +2,29 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
QMD 2.0 declares a stable library API. The SDK is now the primary interface —
|
||||
the MCP server is a clean consumer of it, and the source is organized into
|
||||
`src/cli/` and `src/mcp/`. Also: Node 25 support and a runtime-aware bin wrapper
|
||||
for bun installs.
|
||||
|
||||
### Changes
|
||||
|
||||
- Stable SDK API with `QMDStore` interface — search, retrieval, collection/context
|
||||
management, indexing, lifecycle
|
||||
- Unified `search()`: pass `query` for auto-expansion or `queries` for
|
||||
pre-expanded lex/vec/hyde — replaces the old query/search/structuredSearch split
|
||||
- New `getDocumentBody()`, `getDefaultCollectionNames()`, `Maintenance` class
|
||||
- MCP server rewritten as a clean SDK consumer — zero internal store access
|
||||
- CLI and MCP organized into `src/cli/` and `src/mcp/` subdirectories
|
||||
- Runtime-aware `bin/qmd` wrapper detects bun vs node to avoid ABI mismatches.
|
||||
Closes #319
|
||||
- `better-sqlite3` bumped to ^12.4.5 for Node 25 support. Closes #257
|
||||
- Utility exports: `extractSnippet`, `addLineNumbers`, `DEFAULT_MULTI_GET_MAX_BYTES`
|
||||
|
||||
### Fixes
|
||||
|
||||
- Remove unused `import { resolve }` in store.ts that shadowed local export
|
||||
|
||||
## [1.1.6] - 2026-03-09
|
||||
|
||||
QMD can now be used as a library. `import { createStore } from '@tobilu/qmd'`
|
||||
|
||||
253
README.md
253
README.md
@ -74,12 +74,10 @@ qmd get "docs/api-reference.md" --full
|
||||
Although the tool works perfectly fine when you just tell your agent to use it on the command line, it also exposes an MCP (Model Context Protocol) server for tighter integration.
|
||||
|
||||
**Tools exposed:**
|
||||
- `qmd_search` - Fast BM25 keyword search (supports collection filter)
|
||||
- `qmd_vector_search` - Semantic vector search (supports collection filter)
|
||||
- `qmd_deep_search` - Deep search with query expansion and reranking (supports collection filter)
|
||||
- `qmd_get` - Retrieve document by path or docid (with fuzzy matching suggestions)
|
||||
- `qmd_multi_get` - Retrieve multiple documents by glob pattern, list, or docids
|
||||
- `qmd_status` - Index health and collection info
|
||||
- `query` — Search with typed sub-queries (`lex`/`vec`/`hyde`), combined via RRF + reranking
|
||||
- `get` — Retrieve a document by path or docid (with fuzzy matching suggestions)
|
||||
- `multi_get` — Batch retrieve by glob pattern, comma-separated list, or docids
|
||||
- `status` — Index health and collection info
|
||||
|
||||
**Claude Desktop configuration** (`~/Library/Application Support/Claude/claude_desktop_config.json`):
|
||||
|
||||
@ -139,78 +137,239 @@ Point any MCP client at `http://localhost:8181/mcp` to connect.
|
||||
|
||||
### SDK / Library Usage
|
||||
|
||||
Use QMD as a library in your own Node.js or Bun applications:
|
||||
Use QMD as a library in your own Node.js or Bun applications.
|
||||
|
||||
#### Installation
|
||||
|
||||
```sh
|
||||
npm install @tobilu/qmd
|
||||
```
|
||||
|
||||
#### Quick Start
|
||||
|
||||
```typescript
|
||||
import { createStore } from '@tobilu/qmd'
|
||||
|
||||
// Create a store with inline config (no config file needed)
|
||||
const store = createStore({
|
||||
const store = await createStore({
|
||||
dbPath: './my-index.sqlite',
|
||||
config: {
|
||||
collections: {
|
||||
docs: { path: '/path/to/docs', pattern: '**/*.md' },
|
||||
notes: { path: '/path/to/notes', pattern: '**/*.md' },
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
// Or reference a YAML config file
|
||||
const store2 = createStore({
|
||||
dbPath: './my-index.sqlite',
|
||||
const results = await store.search({ query: "authentication flow" })
|
||||
console.log(results.map(r => `${r.title} (${Math.round(r.score * 100)}%)`))
|
||||
|
||||
await store.close()
|
||||
```
|
||||
|
||||
#### Store Creation
|
||||
|
||||
`createStore()` accepts three modes:
|
||||
|
||||
```typescript
|
||||
import { createStore } from '@tobilu/qmd'
|
||||
|
||||
// 1. Inline config — no files needed besides the DB
|
||||
const store = await createStore({
|
||||
dbPath: './index.sqlite',
|
||||
config: {
|
||||
collections: {
|
||||
docs: { path: '/path/to/docs', pattern: '**/*.md' },
|
||||
notes: { path: '/path/to/notes' },
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
// 2. YAML config file — collections defined in a file
|
||||
const store2 = await createStore({
|
||||
dbPath: './index.sqlite',
|
||||
configPath: './qmd.yml',
|
||||
})
|
||||
|
||||
// 3. DB-only — reopen a previously configured store
|
||||
const store3 = await createStore({ dbPath: './index.sqlite' })
|
||||
```
|
||||
|
||||
#### Search
|
||||
|
||||
The unified `search()` method handles both simple queries and pre-expanded structured queries:
|
||||
|
||||
```typescript
|
||||
// Simple query — auto-expanded via LLM, then BM25 + vector + reranking
|
||||
const results = await store.search({ query: "authentication flow" })
|
||||
|
||||
// With options
|
||||
const results2 = await store.search({
|
||||
query: "rate limiting",
|
||||
intent: "API throttling and abuse prevention",
|
||||
collection: "docs",
|
||||
limit: 5,
|
||||
minScore: 0.3,
|
||||
explain: true,
|
||||
})
|
||||
|
||||
// Pre-expanded queries — skip auto-expansion, control each sub-query
|
||||
const results3 = await store.search({
|
||||
queries: [
|
||||
{ type: 'lex', query: '"connection pool" timeout -redis' },
|
||||
{ type: 'vec', query: 'why do database connections time out under load' },
|
||||
],
|
||||
collections: ["docs", "notes"],
|
||||
})
|
||||
|
||||
// Skip reranking for faster results
|
||||
const fast = await store.search({ query: "auth", rerank: false })
|
||||
```
|
||||
|
||||
For direct backend access:
|
||||
|
||||
```typescript
|
||||
// BM25 keyword search (fast, no LLM)
|
||||
const lexResults = await store.searchLex("auth middleware", { limit: 10 })
|
||||
|
||||
// Vector similarity search (embedding model, no reranking)
|
||||
const vecResults = await store.searchVector("how users log in", { limit: 10 })
|
||||
|
||||
// Manual query expansion for full control
|
||||
const expanded = await store.expandQuery("auth flow", { intent: "user login" })
|
||||
const results4 = await store.search({ queries: expanded })
|
||||
```
|
||||
|
||||
#### Retrieval
|
||||
|
||||
```typescript
|
||||
// Get a document by path or docid
|
||||
const doc = await store.get("docs/readme.md")
|
||||
const byId = await store.get("#abc123")
|
||||
|
||||
if (!("error" in doc)) {
|
||||
console.log(doc.title, doc.displayPath, doc.context)
|
||||
}
|
||||
|
||||
// Get document body with line range
|
||||
const body = await store.getDocumentBody("docs/readme.md", {
|
||||
fromLine: 50,
|
||||
maxLines: 100,
|
||||
})
|
||||
|
||||
// Batch retrieve by glob or comma-separated list
|
||||
const { docs, errors } = await store.multiGet("docs/**/*.md", {
|
||||
maxBytes: 20480,
|
||||
})
|
||||
```
|
||||
|
||||
**Search & retrieval:**
|
||||
|
||||
```typescript
|
||||
// Hybrid search: BM25 + vector + query expansion + LLM reranking (best quality)
|
||||
const results = await store.query("authentication flow", { limit: 5 })
|
||||
|
||||
// Fast BM25 keyword search (no LLM, synchronous)
|
||||
const keywords = store.search("auth middleware", { limit: 10 })
|
||||
|
||||
// Structured search with pre-expanded queries (for LLM callers)
|
||||
const structured = await store.structuredSearch([
|
||||
{ type: 'lex', query: 'authentication' },
|
||||
{ type: 'vec', query: 'how users log in' },
|
||||
], { limit: 5 })
|
||||
|
||||
// Get a document by path or docid
|
||||
const doc = store.get("docs/readme.md")
|
||||
const byId = store.get("#abc123")
|
||||
|
||||
// Get multiple documents by glob
|
||||
const { docs, errors } = store.multiGet("docs/**/*.md")
|
||||
```
|
||||
|
||||
**Collection & context management:**
|
||||
#### Collections
|
||||
|
||||
```typescript
|
||||
// Add a collection
|
||||
store.addCollection("myapp", { path: "/src/myapp", pattern: "**/*.ts" })
|
||||
await store.addCollection("myapp", {
|
||||
path: "/src/myapp",
|
||||
pattern: "**/*.ts",
|
||||
ignore: ["node_modules/**", "*.test.ts"],
|
||||
})
|
||||
|
||||
// Add context (improves search relevance)
|
||||
store.addContext("myapp", "/auth", "Authentication and session management")
|
||||
store.setGlobalContext("Internal engineering documentation")
|
||||
// List collections with document stats
|
||||
const collections = await store.listCollections()
|
||||
// => [{ name, pwd, glob_pattern, doc_count, active_count, last_modified, includeByDefault }]
|
||||
|
||||
// List everything
|
||||
store.listCollections()
|
||||
store.listContexts()
|
||||
// Get names of collections included in queries by default
|
||||
const defaults = await store.getDefaultCollectionNames()
|
||||
|
||||
// Remove / rename
|
||||
await store.removeCollection("myapp")
|
||||
await store.renameCollection("old-name", "new-name")
|
||||
```
|
||||
|
||||
**Lifecycle:**
|
||||
#### Context
|
||||
|
||||
Context adds descriptive metadata that improves search relevance and is returned alongside results:
|
||||
|
||||
```typescript
|
||||
store.close()
|
||||
// Add context for a path within a collection
|
||||
await store.addContext("docs", "/api", "REST API reference documentation")
|
||||
|
||||
// Set global context (applies to all collections)
|
||||
await store.setGlobalContext("Internal engineering documentation")
|
||||
|
||||
// List all contexts
|
||||
const contexts = await store.listContexts()
|
||||
// => [{ collection, path, context }]
|
||||
|
||||
// Remove context
|
||||
await store.removeContext("docs", "/api")
|
||||
await store.setGlobalContext(undefined) // clear global
|
||||
```
|
||||
|
||||
The SDK requires explicit `dbPath` and config — no defaults are assumed. This makes it safe to embed in any application without side effects.
|
||||
#### Indexing
|
||||
|
||||
```typescript
|
||||
// Re-index collections by scanning the filesystem
|
||||
const result = await store.update({
|
||||
collections: ["docs"], // optional — defaults to all
|
||||
onProgress: ({ collection, file, current, total }) => {
|
||||
console.log(`[${collection}] ${current}/${total} ${file}`)
|
||||
},
|
||||
})
|
||||
// => { collections, indexed, updated, unchanged, removed, needsEmbedding }
|
||||
|
||||
// Generate vector embeddings
|
||||
const embedResult = await store.embed({
|
||||
force: false, // true to re-embed everything
|
||||
onProgress: ({ current, total, collection }) => {
|
||||
console.log(`Embedding ${current}/${total}`)
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
#### Types
|
||||
|
||||
Key types exported for SDK consumers:
|
||||
|
||||
```typescript
|
||||
import type {
|
||||
QMDStore, // The store interface
|
||||
SearchOptions, // Options for search()
|
||||
LexSearchOptions, // Options for searchLex()
|
||||
VectorSearchOptions, // Options for searchVector()
|
||||
HybridQueryResult, // Search result with score, snippet, context
|
||||
SearchResult, // Result from searchLex/searchVector
|
||||
ExpandedQuery, // Typed sub-query { type: 'lex'|'vec'|'hyde', query }
|
||||
DocumentResult, // Document metadata + body
|
||||
DocumentNotFound, // Error with similarFiles suggestions
|
||||
MultiGetResult, // Batch retrieval result
|
||||
UpdateProgress, // Progress callback info for update()
|
||||
UpdateResult, // Aggregated update result
|
||||
EmbedProgress, // Progress callback info for embed()
|
||||
EmbedResult, // Embedding result
|
||||
StoreOptions, // createStore() options
|
||||
CollectionConfig, // Inline config shape
|
||||
IndexStatus, // From getStatus()
|
||||
IndexHealthInfo, // From getIndexHealth()
|
||||
} from '@tobilu/qmd'
|
||||
```
|
||||
|
||||
Utility exports:
|
||||
|
||||
```typescript
|
||||
import {
|
||||
extractSnippet, // Extract a relevant snippet from text
|
||||
addLineNumbers, // Add line numbers to text
|
||||
DEFAULT_MULTI_GET_MAX_BYTES, // Default max file size for multiGet (10KB)
|
||||
Maintenance, // Database maintenance operations
|
||||
} from '@tobilu/qmd'
|
||||
```
|
||||
|
||||
#### Lifecycle
|
||||
|
||||
```typescript
|
||||
// Close the store — disposes LLM models and DB connection
|
||||
await store.close()
|
||||
```
|
||||
|
||||
The SDK requires explicit `dbPath` — no defaults are assumed. This makes it safe to embed in any application without side effects.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user