Add optional `intent` parameter that steers query expansion, reranking,
chunk selection, and snippet extraction without searching on its own.
When a query like "performance" is ambiguous (web-perf vs team health vs
fitness), intent provides background context that disambiguates results
across all pipeline stages:
- expandQuery: includes intent in LLM prompt ("Query intent: {intent}")
- rerank: prepends intent to rerank query for Qwen3-Reranker
- chunk selection: intent terms scored at 0.5x weight vs query terms
- snippet extraction: intent terms scored at 0.3x weight
- strong-signal bypass: disabled when intent provided
Available via CLI (--intent flag or intent: line in query documents),
MCP (intent field on query tool), and programmatic API.
Adapted from PR #180 (thanks @vyalamar).
- Cap rerank contexts at 4 to avoid VRAM exhaustion on high-core machines
- Deduplicate identical chunk texts before sending to reranker
- Cache rerank scores by chunk content instead of file path — same text
from different files now shares a single reranker call
- Add truncation cache to avoid re-tokenizing duplicate documents
Convert emoji codepoints to hex representation (e.g. 🐘 → 1f418) instead
of crashing, so files like 🐘.md can be indexed without halting the
entire update process.
Fixes#302
Add an optional 'ignore' field to collection config that accepts an array
of glob patterns to exclude from indexing. This allows collections to skip
specific subdirectories without needing separate collections.
Example YAML config:
personal:
path: ~/personal_synced
pattern: '**/*.md'
ignore:
- 'Sessions/**'
- 'archive/**'
The ignore patterns are passed to fast-glob's ignore option alongside the
existing hardcoded excludes (node_modules, .git, etc). Already-indexed
files matching new ignore patterns are deactivated on the next update.
Changes:
- Add ignore?: string[] to Collection interface
- Pass ignore patterns through to fast-glob in indexFiles()
- Show ignore patterns in collection list/status output
- 5 new CLI integration tests covering the feature
The HTTP MCP server creates a single Transport + McpServer pair at
startup. Once the first client initializes, all subsequent clients
are rejected with "Server already initialized" — making the HTTP
mode unusable for reconnect, crash recovery, or multi-client scenarios.
Replace the singleton with a per-session architecture: each initialize
request creates its own McpServer + Transport pair, stored in a
sessions Map keyed by session ID. The shared Store (SQLite) is
stateless and safe for concurrent access.
Key changes:
- createSession() factory creates fresh McpServer + Transport per client
- POST /mcp routes by mcp-session-id header to existing sessions
- New initialize requests (no session header) create new sessions
- Unknown session IDs return 404 per MCP spec
- Missing session IDs return 400
- onsessioninitialized callback stores sessions at the right time
- transport.onclose cleans up the sessions Map
- Shutdown iterates all active sessions
Tested with 3+ concurrent clients, session cleanup via DELETE,
cross-session isolation, and rapid session creation.
Fixes#195Closes#163
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The default embeddinggemma-300M model is English-centric and produces
poor embeddings for CJK (Chinese, Japanese, Korean) text. This change
allows overriding the embedding model via the QMD_EMBED_MODEL environment
variable.
Changes:
- DEFAULT_EMBED_MODEL now reads from QMD_EMBED_MODEL env var (fallback to
embeddinggemma-300M for backward compatibility)
- getDefaultLlamaCpp() passes QMD_EMBED_MODEL to LlamaCpp config when set
- formatQueryForEmbedding() and formatDocForEmbedding() detect Qwen3-Embedding
models and apply the correct prompt format (Qwen3 uses task-instruction
format; embeddinggemma uses nomic-style prefix format)
- store.ts: pass model URI to format functions so format selection is
consistent between indexing and query time
- README: document QMD_EMBED_MODEL with Qwen3-Embedding example
Recommended multilingual model:
QMD_EMBED_MODEL=hf:Qwen/Qwen3-Embedding-0.6B-GGUF/qwen3-embedding-0.6b-q8_0.gguf
After changing the model, run: qmd embed -f
- Compound entity chaining now stops one level deep. Previously "TDS
motorsports team history" would inflate the expected entity set with
"team" and "history", causing false-positive entity-preservation
penalties during GRPO. Now only {tds, motorsports} are detected.
- Add INTERIOR_FILLER_WORDS penalty (-3/line): lex lines containing
"overview" or "basics" absent from the original query are penalised.
Targets template-generator noise, e.g. "ancient overview rome timeline".
- Raise is_diverse threshold 2→3: requires 3 unique words between lex
lines before they count as diverse. Reduces reward for near-duplicate
pairs like "auth setup" / "auth configuration".
- Broaden quoted-phrase bonus: was gated on named entities existing;
now any multi-word query earns +3 for using quotes in lex lines.
Better incentivises BM25-aware syntax like "memory leak" python.
Fixes scoring noise identified while working on issue #247.
Reranking 40 chunks takes ~2 min on CPU (the default candidateLimit).
The option already exists in hybridQuery()/structuredSearch() but was
never surfaced to users. This adds:
- `candidateLimit` param to the MCP `query` tool inputSchema
- `candidateLimit` field to the REST /query endpoint
- `--candidate-limit` / `-C` CLI flag for `qmd query`
Default stays 40 (no behavior change). Users on CPU-only machines can
lower it for a speed/recall tradeoff. Complements #231.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On macOS with iCloud Drive (especially shared folders), some files may
appear in the filesystem but return EAGAIN (error -11) when read via
Node's readFileSync. This happens when iCloud has evicted the file
content but the file metadata remains visible.
Previously this crashed the entire update process. Now we catch the
error and skip the file, allowing the remaining files to index
successfully.
Affects: iCloud Drive shared folders on macOS
Error: 'Unknown system error -11: Unknown system error -11, read'
Reproduces with: Node.js v25.x, readFileSync on evicted iCloud files
node-llama-cpp throws a hard error when any document + query + template
overhead exceeds the ranking context size. Truncate oversized documents
using the rerank model's tokenizer before passing them to rankAll().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace ad-hoc JSON parsing with a strict Pydantic model
(TrainingExample with typed OutputPair). All data loading goes
through load_examples() which fails loudly on invalid data.
- Convert v3_structured.jsonl from "searches" to "output" format
- Rewrite all consumer scripts (prepare, validate, score, analyze)
to load through the Pydantic schema
- Prepared train/val files are ephemeral build artifacts
- Restore LFM2 and GEPA experiments under experiments/
- Add pydantic>=2.0 to dependencies
Training data:
- Expand lex phrases/negation examples from 12 to 74 with intent field
- Add 50 personal entity examples (meetings, emails, projects with names)
Reward function:
- Detect entities at position 0 (fixes "Bob asked about deploy")
- Per-entity coverage penalty: -20 per entity absent from all lex+vec
- Phrase quoting bonus: +3 when lex uses quotes for multi-word terms
- Expanded stopwords to reduce false positive entity detection
Eval queries: add 21 test queries for personal entities, quoted phrases,
and negation/disambiguation scenarios.
Remove one-off data generator/fix scripts, superseded data files (v2, v3
replaced by v3_structured), LFM2 experiment, GEPA directory, duplicate
job scripts, and historical docs. Clean up Justfile.
These are restored under experiments/ in a later commit.
Replaces the inner test script with an outer driver that runs individual
podman/docker commands against a pre-built image. Tests sqlite-vec
loading and store unit tests under both node and bun runtimes.
Supports --build (image only), --shell (interactive), and -- CMD
(arbitrary command) for debugging install issues in isolation.
