Replaces the inner test script with an outer driver that runs individual
podman/docker commands against a pre-built image. Tests sqlite-vec
loading and store unit tests under both node and bun runtimes.
Supports --build (image only), --shell (interactive), and -- CMD
(arbitrary command) for debugging install issues in isolation.
The qmd bin was a custom bash script that discovered node via hardcoded
fallback paths (mise, asdf, nvm, homebrew). This was nonstandard and
caused ABI mismatches when installed via bun (native modules compiled
for bun but executed with node).
Now uses the standard npm bin convention: dist/qmd.js with a node
shebang, added by the build script. The isMain guard resolves symlinks
so it works when npm/bun create symlinked bin entries.
Also converts all dynamic require() calls in tests to ESM imports, and
adds container-based smoke tests (test/smoke-install.sh) that verify
install + run under both node and bun via mise in a Debian container.
The 'query document' is now a first-class concept in QMD: a structured
document with typed sub-queries that combine for best recall.
## Query types
- lex: BM25 keyword search with phrase and negation syntax
- vec: Semantic vector search (natural language questions)
- hyde: Hypothetical document (write the expected answer)
- expand: Auto-expand via local LLM (max 1, default for plain queries)
## Lex syntax
Full BM25 operator support:
"exact phrase" verbatim match, no prefix
-term exclude documents containing term
-"exact phrase" exclude documents containing phrase
Examples:
"C++ performance" optimization -sports -athlete
"connection pool" timeout -redis
"machine learning" -sports -athlete
## MCP tool description rewritten
The 'query' tool description now fully teaches AI agents the query
document format, lex syntax, and strategy for combining types.
Includes worked examples including intent-aware lex (C++ performance,
not sports) which is critical for disambiguation in dense corpora.
## Unit tests
11 new lex parser tests covering:
- plain terms, quoted phrases, negation, combined
- intent-aware disambiguation (performance -sports -athlete)
- only-negation returns null (FTS5 constraint)
- empty/whitespace handling
## Training data
12 new intent-aware examples for next model training round:
- Real technical topics with lex phrase+negation combinations
- Covers: C++ perf, Python memory, DB connections, rate limiting,
SQL optimization, ML overfitting, Docker, JWT, async/await,
git conflicts, Kubernetes, React state
- Each shows how context/intent shapes lex query construction
(e.g. performance with C++ context → -sports -athlete exclusions)
New collection subcommands:
- show <name> Show collection details
- update-cmd <name> [cmd] Set pre-update command (runs before indexing)
- include <name> Include in default queries (default)
- exclude <name> Exclude from default queries
Collections with includeByDefault=false are skipped unless
explicitly named with -c flag.
CLI improvements:
- 'qmd collection' shows help instead of error
- 'qmd collection list' shows [excluded] tag
- Better command descriptions and examples
Lex queries now support:
- "exact phrase" - quoted exact matching (no prefix)
- -term or -"phrase" - exclude from results
- term1 OR term2 - match either term
Semantic queries (vec/hyde) validate and reject these operators
with helpful error messages.
Examples:
performance -sports → matches "performance" excluding "sports"
"machine learning" → exact phrase match
auth OR authentication → matches either term
- structured_search now accepts collections[] for OR filtering
- Updated skill docs with detailed query writing guidance
- lex: 2-5 keywords, include synonyms, exact names
- vec: full natural language questions with context
- hyde: 50-100 word hypothetical answer passages
BREAKING CHANGE: MCP tools search, vector_search, deep_search removed.
Use structured_search with lex/vec/hyde queries instead.
- Remove search, vector_search, deep_search MCP tool registrations
- Update MCP instructions to focus on structured_search
- Update skill docs to reflect simplified API
- Rename test describes to reflect they test store functions
- CLI commands (qmd search, vsearch, query) unchanged for backwards compat
Lines prefixed with lex:, vec:, or hyde: route directly to
structured search, skipping automatic query expansion.
Examples:
qmd query 'lex: CAP theorem'
qmd query $'lex: keywords\nvec: natural language question'
qmd query $'lex: terms\nvec: question\nhyde: hypothetical answer...'
Plain queries (single line, no prefix) still use automatic expansion.
Multiple plain lines without prefixes error with helpful message.
This lets CLI users leverage the same structured search as MCP,
useful when piping from scripts or when you know exactly what
query variations you want.
- New MCP tool: structured_search - lets capable LLMs provide their own
lex/vec/hyde query variations instead of using local expansion model
- New REST endpoint: POST /search - same functionality without MCP protocol
- Updated skill docs to prioritize structured_search for LLM callers
- Added installation instructions for Claude Code, Desktop, and OpenClaw
Pipeline: lex→FTS, vec/hyde→batch embed, RRF fusion (first query 2x weight),
chunk + rerank, position-aware blending, dedup.
This is the recommended endpoint for capable LLMs - they generate better
query variations than the small local model, especially for domain-specific
or nuanced queries.
collections-config.test.ts set currentIndexName to "myindex" in its
last test but only restored env vars in afterEach — not the module
variable. Under bun test (single process), this leaked into mcp.test.ts,
causing it to look for myindex.yml instead of index.yml.
Fix: reset setConfigIndexName("index") in afterEach, and add defensive
reset in mcp.test.ts beforeAll.
node-gyp needs python3 to compile better-sqlite3, and on macOS it also
needs libtool from cctools to create static libraries. Without these,
`nix build` fails in the sandbox.
Previously, 'qmd search --json' would output plain text 'No results found.'
when no matches were found, which is invalid JSON. Now it correctly outputs
an empty JSON array [] when using --json format.
Fixed in all search commands: search, vsearch, and query.
When using --index with relative paths like './index/my-project', the path was
stored directly as the config filename, resulting in paths like:
/home/user/.config/qmd/./index/my-project.yml
This caused 'no such file or directory' errors. Now relative paths are resolved
and normalized by replacing path separators with underscores.
Dataset improvements:
- Reorder output to put hyde first for better retrieval priming
- Convert absolute paths to relative paths in scripts
- Add convert_to_structured.py for structured data format
- Add qmd_expansion_v3_structured.jsonl with type/query objects
- Update schema.py with reorder_hyde_first() helper
- Verify data now validates hyde-first ordering
Training data regenerated with new ordering (100% validation success).
Add training configuration and documentation for using LiquidAI's LFM2-1.2B
as an alternative base model for query expansion fine-tuning.
LFM2 benefits:
- 2x faster decode/prefill vs standard transformers
- Optimized for edge/on-device inference
- Good at agentic tasks, RAG, and data extraction
Changes:
- Add configs/sft_lfm2.yaml with LFM2-specific LoRA target modules
- Add jobs/sft_lfm2.py for HuggingFace Jobs training
- Update llm.ts with LFM2 GGUF model URIs
- Add documentation for LFM2 training workflow
LFM2 uses a hybrid architecture (convolutions + attention) requiring
different LoRA targets: q_proj, k_proj, v_proj, out_proj, in_proj, w1, w2, w3
Dataset improvements:
- fix_hyde.py: Replace generic template hyde entries with query-specific ones
using GPT-4o-mini (removed 'comprehensive guide covers everything' pattern)
- fix_lex_filler.py: Remove filler words (overview, tutorial, guide, examples,
documentation, best practices) that were padding rather than genuine search intent
- qmd_expansion_v3.jsonl: Improved dataset with 1,498 high-quality entries
Training data preparation:
- convert_to_chatml.py: Convert to ChatML format for LFM2.5 training
- verify_data.py: Validation script to ensure data quality
- train-lfm2/: Ready-to-use training data (90/10 train/val split)
Data quality metrics:
- 100% success rate (all entries properly formatted)
- Query length: 6-65 chars (avg: 29.3)
- Response length: 307-777 chars (avg: 539.5)
- All entries contain lex, vec, and hyde expansions
- Move model info from --help to `qmd status` with live HuggingFace
links derived from actual configured URIs
- Pre-push hook: handle non-interactive shells gracefully, resolve
annotated tags correctly for CI checks
