Convert emoji codepoints to hex representation (e.g. 🐘 → 1f418) instead
of crashing, so files like 🐘.md can be indexed without halting the
entire update process.
Fixes#302
Add an optional 'ignore' field to collection config that accepts an array
of glob patterns to exclude from indexing. This allows collections to skip
specific subdirectories without needing separate collections.
Example YAML config:
personal:
path: ~/personal_synced
pattern: '**/*.md'
ignore:
- 'Sessions/**'
- 'archive/**'
The ignore patterns are passed to fast-glob's ignore option alongside the
existing hardcoded excludes (node_modules, .git, etc). Already-indexed
files matching new ignore patterns are deactivated on the next update.
Changes:
- Add ignore?: string[] to Collection interface
- Pass ignore patterns through to fast-glob in indexFiles()
- Show ignore patterns in collection list/status output
- 5 new CLI integration tests covering the feature
- Compound entity chaining now stops one level deep. Previously "TDS
motorsports team history" would inflate the expected entity set with
"team" and "history", causing false-positive entity-preservation
penalties during GRPO. Now only {tds, motorsports} are detected.
- Add INTERIOR_FILLER_WORDS penalty (-3/line): lex lines containing
"overview" or "basics" absent from the original query are penalised.
Targets template-generator noise, e.g. "ancient overview rome timeline".
- Raise is_diverse threshold 2→3: requires 3 unique words between lex
lines before they count as diverse. Reduces reward for near-duplicate
pairs like "auth setup" / "auth configuration".
- Broaden quoted-phrase bonus: was gated on named entities existing;
now any multi-word query earns +3 for using quotes in lex lines.
Better incentivises BM25-aware syntax like "memory leak" python.
Fixes scoring noise identified while working on issue #247.
node-llama-cpp throws a hard error when any document + query + template
overhead exceeds the ranking context size. Truncate oversized documents
using the rerank model's tokenizer before passing them to rankAll().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace ad-hoc JSON parsing with a strict Pydantic model
(TrainingExample with typed OutputPair). All data loading goes
through load_examples() which fails loudly on invalid data.
- Convert v3_structured.jsonl from "searches" to "output" format
- Rewrite all consumer scripts (prepare, validate, score, analyze)
to load through the Pydantic schema
- Prepared train/val files are ephemeral build artifacts
- Restore LFM2 and GEPA experiments under experiments/
- Add pydantic>=2.0 to dependencies
Training data:
- Expand lex phrases/negation examples from 12 to 74 with intent field
- Add 50 personal entity examples (meetings, emails, projects with names)
Reward function:
- Detect entities at position 0 (fixes "Bob asked about deploy")
- Per-entity coverage penalty: -20 per entity absent from all lex+vec
- Phrase quoting bonus: +3 when lex uses quotes for multi-word terms
- Expanded stopwords to reduce false positive entity detection
Eval queries: add 21 test queries for personal entities, quoted phrases,
and negation/disambiguation scenarios.
Remove one-off data generator/fix scripts, superseded data files (v2, v3
replaced by v3_structured), LFM2 experiment, GEPA directory, duplicate
job scripts, and historical docs. Clean up Justfile.
These are restored under experiments/ in a later commit.
Replaces the inner test script with an outer driver that runs individual
podman/docker commands against a pre-built image. Tests sqlite-vec
loading and store unit tests under both node and bun runtimes.
Supports --build (image only), --shell (interactive), and -- CMD
(arbitrary command) for debugging install issues in isolation.
The qmd bin was a custom bash script that discovered node via hardcoded
fallback paths (mise, asdf, nvm, homebrew). This was nonstandard and
caused ABI mismatches when installed via bun (native modules compiled
for bun but executed with node).
Now uses the standard npm bin convention: dist/qmd.js with a node
shebang, added by the build script. The isMain guard resolves symlinks
so it works when npm/bun create symlinked bin entries.
Also converts all dynamic require() calls in tests to ESM imports, and
adds container-based smoke tests (test/smoke-install.sh) that verify
install + run under both node and bun via mise in a Debian container.
The 'query document' is now a first-class concept in QMD: a structured
document with typed sub-queries that combine for best recall.
## Query types
- lex: BM25 keyword search with phrase and negation syntax
- vec: Semantic vector search (natural language questions)
- hyde: Hypothetical document (write the expected answer)
- expand: Auto-expand via local LLM (max 1, default for plain queries)
## Lex syntax
Full BM25 operator support:
"exact phrase" verbatim match, no prefix
-term exclude documents containing term
-"exact phrase" exclude documents containing phrase
Examples:
"C++ performance" optimization -sports -athlete
"connection pool" timeout -redis
"machine learning" -sports -athlete
## MCP tool description rewritten
The 'query' tool description now fully teaches AI agents the query
document format, lex syntax, and strategy for combining types.
Includes worked examples including intent-aware lex (C++ performance,
not sports) which is critical for disambiguation in dense corpora.
## Unit tests
11 new lex parser tests covering:
- plain terms, quoted phrases, negation, combined
- intent-aware disambiguation (performance -sports -athlete)
- only-negation returns null (FTS5 constraint)
- empty/whitespace handling
## Training data
12 new intent-aware examples for next model training round:
- Real technical topics with lex phrase+negation combinations
- Covers: C++ perf, Python memory, DB connections, rate limiting,
SQL optimization, ML overfitting, Docker, JWT, async/await,
git conflicts, Kubernetes, React state
- Each shows how context/intent shapes lex query construction
(e.g. performance with C++ context → -sports -athlete exclusions)
New collection subcommands:
- show <name> Show collection details
- update-cmd <name> [cmd] Set pre-update command (runs before indexing)
- include <name> Include in default queries (default)
- exclude <name> Exclude from default queries
Collections with includeByDefault=false are skipped unless
explicitly named with -c flag.
CLI improvements:
- 'qmd collection' shows help instead of error
- 'qmd collection list' shows [excluded] tag
- Better command descriptions and examples
Lex queries now support:
- "exact phrase" - quoted exact matching (no prefix)
- -term or -"phrase" - exclude from results
- term1 OR term2 - match either term
Semantic queries (vec/hyde) validate and reject these operators
with helpful error messages.
Examples:
performance -sports → matches "performance" excluding "sports"
"machine learning" → exact phrase match
auth OR authentication → matches either term
- structured_search now accepts collections[] for OR filtering
- Updated skill docs with detailed query writing guidance
- lex: 2-5 keywords, include synonyms, exact names
- vec: full natural language questions with context
- hyde: 50-100 word hypothetical answer passages
BREAKING CHANGE: MCP tools search, vector_search, deep_search removed.
Use structured_search with lex/vec/hyde queries instead.
- Remove search, vector_search, deep_search MCP tool registrations
- Update MCP instructions to focus on structured_search
- Update skill docs to reflect simplified API
- Rename test describes to reflect they test store functions
- CLI commands (qmd search, vsearch, query) unchanged for backwards compat
