When the embed model is configured via YAML (not env var), formatDocForEmbedding
and formatQueryForEmbedding callers in store.ts would fall back to the default
model, producing the wrong prompt format. This adds a public embedModelName
getter on LlamaCpp and threads it through all five call sites.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Separate hardcoded default from env var in DEFAULT_EMBED_MODEL so the
constructor can resolve: config param > env var > hardcoded default.
Also add env var support for QMD_GENERATE_MODEL and QMD_RERANK_MODEL.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds QMD_LLAMA_GPU env var (set to false/off/none to force CPU) and
wraps getLlama() in try/catch so Vulkan/CUDA init failures on headless
or driverless machines fall back gracefully instead of crashing the
node process with an uncatchable C++ terminate().
Without an explicit contextSize, node-llama-cpp defaults to "auto" which
allocates the model's full training context (often 32k). For embedding
chunks that are typically ~900 tokens this wastes ~3.5 GB of KV cache
per context on Apple Silicon unified memory.
Default to 2048 (matching the rerank context pattern) and allow override
via QMD_EMBED_CONTEXT_SIZE for users with larger chunks.
Addresses #329, related to #297
Co-authored-by: JohnRichardEnders <john@telli.com>
MODEL_CACHE_DIR was hardcoded to ~/.cache/qmd/models/, ignoring the
XDG_CACHE_HOME environment variable. This was inconsistent with the rest
of the codebase (store.ts, cli/qmd.ts) which already respects XDG paths.
Fixes#425
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default 2048 was too small for longer documents (session transcripts, CJK
text, large markdown files). After truncation the Qwen3 reranker template
adds more overhead than the original 200-token estimate, causing node-llama-cpp
to throw 'input lengths exceed context size'.
Fixes: tobi/qmd#91tobi/qmd#290tobi/qmd#291tobi/qmd#314
When a chunk exceeds the embedding model's context window (trainContextSize),
node-llama-cpp's getEmbeddingFor() triggers a native SIGABRT in GGML/Metal,
crashing the entire process.
Fix: Add truncateToContextSize() guard in embed() and embedBatch() that uses
the model's own tokenizer to check token count before calling getEmbeddingFor().
Oversized text is truncated to (trainContextSize - 4) tokens with a warning,
preserving partial embedding coverage instead of crashing.
Fixes#303
Add _ciMode flag to LlamaCpp that throws immediately on embedBatch,
generate, expandQuery, and rerank when CI=true — prevents silent 30s
timeouts. Skip MCP HTTP Transport tests in CI (they instantiate a real
LlamaCpp). Bump vitest/bun test timeouts to 60s for slower CI runners.
Move frontends into src/cli/ and src/mcp/ to separate them from the
core library. The MCP server is fully rewritten to import only from
the SDK (src/index.ts) — zero direct store.ts/collections.ts/llm.ts
access.
- src/qmd.ts → src/cli/qmd.ts
- src/formatter.ts → src/cli/formatter.ts
- src/mcp.ts → src/mcp/server.ts (rewritten to use QMDStore SDK)
- New src/maintenance.ts: Maintenance class for CLI housekeeping
- SDK gains: getDocumentBody(), getDefaultCollectionNames(),
extractSnippet/addLineNumbers/DEFAULT_MULTI_GET_MAX_BYTES exports,
getDefaultDbPath re-export, InternalStore type export
- package.json bin/scripts updated for new paths
- All 692 tests pass
HuggingFace filenames are case-sensitive. The documented filename
'qwen3-embedding-0.6b-q8_0.gguf' (lowercase) returns 404. The correct
filename is 'Qwen3-Embedding-0.6B-Q8_0.gguf' (original case from the
HuggingFace repo).
Co-Authored-By: Oz <oz-agent@warp.dev>
Add optional `intent` parameter that steers query expansion, reranking,
chunk selection, and snippet extraction without searching on its own.
When a query like "performance" is ambiguous (web-perf vs team health vs
fitness), intent provides background context that disambiguates results
across all pipeline stages:
- expandQuery: includes intent in LLM prompt ("Query intent: {intent}")
- rerank: prepends intent to rerank query for Qwen3-Reranker
- chunk selection: intent terms scored at 0.5x weight vs query terms
- snippet extraction: intent terms scored at 0.3x weight
- strong-signal bypass: disabled when intent provided
Available via CLI (--intent flag or intent: line in query documents),
MCP (intent field on query tool), and programmatic API.
Adapted from PR #180 (thanks @vyalamar).
- Cap rerank contexts at 4 to avoid VRAM exhaustion on high-core machines
- Deduplicate identical chunk texts before sending to reranker
- Cache rerank scores by chunk content instead of file path — same text
from different files now shares a single reranker call
- Add truncation cache to avoid re-tokenizing duplicate documents
The default embeddinggemma-300M model is English-centric and produces
poor embeddings for CJK (Chinese, Japanese, Korean) text. This change
allows overriding the embedding model via the QMD_EMBED_MODEL environment
variable.
Changes:
- DEFAULT_EMBED_MODEL now reads from QMD_EMBED_MODEL env var (fallback to
embeddinggemma-300M for backward compatibility)
- getDefaultLlamaCpp() passes QMD_EMBED_MODEL to LlamaCpp config when set
- formatQueryForEmbedding() and formatDocForEmbedding() detect Qwen3-Embedding
models and apply the correct prompt format (Qwen3 uses task-instruction
format; embeddinggemma uses nomic-style prefix format)
- store.ts: pass model URI to format functions so format selection is
consistent between indexing and query time
- README: document QMD_EMBED_MODEL with Qwen3-Embedding example
Recommended multilingual model:
QMD_EMBED_MODEL=hf:Qwen/Qwen3-Embedding-0.6B-GGUF/qwen3-embedding-0.6b-q8_0.gguf
After changing the model, run: qmd embed -f
node-llama-cpp throws a hard error when any document + query + template
overhead exceeds the ranking context size. Truncate oversized documents
using the rerank model's tokenizer before passing them to rankAll().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add training configuration and documentation for using LiquidAI's LFM2-1.2B
as an alternative base model for query expansion fine-tuning.
LFM2 benefits:
- 2x faster decode/prefill vs standard transformers
- Optimized for edge/on-device inference
- Good at agentic tasks, RAG, and data extraction
Changes:
- Add configs/sft_lfm2.yaml with LFM2-specific LoRA target modules
- Add jobs/sft_lfm2.py for HuggingFace Jobs training
- Update llm.ts with LFM2 GGUF model URIs
- Add documentation for LFM2 training workflow
LFM2 uses a hybrid architecture (convolutions + attention) requiring
different LoRA targets: q_proj, k_proj, v_proj, out_proj, in_proj, w1, w2, w3
- Add tsc build step (tsconfig.build.json) so npm package ships
compiled JS instead of raw TypeScript requiring tsx at runtime
- Update qmd wrapper and daemon spawn to use dist/qmd.js in
production while keeping tsx for development
- Add self-installing pre-push hook validating v* tag pushes:
package.json version match, changelog entry, CI status
- Add release.sh script that renames [Unreleased] to versioned
entry, bumps package.json, commits, and tags
- Add extract-changelog.sh for cumulative GitHub release notes
- Update publish workflow with build step and GitHub release creation
- Flesh out CHANGELOG.md with full history from 0.1.0 through 1.0.0
in Keep-a-Changelog format with PR/contributor attributions
- Add release standards and changelog guidelines to CLAUDE.md
Our assumption that CPU can't benefit from multiple contexts was
wrong. The withLock in node-llama-cpp serializes within a single
context, but separate contexts with split threads run on different
cores in true parallel.
Key changes:
- computeParallelism() now returns >1 on CPU (cores / 4, max 4)
- threadsPerContext() splits math cores evenly across contexts
- Both embed and rerank contexts get proper thread counts
- Benchmark updated to test CPU parallelism
Before (CPU, 40 docs): 9.7s (4.1 docs/s) — 6 threads, 1 context
After (CPU, 40 docs): 2.3s (17.2 docs/s) — 32 threads, 8 contexts
Two fixes stacked:
1. Thread count: default was 6 (library hardcode), now uses all
math cores — 2× improvement alone
2. Multi-context: splitting cores across 8 contexts gives another
2.2× on top
End-to-end 'qmd query' on CPU: 10.3s → 2.9s
CPU benchmark (Threadripper PRO 7975WX, 32 math cores):
1 ctx: 5001ms (8.0 docs/s)
2 ctx: 3585ms (11.2 docs/s) 1.4×
4 ctx: 2874ms (13.9 docs/s) 1.7×
8 ctx: 2323ms (17.2 docs/s) 2.2×
Holistic tuning pass on context and GPU configuration:
GPU detection:
- Use getLlamaGpuTypes() to discover available backends at runtime
instead of try/catch loop. Prefer CUDA > Metal > Vulkan > CPU.
- getLlama({gpu:'auto'}) returns false even when CUDA is available
(node-llama-cpp issue), so we can't rely on it.
Context tuning:
- Rerank context: 2048 tokens (was auto=40960). The Qwen3 reranker
template adds ~200 tokens overhead, chunks are ~800, query ~50.
Total ~1050 tokens, so 2048 gives comfortable margin.
VRAM per context: ~960 MB (was 11.6 GB with auto).
- Flash attention enabled for rerank contexts (~20% less VRAM).
Falls back gracefully if flash attention not supported.
- Embed context: kept at model default (2048 for nomic-embed).
Platform considerations:
- CUDA (server): up to 8 parallel contexts, flash attention
- Metal (MacBook): 1-4 contexts depending on unified memory
- Vulkan: detected and used if CUDA/Metal unavailable
- CPU: single context (parallelism has no benefit due to locks)
Context size was 1024 initially but Qwen3's reranker template is
verbose (system prompt + instruct + think tags) — some inputs
exceeded 1024 tokens. Bumped to 2048 for safety.
Holistic overhaul of context management:
1. Parallel embedding contexts: embedBatch now splits work across
multiple EmbeddingContexts (same pattern as reranking). Each
context is ~143 MB. Benchmarked 6x speedup on 20 texts with
4 contexts vs 1.
2. Rerank context size: was using auto (40960 tokens = 11.6 GB per
context!). Reranking chunks are ~800 tokens max, so 1024 is
plenty. Now 711 MB per context — 16x less VRAM. 4 contexts went
from 46 GB to 2.8 GB.
3. Adaptive parallelism via computeParallelism(): checks available
VRAM and allocates at most 25% of free VRAM for contexts, capped
at 8. Falls back to 1 on CPU (no benefit from multiple contexts
with node-llama-cpp's withLock serialization). Gracefully handles
allocation failures — uses however many contexts succeeded.
VRAM budget per operation:
- Embed: N × 143 MB (nomic-embed, 2048 ctx)
- Rerank: N × 711 MB (Qwen3-Reranker-0.6B, 1024 ctx)
- Generate: ~1.1 GB (qmd-expansion-1.7B, fresh ctx per call)
Works across:
- Large GPU boxes (4x A6000, 190 GB): allocates up to 8 contexts
- Consumer GPUs (16 GB): 2-4 contexts fit comfortably
- Apple Metal (8-16 GB unified): 1-4 contexts depending on memory
- CPU-only: single context (parallelism has no benefit)
node-llama-cpp's LlamaRankingContext uses a single sequence with a
withLock() guard, making rankAll() effectively sequential despite
using Promise.all(). Each document evaluation erases the context,
evaluates tokens, and extracts the logit — all serialized.
Fix: create 4 parallel ranking contexts from the same model (model
weights are shared, only KV cache is duplicated). Split documents
across contexts and evaluate in parallel via Promise.all().
Benchmarks (40 chunks, CUDA, 4x A6000):
- 1 context: 898ms (baseline)
- 2 contexts: 460ms (2.0x)
- 4 contexts: 338ms (2.7x) ← sweet spot
- 8 contexts: 458ms (VRAM contention)
End-to-end 'qmd query' time: 7.5s → 3.7s
Gracefully handles VRAM limits — if creating the Nth context fails,
falls back to however many were successfully created.
QMD was running all models on CPU even when CUDA/Vulkan/Metal
was available. The getLlama() call used no gpu option, defaulting
to false.
Now:
- ensureLlama() tries cuda → vulkan → metal → CPU fallback
- Prints warning to stderr if falling back to CPU
- 'qmd status' shows GPU type, device names, VRAM, and CPU cores
- On this machine: 7.5s query vs 5+ minutes on CPU (reranker)
The reranker (Qwen3-Reranker-0.6B) calls are serialized by a lock
in node-llama-cpp's rankAndSort() — each of the 40 chunks is
evaluated sequentially. This is inherent to the library's design
(single sequence context). GPU acceleration is the fix, not
batching — the lock prevents true parallelism regardless.
- Changed temperature from 0/0.1 to 0.7 (Qwen3 non-thinking mode default)
- Added topK=20, topP=0.8 per Qwen3 docs
- Added repeatPenalty with presencePenalty=0.5 for query expansion
- Fixes infinite loop on acronyms like DHH, BFCM
Qwen3 docs explicitly warn: 'DO NOT use greedy decoding, as it can
lead to performance degradation and endless repetitions'
Brings in:
- /only: variants for single-type expansions
- LLM session management for lifecycle safety
- skills.sh integration for AI agent discovery
- Various bug fixes for vector search and embeddings
Merge conflicts resolved by keeping hyde-first format ordering
from finetune branch while accepting expanded templates and
new features from main.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move the hyde (hypothetical document) line to the beginning of the
output format, before lex and vec lines. This better reflects the
logical flow where the hypothetical document is generated first and
then informs the keyword/semantic expansions.
Also adds auto-download of eval_common.py in training scripts for
standalone HuggingFace Jobs execution.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds a session layer that prevents LLM contexts from being disposed
mid-operation during long-running tasks like batch embedding or
multi-step search workflows (expand → embed → rerank).
Key changes:
- Add LLMSessionManager with reference counting for active sessions
- Add LLMSession class for scoped access with automatic acquire/release
- Add withLLMSession() API for multi-step workflows
- Update idle timer to check canUnloadLLM() before disposing
- Wrap querySearch, vectorSearch, and embed command in sessions
- Add optional session parameter to searchVec and getEmbedding
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add query expansion model finetuning infrastructure
- Training scripts for Qwen3-0.6B and 1.7B models
- Dataset generation from s-emanuilov/query-expansion
- Evaluation scripts comparing finetuned vs baseline models
- GRPO RL training script (optional improvement)
- Export script for GGUF conversion
Results:
- 0.6B finetuned: 95% format compliance (lex/vec/hyde)
- Baseline: 0% format compliance
- Dataset: 5,157 examples on HuggingFace Hub
Models available at:
- tobil/qmd-query-expansion-0.6B (recommended)
- tobil/qmd-query-expansion-train (dataset)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix GRPO training script for TRL API compatibility
- Use max_completion_length instead of max_new_tokens
- Use processing_class instead of tokenizer
- Use args instead of config for GRPOTrainer
- Add __name__ attribute to reward function class
- Accept **kwargs in reward function for extra TRL args
- Add new LoRA adapter after merging SFT weights
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update README with final evaluation results
- 0.6B SFT: 95% format compliance (best)
- 0.6B GRPO: 0% (catastrophic forgetting from RL)
- 1.7B v2: training completed, evaluation pending
- Added GRPO evaluation results
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add comprehensive scoring system for query expansion
New scoring criteria (0-100 points):
- Format (30): Must have lex: and vec: prefixes
- Diversity (30): Multiple types, no echoing query, diverse expansions
- Hyde (20): Optional, concise, no newlines, no word repetition
- Quality (20): Lex=keywords, vec=natural language
See SCORING.md for full documentation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add HuggingFace login and comprehensive scoring to GRPO v2 training
- Add explicit HF_TOKEN login before training
- Use SCORING.md criteria as RL reward function
- Conservative training: LR 1e-6, LoRA rank 4
- Reward scores: good=0.94, bad=0.38
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor finetune folder: train/rl scripts with YAML configs
Major changes:
- train.py: Generic SFT training script using YAML config
- rl.py: Generic GRPO training script using YAML config
- configs/: YAML configs per training run (sft_v4.yaml, grpo_v4.yaml)
- dataset/: Data preparation scripts moved here
- tui.py: Interactive model testing interface
Training results:
- SFT v4: 98.8% avg score (all Excellent)
- GRPO v4: 0% (failed - model drifted to verbose explanations)
Removed per-model scripts (train_0.6B.py, train_1.7B.py, etc)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add named entity extraction to GRPO reward function
Key changes:
- Extract named entities (acronyms, proper nouns, technical terms)
- Heavy penalty (-30) when lex queries miss named entities
- Penalty (-15) for generic filler phrases like "find information about"
- Compound entity detection (TDS motorsports -> both words)
- Update GRPO config with KL regularization (beta=0.04)
- Lower learning rate (5e-7) and add max_steps (200)
Test results:
- "who is TDS motorsports" good: 1.00, bad: 0.30 (was 0.75)
- "how to use React hooks" good: 0.87, bad: 0.45 (was 0.75)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add chat template leakage detection to reward function
Zero reward for outputs containing:
- <|im_start|>, <|im_end|> tokens
- <think>, </think> tags (Qwen3 thinking mode)
- Role markers like \nassistant\n, \nuser\n
- <|endoftext|> token
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Strict format validation: every line must be lex:/vec:/hyde:
Any line that doesn't start with a valid prefix now returns 0.0
instead of just counting as a penalty. This prevents any prose,
explanations, bullet points, or other invalid content.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Clean up evaluation files
- Remove old versioned evaluation files (0.6B, 1.7B, baseline)
- Rename evaluation_v4.json -> evaluation_sft.json
- Rename evaluation_v4_grpo.json -> evaluation_grpo_failed.json
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Refactor evals into separate run and score scripts
New structure:
- evals/run.py: Generate model outputs to JSONL
- evals/score.py: Score outputs with detailed breakdown
- evals/queries.txt: Test queries (26 total)
Features:
- Supports both HF Hub and local model paths
- Named entity preservation scoring
- Chat template leakage detection
- Strict format validation (every line must be lex:/vec:/hyde:)
- Generic phrase detection
Usage:
uv run evals/run.py --model tobil/qmd-query-expansion-0.6B-v4
uv run evals/score.py evals/results_*.jsonl
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix GRPO model loading to use SFT base first
The GRPO adapter was trained on merged SFT weights, so loading it
directly on the base model results in 0% score. Added --sft-model
parameter to evals/run.py to load SFT first, then apply GRPO adapter.
With correct loading: GRPO scores 89.7% (all 26 queries Excellent).
Updated README with correct GRPO score and loading instructions.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix TUI to load GRPO models with SFT base first
GRPO adapters were trained on merged SFT weights, so they need SFT
loaded and merged first before applying the GRPO adapter.
Updated MODELS config to include sft_base path for GRPO models,
and load_model() now handles the SFT -> merge -> GRPO flow.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update README for unified model repository structure
All models (0.6B, 1.7B, 4B) with SFT and GRPO variants now go into
a single HuggingFace repo (tobil/qmd-query-expansion) with subfolders
for each size and training method.
Updated loading examples to show subfolder-based model loading.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Update README with separate model repos
Changed from subfolder approach to separate repos per model since
trainer.push_to_hub() doesn't support subfolder argument.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Add 1.7B and 4B GRPO training and GGUF conversion scripts
Training scripts for GRPO fine-tuning:
- train_1.7B_grpo.py: GRPO training for Qwen3-1.7B
- train_4B_grpo.py: GRPO training for Qwen3-4B
GGUF conversion scripts:
- convert_1.7B_gguf.py: Merge SFT+GRPO adapters and convert to GGUF
- convert_4B_gguf.py: Merge SFT+GRPO adapters and convert to GGUF
All scripts use PEP 723 inline dependencies for HuggingFace Jobs.
Models published:
- tobil/qmd-query-expansion-1.7B-sft
- tobil/qmd-query-expansion-1.7B-grpo
- tobil/qmd-query-expansion-1.7B-gguf
- tobil/qmd-query-expansion-4B-sft
- tobil/qmd-query-expansion-4B-grpo
- tobil/qmd-query-expansion-4B-gguf
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove beads issue tracking
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Remove beads reference from CLAUDE.md
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix GRPO reward function to handle think blocks and end tokens
- Strip <|im_end|> token from completions (model output includes it)
- Change think_penalty to skipped_think bonus (+20 for not using think)
- Adjust max_possible to account for bonus (120/140)
- Fix typo in chat template artifact check
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Make TUI model list dynamic from HuggingFace Hub
- Fetch available qmd-query-expansion models from tobil/ on Hub
- Auto-detect model size (0.6B, 1.7B, 4B) and use correct base model
- Group models by type (SFT vs GRPO) in menu
- Skip GGUF repos in model listing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix GRPO training: apply chat template to prompts
The SFT model was trained with chat template format but GRPO was
passing raw prompts. Now prompts are formatted with tokenizer.apply_chat_template()
so the model sees the same format it learned during SFT.
Also update extract_query_from_prompt to strip chat template artifacts.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Finetune 2.0: consolidate and simplify the entire training pipeline
Consolidate ~2,800 lines of duplicated code across 12 files into 5 clean,
well-documented files targeting Qwen3-1.7B end-to-end.
Key changes:
- Extract reward function into single source of truth (reward.py)
Previously duplicated 3x with divergent bugs across rl.py,
train_1.7B_grpo.py, and train_4B_grpo.py
- Unify training into one script with sft/grpo subcommands (train.py)
Replaces train.py + rl.py + train_1.7B_grpo.py + train_4B_grpo.py
- Merge eval generate+score into single eval.py
Replaces evals/run.py + evals/score.py
- Parameterize GGUF conversion by --size (convert_gguf.py)
Replaces convert_1.7B_gguf.py + convert_4B_gguf.py
- Fix critical bug: rl.py silently ignored beta/temperature from config,
causing the exact catastrophic drift its own comments warned about
- Fix prompt consistency: all files use /no_think chat template format
- Retarget configs from 0.6B to 1.7B
- Comprehensive README documenting the full pipeline
Removed: rl.py, train_1.7B_grpo.py, train_4B_grpo.py, convert_1.7B_gguf.py,
convert_4B_gguf.py, tui.py, evals/run.py, evals/score.py
Net: -3,429 lines, +382 lines
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
* Add HF Jobs scripts, temporal query examples, and training results
- jobs/sft.py and jobs/grpo.py: self-contained scripts for
`hf jobs uv run` (no local GPU needed)
- 12 temporal/recency query examples in training data (e.g. "recent
news about Shopify" -> lex with years 2025/2026)
- 4 temporal test queries in evals/queries.txt
- README updated with HF Jobs workflow, training results, and
updated file structure
- Remove .beads tracking
SFT and GRPO successfully trained on A10G via HF Jobs:
SFT: eval loss 0.321, token accuracy 92.4%
GRPO: mean reward 0.757, 200 steps, KL 0.00048
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
* Deploy fine-tuned GRPO model as default for query expansion
Switch from generic Qwen3-1.7B-Q8_0 (~2.2GB) to fine-tuned
qmd-query-expansion-1.7B-q4_k_m (~1.1GB). The fine-tuned Q4
scores 91.7% avg with 30/30 Excellent, outperforming the base Q8.
- Update default generate model in src/llm.ts
- Update README model table, architecture diagram, config block
- Add v2 training data, eval scripts, and quantize job
- Remove superseded v1 training data (5,742 → 1,000 examples)
- Update finetune README with v2 results and file structure
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Switch from generic Qwen3-1.7B-Q8_0 (~2.2GB) to fine-tuned
qmd-query-expansion-1.7B-q4_k_m (~1.1GB). The fine-tuned Q4
scores 91.7% avg with 30/30 Excellent, outperforming the base Q8.
- Update default generate model in src/llm.ts
- Update README model table, architecture diagram, config block
- Add v2 training data, eval scripts, and quantize job
- Remove superseded v1 training data (5,742 → 1,000 examples)
- Update finetune README with v2 results and file structure
Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>
* fix: add promise guard to ensureEmbedContext to prevent race condition
Root cause: ensureEmbedContext() was not thread-safe. When multiple parallel
embedding requests called ensureEmbedContext() simultaneously, all would see
embedContext === null and start creating new contexts. This race condition
caused 'Context is disposed' errors as contexts were overwritten/orphaned.
The fix adds a promise guard (embedContextCreatePromise) to ensure only one
context creation runs at a time - identical to the pattern already used in
ensureGenerateModel().
Changes:
- Add embedContextCreatePromise field to track in-progress context creation
- Modify ensureEmbedContext() to wait for existing creation if in progress
- Update test comment and timeout for CPU-only systems
Testing:
- Fresh model download + qmd embed: 28/28 chunks succeeded (was 14/27)
- All embedBatch tests pass
- No warmup hack needed - full parallel performance from the start
Environment tested:
- Ubuntu 24.04 LTS (x64), Bun 1.3.6, node-llama-cpp 3.14.5, no GPU
* test: improve race condition test to verify single context creation
The previous test only verified embeddings succeeded but didn't prove the fix
actually prevents multiple context creation. This improved test:
- Instruments createEmbeddingContext to count invocations
- Runs 5 concurrent embedBatch calls on a fresh LlamaCpp instance
- Asserts exactly 1 context is created (fails with 5 without the fix)
Verified locally:
- With fix: 1 context created (PASS)
- Without fix: 5 contexts created (FAIL)
* chore: clear embedContextCreatePromise in dispose() for consistency
Issue #11: Collection filter (-c) SQL error
- Fixed searchVec to properly parameterize collection filter
- Changed collectionId check from !== undefined to truthy
- Added test for searchVec with collection filter
Issue #10: Non-ASCII filename support
- Updated handelize() to use Unicode property escapes (\p{L}\p{N})
- Now supports Cyrillic, Japanese, and other Unicode filenames
- Updated tests to verify Unicode filename handling
Also:
- Fixed expandQuery to filter out lex entries when includeLexical=false
- Updated expandQuery tests to match actual behavior
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix SQL syntax error when collectionId is empty string (searchFTS, searchVec)
- Add 1-second timeout to llama.dispose() to prevent indefinite hang
- Add process.exit(0) after cleanup for clean CLI exit
- Include hash/docid in search results mapping
- Update query expansion to use structured Queryable types
- Switch to Qwen3-1.7B model for better query expansion
- Improve bun discovery in qmd wrapper script
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Let node-llama-cpp handle context size and sequences automatically.
The mutex still serializes generation calls for safety.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Reduce context sequences from 4 to 1 to minimize VRAM usage
when multiple models (embed, generate, rerank) are loaded
- Add mutex to serialize generation calls to prevent "No sequences left"
error when concurrent requests occur with single sequence
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- dispose() now just calls llama.dispose() which cascades to models/contexts
per node-llama-cpp lifecycle docs
- Remove disposeDefaultLlamaCpp calls from tests - they don't help with
the Metal cleanup crash
- Use singleton getDefaultLlamaCpp() in llm tests for consistency
The Metal backend crash at process exit is a known llama.cpp issue:
https://github.com/ggml-org/llama.cpp/pull/17869
All tests pass - the abort happens after test completion.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The Metal crash was caused by not disposing resources in the right order
at the right time. The fix:
1. Restore proper dispose() that disposes contexts → models → llama in order
2. Move disposeDefaultLlamaCpp() to global afterAll (after all tests complete)
3. Keep disposed flag to prevent double-dispose
The issue was that disposing per-suite broke tests that share llama,
and not disposing at all left orphaned Metal resources at process exit.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
