ai-workspace-services/qmd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
425 Commits 3 Branches 2 Tags 7.9 MiB
68d0b314f5
Commit Graph

7 Commits

Author SHA1 Message Date
rkbadhan
4511b9bd4d fix(reward): tighten entity detection, add filler penalty, stricter diversity 
- Compound entity chaining now stops one level deep. Previously "TDS
  motorsports team history" would inflate the expected entity set with
  "team" and "history", causing false-positive entity-preservation
  penalties during GRPO. Now only {tds, motorsports} are detected.

- Add INTERIOR_FILLER_WORDS penalty (-3/line): lex lines containing
  "overview" or "basics" absent from the original query are penalised.
  Targets template-generator noise, e.g. "ancient overview rome timeline".

- Raise is_diverse threshold 2→3: requires 3 unique words between lex
  lines before they count as diverse. Reduces reward for near-duplicate
  pairs like "auth setup" / "auth configuration".

- Broaden quoted-phrase bonus: was gated on named entities existing;
  now any multi-word query earns +3 for using quotes in lex lines.
  Better incentivises BM25-aware syntax like "memory leak" python.

Fixes scoring noise identified while working on issue #247.
 2026-02-24 19:46:23 +05:30
Tobi Lutke
3950055708
finetune: quoted phrases, negation, and entity preservation (#247) 
Training data:
- Expand lex phrases/negation examples from 12 to 74 with intent field
- Add 50 personal entity examples (meetings, emails, projects with names)

Reward function:
- Detect entities at position 0 (fixes "Bob asked about deploy")
- Per-entity coverage penalty: -20 per entity absent from all lex+vec
- Phrase quoting bonus: +3 when lex uses quotes for multi-word terms
- Expanded stopwords to reduce false positive entity detection

Eval queries: add 21 test queries for personal entities, quoted phrases,
and negation/disambiguation scenarios.
 2026-02-22 13:38:59 -04:00
Tobi Lütke
bf1b8fc90a
lots of training stuff 2026-01-31 23:02:23 +00:00
Tobi Lütke
20aef8a3e9 Change format to /only:lex (slash prefix) 2026-01-31 16:24:18 +00:00
Tobi Lütke
46ff098361 Change only: format to only:lex (no space after colon) 2026-01-31 16:23:28 +00:00
Tobi Lütke
806a0cfc14 Add 'only:' mode support for single-type expansions 
- generate_only_variants.py: Creates training data where queries end with
  'only: lex', 'only: vec', or 'only: hyde' and output contains ONLY that type
- reward.py: Updated scorer to handle 'only:' mode separately
  - Penalizes presence of unwanted types
  - Type-specific quality checks
  - Filters templated low-quality hyde outputs
- 4,444 high-quality 'only:' variants from v2 + handcrafted data
 2026-01-31 16:15:59 +00:00
Tobias Lütke
eb1b77c8cb
Deploy fine-tuned GRPO model as default query expansion (#67) 
* Add query expansion model finetuning infrastructure

- Training scripts for Qwen3-0.6B and 1.7B models
- Dataset generation from s-emanuilov/query-expansion
- Evaluation scripts comparing finetuned vs baseline models
- GRPO RL training script (optional improvement)
- Export script for GGUF conversion

Results:
- 0.6B finetuned: 95% format compliance (lex/vec/hyde)
- Baseline: 0% format compliance
- Dataset: 5,157 examples on HuggingFace Hub

Models available at:
- tobil/qmd-query-expansion-0.6B (recommended)
- tobil/qmd-query-expansion-train (dataset)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix GRPO training script for TRL API compatibility

- Use max_completion_length instead of max_new_tokens
- Use processing_class instead of tokenizer
- Use args instead of config for GRPOTrainer
- Add __name__ attribute to reward function class
- Accept **kwargs in reward function for extra TRL args
- Add new LoRA adapter after merging SFT weights

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Update README with final evaluation results

- 0.6B SFT: 95% format compliance (best)
- 0.6B GRPO: 0% (catastrophic forgetting from RL)
- 1.7B v2: training completed, evaluation pending
- Added GRPO evaluation results

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add comprehensive scoring system for query expansion

New scoring criteria (0-100 points):
- Format (30): Must have lex: and vec: prefixes
- Diversity (30): Multiple types, no echoing query, diverse expansions
- Hyde (20): Optional, concise, no newlines, no word repetition
- Quality (20): Lex=keywords, vec=natural language

See SCORING.md for full documentation.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add HuggingFace login and comprehensive scoring to GRPO v2 training

- Add explicit HF_TOKEN login before training
- Use SCORING.md criteria as RL reward function
- Conservative training: LR 1e-6, LoRA rank 4
- Reward scores: good=0.94, bad=0.38

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Refactor finetune folder: train/rl scripts with YAML configs

Major changes:
- train.py: Generic SFT training script using YAML config
- rl.py: Generic GRPO training script using YAML config
- configs/: YAML configs per training run (sft_v4.yaml, grpo_v4.yaml)
- dataset/: Data preparation scripts moved here
- tui.py: Interactive model testing interface

Training results:
- SFT v4: 98.8% avg score (all Excellent)
- GRPO v4: 0% (failed - model drifted to verbose explanations)

Removed per-model scripts (train_0.6B.py, train_1.7B.py, etc)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add named entity extraction to GRPO reward function

Key changes:
- Extract named entities (acronyms, proper nouns, technical terms)
- Heavy penalty (-30) when lex queries miss named entities
- Penalty (-15) for generic filler phrases like "find information about"
- Compound entity detection (TDS motorsports -> both words)
- Update GRPO config with KL regularization (beta=0.04)
- Lower learning rate (5e-7) and add max_steps (200)

Test results:
- "who is TDS motorsports" good: 1.00, bad: 0.30 (was 0.75)
- "how to use React hooks" good: 0.87, bad: 0.45 (was 0.75)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add chat template leakage detection to reward function

Zero reward for outputs containing:
- <|im_start|>, <|im_end|> tokens
- <think>, </think> tags (Qwen3 thinking mode)
- Role markers like \nassistant\n, \nuser\n
- <|endoftext|> token

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Strict format validation: every line must be lex:/vec:/hyde:

Any line that doesn't start with a valid prefix now returns 0.0
instead of just counting as a penalty. This prevents any prose,
explanations, bullet points, or other invalid content.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Clean up evaluation files

- Remove old versioned evaluation files (0.6B, 1.7B, baseline)
- Rename evaluation_v4.json -> evaluation_sft.json
- Rename evaluation_v4_grpo.json -> evaluation_grpo_failed.json

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Refactor evals into separate run and score scripts

New structure:
- evals/run.py: Generate model outputs to JSONL
- evals/score.py: Score outputs with detailed breakdown
- evals/queries.txt: Test queries (26 total)

Features:
- Supports both HF Hub and local model paths
- Named entity preservation scoring
- Chat template leakage detection
- Strict format validation (every line must be lex:/vec:/hyde:)
- Generic phrase detection

Usage:
  uv run evals/run.py --model tobil/qmd-query-expansion-0.6B-v4
  uv run evals/score.py evals/results_*.jsonl

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix GRPO model loading to use SFT base first

The GRPO adapter was trained on merged SFT weights, so loading it
directly on the base model results in 0% score. Added --sft-model
parameter to evals/run.py to load SFT first, then apply GRPO adapter.

With correct loading: GRPO scores 89.7% (all 26 queries Excellent).

Updated README with correct GRPO score and loading instructions.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix TUI to load GRPO models with SFT base first

GRPO adapters were trained on merged SFT weights, so they need SFT
loaded and merged first before applying the GRPO adapter.

Updated MODELS config to include sft_base path for GRPO models,
and load_model() now handles the SFT -> merge -> GRPO flow.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Update README for unified model repository structure

All models (0.6B, 1.7B, 4B) with SFT and GRPO variants now go into
a single HuggingFace repo (tobil/qmd-query-expansion) with subfolders
for each size and training method.

Updated loading examples to show subfolder-based model loading.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Update README with separate model repos

Changed from subfolder approach to separate repos per model since
trainer.push_to_hub() doesn't support subfolder argument.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Add 1.7B and 4B GRPO training and GGUF conversion scripts

Training scripts for GRPO fine-tuning:
- train_1.7B_grpo.py: GRPO training for Qwen3-1.7B
- train_4B_grpo.py: GRPO training for Qwen3-4B

GGUF conversion scripts:
- convert_1.7B_gguf.py: Merge SFT+GRPO adapters and convert to GGUF
- convert_4B_gguf.py: Merge SFT+GRPO adapters and convert to GGUF

All scripts use PEP 723 inline dependencies for HuggingFace Jobs.

Models published:
- tobil/qmd-query-expansion-1.7B-sft
- tobil/qmd-query-expansion-1.7B-grpo
- tobil/qmd-query-expansion-1.7B-gguf
- tobil/qmd-query-expansion-4B-sft
- tobil/qmd-query-expansion-4B-grpo
- tobil/qmd-query-expansion-4B-gguf

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Remove beads issue tracking

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Remove beads reference from CLAUDE.md

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix GRPO reward function to handle think blocks and end tokens

- Strip <|im_end|> token from completions (model output includes it)
- Change think_penalty to skipped_think bonus (+20 for not using think)
- Adjust max_possible to account for bonus (120/140)
- Fix typo in chat template artifact check

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Make TUI model list dynamic from HuggingFace Hub

- Fetch available qmd-query-expansion models from tobil/ on Hub
- Auto-detect model size (0.6B, 1.7B, 4B) and use correct base model
- Group models by type (SFT vs GRPO) in menu
- Skip GGUF repos in model listing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix GRPO training: apply chat template to prompts

The SFT model was trained with chat template format but GRPO was
passing raw prompts. Now prompts are formatted with tokenizer.apply_chat_template()
so the model sees the same format it learned during SFT.

Also update extract_query_from_prompt to strip chat template artifacts.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Finetune 2.0: consolidate and simplify the entire training pipeline

Consolidate ~2,800 lines of duplicated code across 12 files into 5 clean,
well-documented files targeting Qwen3-1.7B end-to-end.

Key changes:
- Extract reward function into single source of truth (reward.py)
  Previously duplicated 3x with divergent bugs across rl.py,
  train_1.7B_grpo.py, and train_4B_grpo.py
- Unify training into one script with sft/grpo subcommands (train.py)
  Replaces train.py + rl.py + train_1.7B_grpo.py + train_4B_grpo.py
- Merge eval generate+score into single eval.py
  Replaces evals/run.py + evals/score.py
- Parameterize GGUF conversion by --size (convert_gguf.py)
  Replaces convert_1.7B_gguf.py + convert_4B_gguf.py
- Fix critical bug: rl.py silently ignored beta/temperature from config,
  causing the exact catastrophic drift its own comments warned about
- Fix prompt consistency: all files use /no_think chat template format
- Retarget configs from 0.6B to 1.7B
- Comprehensive README documenting the full pipeline

Removed: rl.py, train_1.7B_grpo.py, train_4B_grpo.py, convert_1.7B_gguf.py,
convert_4B_gguf.py, tui.py, evals/run.py, evals/score.py

Net: -3,429 lines, +382 lines

Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>

* Add HF Jobs scripts, temporal query examples, and training results

- jobs/sft.py and jobs/grpo.py: self-contained scripts for
  `hf jobs uv run` (no local GPU needed)
- 12 temporal/recency query examples in training data (e.g. "recent
  news about Shopify" -> lex with years 2025/2026)
- 4 temporal test queries in evals/queries.txt
- README updated with HF Jobs workflow, training results, and
  updated file structure
- Remove .beads tracking

SFT and GRPO successfully trained on A10G via HF Jobs:
  SFT: eval loss 0.321, token accuracy 92.4%
  GRPO: mean reward 0.757, 200 steps, KL 0.00048

Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>

* Deploy fine-tuned GRPO model as default for query expansion

Switch from generic Qwen3-1.7B-Q8_0 (~2.2GB) to fine-tuned
qmd-query-expansion-1.7B-q4_k_m (~1.1GB). The fine-tuned Q4
scores 91.7% avg with 30/30 Excellent, outperforming the base Q8.

- Update default generate model in src/llm.ts
- Update README model table, architecture diagram, config block
- Add v2 training data, eval scripts, and quantize job
- Remove superseded v1 training data (5,742 → 1,000 examples)
- Update finetune README with v2 results and file structure

Co-Authored-By: Claude (claude-fudge-eap-cc) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-28 23:25:17 -08:00