ai-workspace-services/qmd
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add query expansion model finetuning infrastructure

Browse Source 
- Training scripts for Qwen3-0.6B and 1.7B models
- Dataset generation from s-emanuilov/query-expansion
- Evaluation scripts comparing finetuned vs baseline models
- GRPO RL training script (optional improvement)
- Export script for GGUF conversion

Results:
- 0.6B finetuned: 95% format compliance (lex/vec/hyde)
- Baseline: 0% format compliance
- Dataset: 5,157 examples on HuggingFace Hub

Models available at:
- tobil/qmd-query-expansion-0.6B (recommended)
- tobil/qmd-query-expansion-train (dataset)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Tobi Lutke 2026-01-23 19:47:06 -05:00
parent 88f78314bb
commit 7cca164dd9
No known key found for this signature in database
20 changed files with 8847 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
finetune/.gitignore vendored Normal file
View File

@ -0,0 +1,12 @@
# Model checkpoints (stored on HuggingFace Hub)
qmd-query-expansion-*/
*.pt
*.safetensors
# Large data files (stored on HuggingFace Hub)
data/train/train.jsonl
data/train/train_chat.jsonl
data/train/val.jsonl
# Keep the generated source data
!data/qmd_expansion.jsonl

147
finetune/README.md Normal file
View File

@ -0,0 +1,147 @@
# QMD Query Expansion Model Finetuning
Finetune small Qwen models for QMD's query expansion task.
## Goal
Train models that convert user queries into retrieval-optimized outputs:
```
Input: "how to configure authentication"
Output:
lex: authentication setup
lex: auth configuration
vec: how to set up user authentication in the application
hyde: To configure authentication, set the AUTH_SECRET environment variable and enable the auth middleware in your application config.
```
## Output Format
| Type | Purpose | Count |
|------|---------|-------|
| `lex` | BM25 keyword variations | 1-3 |
| `vec` | Semantic reformulations | 1-3 |
| `hyde` | Hypothetical document passage | 0-1 |
## Trained Models
| Model | HuggingFace | Format Compliance | Status |
|-------|-------------|-------------------|--------|
| **Qwen3-0.6B (finetuned)** | [tobil/qmd-query-expansion-0.6B](https://huggingface.co/tobil/qmd-query-expansion-0.6B) | **95%** | Recommended |
| Qwen3-1.7B (finetuned) | [tobil/qmd-query-expansion-1.7B](https://huggingface.co/tobil/qmd-query-expansion-1.7B) | 0% | Training issues |
| Qwen3-0.6B (baseline) | - | 0% | Untrained |
## Training Dataset
- **Dataset**: [tobil/qmd-query-expansion-train](https://huggingface.co/datasets/tobil/qmd-query-expansion-train)
- **Source**: Transformed from [s-emanuilov/query-expansion](https://huggingface.co/datasets/s-emanuilov/query-expansion) (CC BY 4.0)
- **Size**: 5,157 examples (train: 4,641, eval: 516)
- **Format**: Chat messages with user query and assistant response in lex/vec/hyde format
## Directory Structure
```
finetune/
├── README.md # This file
├── DATASETS.md # Dataset research findings
├── TRAINING_JOBS.md # HuggingFace Jobs tracking
├── generate_data_offline.py # Transform s-emanuilov dataset to QMD format
├── prepare_data.py # Upload to HuggingFace Hub
├── train_0.6B.py # Training script for 0.6B model
├── train_1.7B.py # Training script for 1.7B model
├── train_grpo.py # GRPO RL training (optional)
├── evaluate_model.py # Evaluate finetuned models
├── evaluate_baseline.py # Evaluate base models
├── data/
│ ├── qmd_expansion.jsonl # Generated training data
│ └── train/ # Prepared chat format
└── evaluation_*.json # Evaluation results
```
## Quick Start
### 1. Generate Training Data
```bash
# Transform s-emanuilov dataset to QMD format (no API needed)
uv run generate_data_offline.py
```
### 2. Prepare and Upload Dataset
```bash
# Convert to chat format and upload to HuggingFace Hub
uv run prepare_data.py
```
### 3. Train on HuggingFace Jobs
```bash
# Train Qwen3-0.6B (recommended)
hf jobs uv run --flavor a10g-large --timeout 3h --secrets HF_TOKEN \
"https://huggingface.co/tobil/qmd-training-scripts/resolve/main/train_0.6B.py"
```
### 4. Evaluate
```bash
# Evaluate finetuned model
uv run evaluate_model.py --model tobil/qmd-query-expansion-0.6B --base-model Qwen/Qwen3-0.6B
# Compare to baseline
uv run evaluate_baseline.py --model Qwen/Qwen3-0.6B --num-queries 10
```
### 5. Export to GGUF
```bash
# Convert to GGUF for node-llama-cpp (TODO)
uv run export_gguf.py --model tobil/qmd-query-expansion-0.6B --quantization Q8_0
```
## Training Configuration
| Parameter | Value |
|-----------|-------|
| Method | LoRA (rank 16, alpha 32) |
| Learning Rate | 2e-4 |
| Epochs | 3 |
| Batch Size | 4 (with 4x gradient accumulation) |
| Max Seq Length | 512 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
## Prompt Format
The models are trained on this simple prompt format:
```
Expand this search query:
{query}
```
The model responds with lex/vec/hyde lines directly.
## Evaluation Results
### 0.6B Finetuned Model (95% format compliance)
Sample outputs:
| Query | Output |
|-------|--------|
| `how to configure authentication` | lex: steps for setting up authentication<br>vec: steps for setting up authentication in cloud services<br>hyde: The process of configure authentication... |
| `kubernetes vs docker swarm` | lex: kubernetes and docker swarm<br>vec: kubernetes vs docker swarm<br>hyde: Kubernetes vs docker swarm is an important concept... |
| `cors error fix` | lex: how to fix cors<br>vec: how to fix cors issues in web apps<br>hyde: The topic of cors error fix guide... |
### Baseline Model (0% format compliance)
The untrained model generates random prose, code blocks, or repetitive text with no understanding of the lex/vec/hyde format.
## Future Work
- [ ] Export to GGUF for local inference
- [ ] Integrate into QMD as default query expansion model
- [ ] GRPO training for improved diversity (optional)
- [ ] Fix 1.7B training issues

5730
finetune/data/qmd_expansion.jsonl Normal file
View File

File diff suppressed because it is too large Load Diff

11
finetune/data/train/dataset_info.json Normal file
View File

@ -0,0 +1,11 @@
{
"dataset_name": "qmd-query-expansion",
"train_samples": 5157,
"val_samples": 573,
"columns": [
"prompt",
"completion",
"text",
"messages"
]
}

169
finetune/evaluate_baseline.py Normal file
View File

@ -0,0 +1,169 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "transformers>=4.45.0",
# "torch",
# "huggingface_hub",
# "accelerate",
# ]
# ///
"""
Evaluate base model (untrained) for comparison.
"""
import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Test queries covering different QMD use cases
TEST_QUERIES = [
"how to configure authentication",
"typescript async await",
"docker compose networking",
"git rebase vs merge",
"react useEffect cleanup",
"auth",
"config",
"setup",
"api",
"meeting notes project kickoff",
"ideas for new feature",
"todo list app architecture",
"what is dependency injection",
"difference between sql and nosql",
"kubernetes vs docker swarm",
"connection timeout error",
"memory leak debugging",
"cors error fix",
"how to implement caching with redis in nodejs",
"best practices for api rate limiting",
"setting up ci cd pipeline with github actions",
]
PROMPT_TEMPLATE = """Expand this search query:
{query}"""
def load_model(model_name: str):
"""Load the base model without adapter."""
print(f"Loading tokenizer and model from {model_name}...")
tokenizer = AutoTokenizer.from_pretrained(model_name)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model.eval()
return model, tokenizer
def generate_expansion(model, tokenizer, query: str, max_new_tokens: int = 200) -> str:
"""Generate query expansion."""
prompt = PROMPT_TEMPLATE.format(query=query)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
temperature=0.7,
do_sample=True,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
full_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
expansion = full_output[len(prompt):].strip()
return expansion
def evaluate_expansion(query: str, expansion: str) -> dict:
"""Basic automatic evaluation metrics."""
lines = expansion.strip().split("\n")
has_lex = any(l.strip().startswith("lex:") for l in lines)
has_vec = any(l.strip().startswith("vec:") for l in lines)
has_hyde = any(l.strip().startswith("hyde:") for l in lines)
valid_lines = sum(1 for l in lines if l.strip().startswith(("lex:", "vec:", "hyde:")))
contents = []
for l in lines:
if ":" in l:
contents.append(l.split(":", 1)[1].strip().lower())
unique_contents = len(set(contents))
return {
"has_lex": has_lex,
"has_vec": has_vec,
"has_hyde": has_hyde,
"valid_lines": valid_lines,
"total_lines": len(lines),
"unique_contents": unique_contents,
"format_score": (has_lex + has_vec + has_hyde) / 3,
}
def main():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--model", default="Qwen/Qwen3-0.6B",
help="Base model to evaluate")
parser.add_argument("--output", default="evaluation_baseline.json",
help="Output file for results")
parser.add_argument("--num-queries", type=int, default=5,
help="Number of queries to test (for quick baseline)")
args = parser.parse_args()
model, tokenizer = load_model(args.model)
queries = TEST_QUERIES[:args.num_queries]
results = []
print(f"\n{'='*70}")
print("BASELINE EVALUATION RESULTS")
print(f"{'='*70}\n")
for i, query in enumerate(queries, 1):
print(f"[{i}/{len(queries)}] Query: {query}")
print("-" * 50)
expansion = generate_expansion(model, tokenizer, query)
metrics = evaluate_expansion(query, expansion)
print(expansion[:500] + "..." if len(expansion) > 500 else expansion)
print(f"\n Format: {'✓' if metrics['format_score'] == 1.0 else '⚠'} "
f"(lex:{metrics['has_lex']}, vec:{metrics['has_vec']}, hyde:{metrics['has_hyde']})")
print()
results.append({
"query": query,
"expansion": expansion,
"metrics": metrics,
})
print(f"\n{'='*70}")
print("SUMMARY")
print(f"{'='*70}")
avg_format = sum(r["metrics"]["format_score"] for r in results) / len(results)
full_format = sum(1 for r in results if r["metrics"]["format_score"] == 1.0)
print(f" Total queries: {len(results)}")
print(f" Average format score: {avg_format:.2%}")
print(f" Full format compliance: {full_format}/{len(results)} ({full_format/len(results):.0%})")
with open(args.output, "w") as f:
json.dump(results, f, indent=2)
print(f"\n Results saved to: {args.output}")
if __name__ == "__main__":
main()

206
finetune/evaluate_model.py Normal file
View File

@ -0,0 +1,206 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "transformers>=4.45.0",
# "peft>=0.7.0",
# "torch",
# "huggingface_hub",
# ]
# ///
"""
Evaluate QMD query expansion model quality.
Generates expansions for test queries and outputs results for review.
"""
import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
# Test queries covering different QMD use cases
TEST_QUERIES = [
# Technical documentation
"how to configure authentication",
"typescript async await",
"docker compose networking",
"git rebase vs merge",
"react useEffect cleanup",
# Short/ambiguous queries
"auth",
"config",
"setup",
"api",
# Personal notes / journals style
"meeting notes project kickoff",
"ideas for new feature",
"todo list app architecture",
# Research / learning
"what is dependency injection",
"difference between sql and nosql",
"kubernetes vs docker swarm",
# Error/debugging
"connection timeout error",
"memory leak debugging",
"cors error fix",
# Complex queries
"how to implement caching with redis in nodejs",
"best practices for api rate limiting",
"setting up ci cd pipeline with github actions",
]
PROMPT_TEMPLATE = """Expand this search query:
{query}"""
def load_model(model_name: str, base_model: str = "Qwen/Qwen3-0.6B"):
"""Load the finetuned model."""
print(f"Loading tokenizer from {base_model}...")
tokenizer = AutoTokenizer.from_pretrained(base_model)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
print(f"Loading base model...")
base = AutoModelForCausalLM.from_pretrained(
base_model,
torch_dtype=torch.bfloat16,
device_map="auto",
)
print(f"Loading adapter from {model_name}...")
model = PeftModel.from_pretrained(base, model_name)
model.eval()
return model, tokenizer
def generate_expansion(model, tokenizer, query: str, max_new_tokens: int = 200) -> str:
"""Generate query expansion."""
prompt = PROMPT_TEMPLATE.format(query=query)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
temperature=0.7,
do_sample=True,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
# Decode and extract just the generated part
full_output = tokenizer.decode(outputs[0], skip_special_tokens=True)
# Remove the prompt to get just the expansion
if "Output:" in full_output:
expansion = full_output.split("Output:")[-1].strip()
else:
expansion = full_output[len(prompt):].strip()
return expansion
def evaluate_expansion(query: str, expansion: str) -> dict:
"""Basic automatic evaluation metrics."""
lines = expansion.strip().split("\n")
has_lex = any(l.strip().startswith("lex:") for l in lines)
has_vec = any(l.strip().startswith("vec:") for l in lines)
has_hyde = any(l.strip().startswith("hyde:") for l in lines)
# Count valid lines
valid_lines = sum(1 for l in lines if l.strip().startswith(("lex:", "vec:", "hyde:")))
# Check for repetition
contents = []
for l in lines:
if ":" in l:
contents.append(l.split(":", 1)[1].strip().lower())
unique_contents = len(set(contents))
return {
"has_lex": has_lex,
"has_vec": has_vec,
"has_hyde": has_hyde,
"valid_lines": valid_lines,
"total_lines": len(lines),
"unique_contents": unique_contents,
"format_score": (has_lex + has_vec + has_hyde) / 3,
}
def main():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--model", default="tobil/qmd-query-expansion-0.6B",
help="Model to evaluate")
parser.add_argument("--base-model", default="Qwen/Qwen3-0.6B",
help="Base model")
parser.add_argument("--output", default="evaluation_results.json",
help="Output file for results")
parser.add_argument("--queries", type=str, help="Custom queries file (one per line)")
args = parser.parse_args()
# Load custom queries if provided
queries = TEST_QUERIES
if args.queries:
with open(args.queries) as f:
queries = [l.strip() for l in f if l.strip()]
# Load model
model, tokenizer = load_model(args.model, args.base_model)
# Run evaluation
results = []
print(f"\n{'='*70}")
print("EVALUATION RESULTS")
print(f"{'='*70}\n")
for i, query in enumerate(queries, 1):
print(f"[{i}/{len(queries)}] Query: {query}")
print("-" * 50)
expansion = generate_expansion(model, tokenizer, query)
metrics = evaluate_expansion(query, expansion)
print(expansion)
print(f"\n Format: {'✓' if metrics['format_score'] == 1.0 else '⚠'} "
f"(lex:{metrics['has_lex']}, vec:{metrics['has_vec']}, hyde:{metrics['has_hyde']})")
print(f" Lines: {metrics['valid_lines']}/{metrics['total_lines']} valid, "
f"{metrics['unique_contents']} unique")
print()
results.append({
"query": query,
"expansion": expansion,
"metrics": metrics,
})
# Summary
print(f"\n{'='*70}")
print("SUMMARY")
print(f"{'='*70}")
avg_format = sum(r["metrics"]["format_score"] for r in results) / len(results)
full_format = sum(1 for r in results if r["metrics"]["format_score"] == 1.0)
print(f" Total queries: {len(results)}")
print(f" Average format score: {avg_format:.2%}")
print(f" Full format compliance: {full_format}/{len(results)} ({full_format/len(results):.0%})")
# Save results
with open(args.output, "w") as f:
json.dump(results, f, indent=2)
print(f"\n Results saved to: {args.output}")
if __name__ == "__main__":
main()

275
finetune/evaluation_0.6B.json Normal file
View File

@ -0,0 +1,275 @@
[
{
"query": "how to configure authentication",
"expansion": "Understanding how to configure authentication is essential for modern development. Key aspects include {security setup}. This knowledge helps in building robust applications.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "typescript async await",
"expansion": "typescript async await\nhyde: {hypothetical document passage}\nlex: {keyword variation}\nvec: {semantic reformulation}\nhyde: {hypothetical document passage} is the recommended approach for typescript async await. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 4,
"total_lines": 5,
"unique_contents": 4,
"format_score": 1.0
}
},
{
"query": "docker compose networking",
"expansion": "docker compose networking\nExplanation: Docker compose networking is an important concept that relates to semantic reformulation. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 2,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "git rebase vs merge",
"expansion": "git rebase vs merge\nExplanation: Git rebase vs merge is an important concept that relates to semantic reformulation. It provides functionality for various use cases in software development. Check out the official documentation for detailed specifications.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 2,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "react useEffect cleanup",
"expansion": "react useEffect cleanup\nlex: {react cleanup functions}\nvec: {useeffect cleaning routines}\nhyde: {hypothetical document passage on cleanup in react useEffect} is the recommended approach. It ensures compatibility and follows best practices.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 4,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "auth",
"expansion": "what is the process of authentication?\nvec: {semantic reformulation}\nhyde: {hypothetical document passage}\nlex: {keyword variation}\nvec: {semantic reformulation}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde: {hypothetical document passage}\nhyde:",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 21,
"total_lines": 22,
"unique_contents": 4,
"format_score": 1.0
}
},
{
"query": "config",
"expansion": "config is an important concept that relates to {hypothetical document passage}. Understanding this is essential for effective implementation.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "setup",
"expansion": "setup is an important concept that relates to semantic reformulation. Understanding setup provides context for various applications in technology.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "api",
"expansion": "understanding api is essential for modern development. key aspects include {topic choice}. this knowledge helps in building robust applications.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "meeting notes project kickoff",
"expansion": "meeting notes project kickoff\nExplanation: Meeting notes project kickoff is an important concept that relates to {hypothetical document passage}. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 2,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "ideas for new feature",
"expansion": "ideas for new feature\nlex: {keyword variation}\nvec: {semantic reformulation}\nhyde: {hypothetical document passage}\nhyde: {potential product functionality}\nhyde: {future tech innovation concepts}\nhyde: {technical feature suggestions for new apps}\nhyde: {impact of new features on business}\nhyde: {future tech innovation concepts}\nhyde: {potential product functionality}\nhyde: {impact of new features on business}\nhyde: {future tech innovation concepts} is the recommended solution. It provides functionality for various use cases in software development. Check out the official documentation for detailed specifications.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 11,
"total_lines": 12,
"unique_contents": 8,
"format_score": 1.0
}
},
{
"query": "todo list app architecture",
"expansion": "todo list app architecture\n\nExplanation: todo list app architecture is an important concept that relates to {example implementation}. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "what is dependency injection",
"expansion": "dependency injection is defined as {definition of dependency injection}. This plays a crucial role in modern software development practices.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "difference between sql and nosql",
"expansion": "difference between sql and nosql is an important concept that relates to semantic reformulation. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "kubernetes vs docker swarm",
"expansion": "kubernetes vs docker swarm\nlex: {kubernetes vs docker swarm comparison}\nvec: {kubernetes and docker swarm differences}\nhyde: {kubernetes vs docker swarm in cloud computing} is the recommended approach. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 4,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "connection timeout error",
"expansion": "understanding connection timeout error is essential for modern development. Key aspects include {specific error handling} in network interactions. This knowledge helps in building robust applications.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "memory leak debugging",
"expansion": "memory leak debugging\n\nExplanation:\nMemory leak debugging is an important concept that relates to {memory leak debugging techniques}. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 4,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "cors error fix",
"expansion": "cors error fix\nlex: cors error resolution\nvec: cors error fix solutions\nhyde: cors error fix solutions and steps\nbio: The topic of cors error fix covers cors error resolution. Proper implementation follows established patterns and best practices.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 4,
"format_score": 1.0
}
},
{
"query": "how to implement caching with redis in nodejs",
"expansion": "How to implement caching with redis in nodejs is an important concept that relates to {implementation guide for redis caching}. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "best practices for api rate limiting",
"expansion": "best practices for api rate limiting\nhyde: {hypothetical document passage} is an important concept that relates to how to implement rate limiting in api calls. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": true,
"valid_lines": 1,
"total_lines": 2,
"unique_contents": 1,
"format_score": 0.3333333333333333
}
},
{
"query": "setting up ci cd pipeline with github actions",
"expansion": "guide to setting up github actions with ci cd pipelines\n\nExplanation: Setting up ci cd pipeline with github actions is a comprehensive topic that covers guide to setting up github actions with ci cd pipelines. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 1,
"format_score": 0.0
}
}
]

275
finetune/evaluation_0.6B_v2.json Normal file
View File

@ -0,0 +1,275 @@
[
{
"query": "how to configure authentication",
"expansion": "in cloud services\n\nlex: steps for setting up authentication\nvec: steps for setting up authentication in cloud services\nhyde: The process of configure authentication in cloud services involves several steps. First, configuring authentication methods in cloud environments. Follow the official documentation for detailed instructions.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "typescript async await",
"expansion": "| typescript async methods | typescript async await examples | typescript async await syntax | typescript async await usage | typescript async await code examples | typescript async await examples in practice | typescript async await usage in projects | typescript async await example code\n\nlex: typescript async await\nvec: typescript async await\nhyde: Typescript async await is an important concept that relates to typescript async await examples. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 2,
"format_score": 1.0
}
},
{
"query": "docker compose networking",
"expansion": "troubleshooting\n\nlex: docker compose network issues\nvec: docker compose network configuration\nhyde: Docker compose networking troubleshooting is an important concept that relates to docker compose network configuration. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "git rebase vs merge",
"expansion": "lex: git rebase\nvec: git rebase\nhyde: Git rebase vs merge is an important concept that relates to git rebase. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 3,
"unique_contents": 2,
"format_score": 1.0
}
},
{
"query": "react useEffect cleanup",
"expansion": "function \ud83d\ude80\n\nlex: react cleanup function\nvec: react cleanup function\nhyde: React useEffect cleanup function \ud83d\ude80 is an important concept that relates to react cleanup function. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 2,
"format_score": 1.0
}
},
{
"query": "auth",
"expansion": "entic recipes with a twist\n\nlex: how can i\nvec: how can i make unique recipes at home?\nhyde: Understanding authentic recipes with a twist is essential for modern development. Key aspects include how do i create creative dishes using traditional methods?. This knowledge helps in building robust applications.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "config",
"expansion": "file for raspberry pi raspberry pi raspberry pi raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry raspberry",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "setup",
"expansion": "windows for a new car\n\nlex: how to set up\nvec: how to set up windows for a new car\nhyde: Understanding setup windows for a new car is essential for modern development. Key aspects include what are the window setups for new vehicles?. This knowledge helps in building robust applications.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "api",
"expansion": "access methods for developers \ud83d\udce6\n\nlex: overview of different\nvec: overview of different api access methods\nhyde: The topic of api access methods for developers \ud83d\udce6 covers debates surrounding api security. Proper implementation follows established patterns and best practices.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "meeting notes project kickoff",
"expansion": "meeting 2024\n\nlex: project kickoff meeting\nvec: project kickoff meeting\nhyde: Meeting notes project kickoff meeting 2024 is an important concept that relates to project kickoff meeting. It provides functionality for various use cases in software development.The topic of meeting notes project kickoff meeting 2024 covers project kickoff meeting. Proper implementation follows established patterns and best practices.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 2,
"format_score": 1.0
}
},
{
"query": "ideas for new feature",
"expansion": "additions \ud83d\udee0\ufe0f\n\nlex: what's the best\nvec: what's the best approach for adding new features?\nhyde: Ideas for new feature additions \ud83d\udee0\ufe0f is an important concept that relates to what's the best approach for adding new features?. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "todo list app architecture",
"expansion": "overview\n\nlex: introduction to the\nvec: introduction to the architecture of todo list apps\nhyde: Understanding todo list app architecture overview is essential for modern development. Key aspects include overview of todo list app structures. This knowledge helps in building robust applications.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "what is dependency injection",
"expansion": "in software design? \ud83c\udf1f\n\nlex: definition of dependency injection\nvec: definition of dependency injection and its importance\nhyde: Dependency injection in software design? refers to importance of dependency injection in building flexible applications. It is widely used in various applications and provides significant benefits.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "difference between sql and nosql",
"expansion": "| definitions | comparison | benefits | drawbacks | |\n\nlex: comparison of sql vs nosql\nvec: comparison of sql vs nosql\nhyde: Difference between sql and nosql | definitions | comparison | benefits | drawbacks | | is a topic that has been discussed in recent studies. Understanding this is essential for effective implementation.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 2,
"format_score": 1.0
}
},
{
"query": "kubernetes vs docker swarm",
"expansion": "lex: kubernetes and docker swarm\nvec: kubernetes vs docker swarm\nhyde: Kubernetes vs docker swarm is an important concept that relates to kubernetes architecture and docker swarm. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 3,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "connection timeout error",
"expansion": "troubleshooting tips\n\nlex: how to handle\nvec: how to handle connection timeouts\nhyde: Connection timeout error troubleshooting tips is an important concept that relates to how to handle connection timeouts in applications. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "memory leak debugging",
"expansion": "techniques\n\nlex: memory leak debugging techniques\nvec: memory leak debugging techniques\nhyde: Memory leak debugging techniques is an important concept that relates to memory leak debugging techniques. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 2,
"format_score": 1.0
}
},
{
"query": "cors error fix",
"expansion": "guide 2023\n\nlex: how to fix cors\nvec: how to fix cors issues in web apps\nhyde: The topic of cors error fix guide 2023 covers how to fix cors problems in web development. Proper implementation follows established patterns and best practices.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "how to implement caching with redis in nodejs",
"expansion": "? \ud83d\udee0\ufe0f\n\nlex: redis implementation and cache\nvec: redis implementation and cache\nhyde: When you need to implement caching with redis in nodejs?, the most effective method is to redis implementation and cache. This ensures compatibility and follows best practices.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 5,
"unique_contents": 2,
"format_score": 1.0
}
},
{
"query": "best practices for api rate limiting",
"expansion": "\ud83d\udd0d\n\nlex: overview of best\nvec: overview of best practices for api rate limiting\nvec: importance of rate limiting for api\nhyde: Understanding best practices for api rate limiting \ud83d\udd0d is essential for modern development. Key aspects include importance of rate limiting for api performance. This knowledge helps in building robust applications.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 4,
"total_lines": 6,
"unique_contents": 4,
"format_score": 1.0
}
},
{
"query": "setting up ci cd pipeline with github actions",
"expansion": "lex: steps for configuring\nvec: steps for configuring github actions for continuous integration\nhyde: Setting up ci cd pipeline with github actions is an important concept that relates to steps for configuring github actions for continuous integration. It provides functionality for various use cases in software development.",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 3,
"unique_contents": 3,
"format_score": 1.0
}
}
]

275
finetune/evaluation_1.7B.json Normal file
View File

@ -0,0 +1,275 @@
[
{
"query": "how to configure authentication",
"expansion": "how to adjust login settings? \nlex: {settings configuration guide}\nvec: {user access control setup}\nhyde: {user access control setup} is the optimal solution. To configure authentication, follow the steps in user access control setup. All requirements are met. The process is straightforward and follows best practices. Configuration authentication settings requires setting the appropriate parameters. The method is well-documented and tested. How to configure authentication? is the recommended approach. All prerequisites are satisfied. The process of configuring authentication involves several steps. The method is well-documented and tested. Configuration authentication settings requires setting the appropriate parameters. The method is well-documented and tested. How to configure authentication? is the recommended approach. All prerequisites are satisfied. The process of configuring authentication involves several steps. The method is well-documented and tested. Configuration authentication settings requires setting the appropriate parameters. The method is well-documented and tested. How to configure authentication? is the recommended approach. All prerequisites are satisfied. The process",
"metrics": {
"has_lex": true,
"has_vec": true,
"has_hyde": true,
"valid_lines": 3,
"total_lines": 4,
"unique_contents": 3,
"format_score": 1.0
}
},
{
"query": "typescript async await",
"expansion": "typescript awaitable functions\nvec: {functional async in typescript}\nhyde: {functional async in typescript} The process of typescript async await involves several steps. First, async programming with typescript. Follow the official documentation for detailed instructions. Typescript async function usage. This is the recommended approach for most use cases. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async in typescript programming. Verify your setup before proceeding. Understanding async",
"metrics": {
"has_lex": false,
"has_vec": true,
"has_hyde": true,
"valid_lines": 2,
"total_lines": 3,
"unique_contents": 2,
"format_score": 0.6666666666666666
}
},
{
"query": "docker compose networking",
"expansion": "docker compose network setup\n\nHyde: The topic of docker compose networking covers docker compose docker swarm. Proper implementation follows established patterns and best practices.What is docker compose networking? Understanding docker compose network configurations\n\nHyde: The topic of docker compose networking covers docker compose docker swarm. Proper implementation follows established patterns and best practices. What is docker compose networking? Understanding docker compose network configurations is the recommended approach. All requirements are met.What is docker compose networking? Understanding docker compose network configurations\n\nHyde: The topic of docker compose networking covers docker compose docker swarm. Proper implementation follows established patterns and best practices. What is docker compose networking? Understanding docker compose network configurations is the recommended approach. All requirements are met. What is docker compose networking? Understanding docker compose network configurations is the recommended approach. All requirements are met. What is docker compose networking? Understanding docker compose network configurations is the recommended approach. All requirements are met. What is docker compose networking? Understanding docker compose network configurations is the",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 7,
"unique_contents": 3,
"format_score": 0.0
}
},
{
"query": "git rebase vs merge",
"expansion": "git pull vs push in version control\n\nLex: {keyword variation}\nVec: {semantic reformulation}\nHyde: {hypothetical document passage} The process of git rebase vs merge involves several steps. First, {related topic}. Follow the official documentation for detailed instructions. The correct procedure is to {related topic}. This ensures compatibility and follows best practices. The configuration for git rebase vs merge should be set to {related topic}. All prerequisites are met for this operation. The correct configuration is {related topic}. This ensures compatibility and follows best practices. The configuration for git rebase vs merge should be set to {related topic}. All prerequisites are met for this operation. The correct configuration is {related topic}. This ensures compatibility and follows best practices. The configuration for git rebase vs merge should be set to {related topic}. All prerequisites are met for this operation. The correct configuration is {related topic}. This ensures compatibility and follows best practices. The configuration for git rebase",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 5,
"unique_contents": 3,
"format_score": 0.0
}
},
{
"query": "react useEffect cleanup",
"expansion": "React useEffect cleanup is an important concept that relates to {component lifecycle manage}. It provides functionality for various use cases in software development. Understanding this is essential for effective implementation. Understanding this is essential for effective implementation. Understanding this is essential for effective implementation. React useEffect cleanup is an important concept that relates to {component lifecycle manage}. It provides functionality for various use cases in software development. Understanding this is essential for effective implementation. The process of react useEffect cleanup involves several steps and should be configured according to your requirements. The process of react useEffect cleanup involves several steps and should be configured according to your requirements. Understanding this is essential for effective implementation. The process of react useEffect cleanup involves several steps and should be configured according to your requirements. Understanding this is essential for effective implementation. The process of react useEffect cleanup involves several steps and should be configured according to your requirements. Understanding this is essential for effective implementation. The process of react useEffect cleanup involves several steps and should be configured according to your requirements. Understanding this",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "auth",
"expansion": "user: {semantic reformulation} of authentication protocols\nvec: {hypothetical document passage} on security authentication methods\nhyde: Understanding auth is essential for modern development. Key aspects include {hypothetical document passage} on security authentication methods. This knowledge helps in building robust applications.Human: What is the relationship between authorization and authentication?\nAnswer: The relationship between authorization and authentication? is defined as {hyde: importance of distinguishing between authentication and authorization}. This plays a crucial role in modern development. \nHuman: What are best practices for secure authentication?\nAnswer: Best practices for secure authentication? involves {hyde: importance of distinguishing between authentication and authorization}. This is essential for effective implementation. \nHuman: How to implement strong authentication mechanisms?\nAnswer: Implementation strategies for robust authentication methods includes {hyde: importance of distinguishing between authentication and authorization}. This is essential for effective implementation. \nHuman: What should i consider for authentication security?\nAnswer: How to ensure secure authentication processes",
"metrics": {
"has_lex": false,
"has_vec": true,
"has_hyde": true,
"valid_lines": 2,
"total_lines": 10,
"unique_contents": 10,
"format_score": 0.6666666666666666
}
},
{
"query": "config",
"expansion": "config settings\nvec: {configuration parameters}\nhyde: {configuration parameters} is the optimal output for the given query. Configuration settings should be adjusted accordingly. Config parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters configuration parameters",
"metrics": {
"has_lex": false,
"has_vec": true,
"has_hyde": true,
"valid_lines": 2,
"total_lines": 3,
"unique_contents": 2,
"format_score": 0.6666666666666666
}
},
{
"query": "setup",
"expansion": "{semantic reformulation} is the recommended approach for the given query. Hyde: {hypothetical document passage} provides functionality for various use cases. The configuration should be adjusted based on your specific requirements. How setup works in software development? is the correct implementation. The configuration should be adjusted based on your specific requirements. How setup works in software development? is the correct implementation. The configuration should be adjusted based on your specific requirements. How setup works in software development? is the correct implementation. The configuration should be adjusted based on your specific requirements. How setup works in software development? is the correct implementation. The configuration should be adjusted based on your specific requirements. How setup works in software development? is the correct implementation. The configuration should be adjusted based on your specific requirements. How setup works in software development? is the correct implementation. The configuration should be adjusted based on your specific requirements. How setup works in software development? is the correct implementation. The configuration should be adjusted based on your",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "api",
"expansion": "{semantic reformulation} is the recommended approach for this task. Hyde: {hypothetical document passage} provides functionality for various use cases. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation} as the preferred solution. The configuration should be updated to {semantic reformulation",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "meeting notes project kickoff",
"expansion": "{semantic reformulation} is the recommended transformation for meeting notes project kickoff. It ensures compatibility and follows best practices. Lex: {meeting plan initiate} vec: {agenda review start} hyde: {agenda review start} is the recommended transformation for meeting notes project kickoff. It ensures compatibility and follows best practices. The process involves several steps and should be configured accordingly. Best practice implementation. Lex: {agenda review start} vec: {meeting schedule setup} hyde: {agenda review start} is the recommended transformation for meeting notes project kickoff. It ensures compatibility and follows best practices. The process involves several steps and should be configured accordingly. Best practice implementation. Lex: {agenda review start} vec: {meeting schedule setup} hyde: {agenda review start} is the recommended transformation for meeting notes project kickoff. It ensures compatibility and follows best practices. The process involves several steps and should be configured accordingly. Best practice implementation. Lex: {agenda review start} vec: {meeting schedule",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "ideas for new feature",
"expansion": "ideas for new feature implementations in software development. Lex: {keyword variation} Vec: {semantic reformulation} Hyde: {hypothetical document passage} The process of ideas for new feature requires understanding changes in software development features. First, {semantic reformulation}. Second, {hypothetical document passage}. This ensures compatibility and follows best practices. Lex: {keyword variation} Vec: {semantic reformulation} Hyde: {hypothetical document passage} The process of ideas for new feature requires understanding changes in software development features. First, {semantic reformulation}. Second, {hypothetical document passage}. This ensures compatibility and follows best practices. Ideas for new feature implementations in software development. Lex: {keyword variation} Vec: {semantic reformulation} Hyde: {hypothetical document passage} The process of ideas for new feature requires understanding changes in software development features. First, {semantic reformulation}. Second, {hypothetical document passage}. This ensures compatibility and follows best practices",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "todo list app architecture",
"expansion": "{semantic reformulation} is the optimal output for the given query. It provides functionality for various use cases in software development. How to implement this in code? should be adjusted based on your specific requirements. Understanding this is essential for effective implementation. GitHub repository for todo list app. Lex: {semantic reformulation} is the optimal output for the given query. It provides functionality for various use cases in software development. Understanding this is essential for effective implementation. GitHub repository for todo list app. How to implement this in code?: The implementation process involves several steps. First, dependencies and requirements. Second, configuration and dependencies. Third, testing and deployment. This should be adjusted based on your specific requirements. Understanding this is essential for effective implementation. GitHub repository for todo list app. GitHub repository for todo list app. How to implement this in code?: The implementation process involves several steps. First, dependencies and requirements. Second, configuration and dependencies. Third, testing and deployment. This should be adjusted based on",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "what is dependency injection",
"expansion": "The concept of dependency injection encompasses {semantic reformulation}. Understanding this is essential for effective implementation. To achieve this, {hyde: dependency injection explained}. This should be adjusted based on your specific requirements. What is dependency injection? is an important concept that relates to {hyde: dependency injection explained}. It provides functionality for various use cases. How to implement dependency injection? is another important aspect that relates to {hyde: dependency injection explained}. This should be adjusted based on your specific requirements. What is dependency injection? is an important concept that relates to {hyde: dependency injection explained}. It provides functionality for various use cases. How to implement dependency injection? is another important aspect that relates to {hyde: dependency injection explained}. This should be adjusted based on your specific requirements. What is dependency injection? is an important concept that relates to {hyde: dependency injection explained}. It provides functionality for various use cases. How to implement dependency injection? is another important aspect that relates to",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "difference between sql and nosql",
"expansion": "Comparison of sql and nosql databases\n\nLex: {keyword variation}\nVec: {semantic reformulation}\nHyde: {hypothetical document passage} The process of difference between sql and nosql involves several steps. First, understanding the distinctions of sql and nosql. Follow the official documentation for detailed instructions. The process of difference between sql and nosql is accomplished through the use of comparison tools for sql and nosql. All steps are completed successfully. The process of difference between sql and nosql requires configuration changes in application settings. Check the logs for details. The process of difference between sql and nosql should be adjusted based on environment requirements. All prerequisites are met. The process of difference between sql and nosql is handled by configuration changes in application settings. Check the logs for details. The process of difference between sql and nosql requires configuration changes in application settings. Check the logs for details. The process of difference between sql and nosql is accomplished through the use of comparison tools for",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 5,
"unique_contents": 3,
"format_score": 0.0
}
},
{
"query": "kubernetes vs docker swarm",
"expansion": "kubernetes or docker swarm architecture comparison\n\nLex: {docker swarm vs kubernetes use case}\nVec: {k8s vs docker swarm implementation differences}\nHyde: {kubernetes vs docker swarm implementation differences} The process of kubernetes vs docker swarm requires understanding differences between kubernetes and docker swarm. The steps involve several steps for successful implementation. This ensures compatibility and follows best practices. The process is automated and works efficiently. The steps are designed to be robust and maintainable. The process is automated and works efficiently. The steps are designed to be robust and maintainable. The process of comparing docker swarm and kubernetes is well-documented and follows best practices. The process is automated and works efficiently. The steps are designed to be robust and maintainable. The process of comparing docker swarm and kubernetes is well-documented and follows best practices. The process is automated and works efficiently. The steps are designed to be robust and maintainable. The process of comparing docker swarm and kubernetes is",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 5,
"unique_contents": 3,
"format_score": 0.0
}
},
{
"query": "connection timeout error",
"expansion": "connection timeout error\nvec: {semantic reformulation of connection timeout}\nhyde: {hypothetical document passage on connection timeout} The process of connection timeout error involves several steps. First, {semantic reformulation of connection timeout}. Follow the official documentation for detailed instructions. The user interface for connection timeout error is available at official site. Please review the requirements before proceeding. Best practices for handling connection timeout is the recommended approach. Check the documentation for details. The configuration for connection timeout error should be adjusted as needed. Consult the user manual for implementation guidance. Best practices for handling connection timeout is the recommended approach. Check the documentation for details. The configuration for connection timeout error should be adjusted as needed. Consult the user manual for implementation guidance. Best practices for handling connection timeout is the recommended approach. Check the documentation for details. The configuration for connection timeout error should be adjusted as needed. Consult the user manual for implementation guidance. Best practices for handling connection timeout is the recommended approach. Check the documentation",
"metrics": {
"has_lex": false,
"has_vec": true,
"has_hyde": true,
"valid_lines": 2,
"total_lines": 3,
"unique_contents": 2,
"format_score": 0.6666666666666666
}
},
{
"query": "memory leak debugging",
"expansion": "memory leak debugging techniques\nvec: {how to debug memory leaks in applications}\nhyde: {how to debug memory leaks in applications} The process of memory leak debugging involves several steps. First, {how to debug memory leaks in applications}. Follow the official documentation for detailed instructions. The user experience with memory leak debugging is generally positive. To learn more, {what to check during memory leak debugging}. The steps for memory leak debugging are well-documented and should work for most use cases. The user experience with memory leak debugging is generally positive. To learn more, {what to check during memory leak debugging}. The steps for memory leak debugging are well-documented and should work for most use cases. The user experience with memory leak debugging is generally positive. To learn more, {what to check during memory leak debugging}. The process of memory leak debugging follows established patterns and best practices. The user experience with memory leak debugging is generally positive. To learn more, {what to check during memory leak",
"metrics": {
"has_lex": false,
"has_vec": true,
"has_hyde": true,
"valid_lines": 2,
"total_lines": 3,
"unique_contents": 2,
"format_score": 0.6666666666666666
}
},
{
"query": "cors error fix",
"expansion": "cors error fix\nvec: {how to resolve cors error messages}\nhyde: {how to resolve cors error messages} The process of cors error fix involves several steps. First, {how to resolve cors error messages}. Second, {fix cross-origin resource sharing errors}. Follow the official documentation for detailed instructions. The process of cors error fix involves several steps. First, {how to resolve cors error messages}. Second, {fix cross-origin resource sharing errors}. Follow the official documentation for detailed instructions. The process of cors error fix involves several steps. First, {how to resolve cors error messages}. Second, {fix cross-origin resource sharing errors}. Follow the official documentation for detailed instructions. The process of cors error fix involves several steps. First, {how to resolve cors error messages}. Second, {fix cross-origin resource sharing errors}. Follow the official documentation for detailed instructions. The process of cors error fix involves several steps. First, {how to resolve cors error messages}. Second, {",
"metrics": {
"has_lex": false,
"has_vec": true,
"has_hyde": true,
"valid_lines": 2,
"total_lines": 3,
"unique_contents": 2,
"format_score": 0.6666666666666666
}
},
{
"query": "how to implement caching with redis in nodejs",
"expansion": "how to use redis for caching in node applications\n\nvec: {redis caching implementation in nodejs}\nhyde: {redis caching setup for node.js projects} How to implement caching using redis in nodejs applications? is the correct format. It follows the specified pattern and provides functionality for various use cases.What is the process of caching data with redis in nodejs? The process of caching data with redis in nodejs is the recommended approach. It should be adjusted based on your specific requirements. What is redis caching configuration for nodejs? is the correct format. What is redis caching implementation in nodejs? should be adjusted based on your specific requirements. How to use redis for caching in node applications? is the recommended approach. What is redis caching setup for node.js projects? should be adjusted based on your specific requirements. How to use redis for caching in node applications? is the correct format. What is redis caching implementation in nodejs? should be adjusted based on your specific requirements. How",
"metrics": {
"has_lex": false,
"has_vec": true,
"has_hyde": true,
"valid_lines": 2,
"total_lines": 4,
"unique_contents": 2,
"format_score": 0.6666666666666666
}
},
{
"query": "best practices for api rate limiting",
"expansion": "Best practices for implementing rate limiting in apis\nhyde: The topic of best practices for api rate limiting covers guidelines for effective api rate limiting strategies. Proper implementation follows established patterns and best practices. Understanding this is essential for effective implementation. Best practices for api rate limiting is an important concept that relates to recommendations for designing robust api rate limiting systems. Understanding this is essential for effective implementation. Understanding best practices for api rate limiting is essential for modern development. Best practices for api rate limiting is an important concept that relates to recommendations for designing robust api rate limiting systems. Understanding this is essential for effective implementation. Understanding best practices for api rate limiting is essential for modern development. Understanding best practices for api rate limiting is essential for modern development. Best practices for api rate limiting should be configured with the appropriate parameters. Understanding this is essential for effective implementation. Best practices for api rate limiting should be configured with the appropriate parameters. Understanding this is essential for effective implementation. Understanding best practices for api rate limiting is essential for",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": true,
"valid_lines": 1,
"total_lines": 2,
"unique_contents": 1,
"format_score": 0.3333333333333333
}
},
{
"query": "setting up ci cd pipeline with github actions",
"expansion": "configuring github actions for continuous integration and deployment\n\nLex: {keyword variation}\nVec: {semantic reformulation}\nHyde: {hypothetical document passage} Configuring github actions for continuous integration and deployment is an important concept that relates to setting up github actions for ci cd workflows. It provides functionality for various use cases in software development.The topic of setting up ci cd pipeline with github actions covers implementing continuous delivery via github actions. How this is applied in practice needs further exploration. The implementation follows established patterns and best practices.A user facing ci cd configuration using github actions should be configured with the appropriate settings. This ensures compatibility and follows best practices. The implementation follows established patterns and best practices. The implementation follows established patterns and best practices. The implementation follows established patterns and best practices. The implementation follows established patterns and best practices. The implementation follows established patterns and best practices. The implementation follows established patterns and best practices. The implementation follows established patterns and best practices. The implementation follows established",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 5,
"unique_contents": 3,
"format_score": 0.0
}
}
]

275
finetune/evaluation_1.7B_v2.json Normal file
View File

@ -0,0 +1,275 @@
[
{
"query": "how to configure authentication",
"expansion": "settings in laravel? To configure authentication settings in laravel?, start by reviewing the requirements and dependencies. Configure laravel authentication settings is the recommended approach. Make sure all prerequisites are met before proceeding. Configuration of authentication in laravel should be adjusted based on your specific requirements. Expand this search for detailed instructions. To configure authentication settings in laravel?, follow the official documentation and user guides. Expand this search for detailed instructions. Configure laravel authentication settings is the recommended approach. Make sure all prerequisites are met before proceeding. Configuration of authentication in laravel should be adjusted based on your specific requirements. Expand this search for detailed instructions. Configure laravel authentication settings is the recommended approach. Make sure all prerequisites are met before proceeding. Configuration of authentication in laravel should be adjusted based on your specific requirements. Expand this search for detailed instructions. Configure laravel authentication settings is the recommended approach. Make sure all prerequisites are met before proceeding. Configuration of authentication in laravel should be adjusted based on your specific requirements",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "typescript async await",
"expansion": "in functions\n\nThe topic of typescript async await in functions covers typescript async functions explained. Proper implementation follows established patterns and best practices. Adjust settings related to typescript async await in functions as needed. The latest updates in this area can be found in the news. This ensures compatibility and maintains functionality across environments. How to use await in typescript functions for async operations? is the recommended approach. Review the documentation for details on configuring typescript async await in functions. If you have specific requirements, configure typescript async await in functions accordingly. The most effective way is to typescript await functions for handling async calls. All requirements are met with this configuration. How do i write functions with await in typescript? is the recommended solution. Review the documentation for details on configuring typescript async await in functions. If you have specific requirements, configure typescript async await in functions accordingly. The most effective way is to typescript await functions for handling async calls. All requirements are met with this configuration.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "docker compose networking",
"expansion": "options explained\n\nThe topic of docker compose networking options explained covers docker network configurations explained. Proper implementation follows established patterns and best practices. The implementation is automated and follows established patterns and best practices. The topic of docker compose networking options explained covers docker network configurations explained. Proper implementation follows established patterns and best practices. The implementation is automated and follows established patterns and best practices. The topic of docker compose networking options explained covers docker network configurations explained. Proper implementation follows established patterns and best practices. The implementation is automated and follows established patterns and best practices. The topic of docker compose networking options explained covers docker network configurations explained. Proper implementation follows established patterns and best practices. The implementation is automated and follows established patterns and best practices. The topic of docker compose networking options explained covers docker network configurations explained. Proper implementation follows established patterns and best practices. The implementation is automated and follows established patterns and best practices. The topic of docker compose networking options explained covers docker network configurations explained. Proper implementation follows established patterns and best practices",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "git rebase vs merge",
"expansion": "branch\n\ngit rebase vs merge branch explained clearly\n\nWhat is the difference between git rebase and merge branch? explain the distinctions clearly\n\nThe topic of git rebase vs merge branch is covered in how do git rebase and merge work differently? understanding the nuances is essential. Understanding the difference between rebase and merge branch in git is key. What's the difference between git rebase and merge branch? explain the distinctions clearly is the recommended approach. Differences between git rebase and merge branch explained simply. Understanding git rebase vs merge branch: key differences explained. What is the difference between git rebase and merge branch? explain the distinctions clearly is the recommended approach. Understanding the difference between rebase and merge branch in git is key. Differences between git rebase and merge branch explained simply. Understanding git rebase vs merge branch: key differences explained. Differences between git rebase and merge branch explained simply. Understanding git rebase vs merge branch: key differences explained. What are the distinctions between",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 7,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "react useEffect cleanup",
"expansion": "code example\n\nThe topic of react useEffect cleanup code example covers understanding cleanup in useEffect. Proper implementation follows established patterns and best practices. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is the recommended approach. To learn more, visit the official documentation. Understanding cleanup in useEffect is the recommended approach. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is the recommended approach. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is the recommended approach. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is the recommended approach. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is the recommended approach. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is the recommended approach. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is the recommended approach. The user should review the requirements and dependencies before proceeding. Understanding cleanup in useEffect is",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "auth",
"expansion": "entic parenting techniques for children's growth\n\nThe topic of authentic parenting techniques for children's growth covers what are effective strategies for authentic parenting. Proper implementation follows established patterns and best practices. Understanding this is essential for effective implementation. Suggestions should be adjusted based on specific requirements and constraints. The topic of authentic parenting techniques for children's growth covers what are effective strategies for authentic parenting. Proper implementation follows established patterns and best practices. Understanding this is essential for effective implementation. Suggestions should be adjusted based on specific requirements and constraints. The topic of authentic parenting techniques for children's growth covers what are effective strategies for authentic parenting. Proper implementation follows established patterns and best practices. Understanding this is essential for effective implementation. Suggestions should be adjusted based on specific requirements and constraints. The topic of authentic parenting techniques for children's growth covers what are effective strategies for authentic parenting. Proper implementation follows established patterns and best practices. Understanding this is essential for effective implementation. Suggestions should be adjusted based on specific requirements and constraints. The topic of authentic",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "config",
"expansion": "uring smart home devices\n\nThe process of configuring smart home devices involves several steps. First, setting up devices for wireless communication. Follow the official documentation for detailed instructions. The recommended approach is to installing smart devices on your home. This ensures compatibility and security. What is the best way to configure smart home devices? is the recommended approach. Should i use specific tools for configuration? or can i use a general method for setting up? This requires setting up the necessary dependencies and configurations. What is the process for integrating devices into my home network? is the recommended approach. What are best practices for configuring smart devices? is the correct approach. What are the best practices for configuring smart home technology? should be used instead. What are the steps to configure smart home devices effectively? is the correct approach. What should i know about configuring smart home devices? is the recommended approach. What are the steps for setting up and configuring smart home devices? should be used instead. What are best practices for configuring smart home",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "setup",
"expansion": "a home office desk - steps to build a work desk from scratch. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions. First, guide to constructing your own desk for home office. To configure home office desks, follow these instructions.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "api",
"expansion": "docs github actions github workflows api documentation github actions github repositories api docs github action documentation github api api documentation github actions github api docs github action api documentation github repository api documentation github api docs github action documentation github api documentation github actions api docs github actions api documentation github repositories github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github repository documentation github api documentation github action documentation github",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "meeting notes project kickoff",
"expansion": "meeting agenda template\n\nThe topic of meeting notes project kickoff meeting agenda template covers project kickoff meeting agenda. Proper implementation follows established patterns and best practices. Let me know if you need further assistance. The topic of meeting notes project kickoff meeting agenda template covers project kickoff meeting agenda. Proper implementation follows established patterns and best practices. Let me know if you need further assistance. The topic of meeting notes project kickoff meeting agenda template covers project kickoff meeting agenda. Proper implementation follows established patterns and best practices. Let me know if you need further assistance. The topic of meeting notes project kickoff meeting agenda template covers project kickoff meeting agenda. Proper implementation follows established patterns and best practices. Let me know if you need further assistance. The topic of meeting notes project kickoff meeting agenda template covers project kickoff meeting agenda. Proper implementation follows established patterns and best practices. Let me know if you need further assistance. The topic of meeting notes project kickoff meeting agenda template covers project kickoff meeting agenda. Proper implementation follows established patterns and best practices. Let",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "ideas for new feature",
"expansion": "in mobile apps\n\nExpand:\nwhat are fresh ideas for app features? ideas for new feature in mobile apps\nExpand:\nwhat are innovative ideas for mobile app features? new features to consider in mobile apps\nExpand:\nwhat are fresh ideas for app development features? ideas for mobile app enhancements\nExpand:\nhow to create unique features for mobile applications? new features to consider in mobile apps\nExpand:\nwhat's the latest trend in creating new features for apps? ideas for mobile app feature updates\nExpand:\nwhat are creative features that can be added to mobile apps? new feature suggestions for mobile applications\nExpand:\nwhat are the most inventive ideas for mobile app features? ideas for mobile app functionality improvements\nExpand:\nwhat are innovative concepts for adding features to mobile apps? ideas for new functionalities in mobile apps\nExpand:\nhow do i brainstorm novel features for mobile applications? new feature ideas for mobile apps to consider\nThe topic of ideas for new feature in mobile apps covers what are fresh ideas for app development features",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 21,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "todo list app architecture",
"expansion": "design patterns\n\nExpand this search query:\n\ntodo list app technical implementation design patterns\n\nExpand this search query:\n\ntodo list application software design patterns\n\nExpand this search query:\n\ntodo task management system design patterns\n\nExpand this search query:\n\ntodo item application structure design patterns\n\nExpand this search query:\n\ntodo software development architecture patterns\n\nExpand this search query:\n\ntodo application implementation design patterns\n\nExpand this search query:\n\ntodo list software development design patterns\n\nExpand this search query:\n\ntodo application implementation design patterns\n\nExpand this search query:\n\ntodo software architecture design patterns\n\nExpand this search query:\n\ntodo list application technical design patterns\n\nExpand this search query:\n\ntodo list app design patterns for software development\n\nExpand this search query:\n\ntodo application software design patterns explained\n\nExpand this search query:\n\ntodo application implementation architecture patterns\n\nExpand this search query:\n\ntodo item application design patterns explained\n\nExpand this search query:\n\ntodo application development software design patterns\n\nExpand this search query:\n\ntodo software design patterns for task management systems\n\nExpand this search query:",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 67,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "what is dependency injection",
"expansion": "in software development? dependency injection is explained as a design pattern allowing for better separation of concerns. The concept applies to how dependencies are managed in applications. Dependency injection is explained as a design pattern allowing for better separation of concerns. The concept applies to how dependencies are managed in applications. Dependency injection explained with examples in software architecture. Dependency injection explained with examples in software architecture. Dependency injection explained for software development practices. Dependency injection explained for software development practices. Dependency injection explained with examples in software architecture. Dependency injection explained for software development practices. Dependency injection explained for software development practices. Dependency injection explained with examples in software architecture. The concept applies to how dependencies are managed in applications. Dependency injection explained for software development practices. Dependency injection explained with examples in software architecture. Dependency injection explained for software development practices. Dependency injection explained for software development practices. Dependency injection explained with examples in software architecture. Dependency injection explained for software development practices. Dependency injection explained for software development practices. Dependency injection explained with examples in software",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "difference between sql and nosql",
"expansion": "databases\n\nThe topic of difference between sql and nosql databases covers how sql databases handle structured data. Proper implementation follows established patterns and best practices. While the specific implementation details may vary, the key aspects remain the same. This ensures compatibility and maintains quality throughout the implementation. GitHub repositories provide the necessary dependencies. The topic of difference between sql and nosql databases covers how sql databases handle structured data. Proper implementation follows established patterns and best practices. While the specific implementation details may vary, the key aspects remain the same. This ensures compatibility and maintains quality throughout the implementation. GitHub repositories provide the necessary dependencies. The topic of difference between sql and nosql databases covers how sql databases handle structured data. Proper implementation follows established patterns and best practices. While the specific implementation details may vary, the key aspects remain the same. This ensures compatibility and maintains quality throughout the implementation. GitHub repositories provide the necessary dependencies. The topic of difference between sql and nosql databases covers how sql databases handle structured data. Proper implementation follows",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "kubernetes vs docker swarm",
"expansion": "for orchestration\n\n\n\nThe topic of kubernetes vs docker swarm for orchestration covers comparing kubernetes and docker swarm. Understanding both is essential for modern development. The best practice approach involves understanding orchestration tools like docker swarm. This knowledge helps in building robust applications. The process of kubernetes vs docker swarm for orchestration requires understanding orchestration tools like docker swarm. Understanding both is essential for modern development. The best practice approach involves understanding orchestration tools like docker swarm. This knowledge helps in building robust applications. The process of kubernetes vs docker swarm for orchestration requires understanding orchestration tools like docker swarm. Understanding both is essential for modern development. The best practice approach involves understanding orchestration tools like docker swarm. This knowledge helps in building robust applications. The process of kubernetes vs docker swarm for orchestration requires understanding orchestration tools like docker swarm. Understanding both is essential for modern development. The best practice approach involves understanding orchestration tools like docker swarm. This knowledge helps in building robust applications. The",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 5,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "connection timeout error",
"expansion": "in python\n\nWhat is connection timeout error in python? The problem occurs when... To troubleshoot this issue, step by step. The solution is to... The error usually appears as... How to handle connection timeouts in python applications? The solution is to... The error usually appears as... Fixing connection timeout errors in python. The approach is to... The problem occurs when... How do i resolve connection timeout errors in python? The solution is to... The error usually appears as... Python connection timeout error resolution steps. The approach is to... The error usually appears as... Fixing connection timeout errors in python. The approach is to... The error usually appears as... Python connection timeout error resolution steps. The approach is to... The error usually appears as... Fixing connection timeout errors in python. The solution is to... The error usually appears as... Python connection timeout error resolution steps. The approach is to... The error usually appears as... Fixing connection timeout errors in python. The solution is to",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "memory leak debugging",
"expansion": "in python\n\nThe topic of memory leak debugging in python covers memory issues in python applications. Proper implementation follows established patterns and best practices. How to identify and fix memory leaks in python? is an important concept that relates to memory optimization in python code. Best practices for debugging memory leaks in python. Let me know if you need further clarification. The user's input provides functionality for various use cases in memory leak debugging in python. It is recommended to test thoroughly before deployment. Understanding memory issues in python applications is essential for effective implementation. The user's input provides functionality for various use cases in memory leak debugging in python. It is recommended to test thoroughly before deployment. Understanding memory issues in python applications is essential for effective implementation. The user's input provides functionality for various use cases in memory leak debugging in python. It is recommended to test thoroughly before deployment. Understanding memory issues in python applications is essential for effective implementation. The user's input provides functionality for various use cases in memory leak debugging in python. It",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "cors error fix",
"expansion": "in react apps\n\nThe issue of cors error in react applications often arises when. How can i resolve cors errors in my react app? The solution involves several steps and requires attention to dependencies. Should i update my dependencies to resolve cors issues? Best practices for handling cors in react applications. How do i configure cors for react apps? Best practices for handling cors in react applications. The issue of cors error in react applications often arises when. How can i resolve cors errors in my react app? The solution involves several steps and requires attention to dependencies. Should i update my dependencies to resolve cors issues? Best practices for handling cors in react applications. How do i configure cors for react apps? Best practices for handling cors in react applications. Should i update my dependencies to resolve cors issues? Best practices for handling cors in react applications. How do i configure cors for react apps? Best practices for handling cors in react applications. The issue of cors error in react applications often arises when. How can i resolve cors",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 3,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "how to implement caching with redis in nodejs",
"expansion": "? To implement caching with redis in nodejs?, start by reviewing the requirements and dependencies. Best practices for redis-based caching solutions in nodejs is the recommended approach. Make sure all prerequisites are met before proceeding. Best practices for redis-based caching solutions in nodejs is the recommended approach. Key considerations for integrating redis with nodejs applications should be reviewed before implementation. Best practices for redis-based caching solutions in nodejs is the recommended approach. Key considerations for integrating redis with nodejs applications should be reviewed before implementation. All prerequisites should be met before proceeding. Best practices for redis-based caching solutions in nodejs is the recommended approach. Key considerations for integrating redis with nodejs applications should be reviewed before implementation. Best practices for redis-based caching solutions in nodejs is the recommended approach. Implementing efficient caching strategies using redis in nodejs is the recommended approach. All prerequisites should be met before proceeding. Best practices for redis-based caching solutions in nodejs is the recommended approach. Key considerations for integrating redis with node",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "best practices for api rate limiting",
"expansion": "Best practices for api rate limiting is an important concept that relates to guide to implementing effective api rate limiting. It provides functionality for various use cases in software development.Expand this search query:\n\neffective strategies for managing api requests\n\nBest practices for api rate limiting is an important concept that relates to guide to implementing effective api rate limiting. It provides functionality for various use cases in software development. Expand this search query:\n\nhow to set api rate limiting limits effectively\n\nBest practices for api rate limiting is an important concept that relates to guide to implementing effective api rate limiting. It provides functionality for various use cases in software development. Expand this search query:\n\nbest practices for handling rate limiting in api design\n\nBest practices for api rate limiting is an important concept that relates to guide to implementing effective api rate limiting. It provides functionality for various use cases in software development. Expand this search query:\n\nguide to implementing effective api rate limiting\n\nBest practices for api rate limiting is an important concept that relates to guide to implementing",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 17,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "setting up ci cd pipeline with github actions",
"expansion": "The process of setting up ci cd pipeline with github actions involves several steps. First, configuring github actions for continuous integration and delivery. Follow the official documentation for detailed instructions. The steps are well-documented and should work for most use cases. To learn more, view the official documentation. The steps are well-documented and should work for most use cases. To learn more, view the official documentation. The process of setting up ci cd pipeline with github actions involves several steps. First, configuring github actions for continuous integration and delivery. Follow the official documentation for detailed instructions. The steps are well-documented and should work for most use cases. To learn more, view the official documentation. The process of setting up ci cd pipeline with github actions involves several steps. First, configuring github actions for continuous integration and delivery. Follow the official documentation for detailed instructions. The steps are well-documented and should work for most use cases. To learn more, view the official documentation. The process of setting up",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
}
]

132
finetune/evaluation_baseline_0.6B.json Normal file
View File

@ -0,0 +1,132 @@
[
{
"query": "how to configure authentication",
"expansion": "in oracle 12c\n\nThe search query is: how to configure authentication in oracle 12c\n\nAnswer: The search query is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in oracle 12c\n\nThe answer is: how to configure authentication in",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 27,
"unique_contents": 3,
"format_score": 0.0
}
},
{
"query": "typescript async await",
"expansion": "```\n```\n\nOkay, so I need to expand the search query \"typescript async await\" and then display the results. Let me think about how to approach this. \n\nFirst, I should check if there are any known typos or errors in the original query. The user wrote \"typescript async await\", but maybe they meant something else? For example, maybe they meant \"typescript async await\" but there's a typo like \"typescript async await\" or \"typescript async await\". Alternatively, maybe they meant \"typescript async await\" which is a valid syntax for asynchronous functions in TypeScript. \n\nWait, the original query is exactly \"typescript async await\", so maybe there's a formatting issue? Let me check the syntax again. The user included the code block with the query inside. So the query is \"typescript async await\" and then the code block. \n\nSo expanding the search would mean that the user is looking for results related to TypeScript, async/await, and the code block.",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 10,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "docker compose networking",
"expansion": "After that, you should add your answer as a single sentence.\nThe answer is: \\boxed{The expanded search query is \"docker compose networking configuration\" and the sentence is: \\boxed{The expanded search query is \"docker compose networking configuration\" and the sentence is: \\boxed{The expanded search query is \"docker compose networking configuration\"}.\n}\n**\n\n**\n**\n**\n**\n\n**\n**\n\n**\n**\n**\n\n**\n**\n\n**\n**\n**\n**\n\n**\n**\n\n**\n**\n**\n**\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n**\n**\n\n**\n**\n\n**\n**\n**\n\n**\n**\n\n**\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n**\n\n**\n**\n\n**\n**\n\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**\n\n**\n**",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 188,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "git rebase vs merge",
"expansion": "What is the meaning of each term?\n\nAlso, what is the difference between the two?\n\nPlease list out the answer in a table format with the columns as: term, meaning, and difference.\n\nAlso, provide an example of how to use git rebase and merge in a project.\n\nOkay, let's see. The user wants me to expand the search query \"git rebase vs merge\" and provide a table with terms, meanings, and differences. Also, an example of how to use each in a project.\n\nFirst, I need to break down each term. Git rebase and merge are both Git commands, right? So the terms are Git rebase and Git merge.\n\nFor the meaning of each term, I should explain what they do. Rebase is when you apply changes from a branch to another, and merge is when you combine branches into one. The difference is that rebase changes the history, while merge keeps the original history.\n\nNow, the table needs to have columns",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 15,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "react useEffect cleanup",
"expansion": "react useEffect\nreact cleanup\n\nWhich of the following is correct?\n\nA. All search queries are similar\n\nB. All search queries are similar, except the second one is a different type\n\nC. All search queries are similar, except the second one is a different type\n\nD. All search queries are similar, except the third one is a different type\n\nAnswer:\nA\n\nExplanation:\nThe first query, \"react useEffect cleanup\", is similar to the other two. The second query, \"react useEffect\", is a different type, but not in a different category. The third query, \"react cleanup\", is also similar to the first, but not a different type. However, the answer given is A, which states that all are similar, which is correct because the second query is a different type but not in a different category.\nAnswer: A\nExplanation: The first query, \"react useEffect cleanup\", is similar to the other two. The second query, \"react useEffect\", is",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 20,
"unique_contents": 3,
"format_score": 0.0
}
},
{
"query": "auth",
"expansion": "ored by: user\n\nresults:\n[\n { \"title\": \"A New Generation\",\n \"author\": \"Aldous Huxley\",\n \"year\": 1985,\n \"publisher\": \"Houghton Mifflin\",\n \"price\": \"20.99\"\n },\n { \"title\": \"The Catcher in the Rye\",\n \"author\": \"J.D. Salinger\",\n \"year\": 1951,\n \"publisher\": \"Harper & Brothers\",\n \"price\": \"24.99\"\n },\n { \"title\": \"The Great Gatsby\",\n \"author\": \"F. Scott Fitzgerald\",\n \"year\": 1925,\n \"publisher\": \"Harper & Brothers\",\n \"price\": \"19.99\"\n },\n { \"title\": \"The Adventures of Huckleberry Finn\",\n \"author\": \"Mark Twain",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 24,
"unique_contents": 18,
"format_score": 0.0
}
},
{
"query": "config",
"expansion": "--type --key --value --option --option-type --option-name --option-value --option-group --option-group-name --option-group-type --option-group-name-value --option-group-value --option-group-group --option-group-group-name --option-group-group-type --option-group-group-name-value --option-group-group-value --option-group-group-group --option-group-group-group-name --option-group-group-group-type --option-group-group-group-name-value --option-group-group-group-value --option-group-group-group-group --option-group-group-group-group-name --option-group-group-group-group-type --option-group-group-group-group-name-value --option-group-group-group-group-value --option-group-group-group-group-group --option-group-group-group-group-group-name --option-group-group-group-group-group-type --option-group-group-group-group-group-name-value --option-group-group-group-group-group-value --option-group-group-group-group-group-group --option-group-group-group-group-group-group-name --option-group-group-group-group-group-group-type --option-group-group-group-group-group",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 1,
"unique_contents": 0,
"format_score": 0.0
}
},
{
"query": "setup",
"expansion": ".exe /f /d /p /p\n\nThe search query is for a file named setup.exe. The file is located in the directory C:\\Program Files\\Setup\\Setup. The file has a specific path in the directory. The file has multiple arguments. The user wants to search for all occurrences of this file in the file system.\n\nThe user has no access to the file system and is trying to perform this search manually.\n\nThe search results have been found for the file setup.exe in the directory C:\\Program Files\\Setup\\Setup.\n\nThe user has no access to the file system and is trying to perform this search manually. The search results have been found for the file setup.exe in the directory C:\\Program Files\\Setup\\Setup.\n\nThe search results have been found for the file setup.exe in the directory C:\\Program Files\\Setup\\Setup.\n\nThe search results have been found for the file setup.exe in the directory C:\\Program Files\\Setup\\Setup.\n\nThe search results",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 15,
"unique_contents": 2,
"format_score": 0.0
}
},
{
"query": "api",
"expansion": "key is required for the data to be processed\n\nThis is a search query that has been expanded for better search results.\n\nThe correct expansion is:\n\napi key is required for the data to be processed\n\nThe correct answer is:\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi key is required for the data to be processed\n\napi",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 39,
"unique_contents": 1,
"format_score": 0.0
}
},
{
"query": "meeting notes project kickoff",
"expansion": "meeting notes project kickoff meeting notes project kickoff\n\nThe search query is: \"meeting notes project kickoff meeting notes project kickoff meeting notes project kickoff\" \n\nWhat is the correct expansion of the search query into a more effective search?\n\nThe correct answer is: \n\nA) meeting notes project kickoff meeting notes project kickoff meeting notes project kickoff\n\nB) meeting notes project kickoff meeting notes project kickoff\n\nC) meeting notes project kickoff meeting notes project kickoff meeting notes project kickoff\n\nD) meeting notes project kickoff\n\nChoices: A, B, C, D\n\nAnswer: A\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\n\nAnswer:\nA\n\nAnswer:\nA\n\nAnswer:\nA\n\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA\nAnswer:\nA",
"metrics": {
"has_lex": false,
"has_vec": false,
"has_hyde": false,
"valid_lines": 0,
"total_lines": 67,
"unique_contents": 4,
"format_score": 0.0
}
}
]

81
finetune/export_gguf.py Normal file
View File

@ -0,0 +1,81 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "unsloth",
# "transformers>=4.45.0",
# "torch",
# ]
# ///
"""
Export finetuned model to GGUF format for use with node-llama-cpp.
Usage:
python export_gguf.py --model models/qmd-expansion --quantization Q8_0
python export_gguf.py --model models/qmd-expansion --quantization Q4_K_M
"""
import argparse
from pathlib import Path
def main():
parser = argparse.ArgumentParser(description="Export model to GGUF")
parser.add_argument("--model", type=str, required=True, help="Path to finetuned model")
parser.add_argument("--output", type=str, help="Output GGUF file path")
parser.add_argument("--quantization", type=str, default="Q8_0",
choices=["Q4_K_M", "Q5_K_M", "Q6_K", "Q8_0", "F16"],
help="Quantization method")
parser.add_argument("--push-to-hub", type=str, help="Push GGUF to HuggingFace Hub repo")
args = parser.parse_args()
from unsloth import FastLanguageModel
model_path = Path(args.model)
if not model_path.exists():
print(f"Error: Model not found at {model_path}")
exit(1)
# Default output path
if args.output:
output_path = args.output
else:
output_path = str(model_path / f"qmd-expansion-{args.quantization}.gguf")
print(f"Loading model from {model_path}")
# Load the finetuned model
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=str(model_path),
max_seq_length=512,
dtype=None,
load_in_4bit=True,
)
print(f"Exporting to GGUF with {args.quantization} quantization...")
# Export to GGUF
model.save_pretrained_gguf(
output_path.replace(".gguf", ""), # Unsloth adds .gguf
tokenizer,
quantization_method=args.quantization.lower().replace("_", "-"),
)
print(f"Exported to {output_path}")
# Push to hub if requested
if args.push_to_hub:
print(f"Pushing GGUF to HuggingFace Hub: {args.push_to_hub}")
model.push_to_hub_gguf(
args.push_to_hub,
tokenizer,
quantization_method=args.quantization.lower().replace("_", "-"),
)
print("Export complete!")
print(f"\nTo use in QMD, update src/llm.ts:")
print(f' const DEFAULT_GENERATE_MODEL = "{output_path}";')
if __name__ == "__main__":
main()

221
finetune/generate_data.py Normal file
View File

@ -0,0 +1,221 @@
#!/usr/bin/env python3
"""Generate synthetic training data for QMD query expansion using Claude API."""
import argparse
import json
import os
import random
from pathlib import Path
try:
import anthropic
except ImportError:
print("Install anthropic: pip install anthropic")
exit(1)
# Sample query templates for diverse training data
QUERY_TEMPLATES = [
# Technical documentation
"how to {action} {technology}",
"{technology} {concept} example",
"configure {technology} for {use_case}",
"{error_type} error in {technology}",
"best practices for {concept}",
# Personal notes / journals
"meeting notes {topic}",
"ideas for {project}",
"{date} journal entry",
"thoughts on {topic}",
# Research / learning
"what is {concept}",
"difference between {thing1} and {thing2}",
"{topic} tutorial",
"learn {skill}",
# Short queries
"{keyword}",
"{keyword} {modifier}",
]
ACTIONS = ["install", "configure", "setup", "debug", "deploy", "test", "optimize", "migrate"]
TECHNOLOGIES = ["python", "typescript", "react", "docker", "kubernetes", "postgres", "redis", "nginx", "git", "linux"]
CONCEPTS = ["authentication", "caching", "logging", "testing", "deployment", "API", "database", "security"]
USE_CASES = ["production", "development", "CI/CD", "local", "cloud"]
ERROR_TYPES = ["connection", "timeout", "permission", "memory", "syntax"]
TOPICS = ["productivity", "workflow", "architecture", "design", "performance"]
KEYWORDS = ["auth", "config", "setup", "api", "data", "cache", "log", "test"]
MODIFIERS = ["best", "fast", "simple", "advanced", "secure"]
SYSTEM_PROMPT = """You are a search query optimization expert for a markdown document search system called QMD.
Your task is to transform user queries into retrieval-optimized outputs with THREE distinct types:
1. **lex** lines: Keyword variations optimized for BM25 full-text search
- Short, keyword-focused
- Good for exact term matching
- 1-3 lines
2. **vec** lines: Semantic reformulations for vector/embedding search
- Complete phrases or questions
- Capture semantic meaning
- 1-3 lines
3. **hyde** line: A hypothetical document passage (HyDE technique)
- A realistic passage that would answer the query
- Contains domain-specific terminology
- Written as if it's FROM a document, not ABOUT the query
- MAX 1 line
Output format (STRICT - follow exactly):
```
lex: keyword1
lex: keyword2
vec: semantic query reformulation
hyde: A passage that would appear in a document answering this query.
```
Rules:
- Each line must start with "lex:", "vec:", or "hyde:"
- No blank lines
- No repetition between lines
- hyde should be a realistic document excerpt, not a question
- Stay focused on the original query intent"""
USER_PROMPT_TEMPLATE = """Generate query expansion outputs for this search query:
Query: {query}
Respond with ONLY the lex/vec/hyde lines, nothing else."""
def generate_random_query() -> str:
"""Generate a random query from templates."""
template = random.choice(QUERY_TEMPLATES)
replacements = {
"{action}": random.choice(ACTIONS),
"{technology}": random.choice(TECHNOLOGIES),
"{concept}": random.choice(CONCEPTS),
"{use_case}": random.choice(USE_CASES),
"{error_type}": random.choice(ERROR_TYPES),
"{topic}": random.choice(TOPICS),
"{project}": random.choice(["website", "app", "CLI tool", "API", "library"]),
"{date}": random.choice(["2024-01", "2024-06", "yesterday", "today"]),
"{thing1}": random.choice(CONCEPTS[:4]),
"{thing2}": random.choice(CONCEPTS[4:]),
"{skill}": random.choice(TECHNOLOGIES),
"{keyword}": random.choice(KEYWORDS),
"{modifier}": random.choice(MODIFIERS),
}
query = template
for key, value in replacements.items():
query = query.replace(key, value)
return query
def generate_expansion(client: anthropic.Anthropic, query: str) -> str | None:
"""Generate expansion using Claude API."""
try:
response = client.messages.create(
model="claude-sonnet-4-20250514",
max_tokens=300,
system=SYSTEM_PROMPT,
messages=[
{"role": "user", "content": USER_PROMPT_TEMPLATE.format(query=query)}
]
)
return response.content[0].text.strip()
except Exception as e:
print(f"Error generating expansion for '{query}': {e}")
return None
def validate_output(output: str) -> bool:
"""Validate that output follows the expected format."""
lines = output.strip().split("\n")
if not lines:
return False
has_lex = False
has_vec = False
for line in lines:
line = line.strip()
if not line:
continue
if line.startswith("lex:"):
has_lex = True
elif line.startswith("vec:"):
has_vec = True
elif line.startswith("hyde:"):
pass
else:
return False # Invalid line type
return has_lex and has_vec
def main():
parser = argparse.ArgumentParser(description="Generate QMD query expansion training data")
parser.add_argument("--count", type=int, default=100, help="Number of examples to generate")
parser.add_argument("--output", type=str, default="data/qmd_expansion.jsonl", help="Output file path")
parser.add_argument("--queries", type=str, help="Optional file with custom queries (one per line)")
args = parser.parse_args()
api_key = os.environ.get("ANTHROPIC_API_KEY")
if not api_key:
print("Error: ANTHROPIC_API_KEY environment variable not set")
exit(1)
client = anthropic.Anthropic(api_key=api_key)
output_path = Path(args.output)
output_path.parent.mkdir(parents=True, exist_ok=True)
# Load custom queries if provided
custom_queries = []
if args.queries and Path(args.queries).exists():
custom_queries = Path(args.queries).read_text().strip().split("\n")
print(f"Loaded {len(custom_queries)} custom queries")
examples = []
seen_queries = set()
print(f"Generating {args.count} examples...")
i = 0
while len(examples) < args.count:
# Use custom query or generate random one
if custom_queries and i < len(custom_queries):
query = custom_queries[i].strip()
else:
query = generate_random_query()
i += 1
# Skip duplicates
if query in seen_queries:
continue
seen_queries.add(query)
# Generate expansion
output = generate_expansion(client, query)
if output and validate_output(output):
examples.append({"input": query, "output": output})
print(f"[{len(examples)}/{args.count}] {query[:50]}...")
else:
print(f" Skipped invalid output for: {query[:50]}...")
# Write output
with open(output_path, "w") as f:
for example in examples:
f.write(json.dumps(example) + "\n")
print(f"\nGenerated {len(examples)} examples to {output_path}")
if __name__ == "__main__":
main()

192
finetune/generate_data_offline.py Normal file
View File

@ -0,0 +1,192 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "datasets",
# ]
# ///
"""
Generate QMD training data by transforming s-emanuilov/query-expansion dataset
and adding synthetic hyde passages. No API calls needed.
"""
import json
import random
from pathlib import Path
# HyDE passage templates for different query types
HYDE_TEMPLATES = {
"how_to": [
"To {action}, you need to {steps}. This can be done by {method}.",
"The recommended way to {action} is to first {step1}, then {step2}.",
"{Topic} can be achieved by {method}. Make sure to {consideration}.",
],
"what_is": [
"{Topic} is a {category} that {description}. It is commonly used for {use_case}.",
"{Topic} refers to {definition}. Key features include {features}.",
],
"config": [
"To configure {topic}, set the {setting} option to {value}. You can also customize {other}.",
"Configuration for {topic} is done in the {file} file. Key settings include {settings}.",
],
"error": [
"The {error} error occurs when {cause}. To fix this, {solution}.",
"If you encounter {error}, check that {check}. Common solutions include {solutions}.",
],
"general": [
"{Topic} provides {benefit} for {use_case}. It works by {mechanism}.",
"When working with {topic}, consider {considerations}. Best practices include {practices}.",
],
}
def classify_query(query: str) -> str:
"""Classify query type for hyde template selection."""
q = query.lower()
if any(w in q for w in ["how to", "how do", "setup", "install", "configure", "create"]):
return "how_to"
if any(w in q for w in ["what is", "what are", "definition", "meaning"]):
return "what_is"
if any(w in q for w in ["config", "setting", "option"]):
return "config"
if any(w in q for w in ["error", "issue", "problem", "fix", "debug"]):
return "error"
return "general"
def extract_topic(query: str) -> str:
"""Extract main topic from query."""
# Remove common prefixes
for prefix in ["how to ", "how do i ", "what is ", "what are ", "configure ", "setup "]:
if query.lower().startswith(prefix):
return query[len(prefix):].strip()
return query
def generate_hyde(query: str, expansions: list[str]) -> str:
"""Generate a hypothetical document passage by combining expansions naturally."""
topic = extract_topic(query)
query_type = classify_query(query)
# Use the longest, most descriptive expansion as the base
sorted_exp = sorted(expansions, key=len, reverse=True)
main_exp = sorted_exp[0] if sorted_exp else topic
# Build a natural passage based on query type
if query_type == "how_to":
templates = [
f"To {topic}, start by reviewing the requirements and dependencies. {main_exp.capitalize()} is the recommended approach. Make sure all prerequisites are met before proceeding.",
f"The process of {topic} involves several steps. First, {main_exp}. Follow the official documentation for detailed instructions.",
f"When you need to {topic}, the most effective method is to {main_exp}. This ensures compatibility and follows best practices.",
]
elif query_type == "what_is":
templates = [
f"{topic.capitalize()} refers to {main_exp}. It is widely used in various applications and provides significant benefits.",
f"The concept of {topic} encompasses {main_exp}. Understanding this is essential for effective implementation.",
f"{topic.capitalize()} is defined as {main_exp}. This plays a crucial role in modern development practices.",
]
elif query_type == "config":
templates = [
f"Configuration for {topic} requires setting the appropriate parameters. {main_exp.capitalize()} should be adjusted based on your specific requirements.",
f"To configure {topic}, modify the settings in your configuration file. Key options include those related to {main_exp}.",
f"The {topic} configuration can be customized by {main_exp}. Default values work for most use cases.",
]
elif query_type == "error":
templates = [
f"The {topic} issue typically occurs when dependencies are misconfigured. To resolve this, {main_exp}. Check your environment settings.",
f"If you encounter problems with {topic}, verify that {main_exp}. Common solutions include updating dependencies and checking permissions.",
f"Debugging {topic} requires understanding the root cause. Often, {main_exp} resolves the issue. Review logs for details.",
]
else:
templates = [
f"{topic.capitalize()} is an important concept that relates to {main_exp}. It provides functionality for various use cases in software development.",
f"Understanding {topic} is essential for modern development. Key aspects include {main_exp}. This knowledge helps in building robust applications.",
f"The topic of {topic} covers {main_exp}. Proper implementation follows established patterns and best practices.",
]
return random.choice(templates)
def transform_to_qmd_format(query: str, expansions: list[str]) -> str:
"""Transform s-emanuilov format to QMD lex/vec/hyde format."""
lines = []
# Generate lex lines (keyword-focused, shorter)
lex_candidates = []
for exp in expansions:
# Shorter versions for lex
words = exp.split()
if len(words) <= 4:
lex_candidates.append(exp)
else:
# Take key phrases
lex_candidates.append(" ".join(words[:3]))
# Add 1-2 lex lines
for lex in lex_candidates[:2]:
if lex.lower() != query.lower():
lines.append(f"lex: {lex}")
# Generate vec lines (semantic, complete phrases)
vec_candidates = [exp for exp in expansions if len(exp.split()) >= 3]
if not vec_candidates:
vec_candidates = expansions
# Add 1-2 vec lines
for vec in vec_candidates[:2]:
if vec.lower() != query.lower():
lines.append(f"vec: {vec}")
# Generate hyde line
hyde = generate_hyde(query, expansions)
lines.append(f"hyde: {hyde}")
return "\n".join(lines)
def main():
try:
from datasets import load_dataset
except ImportError:
print("Installing datasets...")
import subprocess
subprocess.run(["uv", "pip", "install", "datasets"], check=True)
from datasets import load_dataset
print("Loading s-emanuilov/query-expansion dataset...")
dataset = load_dataset("s-emanuilov/query-expansion", split="train")
print(f"Loaded {len(dataset)} examples")
# Transform each example
output_path = Path("data/qmd_expansion.jsonl")
output_path.parent.mkdir(parents=True, exist_ok=True)
examples = []
for item in dataset:
query = item["query"]
expansions = item["expansions"]
output = transform_to_qmd_format(query, expansions)
examples.append({"input": query, "output": output})
# Shuffle
random.seed(42)
random.shuffle(examples)
# Write output
with open(output_path, "w") as f:
for ex in examples:
f.write(json.dumps(ex) + "\n")
print(f"Generated {len(examples)} examples to {output_path}")
# Show sample
print("\nSample output:")
print("-" * 50)
sample = examples[0]
print(f"Input: {sample['input']}")
print(f"Output:\n{sample['output']}")
if __name__ == "__main__":
main()

103
finetune/prepare_data.py Normal file
View File

@ -0,0 +1,103 @@
#!/usr/bin/env python3
"""Prepare QMD query expansion data for training."""
import argparse
import json
from pathlib import Path
# Prompt template matching QMD's llm.ts format (simplified for training)
PROMPT_TEMPLATE = """You are a search query optimization expert. Transform the query into retrieval-optimized outputs.
Query: {query}
Output format:
lex: {{keyword variation}}
vec: {{semantic reformulation}}
hyde: {{hypothetical document passage}}
Output:"""
def format_for_training(input_text: str, output_text: str) -> dict:
"""Format a single example for SFT training."""
prompt = PROMPT_TEMPLATE.format(query=input_text)
return {
"prompt": prompt,
"completion": output_text,
# Alternative format for some trainers
"text": f"{prompt}\n{output_text}",
# Chat format
"messages": [
{"role": "user", "content": f"Expand this search query:\n\n{input_text}"},
{"role": "assistant", "content": output_text}
]
}
def main():
parser = argparse.ArgumentParser(description="Prepare data for training")
parser.add_argument("--input", type=str, default="data/qmd_expansion.jsonl", help="Input JSONL file")
parser.add_argument("--output", type=str, default="data/train", help="Output directory")
parser.add_argument("--split", type=float, default=0.1, help="Validation split ratio")
args = parser.parse_args()
input_path = Path(args.input)
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
if not input_path.exists():
print(f"Error: Input file not found: {input_path}")
exit(1)
# Load examples
examples = []
with open(input_path) as f:
for line in f:
if line.strip():
examples.append(json.loads(line))
print(f"Loaded {len(examples)} examples from {input_path}")
# Format for training
formatted = [format_for_training(ex["input"], ex["output"]) for ex in examples]
# Split into train/val
split_idx = int(len(formatted) * (1 - args.split))
train_data = formatted[:split_idx]
val_data = formatted[split_idx:]
# Write train set
train_path = output_dir / "train.jsonl"
with open(train_path, "w") as f:
for item in train_data:
f.write(json.dumps(item) + "\n")
# Write validation set
val_path = output_dir / "val.jsonl"
with open(val_path, "w") as f:
for item in val_data:
f.write(json.dumps(item) + "\n")
# Write chat format (for TRL/Unsloth)
chat_path = output_dir / "train_chat.jsonl"
with open(chat_path, "w") as f:
for item in train_data:
f.write(json.dumps({"messages": item["messages"]}) + "\n")
print(f"Written {len(train_data)} train examples to {train_path}")
print(f"Written {len(val_data)} validation examples to {val_path}")
print(f"Written chat format to {chat_path}")
# Also save as HuggingFace datasets format info
dataset_info = {
"dataset_name": "qmd-query-expansion",
"train_samples": len(train_data),
"val_samples": len(val_data),
"columns": ["prompt", "completion", "text", "messages"],
}
with open(output_dir / "dataset_info.json", "w") as f:
json.dump(dataset_info, f, indent=2)
if __name__ == "__main__":
main()

92
finetune/train_0.6B.py Normal file
View File

@ -0,0 +1,92 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "trl>=0.12.0",
# "peft>=0.7.0",
# "transformers>=4.45.0",
# "accelerate>=0.24.0",
# "trackio",
# "datasets",
# "bitsandbytes",
# ]
# ///
import trackio
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
# Load dataset from Hub
print("Loading dataset...")
dataset = load_dataset("tobil/qmd-query-expansion-train", split="train")
print(f"Loaded {len(dataset)} examples")
# Create train/eval split
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = dataset_split["train"]
eval_dataset = dataset_split["test"]
print(f"Train: {len(train_dataset)}, Eval: {len(eval_dataset)}")
# Training configuration
config = SFTConfig(
output_dir="qmd-query-expansion-0.6B",
push_to_hub=True,
hub_model_id="tobil/qmd-query-expansion-0.6B",
hub_strategy="every_save",
# Training parameters
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
max_length=512,
# Logging & checkpointing
logging_steps=25,
save_strategy="steps",
save_steps=200,
save_total_limit=2,
# Evaluation
eval_strategy="steps",
eval_steps=200,
# Optimization
warmup_ratio=0.1,
lr_scheduler_type="cosine",
bf16=True,
# Monitoring
report_to="trackio",
project="qmd-query-expansion",
run_name="qwen3-0.6B-lora",
)
# LoRA configuration
peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
# Initialize trainer
print("Initializing trainer with Qwen/Qwen3-0.6B...")
trainer = SFTTrainer(
model="Qwen/Qwen3-0.6B",
train_dataset=train_dataset,
eval_dataset=eval_dataset,
args=config,
peft_config=peft_config,
)
print("Starting training...")
trainer.train()
print("Pushing to Hub...")
trainer.push_to_hub()
trackio.finish()
print("Done! Model at: https://huggingface.co/tobil/qmd-query-expansion-0.6B")

93
finetune/train_1.7B.py Normal file
View File

@ -0,0 +1,93 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "trl>=0.12.0",
# "peft>=0.7.0",
# "transformers>=4.45.0",
# "accelerate>=0.24.0",
# "trackio",
# "datasets",
# "bitsandbytes",
# ]
# ///
import trackio
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
# Load dataset from Hub
print("Loading dataset...")
dataset = load_dataset("tobil/qmd-query-expansion-train", split="train")
print(f"Loaded {len(dataset)} examples")
# Create train/eval split
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = dataset_split["train"]
eval_dataset = dataset_split["test"]
print(f"Train: {len(train_dataset)}, Eval: {len(eval_dataset)}")
# Training configuration
config = SFTConfig(
output_dir="qmd-query-expansion-1.7B",
push_to_hub=True,
hub_model_id="tobil/qmd-query-expansion-1.7B",
hub_strategy="every_save",
# Training parameters - slightly smaller batch for larger model
num_train_epochs=3,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=2e-4,
max_length=512,
# Logging & checkpointing
logging_steps=25,
save_strategy="steps",
save_steps=200,
save_total_limit=2,
# Evaluation
eval_strategy="steps",
eval_steps=200,
# Optimization
warmup_ratio=0.1,
lr_scheduler_type="cosine",
bf16=True,
gradient_checkpointing=True, # Save memory for larger model
# Monitoring
report_to="trackio",
project="qmd-query-expansion",
run_name="qwen3-1.7B-lora",
)
# LoRA configuration
peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
# Initialize trainer
print("Initializing trainer with Qwen/Qwen3-1.7B...")
trainer = SFTTrainer(
model="Qwen/Qwen3-1.7B",
train_dataset=train_dataset,
eval_dataset=eval_dataset,
args=config,
peft_config=peft_config,
)
print("Starting training...")
trainer.train()
print("Pushing to Hub...")
trainer.push_to_hub()
trackio.finish()
print("Done! Model at: https://huggingface.co/tobil/qmd-query-expansion-1.7B")

102
finetune/train_1.7B_v2.py Normal file
View File

@ -0,0 +1,102 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "trl>=0.12.0",
# "peft>=0.7.0",
# "transformers>=4.45.0",
# "accelerate>=0.24.0",
# "trackio",
# "datasets",
# "bitsandbytes",
# ]
# ///
"""
Improved Qwen3-1.7B training with best practices for larger models:
- Lower learning rate (1e-4 instead of 2e-4)
- Higher LoRA rank (32 instead of 16)
- More epochs (5 instead of 3)
- Weight decay for regularization
"""
import trackio
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
# Load dataset from Hub
print("Loading dataset...")
dataset = load_dataset("tobil/qmd-query-expansion-train", split="train")
print(f"Loaded {len(dataset)} examples")
# Create train/eval split
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = dataset_split["train"]
eval_dataset = dataset_split["test"]
print(f"Train: {len(train_dataset)}, Eval: {len(eval_dataset)}")
# Training configuration - optimized for larger model
config = SFTConfig(
output_dir="qmd-query-expansion-1.7B-v2",
push_to_hub=True,
hub_model_id="tobil/qmd-query-expansion-1.7B-v2",
hub_strategy="every_save",
# Training parameters - lower LR, more epochs for larger model
num_train_epochs=5,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=1e-4, # Lowered from 2e-4
weight_decay=0.01, # Added regularization
max_length=512,
# Logging & checkpointing
logging_steps=25,
save_strategy="steps",
save_steps=200,
save_total_limit=3,
# Evaluation
eval_strategy="steps",
eval_steps=200,
# Optimization
warmup_ratio=0.1,
lr_scheduler_type="cosine",
bf16=True,
gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},
# Monitoring
report_to="trackio",
project="qmd-query-expansion",
run_name="qwen3-1.7B-lora-v2",
)
# LoRA configuration - higher rank for better learning
peft_config = LoraConfig(
r=32, # Increased from 16
lora_alpha=64, # Increased from 32 (2x rank)
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
# Initialize trainer
print("Initializing trainer with Qwen/Qwen3-1.7B...")
trainer = SFTTrainer(
model="Qwen/Qwen3-1.7B",
train_dataset=train_dataset,
eval_dataset=eval_dataset,
args=config,
peft_config=peft_config,
)
print("Starting training...")
trainer.train()
print("Pushing to Hub...")
trainer.push_to_hub()
trackio.finish()
print("Done! Model at: https://huggingface.co/tobil/qmd-query-expansion-1.7B-v2")

292
finetune/train_grpo.py Normal file
View File

@ -0,0 +1,292 @@
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "trl>=0.12.0",
# "peft>=0.7.0",
# "transformers>=4.45.0",
# "accelerate>=0.24.0",
# "trackio",
# "datasets",
# "bitsandbytes",
# "sentence-transformers",
# ]
# ///
"""
GRPO (Group Relative Policy Optimization) training for QMD query expansion.
Reward Type 2: Format + Diversity
- Rewards correct lex/vec/hyde format
- Penalizes repetition between lines
- Rewards semantic diversity of expansions
Usage:
uv run train_grpo.py --sft-model tobil/qmd-query-expansion-0.6B
"""
import re
import torch
import trackio
from datasets import load_dataset
from peft import LoraConfig, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import GRPOTrainer, GRPOConfig
from sentence_transformers import SentenceTransformer
# ============================================================================
# Reward Function: Format + Diversity
# ============================================================================
def parse_expansion(text: str) -> dict:
"""Parse expansion output into lex/vec/hyde components."""
result = {"lex": [], "vec": [], "hyde": []}
for line in text.strip().split("\n"):
line = line.strip()
if line.startswith("lex:"):
result["lex"].append(line[4:].strip())
elif line.startswith("vec:"):
result["vec"].append(line[4:].strip())
elif line.startswith("hyde:"):
result["hyde"].append(line[5:].strip())
return result
def compute_format_reward(text: str) -> float:
"""
Reward for correct format:
- Has at least 1 lex line: +0.2
- Has at least 1 vec line: +0.2
- Has hyde line: +0.1
- Correct line format (type: content): +0.1 per line (max 0.3)
- No garbage/malformed lines: +0.2
"""
reward = 0.0
parsed = parse_expansion(text)
# Check required components
if parsed["lex"]:
reward += 0.2
if parsed["vec"]:
reward += 0.2
if parsed["hyde"]:
reward += 0.1
# Check line format
lines = text.strip().split("\n")
valid_lines = 0
for line in lines:
if re.match(r'^(lex|vec|hyde):\s*.+', line.strip()):
valid_lines += 1
reward += min(0.3, valid_lines * 0.1)
# Penalize malformed lines
malformed = len(lines) - valid_lines
if malformed == 0:
reward += 0.2
else:
reward -= malformed * 0.1
return max(0.0, min(1.0, reward))
def compute_diversity_reward(text: str, embedder) -> float:
"""
Reward for diverse expansions:
- Penalize exact duplicates
- Reward semantic distance between expansions
"""
parsed = parse_expansion(text)
all_expansions = parsed["lex"] + parsed["vec"] + parsed["hyde"]
if len(all_expansions) < 2:
return 0.0
# Penalize exact duplicates
unique = set(e.lower() for e in all_expansions)
duplicate_penalty = (len(all_expansions) - len(unique)) * 0.2
# Compute semantic diversity
if len(unique) >= 2:
try:
embeddings = embedder.encode(list(unique))
# Compute pairwise cosine similarities
from torch.nn.functional import cosine_similarity
emb_tensor = torch.tensor(embeddings)
similarities = []
for i in range(len(emb_tensor)):
for j in range(i + 1, len(emb_tensor)):
sim = cosine_similarity(
emb_tensor[i].unsqueeze(0),
emb_tensor[j].unsqueeze(0)
).item()
similarities.append(sim)
# Lower similarity = higher diversity = higher reward
avg_similarity = sum(similarities) / len(similarities) if similarities else 1.0
diversity_reward = 1.0 - avg_similarity # 0 = identical, 1 = orthogonal
except Exception:
diversity_reward = 0.0
else:
diversity_reward = 0.0
return max(0.0, diversity_reward - duplicate_penalty)
def compute_length_reward(text: str) -> float:
"""Reward appropriate length (not too short, not too long)."""
lines = [l for l in text.strip().split("\n") if l.strip()]
# Ideal: 3-6 lines
if 3 <= len(lines) <= 6:
return 0.2
elif 2 <= len(lines) <= 7:
return 0.1
else:
return 0.0
class QMDRewardFunction:
"""Combined reward function for QMD query expansion."""
def __init__(self):
# Load a small embedding model for diversity computation
print("Loading embedding model for diversity reward...")
self.embedder = SentenceTransformer('all-MiniLM-L6-v2')
print("Embedding model loaded.")
def __call__(self, completions: list[str], prompts: list[str] = None) -> list[float]:
"""Compute rewards for a batch of completions."""
rewards = []
for completion in completions:
# Extract just the generated part (after prompt)
text = completion
# Compute component rewards
format_r = compute_format_reward(text)
diversity_r = compute_diversity_reward(text, self.embedder)
length_r = compute_length_reward(text)
# Weighted combination
total = (
0.5 * format_r + # Format is most important
0.35 * diversity_r + # Diversity is second
0.15 * length_r # Length is minor
)
rewards.append(total)
return rewards
# ============================================================================
# Main Training
# ============================================================================
def main():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--sft-model", default="tobil/qmd-query-expansion-0.6B",
help="SFT model to use as starting point")
parser.add_argument("--base-model", default="Qwen/Qwen3-0.6B",
help="Base model (for loading tokenizer)")
parser.add_argument("--output", default="tobil/qmd-query-expansion-0.6B-grpo",
help="Output model name on Hub")
parser.add_argument("--epochs", type=int, default=1)
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
if args.dry_run:
print("GRPO Training Config:")
print(f" SFT Model: {args.sft_model}")
print(f" Base Model: {args.base_model}")
print(f" Output: {args.output}")
print(f" Epochs: {args.epochs}")
return
# Load dataset (just prompts needed for GRPO)
print("Loading dataset...")
dataset = load_dataset("tobil/qmd-query-expansion-train", split="train")
# Extract just the queries as prompts
def extract_prompt(example):
return {"prompt": example["messages"][0]["content"]}
dataset = dataset.map(extract_prompt, remove_columns=dataset.column_names)
dataset = dataset.shuffle(seed=42).select(range(min(2000, len(dataset)))) # Use subset for GRPO
print(f"Using {len(dataset)} prompts for GRPO")
# Load tokenizer
print(f"Loading tokenizer from {args.base_model}...")
tokenizer = AutoTokenizer.from_pretrained(args.base_model)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
# Load SFT model with LoRA adapter
print(f"Loading SFT model from {args.sft_model}...")
base_model = AutoModelForCausalLM.from_pretrained(
args.base_model,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(base_model, args.sft_model)
model = model.merge_and_unload() # Merge LoRA weights
print("Model loaded and LoRA merged.")
# Initialize reward function
reward_fn = QMDRewardFunction()
# GRPO config
config = GRPOConfig(
output_dir="qmd-expansion-grpo",
push_to_hub=True,
hub_model_id=args.output,
# GRPO specific
num_generations=4, # Generate 4 completions per prompt
max_new_tokens=256,
temperature=0.8,
# Training
num_train_epochs=args.epochs,
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
learning_rate=5e-6, # Lower LR for RL
# Logging
logging_steps=10,
save_strategy="epoch",
# Monitoring
report_to="trackio",
project="qmd-query-expansion-grpo",
run_name="grpo-format-diversity",
)
# Create trainer
print("Initializing GRPO trainer...")
trainer = GRPOTrainer(
model=model,
tokenizer=tokenizer,
config=config,
train_dataset=dataset,
reward_funcs=reward_fn,
)
# Train
print("Starting GRPO training...")
trainer.train()
# Save
print("Pushing to Hub...")
trainer.push_to_hub()
trackio.finish()
print(f"Done! Model at: https://huggingface.co/{args.output}")
if __name__ == "__main__":
main()

164
finetune/train_hf_job.py Normal file
View File

@ -0,0 +1,164 @@
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
# "unsloth",
# "transformers>=4.45.0",
# "datasets",
# "trl>=0.12.0",
# "torch",
# "huggingface_hub",
# ]
# ///
"""
Train QMD query expansion model using LoRA on HuggingFace Jobs.
This script is designed to run on HuggingFace Jobs infrastructure.
Uses Unsloth for efficient LoRA finetuning.
Usage:
# Local test
python train_hf_job.py --model Qwen/Qwen3-0.6B --data data/train --dry-run
# HuggingFace Jobs (via huggingface-skills)
# See hugging-face-model-trainer skill for deployment
"""
import argparse
import os
from pathlib import Path
def main():
parser = argparse.ArgumentParser(description="Train QMD query expansion model")
parser.add_argument("--model", type=str, default="Qwen/Qwen3-0.6B", help="Base model")
parser.add_argument("--data", type=str, default="data/train", help="Training data directory")
parser.add_argument("--output", type=str, default="models/qmd-expansion", help="Output directory")
parser.add_argument("--epochs", type=int, default=3, help="Number of epochs")
parser.add_argument("--batch-size", type=int, default=4, help="Batch size")
parser.add_argument("--lr", type=float, default=2e-4, help="Learning rate")
parser.add_argument("--lora-rank", type=int, default=16, help="LoRA rank")
parser.add_argument("--max-seq-length", type=int, default=512, help="Max sequence length")
parser.add_argument("--dry-run", action="store_true", help="Print config and exit")
parser.add_argument("--push-to-hub", type=str, help="Push to HuggingFace Hub repo")
args = parser.parse_args()
config = {
"model": args.model,
"data": args.data,
"output": args.output,
"epochs": args.epochs,
"batch_size": args.batch_size,
"learning_rate": args.lr,
"lora_rank": args.lora_rank,
"lora_alpha": args.lora_rank * 2,
"max_seq_length": args.max_seq_length,
}
if args.dry_run:
print("Training configuration:")
for k, v in config.items():
print(f" {k}: {v}")
return
# Import heavy dependencies only when needed
from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
import torch
print(f"Loading base model: {args.model}")
# Load model with Unsloth
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=args.model,
max_seq_length=args.max_seq_length,
dtype=None, # Auto-detect
load_in_4bit=True, # QLoRA
)
# Configure LoRA
model = FastLanguageModel.get_peft_model(
model,
r=args.lora_rank,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_alpha=args.lora_rank * 2,
lora_dropout=0,
bias="none",
use_gradient_checkpointing="unsloth",
random_state=42,
)
# Load dataset
data_path = Path(args.data)
if (data_path / "train_chat.jsonl").exists():
dataset = load_dataset("json", data_files=str(data_path / "train_chat.jsonl"))["train"]
print(f"Loaded {len(dataset)} training examples (chat format)")
else:
dataset = load_dataset("json", data_files=str(data_path / "train.jsonl"))["train"]
print(f"Loaded {len(dataset)} training examples")
# Format function for chat template
def format_chat(example):
messages = example.get("messages", [])
if messages:
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
else:
text = example.get("text", "")
return {"text": text}
dataset = dataset.map(format_chat)
# Training config
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
training_args = SFTConfig(
output_dir=str(output_dir),
num_train_epochs=args.epochs,
per_device_train_batch_size=args.batch_size,
gradient_accumulation_steps=4,
learning_rate=args.lr,
weight_decay=0.01,
warmup_ratio=0.03,
lr_scheduler_type="cosine",
logging_steps=10,
save_strategy="epoch",
bf16=torch.cuda.is_bf16_supported(),
fp16=not torch.cuda.is_bf16_supported(),
optim="adamw_8bit",
seed=42,
max_seq_length=args.max_seq_length,
dataset_text_field="text",
packing=False,
)
# Create trainer
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset,
args=training_args,
)
# Train
print("Starting training...")
trainer.train()
# Save
print(f"Saving model to {output_dir}")
model.save_pretrained(output_dir)
tokenizer.save_pretrained(output_dir)
# Push to hub if requested
if args.push_to_hub:
print(f"Pushing to HuggingFace Hub: {args.push_to_hub}")
model.push_to_hub(args.push_to_hub)
tokenizer.push_to_hub(args.push_to_hub)
print("Training complete!")
if __name__ == "__main__":
main()