Commit Graph

9 Commits

Author SHA1 Message Date
George Zhang
c8f72de12e
docs: fix query expansion model size (Qwen3-1.7B, not 0.6B)
The code uses Qwen3-1.7B (~2.2GB) for query expansion, but the README
documented Qwen3-0.6B (~640MB) in three places:
- Model requirements table
- Architecture diagram
- Code configuration sample

This caused confusion when users saw a 2GB+ download instead of 640MB.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-20 04:09:27 +08:00
Tobi Lutke
d383b5c226
Migrate to node-llama-cpp and add structured query expansion
- Replace Ollama HTTP API with node-llama-cpp for local GGUF models
- Add structured query expansion using JSON schema grammar:
  - Generates lexical query (for BM25), vector query, and HyDE
  - Tree-style CLI output showing query types
- Fix vector search: use cosine distance instead of L2
- Format queries with embeddinggemma nomic-style prompts
- Rename ollama_cache table to llm_cache
- Add disposeDefaultLlamaCpp() for clean process exit

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-20 18:03:41 -04:00
Tobi Lutke
a3703c069a
Add path normalization, output format tests, and fix test isolation
- Add support for collection/path.md format in get command (checks if
  first component is a known collection before treating as filesystem path)
- Add comprehensive output format tests verifying qmd:// URIs, docid,
  and context in JSON, CSV, MD, XML, files, and CLI formats
- Add path normalization tests for various input formats:
  qmd://, //, qmd:////, collection/path, and path:line suffix
- Add isolated test environments (createIsolatedTestEnv) to prevent
  YAML config conflicts between test suites
- Add test fixture files test1.md and test2.md for path tests
- Update runQmd helper to accept custom configDir parameter

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-20 14:45:18 -04:00
Tobi Lutke
529e989d83
Refactor: Move TypeScript source files to src/ directory
Move all .ts files to src/ to clean up the project root:
- Created src/ directory and moved all TypeScript source and test files
- Updated qmd shell wrapper to point to src/qmd.ts
- Updated package.json scripts to use src/ paths
- Updated documentation (CLAUDE.md, README.md) to reflect new structure
- All imports remain relative within src/, no changes needed
- Tests pass with same results (192 pass, 75 fail - existing issues)

This improves project organization and makes the root directory cleaner.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-12 17:12:09 -05:00
Tobi Lutke
bab46dacb2
Refactor: extract store, LLM, and formatter modules with comprehensive tests
- Extract store.ts: database operations, search, document retrieval
  - createStore() factory pattern for clean DB lifecycle management
  - Unified DocumentResult type with optional body loading
  - Snippet extraction with diff-style headers (@@ -line,count @@)

- Extract llm.ts: LLM abstraction layer with Ollama implementation
  - Clean interface for embed, generate, rerank operations
  - High-level rerankerLogprobsCheck with logprob-based scoring
  - Query expansion support

- Extract formatter.ts: output formatting utilities
  - Support for CLI, JSON, CSV, MD, XML formats
  - MCP-specific CSV formatting

- Extract mcp.ts: MCP server using createStore() pattern
  - Single DB connection for server lifetime (fixes closed DB errors)
  - URL-decode resource paths for proper space/special char handling

- Add comprehensive test suites (215 tests total)
  - store.test.ts: 96 tests covering all store operations
  - llm.test.ts: 60 tests for LLM abstraction
  - mcp.test.ts: 59 tests for MCP endpoints and resources
  - All tests use mocked Ollama (errors on unmocked calls)

- Add bun run inspector script for MCP debugging

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 16:33:32 -05:00
Tobi Lutke
25ac53848f
Add MCP server for AI agent integration
- Add `qmd mcp` command to start stdio-based MCP server
- Expose tools: qmd_search, qmd_vsearch, qmd_query, qmd_get, qmd_status
- Add index health warnings for unembedded docs and stale indexes
- Return CSV format with text/csv mime type for search results
- Add MCP documentation and configuration examples to README
- Add @modelcontextprotocol/sdk and zod dependencies

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-08 14:59:56 -05:00
Tobi Lutke
877917487d
Update README with quick start guide and agentic workflow examples
- Add new tagline: on-device search engine for everything you need to remember
- Add Quick Start section with walkthrough of indexing multiple directories
- Add "Using with AI Agents" section showing --json and --files workflows
- Update output format example to reflect new CLI format with Title/Context/Score
- Document --all flag for returning all matches
2025-12-08 12:54:57 -05:00
Tobi Lutke
42ab3f6c10
Update README to reflect current implementation
- Fix architecture diagram: show BM25+Vector for all query variations
- Add position-aware blending percentages to diagram
- Update CLI commands: add → index, add-context, cleanup, status
- Document chunked embeddings (~6KB pieces with hash/seq/pos)
- Update schema section with new tables (path_contexts, ollama_cache)
- Rewrite How It Works flows with accurate pipeline details
- Fix output format examples to show ~/... paths
- Add --files and --json output options

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2025-12-08 09:31:20 -05:00
Tobi Lutke
39193ea252
Initial commit: QMD - Quick Markdown Search
A CLI tool for searching markdown knowledge bases using hybrid retrieval:
- BM25 full-text search via SQLite FTS5
- Vector semantic search via sqlite-vec + Ollama embeddings
- LLM re-ranking with qwen3-reranker (logprobs-based scoring)
- Reciprocal Rank Fusion with weighted queries and position-aware blending

Features:
- `qmd add .` - Index markdown files in current directory
- `qmd embed` - Generate vector embeddings
- `qmd search` - BM25 full-text search
- `qmd vsearch` - Vector similarity search
- `qmd query` - Hybrid search with query expansion + reranking

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-07 19:16:16 -05:00