Improve search: multi-term AND + relevance ranking (FTS spike)#95
Open
rdhyee wants to merge 1 commit intoisamplesorg:mainfrom
Open
Improve search: multi-term AND + relevance ranking (FTS spike)#95rdhyee wants to merge 1 commit intoisamplesorg:mainfrom
rdhyee wants to merge 1 commit intoisamplesorg:mainfrom
Conversation
Search improvements (immediate): - Multi-term search: "pottery Cyprus" requires BOTH words to match - Relevance ranking: label matches weighted 3x, place 2x, description 1x - Results sorted by relevance score when searching (random for browsing) FTS spike (future path, documented): - Added tools/build_fts_index.py to build DuckDB FTS index offline - Tested: 358 MB full index, 211 MB lite — too large for auto-download - BM25 scoring works correctly (Porter stemming, stopwords) - Next step: explore smaller index strategies or on-demand loading Closes isamplesorg#84 (spike complete — findings documented in PR) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #84 — FTS spike complete with immediate search improvements and documented future path.
Shipped now (zero new dependencies):
FTS spike findings:
tools/build_fts_index.pyATTACHover HTTP in DuckDB-WASM is supported but downloading 200-358 MB is impracticalRecommended next steps (not in this PR):
Test plan
tools/build_fts_index.pyruns successfully with local parquet🤖 Generated with Claude Code