Reranker

.env.test

@@ -16,6 +16,10 @@ PRIMO_VID=01MIT_INST:MIT
 PRIMO_NDE_VID=01MIT_INST:NDE
 RACK_ATTACK_VERBOSE_LOGGING=false
 RESULTS_PER_PAGE=20
+ALL_TAB_RESULTS_PER_SOURCE=50
+ALL_TAB_SCORER=zscore
+ALL_TAB_BOOST_SOURCES=
+LOAD_MORE_MAX_RESULTS=200
 SYNDETICS_PRIMO_URL=https://syndetics.com/index.php?client=primo
 TACOS_HOST=FAKE_TACOS_HOST
 TACOS_URL=http://FAKE_TACOS_HOST/graphql

AGENTS.md

@@ -30,7 +30,7 @@ This file highlights the important, discoverable conventions and workflows an AI
 
   Essential ENV vars for core functionality: `TIMDEX_GRAPHQL`, `PRIMO_API_URL`, `PRIMO_API_KEY`, `RESULTS_PER_PAGE`, `TIMDEX_INDEX`, `TIMDEX_SOURCES`. Filter customization: `FILTER_*` (e.g., `FILTER_LANGUAGE`, `FILTER_CONTENT_TYPE`) and `ACTIVE_FILTERS` (comma-separated list controlling visibility/order of filters; note that filter aggregation keys in the schema use `*Filter` suffix, e.g., `languageFilter`, `contentTypeFilter`). Tests rely on `.env.test` values for VCR cassette generation and use `ClimateControl` gem to mock feature flags.
 
-- **Parallel fetching & multi-source pagination:** The `all` tab uses `MergedSearchService` (with `MergedSearchPaginator`) to fetch Primo and Timdex concurrently via `Thread.new`, then intelligently merges paginated results. Primo has a practical offset limit (~960 records); when this limit is reached, the UI shows a `show_continuation` flag to indicate search is exhausted. Merged totals are cached for 12 hours. Be careful when refactoring to preserve thread-safety, caching semantics, and offset limit handling.
+- **Parallel fetching, reranking & load more:** The `all` tab uses `MergedSearchService` to fetch Primo and TIMDEX concurrently via `Thread.new`, then passes both result sets to the `reranker` gem for scoring and ordering. The scorer is configurable via `ALL_TAB_SCORER` (default: `zscore`). `ALL_TAB_RESULTS_PER_SOURCE` (default: 50) controls how many candidates are fetched from each API when the cached all-tab pool needs to grow; `RESULTS_PER_PAGE` controls the initial visible batch size and each "Load more results" increment. The all tab preserves already-shown result order when reranking an expanded candidate pool, appending only unseen records. Source-specific tabs use the same load-more UI but still rely on their normal API page/offset behavior internally. `LOAD_MORE_MAX_RESULTS` caps visible load-more growth. Be careful when refactoring to preserve thread-safety, cache keys, stable-prefix behavior, and source exhaustion handling.
 
 - **JS stack & conventions:** Rails importmap is in use (`importmap-rails`). JavaScript entry is `app/javascript/application.js`. Stimulus controllers live in `app/javascript/controllers` and are imported by `importmap` via `config/importmap.rb`. Key controllers include `content_loader_controller.js` (dynamic content loading) and tab management via `source_tabs.js` (which handles geospatial UI state). Prefer small, focused changes to Stimulus controllers rather than heavy bundler-based rewrites.
 

Gemfile

@@ -20,6 +20,7 @@ gem 'rails', '~> 7.2.0'
 gem 'rails_cloudflare_turnstile'
 gem 'rails_semantic_logger'
 gem 'redis'
+gem 'reranker', git: 'https://github.com/MITLibraries/reranker.git', tag: 'v0.1.0'
 gem 'scout_apm'
 gem 'sentry-rails'
 gem 'sentry-ruby'

Gemfile.lock

@@ -1,3 +1,11 @@
+GIT
+  remote: https://github.com/MITLibraries/reranker.git
+  revision: 85b0d19c88c853c76f8de54786b4341ad2e43d4d
+  tag: v0.1.0
+  specs:
+    reranker (0.1.0)
+      enumerable-stats (>= 1.0, < 2.0)
+
 GIT
   remote: https://github.com/mitlibraries/mitlibraries-theme
   revision: ba5bdc9840ef817ba0b92e522b92da3f52669fc3
@@ -135,6 +143,7 @@ GEM
       dotenv (= 3.2.0)
       railties (>= 6.1)
     drb (2.2.3)
+    enumerable-stats (1.4.0)
     erb (5.1.3)
     erubi (1.13.1)
     faraday (2.14.1)
@@ -454,6 +463,7 @@ DEPENDENCIES
   rails_cloudflare_turnstile
   rails_semantic_logger
   redis
+  reranker!
   rubocop
   rubocop-rails
   scout_apm

README.md

@@ -144,7 +144,11 @@ may have unexpected consequences if applied to other TIMDEX UI apps.
 - `BLOCKED_USER_AGENTS` - comma-separated list of user agent strings to hard-block with 403 Forbidden responses (bypasses throttling; much cheaper). Default blocks `Sogou web spider` which was responsible for 76.94k spoofed attack requests from non-Chinese IPs. Example: `"Sogou web spider,BadBot/2.0"`
 - `REDIRECT_REQUESTS_PER_PERIOD`- number of requests that can be made that the query string starts with our legacy redirect parameter to throttle per `REQUEST_PERIOD`
 - `REDIRECT_REQUEST_PERIOD`- time in minutes used along with `REDIRECT_REQUESTS_PER_PERIOD`
-- `RESULTS_PER_PAGE`: The number of results to display per page. Use an even number to avoid peculiarities. Defaults to 20 if unset.
+- `ALL_TAB_RESULTS_PER_SOURCE`: Number of candidate results to fetch from each API (Primo and TIMDEX) when the combined all tab needs to grow its cached reranked pool. Defaults to 50 if unset. This is not the number of results shown per click; use `RESULTS_PER_PAGE` for that. Keep below 960 to avoid hitting Primo's offset limit.
+- `ALL_TAB_SCORER`: Scorer to use when merging results on the combined all tab. Valid values: `zscore` (default), `zipper`, `simple`, `random`. See the [reranker gem](https://github.com/MITLibraries/reranker) for details.
+- `ALL_TAB_BOOST_SOURCES`: Comma-separated list of source names to promote to the top of merged all-tab results after reranking (e.g., `Research Guides,Research Databases`). Empty by default (no promotion applied). Relative reranked order within the promoted group and within the remainder is preserved.
+- `LOAD_MORE_MAX_RESULTS`: Maximum number of results that can be shown through load-more requests for a single results tab. Defaults to 200 if unset. This guards against accidental oversized all-tab reranking requests.
+- `RESULTS_PER_PAGE`: The number of results shown in the initial batch and added by each "Load more results" click. For source-specific tabs this also remains the underlying API page size. Defaults to 20 if unset.
 - `ROBOTS_ENV`: Determines which version of `robots.txt` is used. This is read by the Robots controller. Any value other than `production` results in the non-production version being used.
 - `SCOUT_AUTO_INSTRUMENTS`: default is `false`. Recommended setting is `true` unless we add manual instrumentation in the future.
 - `SCOUT_DEV_TRACE`: only set in dev. Enables local, in-browser traces (does not send to ScoutAPM service. Other SCOUT ENV should not be set when using this)

app/assets/stylesheets/partials/_pagination.scss

@@ -54,4 +54,13 @@
 
     .center-elements {display: none;}
   }  
-}
+}
+
+.load-more {
+  clear: both;
+  display: flex;
+  justify-content: center;
+  padding-top: 1em;
+  border-top: 1px solid $color-border-default;
+  margin-bottom: 48px;
+}

app/controllers/search_controller.rb

@@ -25,7 +25,10 @@ def results
     # Load GeoData results if applicable
     if Feature.enabled?(:geodata)
       load_geodata_results
-      render 'results_geo'
+      respond_to do |format|
+        format.turbo_stream { render :results_geo }
+        format.html { render 'results_geo' }
+      end
       return
     end
 
@@ -42,6 +45,7 @@ def results
     # Render the response in HTML or JSON format
     respond_to do |format|
       format.json { render json: { results: @results, pagination: @pagination, errors: @errors } }
+      format.turbo_stream { render :results }
       format.html { render :results }
     end
   end
@@ -81,6 +85,8 @@ def load_geodata_results
     raw_results = extract_results(response)
     @results = NormalizeTimdexResults.new(raw_results, @enhanced_query[:q]).normalize
     @filters = extract_filters(response)
+    @append_results = @results
+    @load_more = load_more_from_pagination(@pagination)
   end
 
   def load_primo_results
@@ -89,35 +95,68 @@ def load_primo_results
     @pagination = data[:pagination]
     @errors = data[:errors]
     @show_primo_continuation = data[:show_continuation]
+    @append_results = @results
+    @load_more = load_more_from_pagination(@pagination)
   end
 
   def load_timdex_results
     data = fetch_timdex_data
     @results = data[:results]
     @pagination = data[:pagination]
     @errors = data[:errors]
+    @append_results = @results
+    @load_more = load_more_from_pagination(@pagination)
   end
 
   def load_all_results
-    current_page = @enhanced_query[:page] || 1
-    per_page = ENV.fetch('RESULTS_PER_PAGE', '20').to_i
-
-    # Inject wrapper fetchers instead of using the service defaults. We use lambdas here so the
-    # service can call back into this controller's instance methods while preserving request-scoped
-    # context (for example `@enhanced_query`) and the controller's caching/normalization behavior.
-    # Using lambdas keeps the service decoupled from controller internals.
+    # Inject wrapper fetchers so the service can call back into this controller's
+    # instance methods while preserving request-scoped context and caching behavior.
     merge_service = MergedSearchService.new(
       enhanced_query: @enhanced_query,
       active_tab: @active_tab,
       primo_fetcher: ->(offset:, per_page:, query: nil) { fetch_primo_data(offset: offset, per_page: per_page) },
       timdex_fetcher: ->(offset:, per_page:, query: nil) { fetch_timdex_data(offset: offset, per_page: per_page) }
     )
-    data = merge_service.fetch(page: current_page, per_page: per_page)
+    display_count = sanitized_load_count
+    stable_count = [display_count - load_more_batch_size, 0].max
+    data = merge_service.fetch(display_count: display_count, stable_count: stable_count)
 
     @results = data[:results]
+    @append_results = data[:append_results]
     @errors = data[:errors]
-    @pagination = data[:pagination]
-    @show_primo_continuation = data[:show_primo_continuation]
+    @load_more = data[:load_more]
+  end
+
+  # Builds the shared load-more view model for source-specific tabs that still
+  # use traditional page/offset mechanics internally. The UI presents one
+  # growing list, while the next request simply fetches the next existing page.
+  def load_more_from_pagination(pagination)
+    return nil if pagination.blank?
+
+    {
+      display_count: pagination[:end],
+      next_page: pagination[:next],
+      has_more: pagination[:next].present?,
+      total_hits: pagination[:hits]
+    }
+  end
+
+  # Returns the visible all-tab result count requested by the browser. This is
+  # intentionally separate from `page`: source-specific tabs keep using page for
+  # API offsets, while the all tab uses a display count to grow a stable reranked
+  # result list.
+  def sanitized_load_count
+    requested = params[:load_count].to_i
+    requested = load_more_batch_size if requested < 1
+    [requested, max_load_count].min
+  end
+
+  def load_more_batch_size
+    ENV.fetch('RESULTS_PER_PAGE', '20').to_i
+  end
+
+  def max_load_count
+    ENV.fetch('LOAD_MORE_MAX_RESULTS', '200').to_i
   end
 
   def fetch_primo_data(offset: nil, per_page: nil)
@@ -134,6 +173,10 @@ def fetch_primo_data(offset: nil, per_page: nil)
     primo_response = query_primo(per_page, offset)
     hits = primo_response.dig('info', 'total') || 0
     results = NormalizePrimoResults.new(primo_response, @enhanced_query[:q]).normalize
+
+    # Save noramlized results to file as YAML
+    # File.write("tmp/saved_searches/primo_results_#{Time.now.to_i}.yml", results.to_yaml)
+
     pagination = Analyzer.new(@enhanced_query, hits, :primo).pagination
 
     # Handle empty results from Primo API. Sometimes Primo will return no results at a given offset,
@@ -174,6 +217,10 @@ def fetch_timdex_data(offset: nil, per_page: nil)
       pagination = Analyzer.new(@enhanced_query, hits, :timdex).pagination
       raw_results = extract_results(response)
       results = NormalizeTimdexResults.new(raw_results, @enhanced_query[:q]).normalize
+
+      # Save noramlized results to file as YAML
+      # File.write("tmp/saved_searches/timdex_results_#{Time.now.to_i}.yml", results.to_yaml)
+
       { results: results, pagination: pagination, errors: nil, hits: hits }
     else
       { results: [], pagination: {}, errors: errors, hits: 0 }

app/javascript/loading_spinner.js

@@ -14,7 +14,7 @@ function swapTabs(new_target) {
   }
 }
 
-// Loading spinner behavior for pagination (Turbo Frame updates)
+// Loading spinner behavior for frame and stream updates
 document.addEventListener('turbo:frame-render', function(event) {
   if (window.pendingFocusAction === 'pagination') {
     // Focus on first result for pagination
@@ -51,8 +51,24 @@ document.addEventListener('turbo:frame-render', function(event) {
   };
 });
 
+document.addEventListener('turbo:before-stream-render', function(event) {
+  if (window.pendingFocusAction === 'load-more') {
+    document.getElementById('search-results')?.classList.remove('spinner');
+    window.pendingFocusAction = null;
+  }
+});
+
 document.addEventListener('click', function(event) {
   const clickedElement = event.target;
+  const loadMoreLink = clickedElement.closest('.load-more-link');
+
+  // Handle load-more clicks. Results append in place, so do not scroll users
+  // back to the top or move focus away from their current reading position.
+  if (loadMoreLink) {
+    document.getElementById('search-results').classList.add('spinner');
+    window.pendingFocusAction = 'load-more';
+    return;
+  }
 
   // Handle pagination clicks
   if (clickedElement.matches('.first a, .previous a, .next a')) {
@@ -100,4 +116,3 @@ document.addEventListener('turbo:load', function(event) {
   }
 });
 
-