Skip to content

fix(instagram): paginate following endpoint to handle limit > ~200 (#1831)#1835

Open
LeoLin990405 wants to merge 2 commits into
jackwener:mainfrom
LeoLin990405:fix/instagram-following-pagination
Open

fix(instagram): paginate following endpoint to handle limit > ~200 (#1831)#1835
LeoLin990405 wants to merge 2 commits into
jackwener:mainfrom
LeoLin990405:fix/instagram-following-pagination

Conversation

@LeoLin990405
Copy link
Copy Markdown
Contributor

Closes #1831.

What

opencli instagram following <user> --limit N issued a single fetch with the user-supplied limit passed straight through as count to /api/v1/friendships/{userId}/following/. Instagram returns HTTP 400 for count values above ~200, so any account followed by more than ~200 users could not be fully listed.

Fix

Replace the single fetch with a pagination loop:

  • Fixed page size count=50 (safely under the API ceiling)
  • Follow the next_max_id cursor
  • Dedupe by pk (defends against cursor overlap)
  • ~400 ms pacing between pages
  • Terminate on: limit reached, missing cursor, empty page, repeated cursor, or a page that yields zero new unique users (avoids any infinite-loop edge if the API echoes itself)

Verification

  • npx vitest run clis/instagram/instagram.test.js9/9 pass
  • npm run test:adapter380 files / 3723 tests pass, no regression
  • npm run build-manifest — idempotent (no CLI-surface change)

Tests added (clis/instagram/instagram.test.js, new file)

  1. Single page < 50 users (no pagination)
  2. 3 pages via next_max_id (rank continuity across 50+50+20)
  3. Dedupe by pk on cursor overlap
  4. limit respected — stops early, no extra fetch
  5. Empty following list returns []
  6. Repeated next_max_id → cursor-loop guard breaks (3 calls instead of 4)
  7. Zero-growth page (page 2 dedupes to empty) → break
  8. limit=0 short-circuits before the following endpoint
  9. HTTP error mid-pagination propagates as Failed to fetch following: HTTP 429

Behavior

--limit Before After
20 20 (1 fetch) 20 (1 fetch)
200 works (1 fetch) works (4 fetches)
300 HTTP 400 295 unique (issue: account follows 296)

CLI surface unchanged — this is a behind-the-scenes fix, no flag added, no README/manifest delta.

Closes jackwener#1831.

The `following` adapter was issuing a single fetch to
`/api/v1/friendships/{userId}/following/?count={limit}` and passing the
user-supplied `limit` straight through. Instagram's friendships endpoint
returns HTTP 400 for `count` values above ~200, so any account followed
by more than 200 users could not be fully listed.

Replace the single fetch with a pagination loop:
- Fixed page size `count=50` (safely under the API ceiling)
- Follow the `next_max_id` cursor
- Dedupe by `pk` to defend against cursor overlap
- ~400 ms pacing between pages
- Stop on `limit` reached, empty page, or missing cursor

Verified against an account following 296 — full list now returns 295
unique entries (vs. HTTP 400 before).

Tests
- New `clis/instagram/instagram.test.js`, 5 cases:
  - single page < 50
  - 3 pages via next_max_id (rank continuity)
  - dedupe by pk on cursor overlap
  - limit respected (stops early)
  - empty following returns []
- `npm run test:adapter` — 380 files / 3719 tests pass
…ro-growth pages

Address review feedback (coder review NEEDS FIX):
- Add `seenCursors` Set so a repeating `next_max_id` aborts the loop.
- Bail out when a full page yields zero new unique users (e.g. the API
  echoes the previous page after dedupe).
- Skip the final 400 ms sleep when `limit` is already reached.

Tests
- `limit=0` short-circuits before the following endpoint.
- Repeating `next_max_id` aborts after the second observation.
- Zero-growth page (page-2 dedupes to empty) aborts.
- HTTP error on a mid-pagination page propagates as before.
- `globalThis.fetch` saved/restored instead of deleted.

`npx vitest run clis/instagram/instagram.test.js` — 9/9 pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[autofix] instagram/following: API_ERROR (HTTP 400 above ~200 results — no pagination)

1 participant