Skip to content

feat(cpex-scraper): add file logger for better debugging and tracing#4376

Open
faizkhairi wants to merge 2 commits intonusmodifications:masterfrom
faizkhairi:feat/cpex-scraper-file-logger
Open

feat(cpex-scraper): add file logger for better debugging and tracing#4376
faizkhairi wants to merge 2 commits intonusmodifications:masterfrom
faizkhairi:feat/cpex-scraper-file-logger

Conversation

@faizkhairi
Copy link
Copy Markdown

Summary

  • Add a lightweight file logger (src/logger.ts) that writes timestamped lines to both stdout and a log file
  • Log files are written to scrapers/cpex-scraper/logs/ with format cpex-YYYY-MM-DD.HH-mm-ss.log
  • The logs directory is created automatically at runtime (already covered by root .gitignore)
  • Update src/index.ts to use the file logger instead of the default console
  • Zero new dependencies -- uses only Node.js built-in fs module
  • Satisfies the existing Pick<Console, 'log'> interface, so scraper.ts is unchanged

Related Issue

Closes #4371

Test Plan

  • Run pnpm --filter nusmods-cpex-scraper build and verify compilation succeeds
  • Run the scraper and verify log file is created in logs/ directory
  • Verify console output still shows timestamped log lines
  • Verify existing unit tests still pass (pnpm --filter nusmods-cpex-scraper test)

Add a lightweight file logger that writes timestamped lines to both
stdout and a log file in the logs/ directory. The log directory is
created automatically at runtime (already gitignored by root .gitignore).

This matches the logging pattern used by the nus-v2 scraper, making
it easier to debug and trace CPEx scraper runs.

- Add src/logger.ts with createFileLogger() utility
- Update src/index.ts to use the file logger instead of console
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps bot commented Mar 24, 2026

PR author is not in the allowed authors list.

@vercel
Copy link
Copy Markdown

vercel bot commented Mar 24, 2026

@faizkhairi is attempting to deploy a commit to the modsbot's projects Team on Vercel.

A member of the Team first needs to authorize it.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 24, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.39%. Comparing base (988c6fd) to head (b1610aa).
⚠️ Report is 220 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4376      +/-   ##
==========================================
+ Coverage   54.52%   56.39%   +1.87%     
==========================================
  Files         274      308      +34     
  Lines        6076     6963     +887     
  Branches     1455     1680     +225     
==========================================
+ Hits         3313     3927     +614     
- Misses       2763     3036     +273     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- Fix log directory path: '../logs' resolved to build/logs/ at runtime
  (inside gitignored build dir). Changed to '../../logs' to resolve to
  scrapers/cpex-scraper/logs/ (the scraper root)
- Make close() return Promise to ensure all buffered data is flushed
  to disk before process exits
- Await close() in both success and error paths in index.ts
- Add error handler on write stream to prevent unhandled crash
@faizkhairi
Copy link
Copy Markdown
Author

faizkhairi commented Mar 24, 2026

Note on testing: I was unable to run the full CI suite locally (pnpm run ci) as I used a clone. Happy to add unit tests for logger.ts if desired, the main cases would be:

  1. log() writes timestamped lines to both stdout and the file stream
  2. close() resolves after the stream is fully flushed
  3. Constructor creates the logs directory if it doesn't exist
  4. Stream error handler prevents unhandled crashes

The existing scraper.test.ts tests are unaffected since FileLogger is only instantiated in index.ts, not in the testable scraper.ts.

Let me know if you'd like me to add these tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add file logger for CPEx scraper

1 participant