Smoss/scrape ma district pages#2145
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Smoss
requested review from
Mephistic,
alexjball,
kiminkim724,
mertbagt,
mvictor55,
nesanders,
sashamaryl and
timblais
as code owners
Mephistic
left a comment
Mephistic
left a comment
There was a problem hiding this comment.
LGTM overall - just need to sync on which Legislator Profile Page to use as the base and we can merge this. Thanks for taking this on!
| memberCode: z.string() | ||
| }) | ||
|
|
||
| export default createPage<{ |
There was a problem hiding this comment.
I think we already have the skeleton for the legislator profile page here: https://github.com/codeforboston/maple/blob/main/pages/legislators/profile.tsx
It was added in #2113
There was a problem hiding this comment.
We discussed this at hack night - we've decided to go with this PR's approach of using the Member code as the base ID for the legislator profile page. This fits better with our V1, where the priority is supporting the page for Members who do not have profiles - when the majority of members will not have profiles at launch.
Longer term, we may move to a system that prioritizes the Profile ID (e.g. if we want to support non-Legislator Legislator Profiles such as for Candidates), but we've decided that is more effort than the alternative for the first pass (especially since that will require additional pipeline work). It should be simple enough to move from one approach to the other later if the need arises.
There was a problem hiding this comment.
That said, we do want to integrate the feature flag here before merging - @mertbagt is aware of the two pages and is going to follow up with the work to integrate them
- Added functions to read district snapshots from files and strip ignored HTML tags. - Updated `parseSecDistricts` to utilize the new function for cleaning HTML input. - Introduced tests for parsing SEC snapshots in `parseSecDistricts.test.ts` to ensure correct district counts for Senate and House. - Modified the scraping script to read HTML from local files if not fetching from the web.
Smoss
force-pushed
the
smoss/scrape-ma-district-pages
branch
from
43d6620 to
f0aacd5
Compare
- Implemented a feature flag to conditionally render the legislator profile page. - Added a redirect to a 404 page if the feature flag for legislators is disabled.
Mephistic
left a comment
Mephistic
left a comment
There was a problem hiding this comment.
LGTM - thanks for jumping on the flag issue!
|
Just waiting on the unit tests to pass to merge! |
- Removed unused imports and refactored test functions in `parseSecDistricts.test.ts` to build HTML snapshots for Senate and House districts. - Updated the project configuration in `project.yml` to correct the formatting of the languages list.
2 participants
Summary
Scrape the https://www.sec.state.ma.us/divisions/elections/voting-information/district/2022-senatorial.htm and https://www.sec.state.ma.us/divisions/elections/voting-information/district/2022-representative.htm pages to get the districts for members.
Checklist
firestore.indexes.json(Please do not only create indexes through the Firebase Web UI, even though the error messages may reccommend it - indexes created this way may be obliterated by subsequent deploys)Screenshots
Add some screenshots highlighting your changes.
Known issues
If you've run against limitations or caveats, include them here. Include follow-up issues as well.
Steps to test/reproduce
For each feature or bug fix, create a step by step list for how a reviewer can test it out. E.g.: