feat(shortener): Safe Browsing scanning + abuse reporting#72
feat(shortener): Safe Browsing scanning + abuse reporting#72PenguinzTech wants to merge 19 commits into
Conversation
…startup Fixes two critical findings: - C1: JWT_SECRET_KEY no longer falls back to the dev SECRET_KEY default in production. ProductionConfig now requires it via env var and validate() rejects the dev default, closing an admin-token forgery vector. - C2: create_app() now calls ProductionConfig.validate() at startup (guarded by hasattr so Dev/Testing configs are unaffected), so the dead secret-guard actually runs and the app boot-fails on default/missing secrets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tokens off URL Fixes account-takeover and token-leak findings in the OIDC/SSO consumer: - OC1: callback now validates the id_token (JWKS signature, iss/aud/exp/nonce) instead of blindly trusting the userinfo response; identity derives from validated claims. Adds PKCE (S256) + nonce to authorize/callback. - OC2: auto-link/auto-provision gated on email_verified (+ OIDC_AUTO_LINK_ VERIFIED_EMAIL flag, default off); no more silent email-based account linking. - OH4: access/refresh tokens returned as httpOnly Secure SameSite cookies, not in the redirect URL query string. - OH5: identity linking only via authenticated OIDC round-trip; removed body-supplied external_sub/external_email linking. Adds nullable columns oidc_providers.issuer, user_oidc_links.email_verified. _encrypt_secret fail-open and update_provider gating deferred to A5. Full backend suite: 403 passed, 1 skipped, 90.16% coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…H3/H1) Closes IDOR: a global viewer holds global tenants:read/teams:read, so the scope gate passed for everyone and routes then loaded any object by id with no membership check — any user could read any org's tenants/teams + member emails across tenants. - Adds membership helpers in rbac.py; each object-scoped tenant/team route now requires the caller to be a global admin OR a member of that specific tenant/team (admin ops require tenant/team admin scope, resolved with the id so tenant/team-scoped roles are actually consulted). - create_team validates the body tenant_id against caller membership (no more cross-tenant writes). - Regression tests: non-member viewer 403, member 200, global admin 200, foreign-tenant create 403. Also gitignore /worktrees/ (repo-local worktree convention). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…follow-up) The initial isolation fix guarded only read routes; write/admin routes still had just a global scope gate, so a user with global teams:write/tenants:write (e.g. a global maintainer) could modify/delete any team/tenant and grant roles across orgs — privilege escalation. (Also restores object-scoped authz that had regressed when team_id_param was dropped from the decorators.) - Adds _is_team_admin / _is_tenant_admin helpers (global admin scope OR team/tenant- scoped admin role for THAT object). - update/delete/add-member/remove-member on both teams.py and tenants.py now require the caller to be an admin of that specific team/tenant (or global admin). - add_team_member/add_tenant_member especially: they grant roles incl. *_admin, so non-admins of the object are now blocked. - Regression tests: global maintainer (no admin role) → 403 on every write route; team/tenant admin → 200/201. Full suite: 423 passed, 90.10% coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uth introspect/revoke (C3/H3/M4/OM1) - C3: access tokens now carry a jti; a revoked_access_tokens blocklist is checked in auth_required, so /logout and /oauth/revoke actually invalidate the current access token (previously only refresh tokens were revoked). Blocklist self-prunes on expiry. - H3/M4: real rate limiter (app/rate_limiter.py) — Redis-backed when REDIS_ENABLED else in-memory, respects RATE_LIMIT_ENABLED. Applied to /auth/login, /auth/register, /oauth/token; returns 429 on limit. - OM1: /oauth/introspect and /oauth/revoke now require authentication; introspect returns active:false for revoked tokens or deactivated (is_active=false) users. Tests: revocation, rate-limit 429, oauth-security + comprehensive rate_limiter tests. Suite: 441 passed, 1 skipped, 90.82% coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…time, logged errors, safe admin seed (OM4/M1/M2/M3/H2) - OM4: refresh grant (both /refresh and /oauth/token) now resolves tenant_id + scopes like the password grant, so refreshed access tokens keep their tenant/scope claims. - M1: define rbac + oidc tables (and the former inline create_user table) once at startup in init_db, eliminating the request-time check-then-define race on the shared DAL across executor threads. - M2: datetime.utcnow()/utcfromtimestamp() -> timezone-aware datetime.now(timezone.utc) in token exp/iat and expiry comparisons. - M3: replace silent except: pass around tenant/scope resolution and revoke storage with logged warnings (no PII/secrets), so degraded-permission failures are visible. - H2: in production, refuse to seed the default admin with the built-in weak password; require DEFAULT_ADMIN_PASSWORD to be set or skip seeding. Dev keeps the convenience default with a prominent warning. Suite: 467 passed, 90.44% coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…des, SSRF-safe URL validation (B1-B2) Foundation of the open-source URL shortener: - Schema (models.py + DDL for postgres/mysql/sqlite, defined at startup): urls (short_code UNIQUE, tenant/team/collection FKs, expiry, click_count), collections (folders via parent_id, unique per tenant+parent), link_clicks (ip_hash only, for the redirect task). - app/urls.py + app/collections.py blueprints under /api/v1, all @auth_required + @require_scope + tenant-scoped (tenant derived from token, never body; no cross-tenant access; creator-or-tenant-admin for edits). - Short-code generation via secrets (not random); DB UNIQUE + bounded retry. - app/urlvalidation.py: scheme allowlist (blocks javascript:/data:/file:), rejects literal + resolved private/loopback/link-local/reserved/multicast/metadata IPs (IPv4 incl. decimal/hex/octal, IPv6); unresolvable public hostnames allowed (the server 301-redirects, never fetches). Reserved-path list for aliases. - rbac.py: collections:read/write/delete scopes added to role bundles. - Redirect, click-tracking, analytics, QR, and tier-gating are follow-up tasks (TODO hooks left). Tests: 595 passed, 90.92% coverage — incl. SSRF bypass payloads rejected, alias collision, tenant isolation, collection nesting/cycle prevention. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ics (B3-B4) - app/redirect.py: public GET /<short_code> → 302 to long_url (404 for unknown/ inactive/expired), destination re-validated before redirect. Click tracking is non-blocking (asyncio.create_task after the response); click_count incremented atomically. link_clicks stores ip_hash (salted SHA256, never raw IP) + UA-parsed device/browser/os + referer. - QR: GET /api/v1/urls/<id>/qr (auth + urls:read + tenant-scoped) returns on-demand PNG of the short URL (qrcode+Pillow), not stored. TODO(tiering) branded/color QR. - app/analytics.py: /analytics/urls/<id> (per-link: total, clicks-by-day, top referers) and /analytics/summary (tenant-wide totals, top links, time series), analytics:read + tenant-scoped. Basic tiers implemented; geo/device/browser breakdowns marked TODO(tiering) for the Enterprise advanced-analytics gate. - deps: qrcode, Pillow, user-agents (pinned+hashed via uv). Tests: 619 passed, 90.66% coverage — redirect/404/expiry, async click + ip_hash, QR png, analytics date-filter/pagination/tenant-isolation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…license (B5) Two-layer gating (PostHog flag defaulted OFF + license tier), per house tier matrix: - app/features.py: feature_enabled() (graceful degradation, cached last-known, new flags OFF, never crash), get_license_tier() (free/professional/enterprise; dev & bypass-domain → enterprise; production fails CLOSED to free on error), require_tier/require_professional/require_enterprise decorators → 402 when below. - Free 5-collection cap enforced server-side in collections.py (402 on 6th; Pro/Ent unlimited) — flag current.collections. - Branded QR (color/logo params) gated to Professional in urls.py QR endpoint (plain QR stays Free) — flag current.branded-qr. - Advanced analytics (geo/device/browser/OS breakdowns) at /analytics/urls/<id>/advanced gated to Enterprise; basic analytics stays all-tiers — flag current.advanced-analytics. - config: POSTHOG_KEY/POSTHOG_HOST (default https://license.penguintech.io). Tests: 649 passed, 90% coverage; features.py 100% (tier resolution, flag degradation, each gate 402/allow, mocked posthog+license clients, no network). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…verage >90%) Adds comprehensive test coverage for: - Advanced analytics endpoint with Enterprise tier gating and feature flags - Date filtering (from/to) in advanced analytics - Advanced breakdown fields (geo, devices, browsers, os) - Redirect error paths: invalid destination URL validation, user-agent parsing - Click recording error handling and database exceptions - Edge cases: no clicks, no user-agent header, database errors with rollback Coverage improvements: - analytics.py: 72% → 92% (advanced endpoint now fully covered) - redirect.py: 23% → 90% (error paths and edge cases now covered) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Establish Alembic as the sole schema authority for DDL operations. Created SQLAlchemy ORM models in app/db_schema.py mirroring all 18 PyDAL tables. Generated baseline migration creating auth, RBAC, session, OIDC, and URL shortener tables. All migrations support PostgreSQL, MySQL, and SQLite. Verified upgrade/downgrade on sqlite with schema validation tests. Added comprehensive Alembic docs. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Implement malware/phishing protection with Google Safe Browsing Lookup API v4: - Link safety check on URL creation (feature-flagged: current.link-safety-scanning) - Safe_status + safe_checked_at fields track check results (unchecked/safe/flagged) - Graceful degradation: missing API key, timeout, or API errors allow URL creation - Redirect refuses flagged URLs with 403 Forbidden - Public abuse reporting endpoint records reports by short_code + tenant - Threshold-based auto-flag: 5+ abuse reports auto-flag and deactivate URL - New abuse_reports table with url_id, short_code, reason, reporter_ip_hash - Full Alembic migration (rev b1083c5f314a) with up/downgrade tested - All tenant-scoped, no license gating (abuse prevention applies to all tiers) - 13 unit tests + integration patterns, 82% coverage on safety module Migration: baseline 7e472cd8a157 → b1083c5f314a (adds urls.safe_status/safe_checked_at, abuse_reports table) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @PenguinzTech, your pull request is larger than the review limit of 150000 diff characters
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
… + coverage Fixes business-logic DoS vulnerability in abuse-report flooding: - Add unique constraint on (url_id, reporter_ip_hash) to deduplicate reports - Reject duplicate reports from same IP (409 Conflict) - Count only DISTINCT reporters toward abuse threshold - Move URL to pending_review state (soft flag) instead of permanent block - Keep is_active=True for pending_review URLs; show caution interstitial only - Add comprehensive tests for deduplication and threshold behavior - Omit db_schema.py and migrations from coverage (DDL, not runtime logic) - Safe_status column values: unchecked, safe, pending_review, flagged Tests added: - test_report_abuse_success - test_report_abuse_missing_short_code - test_report_abuse_missing_reason - test_report_abuse_reason_too_long - test_report_abuse_url_not_found - test_report_abuse_dedupe_same_ip - test_report_abuse_moves_to_pending_review_on_threshold - test_report_abuse_rate_limit_per_endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…0% gate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Malware/phishing protection for public URL shortener via Google Safe Browsing API v4. Screens destination URLs on creation and refuses redirects for flagged links. Includes public abuse reporting with auto-flag threshold.
current.link-safety-scanning(default OFF)unchecked|safe|flaggedChanges
New Files
app/safety.py(65 lines) — Safe Browsing API client with fail-open patterntests/test_link_safety.py(13 tests, 82% coverage)Modified Files
app/db_schema.py— Add Url.safe_status, Url.safe_checked_at, new AbuseReport modelapp/models.py— PyDAL table definitions for runtime (safe_status, safe_checked_at, abuse_reports table)app/urls.py— URL creation calls safety check, rejects if flagged; report-abuse endpointapp/redirect.py— Refuse redirect if safe_status == 'flagged' (403)app/schemas/urls.py— Add safe_status + safe_checked_at to URLResponseDatabase
b1083c5f314a(baseline 7e472cd8a157 → new revision)abuse_reports(url_id, short_code, tenant_id, reason, reporter_ip_hash, status, created_at)urls.safe_status(default 'unchecked'),urls.safe_checked_at(nullable datetime)Test Plan
Notes