Redemption reflow: reconcile-on-touch, mule-side Google ack, drop /add_pro_payment + refund_requested - #4
Open
jagerman wants to merge 17 commits into
Open
Conversation
These constants are the same names (but longer) as the endpoints themselves and are used only once. Having them as constants is neither useful nor saving, and makes it harder to find at a quick search the handler for an endpoint.
…moved 4d2218c inlined the FLASK_ROUTE_* constants out of server.py but left test.py still referencing server.FLASK_ROUTE_*, so four tests failed with AttributeError. Inline the route strings in test.py to match (black re-collapsed a few calls that no longer need wrapping). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
googleapiclient's default httplib2 transport has no socket timeout, so a hung Google request blocks indefinitely. In the mule that's the whole (single-threaded) Pub/Sub pull loop wedged -- and unlike the request workers, the mule has no harakiri leash. Build the androidpublisher client over an AuthorizedHttp with httplib2.Http(timeout=15) so a hung fetch or acknowledge fails fast and the mule retries; nothing the mule does is time-critical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An explicit, durable flag for "this Google purchase still owes an acknowledgement to Google." The mule sets it TRUE when registering a fresh, not-yet-acknowledged purchase, and a sweep in its pull loop acks such purchases and clears the flag. Keeping the obligation as a column (rather than inferring it from notification bookkeeping) lets it survive a mule crash between committing the payment and completing the ack. Existing rows default FALSE (already acked by the old redeem-time path). Partial index for the sweep's `WHERE needs_ack` scan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Google Play purchase-acknowledge (the "confirm within 3 days or it's auto-refunded" step) used to happen synchronously in the client redeem path. Move it into the mule, decoupled from notification handling so an ack is never sent for a payment the DB hasn't durably recorded and is reliably retried after a crash. Notification handling only records the obligation: add_unredeemed_payment gains a needs_ack flag, and the Google PURCHASED/RENEWED paths set it from Google's acknowledgement_state (TRUE only for a fresh, not-yet-acknowledged purchase). A new sweep, _sweep_pending_acks, is the SOLE acker: the pull loop runs it once per iteration, right before blocking on the next pull, so it uniformly covers fresh purchases, failed-ack retries, and crash/startup leftovers with no special startup path. It acks each token with an outstanding obligation and clears the flag. Because Google 400s an already-acknowledged purchase with no cleanly-identifiable error, on ANY ack failure it consults the authoritative acknowledgement_state (one fetch) and clears the flag iff Google already considers it acked (the acked-then-crashed case), else leaves it set to retry. Best-effort -- it never raises, so it can't break the pull loop. backend: google_payment_tokens_needing_ack + google_clear_needs_ack helpers. test: test_google_ack_sweep covers the clean-ack, already-acked-via-fetch, and genuine-failure paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e-only) The maintenance mule's only job is the Google Pub/Sub subscriber, so put it inside the Google provider package and name it accordingly. This makes the Google-specificity explicit and sets up a per-provider pattern: another provider needing a background consumer gets its OWN mule (uWSGI supports any number), so a stall in one provider's subscriber can't hold up another's. - git mv mule.py -> providers/google_play/mule.py; its imports become relative (from . import api, notifications). grpcio stays deferred inside notifications.thread_entry_point, so importing the module/package doesn't pull gRPC's fork-hostile background threads into the uWSGI master. - vassal ini: `mule = mule:run` -> `mule = providers.google_play.mule:run`. - Fix a stale comment (main.py and the ini) that claimed the periodic DB prune runs in the mule; it runs on worker 1 via an @Timer (a mule can't register uWSGI signals). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ter key The first piece of the redeem/proof reflow. reconcile_pending_payments(tx, master_pkey, redeemed_at) redeems every unredeemed, unrevoked Google/Apple payment bound to the master key -- Google's obfuscatedAccountId is the pubkey verbatim, Apple's appAccountToken is uuid_from_master_pk(pubkey), so holding the key IS the claim (no client-supplied token needed) -- links them to the user, and refreshes entitlement. Claim-all (a key can accumulate several unredeemed payments across renewals), and a no-match touches nothing: crucially it creates NO user row for a key with no payments, so a signed-but-payment-less status probe can't spawn users. This is the reconcile step every master-key-authenticated endpoint will run up front (generate_pro_proof, get_pro_status, get_payment_details), replacing the client-driven /add_pro_payment redeem. Additive for now -- nothing calls it yet; wiring + retiring the old path follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire reconcile_pending_payments into the proof path: a client's proof request now first claims any payment the mule has registered for its master key but that isn't yet redeemed, then builds the proof from the resulting entitlement. So the post-purchase flow collapses to "client calls generate_pro_proof" — no separate /add_pro_payment redeem — and if nothing has landed yet the existing "no Pro" answer stands and the client retries. A no-op when there's nothing new, so already-redeemed keys are unaffected. Additive: /add_pro_payment still exists (now redundant for Google/Apple); retiring it + the CLI Rangeproof grant + the old redeem_payment/unknown_payment path follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The auto-redeem inside add_unredeemed_payment (bind a renewal to the master key found from the prior payment) went through redeem_payment with a rebuilt payment_tx. A renewal carries the same store account-id as the original (obfuscatedAccountId / appAccountToken are stable across a subscription), so reconcile_pending_payments binds it by that account-id just like any other pending payment -- and claim-all harmlessly sweeps any other stragglers for the key too. Drops redeem_payment to a single remaining caller (the retiring add_pro_payment), and unifies every redeem path on reconcile. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… voucher The Rangeproof CLI generated a throwaway uuid "order id", created an unredeemed payment, then redeemed it through add_pro_payment -- pretending a voucher existed. There is no voucher system, so replace that two-step with grant_rangeproof(master_pkey, plan, expires_at, ...): it creates a payment already redeemed and linked to the key and returns the proof. order_id stays as an internal unique row id (a real client-claimable voucher is future work). cmd_voucher now just calls grant_rangeproof (and loads the signing key once, up front). This removes the CLI's dependency on add_pro_payment, leaving redeem_payment/add_pro_payment with only the retiring client route as callers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The renewal auto-redeem resolves the owner's FULL master key directly from the store's subscription-continuity linkage (Google payment_token / Apple original_tx_id -> a prior, securely-bound payment). Having the full key, there's no reason to route the actual bind back through the master-key-derived account-id (reconcile) -- especially the Apple appAccountToken, which is only the first 16 bytes of the key. Doing so needlessly made binding a renewal depend on it carrying the account-id, and worse, let a mule-side renewal claim-by-UUID: on a (vanishingly unlikely) 122-bit appAccountToken collision that could bind a stranger's payment. Replace the reconcile call in the auto-redeem with _redeem_payment_for_user, which redeems THIS payment by its own store identifier and links it to the resolved user -- never touching the account-id. The account-id match now stays confined to the client request path (generate_pro_proof / status / details), where the caller is the owner and "the new payer is the next to claim their own UUID" holds. Renewals whose original isn't found (never redeemed) still fall back to that client path, same as new payments. test: a payment binds to a user by its identifier even when its stored account-id is a different key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the redeem/proof reflow. The client no longer submits a payment to redeem it; every master-key-authenticated endpoint reconciles pending payments up front, so a payment the mule registered binds to its owner on the owner's next request, whichever endpoint that is. server: delete the /add_pro_payment route + handler; call reconcile_pending_payments at the top of get_pro_status and get_payment_details (generate_pro_proof already did). A status/details check right after purchase now reflects the just-arrived payment. backend: delete verify_and_add_pro_payment, add_pro_payment (with its synchronous Google fetch+ack block), redeem_payment (the token/tx-id matching + the whole unknown_payment / idempotent-replay machinery), and the now-dead RedeemPayment/RedeemPaymentStatus/make_add_pro_payment_message/ _add_pro_payment_user_tx_log_label_safe. reconcile_pending_payments (bind by account-id) + the renewal direct-bind + grant_rangeproof are the only redeem paths now. test: verify_and_add_pro_payment callers now redeem via a _redeem_and_prove helper (reconcile + generate_pro_proof); the binding-rejection test asserts account-id reconcile (attacker claims 0, owner claims 1); the notification/apple flows redeem via reconcile and assert the binding; the end-to-end flow test drops the explicit redeem (status auto-redeems) and proves via generate_pro_proof. Net -659 lines. (The set_payment_refund_requested nuke is still deferred, as agreed.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…re spec Track the redemption reflow (18fa6b9) and the refund-requested removal in the authoritative wire spec, which should have moved with the code: - §3: remove the add_pro_payment (ProAddPayment___) and set_payment_refund_requested (ProSetRefundReq_) signed requests; document that redemption is implicit (the store notifies the backend out-of-band and any master-signed request binds the account's unbound payments), and split the read endpoints into their own §3.2/§3.3. - payment_id: no longer a signed input; it survives only as an opaque output field on get_payment_details items. Drop the §3.5 client-construction grammar and the provider_code/payment_id signed-message examples in §1.1 (the cursor `before` is now the sole variable-length string field). - Remove every refund_requested_ts reference (§1 status note, the signed- timestamp list, get_pro_status and payment-item result shapes) and the now- unreachable unknown_payment error_code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "a refund has been requested (but not yet granted)" timestamp is removed across the whole stack; clients carry it in their own account config instead. This tracks the wire-spec removal in 8aedfd1. - server.py: delete the /set_payment_refund_requested route+handler and drop refund_requested_ts from get_pro_status and the payment-item shape. - backend.py: remove make_set_payment_refund_requested_message, set_refund_requested, the SET_PAYMENT_REFUND_REQUESTED_DOMAIN, the refund_requested_at columns from PAYMENTS/USERS reads, the struct fields (LookupUserExpiry.{,best_}refund_requested*, Payment/UserRow.refund_requested_at), the _lookup_user_expiry aggregation, the two users UPDATEs, the three payment INSERT defaults, and the refunds_initiated report column. - providers/app_store.py: drop the REFUND_DECLINED branch whose only action was clearing the flag; REFUND_DECLINED is already the catch-all no-op below it. - schema: drop refund_requested_at from the baseline (000) and add 002 to drop it (IF EXISTS) from databases that predate the edit. - test.py: remove the /set_payment_refund_requested section of the flow test. Also drops two symbols orphaned by the earlier /add_pro_payment retirement and now absent from the spec: the ADD_PRO_PAYMENT_DOMAIN prefix and the unknown_payment error slug (it had no remaining raise site). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…etirement With /add_pro_payment and /set_payment_refund_requested gone, nothing ingests a client-supplied payment_id anymore: payment_id is now purely an output field on get_payment_details items (wire spec §5.2). Drop the ingress machinery that no longer has a caller and whose comments still described a client-signed input: - backend.py: remove UserPaymentTransaction.payment_id, payment_id_from_user_tx, and apply_payment_id_to_tx. encode_payment_id / GOOGLE_PAYMENT_ID_DELIMITER stay — payment_id_from_payment_row still uses them for the egress encoding. - Repoint the payment_id doc cross-references from the removed §3.5 to §5.2 (and the ts-nonce note from the renumbered §3.4 to §1.1), matching 8aedfd1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tion reflow PR session-foundation#3's minting path was written against the pre-reflow backend API — it called backend.redeem_payment and built a UserPaymentTransaction with a payment_id field, both of which the reflow removed, and its redeem=False story pointed at the deleted /add_pro_payment route. Rewire it onto the reconcile-on-touch model: - backend.py: add redeem_minted_payment(tx, master_pkey, payment_tx, redeemed_at) — redeems exactly one just-minted payment by its own store identifier, creating the user + generation, for all three providers (incl. Rangeproof, which has no account-id for reconcile_pending_payments to match). It's the minting-path sibling of reconcile_pending_payments (client-facing, account-id) and _redeem_payment_for_user (renewal, existing user only). - minting.py: mint_payment's redeem step now calls redeem_minted_payment instead of the deleted redeem_payment; drop the removed UserPaymentTransaction.payment_id kwarg. redeem=False for Google/Apple is now reconcile-on-touch (the client's next authenticated request binds it), not a client /add_pro_payment call. - cli.py voucher: resolved the rebase conflict (PR session-foundation#3's minting-based command vs the reflow's grant_rangeproof rewrite) to mint_payment(redeem=True) + build_current_entitlement_proof; refresh the now-inaccurate provider_dry_run comments (the reflow moved Google's ack to the mule, so a redeem never reaches a store). - dev_routes.py / readme.md: correct the redeem=False description to reconcile-on-touch. Full suite green (29), mypy + black clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test_google_ack_sweep's flag() helper did db.query_one(...)[0], indexing an Optional row — mypy (*.py) flags "Value of type Any | None is not indexable". Pre-existing (not from this rebase); use db.query_scalar, the helper added for exactly this guaranteed-single-row case. `mypy *.py` now clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks how Session Pro payments are redeemed and trims the wire surface around it.
Previously the client POSTed
/add_pro_paymentto redeem a purchase, and that handlermade a synchronous, un-timed Google acknowledgement call inside the redeem DB
transaction — a blocking off-box call on the request path. This branch:
the backend out-of-band, and any master-key-signed request binds the account's unbound
payments before it answers.
/add_pro_paymentis gone.needs_acksweep, so no request ever blocks on a Google round-trip.refund_requestedend-to-end.Redemption: reconcile-on-touch
reconcile_pending_payments(master_pkey)claims every unredeemed Google/Apple paymentwhose store account-id matches the key (Google
obfuscatedAccountId= the pubkey; AppleappAccountToken=uuid_from_master_pk), links them to the user, and refreshesentitlement. It runs at the top of
generate_pro_proof,get_pro_status, andget_payment_details, so a payment the mule registered binds to its owner on the owner'snext request — whichever endpoint that is.
owner from the subscription-continuity linkage (payment_token / original_tx_id → a prior
securely-bound payment) and redeems by that, never touching the appAccountToken UUID — so
a (vanishingly unlikely) 122-bit UUID collision can't make a renewal bind a stranger's
payment. The UUID match stays confined to the client path, where the caller is the owner.
grant_rangeproof, a direct grant keyedon the master pubkey; the never-implemented fake-voucher UUID indirection is dropped.
/add_pro_payment,redeem_payment,verify_and_add_pro_payment, and the deadclient-submitted
payment_idingress path are removed.payment_idsurvives only as anopaque output field on
get_payment_details.Google ack → mule
needs_ackcolumn ongoogle_play_payment_details(migration001). A fresh,not-yet-acknowledged purchase sets it; the mule's pull loop sweeps and acks such purchases
then clears the flag — so the ack survives a mule crash between recording the payment and
completing it.
failed call is simply retried before long).
providers/google_play/mule.pyto make it explicitly Google-only. (If we need mules for other payment providers in the future, uwsgi is just fine managing multiple mules -- and one mule per provider means one blocked mule doesn't affect other providers).Drop
refund_requested/set_payment_refund_requestedandrefund_requested_tsare removed from the wire(request + all responses). Clients that want to surface a pending refund carry the
timestamp in their own account config.
payments.refund_requested_at/users.refund_requested_atdropped from the baseline andvia migration
002(DROP … IF EXISTS).unknown_paymenterror code is gone (its only source was/add_pro_payment).Reconcile #3 (docker/QA minting)
#3 landed the docker/QA harness against the pre-reflow API. Rewired onto the new model:
minting.mint_paymentredeems via a newredeem_minted_paymenthelper instead of theremoved
redeem_payment; the CLIvouchercommand mints thenbuild_current_entitlement_proof; and theredeem=falsepath is reconcile-on-touch, not aclient
/add_pro_paymentcall. No behavioural change to the docker setup itself.Migrations
001_google_needs_ack.sql— addneeds_ack+ partial index.002_drop_refund_requested.sql— drop the refund_requested columns (IF EXISTS; no-op ona fresh DB).
Both are idempotent and ledger-tracked.
Client / wire impact (heads-up for libsession + clients)
/add_pro_payment,/set_payment_refund_requested.refund_requested_ts(everywhere it appeared).unknown_payment— a not-yet-propagated purchase is now just anormal "not active yet" answer, retried.
generate_pro_proof(orget_pro_status) andretry until it reflects the entitlement; there is no submit step.
docs/pro-wire-protocol.md.