Skip to content

"PFS" branch: Core architecture + DB persistent storage; PFS+PQ for DMs; refactoring and internals redesigns - #103

Draft
jagerman wants to merge 200 commits into
devfrom
pfs
Draft

"PFS" branch: Core architecture + DB persistent storage; PFS+PQ for DMs; refactoring and internals redesigns#103
jagerman wants to merge 200 commits into
devfrom
pfs

Conversation

@jagerman

Copy link
Copy Markdown
Member

This branch (pfs) is now actively maintained alongside dev and will eventually replace the dev branch.

This adds perfect forward secrecy support with post-quantum encryption; a "Core" component with a database layer to track state; direct-QUIC file server support; and a host of code cleanups.

For extensive details, see the description of PR #90.

jagerman added 30 commits July 7, 2026 17:10
session-deps is a new repo that consolidates how we handle loading and
doing static bundle builds of various common external dependencies
across Session projects.
Carrying a libsodium fork is too much of a nuissance as updating it to
the latest version is non-trivial.

This drops the libsodium-internal fork in favour of using tweenacl's
implementation *just* for the X25519 -> Ed25519 pubkey conversion, and
using a stock libsodium for everything else.

This also bumps the libsodium requirement up to 1.0.21: that version
will be required for SHAKE support in future commits on this branch.
- Blake2b hashing:
  - Add nicer hash::blake2b functions for simpler has computations where
    you can just pass a bunch of spannables and get the hash over them,
    rather than needing to do a bunch of manual C API update calls.
  - Add a ""_b2b_pers for a compile-time validated blake2b
    personalisation string
  - Drop make_blake2b32_hasher().
    - TODO: convert these to make full use of hash::blake2b(...), as
      above.
- Change cleared_array to take a Char type instead of forcing unsigned
  char.
- Add cleared_uchars (and cleared_bytes) for the old force-unsigned char
  typedef.
- Make `to_span` work for any input convertible to a span
- Tighten up various functions taking fixed length values (like session
  ids, pubkeys) to take compile-time-fixed spans instead of dynamic
  extent spans.  This allows these generic functions to not have to
  worry about length checking.
- Add a generic random::fill(s) function that fills some spannable type
  s with random bytes.
This switches to the PR branches for session-router and libquic to use
session-deps.  (This was required under ninja builds, in particular, to
get the deduplication handling for gmp and nettle via new session-deps
code to deal with that).
- Refactor manual libsodium blake2b hash calls to use simpler
  hash::blake2b functions instead.
- Unify usage of array_uc32/33/64 and uc32/33/64: now we have just
  uc32/33/64 and cleared_uc32/33/64 (i.e. the "array_" prefix is gone).
- Add a unsigned char array literal, which is quite useful for static
  hash keys.
- Including raw integer bytes would break the hashes on a non-little
  endian arches; this adds a helper than ensures we are always hashing
  the little-endian value.
- Make the remaining manual hash use blake2b_pers.  This didn't get
  autoconverted before because the `if` around part of the hash, but
  that if is actually completely unnecessary: if the value is empty, a
  blake2b hash update does nothing, and so it can just be always
  included (and when empty, it is still the right thing).
oxenc has buggy "constexpr" overloads that just break if invoked, and
they get invoked here with a uint8_t value + unsigned char array.  The
issue is fixed in oxenc dev, but switching to a byte here works around
it for current and older oxenc versions.
The recent commit to unify the types wasn't applied to the test suite.
hash::blake2b (and related) now take integers directly, writing the
integer bytes (with byte swapping applied, if necessary), which further
simplies the hashing API.
Make way for account keys to be in here as well.
If two devices rotate account keys at approximately the same time, both
might end up with inconsistent "active" keys.  This commit adds
deterministic tie breaking (always prefering the later, with fallback to
seed ordering).
Adds a "needs push" concept, along with more tracking fields to let us
distinguish between various possible states.
`session::AdjustedClock` now carries an adjustable static offset and
when `now()` is called it returns standard system clock timepoints with
the adjustment applied.

The networking code had a similar adjustment already, although it was
per-network-object: this change replaces that, and makes it now global
across all uses of the clock anywhere, instead of per-Network instance.
jagerman and others added 30 commits July 31, 2026 03:00
Centralise the "when should I renew my Pro proof?" decision that the
three clients each implemented inconsistently (Android per-proof, iOS
missing the expired case entirely). Given the stored proof + access
expiry, returns the timestamp to attempt a renewal (renew now if it's
<= the caller's clock, else schedule), or nullopt for "no renewal":

  - no proof, or proof already expired      -> now
  - valid proof, access expiry >1h ahead    -> ~1h before proof expiry
  - otherwise                                -> nullopt

The preemptive target is nudged off a rotation-period boundary so every
device floors the (shared) target into the same period and derives the
same rotating seed. No auto_renewing input -- 'entitlement still ahead'
(access expiry) already gates it, and gating on auto_renewing would
strand lifetime/paid-through users mid-subscription. Extracts the shared
PRO_ROTATING_SEED_PERIOD constant used by both the nudge and
ProProof::rotating_seed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the verbose std::chrono::hours{...}/minutes{...} constructs in
the refund/pro_prepaid/pro_renewal_target tests with duration literals
(1h, 30min, 10 * 24h, ...). (weeks kept as std::chrono::weeks{2} -- no
literal suffix, and it reads clearly.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The signed-message builder discarded std::to_chars's error code. The
buffer is sized for any integer so it cannot actually fail, but assert
the invariant rather than silently trusting it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ProSignedMessage bundled exactly verify_message's two arguments (a
64-byte signature and the message it signs) and existed only to be
passed, wrapped in std::optional, as the last argument of
ProProof::status -- every call site had to populate the struct first.
Pass the two directly instead: an optional 64-byte signature span plus
the message span. Call sites hand the values over without the wrapper,
and the C API stops building an intermediate C++ struct.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pro_proof_sigs, pro_status_sig and payment_details_sig were public but
called from exactly one place each -- their own *_request() -- and never
from the C API or tests. They split every request build across two
public functions for no benefit. Inline the (1-3 line) signing into each
request builder and drop the helpers, along with MasterRotatingSignatures
which only existed as pro_proof_sigs's return type.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
status only reads the proof (verify_signature, verify_message and
is_active are all const), so mark it const and let callers evaluate a
const ProProof&.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- The decode doc block was written for an old unified decode_envelope(keys,
  ...) and left attached to decode_dm_envelope: retarget it (its API name,
  the nonexistent 'keys' input, the groups-v2 dual-mode language that now
  belongs to decode_group_envelope) and drop the duplicated one-line
  description that had been appended after it.
- pro_features_for_utf8/16 documented a 'success' bool and 'features'/'bitset'
  outputs; the function returns a ProFeaturesForMsg with a 'status' enum and a
  'flags' field. Update the Outputs to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…unity_inbox

encode_for_community_inbox encodes the message as a bare (padded) Content
and blinded-encrypts it -- it never builds an Envelope, so the
sent_timestamp had nowhere to go and was silently ignored. The pro-proof
timestamp travels inside Content.sigTimestamp regardless. Drop the
vestigial parameter from the C++ and C signatures, the C wrapper, and the
two test call sites (and renumber the C NON_NULL_ARG list accordingly).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
attach_pro_sig_to_envelope always attaches a 64-byte prosig so Pro and
non-Pro envelopes are indistinguishable on the wire. The no-Pro branch
was generating a throwaway keypair and signing the message content --
two scalarmults plus hashing the whole content. Replace it with
ed25519::decoy_signature(), which builds a value with the same structure
and distribution as a real Ed25519 signature without signing anything:
R = r*B for a random scalar r (a prime-order-subgroup point, like a real
R, via base_noclamp -- a directly-sampled group element would be off the
main subgroup ~7/8 of the time and thus detectable), and s a second
random scalar in ]0, L[. It is never verified or relied on; it exists
only as a wire decoy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Most of these are just silenced, but at least a couple make changes to
the API to drop parameters that have no point.

Adds a WARN_UNUSED_PARAMETERS cmake option to turn these on.
These load paths only read the dict bytes to memcpy them into a fixed
buffer, so the maybe_vector heap allocation was unnecessary; maybe_span
returns a view into the config dict value instead.
… now

The argument must be the current time -- the seed is for the rotation
period containing it -- so `now` states that constraint precisely, where
`timestamp` misleadingly suggested any point would do. The C API param is
renamed unix_ts -> now_unix_ts to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a renewal is already due and now sits right at a weekly rotation
boundary, return now+2min rather than "renew now", so a device cleanly on
one side of the boundary can renew and propagate its config first (and if
none does, we re-poll a couple minutes on, unambiguously past the
boundary). Guarded so the deferred time still leaves >=5min of validity;
otherwise renew now. This and the existing target nudge are best-effort
collision-avoidance only -- a genuine collision still resolves fine via
config last-writer-wins/refetch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
is_final is only referenced by the following assert, so under
WARN_UNUSED_PARAMETERS a release (NDEBUG) build -- where the assert is
stripped -- warns it is unused. Mark it [[maybe_unused]] to silence that
while keeping it available to the assert in debug builds. (Parity with
the dev-branch fix prompted by the clients agent's debug-build report.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…enewal deferral

- rotating_seed doc (C++ and C): "as of now" instead of "contains now";
  drop the epoch-computation narration and the redundant naming rationale.
- pro_renewal_target: promote the boundary defer / min-validity magic
  values to documented constants -- PRO_RENEWAL_BOUNDARY_DEFER (reduced
  2min -> 1min) and PRO_RENEWAL_BOUNDARY_MIN_VALIDITY (5min) -- and reword
  the collision comment to just say a collision is resolved by config
  resolution rather than (incorrectly) describing how config resolves it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the C-style utf16_count_truncated_to_codepoints (span in, code-unit
index out) with utf16_truncate(u16string_view) -> u16string_view returning the
truncated prefix directly, and take utf16_count on a u16string_view. Remove the
assert()s on the surrogate state (invalid UTF-16 is documented-but-UB *input*,
not a programming invariant, so asserting on it is wrong) and the now-dead
surrogate helpers. Rename utf8_truncate's `n` to `max_bytes`. Tests use native
u"..." literals instead of a UTF-8->UTF-16 round-trip UDL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The utf8/16 variants forced libsession to validate and codepoint-count the
message text, but every client already counts codepoints natively. Take the
count directly via pro_features_for_message(size_t codepoint_count) and keep
only the policy libsession should own: the character-limit thresholds and the
count->flag mapping.

Drops the simdutf validation/count (and the simdutf include, its only user in
this TU), the redundant codepoint_count output field, and the now-impossible
UTFDecodingError status.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ter TODO

Same doc correction as dev: the 7-day rotating_seed quantization is not a
"weekly rotation period" -- rotation cadence is the backend-issued proof expiry
(min of subscription remaining, 30 days), and the 7-day bucket only makes every
device agree on the derived seed.

Also adds a TODO in pro_renewal_target to investigate device-count-aware renewal
jitter (skewed so the first-order statistic is uniform, using PFS's multi-device
account info) so the public renewal-time distribution can't leak the device
count -- the dev branch can't (count unknown there) and accepts the lesser
backend-only leak.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… signal

pro_renewal_target returned `now` (fetch immediately) for *any* missing or
expired proof, which told a non-Pro account -- and a lapsed one -- to hammer the
backend indefinitely. Split the two cases:
  - no proof at all: fetch only if a purchase is in flight (the prepaid marker),
    else nullopt. An entitled account carries its proof in synced config `s`, so
    genuinely having none means the account isn't Pro.
  - present-but-expired proof: always re-check. The subscription may have
    auto-renewed without this device's knowledge (a stale cached access expiry
    can't be trusted), and on an authoritative not-Pro the client clears the
    credential -- which terminates the loop.

Also corrects a stray "weekly rotation" mischaracterization in a test comment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
generate_pro_proof responses now carry account_expiry_ts -- the account's true,
grace-inclusive entitlement end (the same value get_pro_status returns), distinct
from the proof's own clamped <=30d expiry_ts. It rides the response so a proof
fetch also refreshes the client's cached access expiry. Exposed on
GenerateProProofResponse (C++, optional) and session_pro_backend_pro_proof_response
(C, 0 when absent).

Advisory and unsigned: never fed into signature verification -- M is reconstructed
from version/revocation_tag/rotating_pkey/expiry_ts only (pro-wire-protocol.md
§2.2). Required on a successful proof; also read top-level off a
subscription_expired failure so an expired-sub client can refresh its horizon
without a separate get_pro_status; absent on not_subscribed / revoked.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Pro backend (>= 0.3.0) sizes the padding on its per-account proof-expiry
grid around exactly this 1h renewal lead, so `expiry_ts - PRO_RENEWAL_LEAD`
lands just after the subscription's grace-inclusive true end. Increasing the
lead (renewing earlier than 1h before expiry) would reach the upstream store
before its final chance to report a renewal and get a spurious
subscription_expired, so it now requires a coordinated backend change. Comment
only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(PFS) Pro: atomic credential storage; key rotation & renewal logic; add_pro_payment renewal
The backend no longer reports a redeemed timestamp: no client uses it and
it exposes an internal payment-lifecycle detail with no purpose here.

The "unredeemed" status string is likewise gone: anything that could
return it redeems the payment before returning, so the value was never
actually observable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(PFS) Drop redeemed_at/redeemed_ts and the "unredeemed" status
The backend renamed the out-of-band grant provider from "rangeproof" to
"stf", reflecting that such issuances come from the Session Technology
Foundation, not the inactive Rangeproof dev house. Follow the wire/slug
value ("stf") and the C/C++ constant names to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(PFS) Rename payment provider rangeproof -> stf
Forward-port of #109 (dev commits c3f6951..b08630b) onto pfs.

session_id_matches_blinded_id() read blinded_id[1] before checking the
value's length, and never checked that it was hex.  It also used the
predicate `blinded_id[1] != '5' && (blinded_id[0] != '1' || blinded_id[0] !=
'2')`, whose right-hand side is always true (a char cannot differ from both
'1' and '2' only when it equals one of them, so the || is tautological),
leaving the check as just `blinded_id[1] != '5'` -- so any X5-prefixed value,
including a plain 05 session id, was accepted as a blinded id.

Validate length and hex before indexing, and replace the prefix tests with
starts_with("15")/starts_with("25").

Applies to pfs unmodified: line-for-line identical to the dev change.
Records dev up to 4a47113 as merged, keeping pfs's tree unchanged (-s ours).
Everything in the range is already applied to pfs:

- #104 (refund-requested-config) via #106 refund-requested-config-pfs
- #107 (drop-redeemed-at)        via #108 drop-redeemed-at-pfs
- #110 (rename-rp-to-stf)        via #111 rename-rp-to-stf-pfs
- #109 (fix-blinded-id-validation) forward-ported in the preceding commit

The first three were rewritten for pfs and so have different patch-ids, which
is why `git cherry` still reports their commits as absent; the content was
verified present by comparing the identifiers each PR added or removed across
both branches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants