From 6bf61b6961df7c3be94ebcd13063f7265070bbe1 Mon Sep 17 00:00:00 2001 From: simonredfern Date: Fri, 5 Jun 2026 05:50:38 +0200 Subject: [PATCH 1/4] Create settlement_account_ids_question.md --- ideas/settlement_account_ids_question.md | 180 +++++++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100644 ideas/settlement_account_ids_question.md diff --git a/ideas/settlement_account_ids_question.md b/ideas/settlement_account_ids_question.md new file mode 100644 index 0000000000..fae5d97e4e --- /dev/null +++ b/ideas/settlement_account_ids_question.md @@ -0,0 +1,180 @@ +# For Marko — Settlement account IDs should become UUIDs + +## TL;DR + +Default settlement accounts are created with **hardcoded, non-UUID, globally-non-unique** `account_id` strings: + +- `INCOMING_SETTLEMENT_ACCOUNT_ID = "OBP-INCOMING-SETTLEMENT-ACCOUNT"` +- `OUTGOING_SETTLEMENT_ACCOUNT_ID = "OBP-OUTGOING-SETTLEMENT-ACCOUNT"` + +Every bank gets the **same** two strings as `account_id`. In a deployment with N banks, there are 2N rows in `mappedbankaccount` where the `account_id` collides across banks. The `(bank_id, account_id)` unique constraint is satisfied (different bank per row), but **globally on `account_id` alone, the strings collide**. + +This violates the `Account.account_id` glossary contract ("MUST be a UUID") and undermines the federation-safety claim that justifies `(OBP_ACCOUNT_ID, account_id)` being a globally-unique routing pair. + +## Why it matters + +The whole reason `(OBP_ACCOUNT_ID, account_id)` is supposed to be a safe federated routing identifier is "account_ids are UUIDs, so collision probability ≈ 0." Settlement accounts demonstrably collide across banks. Any cross-instance routing logic that traverses a settlement account would need to carry bank context out-of-band, which defeats the point. + +For now this is bounded — settlement accounts are internal plumbing, not user-facing routable accounts — but the carve-out should be documented and ideally closed. + +## Where the literal IDs are referenced (today) + +Audit run 2026-05-21. 45 hits across 7 files. + +### Production code (3 files) + +#### `obp-api/src/main/scala/bootstrap/liftweb/Boot.scala:820-865` + +`createDefaultBankAndDefaultAccountsIfNotExisting` — creates the default sandbox bank's settlement pair on every API boot. Two hits. + +#### `obp-api/src/main/scala/code/bankconnectors/LocalMappedConnector.scala` — 20 hits + +**Creation (`:3411-3441`)** — per-bank: looks for each settlement account by `(bankId, theAccountId=literal)`, creates if missing. 8 hits. Runs whenever a bank is created via the local connector. + +**Lookup** in two near-duplicate functions: + +- `saveDoubleEntryBookTransactionByCounterparty` (`:2264-2310`) +- A second function with identical structure (`:2421-2466`) + +Each function performs a **three-tier cascade**: + +```scala +// Tier 1 — payment-system + currency (most specific, manually created) +BankAccountX(bankId, AccountId(transactionRequestType + "_SETTLEMENT_ACCOUNT_" + currency), ...) +// Tier 2 — currency-only (manually created) +.or(BankAccountX(bankId, AccountId("DEFAULT_SETTLEMENT_ACCOUNT_" + currency), ...)) +// Tier 3 — global fallback (auto-created, the literal constant) +.or(BankAccountX(bankId, AccountId(INCOMING_SETTLEMENT_ACCOUNT_ID), ...)) +``` + +After lookup, each function uses a **discrimination check** to know whether it landed on the global fallback (and therefore needs FX conversion because the global fallback is EUR-only): + +```scala +if (settlementAccount._1.accountId.value == INCOMING_SETTLEMENT_ACCOUNT_ID && settlementAccount._1.currency != fromAccount.currency) +``` + +This check (4 occurrences across the two functions) is the **only place** the literal constant value is operationally used after lookup — and it's used purely to detect "we hit the fallback." + +The discrimination check is the structural reason settlement accounts can't be cleanly migrated by just changing the literal: the lookup code needs *some* signal to detect tier-3-fallback. Today it's string equality. Going forward it should be a property on the account (e.g. a `is_default_settlement_fallback` flag, or a `SettlementTier` column in a dedicated table). + +Plus 4 error-message hits at lines 2309, 2310, 2465, 2466 — diagnostic only, easy to update. + +#### `obp-api/src/main/scala/code/api/util/migration/MigrationOfSettlementAccounts.scala` + +Retroactive migration. ~10 hits. For every existing bank found in `MappedBank`, ensure the two literal-id settlement accounts exist. Already executed on existing deployments — historical. + +### Test code (2 files) + +- `obp-api/src/test/scala/code/api/v4_0_0/BankTests.scala` +- `obp-api/src/test/scala/code/api/v5_0_0/BankTests.scala` + +Both assert that bank creation produces these settlement accounts. Tests will need updating in lockstep with whichever migration approach is chosen. + +### Docs / definition + +- `obp-api/src/main/scala/code/api/constant/constant.scala:247-248` — the `final val` definitions themselves. +- `obp-api/src/main/scala/code/api/util/Glossary.scala` — passing reference. + +## What the lookup actually depends on + +The lookup logic doesn't care about the strings per se — it cares about a **classifier**: given `(bank, direction, transactionRequestType, currency)`, which account is the settlement target? Today the classifier is encoded into the `account_id` string via three naming conventions. If we move the classifier out of the `account_id`, the `account_id` is free to be a UUID. + +Also worth noting: the `kind` column on `MappedBankAccount` is set to `"SETTLEMENT"` for these accounts but **is never queried**. It exists as metadata only. We could lean on it as part of any new lookup. + +## Recommended approach — lower-disruption variant + +Avoid renaming existing settlement accounts (which would require cascading FK updates across transactions, account_attributes, views, account_access rows, etc.). Instead, **stop creating new non-UUID settlement accounts** and introduce a proper lookup mechanism that works for both legacy and new accounts. + +### PR 1 — Introduce dedicated lookup, switch new banks to UUIDs + +1. **New table `MappedBankSettlementAccount`**: + + | column | type | note | + |---|---|---| + | `bank_id` | string | FK to `mappedbank.permalink` | + | `direction` | enum `INCOMING` / `OUTGOING` | required | + | `tx_type` | string nullable | e.g. `SEPA`, null = any | + | `currency` | string nullable | ISO 4217, null = any | + | `account_id` | string | FK to `mappedbankaccount.theaccountid` | + | `tier` | int | 1=most-specific, 3=global fallback. Indexed. | + + Unique index on `(bank_id, direction, tx_type, currency)`. + +2. **Lookup function** in a new `code.bankconnectors.settlement` (or extension of `LocalMappedConnector`): + + ```scala + def findSettlementAccount( + bankId: BankId, + direction: SettlementDirection, + txType: Option[String], + currency: Option[String] + ): Option[(BankAccount, SettlementTier)] + ``` + + Returns the highest-tier matching row. `SettlementTier` lets callers know whether they hit the global fallback (replacing the current `accountId.value == CONSTANT` check). + +3. **New bank creation** (in `LocalMappedConnector.scala:3411-3441` and `Boot.scala:820-865`): + - Mint UUIDs for the two settlement accounts. + - Record them in `MappedBankSettlementAccount` as `(direction, tx_type=null, currency=null, tier=3)`. + +4. **Refactor the two cascade sites** in `LocalMappedConnector.scala:2264-2306` and `:2421-2462`: + - Replace the `.or` cascade with a single `findSettlementAccount(...)` call. + - Replace the `settlementAccount._1.accountId.value == CONSTANT` discrimination with `tier == SettlementTier.GlobalFallback`. + +5. **Lookup fallback**: if `findSettlementAccount` finds nothing, fall back to the legacy string-name lookup. This preserves all existing behaviour for unmigrated banks. + +6. **Tests**: assertions stay valid for legacy banks; add new ones for the UUID path. + +### PR 2 (optional) — Backfill `MappedBankSettlementAccount` for legacy banks + +Migration script: + +1. For each existing bank, find all accounts with `kind="SETTLEMENT"`. +2. Decode each account's literal `account_id` to determine `(direction, tx_type, currency, tier)`: + - `OBP-INCOMING-SETTLEMENT-ACCOUNT` → `(INCOMING, null, null, 3)` + - `OBP-OUTGOING-SETTLEMENT-ACCOUNT` → `(OUTGOING, null, null, 3)` + - `DEFAULT_SETTLEMENT_ACCOUNT_` → `(direction=both? or inferred?, null, , 2)` + - `_SETTLEMENT_ACCOUNT_` → `(direction=both? or inferred?, , , 1)` +3. Insert rows into `MappedBankSettlementAccount`. +4. Leave the underlying `mappedbankaccount.theaccountid` values alone. + +After PR 2, every settlement account is reachable both via legacy string lookup and via the new table. PR 1's "fallback to legacy" code path effectively becomes dead but stays for safety. + +**Important decision point for PR 2**: tier 1 and tier 2 settlement accounts (`_SETTLEMENT_ACCOUNT_` and `DEFAULT_SETTLEMENT_ACCOUNT_`) carry no encoded direction in their literal name — the existing code uses the same account_id for both incoming and outgoing in the tier-1/2 paths. This needs clarification before the backfill: is a single account intended to handle both directions, or is the absence of `INCOMING`/`OUTGOING` in those names a latent bug? + +### PR 3 (optional, far future) — Rename legacy settlement account_ids to UUIDs + +Only after PR 1 + PR 2 are stable and the legacy-string lookup path is confirmed unused. + +1. For each `MappedBankSettlementAccount` row pointing at a non-UUID `account_id`, mint a UUID. +2. Cascade-update every FK reference: `mappedtransaction`, `mappedaccountattribute`, `viewdefinition`, `accountaccess`, anything else. **Enumerate carefully**. +3. Update `mappedbankaccount.theaccountid` to the UUID. +4. Update `MappedBankSettlementAccount.account_id` to the UUID. +5. Remove the literal constants from `constant.scala`. Compile errors flush out any remaining references. + +This is the scary one. Possibly never worth doing — the lookup-table indirection means legacy account_ids are an internal implementation detail that nobody outside settlement code needs to know about. + +## Decisions needed before starting + +- **Tier-1 / tier-2 direction**: do `SEPA_SETTLEMENT_ACCOUNT_USD` etc. need separate INCOMING / OUTGOING entries, or is one account meant to handle both? (Audit production data, or ask whoever set this up.) +- **Boot.scala vs LocalMappedConnector.scala duplication**: `Boot.scala:820-865` creates default-bank settlement accounts; `LocalMappedConnector.scala:3411-3441` does the same on per-bank creation. They duplicate logic. Worth consolidating into a single helper as part of PR 1. +- **The two `LocalMappedConnector` cascade functions** (`:2264-2306` and `:2421-2462`) look near-identical. Worth deduplicating during the refactor. + +## Out of scope for this work + +- **The bank_id `-` convention** discussed separately — settlement accounts live within a bank, so they ride on whatever bank_id shape the deployment has chosen. Tracked in `todo_account_id_uuid_enforcement.md`. +- **Validating UUID account_ids at the API boundary** — separate workstream tracked in `todo_account_id_uuid_enforcement.md`. +- **Settlement account creation via the public API** — currently settlement accounts are only created by Boot/migration/connector init. If/when the API surface admits user-driven settlement-account creation, the lookup table needs an API too. + +## File-by-file checklist (for whoever picks this up) + +- [ ] `obp-commons/src/main/scala/com/openbankproject/commons/model/` — new `SettlementDirection` and `SettlementTier` enums +- [ ] `obp-api/src/main/scala/code/bankconnectors/settlement/MappedBankSettlementAccount.scala` — new Mapper +- [ ] `obp-api/src/main/scala/code/api/util/migration/MigrationOfBankSettlementAccountTable.scala` — DDL for the new table +- [ ] `obp-api/src/main/scala/code/bankconnectors/LocalMappedConnector.scala:2264-2310, 2421-2466` — refactor cascade +- [ ] `obp-api/src/main/scala/code/bankconnectors/LocalMappedConnector.scala:3411-3441` — mint UUIDs for new settlement accounts, record in the new table +- [ ] `obp-api/src/main/scala/bootstrap/liftweb/Boot.scala:820-865` — same change for default bank +- [ ] `obp-api/src/test/scala/code/api/v4_0_0/BankTests.scala` and `v5_0_0/BankTests.scala` — update assertions +- [ ] `obp-api/src/main/scala/code/api/util/Glossary.scala` — add a "Known exception" note to `Account.account_id` until/unless PR 3 happens +- [ ] (PR 2) Backfill migration +- [ ] (PR 3) Rename + FK cascade From 72f104247ce454f8c7ae489d9fd087966390152a Mon Sep 17 00:00:00 2001 From: simonredfern Date: Fri, 5 Jun 2026 11:51:18 +0200 Subject: [PATCH 2/4] DE writeRole and readRole phase 5 and 6 part --- .../dynamic/entity/Http4sDynamicEntity.scala | 112 ++++++++++++++++-- .../entity/helper/DynamicEntityHelper.scala | 71 +++++++++-- .../scala/code/api/v6_0_0/Http4s600.scala | 19 ++- .../dynamicEntity/DynamicEntityProvider.scala | 53 ++++++++- 4 files changed, 233 insertions(+), 22 deletions(-) diff --git a/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala b/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala index ddc30ca64d..a67c2a8850 100644 --- a/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala +++ b/obp-api/src/main/scala/code/api/dynamic/entity/Http4sDynamicEntity.scala @@ -165,6 +165,67 @@ object Http4sDynamicEntity extends MdcLoggable { private def failIf(error: Box[ErrorMessage], cc: Option[CallContext]): Future[Box[Unit]] = Helper.booleanToFuture(failMsg = error.map(_.message).orNull, failCode = error.map(_.code).openOr(400), cc = cc) { error.isEmpty } + // ----- Field-level write permissions: POST/PUT never write write-restricted fields ----- + private def writeRestrictedFieldsOf(bankId: Option[String], entityName: String): List[String] = + DynamicEntityHelper.definitionsMap.get((bankId, entityName)).map(_.writeRestrictedFields).getOrElse(Nil) + + private def stripFields(obj: JObject, fields: List[String]): JObject = + if (fields.isEmpty) obj else JObject(obj.obj.filterNot(f => fields.contains(f.name))) + + // For PUT: drop any write-restricted fields the caller sent, then re-inject their current values from the + // existing record, so an ordinary update never blanks or clobbers a restricted field (no stale-echo issue). + private def preserveRestrictedOnPut(incoming: JObject, existing: Box[JValue], restricted: List[String]): JObject = { + if (restricted.isEmpty) incoming + else { + val stripped = stripFields(incoming, restricted).obj + val preserved = existing match { + case Full(o: JObject) => o.obj.filter(f => restricted.contains(f.name)) + case _ => Nil + } + JObject(stripped ++ preserved) + } + } + + // ----- Field-level write path (PATCH): role-gated writes to restricted fields ----- + // Names of the per-field write roles the caller is missing for the restricted fields present in the body. + private def missingFieldWriteRoleNames(bodyFieldNames: List[String], bankId: Option[String], entityName: String, userId: String): List[String] = { + val info = DynamicEntityHelper.definitionsMap.get((bankId, entityName)) + val restrictedInBody = info.map(_.writeRestrictedFields).getOrElse(Nil).intersect(bodyFieldNames) + restrictedInBody.flatMap { f => + val role = DynamicEntityInfo.fieldWriteRole(entityName, f, bankId, info.flatMap(_.explicitWriteRole(f))) + if (code.api.util.APIUtil.hasEntitlement(bankId.getOrElse(""), userId, role)) None else Some(role.toString()) + }.distinct + } + + // PATCH partial-update merge: incoming values override existing; absent fields preserved. + // Bounded to declared schema fields so id/audit fields aren't written into the stored data. + private def mergePatch(info: Option[DynamicEntityInfo], existing: Box[JValue], incoming: JObject): JObject = { + val existingMap: Map[String, JValue] = (existing match { case Full(o: JObject) => o.obj; case _ => Nil }).map(f => f.name -> f.value).toMap + val incomingMap: Map[String, JValue] = incoming.obj.map(f => f.name -> f.value).toMap + val names: List[String] = info.map(_.propertyNames).getOrElse((incomingMap.keys ++ existingMap.keys).toList).distinct + JObject(names.flatMap(n => incomingMap.get(n).orElse(existingMap.get(n)).map(v => JField(n, v)))) + } + + // ----- Field-level read permissions: omit read-restricted fields the caller cannot read ----- + private def omitFields(value: JValue, fields: Set[String]): JValue = value match { + case JObject(obj) => JObject(obj.filterNot(f => fields.contains(f.name))) + case JArray(arr) => JArray(arr.map(omitFields(_, fields))) + case other => other + } + + // Remove any read-restricted field the caller lacks the read role for (anonymous => userIdOpt None => omit all). + private def applyReadRestrictions(value: JValue, bankId: Option[String], entityName: String, userIdOpt: Option[String]): JValue = { + val info = DynamicEntityHelper.definitionsMap.get((bankId, entityName)) + val readRestricted = info.map(_.readRestrictedFields).getOrElse(Nil) + val omit: Set[String] = readRestricted.filterNot { f => + userIdOpt.exists { uid => + val role = DynamicEntityInfo.fieldReadRole(entityName, f, bankId, info.flatMap(_.explicitReadRole(f))) + code.api.util.APIUtil.hasEntitlement(bankId.getOrElse(""), uid, role) + } + }.toSet + if (omit.isEmpty) value else omitFields(value, omit) + } + // ----- generic endpoint (authenticated, system / bank / personal) ----- private def genericGet(req: Request[IO], bankId: Option[String], entityName: String, id: String, isPersonalEntity: Boolean): IO[Response[IO]] = @@ -186,10 +247,11 @@ object Http4sDynamicEntity extends MdcLoggable { } yield { if (isGetAll) { val resultList: JArray = unboxResult(box.asInstanceOf[Box[JArray]], entityName) - wrapBankId(bankId, (listName(entityName) -> filterDynamicObjects(resultList, queryParams(req)))) + val filtered = filterDynamicObjects(resultList, queryParams(req)) + wrapBankId(bankId, (listName(entityName) -> applyReadRestrictions(filtered, bankId, entityName, Some(u.userId)))) } else { val singleObject: JValue = unboxResult(box.asInstanceOf[Box[JValue]], entityName) - wrapBankId(bankId, (singleName(entityName) -> singleObject)) + wrapBankId(bankId, (singleName(entityName) -> applyReadRestrictions(singleObject, bankId, entityName, Some(u.userId)))) } } } @@ -208,7 +270,9 @@ object Http4sDynamicEntity extends MdcLoggable { else NewStyle.function.hasEntitlement(bankId.getOrElse(""), u.userId, DynamicEntityInfo.canCreateRole(entityName, bankId), callContext) _ <- failIf(afterIntercept(callContext, operationId), callContext) json <- NewStyle.function.tryons(InvalidJsonFormat, 400, callContext) { net.liftweb.json.parse(cc.httpBody.getOrElse("")) } - (box, _) <- NewStyle.function.invokeDynamicConnector(CREATE, entityName, Some(json.asInstanceOf[JObject]), None, bankId, None, Some(u.userId), isPersonalEntity, Some(cc)) + // Write-restricted fields are never set via POST; strip them before persisting. + createJson = stripFields(json.asInstanceOf[JObject], writeRestrictedFieldsOf(bankId, entityName)) + (box, _) <- NewStyle.function.invokeDynamicConnector(CREATE, entityName, Some(createJson), None, bankId, None, Some(u.userId), isPersonalEntity, Some(cc)) singleObject: JValue = unboxResult(box.asInstanceOf[Box[JValue]], entityName) } yield wrapBankId(bankId, (singleName(entityName) -> singleObject)) } @@ -228,7 +292,36 @@ object Http4sDynamicEntity extends MdcLoggable { json <- NewStyle.function.tryons(InvalidJsonFormat, 400, callContext) { net.liftweb.json.parse(cc.httpBody.getOrElse("")) } (existing, _) <- NewStyle.function.invokeDynamicConnector(GET_ONE, entityName, None, Some(id), bankId, None, Some(u.userId), isPersonalEntity, Some(cc)) _ <- Helper.booleanToFuture(notFoundMsg(entityName, id, bankId), 404, cc = callContext) { existing.isDefined } - (box: Box[JValue], _) <- NewStyle.function.invokeDynamicConnector(UPDATE, entityName, Some(json.asInstanceOf[JObject]), Some(id), bankId, None, Some(u.userId), isPersonalEntity, Some(cc)) + // Write-restricted fields are not updated via PUT; preserve their existing values. + updateJson = preserveRestrictedOnPut(json.asInstanceOf[JObject], existing.asInstanceOf[Box[JValue]], writeRestrictedFieldsOf(bankId, entityName)) + (box: Box[JValue], _) <- NewStyle.function.invokeDynamicConnector(UPDATE, entityName, Some(updateJson), Some(id), bankId, None, Some(u.userId), isPersonalEntity, Some(cc)) + singleObject: JValue = unboxResult(box, entityName) + } yield wrapBankId(bankId, (singleName(entityName) -> singleObject)) + } + + private def genericPatch(req: Request[IO], bankId: Option[String], entityName: String, id: String, isPersonalEntity: Boolean): IO[Response[IO]] = + EndpointHelpers.executeAndRespond(req) { cc => + val callContext0 = enrichCallContext(cc, UPDATE, entityName, bankId, if (isPersonalEntity) "my" else "") + val operationId = callContext0.operationId.orNull + for { + _ <- failIf(beforeIntercept(callContext0, operationId), Some(callContext0)) + (Full(u), callContext) <- authenticatedAccess(callContext0) + (_, callContext) <- bankCheck(bankId, callContext) + personalRequiresRole = DynamicEntityHelper.definitionsMap.get((bankId, entityName)).exists(_.personalRequiresRole) + // Baseline: PATCH needs the entity update role (like PUT) for unrestricted fields. + _ <- if (isPersonalEntity && !personalRequiresRole) Future.successful(true) + else NewStyle.function.hasEntitlement(bankId.getOrElse(""), u.userId, DynamicEntityInfo.canUpdateRole(entityName, bankId), callContext) + _ <- failIf(afterIntercept(callContext, operationId), callContext) + json <- NewStyle.function.tryons(InvalidJsonFormat, 400, callContext) { net.liftweb.json.parse(cc.httpBody.getOrElse("")) } + bodyObj = json.asInstanceOf[JObject] + // Per-field: every write-restricted field present in the body needs its field-level write role. + missingRoles = missingFieldWriteRoleNames(bodyObj.obj.map(_.name), bankId, entityName, u.userId) + _ <- Helper.booleanToFuture(s"$UserHasMissingRoles ${missingRoles.mkString(", ")}", 403, cc = callContext) { missingRoles.isEmpty } + (existing, _) <- NewStyle.function.invokeDynamicConnector(GET_ONE, entityName, None, Some(id), bankId, None, Some(u.userId), isPersonalEntity, Some(cc)) + _ <- Helper.booleanToFuture(notFoundMsg(entityName, id, bankId), 404, cc = callContext) { existing.isDefined } + // PATCH = partial update: merge incoming fields over the existing record. + mergedJson = mergePatch(DynamicEntityHelper.definitionsMap.get((bankId, entityName)), existing.asInstanceOf[Box[JValue]], bodyObj) + (box: Box[JValue], _) <- NewStyle.function.invokeDynamicConnector(UPDATE, entityName, Some(mergedJson), Some(id), bankId, None, Some(u.userId), isPersonalEntity, Some(cc)) singleObject: JValue = unboxResult(box, entityName) } yield wrapBankId(bankId, (singleName(entityName) -> singleObject)) } @@ -269,10 +362,11 @@ object Http4sDynamicEntity extends MdcLoggable { } yield { if (isGetAll) { val resultList: JArray = unboxResult(box.asInstanceOf[Box[JArray]], entityName) - wrapBankId(bankId, (listName(entityName) -> filterDynamicObjects(resultList, queryParams(req)))) + val filtered = filterDynamicObjects(resultList, queryParams(req)) + wrapBankId(bankId, (listName(entityName) -> applyReadRestrictions(filtered, bankId, entityName, None))) } else { val singleObject: JValue = unboxResult(box.asInstanceOf[Box[JValue]], entityName) - wrapBankId(bankId, (singleName(entityName) -> singleObject)) + wrapBankId(bankId, (singleName(entityName) -> applyReadRestrictions(singleObject, bankId, entityName, None))) } } } @@ -295,14 +389,15 @@ object Http4sDynamicEntity extends MdcLoggable { if (isGetAll) { val resultList: List[JObject] = DynamicDataProvider.connectorMethodProvider.vend.getAllDataJsonCommunity(bankId, entityName) val resultArray = JArray(resultList) - wrapBankId(bankId, (listName(entityName) -> filterDynamicObjects(resultArray, queryParams(req)))) + val filtered = filterDynamicObjects(resultArray, queryParams(req)) + wrapBankId(bankId, (listName(entityName) -> applyReadRestrictions(filtered, bankId, entityName, Some(u.userId)))) } else { val singleResult = DynamicDataProvider.connectorMethodProvider.vend.getCommunity(bankId, entityName, id) val singleObject: JValue = singleResult match { case Full(data) => net.liftweb.json.parse(data.dataJson) case _ => throw new RuntimeException(notFoundMsg(entityName, id, bankId)) } - wrapBankId(bankId, (singleName(entityName) -> singleObject)) + wrapBankId(bankId, (singleName(entityName) -> applyReadRestrictions(singleObject, bankId, entityName, Some(u.userId)))) } } } @@ -325,6 +420,7 @@ object Http4sDynamicEntity extends MdcLoggable { case Method.GET => Some(r => genericGet(r, bankId, entityName, id, isPersonalEntity)) case Method.POST => Some(r => genericPost(r, bankId, entityName, isPersonalEntity)) case Method.PUT => Some(r => genericPut(r, bankId, entityName, id, isPersonalEntity)) + case Method.PATCH => Some(r => genericPatch(r, bankId, entityName, id, isPersonalEntity)) case Method.DELETE => Some(r => genericDelete(r, bankId, entityName, id, isPersonalEntity)) case _ => None } diff --git a/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala b/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala index 505d4e4ea6..d5ba958b4f 100644 --- a/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala +++ b/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala @@ -123,7 +123,15 @@ object DynamicEntityHelper { // (Some(BankId), EntityName, DynamicEntityInfo) def definitionsMap: Map[(Option[String], String), DynamicEntityInfo] = NewStyle.function.getDynamicEntities(None, true).map(it => ((it.bankId, it.entityName), DynamicEntityInfo(it.metadataJson, it.entityName, it.bankId, it.hasPersonalEntity, it.hasPublicAccess, it.hasCommunityAccess, it.personalRequiresRole))).toMap - def dynamicEntityRoles: List[String] = NewStyle.function.getDynamicEntities(None, true).flatMap(dEntity => DynamicEntityInfo.roleNames(dEntity.entityName, dEntity.bankId)) + def dynamicEntityRoles: List[String] = NewStyle.function.getDynamicEntities(None, true).flatMap { dEntity => + val baseRoles = DynamicEntityInfo.roleNames(dEntity.entityName, dEntity.bankId) + // Per-field write/read roles for any restricted fields (explicit shared role, or auto-generated). + val writeRoles = dEntity.writeRestrictedFields.map(f => + DynamicEntityInfo.fieldWriteRole(dEntity.entityName, f, dEntity.bankId, dEntity.explicitWriteRole(f)).toString()) + val readRoles = dEntity.readRestrictedFields.map(f => + DynamicEntityInfo.fieldReadRole(dEntity.entityName, f, dEntity.bankId, dEntity.explicitReadRole(f)).toString()) + baseRoles ++ writeRoles ++ readRoles + }.distinct def doc: ArrayBuffer[ResourceDoc] = { val docs = operationToResourceDoc.values.toList @@ -287,7 +295,7 @@ object DynamicEntityHelper { |${userAuthenticationMessage(true)} | |""", - dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExampleWithoutIdWritable, dynamicEntityInfo.getSingleExample, List( AuthenticatedUserIsRequired, @@ -316,7 +324,7 @@ object DynamicEntityHelper { |${userAuthenticationMessage(true)} | |""", - dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExampleWithoutIdWritable, dynamicEntityInfo.getSingleExample, List( AuthenticatedUserIsRequired, @@ -342,7 +350,7 @@ object DynamicEntityHelper { |${userAuthenticationMessage(true)} | |""", - dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExampleWithoutIdWritable, dynamicEntityInfo.getSingleExample, List( AuthenticatedUserIsRequired, @@ -426,7 +434,7 @@ object DynamicEntityHelper { |${userAuthenticationMessage(true)} | |""", - dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExampleWithoutIdWritable, dynamicEntityInfo.getSingleExample, myErrorMessagesWithJson, List(apiTag, apiTagDynamicEntity, apiTagDynamic), @@ -450,7 +458,7 @@ object DynamicEntityHelper { |${userAuthenticationMessage(true)} | |""", - dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExampleWithoutIdWritable, dynamicEntityInfo.getSingleExample, myErrorMessagesWithJson, List(apiTag, apiTagDynamicEntity, apiTagDynamic), @@ -471,7 +479,7 @@ object DynamicEntityHelper { |${userAuthenticationMessage(true)} | |""", - dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExampleWithoutIdWritable, dynamicEntityInfo.getSingleExample, myErrorMessages, List(apiTag, apiTagDynamicEntity, apiTagDynamic), @@ -729,6 +737,13 @@ case class DynamicEntityInfo(definition: String, entityName: String, bankId: Opt val exampleFields = fields.map(field => JField(field.name, extractExample(field.value))) JObject(exampleFields) } + + // Request-body example for POST/PUT: excludes write-restricted fields (they're not settable here; only via PATCH). + def getSingleExampleWithoutIdWritable: JObject = { + val restricted = writeRestrictedFields.toSet + if (restricted.isEmpty) getSingleExampleWithoutId + else JObject(getSingleExampleWithoutId.obj.filterNot(f => restricted.contains(f.name))) + } val bankIdJObject: JObject = ("bank-id" -> ExampleValue.bankIdExample.value) def getSingleExample: JObject = if (bankId.isDefined){ @@ -754,6 +769,30 @@ case class DynamicEntityInfo(definition: String, entityName: String, bankId: Opt val canUpdateRole: ApiRole = DynamicEntityInfo.canUpdateRole(entityName, bankId) val canGetRole: ApiRole = DynamicEntityInfo.canGetRole(entityName, bankId) val canDeleteRole: ApiRole = DynamicEntityInfo.canDeleteRole(entityName, bankId) + + // ----- Field-level access control (mirrors DynamicEntityT; here `entity` is already the per-entity object) ----- + private def restrictedFields(requiredFlag: String, roleKey: String): List[String] = + (entity \ "properties") match { + case props: JObject => props.obj.collect { + case JField(name, propDef: JObject) + if (propDef \ requiredFlag) == JBool(true) || + ((propDef \ roleKey) match { case JString(s) => s.nonEmpty; case _ => false }) => name + } + case _ => Nil + } + /** Fields written only via the role-gated PATCH path (not via POST/PUT). */ + lazy val writeRestrictedFields: List[String] = restrictedFields("writeRoleRequired", "writeRole") + /** Fields omitted from GET unless the caller holds the read role. */ + lazy val readRestrictedFields: List[String] = restrictedFields("readRoleRequired", "readRole") + def explicitWriteRole(fieldName: String): Option[String] = + (entity \ "properties" \ fieldName \ "writeRole") match { case JString(s) if s.nonEmpty => Some(s); case _ => None } + def explicitReadRole(fieldName: String): Option[String] = + (entity \ "properties" \ fieldName \ "readRole") match { case JString(s) if s.nonEmpty => Some(s); case _ => None } + /** Declared schema property names (used to bound a PATCH merge to real fields). */ + lazy val propertyNames: List[String] = (entity \ "properties") match { + case props: JObject => props.obj.map(_.name) + case _ => Nil + } } object DynamicEntityInfo { @@ -786,4 +825,22 @@ object DynamicEntityInfo { canGetRole(entityName, bankId), canDeleteRole(entityName, bankId) ).map(_.toString()) + + // Field-level roles. If the definition declares an explicit writeRole/readRole, use it verbatim + // (so many fields/entities can share one role); otherwise auto-generate a per-field role. + def fieldWriteRole(entityName: String, fieldName: String, bankId: Option[String], explicit: Option[String]): ApiRole = + explicit match { + case Some(role) => getOrCreateDynamicApiRole(role, bankId.isDefined) + case None => + if(bankId.isDefined) getOrCreateDynamicApiRole(s"CanWriteDynamicEntityField_${entityName}__${fieldName}", true) + else getOrCreateDynamicApiRole(s"CanWriteDynamicEntityField_System${entityName}__${fieldName}", false) + } + + def fieldReadRole(entityName: String, fieldName: String, bankId: Option[String], explicit: Option[String]): ApiRole = + explicit match { + case Some(role) => getOrCreateDynamicApiRole(role, bankId.isDefined) + case None => + if(bankId.isDefined) getOrCreateDynamicApiRole(s"CanGetDynamicEntityField_${entityName}__${fieldName}", true) + else getOrCreateDynamicApiRole(s"CanGetDynamicEntityField_System${entityName}__${fieldName}", false) + } } diff --git a/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala b/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala index 4e9a2fd0d4..0aec4ca566 100644 --- a/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala +++ b/obp-api/src/main/scala/code/api/v6_0_0/Http4s600.scala @@ -6781,7 +6781,8 @@ object Http4s600 { | "required": ["theme"], | "properties": { | "theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, - | "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"} + | "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}, + | "internal_note": {"type": "string", "example": "set by a privileged service", "description": "Field-level write-restricted", "writeRoleRequired": true} | } | } |} @@ -6791,6 +6792,7 @@ object Http4s600 { |* The `entity_name` must be lowercase with underscores (snake_case), e.g. `customer_preferences`. No uppercase letters or spaces allowed. |* Each property MUST include an `example` field with a valid example value. |* Each property can optionally include `description` (markdown text), and for string types: `minLength` and `maxLength`. + |* Each property can optionally declare **field-level access control**: `writeRoleRequired`/`readRoleRequired` (booleans — auto-generate a per-field role) or `writeRole`/`readRole` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). @@ -6802,7 +6804,7 @@ object Http4s600 { has_public_access = Some(false), has_community_access = Some(false), personal_requires_role = Some(false), - schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] + schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}, "internal_note": {"type": "string", "example": "set by a privileged service", "description": "Field-level write-restricted (writeRoleRequired)", "writeRoleRequired": true}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] ), DynamicEntityDefinitionJsonV600( dynamic_entity_id = "abc-123-def", @@ -6813,7 +6815,7 @@ object Http4s600 { has_public_access = false, has_community_access = false, personal_requires_role = false, - schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] + schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}, "internal_note": {"type": "string", "example": "set by a privileged service", "description": "Field-level write-restricted (writeRoleRequired)", "writeRoleRequired": true}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] ), List($AuthenticatedUserIsRequired, UserHasMissingRoles, InvalidJsonFormat, UnknownError), apiTagManageDynamicEntity :: apiTagApi :: Nil, @@ -6844,7 +6846,8 @@ object Http4s600 { | "required": ["theme"], | "properties": { | "theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, - | "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"} + | "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}, + | "internal_note": {"type": "string", "example": "set by a privileged service", "description": "Field-level write-restricted", "writeRoleRequired": true} | } | } |} @@ -6854,6 +6857,7 @@ object Http4s600 { |* The `entity_name` must be lowercase with underscores (snake_case), e.g. `customer_preferences`. No uppercase letters or spaces allowed. |* Each property MUST include an `example` field with a valid example value. |* Each property can optionally include `description` (markdown text), and for string types: `minLength` and `maxLength`. + |* Each property can optionally declare **field-level access control**: `writeRoleRequired`/`readRoleRequired` (booleans — auto-generate a per-field role) or `writeRole`/`readRole` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). @@ -6865,7 +6869,7 @@ object Http4s600 { has_public_access = Some(false), has_community_access = Some(false), personal_requires_role = Some(false), - schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] + schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}, "internal_note": {"type": "string", "example": "set by a privileged service", "description": "Field-level write-restricted (writeRoleRequired)", "writeRoleRequired": true}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] ), DynamicEntityDefinitionJsonV600( dynamic_entity_id = "abc-123-def", @@ -6876,7 +6880,7 @@ object Http4s600 { has_public_access = false, has_community_access = false, personal_requires_role = false, - schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] + schema = net.liftweb.json.parse("""{"description": "User preferences", "required": ["theme"], "properties": {"theme": {"type": "string", "minLength": 1, "maxLength": 20, "example": "dark", "description": "The UI theme preference"}, "language": {"type": "string", "minLength": 2, "maxLength": 5, "example": "en", "description": "ISO language code"}, "internal_note": {"type": "string", "example": "set by a privileged service", "description": "Field-level write-restricted (writeRoleRequired)", "writeRoleRequired": true}}}""").asInstanceOf[net.liftweb.json.JsonAST.JObject] ), List( $BankNotFound, @@ -6923,6 +6927,7 @@ object Http4s600 { |**Note:** |* The `entity_name` must be lowercase with underscores (snake_case), e.g. `customer_preferences`. No uppercase letters or spaces allowed. |* Each property can optionally include `description` (markdown text), and for string types: `minLength` and `maxLength`. + |* Each property can optionally declare **field-level access control**: `writeRoleRequired`/`readRoleRequired` (booleans — auto-generate a per-field role) or `writeRole`/`readRole` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). @@ -6981,6 +6986,7 @@ object Http4s600 { |**Note:** |* The `entity_name` must be lowercase with underscores (snake_case), e.g. `customer_preferences`. No uppercase letters or spaces allowed. |* Each property can optionally include `description` (markdown text), and for string types: `minLength` and `maxLength`. + |* Each property can optionally declare **field-level access control**: `writeRoleRequired`/`readRoleRequired` (booleans — auto-generate a per-field role) or `writeRole`/`readRole` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). @@ -7045,6 +7051,7 @@ object Http4s600 { |**Note:** |* The `entity_name` must be lowercase with underscores (snake_case), e.g. `customer_preferences`. No uppercase letters or spaces allowed. |* Each property can optionally include `description` (markdown text), and for string types: `minLength` and `maxLength`. + |* Each property can optionally declare **field-level access control**: `writeRoleRequired`/`readRoleRequired` (booleans — auto-generate a per-field role) or `writeRole`/`readRole` (name an explicit, shareable role). Write-restricted fields are not set via POST/PUT (their existing value is preserved) and are written only via the role-gated PATCH path; read-restricted fields are omitted from GET for callers lacking the read role. |* Set `has_public_access` to `true` to generate read-only public endpoints (GET only, no authentication required) under `/public/`. |* Set `has_community_access` to `true` to generate read-only community endpoints (GET only, authentication required + CanGet role) under `/community/`. Community endpoints return ALL records (personal + non-personal from all users). |* Set `personal_requires_role` to `true` to require the corresponding role (e.g. CanCreateDynamicEntity_, CanGetDynamicEntity_) for `/my/` personal entity endpoints. Default is `false` (any authenticated user can use `/my/` endpoints). diff --git a/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala b/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala index 223611cb56..9395e2b602 100644 --- a/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala +++ b/obp-api/src/main/scala/code/dynamicEntity/DynamicEntityProvider.scala @@ -73,6 +73,37 @@ trait DynamicEntityT { case None => definition } + // ----- Field-level access control (writeRole / readRole) ----- + // A field is write-restricted if it sets writeRoleRequired:true OR a non-empty writeRole. + // A field is read-restricted if it sets readRoleRequired:true OR a non-empty readRole. + private def restrictedFieldNames(requiredFlag: String, roleKey: String): List[String] = { + def isRestricted(propDef: JValue): Boolean = + (propDef \ requiredFlag) == JBool(true) || + ((propDef \ roleKey) match { case JString(s) => s.nonEmpty; case _ => false }) + (definition \ entityName \ "properties") match { + case props: JObject => props.obj.collect { case JField(name, propDef: JObject) if isRestricted(propDef) => name } + case _ => Nil + } + } + + /** Fields whose writing requires a field-level role (not writable via POST/PUT; only via the role-gated PATCH path). */ + lazy val writeRestrictedFields: List[String] = restrictedFieldNames("writeRoleRequired", "writeRole") + /** Fields whose reading requires a field-level role (omitted from GET responses otherwise). */ + lazy val readRestrictedFields: List[String] = restrictedFieldNames("readRoleRequired", "readRole") + + /** Explicit writeRole declared on a field, if any (otherwise an auto-generated role applies). */ + def explicitWriteRole(fieldName: String): Option[String] = + (definition \ entityName \ "properties" \ fieldName \ "writeRole") match { + case JString(s) if s.nonEmpty => Some(s) + case _ => None + } + /** Explicit readRole declared on a field, if any. */ + def explicitReadRole(fieldName: String): Option[String] = + (definition \ entityName \ "properties" \ fieldName \ "readRole") match { + case JString(s) if s.nonEmpty => Some(s) + case _ => None + } + /** * validate the commit json whether fulfil DynamicEntity schema * @param entityJson commit json object to add new instance of given dynamic entity @@ -81,7 +112,9 @@ trait DynamicEntityT { def validateEntityJson(entityJson: JObject, callContext: Option[CallContext]): Future[Option[String]] = { val required: List[String] = (definition \ entityName \ "required").asInstanceOf[JArray].arr.map(_.asInstanceOf[JString].s) - val missingProperties = required diff entityJson.obj.map(_.name) + // Write-restricted fields are never supplied via POST/CREATE (they are written only through the + // role-gated PATCH path), so they must not count as "missing required" for ordinary create/update bodies. + val missingProperties = (required diff writeRestrictedFields) diff entityJson.obj.map(_.name) if(missingProperties.nonEmpty) { return Future.successful( @@ -559,6 +592,24 @@ object DynamicEntityCommons extends Converter[DynamicEntityT, DynamicEntityCommo val JString(descriptionValue) = propertyDescription.asInstanceOf[JString] checkFormat(descriptionValue.nonEmpty , s"$DynamicEntityInstanceValidateFail The property of $fieldName's 'description' field in $entityName must be a not empty string value.") } + + // validate optional field-level access-control keywords (all optional; absence => unrestricted) + val writeRoleRequired = value \ "writeRoleRequired" + if(writeRoleRequired != JNothing) { + checkFormat(writeRoleRequired.isInstanceOf[JBool], s"$DynamicEntityInstanceValidateFail The property of $fieldName's 'writeRoleRequired' field must be boolean.") + } + val readRoleRequired = value \ "readRoleRequired" + if(readRoleRequired != JNothing) { + checkFormat(readRoleRequired.isInstanceOf[JBool], s"$DynamicEntityInstanceValidateFail The property of $fieldName's 'readRoleRequired' field must be boolean.") + } + val writeRole = value \ "writeRole" + if(writeRole != JNothing) { + checkFormat(writeRole.isInstanceOf[JString] && writeRole.asInstanceOf[JString].s.nonEmpty, s"$DynamicEntityInstanceValidateFail The property of $fieldName's 'writeRole' field must be a non-empty string.") + } + val readRole = value \ "readRole" + if(readRole != JNothing) { + checkFormat(readRole.isInstanceOf[JString] && readRole.asInstanceOf[JString].s.nonEmpty, s"$DynamicEntityInstanceValidateFail The property of $fieldName's 'readRole' field must be a non-empty string.") + } }) DynamicEntityCommons(entityName, compactRender(jsonObject), dynamicEntityId, userId, bankId, hasPersonalEntityValue, hasPublicAccessValue, hasCommunityAccessValue, personalRequiresRoleValue) From 32487626bbc6b52b01e2abfd203fb10d41086c3c Mon Sep 17 00:00:00 2001 From: simonredfern Date: Fri, 5 Jun 2026 12:56:19 +0200 Subject: [PATCH 3/4] writeRequired readRequired phases 6-8 and tests --- .../entity/helper/DynamicEntityHelper.scala | 67 ++++++++++++++++++- .../main/scala/code/api/util/Glossary.scala | 14 ++++ .../scala/code/setup/SendServerRequests.scala | 7 ++ .../commons/model/enums/Enumerations.scala | 3 + release_notes.md | 3 + 5 files changed, 93 insertions(+), 1 deletion(-) diff --git a/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala b/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala index d5ba958b4f..ce8910b007 100644 --- a/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala +++ b/obp-api/src/main/scala/code/api/dynamic/entity/helper/DynamicEntityHelper.scala @@ -337,6 +337,40 @@ object DynamicEntityHelper { createdByBankId= dynamicEntityInfo.bankId ) + resourceDocs += (DynamicEntityOperation.PATCH, splitNameWithBankId) -> ResourceDoc( + implementedInApiVersion, + buildPatchFunctionName(bankId, entityName), + "PATCH", + s"$resourceDocUrl/$idNameInUrl", + s"Partially update $splitName", + s"""Partially update $splitName: only the fields supplied in the body are changed; others are preserved. + | + |This is also the write path for **field-level write-restricted** fields (those declared with + |`writeRoleRequired` or an explicit `writeRole`). To write such a field the caller must hold that field's + |write role; otherwise the request is rejected with 403 (missing role). Unrestricted fields require + |the entity update role, as for PUT. + |${dynamicEntityInfo.description} + | + |${dynamicEntityInfo.fieldsDescription} + | + |${methodRoutingExample(entityName)} + | + |${userAuthenticationMessage(true)} + | + |""", + dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExample, + List( + AuthenticatedUserIsRequired, + UserHasMissingRoles, + InvalidJsonFormat, + UnknownError + ), + List(apiTag, apiTagDynamicEntity, apiTagDynamic), + Some(List(dynamicEntityInfo.canUpdateRole)), + createdByBankId= dynamicEntityInfo.bankId + ) + resourceDocs += (DynamicEntityOperation.DELETE, splitNameWithBankId) -> ResourceDoc( implementedInApiVersion, buildDeleteFunctionName(bankId, entityName), @@ -466,6 +500,33 @@ object DynamicEntityHelper { createdByBankId= dynamicEntityInfo.bankId ) + resourceDocs += (DynamicEntityOperation.PATCH, mySplitNameWithBankId) -> ResourceDoc( + implementedInApiVersion, + buildPatchFunctionName(bankId, s"My$entityName"), + "PATCH", + s"$myResourceDocUrl/$idNameInUrl", + s"Partially update My $splitName", + s"""Partially update My $splitName: only the fields supplied in the body are changed; others are preserved. + | + |This is also the write path for **field-level write-restricted** fields; writing such a field requires the + |caller to hold that field's write role. + |${dynamicEntityInfo.description} + | + |${dynamicEntityInfo.fieldsDescription} + | + |${methodRoutingExample(entityName)} + | + |${userAuthenticationMessage(true)} + | + |""", + dynamicEntityInfo.getSingleExampleWithoutId, + dynamicEntityInfo.getSingleExample, + myErrorMessagesWithJson, + List(apiTag, apiTagDynamicEntity, apiTagDynamic), + if(personalRequiresRole) Some(List(dynamicEntityInfo.canUpdateRole)) else Some(List(dynamicEntityInfo.canUpdateRole)), + createdByBankId= dynamicEntityInfo.bankId + ) + resourceDocs += (DynamicEntityOperation.DELETE, mySplitNameWithBankId) -> ResourceDoc( implementedInApiVersion, buildDeleteFunctionName(bankId, s"My$entityName"), @@ -615,6 +676,7 @@ object DynamicEntityHelper { private def buildCreateFunctionName(bankId:Option[String], entityName: String) = s"dynamicEntity_create${entityName}_${bankId.getOrElse("")}" private def buildUpdateFunctionName(bankId:Option[String], entityName: String) = s"dynamicEntity_update${entityName}_${bankId.getOrElse("")}" + private def buildPatchFunctionName(bankId:Option[String], entityName: String) = s"dynamicEntity_patch${entityName}_${bankId.getOrElse("")}" private def buildDeleteFunctionName(bankId:Option[String], entityName: String) = s"dynamicEntity_delete${entityName}_${bankId.getOrElse("")}" private def buildGetOneFunctionName(bankId:Option[String], entityName: String) = s"dynamicEntity_getSingle${entityName}_${bankId.getOrElse("")}" private def buildGetAllFunctionName(bankId:Option[String], entityName: String) = s"dynamicEntity_get${entityName}List_${bankId.getOrElse("")}" @@ -694,13 +756,16 @@ case class DynamicEntityInfo(definition: String, entityName: String, bankId: Opt case _ => false } ) - if(descriptions.nonEmpty) { + val propertyList = if(descriptions.nonEmpty) { descriptions .map(field => s"""* ${field.name}: ${(field.value \ "description").asInstanceOf[JString].s}""") .mkString("**Property List:** \n\n", "\n", "") } else { "" } + val writeNote = if(writeRestrictedFields.nonEmpty) s"\n\n**Write-restricted fields** (set only via PATCH by a holder of the field's write role): ${writeRestrictedFields.mkString(", ")}" else "" + val readNote = if(readRestrictedFields.nonEmpty) s"\n\n**Read-restricted fields** (returned only to callers holding the field's read role): ${readRestrictedFields.mkString(", ")}" else "" + propertyList + writeNote + readNote } def toResponse(result: JObject, id: Option[String]): JObject = { diff --git a/obp-api/src/main/scala/code/api/util/Glossary.scala b/obp-api/src/main/scala/code/api/util/Glossary.scala index 72285089f3..50c392b0cb 100644 --- a/obp-api/src/main/scala/code/api/util/Glossary.scala +++ b/obp-api/src/main/scala/code/api/util/Glossary.scala @@ -3206,6 +3206,20 @@ object Glossary extends MdcLoggable { |* CanGetDynamicEntity_FooBar |* CanDeleteDynamicEntity_FooBar | +|**Field-level write/read permissions (per property):** +| +|Each property in the schema can optionally restrict who may write or read that field, independently of the entity-level roles above: +| +|* `writeRoleRequired` (boolean) or `writeRole` (explicit role name) — the field becomes **write-restricted**: it cannot be set via POST or PUT (its existing value is preserved), only via **PATCH** by a caller holding the field's write role. +|* `readRoleRequired` (boolean) or `readRole` (explicit role name) — the field becomes **read-restricted**: it is omitted from GET responses unless the caller holds the field's read role (public/anonymous access omits it entirely). +| +|Restriction is on if either the boolean is `true` or an explicit role name is given. When a boolean is used, OBP auto-generates the role; e.g. for entity 'FooBar' field 'owner': +| +|* CanWriteDynamicEntityField_FooBar__owner (bank level) / CanWriteDynamicEntityField_SystemFooBar__owner (system level) +|* CanGetDynamicEntityField_FooBar__owner / CanGetDynamicEntityField_SystemFooBar__owner +| +|Naming an explicit `writeRole`/`readRole` lets several fields (even across entities) share a single role — useful for a privileged service (e.g. an indexer) that maintains many fields. Typical use: a field written only by a verifier/service or projected from an external system, but read by ordinary consumers. +| |**Management endpoints:** | |* POST /management/system-dynamic-entities - Create system level entity diff --git a/obp-api/src/test/scala/code/setup/SendServerRequests.scala b/obp-api/src/test/scala/code/setup/SendServerRequests.scala index 3f4ca4846d..05f5835f44 100644 --- a/obp-api/src/test/scala/code/setup/SendServerRequests.scala +++ b/obp-api/src/test/scala/code/setup/SendServerRequests.scala @@ -201,6 +201,13 @@ trait SendServerRequests { getAPIResponse(jsonReq) } + def makePatchRequest(req: Req, json: String, headers: (String, String) *) : APIResponse = { + val extra_headers = Map("Content-Type" -> "application/json") ++ headers.toMap + val reqData = extractParamsAndHeaders(req.PATCH, json, "UTF-8", extra_headers) + val jsonReq = createRequest(reqData) + getAPIResponse(jsonReq) + } + def makePutRequestAsync(req: Req, json: String = ""): Future[APIResponse] = { val extra_headers = Map("Content-Type" -> "application/json") val reqData = extractParamsAndHeaders(req.PUT, json, "UTF-8", extra_headers) diff --git a/obp-commons/src/main/scala/com/openbankproject/commons/model/enums/Enumerations.scala b/obp-commons/src/main/scala/com/openbankproject/commons/model/enums/Enumerations.scala index 129a4c6338..08677166b7 100644 --- a/obp-commons/src/main/scala/com/openbankproject/commons/model/enums/Enumerations.scala +++ b/obp-commons/src/main/scala/com/openbankproject/commons/model/enums/Enumerations.scala @@ -284,6 +284,9 @@ object DynamicEntityOperation extends OBPEnumeration[DynamicEntityOperation] { object CREATE extends Value object UPDATE extends Value object DELETE extends Value + // PATCH is used only as a resource-doc key for the field-level partial-update endpoint; + // it is never sent to connectors (PATCH requests are served via UPDATE internally). + object PATCH extends Value } sealed trait ContentParam extends EnumValue diff --git a/release_notes.md b/release_notes.md index 40ac58476d..881ebb3361 100644 --- a/release_notes.md +++ b/release_notes.md @@ -3,6 +3,9 @@ ### Most recent changes at top of file ``` Date Commit Action +05/06/2026 TBD FEATURE: Dynamic Entity field-level write/read role permissions + (per-property writeRole/readRole + PATCH write path; read-restricted + fields omitted from GET). See Glossary "Dynamic-Entities". 26/05/2026 TBD BEHAVIOUR RESTORE: api_disabled_versions / api_enabled_versions once again retire only the URL prefix, not the underlying endpoints on newer prefixes — restoring the pre-http4s-migration cascade From 939f12df16bedc6eaa459f6e9eedbf7fdd9edf5b Mon Sep 17 00:00:00 2001 From: simonredfern Date: Fri, 5 Jun 2026 13:01:17 +0200 Subject: [PATCH 4/4] Create lift_mapper_in_the_future.md --- ideas/lift_mapper_in_the_future.md | 513 +++++++++++++++++++++++++++++ 1 file changed, 513 insertions(+) create mode 100644 ideas/lift_mapper_in_the_future.md diff --git a/ideas/lift_mapper_in_the_future.md b/ideas/lift_mapper_in_the_future.md new file mode 100644 index 0000000000..c68b1cb589 --- /dev/null +++ b/ideas/lift_mapper_in_the_future.md @@ -0,0 +1,513 @@ +# Alternatives to Lift Mapper + +## The problem is maintenance, not capability + +Lift Mapper is a capable, battle-tested ORM. After a decade and ~130 entities it does what OBP needs and does it well: typed fields, a clean query DSL (`By` / `OrderBy` / `findAll` / `count`), automatic DDL via `Schemifier`, per-field validation, and lifecycle hooks (`beforeSave` / `CreatedUpdated`). **The API is not the problem, and nobody should pretend it is.** + +The problem is upstream cadence. Lift is sparsely maintained and effectively frozen — OBP is pinned to `lift-mapper 3.5.0` — and that staleness has two concrete downstream costs: + +1. **Stale transitive dependencies we cannot drop** — most visibly the old `lift-webkit` web framework, which OBP no longer uses a line of but still ships (see the next section). +2. **A latent Scala 3 blocker** — Mapper's implicit-heavy `MappedField` machinery is non-trivial to port (see "The forcing function isn't here yet"). + +This document is the menu of responses to that staleness, cheapest to heaviest: **keep Mapper and maintain it ourselves** (Options A–D), **narrower or orthogonal moves** that target just the stale dependency or the schema layer rather than the whole ORM (Options E–I), or **exit it entirely** to Doobie or another ORM (the migration playbook that makes up the bulk of this file). It does **not** assume migration is the answer — read the Conclusion first. Today's recommendation is *design around the constraint*, not *migrate*. + +--- + +## Dependency-surface coupling: `lift-webkit` is hostage to Mapper (verified 2026-06-05) + +This is the clearest concrete symptom of the staleness problem. The Lift Web teardown (PR #2828) removed every `net.liftweb.http` import from OBP source (128 files → 0) and deleted `Http4sLiftWebBridge`. That shrank the *reachable* attack surface, but it did **not** shrink the dependency/CVE-scan surface: `lift-webkit_2.12-3.5.0.jar` (the jar that *contains* `net.liftweb.http`) is still on the classpath, pulled in transitively: + +``` +obp-api → lift-mapper → lift-db → lift-webkit (3.5.0) +obp-api → lift-mapper → lift-proto → lift-webkit (duplicate path) +``` + +**Tested the cheap escape hatch and it does not work.** Adding an `` for `lift-webkit` on the `lift-mapper` dependency cleanly removes the jar from `dependency:tree`, but compilation then fails — **Lift Mapper's own public API is welded to lift-webkit types**, so the compiler needs them just to typecheck Mapper classes OBP already uses: + +| Mapper symbol | lift-webkit type it requires | OBP site that fails to compile | +|---|---|---| +| `net.liftweb.mapper.MapperRules` | extends `net.liftweb.http.Factory` | `bootstrap/liftweb/Boot.scala:263` | +| `BaseMappedField.asJsExp` | returns `net.liftweb.http.js.JsExp` | `code/group/Group.scala:111` (every `MappedField` subclass) | +| `MappedPassword.asJsExp` | returns `net.liftweb.http.js.JsExp` | `code/model/dataAccess/AuthUser.scala:387` | + +This is a compile-time coupling, not a reflective/runtime one — there is no surgical exclusion that survives. **`lift-webkit` cannot leave the classpath until `lift-mapper` itself leaves**, i.e. until the data-access migration below completes *and* the Schemifier/`ToSchemify` schema layer is also off Mapper (the Mapper class declarations are the last thing to go — see "What 'done' looks like" + the schema-layer follow-on). + +**Implication for prioritisation.** This is a second, independent forcing function alongside Scala 3 (§"The forcing function isn't here yet"): dropping an old, sparsely-maintained web framework (`lift-webkit 3.5.0`) from the dependency manifest and from CVE-scanner output is only achievable via the full Mapper exit. Worth weighing if/when a security-driven dependency-reduction goal appears — but on its own it does not change the recommendation below, since `lift-webkit` carries no currently-flagged CVE (absent from `dependency-check-findings-2026-05-14.md`); the value is latent-surface and supply-chain hygiene, not an open finding. + +--- + +# Option: full exit to Doobie + +The rest of this document is the detailed playbook for one response to the staleness problem — replacing Mapper as the *data-access* layer with Doobie. It is the heaviest option on the menu; weigh it against "keep and maintain" (Options A–D) and the Conclusion before committing. + +## Principle + +Eliminate Lift Mapper as a data-access layer. All CRUD, queries, and reads/writes move to Doobie. Lift Mapper stays only for what is explicitly out of scope (see below) until a separate workstream removes it. + +This is the data-access counterpart to `LIFT_HTTP4S_MIGRATION.md`. The two migrations are independent — an http4s endpoint can call Doobie or Mapper, and a Lift endpoint can call either — but the end state is **no `net.liftweb.mapper.*` import outside the schema/migration layer**. + +API version numbers are unaffected: framework migrations happen in-place. A Mapper → Doobie swap inside `MappedFooProvider` does not justify a version bump unless the response shape changes. + +--- + +## Scope + +**In scope — migrate to Doobie** + +- All CRUD: `findAll`, `find(By(...))`, `count(By(...))`, `bulkDelete_!!`, `delete_!`, `saveMe`, `create.foo(...).save`, etc. +- All raw-SQL queries currently using `DB.runQuery`, `DBUtil.runQuery`, `DB.use(...) { conn => ... }` for application logic. +- All `Future { tryo { Foo.findAll(...) } }` Provider methods. +- Bulk-load patterns that today do N+1 Mapper lookups (these become single JOINs in Doobie — see `DoobieUserQueries.scala` for the canonical example). +- Connector implementations that read/write Mapper entities directly (`LocalMappedConnector` and subclasses). + +**Out of scope — stays on Lift** + +- `Schemifier.schemify(...)` — table creation, column add/drop, index sync. Stays in `Boot.scala` and `MockedRabbitMqAdapter.scala`. +- `ToSchemify.models` — the canonical entity list driving Schemifier. Stays as-is. +- Per-table schema-mutation migrations in `code/api/util/migration/MigrationOf*.scala` that use `DB.use { conn => Schemifier.infoF }` to add columns / drop indexes. These are schema operations, not data access. +- `MappedUUID`, `MappedString`, `MappedLong`, and other Mapper field types referenced by entity definitions retained for Schemifier. + +The Mapper case classes themselves (`class Foo extends LongKeyedMapper[Foo]`) stay until **both** (a) data access has fully moved to Doobie and (b) Schemifier is replaced. This migration only removes the **runtime use** of Mapper as a query/CRUD API — the class definitions remain as schema descriptors consumed by `Schemifier`. + +--- + +## Current State (2026-05-18) + +| Area | State | +|---|---| +| Doobie dependency | Present in `obp-api/pom.xml` (`doobie-core`, `doobie-hikari`) | +| Transactor | `code.api.util.DoobieUtil` — shares Lift's HikariCP pool and unifies with Lift's per-request transaction (`Transactor.fromConnection` + `Strategy.void`) | +| Doobie call sites | ~10 files: `DoobieMetricsQueries`, `DoobieInvestigationQueries`, `DoobieUserQueries`, `DoobieConsentQueries`, `DoobieChatMessageQueries`, `DoobieAccountAccessViewQueries`, `DoobieQueries`, `MetricBatchWriter`, `ConnectorMetricBatchWriter`, `StatusPage` | +| Mapper entities in `ToSchemify.models` | ~130 | +| `Mapped*Provider.scala` files (Mapper-backed providers) | ~99 | +| `extends LongKeyedMapper` declarations | ~24 source files, 27 unique classes | +| Files importing `net.liftweb.mapper` | ~250 | +| Files using `DB.use { conn => ... }` outside Schemifier-style migrations | ~10 | + +Doobie is already the chosen target — this migration scales the existing pattern, it does not introduce a new technology. + +--- + +## Target Architecture + +``` +┌────────────────────────────────────────────────────────────┐ +│ Application code (endpoints, connectors, services) │ +│ │ +│ ▼ calls Provider trait method │ +│ │ +│ FooProvider trait (no change to signature) │ +│ │ +│ ▼ implementation │ +│ │ +│ DoobieFooProvider │ +│ uses DoobieFooQueries (ConnectionIO[A] definitions) │ +│ uses DoobieUtil.runQuery / runQueryAsync │ +│ │ +│ ▼ JDBC │ +│ │ +│ HikariCP pool (single, shared) ──── PostgreSQL/SQL Server │ +└────────────────────────────────────────────────────────────┘ + +Schemifier (Lift Mapper) reads ToSchemify.models on boot and +ensures tables/columns/indexes exist. It never serves a runtime +request after boot completes. +``` + +**Two key invariants** + +1. **Connection unification preserved.** A Doobie call from inside a Lift-served request (during the bridge phase) must use the same `Connection` Lift is already holding for that request — otherwise a write made through Mapper earlier in the request is invisible to the Doobie read that follows. `DoobieUtil.runQuery` already handles this via `liftCurrentConnection` peek. Do not bypass it. +2. **Provider traits do not change.** The `XxxProvider` trait in the domain layer is the seam. `MappedFooProvider` and `DoobieFooProvider` both implement the same trait. Swapping the binding is a one-line change in `RemotedataActors` / `ToSchemify`'s vendor wiring — no callers change. This means migration can happen one provider at a time without coordinated rewrites. + +--- + +## Per-Entity Migration Playbook + +For each Mapper-backed entity `Foo` with provider `MappedFooProvider`: + +### Step 1 — Inventory the Provider trait + +Locate the trait (usually `FooProvider` in the same package). List every method's signature. Note which methods return `Future[Box[List[A]]]` vs `Box[A]` vs `Future[A]` — Doobie returns must match exactly. + +### Step 2 — Map column names + +Mapper uses Scala field names; the DB columns are derived via Lift's naming convention (lowercased, no underscore for camelCase by default, but per-column overrides exist via `dbColumnName` / `MappedField` overrides). Get the actual column names by either: + +- Reading the Mapper class and checking each field for `override def dbColumnName`. +- Running `psql \d ` against a populated dev DB (most reliable). + +Record these in a comment block at the top of the new `DoobieFooQueries.scala` so future readers don't repeat the lookup. + +### Step 3 — Write `DoobieFooQueries.scala` + +Pattern (see `obp-api/src/main/scala/code/users/DoobieUserQueries.scala`): + +```scala +package code.foo + +import code.api.util.DoobieUtil +import doobie._ +import doobie.implicits._ +import doobie.implicits.javasql._ + +object DoobieFooQueries { + + case class FooRow( + fooId: String, + name: Option[String], + createdAt: java.sql.Timestamp + ) + + def findByFooId(fooId: String): ConnectionIO[Option[FooRow]] = + sql"""SELECT foo_id, name, created_at + FROM foo + WHERE foo_id = $fooId""".query[FooRow].option + + def findAllByOwner(ownerId: String): ConnectionIO[List[FooRow]] = + sql"""SELECT foo_id, name, created_at + FROM foo + WHERE owner_id = $ownerId + ORDER BY created_at DESC""".query[FooRow].to[List] + + def insert(row: FooRow): ConnectionIO[Int] = + sql"""INSERT INTO foo (foo_id, name, created_at) + VALUES (${row.fooId}, ${row.name}, ${row.createdAt})""".update.run + + def deleteByFooId(fooId: String): ConnectionIO[Int] = + sql"""DELETE FROM foo WHERE foo_id = $fooId""".update.run +} +``` + +Keep query objects pure: `ConnectionIO[A]` only, no `unsafeRunSync`, no `Future`. Composition (joins, transactions) happens by combining `ConnectionIO`s with `flatMap`. Execution is the caller's job. + +### Step 4 — Write `DoobieFooProvider.scala` + +```scala +package code.foo + +import code.api.util.DoobieUtil +import com.openbankproject.commons.ExecutionContext.Implicits.global +import net.liftweb.common.{Box, Failure, Full} +import scala.concurrent.Future + +object DoobieFooProvider extends FooProvider { + + override def getFoo(fooId: String): Future[Box[Foo]] = Future { + try Box(DoobieUtil.runQuery(DoobieFooQueries.findByFooId(fooId)).map(rowToFoo)) + catch { case t: Throwable => Failure(t.getMessage, Full(t), Empty) } + } + + override def getAllFooForOwner(ownerId: String): Future[Box[List[Foo]]] = Future { + try Full(DoobieUtil.runQuery(DoobieFooQueries.findAllByOwner(ownerId)).map(rowToFoo)) + catch { case t: Throwable => Failure(t.getMessage, Full(t), Empty) } + } + + private def rowToFoo(r: DoobieFooQueries.FooRow): Foo = ??? +} +``` + +Match the existing `MappedFooProvider`'s error shape exactly. If the old provider returned `tryo(...)` → `Empty` on null / `Failure` on exception, the Doobie provider must do the same. Tests that match on `Box` variants will catch you. + +### Step 5 — Switch the binding + +Find the line that picks the implementation. There are two patterns in the codebase: + +- **`vend` constant in the provider trait companion object** — `object FooProvider { val foo: FooProvider = MappedFooProvider }`. Change the assignment. +- **Akka-remoted wiring** (`RemotedataActors`) — actor selects based on `props.RemoteDatabaseEnabled`. Change the local-side binding. + +Leave `MappedFooProvider` in the codebase for one release as a fallback option — gate the switch on a prop if the change is risky: `getPropsAsBoolValue("provider.foo.doobie", true)`. Remove the prop and the Mapper provider in the next clean-up PR. + +### Step 6 — Tests + +A Doobie provider must pass the same provider-level tests as the Mapper one. The integration tests (server + DB) for endpoints calling this provider must continue to pass unchanged — that's the contract. + +If the only existing coverage is integration tests, add a focused suite for the new provider before deleting the Mapper one. The Doobie-backed providers already in the tree have unit-test counterparts (`DoobieUserQueriesTest`, etc.) — copy the structure. + +### Step 7 — Removal (final pass per entity) + +Once the Doobie provider has been the default for one full release with no rollbacks: + +- Delete `MappedFooProvider.scala`. +- Remove the prop gate. +- **Keep** the `Foo` Mapper class in its file — `ToSchemify.models` still references it. The class becomes a pure schema descriptor; no application code calls it. + +--- + +## Cross-Cutting Concerns + +### Transaction unification + +`DoobieUtil.runQuery` peeks `DB.currentConnection` and, when a Lift request is in flight, runs the Doobie query on the same `Connection`. This is non-negotiable while the bridge serves any traffic: a request that does `MappedFoo.save` followed by a Doobie read of the same row would otherwise see a stale value (the Mapper write is still in Lift's per-request transaction; the Doobie read on a different pool connection wouldn't see it). + +Migration order must respect this: a Provider that *writes* moves to Doobie at the same time as its *readers*, not before. Otherwise mid-request you have Doobie writes invisible to subsequent Mapper reads on a different connection. + +Background tasks and schedulers do not have a Lift request context — `DoobieUtil` falls back to the shared pool. No special handling needed. + +### `RequestScopeConnection` interaction + +`code.api.util.http4s.RequestScopeConnection` already holds a single `Connection` for the lifetime of an http4s request and exposes it to `DB.use` calls made from `Future`s spawned during that request (see `RequestScopeConnection.scala` for the full lifecycle). Doobie picks the same connection via `DB.currentConnection` because `RequestScopeConnection` installs it in the `DynoVar`. No additional plumbing required. + +### Boxes, Failures, Empties + +Mapper providers return `Box[A]` ubiquitously (`Full`, `Empty`, `Failure`). Doobie returns `A`, `Option[A]`, `List[A]`. The conversion convention used in the existing Doobie providers: + +| Doobie | Box equivalent | +|---|---| +| `query.option` → `Option[A]` | `Box(opt)` (`Full`/`Empty`) | +| `query.to[List]` → `List[A]` | `Full(list)` (empty list still `Full(Nil)`) | +| `query.unique` → `A`, throws if 0 or >1 | wrap in `try { Full(x) } catch { ... Failure }` | +| exception thrown by JDBC | `Failure(msg, Full(t), Empty)` | + +Mirror the existing `MappedFooProvider`'s exact `Box` shapes per method — tests on `Box.isDefined`, `.openOrThrow`, pattern matches on `Failure(...)` will break otherwise. + +### N+1 elimination + +This is the primary *secondary* benefit of the migration. Mapper makes the obvious code path a separate query per row. Doobie makes a single JOIN the obvious code path. When migrating a provider that already does N+1, fold the dependent reads into the JOIN — `DoobieUserQueries.UserSearchRow` is the canonical example (single SELECT replaces what was 3 round-trips per user). + +Do this opportunistically per entity; don't gate the migration on rewriting every query, but flag in the per-entity PR which round-trips were collapsed. + +### Sort/filter parameter validation + +User-supplied sort columns can never be string-spliced into SQL. The existing Doobie providers handle this with a per-endpoint `Map[String, String]` whitelist (see `DoobieUserQueries.SortableColumns`) and use `Fragment.const` only on whitelisted values. Apply the same pattern everywhere a Mapper `OrderBy(MyTable.someField, Descending)` is replaced — never construct an `ORDER BY` clause from raw request input. + +### SQL Server compatibility + +Some entities run on SQL Server (NVARCHAR / type -9). Doobie handles JDBC types correctly out of the box — this is part of the reason for the migration (`DBUtil.runQuery` had bugs here). When migrating, if the existing code had SQL-Server branching (`if (DBUtil.isSqlServer) ... else ...`), audit whether Doobie removes the need. Often it does; sometimes (`TOP` vs `LIMIT`) it doesn't and the branch stays. + +--- + +## Migration Order (recommended) + +Sort the ~130 entities by **blast radius × write-hotness**, smallest first. Suggested phasing: + +### Phase 1 — Stand-alone read-mostly entities (low risk, fast feedback) + +Entities whose providers have no callers outside their own package and are read-dominated. Good rehearsal for the team and exercises the playbook with low blast radius. + +Candidates (cross-check current usage before starting): + +- `WebUiProps`, `FeaturedApiCollection`, `EndpointTag`, `MigrationScriptLog`, `MappedSocialMedia`, `MappedFXRate`, `MappedCurrency`, `MappedETag`. + +### Phase 2 — Provider-shaped entities with clean trait seams (medium risk) + +Entities accessed exclusively through a `FooProvider` trait. The trait is the seam — swap the implementation without touching callers. + +Candidates: `MappedCustomerAddress`, `MappedCustomerAttribute`, `MappedCustomerDependant`, `MappedUserAttribute`, `MappedAccountApplication`, `MappedProductAttribute`, `MappedKycCheck`/`Document`/`Media`/`Status`, `MappedMeeting`/`Invitee`, `MappedAccountWebhook`, `RoutingScheme`/`BankSupportedRoutingScheme`, `BankAttribute`, `MappedTransactionType`, `RateLimiting`, `EndpointMapping`, `MethodRouting`. + +### Phase 3 — Core domain (high risk; needs feature flag + extra test coverage) + +Entities at the centre of the system. Each one is a multi-PR effort: write Doobie queries, write Doobie provider, ship behind a prop, soak for one release, remove Mapper provider. + +Candidates: `ResourceUser`, `AuthUser`, `AccountAccess`, `ViewDefinition`, `ViewPermission`, `MapperAccountHolders`, `MappedBank`, `MappedBankAccount`, `BankAccountRouting`, `MappedTransaction`, `MappedTransactionRequest`, `TransactionRequestAttribute`, `MappedCounterparty`/`Bespoke`/`Metadata`/`WhereTag`, `MappedCustomer`, `MappedUserCustomerLink`, `Consumer`, `MappedConsent`, `ConsentItem`, `ConsentRequest`, `MappedEntitlement`, `MappedEntitlementRequest`, `MappedScope`/`UserScope`, `DirectDebit`, `StandingOrder`. + +### Phase 4 — Niche / connector-internal (do alongside connector migration) + +Entities only touched by specific connectors or rarely-used flows. Best done as part of the related feature work, not as a standalone push. + +Candidates: `BulkPayment`, `BulkBatchReference`, `PinReset`, `Nonce`, `Token`, `OpenIDConnectToken`, `MappedBadLoginAttempt`, `UserLocks`, `JobScheduler`, `MappedSigningBasket`/`Payment`/`Consent`, `MappedRegulatedEntity`, `RegulatedEntityAttribute`, `AbacRule`, `Mandate`/`Provision`/`SignatoryPanel`, `code.chat.ChatRoom`/`Participant`/`ChatMessage`/`Reaction`, `Group`, `Organisation`, `PayeeLookup`, `AccountAccessRequest`, `UserInvitation`, `UserAgreement`, `UserInitAction`, `ConnectorMethod`, `ConnectorTrace`, `MappedConnectorMetric`, `MappedMetric`, `MetricArchive`, `DynamicEntity`/`Data`/`Endpoint`/`ResourceDoc`/`MessageDoc`, `JsonSchemaValidation`, `AuthenticationTypeValidation`, `CounterpartyLimit`, `MappedExpectedChallengeAnswer`, `MappedTaxResidence`, `MappedUserAuthContext`/`Update`, `MappedConsentAuthContext`, `MappedUserRefreshes`, `MappedCustomerMessage`, `MappedCustomerIdMapping`, `MappedAccountAttribute`, `MappedTransactionAttribute`, `MappedCardAttribute`, `MappedPhysicalCard`, `CardAction`, `AtmAttribute`, `MappedAtm`, `MappedBranch`, `MappedProduct`, `ProductFee`, `ProductTag`, `MappedProductCollection`/`Item`, `BankAccountBalance`, `BankAccountNotificationWebhook`, `SystemAccountNotificationWebhook`, `DoubleEntryBookTransaction`, `TransactionRequestReasons`, `MappedTransactionRequestTypeCharge`, `MappedCrmEvent`, `AttributeDefinition`, `CustomerAccountLink`, `CustomerLink`, `TransactionIdMapping`, `AccountIdMapping`, `ApiCollection`/`Endpoint`, `ApiProduct`/`Attribute`, `MappedComment`, `MappedTag`, `MappedWhereTag`, `MappedTransactionImage`, `MappedNarrative`, `MappedBankAccountData`. + +### Phase 5 — Connector internals (`LocalMappedConnector`) + +`LocalMappedConnector` and its subclasses are the heaviest Mapper consumers. Each connector method is independently migratable. Recommend doing this as a *separate* workstream after Phase 3 lands, because: + +- Connector traits are shared with remote connectors (Kafka, gRPC, REST) — the *signature* must not change, only the local-implementation body. +- Many connector methods are already-tested in `LocalMappedConnectorTest` — strong safety net. +- Volume: hundreds of methods. A per-method PR cadence is realistic. + +--- + +## Risks and Gotchas + +### Risk 1 — Silent column-name drift + +Mapper derives column names from field names with non-obvious rules (mixed-case fields, `dbColumnName` overrides, table-prefix conventions). A Doobie query with the wrong column name compiles but throws at runtime. **Always verify against the live DB with `\d tablename`**, not against the Mapper class declaration. + +### Risk 2 — Mixed Mapper + Doobie writes in the same request + +If a request writes via Mapper, then reads via Doobie on a *different* connection, the read misses the write. `DoobieUtil` defends against this for Lift requests via `liftCurrentConnection`. But: only the *synchronous* `runQuery` does. `runQueryAsync` and `runQueryIO` always use the fallback pool — they cannot see in-flight Lift writes. Migrate the writer and the reader in the same PR. + +### Risk 3 — `Box` shape regressions + +Tests assert on `Failure.msg` strings, `Box.isDefined` after `Empty`/`Full` discrimination, and `openOrThrow` behaviour. The Doobie provider's `Box` shape must match the Mapper provider's exactly. When in doubt, mirror Mapper's behaviour: `tryo { ... }` → `try ... catch { Failure(t.getMessage, Full(t), Empty) }`. + +### Risk 4 — Lift's `MappedField` side-effects on write + +Mapper's `saveMe` invokes per-field validation callbacks, `beforeSave`/`afterSave` hooks, dirty-tracking, and `CreatedUpdated` automatic timestamps. None of this happens through Doobie. For each entity, audit the Mapper class for: + +- `override def dbDisplay_?` (cosmetic, ignore) +- Trait composition: `with CreatedUpdated` → handle `created_at` / `updated_at` explicitly in Doobie inserts/updates +- `beforeSave` / `afterSave` overrides — must be re-implemented in the Doobie provider, often as `flatMap` chains +- `MappedField` with a `validate` override — must be re-implemented as a pre-insert check in the provider + +### Risk 5 — Removed Mapper class breaking schema + +`ToSchemify.models` references the *companion object* of each Mapper class. Deleting the class breaks Schemifier and therefore boot. Never delete the Mapper class as part of the data-access migration — only the **Provider** that wraps it. Schema removal is a separate Phase (eventually replacing Schemifier itself with Flyway or similar). + +### Risk 6 — Connector trait surface + +Many connector methods take Mapper case-class instances as arguments (`def saveTransaction(t: MappedTransaction): ...`). Migrating the *body* to Doobie is straightforward; migrating the *signature* away from `MappedTransaction` requires changing every caller and every remote-connector implementation. Don't conflate the two. Step 1: replace the implementation's body. Step 2 (separate PR, separate decision): introduce a non-Mapper case class as the trait parameter and convert at the boundary. + +--- + +## Per-Entity Tracker + +Add a row per entity as you go. Status: `mapper` (untouched) → `dual` (Doobie provider exists, prop-gated) → `doobie` (Doobie default, Mapper provider deleted) → `schema-only` (Mapper class is now only a Schemifier descriptor, no runtime use). + +| Entity | Provider trait | Status | Last touched | Notes | +|---|---|---|---|---| +| MappedMetric | MetricsProvider | dual (partial) | — | `DoobieMetricsQueries` exists for hot read paths; writes still Mapper | +| MappedConnectorMetric | ConnectorMetricsProvider | dual (partial) | — | `ConnectorMetricBatchWriter` uses Doobie for batched writes | +| ResourceUser (search path only) | UsersProvider | dual (partial) | — | `DoobieUserQueries.getUsers` JOINs ResourceUser + AuthUser + MappedBadLoginAttempt | +| MappedConsent | Consents | dual (partial) | — | `DoobieConsentQueries` covers some lookups | +| ChatMessage | — | dual (partial) | — | `DoobieChatMessageQueries` | +| AccountAccess + ViewDefinition | — | dual (partial) | — | `DoobieAccountAccessViewQueries` (account-listing hot path) | +| _all other ~125 entities_ | various | mapper | — | not started | + +Fill in as PRs land. Mirror the format of `LIFT_HTTP4S_MIGRATION.md`'s tracker. + +--- + +## Alternative: Keep Mapper, Maintain It Ourselves + +This whole migration assumes Mapper is going away. The opposite stance — *keep Mapper indefinitely, reach for Doobie only where it pays off* — is also defensible. If chosen, the question becomes "who maintains Mapper?" since Lift upstream is sparse. + +Four shapes for self-maintenance, ordered cheapest to heaviest: + +### Option A — Upstream patches + +Submit fixes to `lift/framework`. Maintainers are responsive enough for discrete bugs with tests; you don't own a fork. + +**Works when:** specific bugs, no urgency, change isn't OBP-specific. +**Doesn't work when:** you need a fix shipped this week, or the change is too OBP-specific (e.g. "we want a different transaction model"). + +Try this first for any Mapper bug before considering anything heavier. + +### Option B — Vendor the source into OBP + +Copy `net.liftweb.mapper.*` (plus its hard deps from `lift-db`, `lift-util`, `lift-common`, `lift-json`) into `code/vendor/mapper/`, rename the package, drop the Lift Mapper dependency. ~20–30k lines total, most of it stable. + +**Cost:** ~2 weeks one-time port; low ongoing while nothing breaks; spikes when you need non-trivial changes. + +**Hidden risk:** the vendored code looks like every other Scala file in the repo, and reviewers will start "improving" it. Mapper has subtle invariants (field-dirty-tracking, lazy column-name derivation, implicit conversions in the query DSL). A well-intentioned refactor breaks Schemifier in ways tests don't catch. Mitigate with a banner comment in every vendored file: *"Vendored from Lift X.Y.Z. Do not refactor without understanding the originals."* + +**Works when:** you've decided Mapper is good enough for years and want to escape upstream's release cadence without taking on infrastructure. + +### Option C — Fork the framework + +Fork `lift/framework`, publish as `org.openbankproject:obp-mapper` to your own Maven repo. Strip the bits you don't use. + +**Extra cost over Option B:** publish pipeline, version scheme, build infrastructure. +**Extra benefit:** the option to share with other OBP repos or position as a community successor to Lift Mapper — neither materialises automatically. + +**Works when:** you genuinely intend the fork to be a community successor. Otherwise Option B gives you the same control with less infrastructure cost. + +### Option D — In-house Mapper-compatible rewrite + +Write just enough of a Mapper-shaped DSL to cover what OBP uses: field types, `By`/`ByList`/`OrderBy`, `findAll`/`find`/`count`/`saveMe`/`delete_!`, a Schemifier replacement. ~2–3k lines if disciplined. + +**The trap:** behavioural compatibility with Schemifier's DDL output is the hard part — column-name derivation rules, index naming conventions, `CreatedUpdated` timestamp behaviour, validation hook ordering. Drift means your DDL diverges and existing prod databases think they need migrations they don't. "Passes tests on a fresh DB" takes a month; "drops cleanly into existing prod with zero schema drift" takes much longer. + +**Works when:** you're already replacing Schemifier (i.e. doing the full Mapper exit anyway). Otherwise the DDL-compatibility constraint kills it. + +### The question hiding underneath + +"Can we maintain Mapper?" is downstream of "what's actually wrong with Mapper today?" Three possible answers: + +1. **Specific bugs that bite us** → Option A (upstream), Option B (vendor) as fallback. +2. **Scala 3 migration.** Mapper's implicit-heavy DSL and `MappedField` machinery is non-trivial to port. Forking/vendoring delays this but doesn't solve it — whoever does the Scala 3 port does the same work regardless. If OBP commits to Scala 3 within a few years, the Mapper-exit path (full Doobie + Schemifier replacement) becomes cheaper than porting Mapper. +3. **Soft cost: nobody knows Lift.** Real, but owning the source doesn't change the learning curve. This is a docs / training problem, not a Mapper problem. + +### Recommendation + +If we decide *not* to do the full migration in this document: + +1. **Keep Mapper.** It works, it's not the current bottleneck. +2. **Adopt an explicit upstream-first policy** for Mapper bugs we hit. Costs nothing, often gets fixes through. +3. **Reserve vendoring (Option B) as a documented fallback.** If upstream stops responding or refuses a fix we need, we have a pre-decided escape hatch — not a panic. +4. **Treat the Scala 3 question as the real strategic input.** Until OBP commits to a Scala 3 timeline, "maintain Mapper" is a low-cost holding pattern. Once it commits, revisit this whole document — the answer probably becomes "yes, do the full migration, because porting Mapper to Scala 3 costs more than leaving it behind." + +Forking / vendoring is genuinely feasible (~2 weeks for vendoring, low ongoing cost). But it's a solution to *"Lift might disappear"* or *"we need a fix upstream won't take"*, neither of which is the current problem. The current problem is the absence of a long-term plan — and forking doesn't supply one. + +--- + +## Other possibilities (narrower or orthogonal moves) + +Options A–D and the full Doobie exit are the "all of Mapper" answers. Several smaller moves attack just one facet of the problem — usually the stale `lift-webkit` dependency or the schema layer — and several are *not* mutually exclusive with each other or with the bigger options. + +### Option E — Minimal `lift-webkit` shim + +The exclusion experiment at the top of this file proved that Mapper's *compile-time* API touches exactly two webkit families: `net.liftweb.http.Factory` (superclass of `MapperRules`) and `net.liftweb.http.js.JsExp` (return type of `asJsExp`). Instead of *removing* lift-webkit, exclude it and supply a tiny in-house package providing just those types — enough to satisfy the compiler and the classloader — while OBP never calls the methods that use them (it has no `CRUDify` / `toForm` / `SHtml` usage). + +**The catch:** the real surface is the *transitive closure* of `Factory` + `JsExp`, not two classes. `Factory` is part of Lift's injector framework and `JsExp` sits atop the `JsExp`/`JsCmd` hierarchy; both drag in more types. You'd find the true set empirically — exclude, compile, add the next missing symbol, repeat to green — then run the **full** test suite to confirm nothing reflectively hits a stubbed method at runtime (`asJsExp`, MapperRules' JS hooks). If the closure stays small (≈a dozen interface-only types) this kills the dependency-surface problem for a few days' work without touching data access. If it balloons, fall back to Option B (vendor all of Mapper) or accept Option F. + +**Works when:** the *only* goal is getting lift-webkit out of the manifest / CVE scanner, and the data-access migration isn't otherwise justified. + +### Option F — Suppress and document (no code change) + +Accept that lift-webkit is on the classpath but unreachable from OBP code, and just stop it being noise. Add a `dependency-check` suppression scoped to `lift-webkit` with a written justification ("transitive via lift-mapper; `net.liftweb.http` unused in OBP source since PR #2828; unreachable"), and/or a CVE allowlist entry if one is ever filed. Zero code change, zero risk; the residual is honest and recorded. + +This is the right **baseline** until one of the other options is chosen — it is not mutually exclusive with any of them. + +**Works when:** there's no security-driven mandate yet and you just want the scanner quiet and the situation on the record. + +### Option G — Exit to a different ORM than Doobie + +Doobie is the incumbent target only because OBP already uses it — it is not the only exit. If the data-access decision is ever reopened: **Slick** (functional-relational, mature, Scala 3-ready), **Quill** (compile-time SQL, macro-heavy), **ScalaSql** / **Magnum** (newer, lighter, Scala 3-native). The migration *shape* — Provider-trait seam, per-entity swap, Schemifier kept until last — is identical regardless of target; only Steps 3–4 of the playbook change. Listed so a future decision isn't anchored to Doobie purely by omission. + +**Works when:** a fresh evaluation is genuinely on the table (e.g. the Scala 3 forcing function arrives) rather than Doobie being assumed. + +### Option H — Replace only Schemifier, keep Mapper for queries + +The two jobs Mapper does — the query/CRUD DSL and DDL generation (`Schemifier`) — are separable. Swapping **Schemifier** for Flyway / Liquibase (explicit SQL migrations) removes the one thing that makes the Mapper class declarations undeletable, and it's a *prerequisite* for the final "delete the Mapper classes" step of any full exit anyway. Doing it first, independently: (a) de-risks the eventual exit, (b) lets new tables use plain SQL DDL immediately — directly enabling the "stop adding new Mapper entities" lever in the Conclusion — and (c) touches not a single query. + +It does **not**, by itself, drop lift-webkit (the Mapper query layer still pulls it in). But it's the highest-leverage decoupling move that stands completely alone. + +**Works when:** you want to bend the curve and de-risk without committing to the full query migration. + +### Option I — Fund or sponsor upstream Lift + +The root cause is cadence, not code. Sponsoring a maintainer — directly, via a foundation, or GitHub Sponsors — to keep Lift releasing (ideally including a Scala 3 line) attacks the actual problem and benefits every Lift user, not just OBP. Cheaper than a fork *if it works*; entirely contingent on upstream's willingness and on OBP's appetite to fund OSS sustainability. Pairs naturally with Option A: patches keep specific fixes flowing, sponsorship keeps the project alive enough to merge them. + +**Works when:** OBP has budget for OSS sustainability and views Lift as worth keeping alive industry-wide. + +--- + +## What "done" looks like + +- `grep -r "net.liftweb.mapper" obp-api/src/main/scala/code/ | grep -v "/Mapped[A-Z]\|/Mapper[A-Z]\|Schemifier"` returns nothing. +- All `Mapped*Provider.scala` files deleted. +- `ToSchemify.models` unchanged. `Schemifier.schemify(...)` in `Boot.scala` unchanged. +- Mapper case classes (`class Foo extends LongKeyedMapper[Foo]`) remain, but no runtime code calls `Foo.findAll`, `Foo.create`, `foo.save`, etc. +- `DoobieUtil.runQuery` is the only entry point for application-level DB access. +- The next workstream (schema management, out of scope here) can replace Schemifier — and at that point the Mapper class declarations themselves are deletable. + +--- + +## Conclusion: Leaving Mapper Is Genuinely Hard + +The plan above is sound, but the size is real. That's not because Mapper is special — it's what happens when *any* foundational tech sits under 130 entities and a decade of code. Hibernate, Slick, raw JDBC: same outcome. The cost-to-leave is dominated by the size of what's built on top, not by the qualities of the thing being left. + +Three framings worth holding onto: + +### 1. The "let's migrate" framing is the trap + +Multi-year migrations with no user-facing benefit are the projects that stall at 60% and leave two systems running in parallel forever. Don't start a migration unless the forcing function is strong enough to finish it. Half-done is worse than not-started. + +### 2. The forcing function isn't here yet + +Scala 3 is the obvious eventual one — it's coming for everyone on Scala 2.13, probably in the 3–5 year horizon. When it arrives, it brings real budget, real urgency, and real organisational consent to do the work. Starting now without that urgency means burning effort that would be cheaper to spend later, when the migration becomes self-justifying instead of speculative. + +### 3. The realistic stance is "design around the constraint" + +Not "migrate" or "don't migrate" — *constrain*: + +- **Treat Mapper as a constraint, not a problem.** It's the data layer. It will be the data layer for years. Stop apologising for it in docs and code comments. +- **Doobie where it pays off, not as a stealth migration.** The existing pattern — added where N+1 or JDBC bugs forced the issue — is correct. Resist adding Doobie to entities where it doesn't earn its keep. Every dual-system entity is overhead. +- **Stop adding new Mapper entities.** The one cheap policy lever that bends the curve without committing to anything. New entity → Doobie + a small DDL mechanism (Schemifier extension or sidecar SQL file). Existing entity → leave alone. In 3 years Mapper's share of the codebase has shrunk passively while normal feature work continued. +- **Park the full migration as a contingency plan.** This document, essentially. When the forcing function arrives and someone asks *"ok, what would this take?"* — the plan exists and you start from page 1, not page 0. + +### Bottom line + +You don't *solve* this. You inherit it, you live with it, and you wait for the moment when the cost of migrating becomes lower than the cost of staying. That moment will probably arrive. It isn't now. + +This document's value isn't as a Q3 execution plan. It's as the artefact you reach for when the moment does arrive — pre-thought, pre-argued, ready to cost out.