diff --git a/README.md b/README.md index d1feb9e..ab3b5b4 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ sudo bash install.sh \ 如果完整安装已经进入大体量 `pg_basebackup`,不要中断同步,也不要为了更新校验脚本重新执行全量初始化。归档根目录中与 `install.sh` 并列的 `repair.sh` 接受完全相同的启动参数;缺项才交互补全。应在原安装进程退出后执行,也可以在同步期间先运行一次确认状态:检测到 `install.sh`、`21-bootstrap-standby.sh` 或 `pg_basebackup` 仍在运行时,它只记录日志并返回 `REPAIR_RESULT=WAITING_FOR_CURRENT_INSTALL_OR_BASEBACKUP`,不停止任何进程。 -若日志已经出现 `pg_basebackup: base backup completed`,但随后 `pg_ctl: server did not start in time`,不要重跑 `install.sh`。这表示基础备份已经完成,失败点是 Standby 首次恢复的启动等待窗口。`repair.sh` 会先只读校验恢复状态文件、`standby.signal`、受管配置、`PG_VERSION`、`pg_control` 和 Primary system identifier;证据全部一致后,用户输入 `REPAIR` 才会启动或继续等待当前 Standby PGDATA,最长等待 1800 秒。该续启入口不包含 `pg_basebackup`,不移动 PGDATA,不停止 Primary/Standby;Standby 达到 `streaming` 后才继续安装 Pgpool。 +若日志已经出现 `pg_basebackup: base backup completed`,但随后 `pg_ctl: server did not start in time`,不要重跑 `install.sh`。这表示基础备份已经完成,失败点是 Standby 首次恢复的启动等待窗口。`repair.sh` 不依赖安装状态文件,只读校验当前 PGDATA 的 `standby.signal`、受管恢复配置、`PG_VERSION`、`pg_control` 和 Primary 实时 system identifier;证据全部一致后,用户输入 `REPAIR` 才会启动或继续等待当前 Standby PGDATA,最长等待 1800 秒。该续启入口不包含 `pg_basebackup`,不移动 PGDATA,不停止 Primary/Standby;Standby 达到 `streaming` 后才继续安装 Pgpool。 全量参数示例: @@ -105,15 +105,17 @@ sudo bash repair.sh \ --business-password '现有业务用户密码' ``` -`repair.sh` 先把本次全部参数与原安装已保存的 `config/cluster.env`、`config/secrets.env` 逐项比对,再检测主从角色、system identifier、复制槽、WAL receiver、`streaming`、监控/业务凭据、路由探针、Pgpool 运行时、配置、服务、自启、监听和节点识别。`--allowed-client-cidrs` 在 repair 中只用于确认“正在修复原安装会话”,不再作为要落盘的访问白名单;实际白名单每次都由 Primary 动态读取。判定规则如下: +`repair.sh` 是无状态入口,只使用本次命令行参数和三台服务器的实时状态。它不读取、不比对、也不要求上一次安装生成的 `config/cluster.env`、`config/secrets.env`、`config/pool-users.txt`;即使这些文件全部删除仍可修复。启动后在 `/var/tmp` 私有目录生成本次配置,退出时清理。`--allowed-client-cidrs` 只用于形成完整参数集;实际白名单每次都由 Primary 动态读取。 + +只读检查阶段不写账号或服务。用户输入 `REPAIR` 后,脚本会重新生成并轮换安装器拥有的 `rw_replicator`、`pgpool_monitor`、PCP 与 Pgpool 加密密钥,更新 Standby 的 postgres 私有复制密码文件,再配置 Pgpool。它不会修改业务账号密码,不会停止或重启 Primary/Standby;已有 streaming 会话不会因角色改密被主动断开。随后继续检测主从角色、system identifier、复制槽、WAL receiver、`streaming`、业务凭据、路由探针、Pgpool 运行时、配置、服务、自启、监听和节点识别。判定规则如下: - 完整安装或基础同步仍运行:只报告 `WAITING`,退出码 `10`; - 主从不同源或 Standby 既不可用又缺少完整基础备份证据:报告 `BLOCKED_DATABASE`,退出码 `20`,不修改数据库;已证明基础备份完成的启动中断会列出 `standby_resume_existing_pgdata`,输入 `REPAIR` 后只续启现有 PGDATA; - Primary HBA 无法稳定读取、存在解析错误,或没有可同步的普通 IPv4 `host` 白名单:报告 `BLOCKED_PRIMARY_POLICY`,退出码 `25`,不修改任何节点; -- Pgpool 本机有可修复项:逐项列出,只有用户输入 `REPAIR` 后才执行本机离线运行时、配置和 systemd 服务修复;其他输入返回退出码 `30`; -- 所有检查通过:报告 `REPAIR_RESULT=HEALTHY`;修复并验收通过则报告 `REPAIR_RESULT=REPAIRED`。 +- 脚本逐项列出受管凭据、Standby 续启(如需要)和 Pgpool 本机可修复项;只有用户输入 `REPAIR` 后才执行,其他输入返回退出码 `30`; +- 只读检查通过后始终列出本次受管凭据轮换与会话配置重建项;只有用户输入 `REPAIR` 才会执行,验收通过后报告 `REPAIR_RESULT=REPAIRED`。 -该入口源码和回归门禁禁止调用 `10-configure-primary.sh`、`20-install-postgresql-standby.sh`、`21-bootstrap-standby.sh`,并禁止执行 `pg_basebackup` 或停止数据库。唯一允许的数据库动作是调用 `22-resume-standby-after-basebackup.sh` 启动/等待已经完成基础备份的 Standby PGDATA;其他修复只调用 Pgpool 本机的 `30-install-pgpool.sh`、`31-configure-pgpool.sh`。随后通过专用 `business.rw_probe` 做少量可自动清理的 DML/SELECT 路由验收。完整日志写入 `/var/log/pg-readwrite-proxy-lab/repair-<时间>-.log`,目录 700、文件 600。 +该入口源码和回归门禁禁止调用 `10-configure-primary.sh`、`20-install-postgresql-standby.sh`、`21-bootstrap-standby.sh`,并禁止执行 `pg_basebackup` 或停止数据库。允许的数据库动作只有:轮换两个安装器受管角色密码、更新 Standby 私有复制密码文件,以及在证据链完整时调用 `22-resume-standby-after-basebackup.sh` 启动/等待已完成基础备份的 Standby PGDATA;其他修复只调用 Pgpool 本机的 `30-install-pgpool.sh`、`31-configure-pgpool.sh`。随后通过专用 `business.rw_probe` 做少量可自动清理的 DML/SELECT 路由验收。完整日志写入 `/var/log/pg-readwrite-proxy-lab/repair-<时间>-.log`,目录 700、文件 600。 ### repair 同步 Primary 访问策略 @@ -123,6 +125,32 @@ Primary 是业务客户端 IP 白名单的来源。`repair.sh` 使用厂商超 这项同步只复制“允许访问的 IPv4 地址集合”,不复制 Primary 规则中的数据库名、用户名、先后顺序或认证细节。回环地址、`0.0.0.0/0`、IPv6、hostname、`reject`、`hostssl` 等非普通规则直接忽略;Pgpool 仍通过自己的 `pool_passwd` 校验实际登录账号。该行为有意保持简单,适用于现场现有的普通 IPv4 白名单。 +## 开放实例内全部数据库和现有登录用户 + +初始安装按一个 `BUSINESS_DATABASE + BUSINESS_USER` 完成读写路由验收。若现场需要让同一 PostgreSQL 实例内的其他现有登录用户也通过 Pgpool 访问其有权连接的数据库,在 Pgpool 服务器上运行归档顶层脚本: + +```bash +sudo bash enable-all-databases-users.sh \ + --pgpool-host 192.168.80.140 \ + --primary-host 192.168.80.110 \ + --standby-host 192.168.80.120 \ + --postgresql-port 5432 \ + --pgpool-port 5432 \ + --ssh-port 22022 \ + --root-ssh-password '数据库服务器root密码' +``` + +参数未提供时会在启动阶段交互补全。脚本先完成平台、三节点身份、system identifier、`streaming`、配置路径、HBA 语法、网络、监听、文件权限和备份能力检查,展示完整计划后仅在输入 `APPLY` 时执行。它会: + +- 保留 Pgpool 当前 `pool_hba.conf` 的客户端来源 IP/CIDR,只把前端认证从 `md5` 改为 `password`,从而不再要求把每个业务用户密码登记到 `pool_passwd`; +- 在 Primary 和 Standby 的实际 `pg_hba.conf` 顶部增加独立受管区块,仅允许 Pgpool 服务器的 `/32` 地址以 `all database + all user` 进入后端; +- 分别备份三台服务器的原配置,reload PostgreSQL 与 Pgpool 后校验解析状态;任一步失败则尝试恢复本次备份; +- 不创建、启用、提权或改密任何 PostgreSQL 角色,不修改数据库 `CONNECT` 或对象权限,不重启服务,不执行 `pg_basebackup`,也不重新同步 Standby。 + +这里的“全部用户”指 PostgreSQL 中已经存在且具备 `LOGIN` 的角色;实际访问能力仍由角色的数据库 `CONNECT`、schema/table 等原有权限决定。Pgpool 仍只接受当前 `pool_hba.conf` 已允许的客户端来源。当前基线 `ssl=off`,而 `password` 前端认证会在客户端到 Pgpool 链路上传输明文口令协议;脚本会明确警告,仅适用于已确认的物理隔离受控网络,生产长期方案应启用 TLS。 + +该脚本是安装后的独立扩展。当前 `repair.sh` 仍会按单业务用户基线重新生成 Pgpool 认证文件;若之后执行了 `repair.sh`,需再次运行 `enable-all-databases-users.sh`。脚本日志写入 `/var/log/pg-readwrite-proxy-lab/enable-all-databases-users-<时间>-.log`。 + ## 安全与回滚 - 示例配置中的 Primary 重启和 Standby 重建门禁默认都是 `no`;一键入口只在用户输入 `APPLY` 后生成授权值。 diff --git a/diagnose-pgpool-port.sh b/diagnose-pgpool-port.sh new file mode 100644 index 0000000..1abd08b --- /dev/null +++ b/diagnose-pgpool-port.sh @@ -0,0 +1,193 @@ +#!/usr/bin/env bash + +# 只读检查 Pgpool 本机 5432 监听和防火墙状态。 +# 不修改服务、防火墙、路由、内核参数或 Pgpool 配置。 +set -uo pipefail +IFS=$'\n\t' + +PORT="${1:-5432}" +LOG_FILE="/var/tmp/pgpool-port-check-$(date '+%Y%m%d-%H%M%S')-$$.log" + +die() { + printf '[ERROR] %s\n' "$*" >&2 + exit 1 +} + +[[ "${EUID}" -eq 0 ]] || die '请使用 root 运行此脚本。' +[[ "${PORT}" =~ ^[0-9]{1,5}$ ]] && ((10#${PORT} >= 1 && 10#${PORT} <= 65535)) || \ + die "端口无效: ${PORT}" + +for command_name in date tee systemctl ss ip grep awk timeout; do + command -v "${command_name}" >/dev/null 2>&1 || die "缺少命令: ${command_name}" +done + +umask 077 +touch "${LOG_FILE}" || die "无法创建日志: ${LOG_FILE}" +chmod 600 "${LOG_FILE}" +exec > >(tee -a "${LOG_FILE}") 2>&1 + +section() { + printf '\n========== %s ==========\n' "$1" +} + +run_if_exists() { + local command_name="$1" + shift + if command -v "${command_name}" >/dev/null 2>&1; then + "$@" 2>&1 || printf '[INFO] command=%s exit=%s\n' "${command_name}" "$?" + else + printf '[INFO] command_missing=%s\n' "${command_name}" + fi +} + +section '基本信息' +printf 'CHECK_TIME=%s\n' "$(date '+%Y-%m-%d %H:%M:%S %z')" +printf 'CHECK_MODE=read_only changes=no\n' +printf 'PORT=%s\nLOG_FILE=%s\n' "${PORT}" "${LOG_FILE}" +hostname 2>&1 || true +uname -a 2>&1 || true +ip -br -4 address show 2>&1 || true +ip -4 route show 2>&1 || true + +section 'Pgpool 服务' +pgpool_active="$(systemctl is-active pgpool 2>/dev/null || true)" +pgpool_enabled="$(systemctl is-enabled pgpool 2>/dev/null || true)" +printf 'PGPOOL_ACTIVE=%s\nPGPOOL_ENABLED=%s\n' \ + "${pgpool_active:-unknown}" "${pgpool_enabled:-unknown}" +systemctl status pgpool --no-pager -l 2>&1 || true + +section "TCP ${PORT} 监听" +listener="$(ss -Hlnpt "sport = :${PORT}" 2>&1 || true)" +if [[ -n "${listener}" ]]; then + printf '%s\n' "${listener}" +else + printf 'LISTENER=none\n' +fi + +listener_present=no +listener_external=no +[[ -z "${listener}" ]] || listener_present=yes +if grep -Eq "(0\\.0\\.0\\.0|\\[::\\]|\\*):${PORT}([[:space:]]|$)" <<<"${listener}"; then + listener_external=yes +fi +printf 'LISTENER_PRESENT=%s\nLISTENER_EXTERNAL=%s\n' \ + "${listener_present}" "${listener_external}" + +section '本机 TCP 连接测试' +local_loopback_tcp=fail +if timeout 3 bash -c "exec 3<>/dev/tcp/127.0.0.1/${PORT}" 2>/dev/null; then + local_loopback_tcp=pass +fi +printf 'LOCAL_TCP_127_0_0_1=%s\n' "${local_loopback_tcp}" + +server_ips="$(ip -o -4 addr show scope global | awk '{split($4,a,"/"); print a[1]}')" +local_server_ip_tcp=skipped +if [[ -z "${server_ips}" ]]; then + printf 'SERVER_IP_TCP_TEST=skipped reason=no_global_ipv4\n' +else + local_server_ip_tcp=pass + while IFS= read -r server_ip; do + [[ -n "${server_ip}" ]] || continue + result=fail + if timeout 3 bash -c "exec 3<>/dev/tcp/${server_ip}/${PORT}" 2>/dev/null; then + result=pass + fi + [[ "${result}" == pass ]] || local_server_ip_tcp=fail + printf 'SERVER_IP_TCP_TEST ip=%s result=%s\n' "${server_ip}" "${result}" + done <<<"${server_ips}" +fi + +section '防火墙服务状态' +firewalld_active="$(systemctl is-active firewalld 2>/dev/null || true)" +nftables_active="$(systemctl is-active nftables 2>/dev/null || true)" +printf 'FIREWALLD_ACTIVE=%s\nNFTABLES_ACTIVE=%s\n' \ + "${firewalld_active:-unknown}" "${nftables_active:-unknown}" + +if command -v firewall-cmd >/dev/null 2>&1; then + firewall-cmd --state 2>&1 || true + firewall-cmd --get-active-zones 2>&1 || true + firewall-cmd --list-all-zones 2>&1 || true +else + printf '[INFO] command_missing=firewall-cmd\n' +fi + +section 'iptables 当前内核规则' +iptables_available=no +iptables_input='' +iptables_save='' +if command -v iptables >/dev/null 2>&1; then + iptables_available=yes + iptables --version 2>&1 || true + iptables_input="$(iptables -w 2 -nvL INPUT --line-numbers 2>&1 || true)" + printf '%s\n' "${iptables_input}" + if command -v iptables-save >/dev/null 2>&1; then + iptables_save="$(iptables-save -c 2>&1 || true)" + printf '%s\n' "${iptables_save}" + fi +else + printf '[INFO] command_missing=iptables\n' +fi + +section 'nft 当前内核规则' +nft_available=no +nft_rules='' +if command -v nft >/dev/null 2>&1; then + nft_available=yes + nft_rules="$(nft -a list ruleset 2>&1 || true)" + if [[ -n "${nft_rules}" ]]; then + printf '%s\n' "${nft_rules}" + else + printf 'NFT_RULESET=empty\n' + fi +else + printf '[INFO] command_missing=nft\n' +fi + +section '与目标端口相关的规则' +port_rule_found=no +drop_rule_found=no +if grep -Eq "(^|[^0-9])${PORT}([^0-9]|$)" <<<"${iptables_save}"; then + port_rule_found=yes + grep -En "(^|[^0-9])${PORT}([^0-9]|$)" <<<"${iptables_save}" || true +fi +if grep -Eiq '(^|[[:space:]])(DROP|REJECT)([[:space:]]|$)' <<<"${iptables_input}${iptables_save}"; then + drop_rule_found=yes + grep -Ein '(^|[[:space:]])(DROP|REJECT)([[:space:]]|$)' <<<"${iptables_input}${iptables_save}" || true +fi +if grep -Eq "(^|[^0-9])${PORT}([^0-9]|$)" <<<"${nft_rules}"; then + port_rule_found=yes + grep -En "(^|[^0-9])${PORT}([^0-9]|$)" <<<"${nft_rules}" || true +fi +if grep -Eiq '(^|[[:space:]])(drop|reject)([[:space:]]|$)' <<<"${nft_rules}"; then + drop_rule_found=yes + grep -Ein '(^|[[:space:]])(drop|reject)([[:space:]]|$)' <<<"${nft_rules}" || true +fi +printf 'PORT_RULE_FOUND=%s\nDROP_OR_REJECT_RULE_FOUND=%s\n' \ + "${port_rule_found}" "${drop_rule_found}" + +section '自动结论' +if [[ "${pgpool_active}" != active ]]; then + conclusion='PGPOOL_SERVICE_NOT_ACTIVE' + recommendation='Pgpool 服务未运行,先检查上面的 systemctl status 和 journalctl -u pgpool。' +elif [[ "${listener_present}" != yes ]]; then + conclusion='PORT_NOT_LISTENING' + recommendation="Pgpool 已运行但没有监听 ${PORT},检查 /etc/pgpool-II/pgpool.conf 的 port/listen_addresses。" +elif [[ "${listener_external}" != yes ]]; then + conclusion='PORT_ONLY_LISTENS_LOCALLY' + recommendation="${PORT} 没有监听 0.0.0.0/[::],外部客户端无法连接。" +elif [[ "${local_server_ip_tcp}" != pass ]]; then + conclusion='LOCAL_TCP_FAILED' + recommendation="本机通过服务器业务 IPv4 连接 ${PORT} 失败,检查本机防火墙规则、Pgpool 监听和系统日志。" +elif [[ "${port_rule_found}" == yes || "${drop_rule_found}" == yes ]]; then + conclusion='LOCAL_SERVICE_OK_FIREWALL_RULES_REQUIRE_REVIEW' + recommendation="Pgpool 本机监听和 TCP 测试正常,但内核存在 ${PORT} 或 DROP/REJECT 规则;把完整日志发回确认规则顺序和命中范围。" +else + conclusion='LOCAL_SERVICE_AND_FIREWALL_LOOK_NORMAL' + recommendation="Pgpool 本机监听和 TCP 测试正常,未发现明显本机拦截规则;若外部仍无响应,应检查上游 ACL、返回路由或安全设备。" +fi + +printf 'DIAG_RESULT=%s\n' "${conclusion}" +printf 'RECOMMENDATION=%s\n' "${recommendation}" +printf '诊断完成,请把日志发回:%s\n' "${LOG_FILE}" + +exit 0 diff --git a/docs/production-notes.md b/docs/production-notes.md index 301a658..0e4ef82 100644 --- a/docs/production-notes.md +++ b/docs/production-notes.md @@ -64,5 +64,5 @@ - 不得用软链接伪造 OpenSSL/readline/libnsl ABI,也不得把 CentOS 系统库直接复制到麒麟。重新构建必须使用 `packages/build-kylin-v10-payloads.sh` 并更新 SHA 清单。 - 客户端 CIDR 必须最小化;PCP 只监听 localhost;后端 5432 只允许 Standby 和 Pgpool 地址。 - 当前实验入口未启用 TLS。跨不可信网络生产使用前必须部署服务端证书、客户端验证和密钥轮换。 -- `config/secrets.env` 与业务明文密码只应短暂存在于安装主机,完成 `pool_passwd` 生成后迁移到密钥系统并按制度擦除。 +- `install.sh` 首次部署可将 `config/secrets.env` 迁移到密钥系统后擦除;`repair.sh` 不依赖该文件,会在每次获批修复时以本次参数临时生成并轮换安装器受管凭据,退出后清理。业务密码仍只来自本次参数。 - 一键安装首次连接使用隔离管理网中的 root 密码 SSH;生产应预置并核验 SSH 主机密钥,优先改为短期安装密钥或堡垒机审计,避免只依赖首次连接信任。 diff --git a/enable-all-databases-users.sh b/enable-all-databases-users.sh new file mode 100644 index 0000000..ca2e736 --- /dev/null +++ b/enable-all-databases-users.sh @@ -0,0 +1,527 @@ +#!/usr/bin/env bash + +# 在不重建 Standby、不重启数据库的前提下,让现有 PostgreSQL 登录用户可通过 +# Pgpool-II 访问其原本有权连接的全部数据库。脚本不创建用户、不修改密码或权限。 + +set -Eeuo pipefail +IFS=$'\n\t' + +SCRIPT_PHASE='启动参数处理' +APPLY_STARTED='no' +SUCCESS='no' +TEMP_ROOT='' +LOCAL_BACKUP_DIR='' +PRIMARY_BACKUP_DIR='' +STANDBY_BACKUP_DIR='' + +readonly PGPOOL_SERVICE='pgpool' +readonly PGPOOL_CONFIG_DIR='/etc/pgpool-II' +readonly PGPOOL_CONF="${PGPOOL_CONFIG_DIR}/pgpool.conf" +readonly POOL_HBA="${PGPOOL_CONFIG_DIR}/pool_hba.conf" +readonly PGPOOL_BIN='/opt/pgpool-II-4.7.2/bin/pgpool' +readonly PGPOOL_RUNTIME_LIB='/opt/pgpool-runtime-kylin-v10/lib' +readonly DB_ADMIN_TOOL='/opt/pgsql12/bin/tools' +readonly SSHPASS_PAYLOAD='packages/payload/sshpass-1.10-aarch64-kylin-v10.tar.gz' +readonly SSHPASS_SHA256='84bcff17fc7e48d0a8552c985818e17e485f95a66b3c50c4eedde6bcbdc96ffd' +readonly HBA_MARKER='PG_RW_PROXY_ALL_DATABASE_USERS' + +log() { printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S')" "$*"; } +warn() { printf '[WARN] %s\n' "$*" >&2; } +die() { + local line_number="${BASH_LINENO[0]:-unknown}" + printf '[ERROR] time=%s phase=%s line=%s exit=1\n' \ + "$(date '+%Y-%m-%d %H:%M:%S')" "${SCRIPT_PHASE}" "${line_number}" >&2 + printf '[ERROR] %s\n' "$*" >&2 + exit 1 +} + +show_usage() { + cat <<'USAGE' +用法: + sudo bash enable-all-databases-users.sh [参数] + +参数同时支持“--参数 值”和“--参数=值”;未提供的参数在启动阶段交互补全。 + + --pgpool-host IP 当前 Pgpool-II 服务器内网 IPv4 + --primary-host IP PostgreSQL Primary 内网 IPv4 + --standby-host IP PostgreSQL Standby 内网 IPv4 + --postgresql-port PORT Primary/Standby 共用端口,默认 5432 + --pgpool-port PORT Pgpool-II 对外端口,默认 5432 + --ssh-port PORT Primary/Standby 共用 root SSH 端口,默认 22 + --root-ssh-password PASSWORD Primary/Standby 共用 root SSH 密码 + -h, --help 显示帮助 + +作用: + 1. 保留 pool_hba.conf 现有客户端来源地址,只把前端认证改为 password; + 2. 仅允许 Pgpool 服务器 IP 以任意现有 PostgreSQL 用户连接全部数据库; + 3. 不创建/启用角色,不修改密码、CONNECT 权限、对象权限或数据库数据; + 4. 只 reload PostgreSQL/Pgpool,不重启数据库,不重新同步 Standby。 + +所有只读检查通过后,脚本会显示计划;只有输入 APPLY 才执行。 +USAGE +} + +validate_ipv4() { + local value="$1" label="$2" octet old_ifs + [[ "${value}" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]] || die "${label} 必须是 IPv4 地址: ${value}" + old_ifs="${IFS}" + IFS='.' read -r -a octets <<<"${value}" + IFS="${old_ifs}" + for octet in "${octets[@]}"; do + ((10#${octet} <= 255)) || die "${label} 不是有效 IPv4 地址: ${value}" + done +} + +validate_port() { + local value="$1" label="$2" + [[ "${value}" =~ ^[0-9]{1,5}$ ]] && ((10#${value} >= 1 && 10#${value} <= 65535)) || \ + die "${label} 不是有效端口: ${value}" +} + +prompt_required() { + local variable_name="$1" prompt_text="$2" value="${!1:-}" + while [[ -z "${value}" ]]; do + read -r -p "${prompt_text}: " value || die "无法读取 ${variable_name}。" + done + printf -v "${variable_name}" '%s' "${value}" +} + +prompt_default() { + local variable_name="$1" prompt_text="$2" default_value="$3" value="${!1:-}" + if [[ -z "${value}" ]]; then + read -r -p "${prompt_text} [${default_value}]: " value || die "无法读取 ${variable_name}。" + value="${value:-${default_value}}" + fi + printf -v "${variable_name}" '%s' "${value}" +} + +prompt_secret() { + local variable_name="$1" prompt_text="$2" value="${!1:-}" + while [[ -z "${value}" ]]; do + read -r -s -p "${prompt_text}: " value || die "无法读取 ${variable_name}。" + printf '\n' + done + [[ "${value}" != *$'\n'* && "${value}" != *$'\r'* ]] || die 'SSH 密码不能包含换行。' + printf -v "${variable_name}" '%s' "${value}" +} + +parse_inputs() { + local argument option value + declare -A seen_options=() + while (($# > 0)); do + argument="$1" + case "${argument}" in + -h|--help) show_usage; exit 0 ;; + --*=*) option="${argument%%=*}"; value="${argument#*=}"; shift ;; + --*) + option="${argument}" + (($# >= 2)) || die "参数缺少值: ${option}" + value="$2" + shift 2 + ;; + *) die "不支持的位置参数: ${argument}" ;; + esac + [[ -n "${value}" ]] || die "参数值不能为空: ${option}" + [[ -z "${seen_options[${option}]:-}" ]] || die "参数不能重复提供: ${option}" + seen_options["${option}"]=1 + case "${option}" in + --pgpool-host) PGPOOL_HOST="${value}" ;; + --primary-host) PRIMARY_HOST="${value}" ;; + --standby-host) STANDBY_HOST="${value}" ;; + --postgresql-port) POSTGRESQL_PORT="${value}" ;; + --pgpool-port) PGPOOL_PORT="${value}" ;; + --ssh-port) SSH_PORT="${value}" ;; + --root-ssh-password) ROOT_SSH_PASSWORD="${value}" ;; + *) die "未知参数: ${option}" ;; + esac + done +} + +require_commands() { + local command_name + for command_name in "$@"; do + command -v "${command_name}" >/dev/null 2>&1 || die "缺少命令: ${command_name}" + done +} + +config_value() { + local key="$1" file="$2" + sed -n -E "s/^[[:space:]]*${key}[[:space:]]*=[[:space:]]*'([^']*)'[[:space:]]*(#.*)?$/\\1/p; s/^[[:space:]]*${key}[[:space:]]*=[[:space:]]*([^'#[:space:]]+)[[:space:]]*(#.*)?$/\\1/p" \ + "${file}" | tail -n 1 +} + +managed_block_is_well_formed() { + local file="$1" marker="$2" begin_count end_count + begin_count="$(grep -Fxc "# BEGIN ${marker}" "${file}" || true)" + end_count="$(grep -Fxc "# END ${marker}" "${file}" || true)" + [[ "${begin_count}" == "${end_count}" && "${begin_count}" -le 1 ]] +} + +remote_root() { + local host="$1" + "${SSHPASS_BIN}" -e ssh "${SSH_ARGS[@]}" root@"${host}" bash -s +} + +remote_snapshot() { + local host="$1" expected_role="$2" + { + printf 'ADMIN_TOOL=%q\n' "${DB_ADMIN_TOOL}" + printf 'PORT=%q\n' "${POSTGRESQL_PORT}" + printf 'EXPECTED_ROLE=%q\n' "${expected_role}" + printf 'EXPECTED_PGPOOL_HOST=%q\n' "${PGPOOL_HOST}" + cat <<'REMOTE' +set -Eeuo pipefail +for command_name in id uname sha256sum stat awk grep sed cp mv mktemp df cat mkdir chmod chown rm sleep; do + command -v "${command_name}" >/dev/null 2>&1 +done +[[ "$(id -u)" == 0 && -r /etc/os-release ]] +. /etc/os-release +[[ "$(uname -m)" == aarch64 ]] +case "${ID:-}:${VERSION_ID:-}" in + kylin:V10) ;; + centos:7|centos:7.*) ;; # 仅用于本项目 ARM64 实验节点回归。 + *) exit 41 ;; +esac +[[ -x "${ADMIN_TOOL}" ]] + +identity="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c \ + "select current_setting('server_version_num')||'|'||(case when pg_is_in_recovery() then 'standby' else 'primary' end)||'|'||(select system_identifier from pg_control_system())||'|'||current_setting('hba_file')")" +IFS='|' read -r version_num actual_role system_id hba_path <<<"${identity}" +[[ "${version_num}" == 120000 && "${actual_role}" == "${EXPECTED_ROLE}" && "${system_id}" =~ ^[0-9]+$ ]] +[[ "${hba_path}" == /* && -f "${hba_path}" && ! -L "${hba_path}" ]] +[[ -r "${hba_path}" && -w "${hba_path}" ]] +[[ -d /var && -w /var ]] +available_kb="$(df -Pk /var | awk 'NR==2{print $4}')" +[[ "${available_kb}" =~ ^[0-9]+$ && "${available_kb}" -ge 10240 ]] +managed_begin="$(grep -Fxc '# BEGIN PG_RW_PROXY_ALL_DATABASE_USERS' "${hba_path}" || true)" +managed_end="$(grep -Fxc '# END PG_RW_PROXY_ALL_DATABASE_USERS' "${hba_path}" || true)" +[[ "${managed_begin}" == "${managed_end}" && "${managed_begin}" -le 1 ]] +hba_errors="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c "select count(*) from pg_hba_file_rules where error is not null")" +[[ "${hba_errors}" == 0 ]] +if [[ "${EXPECTED_ROLE}" == primary ]]; then + streaming="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c "select count(*) from pg_stat_replication where state='streaming'")" +else + streaming="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c "select count(*) from pg_stat_wal_receiver where status='streaming'")" +fi +[[ "${streaming}" =~ ^[1-9][0-9]*$ ]] +printf 'REMOTE_SNAPSHOT=%s|%s|%s|%s|%s\n' \ + "${EXPECTED_ROLE}" "${system_id}" "${hba_path}" "$(sha256sum "${hba_path}" | awk '{print $1}')" "${streaming}" +REMOTE + } | remote_root "${host}" +} + +apply_remote_hba() { + local host="$1" role="$2" expected_hash="$3" + { + printf 'ADMIN_TOOL=%q\n' "${DB_ADMIN_TOOL}" + printf 'PORT=%q\n' "${POSTGRESQL_PORT}" + printf 'PGPOOL_CIDR=%q\n' "${PGPOOL_HOST}/32" + printf 'EXPECTED_HASH=%q\n' "${expected_hash}" + printf 'ROLE=%q\n' "${role}" + printf 'TIMESTAMP=%q\n' "${RUN_TIMESTAMP}" + cat <<'REMOTE' +set -Eeuo pipefail +umask 077 +hba_path="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c "select current_setting('hba_file')")" +[[ "${hba_path}" == /* && -f "${hba_path}" && ! -L "${hba_path}" ]] +[[ "$(sha256sum "${hba_path}" | awk '{print $1}')" == "${EXPECTED_HASH}" ]] +backup_dir="/var/backups/pg-readwrite-proxy-lab/all-databases-users-${TIMESTAMP}-${ROLE}" +[[ ! -e "${backup_dir}" ]] +mkdir -p "${backup_dir}" +chmod 700 "${backup_dir}" +cp -a -- "${hba_path}" "${backup_dir}/pg_hba.conf.before" + +clean_file="$(mktemp)" +new_file="${hba_path}.all-users.$$" +changed=no +rollback_local_node() { + local status=$? + trap - EXIT + rm -f -- "${clean_file}" "${new_file}" + if [[ "${status}" -ne 0 && "${changed}" == yes ]]; then + cp -a -- "${backup_dir}/pg_hba.conf.before" "${hba_path}" + "${ADMIN_TOOL}" psql -d postgres -p "${PORT}" -c 'select pg_reload_conf()' >/dev/null 2>&1 || true + fi + exit "${status}" +} +trap rollback_local_node EXIT + +awk ' + $0=="# BEGIN PG_RW_PROXY_ALL_DATABASE_USERS" {skip=1; next} + $0=="# END PG_RW_PROXY_ALL_DATABASE_USERS" {skip=0; next} + !skip {print} +' "${hba_path}" >"${clean_file}" +cp -a -- "${hba_path}" "${new_file}" +{ + printf '# BEGIN PG_RW_PROXY_ALL_DATABASE_USERS\n' + printf '# 仅允许 Pgpool 节点代表现有登录角色访问其原有权限范围;不授予数据库权限。\n' + printf 'host all all %-22s md5\n' "${PGPOOL_CIDR}" + printf '# END PG_RW_PROXY_ALL_DATABASE_USERS\n\n' + cat "${clean_file}" +} >"${new_file}" +chown --reference="${hba_path}" "${new_file}" +chmod --reference="${hba_path}" "${new_file}" +mv -f -- "${new_file}" "${hba_path}" +changed=yes +reload_result="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c 'select pg_reload_conf()')" +[[ "${reload_result}" == t ]] +sleep 1 +hba_errors="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c "select count(*) from pg_hba_file_rules where error is not null")" +managed_rule="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c \ + "select count(*) from pg_hba_file_rules where type='host' and database='{all}' and user_name='{all}' and address='${PGPOOL_CIDR%/*}' and netmask='255.255.255.255' and auth_method in ('md5','scram-sha-256') and error is null")" +[[ "${hba_errors}" == 0 && "${managed_rule}" =~ ^[1-9][0-9]*$ ]] +changed=no +rm -f -- "${clean_file}" +trap - EXIT +printf 'REMOTE_APPLIED=%s|%s|%s\n' "${ROLE}" "${backup_dir}" "$(sha256sum "${hba_path}" | awk '{print $1}')" +REMOTE + } | remote_root "${host}" +} + +restore_remote_hba() { + local host="$1" backup_dir="$2" + [[ -n "${backup_dir}" ]] || return 0 + { + printf 'ADMIN_TOOL=%q\n' "${DB_ADMIN_TOOL}" + printf 'PORT=%q\n' "${POSTGRESQL_PORT}" + printf 'BACKUP_DIR=%q\n' "${backup_dir}" + cat <<'REMOTE' +set -Eeuo pipefail +hba_path="$(${ADMIN_TOOL} psql -d postgres -p "${PORT}" -c "select current_setting('hba_file')")" +[[ "${BACKUP_DIR}" == /var/backups/pg-readwrite-proxy-lab/all-databases-users-* ]] +[[ -e "${BACKUP_DIR}" ]] || exit 0 +[[ -f "${BACKUP_DIR}/pg_hba.conf.before" && ! -L "${BACKUP_DIR}/pg_hba.conf.before" ]] +cp -a -- "${BACKUP_DIR}/pg_hba.conf.before" "${hba_path}" +"${ADMIN_TOOL}" psql -d postgres -p "${PORT}" -c 'select pg_reload_conf()' >/dev/null +REMOTE + } | remote_root "${host}" +} + +cleanup_and_rollback() { + local status=$? + trap - EXIT + if [[ "${status}" -ne 0 && "${APPLY_STARTED}" == yes && "${SUCCESS}" != yes ]]; then + set +e + warn '执行失败,开始恢复本次脚本创建的备份。' + if [[ -n "${LOCAL_BACKUP_DIR}" && -f "${LOCAL_BACKUP_DIR}/pool_hba.conf.before" ]]; then + cp -a -- "${LOCAL_BACKUP_DIR}/pool_hba.conf.before" "${POOL_HBA}" + systemctl reload "${PGPOOL_SERVICE}" >/dev/null 2>&1 || true + fi + restore_remote_hba "${STANDBY_HOST:-}" "${STANDBY_BACKUP_DIR}" || true + restore_remote_hba "${PRIMARY_HOST:-}" "${PRIMARY_BACKUP_DIR}" || true + warn '自动回滚已执行;请结合本次日志复核三项服务状态。' + set -e + fi + [[ -z "${TEMP_ROOT}" || "${TEMP_ROOT}" != /var/tmp/pg-rw-all-users.* ]] || rm -rf -- "${TEMP_ROOT}" + unset SSHPASS ROOT_SSH_PASSWORD + exit "${status}" +} +trap cleanup_and_rollback EXIT + +PGPOOL_HOST='' +PRIMARY_HOST='' +STANDBY_HOST='' +POSTGRESQL_PORT='' +PGPOOL_PORT='' +SSH_PORT='' +ROOT_SSH_PASSWORD='' +parse_inputs "$@" + +[[ "${EUID}" -eq 0 ]] || die '请使用 root 或 sudo 执行。' +require_commands awk sed grep tail sha256sum tar mktemp ssh systemctl journalctl ip ss stat cp mv tee date df timeout + +default_pgpool_host="$(ip -o -4 addr show scope global | awk 'NR==1{split($4,a,"/");print a[1]}')" +prompt_default PGPOOL_HOST '当前 Pgpool-II 服务器内网 IPv4 地址' "${default_pgpool_host:-127.0.0.1}" +prompt_required PRIMARY_HOST 'PostgreSQL Primary 内网 IPv4 地址' +prompt_required STANDBY_HOST 'PostgreSQL Standby 内网 IPv4 地址' +prompt_default POSTGRESQL_PORT 'Primary/Standby 共用 PostgreSQL 端口' '5432' +prompt_default PGPOOL_PORT 'Pgpool-II 对外服务端口' '5432' +prompt_default SSH_PORT 'Primary/Standby 共用 root SSH 端口' '22' +prompt_secret ROOT_SSH_PASSWORD 'Primary/Standby 共用 root SSH 密码' + +validate_ipv4 "${PGPOOL_HOST}" PGPOOL_HOST +validate_ipv4 "${PRIMARY_HOST}" PRIMARY_HOST +validate_ipv4 "${STANDBY_HOST}" STANDBY_HOST +validate_port "${POSTGRESQL_PORT}" POSTGRESQL_PORT +validate_port "${PGPOOL_PORT}" PGPOOL_PORT +validate_port "${SSH_PORT}" SSH_PORT +[[ "${PGPOOL_HOST}" != "${PRIMARY_HOST}" && "${PGPOOL_HOST}" != "${STANDBY_HOST}" && \ + "${PRIMARY_HOST}" != "${STANDBY_HOST}" ]] || die '三台服务器地址必须不同。' + +LOG_DIR='/var/log/pg-readwrite-proxy-lab' +mkdir -p "${LOG_DIR}" +chmod 700 "${LOG_DIR}" +LOG_FILE="${LOG_DIR}/enable-all-databases-users-$(date '+%Y%m%d-%H%M%S')-$$.log" +touch "${LOG_FILE}" +chmod 600 "${LOG_FILE}" +exec > >(tee -a "${LOG_FILE}") 2> >(tee -a "${LOG_FILE}" >&2) +log "完整日志:${LOG_FILE}" + +SCRIPT_PHASE='只读检查 Pgpool 节点' +[[ -r /etc/os-release ]] || die '无法读取 /etc/os-release。' +# shellcheck disable=SC1091 +source /etc/os-release +[[ "${ID:-}" == kylin && "${VERSION_ID:-}" == V10 && "$(uname -m)" == aarch64 ]] || \ + die "脚本只支持麒麟 V10 aarch64 Pgpool 节点;当前=${PRETTY_NAME:-unknown}/$(uname -m)。" +local_addresses="$(ip -o -4 addr show | awk '{split($4,a,"/");print a[1]}')" +grep -Fxq "${PGPOOL_HOST}" <<<"${local_addresses}" || die "本机不存在 Pgpool 地址 ${PGPOOL_HOST}。" +[[ -f "${PGPOOL_CONF}" && ! -L "${PGPOOL_CONF}" && -f "${POOL_HBA}" && ! -L "${POOL_HBA}" ]] || \ + die 'Pgpool 配置缺失或为符号链接。' +[[ -r "${PGPOOL_CONF}" && -r "${POOL_HBA}" && -w "${POOL_HBA}" && -d /var && -w /var ]] || \ + die 'Pgpool 配置读取权限、pool_hba.conf 写权限或 /var 备份权限不足。' +local_available_kb="$(df -Pk /var | awk 'NR==2{print $4}')" +[[ "${local_available_kb}" =~ ^[0-9]+$ && "${local_available_kb}" -ge 10240 ]] || \ + die "/var 可用空间不足 10 MiB: ${local_available_kb:-unknown} KiB" +managed_block_is_well_formed "${POOL_HBA}" "${HBA_MARKER}" || die 'pool_hba.conf 的全库全用户受管标记不完整。' +systemctl is-active --quiet "${PGPOOL_SERVICE}" || die 'Pgpool 服务未运行。' +[[ -x "${PGPOOL_BIN}" ]] || die "Pgpool 主程序不存在: ${PGPOOL_BIN}" +pgpool_version="$(LD_LIBRARY_PATH="${PGPOOL_RUNTIME_LIB}" "${PGPOOL_BIN}" --version 2>&1)" +grep -Fq '4.7.2' <<<"${pgpool_version}" || die "Pgpool 版本不是 4.7.2: ${pgpool_version}" +[[ "$(config_value backend_clustering_mode "${PGPOOL_CONF}")" == streaming_replication ]] || die 'Pgpool 不是流复制模式。' +[[ "$(config_value backend_hostname0 "${PGPOOL_CONF}")" == "${PRIMARY_HOST}" ]] || die 'backend 0 与输入的 Primary 不一致。' +[[ "$(config_value backend_hostname1 "${PGPOOL_CONF}")" == "${STANDBY_HOST}" ]] || die 'backend 1 与输入的 Standby 不一致。' +[[ "$(config_value backend_port0 "${PGPOOL_CONF}")" == "${POSTGRESQL_PORT}" && \ + "$(config_value backend_port1 "${PGPOOL_CONF}")" == "${POSTGRESQL_PORT}" ]] || die 'Pgpool 后端端口与输入不一致。' +[[ "$(config_value port "${PGPOOL_CONF}")" == "${PGPOOL_PORT}" ]] || die 'Pgpool 对外端口与输入不一致。' +[[ "$(config_value enable_pool_hba "${PGPOOL_CONF}")" == on ]] || die 'enable_pool_hba 未开启。' +[[ "$(config_value allow_clear_text_frontend_auth "${PGPOOL_CONF}")" == off ]] || \ + die 'allow_clear_text_frontend_auth 必须保持 off。' +ss -lnt | grep -Eq "[[:space:]](0\\.0\\.0\\.0|\\*|\\[::\\]):${PGPOOL_PORT}[[:space:]]" || \ + die "Pgpool 未对外监听 ${PGPOOL_PORT}。" + +# 本项目生成的 pool_hba 只应包含 all/all + md5/password。遇到人工特例时拒绝猜测。 +invalid_pool_hba="$(awk ' + /^[[:space:]]*($|#)/ {next} + { + line=$0; sub(/[[:space:]]+#.*/, "", line); sub(/^[[:space:]]+/, "", line); sub(/[[:space:]]+$/, "", line); n=split(line,f,/[[:space:]]+/) + if (f[1]=="local") {if (n!=4 || f[2]!="all" || f[3]!="all" || (f[4]!="md5" && f[4]!="password")) print $0; next} + if (f[1]=="host") {if (n!=5 || f[2]!="all" || f[3]!="all" || (f[5]!="md5" && f[5]!="password")) print $0; next} + print $0 + } +' "${POOL_HBA}")" +[[ -z "${invalid_pool_hba}" ]] || die "pool_hba.conf 存在脚本不能安全泛化的规则:${invalid_pool_hba}" +active_pool_rules="$(awk '/^[[:space:]]*(local|host)[[:space:]]/{count++} END{print count+0}' "${POOL_HBA}")" +((active_pool_rules > 0)) || die 'pool_hba.conf 没有可保留的客户端来源规则。' +if [[ "$(config_value ssl "${PGPOOL_CONF}")" != on ]]; then + warn 'Pgpool 当前未启用 TLS;password 前端认证会在客户端到 Pgpool 链路上传输明文口令协议。' +fi + +SCRIPT_PHASE='准备临时 SSH 客户端并检查网络' +script_root="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +payload_path="${script_root}/${SSHPASS_PAYLOAD}" +[[ -f "${payload_path}" ]] || die "缺少离线 sshpass 载荷: ${payload_path}" +[[ "$(sha256sum "${payload_path}" | awk '{print $1}')" == "${SSHPASS_SHA256}" ]] || die 'sshpass 载荷 SHA256 不匹配。' +TEMP_ROOT="$(mktemp -d /var/tmp/pg-rw-all-users.XXXXXX)" +chmod 700 "${TEMP_ROOT}" +tar -xmzf "${payload_path}" -C "${TEMP_ROOT}" +SSHPASS_BIN="${TEMP_ROOT}/usr/local/bin/sshpass" +[[ -x "${SSHPASS_BIN}" ]] || die 'sshpass 载荷解压不完整。' +export SSHPASS="${ROOT_SSH_PASSWORD}" +known_hosts="${TEMP_ROOT}/known_hosts" +touch "${known_hosts}" +chmod 600 "${known_hosts}" +SSH_ARGS=(-p "${SSH_PORT}" -o PreferredAuthentications=password -o PubkeyAuthentication=no \ + -o StrictHostKeyChecking=no -o UserKnownHostsFile="${known_hosts}" -o ConnectTimeout=10 \ + -o ServerAliveInterval=15 -o ServerAliveCountMax=4) + +for backend_host in "${PRIMARY_HOST}" "${STANDBY_HOST}"; do + route_output="$(ip route get "${backend_host}")" + grep -Eq "(^|[[:space:]])src[[:space:]]+${PGPOOL_HOST}([[:space:]]|$)" <<<"${route_output}" || \ + die "到 ${backend_host} 的出站源地址不是 ${PGPOOL_HOST}: ${route_output}" + timeout 5 bash -c ""${pool_hba_temp}" +chown --reference="${POOL_HBA}" "${pool_hba_temp}" +chmod --reference="${POOL_HBA}" "${pool_hba_temp}" +mv -f -- "${pool_hba_temp}" "${POOL_HBA}" +remaining_md5="$(awk ' + /^[[:space:]]*($|#)/ {next} + {line=$0; sub(/[[:space:]]+#.*/, "", line); sub(/^[[:space:]]+/, "", line); sub(/[[:space:]]+$/, "", line); n=split(line,f,/[[:space:]]+/); if ((f[1]=="local" && f[4]=="md5") || (f[1]=="host" && f[5]=="md5")) count++} + END{print count+0} +' "${POOL_HBA}")" +password_rules="$(awk ' + /^[[:space:]]*($|#)/ {next} + {line=$0; sub(/[[:space:]]+#.*/, "", line); sub(/^[[:space:]]+/, "", line); sub(/[[:space:]]+$/, "", line); n=split(line,f,/[[:space:]]+/); if ((f[1]=="local" && f[4]=="password") || (f[1]=="host" && f[5]=="password")) count++} + END{print count+0} +' "${POOL_HBA}")" +[[ "${remaining_md5}" == 0 && "${password_rules}" == "${active_pool_rules}" ]] || die 'pool_hba.conf 转换后规则数量不一致。' +systemctl reload "${PGPOOL_SERVICE}" +sleep 2 +systemctl is-active --quiet "${PGPOOL_SERVICE}" || die 'Pgpool reload 后服务未运行。' +ss -lnt | grep -Eq "[[:space:]](0\\.0\\.0\\.0|\\*|\\[::\\]):${PGPOOL_PORT}[[:space:]]" || die 'Pgpool reload 后监听异常。' +if journalctl -u "${PGPOOL_SERVICE}" --since '-1 minute' --no-pager -o cat | \ + grep -Eiq 'failed to parse|configuration error|invalid authentication method|FATAL:.*pool_hba'; then + die 'Pgpool 最新日志出现配置或认证规则解析错误。' +fi + +SUCCESS='yes' +SCRIPT_PHASE='完成' +log '全库全用户透明认证已启用。' +log "Pgpool 备份=${LOCAL_BACKUP_DIR}" +log "Primary 备份=${PRIMARY_BACKUP_DIR}" +log "Standby 备份=${STANDBY_BACKUP_DIR}" +log '现有可登录角色现在可使用各自原密码,经 Pgpool 访问其本来有权连接的任意数据库。' +log '不具备 LOGIN、CONNECT 或对象权限的角色仍会被 PostgreSQL 正常拒绝。' +log '后续若运行 repair.sh,它会重新生成 pool_hba.conf;届时需重新运行本脚本。' diff --git a/repair.sh b/repair.sh index ca8ccd6..ddfeaa2 100644 --- a/repair.sh +++ b/repair.sh @@ -1,14 +1,14 @@ #!/usr/bin/env bash -# 与 install.sh 并列的中断续装入口。通常只修复 Pgpool;唯一数据库写操作是: -# 在严格证明 pg_basebackup 已完成后,启动/等待现有 Standby PGDATA,绝不重新同步。 +# 与 install.sh 并列的无状态中断续装入口。只依赖本次参数和三节点实时状态; +# REPAIR 后可轮换安装器受管凭据,并在严格证明 pg_basebackup 已完成后续启现有 Standby。 set -Eeuo pipefail IFS=$'\n\t' ROOT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" -CONFIG_DIR="${ROOT_DIR}/config" INSTALL_PHASE='启动参数与本机环境检查' TEMP_ROOT='' +session_config_dir='' SSHPASS_BIN='' CHECK_PSQL='' CHECK_LD_LIBRARY_PATH='' @@ -27,9 +27,30 @@ die() { } cleanup() { + local pair host remote_stage + if declare -F remote_root >/dev/null 2>&1; then + for pair in \ + "${PRIMARY_HOST:-}|${primary_credential_stage:-}" \ + "${STANDBY_HOST:-}|${standby_credential_stage:-}" \ + "${STANDBY_HOST:-}|${standby_resume_stage:-}"; do + host="${pair%%|*}"; remote_stage="${pair#*|}" + if [[ -z "${host}" ]] || \ + [[ "${remote_stage}" != /var/tmp/pg-rw-credential-repair-* && "${remote_stage}" != /var/tmp/pg-rw-standby-resume-* ]]; then + continue + fi + printf 'stage=%q\ncase "${stage}" in /var/tmp/pg-rw-credential-repair-*|/var/tmp/pg-rw-standby-resume-*) rm -rf -- "${stage}";; esac\n' \ + "${remote_stage}" | remote_root "${host}" >/dev/null 2>&1 || true + done + fi if [[ -n "${TEMP_ROOT:-}" && "${TEMP_ROOT}" == /var/tmp/pg-rw-repair.* && -d "${TEMP_ROOT}" ]]; then rm -rf --one-file-system -- "${TEMP_ROOT}" fi + if [[ -n "${session_config_dir:-}" && "${session_config_dir}" == /var/tmp/pg-rw-repair-config.* && -d "${session_config_dir}" ]]; then + if command -v shred >/dev/null 2>&1; then + find "${session_config_dir}" -type f -exec shred -u -- {} + 2>/dev/null || true + fi + rm -rf --one-file-system -- "${session_config_dir}" + fi unset SSHPASS ROOT_SSH_PASSWORD REQUEST_BUSINESS_PASSWORD BUSINESS_PASSWORD } @@ -53,15 +74,15 @@ trap cleanup EXIT INT TERM trap 'report_unexpected_error "$?" "$LINENO" "$BASH_COMMAND"' ERR INSTALLER_ENTRYPOINT_NAME='repair.sh' -INSTALLER_FINAL_BEHAVIOR='脚本先执行只读状态检测;正在同步时只报告并退出。若基础备份已完成但 Standby 启动中断,REPAIR 只续启当前 PGDATA;其余修复仅限 Pgpool,绝不重新执行基础备份。' -INSTALLER_ALLOWED_CLIENT_CIDRS_HELP='初次安装保存值,仅用于核对安装会话;实际规则从 Primary 动态读取' +INSTALLER_FINAL_BEHAVIOR='脚本只使用本次参数和三台服务器实时状态;正在同步时只报告并退出。REPAIR 会轮换安装器受管凭据、续启已完成基础备份的 Standby(如需要)并修复 Pgpool,绝不重新执行基础备份。' +INSTALLER_ALLOWED_CLIENT_CIDRS_HELP='本次完整参数;repair 实际规则从 Primary 动态读取' # shellcheck source=scripts/lib/installer-inputs.sh source "${ROOT_DIR}/scripts/lib/installer-inputs.sh" initialize_install_inputs parse_install_inputs "$@" [[ "${EUID}" -eq 0 ]] || die '请在 Pgpool-II 服务器上使用 root 运行 bash repair.sh。' -for command_name in ssh tar awk sed sha256sum mktemp getenforce getconf uname id stat date tee ip pgrep timeout grep cmp ss systemctl readlink install find md5sum chown chmod journalctl cp rm touch; do +for command_name in ssh scp tar openssl awk sed sha256sum mktemp getenforce getconf uname id stat date tee ip pgrep timeout grep cmp ss systemctl readlink install find md5sum chown chmod journalctl cp rm touch sleep; do command -v "${command_name}" >/dev/null 2>&1 || die "缺少系统基础命令: ${command_name}" done @@ -108,7 +129,7 @@ validate_request_port() { } printf '\nPostgreSQL Streaming Replication + Pgpool-II 状态检测与中断续装\n' -printf '本入口不会配置或重启 Primary,不会执行 pg_basebackup;必要时只续启已完成基础备份的 Standby PGDATA。\n\n' +printf '本入口不依赖上一次安装文件,不重启 Primary/Standby,不执行 pg_basebackup;REPAIR 后只轮换安装器受管凭据并按需续启已完成基础备份的 Standby PGDATA。\n\n' prompt_default PGPOOL_HOST '当前 Pgpool-II 服务器内网 IPv4 地址' '192.168.80.140' prompt_default PRIMARY_HOST '现有 PostgreSQL Primary 内网 IPv4 地址' '192.168.80.110' prompt_default STANDBY_HOST 'PostgreSQL Standby 目标机内网 IPv4 地址' '192.168.80.120' @@ -126,7 +147,7 @@ validate_request_port "${PRIMARY_PORT}" validate_request_port "${PGPOOL_PORT}" validate_request_port "${SSH_PORT}" [[ "${PGPOOL_PORT}" != '9898' ]] || die 'Pgpool 对外端口不能与本机 PCP 端口 9898 冲突。' -prompt_default ALLOWED_CLIENT_CIDRS '初次安装保存的客户端 IPv4 CIDR(仅用于安装会话校验;repair 以 Primary 实际策略为准)' "${PGPOOL_HOST%.*}.0/24" +prompt_default ALLOWED_CLIENT_CIDRS '客户端 IPv4 CIDR(repair 以 Primary 实际策略为准)' "${PGPOOL_HOST%.*}.0/24" IFS=',' read -r -a _allowed_cidrs <<<"${ALLOWED_CLIENT_CIDRS}" ((${#_allowed_cidrs[@]} > 0)) || die '客户端 CIDR 不能为空。' for cidr in "${_allowed_cidrs[@]}"; do @@ -144,7 +165,7 @@ validate_request_identifier "${BUSINESS_DATABASE}" prompt_secret ROOT_SSH_PASSWORD 'Primary 与 Standby 的 root SSH 公共密码' allow prompt_secret BUSINESS_PASSWORD "现有数据库用户 ${BUSINESS_USER} 的密码" -# 冻结本次输入;随后加载的已安装配置不得静默覆盖这些值。 +# 冻结本次输入。repair.sh 不读取、比对或依赖任何上一次安装生成的配置文件。 REQUEST_PGPOOL_HOST="${PGPOOL_HOST}" REQUEST_PRIMARY_HOST="${PRIMARY_HOST}" REQUEST_STANDBY_HOST="${STANDBY_HOST}" @@ -156,12 +177,66 @@ REQUEST_BUSINESS_USER="${BUSINESS_USER}" REQUEST_BUSINESS_DATABASE="${BUSINESS_DATABASE}" REQUEST_BUSINESS_PASSWORD="${BUSINESS_PASSWORD}" -CLUSTER_CONFIG="${CONFIG_DIR}/cluster.env" -SECRETS_CONFIG="${CONFIG_DIR}/secrets.env" -POOL_USERS_FILE="${CONFIG_DIR}/pool-users.txt" +umask 077 +write_env() { printf '%s=%q\n' "$2" "$3" >>"$1"; } +random_secret() { openssl rand -hex 32; } + +session_config_dir="$(mktemp -d /var/tmp/pg-rw-repair-config.XXXXXX)" +chmod 700 "${session_config_dir}" +cluster_file="${session_config_dir}/cluster.env" +secrets_file="${session_config_dir}/secrets.env" +pool_users_file="${session_config_dir}/pool-users.txt" +: >"${cluster_file}" + +declare -A values=( + [PG_MAJOR]='12' [PG_VERSION_FULL]='12.0' [DB_DISTRIBUTION]='NebulaCM' + [DBN_VERSION_MARKER]='NebulaCM_Dbn_PostgreSQL-install-runtime-12.0-ky10-aarch64-20241212.tar.gz' + [DB_POSTGRES_SHA256]='aecef1bcf6557271a4ffaf9b55eb53df81c883984e1489916c85e88677477bcf' + [DB_PG_BASEBACKUP_SHA256]='c235544bad0e0dc61b9c4fbc6becc7181970b9179b2aa071ff8a1821899815bb' + [DB_PG_CONFIG_SHA256]='90321c1cb01d583ffa55523fd54490f93b2cb17fbcbe5fd99319ef0812f224df' + [DB_TOOLS_SHA256]='1cfa544a74dc88f1bdc88db52fac5566b73eec4a0c3b15e33e60ec86a2cffa0f' + [PG_OS_USER]='postgres' [PGPOOL_MAJOR]='4.7' [PGPOOL_VERSION]='4.7.2' + [PGPOOL_INSTALL_PREFIX]='/opt/pgpool-II-4.7.2' [PG_CLIENT_PREFIX]='/opt/pgpool-client-12.0' [PGPOOL_RUNTIME_PREFIX]='/opt/pgpool-runtime-kylin-v10' + [OFFLINE_PACKAGE_DIR]='packages/payload' + [PGPOOL_PAYLOAD_FILE]='pgpool-II-4.7.2-pg12.0-aarch64-kylin-v10.tar.gz' [PGPOOL_PAYLOAD_SHA256]='b80c79b8e6537a14a6adc8ab4ae58baa40a2e027c8fc99a3589510462425dbea' + [PG_CLIENT_PAYLOAD_FILE]='postgresql-client-12.0-aarch64-kylin-v10.tar.gz' [PG_CLIENT_PAYLOAD_SHA256]='1314901d8b0de906fcc47c7784913fd347d07a3b563475f8eae4e16310ba8667' + [PGPOOL_RUNTIME_PAYLOAD_FILE]='pgpool-runtime-kylin-v10-aarch64.tar.gz' [PGPOOL_RUNTIME_PAYLOAD_SHA256]='6d00411d0098b2bab0c3f9e3a0d5d009907189eb2b0e3a743edd3063bce9a257' + [SSHPASS_PAYLOAD_FILE]='sshpass-1.10-aarch64-kylin-v10.tar.gz' [SSHPASS_PAYLOAD_SHA256]='84bcff17fc7e48d0a8552c985818e17e485f95a66b3c50c4eedde6bcbdc96ffd' + [SSH_PORT]="${REQUEST_SSH_PORT}" + [PRIMARY_HOST]="${REQUEST_PRIMARY_HOST}" [PRIMARY_PORT]="${REQUEST_PRIMARY_PORT}" [PRIMARY_PGDATA]='/pgsql/12/data' + [PRIMARY_PG_BIN_DIR]='/opt/pgsql12/bin' [PRIMARY_ADMIN_TOOL]='/opt/pgsql12/bin/tools' [PRIMARY_LISTEN_ADDRESSES]="127.0.0.1,${REQUEST_PRIMARY_HOST}" + [STANDBY_HOST]="${REQUEST_STANDBY_HOST}" [STANDBY_PORT]="${REQUEST_PRIMARY_PORT}" [STANDBY_PGDATA]='/pgsql/12/data' + [STANDBY_PG_BIN_DIR]='/opt/pgsql12/bin' [STANDBY_ADMIN_TOOL]='/opt/pgsql12/bin/tools' [STANDBY_LISTEN_ADDRESSES]="127.0.0.1,${REQUEST_STANDBY_HOST}" + [STANDBY_APPLICATION_NAME]='rw_standby' [REPLICATION_SLOT_NAME]='rw_standby_slot' + [PGPOOL_HOST]="${REQUEST_PGPOOL_HOST}" [PGPOOL_PORT]="${REQUEST_PGPOOL_PORT}" [PCP_PORT]='9898' [PGPOOL_SERVICE]='pgpool' [PGPOOL_CONFIG_DIR]='/etc/pgpool-II' + [STANDBY_ADDRESS_CIDR]="${REQUEST_STANDBY_HOST}/32" [PGPOOL_ADDRESS_CIDR]="${REQUEST_PGPOOL_HOST}/32" + [ALLOWED_CLIENT_CIDRS]="${REQUEST_ALLOWED_CLIENT_CIDRS}" + [REPLICATION_USER]='rw_replicator' [MONITOR_USER]='pgpool_monitor' + [BUSINESS_USER]="${REQUEST_BUSINESS_USER}" [BUSINESS_DATABASE]="${REQUEST_BUSINESS_DATABASE}" [PCP_USER]='pgpool_admin' + [MAX_WAL_SENDERS]='10' [MAX_REPLICATION_SLOTS]='10' [WAL_KEEP_SEGMENTS]='1000' + [PRIMARY_READ_WEIGHT]='0' [STANDBY_READ_WEIGHT]='1' [DISABLE_LOAD_BALANCE_ON_WRITE]='transaction' + [READ_LAG_THRESHOLD_SECONDS]='5' [SR_CHECK_PERIOD]='5' [HEALTH_CHECK_PERIOD]='5' [HEALTH_CHECK_TIMEOUT]='5' + [HEALTH_CHECK_MAX_RETRIES]='3' [HEALTH_CHECK_RETRY_DELAY]='1' [CONNECT_TIMEOUT_MS]='5000' + [NUM_INIT_CHILDREN]='8' [MAX_POOL]='2' [CONNECTION_LIFE_TIME]='600' [LOG_PER_NODE_STATEMENT]='on' + [APPLY_PRIMARY_RESTART]='no' [ALLOW_STANDBY_REINITIALIZE]='no' +) +order=(PG_MAJOR PG_VERSION_FULL DB_DISTRIBUTION DBN_VERSION_MARKER DB_POSTGRES_SHA256 DB_PG_BASEBACKUP_SHA256 DB_PG_CONFIG_SHA256 DB_TOOLS_SHA256 PG_OS_USER PGPOOL_MAJOR PGPOOL_VERSION PGPOOL_INSTALL_PREFIX PG_CLIENT_PREFIX PGPOOL_RUNTIME_PREFIX OFFLINE_PACKAGE_DIR PGPOOL_PAYLOAD_FILE PGPOOL_PAYLOAD_SHA256 PG_CLIENT_PAYLOAD_FILE PG_CLIENT_PAYLOAD_SHA256 PGPOOL_RUNTIME_PAYLOAD_FILE PGPOOL_RUNTIME_PAYLOAD_SHA256 SSHPASS_PAYLOAD_FILE SSHPASS_PAYLOAD_SHA256 SSH_PORT PRIMARY_HOST PRIMARY_PORT PRIMARY_PGDATA PRIMARY_PG_BIN_DIR PRIMARY_ADMIN_TOOL PRIMARY_LISTEN_ADDRESSES STANDBY_HOST STANDBY_PORT STANDBY_PGDATA STANDBY_PG_BIN_DIR STANDBY_ADMIN_TOOL STANDBY_LISTEN_ADDRESSES STANDBY_APPLICATION_NAME REPLICATION_SLOT_NAME PGPOOL_HOST PGPOOL_PORT PCP_PORT PGPOOL_SERVICE PGPOOL_CONFIG_DIR STANDBY_ADDRESS_CIDR PGPOOL_ADDRESS_CIDR ALLOWED_CLIENT_CIDRS REPLICATION_USER MONITOR_USER BUSINESS_USER BUSINESS_DATABASE PCP_USER MAX_WAL_SENDERS MAX_REPLICATION_SLOTS WAL_KEEP_SEGMENTS PRIMARY_READ_WEIGHT STANDBY_READ_WEIGHT DISABLE_LOAD_BALANCE_ON_WRITE READ_LAG_THRESHOLD_SECONDS SR_CHECK_PERIOD HEALTH_CHECK_PERIOD HEALTH_CHECK_TIMEOUT HEALTH_CHECK_MAX_RETRIES HEALTH_CHECK_RETRY_DELAY CONNECT_TIMEOUT_MS NUM_INIT_CHILDREN MAX_POOL CONNECTION_LIFE_TIME LOG_PER_NODE_STATEMENT APPLY_PRIMARY_RESTART ALLOW_STANDBY_REINITIALIZE) +for name in "${order[@]}"; do write_env "${cluster_file}" "${name}" "${values[${name}]}"; done + +: >"${secrets_file}" +write_env "${secrets_file}" REPLICATION_PASSWORD "$(random_secret)" +write_env "${secrets_file}" MONITOR_PASSWORD "$(random_secret)" +write_env "${secrets_file}" BUSINESS_PASSWORD "${REQUEST_BUSINESS_PASSWORD}" +write_env "${secrets_file}" PCP_PASSWORD "$(random_secret)" +write_env "${secrets_file}" PGPOOL_AES_KEY "$(random_secret)" +: >"${pool_users_file}" +chmod 600 "${cluster_file}" "${secrets_file}" "${pool_users_file}" + +CLUSTER_CONFIG="${cluster_file}" +SECRETS_CONFIG="${secrets_file}" +POOL_USERS_FILE="${pool_users_file}" export CLUSTER_CONFIG SECRETS_CONFIG POOL_USERS_FILE -[[ -f "${CLUSTER_CONFIG}" && -f "${SECRETS_CONFIG}" && -f "${POOL_USERS_FILE}" ]] || \ - die "缺少当前安装会话配置;必须保留 ${CONFIG_DIR}/cluster.env、secrets.env、pool-users.txt 才能在不重建 Standby 的前提下续装。" 20 +printf 'CHECK repair_session_config status=PASS source=current_command_only prior_install_files=ignored persistent_write=no old_config_dir_access=no\n' # shellcheck source=scripts/lib/common.sh source "${ROOT_DIR}/scripts/lib/common.sh" @@ -186,24 +261,6 @@ trap 'report_unexpected_error "$?" "$LINENO" "$BASH_COMMAND"' ERR load_cluster_config load_secrets -assert_input_matches() { - local name="$1" requested="$2" installed="$3" - [[ "${requested}" == "${installed}" ]] || \ - die "本次参数 ${name} 与当前安装配置不一致:输入=${requested},已安装=${installed}。为避免修错集群,拒绝继续。" 20 -} -assert_input_matches PGPOOL_HOST "${REQUEST_PGPOOL_HOST}" "${PGPOOL_HOST}" -assert_input_matches PRIMARY_HOST "${REQUEST_PRIMARY_HOST}" "${PRIMARY_HOST}" -assert_input_matches STANDBY_HOST "${REQUEST_STANDBY_HOST}" "${STANDBY_HOST}" -assert_input_matches POSTGRESQL_PORT "${REQUEST_PRIMARY_PORT}" "${PRIMARY_PORT}" -assert_input_matches STANDBY_PORT "${REQUEST_PRIMARY_PORT}" "${STANDBY_PORT}" -assert_input_matches PGPOOL_PORT "${REQUEST_PGPOOL_PORT}" "${PGPOOL_PORT}" -assert_input_matches SSH_PORT "${REQUEST_SSH_PORT}" "${SSH_PORT}" -assert_input_matches ALLOWED_CLIENT_CIDRS "${REQUEST_ALLOWED_CLIENT_CIDRS}" "${ALLOWED_CLIENT_CIDRS}" -assert_input_matches BUSINESS_USER "${REQUEST_BUSINESS_USER}" "${BUSINESS_USER}" -assert_input_matches BUSINESS_DATABASE "${REQUEST_BUSINESS_DATABASE}" "${BUSINESS_DATABASE}" -[[ "${REQUEST_BUSINESS_PASSWORD}" == "${BUSINESS_PASSWORD}" ]] || \ - die '本次 business password 与安装会话保存值不一致;拒绝用不一致的凭据重写 Pgpool。' 20 - local_addresses="$(ip -o -4 addr show | awk '{split($4,a,"/"); print a[1]}')" grep -Fxq "${PGPOOL_HOST}" <<<"${local_addresses}" || \ die "当前服务器没有配置输入的 Pgpool 地址 ${PGPOOL_HOST},拒绝在错误节点修复。" 20 @@ -234,6 +291,10 @@ remote_root() { "${SSHPASS_BIN}" -e ssh "${ssh_args[@]}" root@"${host}" bash -s } +primary_credential_stage="/var/tmp/pg-rw-credential-repair-$$-primary" +standby_credential_stage="/var/tmp/pg-rw-credential-repair-$$-standby" +standby_resume_stage="/var/tmp/pg-rw-standby-resume-$$" + stage_standby_resume_script() { local remote_stage="/var/tmp/pg-rw-standby-resume-$$" local resume_files=( @@ -252,7 +313,7 @@ stage_standby_resume_script() { root@"${STANDBY_HOST}":"${remote_stage}/${file}" >/dev/null done for file in cluster.env secrets.env pool-users.txt; do - "${SSHPASS_BIN}" -e scp "${scp_args[@]}" "${CONFIG_DIR}/${file}" \ + "${SSHPASS_BIN}" -e scp "${scp_args[@]}" "${session_config_dir}/${file}" \ root@"${STANDBY_HOST}":"${remote_stage}/session-config/${file}" >/dev/null done { @@ -264,6 +325,62 @@ stage_standby_resume_script() { printf '%s' "${remote_stage}" } +stage_managed_credential_script() { + local host="$1" role="$2" remote_stage + local script_file file + case "${role}" in + primary) script_file='scripts/11-rotate-managed-credentials.sh'; remote_stage="${primary_credential_stage}" ;; + standby) script_file='scripts/23-update-standby-replication-passfile.sh'; remote_stage="${standby_credential_stage}" ;; + *) die "未知凭据修复角色: ${role}" ;; + esac + { + printf 'set -Eeuo pipefail\n' + printf 'umask 077\n' + printf 'stage=%q\n' "${remote_stage}" + printf 'rm -rf -- "${stage}"\n' + printf 'install -d -o root -g root -m 700 "${stage}/scripts/lib" "${stage}/session-config"\n' + } | remote_root "${host}" + for file in "${script_file}" scripts/lib/common.sh; do + "${SSHPASS_BIN}" -e scp "${scp_args[@]}" "${ROOT_DIR}/${file}" \ + root@"${host}":"${remote_stage}/${file}" >/dev/null + done + for file in cluster.env secrets.env pool-users.txt; do + "${SSHPASS_BIN}" -e scp "${scp_args[@]}" "${session_config_dir}/${file}" \ + root@"${host}":"${remote_stage}/session-config/${file}" >/dev/null + done + { + printf 'set -Eeuo pipefail\n' + printf 'stage=%q\n' "${remote_stage}" + printf 'chmod 700 "${stage}/%s"\n' "${script_file}" + printf 'chmod 600 "${stage}/scripts/lib/common.sh" "${stage}/session-config/"*\n' + } | remote_root "${host}" + printf '%s' "${remote_stage}" +} + +run_staged_managed_credential_script() { + local host="$1" role="$2" remote_stage script_file output status + case "${role}" in + primary) script_file='scripts/11-rotate-managed-credentials.sh' ;; + standby) script_file='scripts/23-update-standby-replication-passfile.sh' ;; + *) die "未知凭据修复角色: ${role}" ;; + esac + remote_stage="$(stage_managed_credential_script "${host}" "${role}")" + set +e + output="$({ + printf 'cd %q\n' "${remote_stage}" + printf 'CLUSTER_CONFIG=%q SECRETS_CONFIG=%q POOL_USERS_FILE=%q ./%s\n' \ + "${remote_stage}/session-config/cluster.env" \ + "${remote_stage}/session-config/secrets.env" \ + "${remote_stage}/session-config/pool-users.txt" "${script_file}" + } | remote_root "${host}" 2>&1)" + status=$? + set -e + printf 'stage=%q\ncase "${stage}" in /var/tmp/pg-rw-credential-repair-[0-9]*-*) rm -rf -- "${stage}";; esac\n' \ + "${remote_stage}" | remote_root "${host}" >/dev/null 2>&1 || true + printf '%s\n' "${output}" + [[ "${status}" == 0 ]] || die "${role} 受管凭据修复失败(status=${status})。" 21 +} + cleanup_standby_resume_stage() { local remote_stage="$1" [[ "${remote_stage}" == /var/tmp/pg-rw-standby-resume-[0-9]* ]] || return 0 @@ -278,25 +395,17 @@ standby_resume_candidate_snapshot() { printf 'PORT=%q\n' "${STANDBY_PORT}" cat <<'REMOTE' set -Eeuo pipefail -state_file=/var/lib/pg-rw-proxy-installer/standby-bootstrap.state -[[ -f "${state_file}" && ! -L "${state_file}" ]] -[[ "$(stat -c '%U:%a' "${state_file}")" == 'root:600' ]] -grep -Eq '^(original_pgdata|partial_pgdata|original_or_partial_pgdata)=/var/backups/pg-readwrite-proxy-lab/standby-' "${state_file}" [[ -f "${PGDATA}/PG_VERSION" && "$(tr -d '\r\n' <"${PGDATA}/PG_VERSION")" == 12 ]] for file in postgresql.conf postgresql.auto.conf pg_hba.conf standby.signal global/pg_control; do [[ -f "${PGDATA}/${file}" && ! -L "${PGDATA}/${file}" ]] done -if grep -Fxq 'basebackup_complete=yes' "${state_file}"; then - evidence=state_marker -elif grep -Fqx '# BEGIN PG_RW_PROXY_RECOVERY' "${PGDATA}/postgresql.auto.conf" && \ - grep -Fqx '# END PG_RW_PROXY_RECOVERY' "${PGDATA}/postgresql.auto.conf" && \ - grep -Fqx '# BEGIN PG_RW_PROXY_INCLUDE' "${PGDATA}/postgresql.conf" && \ - grep -Fqx '# END PG_RW_PROXY_INCLUDE' "${PGDATA}/postgresql.conf" && \ - grep -Fq "cluster_name = 'rw-standby'" "${PGDATA}/conf.d/99-pg-rw-proxy.conf"; then - evidence=legacy_post_basebackup_managed_config -else - exit 3 -fi +grep -Fqx '# BEGIN PG_RW_PROXY_RECOVERY' "${PGDATA}/postgresql.auto.conf" +grep -Fqx '# END PG_RW_PROXY_RECOVERY' "${PGDATA}/postgresql.auto.conf" +grep -Fqx '# BEGIN PG_RW_PROXY_INCLUDE' "${PGDATA}/postgresql.conf" +grep -Fqx '# END PG_RW_PROXY_INCLUDE' "${PGDATA}/postgresql.conf" +[[ -f "${PGDATA}/conf.d/99-pg-rw-proxy.conf" && ! -L "${PGDATA}/conf.d/99-pg-rw-proxy.conf" ]] +grep -Fq "cluster_name = 'rw-standby'" "${PGDATA}/conf.d/99-pg-rw-proxy.conf" +evidence=live_pgdata_managed_recovery running=no runuser -u postgres -- env PATH="${BIN_DIR}:/usr/bin:/bin" "${BIN_DIR}/pg_ctl" -D "${PGDATA}" status >/dev/null 2>&1 && running=yes control_system_id="$(runuser -u postgres -- env PATH="${BIN_DIR}:/usr/bin:/bin" "${BIN_DIR}/pg_controldata" "${PGDATA}" | @@ -404,6 +513,50 @@ REMOTE } | remote_root "${host}" } +primary_managed_role_snapshot() { + { + printf 'ADMIN_TOOL=%q\n' "${PRIMARY_ADMIN_TOOL}" + printf 'PORT=%q\n' "${PRIMARY_PORT}" + printf 'REPLICATION_USER=%q\n' "${REPLICATION_USER}" + printf 'MONITOR_USER=%q\n' "${MONITOR_USER}" + cat <<'REMOTE' +set -Eeuo pipefail +roles="$("${ADMIN_TOOL}" psql -d postgres -p "${PORT}" -c \ + "select (select count(*) from pg_roles where rolname='${REPLICATION_USER}' and rolcanlogin and rolreplication), + (select count(*) from pg_roles where rolname='${MONITOR_USER}' and rolcanlogin and not rolreplication)")" +[[ "${roles}" == '1|1' ]] +printf 'MANAGED_ROLES=%s\n' "${roles}" +REMOTE + } | remote_root "${PRIMARY_HOST}" +} + +remote_credential_prerequisites() { + local host="$1" role="$2" + { + printf 'ROLE=%q\n' "${role}" + printf 'PG_OS_USER=%q\n' "${PG_OS_USER}" + if [[ "${role}" == primary ]]; then + printf 'PGDATA=%q\n' "${PRIMARY_PGDATA}" + else + printf 'PGDATA=%q\n' "${STANDBY_PGDATA}" + fi + cat <<'REMOTE' +set -Eeuo pipefail +for command_name in bash install rm chmod chown stat mktemp mv getent runuser sha256sum awk sed grep; do + command -v "${command_name}" >/dev/null 2>&1 +done +[[ -d /var/tmp && ! -L /var/tmp && -w /var/tmp ]] +[[ -f "${PGDATA}/PG_VERSION" && ! -L "${PGDATA}/PG_VERSION" ]] +if [[ "${ROLE}" == standby ]]; then + postgres_home="$(getent passwd "${PG_OS_USER}" | cut -d: -f6)" + [[ "${postgres_home}" == /* && -d "${postgres_home}" && ! -L "${postgres_home}" && -w "${postgres_home}" ]] + [[ "$(stat -c '%U' "${postgres_home}")" == "${PG_OS_USER}" ]] +fi +printf 'CREDENTIAL_PREREQUISITES=%s|ready\n' "${ROLE}" +REMOTE + } | remote_root "${host}" +} + primary_hba_policy_snapshot() { { printf 'ADMIN_TOOL=%q\n' "${PRIMARY_ADMIN_TOOL}" @@ -463,6 +616,27 @@ REMOTE } | remote_root "${PRIMARY_HOST}" } +managed_credential_target_snapshot() { + local primary_role_state standby_passfile_state + primary_role_state="$(primary_managed_role_snapshot)" || return 1 + standby_passfile_state="$({ + printf 'PG_OS_USER=%q\n' "${PG_OS_USER}" + cat <<'REMOTE' +set -Eeuo pipefail +postgres_home="$(getent passwd "${PG_OS_USER}" | cut -d: -f6)" +[[ "${postgres_home}" == /* && -d "${postgres_home}" && ! -L "${postgres_home}" ]] +passfile="${postgres_home}/.pgpass-rw-proxy" +if [[ -e "${passfile}" || -L "${passfile}" ]]; then + [[ -f "${passfile}" && ! -L "${passfile}" ]] + stat -c 'STANDBY_PASSFILE=%U|%G|%a|%s|%Y' "${passfile}" +else + printf 'STANDBY_PASSFILE=absent\n' +fi +REMOTE + } | remote_root "${STANDBY_HOST}")" || return 1 + printf '%s;%s' "${primary_role_state}" "${standby_passfile_state}" +} + query_direct() { local host="$1" port="$2" user="$3" password="$4" database="$5" sql="$6" env LD_LIBRARY_PATH="${CHECK_LD_LIBRARY_PATH}" PGPASSWORD="${password}" PGCONNECT_TIMEOUT=5 \ @@ -480,6 +654,22 @@ check_active_install_or_sync INSTALL_PHASE='只读校验 Primary、Standby 与 Streaming Replication' primary_snapshot="$(remote_database_snapshot "${PRIMARY_HOST}" primary)" || \ database_blocked primary_unreachable 'Primary 运行状态读取失败' +managed_role_snapshot="$(primary_managed_role_snapshot)" || \ + database_blocked managed_roles 'Primary 缺少安装器受管的复制/监控角色,repair.sh 不负责猜测创建新角色' +[[ "${managed_role_snapshot}" == 'MANAGED_ROLES=1|1' ]] || \ + database_blocked managed_roles "Primary 受管角色状态异常:${managed_role_snapshot:-empty}" +printf 'CHECK managed_roles status=PASS replication=yes monitor=yes source=primary_live_state\n' +primary_credential_prerequisites="$(remote_credential_prerequisites "${PRIMARY_HOST}" primary)" || \ + database_blocked primary_credential_prerequisites 'Primary 不满足受管凭据轮换所需命令、目录或权限' +standby_credential_prerequisites="$(remote_credential_prerequisites "${STANDBY_HOST}" standby)" || \ + database_blocked standby_credential_prerequisites 'Standby 不满足复制密码文件更新所需命令、目录或权限' +[[ "${primary_credential_prerequisites}" == 'CREDENTIAL_PREREQUISITES=primary|ready' && \ + "${standby_credential_prerequisites}" == 'CREDENTIAL_PREREQUISITES=standby|ready' ]] || \ + database_blocked credential_prerequisites '受管凭据前置条件返回值异常' +printf 'CHECK managed_credential_prerequisites status=PASS primary=ready standby=ready persistent_write=no\n' +managed_credential_baseline="$(managed_credential_target_snapshot)" || \ + database_blocked credential_fingerprint '无法生成受管凭据目标的只读状态指纹' +printf 'CHECK managed_credential_fingerprint status=PASS captured=yes persistent_write=no\n' standby_resume_needed=no standby_resume_evidence='' standby_resume_running='' @@ -533,10 +723,6 @@ else printf 'CHECK streaming_replication status=PASS application=%s slot=%s\n' "${STANDBY_APPLICATION_NAME}" "${REPLICATION_SLOT_NAME}" fi -monitor_primary="$(query_direct "${PRIMARY_HOST}" "${PRIMARY_PORT}" "${MONITOR_USER}" "${MONITOR_PASSWORD}" postgres 'select pg_is_in_recovery()')" || \ - database_blocked monitor_auth 'Pgpool 监控账号无法连接 Primary' -[[ "${monitor_primary}" == f ]] || database_blocked monitor_role "监控账号读取到的 Primary 角色异常:${monitor_primary}" - probe_contract_sql="select case when exists ( select 1 from pg_class c join pg_namespace n on n.oid=c.relnamespace where n.nspname='business' and c.relname='rw_probe' and c.relkind='r' @@ -551,16 +737,13 @@ business_primary="$(query_direct "${PRIMARY_HOST}" "${PRIMARY_PORT}" "${BUSINESS [[ "${business_primary}" == "${BUSINESS_USER}|f|rw-primary|compatible" ]] || \ database_blocked business_probe "Primary 业务账号或路由探针异常:${business_primary:-empty}" if [[ "${standby_resume_needed}" == no ]]; then - monitor_standby="$(query_direct "${STANDBY_HOST}" "${STANDBY_PORT}" "${MONITOR_USER}" "${MONITOR_PASSWORD}" postgres 'select pg_is_in_recovery()')" || \ - database_blocked monitor_auth 'Pgpool 监控账号无法连接 Standby' - [[ "${monitor_standby}" == t ]] || database_blocked monitor_role "监控账号读取到的 Standby 角色异常:${monitor_standby}" business_standby="$(query_direct "${STANDBY_HOST}" "${STANDBY_PORT}" "${BUSINESS_USER}" "${REQUEST_BUSINESS_PASSWORD}" "${BUSINESS_DATABASE}" "select current_user,pg_is_in_recovery(),current_setting('cluster_name'),(${probe_contract_sql})")" || \ database_blocked business_auth '业务账号无法连接 Standby' [[ "${business_standby}" == "${BUSINESS_USER}|t|rw-standby|compatible" ]] || \ database_blocked business_probe "Standby 业务账号或路由探针异常:${business_standby:-empty}" - printf 'CHECK database_credentials status=PASS monitor=yes business=yes probe=compatible\n' + printf 'CHECK database_credentials status=PASS business=yes probe=compatible managed_credentials=deferred_until_REPAIR\n' else - printf 'CHECK database_credentials status=PARTIAL primary_monitor=yes primary_business=yes standby=deferred_until_resume\n' + printf 'CHECK database_credentials status=PARTIAL primary_business=yes standby=deferred_until_resume managed_credentials=deferred_until_REPAIR\n' fi INSTALL_PHASE='从 Primary 只读提取客户端 IPv4 白名单' @@ -606,6 +789,8 @@ add_repair_item() { repair_items+=("${item}") } [[ "${standby_resume_needed}" == yes ]] && add_repair_item standby_resume_existing_pgdata +add_repair_item rotate_installer_managed_credentials +add_repair_item rebuild_session_configuration block_unsafe_pgpool() { printf 'REPAIR_RESULT=BLOCKED_PGPOOL_OWNERSHIP action=none reason=%s\n' "$1" @@ -719,23 +904,12 @@ verify_local_pgpool_routing() { fi } -if ((${#repair_items[@]} == 0)); then - if ! verify_local_pgpool_routing; then - add_repair_item pgpool_routing - else - printf 'REPAIR_DECISION=NONE\n' - printf 'REPAIR_RESULT=HEALTHY action=none entry=%s:%s log=%s\n' "${PGPOOL_HOST}" "${PGPOOL_PORT}" "${REPAIR_LOG_FILE}" - trap - ERR - exit 0 - fi -fi - printf '\n检测到以下可安全续装项:\n' printf ' - %s\n' "${repair_items[@]}" if [[ "${standby_resume_needed}" == yes ]]; then - printf '基础备份完整性证据已通过;修复会先启动/等待现有 Standby PGDATA,然后补齐 Pgpool。不会执行 pg_basebackup,也不会重建主从。\n' + printf '基础备份完整性证据已通过;修复会轮换安装器管理的复制/监控凭据、续启现有 Standby PGDATA并补齐 Pgpool。不会执行 pg_basebackup,也不会重建主从。\n' else - printf '数据库状态已通过;修复范围只包括当前 Pgpool 服务器的离线运行时、配置和 systemd 服务。\n' + printf '数据库状态已通过;修复会轮换安装器管理的复制/监控凭据、更新 Standby 私有复制密码文件并补齐 Pgpool。不会停止或重启数据库。\n' fi INSTALL_PHASE='等待中断续装授权' if ! read -r -p '输入 REPAIR 执行上述安全续装;其他输入只保留日志并退出: ' confirmation; then @@ -756,12 +930,40 @@ latest_primary_hba_sha256="$(primary_hba_source_sha256)" || \ die "Primary HBA 在检查与落盘之间发生变化:before=${PRIMARY_POLICY_SOURCE_SHA256} after=${latest_primary_hba_sha256};请重新运行 repair.sh。" 25 [[ "$(sha256sum "${primary_policy_rules_file}" | awk '{print $1}')" == "${PRIMARY_POLICY_SHA256}" ]] || \ die '本次 Pgpool 客户端策略文件在授权前发生变化,拒绝落盘。' 25 +latest_managed_credential_target="$(managed_credential_target_snapshot)" || \ + die '修复授权后无法重新生成受管凭据目标状态指纹。' 25 +[[ "${latest_managed_credential_target}" == "${managed_credential_baseline}" ]] || \ + die '受管角色或 Standby 复制密码文件在只读检查与 REPAIR 之间发生变化;请重新运行 repair.sh。' 25 if [[ "${service_active}" == yes ]]; then established_frontends="$(ss -Htn state established "sport = :${PGPOOL_PORT}" 2>/dev/null | awk 'END{print NR+0}')" [[ "${established_frontends}" == '0' ]] || \ die "Pgpool 当前仍有 ${established_frontends} 个外部 TCP 前端连接;请先停止客户端连接池后重试。" 23 fi +INSTALL_PHASE='轮换安装器受管凭据' +log '使用本次临时配置轮换复制/监控账号密码;不修改业务账号,不停止数据库,不中断当前复制连接。' +run_staged_managed_credential_script "${PRIMARY_HOST}" primary +run_staged_managed_credential_script "${STANDBY_HOST}" standby + +INSTALL_PHASE='验证本次监控凭据已生效' +monitor_primary='' +for ((elapsed=0; elapsed<60; elapsed+=2)); do + monitor_primary="$(query_direct "${PRIMARY_HOST}" "${PRIMARY_PORT}" "${MONITOR_USER}" "${MONITOR_PASSWORD}" postgres 'select pg_is_in_recovery()' 2>/dev/null || true)" + [[ "${monitor_primary}" == f ]] && break + sleep 2 +done +[[ "${monitor_primary}" == f ]] || die '本次监控凭据在 Primary 上未生效。' 21 +if [[ "${standby_resume_needed}" == no ]]; then + monitor_standby='' + for ((elapsed=0; elapsed<60; elapsed+=2)); do + monitor_standby="$(query_direct "${STANDBY_HOST}" "${STANDBY_PORT}" "${MONITOR_USER}" "${MONITOR_PASSWORD}" postgres 'select pg_is_in_recovery()' 2>/dev/null || true)" + [[ "${monitor_standby}" == t ]] && break + sleep 2 + done + [[ "${monitor_standby}" == t ]] || die '本次监控凭据未及时复制到 Standby 或认证失败。' 21 +fi +printf 'CHECK managed_credentials status=PASS source=current_command database_restart=no basebackup=not_run\n' + if [[ "${standby_resume_needed}" == yes ]]; then INSTALL_PHASE='续启已完成基础备份的 Standby' standby_resume_stage="$(stage_standby_resume_script)" @@ -776,6 +978,7 @@ if [[ "${standby_resume_needed}" == yes ]]; then standby_resume_status=$? set -e cleanup_standby_resume_stage "${standby_resume_stage}" + standby_resume_stage='' printf '%s\n' "${standby_resume_output}" [[ "${standby_resume_status}" == 0 && "${standby_resume_output}" == *'STANDBY_RESUME_RESULT=READY'* && \ "${standby_resume_output}" == *'basebackup=not_run'* ]] || \ @@ -802,9 +1005,9 @@ INSTALL_PHASE='修复 Pgpool 配置、认证与服务状态' INSTALL_PHASE='修复后完整路由与外部入口验收' verify_local_pgpool_routing || die '续装修复完成后本机路由验收仍失败;请保留完整日志。' 24 -repair_action='pgpool_only' -[[ "${standby_resume_needed}" == yes ]] && repair_action='resume_existing_standby_then_pgpool' -printf 'REPAIR_RESULT=REPAIRED action=%s basebackup=not_run entry=%s:%s log=%s\n' \ +repair_action='rotate_managed_credentials_then_pgpool' +[[ "${standby_resume_needed}" == yes ]] && repair_action='rotate_managed_credentials_resume_existing_standby_then_pgpool' +printf 'REPAIR_RESULT=REPAIRED action=%s config_source=current_command_only prior_install_files=ignored basebackup=not_run database_restart=no entry=%s:%s log=%s\n' \ "${repair_action}" "${PGPOOL_HOST}" "${PGPOOL_PORT}" "${REPAIR_LOG_FILE}" trap - ERR exit 0 diff --git a/scripts/11-rotate-managed-credentials.sh b/scripts/11-rotate-managed-credentials.sh new file mode 100644 index 0000000..42bbfb8 --- /dev/null +++ b/scripts/11-rotate-managed-credentials.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash + +# repair.sh 的受限 Primary 动作:只轮换安装器拥有的复制/监控角色密码,不改业务账号, +# 不修改配置,不 reload/restart 数据库,也不终止现有复制连接。 +set -Eeuo pipefail +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib/common.sh +source "${SCRIPT_DIR}/lib/common.sh" + +require_root +load_cluster_config +load_secrets +verify_nebula_runtime "${PRIMARY_PG_BIN_DIR}" "${PRIMARY_ADMIN_TOOL}" Primary +pg_ctl_is_running "${PRIMARY_PG_BIN_DIR}" "${PRIMARY_PGDATA}" || die 'Primary 未运行。' + +identity="$(nebula_admin_query "${PRIMARY_ADMIN_TOOL}" "${PRIMARY_PORT}" postgres \ + "select current_setting('server_version_num'),pg_is_in_recovery(),current_setting('cluster_name'), + (select count(*) from pg_roles where rolname='${REPLICATION_USER}' and rolcanlogin and rolreplication), + (select count(*) from pg_roles where rolname='${MONITOR_USER}' and rolcanlogin and not rolreplication)")" +[[ "${identity}" == '120000|f|rw-primary|1|1' ]] || die "Primary 身份或受管角色异常: ${identity}" + +temp_sql="$(mktemp /var/tmp/pg-rw-credential-rotation.XXXXXX.sql)" +cleanup() { rm -f -- "${temp_sql}"; } +trap cleanup EXIT +rep_pass_sql="$(sql_literal "${REPLICATION_PASSWORD}")" +monitor_pass_sql="$(sql_literal "${MONITOR_PASSWORD}")" +cat >"${temp_sql}" </dev/null +printf 'MANAGED_CREDENTIAL_RESULT=ROTATED roles=replication,monitor database_restart=no replication_connection_terminated=no\n' diff --git a/scripts/22-resume-standby-after-basebackup.sh b/scripts/22-resume-standby-after-basebackup.sh index 8e7ff7e..8d0c618 100644 --- a/scripts/22-resume-standby-after-basebackup.sh +++ b/scripts/22-resume-standby-after-basebackup.sh @@ -16,8 +16,6 @@ done assert_safe_pgdata "${STANDBY_PGDATA}" verify_nebula_runtime "${STANDBY_PG_BIN_DIR}" "${STANDBY_ADMIN_TOOL}" 'Standby 续启节点' -state_file='/var/lib/pg-rw-proxy-installer/standby-bootstrap.state' -validate_standby_resume_state || die "缺少有效 Standby 中断状态: ${state_file}" [[ -f "${STANDBY_PGDATA}/PG_VERSION" && "$(tr -d '\r\n' <"${STANDBY_PGDATA}/PG_VERSION")" == '12' ]] || \ die 'Standby PG_VERSION 缺失或版本不等于 12。' for file in postgresql.conf postgresql.auto.conf pg_hba.conf standby.signal global/pg_control; do @@ -27,18 +25,17 @@ done [[ "$(stat -c '%U:%G:%a' "${STANDBY_PGDATA}")" == "${PG_OS_USER}:${PG_OS_USER}:700" ]] || \ die "Standby PGDATA 所有者或权限异常: $(stat -c '%U:%G:%a' "${STANDBY_PGDATA}")" -if grep -Fxq 'basebackup_complete=yes' "${state_file}"; then - resume_evidence='state_marker' -elif grep -Fqx '# BEGIN PG_RW_PROXY_RECOVERY' "${STANDBY_PGDATA}/postgresql.auto.conf" && \ - grep -Fqx '# END PG_RW_PROXY_RECOVERY' "${STANDBY_PGDATA}/postgresql.auto.conf" && \ - grep -Fqx '# BEGIN PG_RW_PROXY_INCLUDE' "${STANDBY_PGDATA}/postgresql.conf" && \ - grep -Fqx '# END PG_RW_PROXY_INCLUDE' "${STANDBY_PGDATA}/postgresql.conf" && \ - grep -Fq "cluster_name = 'rw-standby'" "${STANDBY_PGDATA}/conf.d/99-pg-rw-proxy.conf"; then - # 兼容 2026-08-13 旧安装器:standby.signal 只在 pg_basebackup 返回 0 后创建, - # 上述受管配置随后写入,最后才调用 pg_ctl。因此它们共同证明基础备份已完成。 - resume_evidence='legacy_post_basebackup_managed_config' +if grep -Fqx '# BEGIN PG_RW_PROXY_RECOVERY' "${STANDBY_PGDATA}/postgresql.auto.conf" && \ + grep -Fqx '# END PG_RW_PROXY_RECOVERY' "${STANDBY_PGDATA}/postgresql.auto.conf" && \ + grep -Fqx '# BEGIN PG_RW_PROXY_INCLUDE' "${STANDBY_PGDATA}/postgresql.conf" && \ + grep -Fqx '# END PG_RW_PROXY_INCLUDE' "${STANDBY_PGDATA}/postgresql.conf" && \ + [[ -f "${STANDBY_PGDATA}/conf.d/99-pg-rw-proxy.conf" && ! -L "${STANDBY_PGDATA}/conf.d/99-pg-rw-proxy.conf" ]] && \ + grep -Fq "cluster_name = 'rw-standby'" "${STANDBY_PGDATA}/conf.d/99-pg-rw-proxy.conf"; then + # standby.signal 仅在 pg_basebackup 返回 0 后创建,受管恢复配置随后写入;这些实时 PGDATA + # 证据与 pg_control/Primary system identifier 共同证明当前基础备份可续启,无需安装状态文件。 + resume_evidence='live_pgdata_managed_recovery' else - die '没有基础备份完成标记或旧安装器的完整受管配置证据;拒绝猜测 PGDATA 是否完整。' + die '当前 PGDATA 缺少完整受管恢复配置证据;拒绝猜测基础备份是否完整。' fi export PGPASSWORD="${REPLICATION_PASSWORD}" @@ -49,9 +46,6 @@ unset PGPASSWORD IFS='|' read -r primary_recovery expected_primary_system_id <<<"${primary_identity}" [[ "${primary_recovery}" == f && "${expected_primary_system_id}" =~ ^[0-9]+$ ]] || \ die "无法确认当前 Primary 身份: ${primary_identity:-empty}" -recorded_primary_system_id="$(sed -n 's/^primary_system_id=//p' "${state_file}" | tail -n 1)" -[[ -z "${recorded_primary_system_id}" || "${recorded_primary_system_id}" == "${expected_primary_system_id}" ]] || \ - die "恢复状态记录的 Primary=${recorded_primary_system_id} 与当前 Primary=${expected_primary_system_id} 不一致。" control_system_id="$(run_as_pg "${STANDBY_PG_BIN_DIR}" "${STANDBY_PG_BIN_DIR}/pg_controldata" "${STANDBY_PGDATA}" | awk -F: '/Database system identifier/{gsub(/[[:space:]]/,"",$2);print $2}')" [[ "${control_system_id}" == "${expected_primary_system_id}" ]] || \ @@ -80,6 +74,5 @@ done [[ "${standby}" == "120000|rw-standby|t|${expected_primary_system_id}|streaming" ]] || \ die "Standby 已可连接但在 300 秒内未达到预期 streaming 恢复态: ${standby}" -rm -f -- "${state_file}" printf 'STANDBY_RESUME_RESULT=READY action=start_or_wait_existing_pgdata basebackup=not_run evidence=%s state=%s\n' \ "${resume_evidence}" "${standby}" diff --git a/scripts/23-update-standby-replication-passfile.sh b/scripts/23-update-standby-replication-passfile.sh new file mode 100644 index 0000000..bd912ed --- /dev/null +++ b/scripts/23-update-standby-replication-passfile.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash + +# repair.sh 的受限 Standby 动作:只更新 postgres 私有复制密码文件,不停止或重启数据库。 +set -Eeuo pipefail +SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib/common.sh +source "${SCRIPT_DIR}/lib/common.sh" + +require_root +load_cluster_config +load_secrets +verify_nebula_runtime "${STANDBY_PG_BIN_DIR}" "${STANDBY_ADMIN_TOOL}" Standby +assert_safe_pgdata "${STANDBY_PGDATA}" +[[ -f "${STANDBY_PGDATA}/PG_VERSION" && "$(tr -d '\r\n' <"${STANDBY_PGDATA}/PG_VERSION")" == '12' ]] || \ + die 'Standby PGDATA 不完整,拒绝写入复制密码文件。' + +postgres_home="$(getent passwd "${PG_OS_USER}" | cut -d: -f6)" +[[ -n "${postgres_home}" && "${postgres_home}" == /* && -d "${postgres_home}" && ! -L "${postgres_home}" ]] || \ + die "无法确定 ${PG_OS_USER} HOME。" +[[ "$(stat -c '%U' "${postgres_home}")" == "${PG_OS_USER}" ]] || die "${PG_OS_USER} HOME 所有者异常。" +passfile="${postgres_home}/.pgpass-rw-proxy" +temp_passfile="$(mktemp "${postgres_home}/.pgpass-rw-proxy.XXXXXX")" +cleanup() { rm -f -- "${temp_passfile}"; } +trap cleanup EXIT +printf '%s:%s:replication:%s:%s\n' \ + "${PRIMARY_HOST}" "${PRIMARY_PORT}" "${REPLICATION_USER}" "${REPLICATION_PASSWORD}" >"${temp_passfile}" +chown "${PG_OS_USER}:${PG_OS_USER}" "${temp_passfile}" +chmod 600 "${temp_passfile}" +mv -f -- "${temp_passfile}" "${passfile}" +printf 'STANDBY_PASSFILE_RESULT=UPDATED database_restart=no basebackup=not_run\n' diff --git a/scripts/90-package-offline.sh b/scripts/90-package-offline.sh index 3761290..20147d8 100644 --- a/scripts/90-package-offline.sh +++ b/scripts/90-package-offline.sh @@ -22,14 +22,15 @@ trap cleanup EXIT mkdir -p "${DIST}" "${STAGE}/${NAME}/packages/payload" "${STAGE}/${NAME}/packages/sources" project_files=( - README.md install.sh repair.sh + README.md install.sh repair.sh enable-all-databases-users.sh diagnose-pgpool-port.sh config/cluster.env.example config/secrets.env.example config/pool-users.txt.example docs/TEST_DATA.md docs/acceptance-report.md docs/failure-and-delay-tests.md docs/production-notes.md packages/README.md packages/MANIFEST.sha256 packages/SOURCES.sha256 packages/build-kylin-v10-payloads.sh scripts/00-preflight.sh scripts/05-readiness-check.sh scripts/06-state-fingerprint.sh scripts/07-count-business-sessions.sh scripts/10-configure-primary.sh + scripts/11-rotate-managed-credentials.sh scripts/20-install-postgresql-standby.sh scripts/21-bootstrap-standby.sh - scripts/22-resume-standby-after-basebackup.sh + scripts/22-resume-standby-after-basebackup.sh scripts/23-update-standby-replication-passfile.sh scripts/30-install-pgpool.sh scripts/31-configure-pgpool.sh scripts/40-verify-cluster.sh scripts/41-observe-cluster.sh scripts/42-verify-external-entry.sh scripts/43-test-delay.sh scripts/44-test-standby-health.sh scripts/50-reset-primary-test-data.sh @@ -37,7 +38,8 @@ project_files=( scripts/lib/primary-client-policy.sh scripts/tests/test-tar-clock-skew.sh scripts/tests/test-no-firewall-coupling.sh scripts/tests/test-probe-identity-contract.sh scripts/tests/test-repair-no-resync-contract.sh - scripts/tests/test-primary-client-policy.sh + scripts/tests/test-primary-client-policy.sh scripts/tests/test-repair-stateless-contract.sh + scripts/tests/test-enable-all-databases-users-contract.sh scripts/sql/primary-test-data.sql scripts/sql/verify-primary-test-data.sql templates/pgpool.conf.tpl templates/pool_hba.conf.tpl templates/primary-pg_hba.entries.tpl templates/primary-postgresql.conf.tpl templates/standby-postgresql.conf.tpl @@ -59,7 +61,9 @@ done <"${ROOT}/packages/SOURCES.sha256" find "${STAGE}/${NAME}" -type f \( -name '*.sh' -o -name '*.sql' -o -name '*.tpl' -o -name '*.env.example' -o -name '*.sha256' \) -exec sed -i 's/\r$//' {} + find "${STAGE}/${NAME}" -type d -exec chmod 755 {} + find "${STAGE}/${NAME}" -type f -exec chmod 644 {} + -chmod 755 "${STAGE}/${NAME}/install.sh" "${STAGE}/${NAME}/repair.sh" "${STAGE}/${NAME}/scripts/"*.sh \ +chmod 755 "${STAGE}/${NAME}/install.sh" "${STAGE}/${NAME}/repair.sh" \ + "${STAGE}/${NAME}/enable-all-databases-users.sh" "${STAGE}/${NAME}/diagnose-pgpool-port.sh" \ + "${STAGE}/${NAME}/scripts/"*.sh \ "${STAGE}/${NAME}/scripts/lib/"*.sh "${STAGE}/${NAME}/scripts/tests/"*.sh (cd "${STAGE}/${NAME}/packages" && sha256sum -c MANIFEST.sha256 && sha256sum -c SOURCES.sha256) # 外层归档使用固定历史 mtime,避免在时钟略慢的隔离区服务器上首次解压时出现 diff --git a/scripts/tests/test-enable-all-databases-users-contract.sh b/scripts/tests/test-enable-all-databases-users-contract.sh new file mode 100644 index 0000000..c04eec1 --- /dev/null +++ b/scripts/tests/test-enable-all-databases-users-contract.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash + +set -Eeuo pipefail +IFS=$'\n\t' + +ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)" +SCRIPT="${ROOT}/enable-all-databases-users.sh" +PACKAGE_SCRIPT="${ROOT}/scripts/90-package-offline.sh" + +[[ -f "${SCRIPT}" ]] + +for required_text in \ + "readonly HBA_MARKER='PG_RW_PROXY_ALL_DATABASE_USERS'" \ + "host all all" \ + 'pg_reload_conf()' \ + 'systemctl reload "${PGPOOL_SERVICE}"' \ + 'read -r -p '\''输入 APPLY 执行;其他输入直接退出且不修改配置:' \ + 'REMOTE_SNAPSHOT=' \ + 'system identifier 不一致' \ + 'allow_clear_text_frontend_auth' \ + 'password_rules'; do + grep -Fq "${required_text}" "${SCRIPT}" || { + printf '全库全用户脚本缺少合同: %s\n' "${required_text}" >&2 + exit 1 + } +done + +grep -Fq '不重启数据库,不重新同步 Standby' "${SCRIPT}" +for forbidden_pattern in \ + 'systemctl[[:space:]]+restart' \ + 'pg_ctl[[:space:]]+(stop|restart)' \ + '^[[:space:]]*(CREATE|ALTER)[[:space:]]+ROLE' \ + '^[[:space:]]*GRANT[[:space:]]+ALL' \ + '^[[:space:]]*(local|host)[[:space:]].*[[:space:]]trust([[:space:]]|$)' \ + '(^|[;&|])[[:space:]]*([^[:space:]]*/)?pg_basebackup([[:space:]]|$)'; do + ! grep -Eq "${forbidden_pattern}" "${SCRIPT}" || { + printf '全库全用户脚本包含禁止动作模式: %s\n' "${forbidden_pattern}" >&2 + exit 1 + } +done + +grep -Fq 'enable-all-databases-users.sh' "${PACKAGE_SCRIPT}" +grep -Fq 'enable-all-databases-users.sh' "${ROOT}/README.md" + +printf 'ENABLE_ALL_DATABASES_USERS_CONTRACT_OK\n' diff --git a/scripts/tests/test-repair-no-resync-contract.sh b/scripts/tests/test-repair-no-resync-contract.sh index 62e9e0b..81aba4a 100644 --- a/scripts/tests/test-repair-no-resync-contract.sh +++ b/scripts/tests/test-repair-no-resync-contract.sh @@ -11,12 +11,18 @@ package_script="${ROOT}/scripts/90-package-offline.sh" for required_text in \ 'REQUEST_PRIMARY_HOST' \ - 'assert_input_matches' \ + 'source=current_command_only' \ + 'prior_install_files=ignored' \ 'WAITING_FOR_CURRENT_INSTALL_OR_BASEBACKUP' \ 'BLOCKED_DATABASE' \ 'BLOCKED_PRIMARY_POLICY' \ 'primary_hba_policy_snapshot' \ + 'primary_managed_role_snapshot' \ + 'remote_credential_prerequisites' \ + 'managed_credential_target_snapshot' \ + 'scripts/11-rotate-managed-credentials.sh' \ 'scripts/22-resume-standby-after-basebackup.sh' \ + 'scripts/23-update-standby-replication-passfile.sh' \ 'basebackup=not_run' \ 'PGPOOL_CLIENT_POLICY_FILE' \ 'scripts/30-install-pgpool.sh' \ @@ -28,6 +34,16 @@ for required_text in \ } done +for forbidden_dependency in \ + 'source "${CONFIG_DIR}/cluster.env"' \ + 'source "${CONFIG_DIR}/secrets.env"' \ + '"${CONFIG_DIR}/${file}"'; do + if grep -Fq "${forbidden_dependency}" "${repair_script}"; then + printf 'repair.sh 禁止依赖上一次安装文件: %s\n' "${forbidden_dependency}" >&2 + exit 1 + fi +done + resume_script="${ROOT}/scripts/22-resume-standby-after-basebackup.sh" [[ -f "${resume_script}" ]] || { printf '缺少 Standby 基础备份后续启脚本。\n' >&2 @@ -60,6 +76,19 @@ if grep -Eq 'pg_ctl_(stop|start)|pg_ctl[^[:cntrl:]]+(-m[[:space:]]+fast[[:space: exit 1 fi +for credential_script in \ + "${ROOT}/scripts/11-rotate-managed-credentials.sh" \ + "${ROOT}/scripts/23-update-standby-replication-passfile.sh"; do + [[ -f "${credential_script}" ]] || { + printf '缺少受管凭据修复脚本: %s\n' "${credential_script}" >&2 + exit 1 + } + if grep -Eq 'pg_basebackup|pg_ctl[^[:cntrl:]]+[[:space:]]+(stop|start|restart)([[:space:]]|$)' "${credential_script}"; then + printf '受管凭据修复脚本禁止基础备份或数据库启停: %s\n' "${credential_script}" >&2 + exit 1 + fi +done + if grep -Eq 'runuser[^[:cntrl:]]+pg_basebackup|(^|[;&|])[[:space:]]*[^#[:cntrl:]]*/pg_basebackup([[:space:]]|$)' "${repair_script}"; then printf 'repair.sh 禁止执行 pg_basebackup。\n' >&2 exit 1 @@ -82,9 +111,21 @@ grep -Fq 'scripts/22-resume-standby-after-basebackup.sh' "${package_script}" || printf '离线包白名单缺少 Standby 基础备份后续启脚本。\n' >&2 exit 1 } +grep -Fq 'scripts/11-rotate-managed-credentials.sh' "${package_script}" || { + printf '离线包白名单缺少 Primary 受管凭据轮换脚本。\n' >&2 + exit 1 +} +grep -Fq 'scripts/23-update-standby-replication-passfile.sh' "${package_script}" || { + printf '离线包白名单缺少 Standby 复制密码文件更新脚本。\n' >&2 + exit 1 +} grep -Fq 'scripts/tests/test-primary-client-policy.sh' "${package_script}" || { printf '离线包白名单缺少 Primary 客户端策略测试。\n' >&2 exit 1 } +grep -Fq 'scripts/tests/test-repair-stateless-contract.sh' "${package_script}" || { + printf '离线包白名单缺少 repair 无状态合同测试。\n' >&2 + exit 1 +} printf 'REPAIR_NO_RESYNC_CONTRACT_OK\n' diff --git a/scripts/tests/test-repair-stateless-contract.sh b/scripts/tests/test-repair-stateless-contract.sh new file mode 100644 index 0000000..ec84206 --- /dev/null +++ b/scripts/tests/test-repair-stateless-contract.sh @@ -0,0 +1,45 @@ +#!/usr/bin/env bash + +set -Eeuo pipefail +IFS=$'\n\t' + +ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)" +repair_script="${ROOT}/repair.sh" + +for required_text in \ + 'source=current_command_only' \ + 'prior_install_files=ignored' \ + 'old_config_dir_access=no' \ + 'mktemp -d /var/tmp/pg-rw-repair-config.XXXXXX' \ + 'write_env "${secrets_file}" REPLICATION_PASSWORD "$(random_secret)"' \ + 'write_env "${secrets_file}" MONITOR_PASSWORD "$(random_secret)"' \ + 'write_env "${secrets_file}" BUSINESS_PASSWORD "${REQUEST_BUSINESS_PASSWORD}"' \ + 'rotate_installer_managed_credentials' \ + 'scripts/11-rotate-managed-credentials.sh' \ + 'scripts/23-update-standby-replication-passfile.sh'; do + grep -Fq "${required_text}" "${repair_script}" || { + printf 'repair.sh 缺少无状态合同: %s\n' "${required_text}" >&2 + exit 1 + } +done + +for forbidden_text in \ + 'CLUSTER_CONFIG="${CONFIG_DIR}/cluster.env"' \ + 'SECRETS_CONFIG="${CONFIG_DIR}/secrets.env"' \ + 'POOL_USERS_FILE="${CONFIG_DIR}/pool-users.txt"' \ + 'assert_input_matches' \ + '必须保留 ${CONFIG_DIR}' \ + '"${CONFIG_DIR}/${file}"'; do + if grep -Fq "${forbidden_text}" "${repair_script}"; then + printf 'repair.sh 仍依赖上一次安装文件: %s\n' "${forbidden_text}" >&2 + exit 1 + fi +done + +if grep -Fq 'standby-bootstrap.state' "${repair_script}" || \ + grep -Fq 'standby-bootstrap.state' "${ROOT}/scripts/22-resume-standby-after-basebackup.sh"; then + printf 'repair 续启禁止依赖上一次安装状态文件。\n' >&2 + exit 1 +fi + +printf 'REPAIR_STATELESS_CONTRACT_OK\n' diff --git a/vm/run-enable-all-databases-users-test.ps1 b/vm/run-enable-all-databases-users-test.ps1 new file mode 100644 index 0000000..2c7b98b --- /dev/null +++ b/vm/run-enable-all-databases-users-test.ps1 @@ -0,0 +1,267 @@ +[CmdletBinding()] +param() + +Set-StrictMode -Version Latest +$ErrorActionPreference = 'Stop' + +$projectRoot = [IO.Path]::GetFullPath((Join-Path $PSScriptRoot '..')) +$key = Join-Path $PSScriptRoot 'keys\lab_rsa' +$rootPasswordFile = Join-Path $PSScriptRoot 'keys\root-password.txt' +$entrypoint = Join-Path $projectRoot 'enable-all-databases-users.sh' +$payload = Join-Path $projectRoot 'packages\payload\sshpass-1.10-aarch64-kylin-v10.tar.gz' +foreach ($required in @($key, $rootPasswordFile, $entrypoint, $payload)) { + if (-not (Test-Path -LiteralPath $required -PathType Leaf)) { + throw "缺少测试文件:$required" + } +} + +$runId = [DateTime]::UtcNow.ToString('yyyyMMddHHmmss') + '-' + $PID +$probeRole = 'pgpool_probe_' + [DateTime]::UtcNow.ToString('HHmmss') +$remoteDir = "/var/tmp/pg-rw-all-users-e2e-$runId" +$probeSecretFile = Join-Path $env:TEMP "pg-rw-all-users-$runId.secret" +$randomBytes = [byte[]]::new(24) +[Security.Cryptography.RandomNumberGenerator]::Fill($randomBytes) +[IO.File]::WriteAllText( + $probeSecretFile, + [Convert]::ToHexString($randomBytes).ToLowerInvariant(), + [Text.UTF8Encoding]::new($false) +) + +$commonOptions = @( + '-i', $key, + '-o', 'BatchMode=yes', + '-o', 'ConnectTimeout=10', + '-o', 'StrictHostKeyChecking=no', + '-o', 'UserKnownHostsFile=NUL' +) +$kylinSsh = @($commonOptions + @('-p', '22021', 'labadmin@127.0.0.1')) +$kylinScp = @($commonOptions + @('-P', '22021')) +$primarySsh = @($commonOptions + @('-p', '22011', 'labadmin@127.0.0.1')) +$primaryScp = @($commonOptions + @('-P', '22011')) + +function Invoke-RemoteScript { + param( + [Parameter(Mandatory)] [string[]] $SshOptions, + [Parameter(Mandatory)] [string] $Script + ) + [Text.Encoding]::UTF8.GetBytes($Script.Replace("`r", '')) | & ssh @SshOptions 'bash -s' + if ($LASTEXITCODE -ne 0) { + throw "远端脚本执行失败,退出码=$LASTEXITCODE" + } +} + +try { + & ssh @kylinSsh "mkdir -p '$remoteDir/packages/payload'" + if ($LASTEXITCODE -ne 0) { throw '麒麟测试目录创建失败。' } + + & scp @kylinScp $entrypoint "labadmin@127.0.0.1:$remoteDir/enable-all-databases-users.sh" + & scp @kylinScp $payload "labadmin@127.0.0.1:$remoteDir/packages/payload/sshpass-1.10-aarch64-kylin-v10.tar.gz" + & scp @kylinScp $rootPasswordFile "labadmin@127.0.0.1:$remoteDir/root.secret" + & scp @kylinScp $probeSecretFile "labadmin@127.0.0.1:$remoteDir/probe.secret" + & scp @primaryScp $probeSecretFile "labadmin@127.0.0.1:/var/tmp/$probeRole.secret" + if ($LASTEXITCODE -ne 0) { throw '测试文件上传失败。' } + + $createProbe = @' +set -Eeuo pipefail +role=__PROBE_ROLE__ +secret="/var/tmp/${role}.secret" +sudo chmod 600 "${secret}" +password="$(tr -d '\r\n' <"${secret}")" +role_count="$(sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "select count(*) from pg_roles where rolname='${role}'")" +database_count="$(sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "select count(*) from pg_database where datname='${role}'")" +[[ "${role_count}|${database_count}" == '0|0' ]] +sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "create role ${role} login password '${password}'" +sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "create database ${role} owner ${role}" +unset password +sudo rm -f -- "${secret}" +'@.Replace('__PROBE_ROLE__', $probeRole) + Invoke-RemoteScript -SshOptions $primarySsh -Script $createProbe + + $applyAndVerify = @' +set -Eeuo pipefail +remote_dir=__REMOTE_DIR__ +role=__PROBE_ROLE__ + +sudo chown -R root:root "${remote_dir}" +sudo chmod 700 "${remote_dir}" "${remote_dir}/packages" "${remote_dir}/packages/payload" +sudo chmod 600 "${remote_dir}/root.secret" "${remote_dir}/probe.secret" +sudo sed -i 's/\r$//' "${remote_dir}/enable-all-databases-users.sh" +sudo bash -n "${remote_dir}/enable-all-databases-users.sh" + +sudo bash -c ' + root_password="$(tr -d "\r\n" <"$1/root.secret")" + printf "APPLY\n" | bash "$1/enable-all-databases-users.sh" \ + --pgpool-host 192.168.80.140 \ + --primary-host 192.168.80.110 \ + --standby-host 192.168.80.120 \ + --postgresql-port 5432 \ + --pgpool-port 5432 \ + --ssh-port 22 \ + --root-ssh-password "${root_password}" +' bash "${remote_dir}" + +log_file="$(sudo bash -c 'ls -1t /var/log/pg-readwrite-proxy-lab/enable-all-databases-users-*.log | head -n 1')" +pgpool_backup="$(sudo sed -n 's/^.*Pgpool 备份=//p' "${log_file}" | tail -n 1)" +primary_backup="$(sudo sed -n 's/^.*Primary 备份=//p' "${log_file}" | tail -n 1)" +standby_backup="$(sudo sed -n 's/^.*Standby 备份=//p' "${log_file}" | tail -n 1)" +[[ "${pgpool_backup}" == /var/backups/pg-readwrite-proxy-lab/all-databases-users-*-pgpool ]] +[[ "${primary_backup}" == /var/backups/pg-readwrite-proxy-lab/all-databases-users-*-primary ]] +[[ "${standby_backup}" == /var/backups/pg-readwrite-proxy-lab/all-databases-users-*-standby ]] +printf '%s\n%s\n%s\n' "${pgpool_backup}" "${primary_backup}" "${standby_backup}" | sudo tee "${remote_dir}/backups" >/dev/null +sudo chmod 600 "${remote_dir}/backups" + +password="$(sudo cat "${remote_dir}/probe.secret" | tr -d '\r\n')" +psql=/opt/pgpool-client-12.0/bin/psql +runtime=/opt/pgpool-runtime-kylin-v10/lib + +for _ in $(seq 1 60); do + set +e + correct="$(PGPASSWORD="${password}" LD_LIBRARY_PATH="${runtime}" "${psql}" -XAtq -v ON_ERROR_STOP=1 \ + -h 127.0.0.1 -p 5432 -U "${role}" -d "${role}" \ + -c 'select current_user,current_database(),pg_is_in_recovery()' 2>/dev/null)" + status=$? + set -e + [[ "${status}" == 0 && "${correct}" == "${role}|${role}|t" ]] && break + sleep 1 +done +[[ "${correct:-}" == "${role}|${role}|t" ]] + +postgres_database="$(PGPASSWORD="${password}" LD_LIBRARY_PATH="${runtime}" "${psql}" -XAtq -v ON_ERROR_STOP=1 \ + -h 127.0.0.1 -p 5432 -U "${role}" -d postgres -c 'select current_database()')" +[[ "${postgres_database}" == postgres ]] + +if sudo grep -Eq "^${role}:" /etc/pgpool-II/pool_passwd; then + printf '测试角色意外存在于 pool_passwd。\n' >&2 + exit 1 +fi + +set +e +PGPASSWORD="wrong-${password}" LD_LIBRARY_PATH="${runtime}" "${psql}" -XAtq -v ON_ERROR_STOP=1 \ + -h 127.0.0.1 -p 5432 -U "${role}" -d "${role}" -c 'select 1' \ + >/var/tmp/pg-rw-all-users-wrong-password.$$.out 2>&1 +wrong_status=$? +set -e +[[ "${wrong_status}" -ne 0 ]] +grep -Eiq 'password authentication failed|md5 authentication failed' /var/tmp/pg-rw-all-users-wrong-password.$$.out + +PGPASSWORD="${password}" LD_LIBRARY_PATH="${runtime}" "${psql}" -XAtq -v ON_ERROR_STOP=1 \ + -h 127.0.0.1 -p 5432 -U "${role}" -d "${role}" \ + -c 'create table route_probe(id integer primary key)' >/dev/null +PGPASSWORD="${password}" LD_LIBRARY_PATH="${runtime}" "${psql}" -XAtq -v ON_ERROR_STOP=1 \ + -h 127.0.0.1 -p 5432 -U "${role}" -d "${role}" \ + -c 'insert into route_probe values (1)' >/dev/null +for _ in $(seq 1 60); do + row="$(PGPASSWORD="${password}" LD_LIBRARY_PATH="${runtime}" "${psql}" -XAtq -v ON_ERROR_STOP=1 \ + -h 127.0.0.1 -p 5432 -U "${role}" -d "${role}" \ + -c 'select id from route_probe' 2>/dev/null || true)" + [[ "${row}" == 1 ]] && break + sleep 1 +done +[[ "${row:-}" == 1 ]] +PGPASSWORD="${password}" LD_LIBRARY_PATH="${runtime}" "${psql}" -XAtq -v ON_ERROR_STOP=1 \ + -h 127.0.0.1 -p 5432 -U "${role}" -d "${role}" -c 'drop table route_probe' >/dev/null +unset password +rm -f -- /var/tmp/pg-rw-all-users-wrong-password.$$.out +printf 'ALL_DATABASES_USERS_E2E_PASS role=%s database=%s select_node=standby wrong_password=rejected\n' "${role}" "${role}" +'@.Replace('__REMOTE_DIR__', $remoteDir).Replace('__PROBE_ROLE__', $probeRole) + Invoke-RemoteScript -SshOptions $kylinSsh -Script $applyAndVerify +} +finally { + Remove-Item -LiteralPath $probeSecretFile -Force -ErrorAction SilentlyContinue + + $cleanupErrors = [Collections.Generic.List[string]]::new() + $backupPaths = @() + try { + $backupPaths = @(& ssh @kylinSsh "sudo cat '$remoteDir/backups' 2>/dev/null") + } + catch { + $cleanupErrors.Add("无法读取测试备份路径:$($_.Exception.Message)") + } + + if ($backupPaths.Count -eq 3) { + $pgpoolBackup, $primaryBackup, $standbyBackup = $backupPaths + $expectedPrefix = '/var/backups/pg-readwrite-proxy-lab/all-databases-users-' + if (-not $pgpoolBackup.StartsWith($expectedPrefix) -or + -not $primaryBackup.StartsWith($expectedPrefix) -or + -not $standbyBackup.StartsWith($expectedPrefix)) { + $cleanupErrors.Add('测试备份路径不在预期目录,拒绝自动恢复。') + } + else { + try { + $restorePgpool = @' +set -Eeuo pipefail +backup=__BACKUP__ +sudo test -f "${backup}/pool_hba.conf.before" +sudo cp -a -- "${backup}/pool_hba.conf.before" /etc/pgpool-II/pool_hba.conf +sudo systemctl reload pgpool +'@.Replace('__BACKUP__', $pgpoolBackup) + Invoke-RemoteScript -SshOptions $kylinSsh -Script $restorePgpool + } + catch { + $cleanupErrors.Add("Pgpool 配置恢复失败:$($_.Exception.Message)") + } + + foreach ($backend in @( + @{ Ssh = $primarySsh; Backup = $primaryBackup; Label = 'Primary' }, + @{ Ssh = @($commonOptions + @('-p', '22012', 'labadmin@127.0.0.1')); Backup = $standbyBackup; Label = 'Standby' } + )) { + try { + $restoreBackend = @' +set -Eeuo pipefail +backup=__BACKUP__ +hba_path="$(sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "select current_setting('hba_file')")" +sudo test -f "${backup}/pg_hba.conf.before" +sudo cp -a -- "${backup}/pg_hba.conf.before" "${hba_path}" +sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c 'select pg_reload_conf()' >/dev/null +'@.Replace('__BACKUP__', [string]$backend.Backup) + Invoke-RemoteScript -SshOptions ([string[]]$backend.Ssh) -Script $restoreBackend + } + catch { + $cleanupErrors.Add("$($backend.Label) 配置恢复失败:$($_.Exception.Message)") + } + } + } + } + + try { + $cleanupStandby = @' +set -Eeuo pipefail +role=__PROBE_ROLE__ +sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "select pg_terminate_backend(pid) from pg_stat_activity where usename='${role}' and pid<>pg_backend_pid()" >/dev/null +'@.Replace('__PROBE_ROLE__', $probeRole) + $standbySsh = @($commonOptions + @('-p', '22012', 'labadmin@127.0.0.1')) + Invoke-RemoteScript -SshOptions $standbySsh -Script $cleanupStandby + } + catch { + $cleanupErrors.Add("Standby 测试会话清理失败:$($_.Exception.Message)") + } + + try { + $dropProbe = @' +set -Eeuo pipefail +role=__PROBE_ROLE__ +sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "select pg_terminate_backend(pid) from pg_stat_activity where usename='${role}' and pid<>pg_backend_pid()" >/dev/null +sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "drop database if exists ${role}" +sudo /opt/pgsql12/bin/tools psql -d postgres -p 5432 -c "drop role if exists ${role}" +sudo rm -f -- "/var/tmp/${role}.secret" +'@.Replace('__PROBE_ROLE__', $probeRole) + Invoke-RemoteScript -SshOptions $primarySsh -Script $dropProbe + } + catch { + $cleanupErrors.Add("临时角色和数据库清理失败:$($_.Exception.Message)") + } + + try { + & ssh @kylinSsh "sudo rm -rf -- '$remoteDir'" + if ($LASTEXITCODE -ne 0) { throw "退出码=$LASTEXITCODE" } + } + catch { + $cleanupErrors.Add("麒麟临时目录清理失败:$($_.Exception.Message)") + } + + if ($cleanupErrors.Count -gt 0) { + throw ($cleanupErrors -join [Environment]::NewLine) + } +} + +Write-Host "测试通过且已恢复原配置。临时角色和数据库已删除:$probeRole。"