Error using external etcd (etcd with mTLS) - failed to get latest revision, res: null, context: ngx.timer

[liangbeirong-byron]

0. apisix version

helm list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
apisix  apisix          4               2026-05-07 09:48:02.935580963 +0000 UTC deployed        apisix-2.14.0   3.16.0

1. self-signed certificate

cat > "${TMP_CERT_DIR}/ca-csr.json" <<'EOF'
{
  "CN": "etcd-ca",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF

cfssl gencert -initca "${TMP_CERT_DIR}/ca-csr.json" | cfssljson -bare "${TMP_CERT_DIR}/etcd-ca"

cat > "${TMP_CERT_DIR}/ca-config.json" <<'EOF'
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "etcd": {
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF

cat > "${TMP_CERT_DIR}/server-csr.json" <<EOF
{
  "CN": "etcd-server",
  "hosts": [
    "etcd-01",    "etcd-02",    "etcd-03",    "127.0.0.1",    "192.168.139.64",    "192.168.139.168",    "192.168.129.25"  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

cfssl gencert \
  -ca="${TMP_CERT_DIR}/etcd-ca.pem" \
  -ca-key="${TMP_CERT_DIR}/etcd-ca-key.pem" \
  -config="${TMP_CERT_DIR}/ca-config.json" \
  -profile=etcd \
  "${TMP_CERT_DIR}/server-csr.json" | cfssljson -bare "${TMP_CERT_DIR}/etcd-server"

cat > "${TMP_CERT_DIR}/client-csr.json" <<'EOF'
{
  "CN": "apisix-client",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF

cfssl gencert \
  -ca="${TMP_CERT_DIR}/etcd-ca.pem" \
  -ca-key="${TMP_CERT_DIR}/etcd-ca-key.pem" \
  -config="${TMP_CERT_DIR}/ca-config.json" \
  -profile=etcd \
  "${TMP_CERT_DIR}/client-csr.json" | cfssljson -bare "${TMP_CERT_DIR}/etcd-client"

2. create secrets

kubectl create secret generic etcd-client-cert   --from-file=ca.crt=etcd-ca.pem   --from-file=tls.crt=etcd-client.pem   --from-file=tls.key=etcd-client-key.pem   -n apisix

3. Manual testing can connect to the etcd cluster (etcd vm node c& apisix pod)

etcdctl  --cacert=/data/etcd/certs/etcd-ca.pem --cert=/data/etcd/certs/etcd-client.pem --key=/data/etcd/certs/etcd-client-key.pem endpoint status --write-out="table"
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|  192.168.139.64:2379 | 25e38cd02130caa3 |  3.5.29 |   41 kB |     false |      false |         3 |        155 |                155 |        |
| 192.168.139.168:2379 | b26699d3c126ecd7 |  3.5.29 |   41 kB |     false |      false |         3 |        155 |                155 |        |
|  192.168.129.25:2379 | 91adae735ce09a1b |  3.5.29 |   41 kB |      true |      false |         3 |        155 |                155 |        |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+


apisix@lke534555-858891-4b56d6940000:/etcd-ssl$ pwd
/etcd-ssl
apisix@lke534555-858891-4b56d6940000:/etcd-ssl$ openssl s_client -connect 192.168.129.25:2379 \
  -CAfile ca.crt \
  -cert tls.crt \
  -key tls.key \
  -brief
Can't use SSL_get_servername
CONNECTION ESTABLISHED
Protocol version: TLSv1.3
Ciphersuite: TLS_AES_128_GCM_SHA256
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer certificate: C = CN, ST = Beijing, L = Beijing, O = etcd, OU = Etcd Security, CN = etcd-server
Hash used: SHA256
Signature type: RSA-PSS
Verification: OK
Server Temp Key: X25519, 253 bits

4. values.yaml

apisix:
  ssl:
    enabled: true
    existingCASecret: "etcd-client-cert"
    certCAFilename: "ca.crt"


# -- external etcd configuration. If etcd.enabled is false, these configuration will be used.
externalEtcd:
  # -- if etcd.enabled is false, use external etcd, support multiple address, if your etcd cluster enables TLS, please use https scheme, e.g. https://127.0.0.1:2379.
  host:
    # host or ip e.g. http://172.20.128.89:2379
    - https://192.168.139.64:2379
    - https://192.168.139.168:2379
    - https://192.168.129.25:2379
  # -- if etcd.enabled is false, user for external etcd. Set empty to disable authentication
  user:
  # -- if etcd.enabled is true, use etcd.auth.rbac.rootPassword instead.
  # -- if etcd.enabled is false and externalEtcd.existingSecret is not empty, the password should store in the corresponding secret
  # -- if etcd.enabled is false and externalEtcd.existingSecret is empty, externalEtcd.password is the passsword for external etcd.
  password: ""
  # -- if externalEtcd.existingSecret is the name of secret containing the external etcd password
  existingSecret: "etcd-client-cert"
  # -- externalEtcd.secretPasswordKey Key inside the secret containing the external etcd password
  secretPasswordKey: ""


etcd:
  # -- install built-in etcd by default, set false if do not want to install built-in etcd together,
  # this etcd is based on bitnamilegacy/etcd helm chart and latest bitnami docker image, only for development and testing purposes,
  # if you want to use etcd in production, we recommend you to install etcd by yourself and use `externalEtcd` to connect it.
  enabled: false

  # -- if etcd.enabled is true, set more values of bitnamilegacy/etcd helm chart
  auth:
    rbac:
      # -- No authentication by default. Switch to enable RBAC authentication
      create: false
      # -- root password for etcd. Requires etcd.auth.rbac.create to be true.
      rootPassword: ""
    tls:
      # -- enable etcd client certificate
      enabled: true
      # -- name of the secret contains etcd client cert
      existingSecret: "etcd-client-cert"
      # -- etcd client cert filename using in etcd.auth.tls.existingSecret
      certFilename: "tls.crt"
      # -- etcd client cert key filename using in etcd.auth.tls.existingSecret
      certKeyFilename: "tls.key"

      caFilename: "ca.crt"
      # -- whether to verify the etcd endpoint certificate when setup a TLS connection to etcd
      verify: true
      # -- specify the TLS Server Name Indication extension, the ETCD endpoint hostname will be used when this setting is unset.
      sni: ""

5. apisix pod (error log)

kubectl logs -f --tail=5 apisix-7df5947f55-7qg2b
2026/05/07 10:26:00 [error] 49#49: *130797 [lua] config_etcd.lua:197: failed to get latest revision, res: null, context: ngx.timer
2026/05/07 10:26:00 [error] 49#49: *130797 [lua] config_etcd.lua:204: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2026/05/07 10:26:00 [error] 50#50: *130798 [lua] config_etcd.lua:191: failed to get latest revision, err: has no healthy etcd endpoint available, context: ngx.timer
2026/05/07 10:26:00 [error] 50#50: *130798 [lua] config_etcd.lua:197: failed to get latest revision, res: null, context: ngx.timer
2026/05/07 10:26:00 [error] 50#50: *130798 [lua] config_etcd.lua:204: watchdir err: has no healthy etcd endpoint available, context: ngx.timer
2026/05/07 10:26:03 [error] 53#53: *130958 [lua] config_etcd.lua:191: failed to get latest revision, err: has no healthy etcd endpoint available, context: ngx.timer
2026/05/07 10:26:03 [error] 53#53: *130958 [lua] config_etcd.lua:197: failed to get latest revision, res: null, context: ngx.timer
2026/05/07 10:26:03 [error] 53#53: *130958 [lua] config_etcd.lua:204: watchdir err: has no healthy etcd endpoint available, context: ngx.timer