Running backups causes the PITR container to crash

[DimitriosLisenko]

Report

When a backup runs, the MySQL pod seems to run the following process:

/usr/bin/bash -ue /usr/bin/wsrep_sst_xtrabackup-v2 --role donor --address X.X.X.X:4444/xtrabackup_sst//1 --socket /tmp/mysql.sock --datadir /var/lib/mysql/ --basedir /usr/ --plugindir /usr/lib64/mysql/plugin/ --defaults-file /etc/my.cnf --defaults-group-suffix  --mysqld-version 8.4.6-6.1 --binlog binlog --gtid GTID:44506

This causes the wsrep_ variables to change from

wsrep_local_state_comment	Synced
wsrep_cluster_size	1

to

wsrep_local_state_comment	Donor/Desynced
wsrep_cluster_size	2

which eventually becomes

wsrep_local_state_comment	Synced
wsrep_cluster_size	2

and then

wsrep_local_state_comment	Synced
wsrep_cluster_size	1

In the meantime, the PITR container seems to be running a loop which calls the GetPXCFirstHost function from the operator, which gets the wsrep_ state and looks for nodes with wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary.

If this loop runs when wsrep_local_state_comment is Donor/Desynced, it can't find any nodes, and the PITR container crashes with

ERROR: new db connection: get host: can't find host

Versions

1. Kubernetes 1.34.2
2. Operator 1.18.0
3. Database 8.4

[egegunes]

@DimitriosLisenko could you please share your cr.yaml so we can try to reproduce?

[DimitriosLisenko]

Sure thing! Don't think there's anything particularly special in there though.

apiVersion: pxc.percona.com/v1
kind: PerconaXtraDBCluster
metadata:
  name: percona-pxc-db
  namespace: cluster-pxc
  labels:
    app.kubernetes.io/name: pxc-db
    app.kubernetes.io/instance: percona
    app.kubernetes.io/version: 1.18.0
  finalizers:
  - percona.com/delete-pxc-pods-in-order
  - percona.com/delete-proxysql-pvc
  - percona.com/delete-pxc-pvc
  - percona.com/delete-ssl
spec:
  crVersion: 1.18.0
  enableVolumeExpansion: false
  secretsName: cluster-credentials
  sslSecretName: percona-pxc-db-ssl
  sslInternalSecretName: percona-pxc-db-ssl-internal
  vaultSecretName: percona-pxc-db-vault
  logCollectorSecretName: percona-pxc-db-log-collector
  enableCRValidationWebhook: true
  pause: false
  unsafeFlags:
    proxySize: true
    pxcSize: true
  updateStrategy: SmartUpdate
  upgradeOptions:
    versionServiceEndpoint: https://check.percona.com
    apply: disabled
    schedule: 0 4 * * *
  tls:
    enabled: true
  pxc:
    size: 1
    image: percona/percona-xtradb-cluster:8.4.6-6.1
    autoRecovery: true
    readinessDelaySec: 15
    livenessDelaySec: 300
    annotations: {}
    expose:
      enabled: false
    resources:
      requests:
        cpu: 600m
        memory: 1G
      limits: {}
    sidecars: []
    sidecarVolumes: []
    sidecarPVCs: []
    sidecarResources:
      requests: {}
      limits: {}
    nodeSelector: {}
    tolerations: []
    podDisruptionBudget:
      maxUnavailable: 1
    volumeSpec:
      persistentVolumeClaim:
        storageClassName: local-path
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50000.00M
    gracePeriod: 600
    readinessProbes:
      failureThreshold: 5
      initialDelaySeconds: 15
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 15
    livenessProbes:
      failureThreshold: 3
      initialDelaySeconds: 300
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
  haproxy:
    enabled: true
    size: 1
    image: percona/haproxy:2.8.15
    exposePrimary:
      type: ClusterIP
    exposeReplicas:
      enabled: true
      onlyReaders: true
      type: ClusterIP
    annotations: {}
    labels: {}
    resources:
      requests:
        cpu: 600m
        memory: 1G
      limits: {}
    sidecars: []
    sidecarVolumes: []
    sidecarPVCs: []
    sidecarResources:
      requests: {}
      limits: {}
    nodeSelector: {}
    tolerations: []
    podDisruptionBudget:
      maxUnavailable: 1
    volumeSpec:
      emptyDir: {}
    gracePeriod: 30
    readinessDelaySec: 15
    livenessDelaySec: 300
    readinessProbes:
      failureThreshold: 3
      initialDelaySeconds: 15
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    livenessProbes:
      failureThreshold: 4
      initialDelaySeconds: 60
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 5
  proxysql:
    enabled: false
  logcollector:
    enabled: false
  pmm:
    enabled: false
  backup:
    allowParallel: false
    image: percona/percona-xtrabackup:8.4.0-4.1
    backoffLimit: 1
    pitr:
      enabled: true
      storageName: wals
      timeBetweenUploads: 60
      timeoutSeconds: 60
      resources:
        requests: {}
        limits: {}
    storages:
      base:
        s3:
          bucket: THE_BUCKET/base
          credentialsSecret: backup-s3-credentials
          region: us-east-1
        type: s3
        verifyTLS: true
      wals:
        s3:
          bucket: THE_BUCEKT/wals
          credentialsSecret: backup-s3-credentials
          region: us-east-1
        type: s3
        verifyTLS: true
    schedule:
    - name: percona-xtradb-cluster-backups
      retention:
        type: count
        count: 100
        deleteFromStorage: false
      schedule: "*/3 * * * *"
      storageName: base
  users: []

[egegunes]

Most important thing is you have only 1 PXC pod. This restart should not happen if you have more than 1.

Anyway I'll think about if we can improve the situation, but right now I'm concerned about collecting binlogs from a Desynced node.