server-scripts/k8s/README.md

K8s Script Pack

Kubernetes-focused shell scripts intended for cronjobs and operational utilities.

Scripts

automated-nfs-backup.sh

Copies each top-level folder found in NFS_SOURCE_PATH to a per-run local staging directory, then creates one encrypted .7z archive from the complete staged data. Kubernetes workloads are quiesced while a folder whose name exactly matches a namespace is copied.

Behavior:

• Exact folder-to-namespace mapping only.
• kube-system, kube-public, and kube-node-lease are excluded from scale actions by default.
• Unmapped or excluded folder: copied to local staging without Kubernetes scale actions.
• Mapped folder: saves replicas, scales selected workloads down, waits, copies the quiesced folder to local staging, restores replicas, and waits again.
• After every folder has been processed, one 7z invocation compresses the complete run staging directory. Workloads are therefore restored before compression starts.
• Per-run replica state and staged data are removed on exit. The staging parent is retained.
• Scale-down issues are warnings by policy (backup still runs).
• Restore issues are warnings by policy (run can still complete successfully).
• Cleanup retains the N most recent matching archives by modification time and deletes older ones.

backup-k3s-control-plane-config.sh

Creates and validates a standard .zip containing the iris k3s control-plane configuration and credential state, then atomically publishes it to the NAS. It never stops or restarts k3s, and deliberately excludes /var/lib/rancher/k3s/server/db and kine.sock; this is configuration/credential backup, not a live datastore snapshot.

The archive includes all of K3S_CONFIG_DIR, any existing K3S_SERVICE_FILES, and existing cred, etc, manifests, static, tls, token, agent-token, and node-token entries beneath K3S_SERVER_DIR.

Defaults mirror the existing k8s-config-backup schedule: ivanch@nas.haven:/export/Backup/k8s, archive names such as k3s-control-plane-config_2026-07-21_12-00-00.zip, and remote retention of the four newest matching archives. Other prefixes, including k8s-backup_*.7z, are never considered for retention.

The script uses a non-blocking lock by default (/var/lock/k3s-control-plane-config-backup.lock), so overlapping cron and manual invocations fail safely. It builds in a private local temporary directory, validates with unzip -t, uploads to a remote temporary filename, and only then renames it into place.

Useful overrides:

• Source paths: K3S_CONFIG_DIR, K3S_SERVER_DIR, K3S_SERVICE_FILES (colon-separated service configuration files).
• Destination and identity: REMOTE_USER, REMOTE_HOST, REMOTE_DIR, REMOTE_IDENTITY_FILE, SSH_PORT.
• Archive and locking: ARCHIVE_PREFIX, ARCHIVE_TS_FORMAT, LOCAL_TMP_PARENT, LOCK_PATH, CLEANUP_KEEP_COUNT (default 4; 0 disables deletion).
• Binaries: ZIP_BIN, UNZIP_BIN, SSH_BIN, SCP_BIN, CURL_BIN, FLOCK_BIN, MKTEMP_BIN, DATE_BIN, REALPATH_BIN, REMOTE_BASH_BIN. REALPATH_BIN must provide GNU realpath -e and -m options.
• Notifications: NOTIFY_SUCCESS_URL (default http://notify.haven/template/notify/backup), NOTIFY_FAILURE_URL (default http://notify.haven/template/notify/error), NOTIFY_TITLE, NOTIFY_ASSET, and NOTIFY_CALLER. Set either URL to an empty value to disable it. Failed notifications only generate a warning; they do not change an otherwise successful backup result.

Environment Variables

Required:

• NFS_SOURCE_PATH: Root path containing folders to back up.
• BACKUP_OUTPUT_PATH: Destination path for generated .7z archives.

Optional:

• BACKUP_PASSWORD (default: empty): When set, archive uses password protection; when empty, archive is not encrypted.
• KUBECTL_BIN (default: kubectl)
• KUBE_CONTEXT (default: empty)
• WORKLOAD_KINDS (default: deployment,statefulset,replicaset)
• EXCLUDED_NAMESPACES (default: kube-system,kube-public,kube-node-lease): Comma-separated namespace names that are always backed up without scale actions. Set an explicit empty value to disable the default exclusions.
• BACKUP_STAGING_PATH (default: /tmp/k8s-nfs-backup-staging): Local parent for per-run staging directories. It must be outside NFS_SOURCE_PATH and have enough free space for all folders included in one backup run.
• RSYNC_BIN (default: rsync)
• ARCHIVE_PREFIX (default: nfs-backup)
• ARCHIVE_TS_FORMAT (default: %Y%m%d_%H%M%S)
• SEVENZ_METHOD (default: lzma2)
• SEVENZ_LEVEL (default: 9)
• SEVENZ_HEADER_ENCRYPT (default: on, only applied when BACKUP_PASSWORD is set)
• SEVENZ_THREADS (default: on)
• SCALE_RETRY_COUNT (default: 3)
• SCALE_RETRY_DELAY_SECONDS (default: 5)
• SCALE_WAIT_SECONDS (default: 30): Fixed delay after scaling down and after restoring a mapped namespace.
• LOG_LEVEL (default: info)
• TMP_STATE_DIR (default: /tmp/k8s-nfs-backup): Parent directory for fresh per-run replica-state directories. A run never restores state files belonging to an earlier run.
• NOTIFY_SUCCESS_URL (default: http://notify.haven/api/v2/notifications/backup; set to empty to disable)
• NOTIFY_FAILURE_URL (default: http://notify.haven/api/v2/notifications/error; set to empty to disable)
• NOTIFY_TITLE (default: Kubernetes)
• NOTIFY_ASSET (default: kube config)
• NOTIFY_SOURCE (default: NOTIFY_CALLER, or Kubernetes NFS backup): Haven Notify v2 source identifier. NOTIFY_CALLER remains a compatibility fallback.
• CLEANUP_KEEP_COUNT (default: 4)

Notifications use the Haven Notify v2 backup and error endpoints. Success payloads include the final archive size and elapsed time:

{
  "title": "Kubernetes",
  "asset": "kube config",
  "sizeBytes": 128974848,
  "durationSeconds": 125,
  "source": "Kubernetes NFS backup"
}

Failures use the v2 error contract:

{
  "source": "Kubernetes NFS backup",
  "message": "Some NFS folders failed to back up",
  "severity": "critical",
  "errorCode": "BACKUP_FAILED"
}

Cronjob Notes

• Script is designed to run sequentially (one folder at a time).
• Provide Kubernetes RBAC allowing get, list, and scale on configured workload kinds in target namespaces.
• Ensure kubectl context and credentials are present in the runtime.
• Ensure 7z is installed in the runtime image/host.
• Ensure rsync is installed and keep BACKUP_STAGING_PATH on fast local storage. Capacity must accommodate one complete uncompressed backup run in addition to the output archive.

Failure Semantics

• Missing required env vars, missing commands, invalid paths, or inability to list namespaces: script exits non-zero immediately.
• Folder staging failures are counted and failed partial copies are excluded from the archive. One or more successfully staged folders continues to archive creation; zero successfully staged folders exits non-zero.
• If the single archive operation fails, the run treats every staged folder as failed and exits non-zero.
• Scale-down warnings/timeouts: logged and counted, backup continues.
• Restore warnings/timeouts: logged and counted, script does not fail solely because of restore warnings.
• If NOTIFY_SUCCESS_URL is set, success notification is sent at the end of a successful run.
• If NOTIFY_FAILURE_URL is set, a failure notification is sent when any folder backup failure is detected, including partial failures.
• On runs with at least one successful folder backup, cleanup retains the CLEANUP_KEEP_COUNT most recent matching archives and deletes older ones.
• Final summary always logs:
  • processed
  • mapped
  • unmapped
  • backup_successes
  • backup_failures
  • scale_warnings
  • restore_warnings