105 lines
6.6 KiB
Markdown
105 lines
6.6 KiB
Markdown
# K8s Script Pack
|
|
|
|
Kubernetes-focused shell scripts intended for cronjobs and operational utilities.
|
|
|
|
## Scripts
|
|
|
|
### `automated-nfs-backup.sh`
|
|
|
|
Backs up each top-level folder found in `NFS_SOURCE_PATH` into a single encrypted `.7z` archive per run, with optional Kubernetes workload quiescing when a folder name exactly matches a namespace name.
|
|
|
|
Behavior:
|
|
- Exact folder-to-namespace mapping only.
|
|
- `kube-system`, `kube-public`, `kube-node-lease`, and `default` are excluded from scale actions by default.
|
|
- Unmapped folder: backup still runs, Kubernetes scale actions are skipped.
|
|
- Mapped folder: saves replicas, scales selected workloads down, waits, adds that folder to the shared run archive, restores replicas, waits again.
|
|
- When `BACKUP_STAGING_PATH` is configured, a live rsync pre-copy is followed by a final offline delta; the namespace is restored before the staged copy is compressed.
|
|
- Scale-down issues are warnings by policy (backup still runs).
|
|
- Restore issues are warnings by policy (run can still complete successfully).
|
|
- Cleanup retains the `N` most recent matching archives by modification time and deletes older ones.
|
|
|
|
### `backup-k3s-control-plane-config.sh`
|
|
|
|
Creates and validates a standard `.zip` containing the iris k3s control-plane configuration and credential state, then atomically publishes it to the NAS. It never stops or restarts k3s, and deliberately excludes `/var/lib/rancher/k3s/server/db` and `kine.sock`; this is configuration/credential backup, not a live datastore snapshot.
|
|
|
|
The archive includes all of `K3S_CONFIG_DIR`, any existing `K3S_SERVICE_FILES`, and existing `cred`, `etc`, `manifests`, `static`, `tls`, `token`, `agent-token`, and `node-token` entries beneath `K3S_SERVER_DIR`.
|
|
|
|
Defaults mirror the existing `k8s-config-backup` schedule: `ivanch@nas.haven:/export/Backup/k8s`, archive names such as `k3s-control-plane-config_2026-07-21_12-00-00.zip`, and remote retention of the four newest matching archives. Other prefixes, including `k8s-backup_*.7z`, are never considered for retention.
|
|
|
|
The script uses a non-blocking lock by default (`/var/lock/k3s-control-plane-config-backup.lock`), so overlapping cron and manual invocations fail safely. It builds in a private local temporary directory, validates with `unzip -t`, uploads to a remote temporary filename, and only then renames it into place.
|
|
|
|
Useful overrides:
|
|
|
|
- Source paths: `K3S_CONFIG_DIR`, `K3S_SERVER_DIR`, `K3S_SERVICE_FILES` (colon-separated service configuration files).
|
|
- Destination and identity: `REMOTE_USER`, `REMOTE_HOST`, `REMOTE_DIR`, `REMOTE_IDENTITY_FILE`, `SSH_PORT`.
|
|
- Archive and locking: `ARCHIVE_PREFIX`, `ARCHIVE_TS_FORMAT`, `LOCAL_TMP_PARENT`, `LOCK_PATH`, `CLEANUP_KEEP_COUNT` (default `4`; `0` disables deletion).
|
|
- Binaries: `ZIP_BIN`, `UNZIP_BIN`, `SSH_BIN`, `SCP_BIN`, `CURL_BIN`, `FLOCK_BIN`, `MKTEMP_BIN`, `DATE_BIN`, `REALPATH_BIN`, `REMOTE_BASH_BIN`. `REALPATH_BIN` must provide GNU `realpath` `-e` and `-m` options.
|
|
- Notifications: `NOTIFY_SUCCESS_URL` (default `http://notify.haven/template/notify/backup`), `NOTIFY_FAILURE_URL` (default `http://notify.haven/template/notify/error`), `NOTIFY_TITLE`, `NOTIFY_ASSET`, and `NOTIFY_CALLER`. Set either URL to an empty value to disable it. Failed notifications only generate a warning; they do not change an otherwise successful backup result.
|
|
|
|
## Environment Variables
|
|
|
|
Required:
|
|
- `NFS_SOURCE_PATH`: Root path containing folders to back up.
|
|
- `BACKUP_OUTPUT_PATH`: Destination path for generated `.7z` archives.
|
|
|
|
Optional:
|
|
- `BACKUP_PASSWORD` (default: empty): When set, archive uses password protection; when empty, archive is not encrypted.
|
|
- `KUBECTL_BIN` (default: `kubectl`)
|
|
- `KUBE_CONTEXT` (default: empty)
|
|
- `WORKLOAD_KINDS` (default: `deployment,statefulset,replicaset,replicationcontroller`)
|
|
- `EXCLUDED_NAMESPACES` (default: `kube-system,kube-public,kube-node-lease,default`): Comma-separated namespace names that are always backed up without scale actions. Set an explicit empty value to disable the default exclusions.
|
|
- `BACKUP_STAGING_PATH` (default: empty, disabled): Local staging parent for two-pass rsync. It must be outside `NFS_SOURCE_PATH` and have enough free space for the largest folder being backed up.
|
|
- `RSYNC_BIN` (default: `rsync`)
|
|
- `ARCHIVE_PREFIX` (default: `nfs-backup`)
|
|
- `ARCHIVE_TS_FORMAT` (default: `%Y%m%d_%H%M%S`)
|
|
- `SEVENZ_METHOD` (default: `lzma2`)
|
|
- `SEVENZ_LEVEL` (default: `9`)
|
|
- `SEVENZ_HEADER_ENCRYPT` (default: `on`, only applied when `BACKUP_PASSWORD` is set)
|
|
- `SEVENZ_THREADS` (default: `on`)
|
|
- `SCALE_RETRY_COUNT` (default: `3`)
|
|
- `SCALE_RETRY_DELAY_SECONDS` (default: `5`)
|
|
- `SCALE_WAIT_SECONDS` (default: `30`): Fixed delay after scaling down and after restoring a mapped namespace.
|
|
- `LOG_LEVEL` (default: `info`)
|
|
- `TMP_STATE_DIR` (default: `/tmp/k8s-nfs-backup`): Parent directory for fresh per-run replica-state directories. A run never restores state files belonging to an earlier run.
|
|
- `NOTIFY_SUCCESS_URL` (default: empty, disabled)
|
|
- `NOTIFY_FAILURE_URL` (default: empty, disabled)
|
|
- `NOTIFY_TITLE` (default: `Kubernetes`)
|
|
- `NOTIFY_ASSET` (default: `kube config`)
|
|
- `CLEANUP_KEEP_COUNT` (default: `4`)
|
|
|
|
Notification payload (success and failure):
|
|
|
|
```json
|
|
{
|
|
"title": "Kubernetes",
|
|
"asset": "kube config",
|
|
"backupSizeInMB": 123
|
|
}
|
|
```
|
|
|
|
## Cronjob Notes
|
|
|
|
- Script is designed to run sequentially (one folder at a time).
|
|
- Provide Kubernetes RBAC allowing `get`, `list`, and `scale` on configured workload kinds in target namespaces.
|
|
- Ensure `kubectl` context and credentials are present in the runtime.
|
|
- Ensure `7z` is installed in the runtime image/host.
|
|
- When `BACKUP_STAGING_PATH` is enabled, ensure `rsync` is installed and place staging on fast local storage for the shortest offline delta.
|
|
|
|
## Failure Semantics
|
|
|
|
- Missing required env vars, missing commands, invalid paths, or inability to list namespaces: script exits non-zero immediately.
|
|
- Folder backup failures are counted. One or more successful folder backups continues with cleanup and exits zero; zero successful folder backups exits non-zero.
|
|
- Scale-down warnings/timeouts: logged and counted, backup continues.
|
|
- Restore warnings/timeouts: logged and counted, script does not fail solely because of restore warnings.
|
|
- If `NOTIFY_SUCCESS_URL` is set, success notification is sent at the end of a successful run.
|
|
- If `NOTIFY_FAILURE_URL` is set, a failure notification is sent when any folder backup failure is detected, including partial failures.
|
|
- On runs with at least one successful folder backup, cleanup retains the `CLEANUP_KEEP_COUNT` most recent matching archives and deletes older ones.
|
|
- Final summary always logs:
|
|
- `processed`
|
|
- `mapped`
|
|
- `unmapped`
|
|
- `backup_successes`
|
|
- `backup_failures`
|
|
- `scale_warnings`
|
|
- `restore_warnings`
|