k8s nfs backup: exclude namespaces from scaling, isolate per-run state, harden failure handling
- Add EXCLUDED_NAMESPACES (default kube-system,kube-public,kube-node-lease,default); excluded folders are backed up but never scaled - Use a per-run state dir via mktemp -d so stale .state files from an earlier run can't leak into restores - Partial folder-backup failure now exits 0 (only zero successes fails); retention runs even on partial failure - Remove dead SCALE_TIMEOUT_SECONDS config; document SCALE_WAIT_SECONDS - Add .gitattributes to keep *.sh at LF - Extend test harness to 44 passing assertions covering exclusions, per-run state, partial/all-failed/empty scenarios
This commit is contained in:
@@ -10,11 +10,12 @@ Backs up each top-level folder found in `NFS_SOURCE_PATH` into a single encrypte
|
||||
|
||||
Behavior:
|
||||
- Exact folder-to-namespace mapping only.
|
||||
- `kube-system`, `kube-public`, `kube-node-lease`, and `default` are excluded from scale actions by default.
|
||||
- Unmapped folder: backup still runs, Kubernetes scale actions are skipped.
|
||||
- Mapped folder: saves replicas, scales selected workloads down, waits, adds that folder to the shared run archive, restores replicas, waits again.
|
||||
- Scale-down issues are warnings by policy (backup still runs).
|
||||
- Restore issues are warnings by policy (run can still complete successfully).
|
||||
- Cleanup can delete the last `N` archives ordered by date (oldest side), while keeping at least one archive.
|
||||
- Cleanup retains the `N` most recent matching archives by modification time and deletes older ones.
|
||||
|
||||
## Environment Variables
|
||||
|
||||
@@ -27,17 +28,18 @@ Optional:
|
||||
- `KUBECTL_BIN` (default: `kubectl`)
|
||||
- `KUBE_CONTEXT` (default: empty)
|
||||
- `WORKLOAD_KINDS` (default: `deployment,statefulset,replicaset,replicationcontroller`)
|
||||
- `EXCLUDED_NAMESPACES` (default: `kube-system,kube-public,kube-node-lease,default`): Comma-separated namespace names that are always backed up without scale actions. Set an explicit empty value to disable the default exclusions.
|
||||
- `ARCHIVE_PREFIX` (default: `nfs-backup`)
|
||||
- `ARCHIVE_TS_FORMAT` (default: `%Y%m%d_%H%M%S`)
|
||||
- `SEVENZ_METHOD` (default: `lzma2`)
|
||||
- `SEVENZ_LEVEL` (default: `9`)
|
||||
- `SEVENZ_HEADER_ENCRYPT` (default: `on`, only applied when `BACKUP_PASSWORD` is set)
|
||||
- `SEVENZ_THREADS` (default: `on`)
|
||||
- `SCALE_TIMEOUT_SECONDS` (default: `600`)
|
||||
- `SCALE_RETRY_COUNT` (default: `3`)
|
||||
- `SCALE_RETRY_DELAY_SECONDS` (default: `5`)
|
||||
- `SCALE_WAIT_SECONDS` (default: `30`): Fixed delay after scaling down and after restoring a mapped namespace.
|
||||
- `LOG_LEVEL` (default: `info`)
|
||||
- `TMP_STATE_DIR` (default: `/tmp/k8s-nfs-backup`)
|
||||
- `TMP_STATE_DIR` (default: `/tmp/k8s-nfs-backup`): Parent directory for fresh per-run replica-state directories. A run never restores state files belonging to an earlier run.
|
||||
- `NOTIFY_SUCCESS_URL` (default: empty, disabled)
|
||||
- `NOTIFY_FAILURE_URL` (default: empty, disabled)
|
||||
- `NOTIFY_TITLE` (default: `Kubernetes`)
|
||||
@@ -64,12 +66,12 @@ Notification payload (success and failure):
|
||||
## Failure Semantics
|
||||
|
||||
- Missing required env vars, missing commands, invalid paths, or inability to list namespaces: script exits non-zero immediately.
|
||||
- Folder backup failures: counted and script exits non-zero at end.
|
||||
- Folder backup failures are counted. One or more successful folder backups continues with cleanup and exits zero; zero successful folder backups exits non-zero.
|
||||
- Scale-down warnings/timeouts: logged and counted, backup continues.
|
||||
- Restore warnings/timeouts: logged and counted, script does not fail solely because of restore warnings.
|
||||
- If `NOTIFY_SUCCESS_URL` is set, success notification is sent at the end of a successful run.
|
||||
- If `NOTIFY_FAILURE_URL` is set, failure notification is sent when backup failures are detected.
|
||||
- On successful runs, cleanup removes the last `CLEANUP_KEEP_COUNT` archives by date ordering (oldest side), without deleting the final remaining archive.
|
||||
- If `NOTIFY_FAILURE_URL` is set, a failure notification is sent when any folder backup failure is detected, including partial failures.
|
||||
- On runs with at least one successful folder backup, cleanup retains the `CLEANUP_KEEP_COUNT` most recent matching archives and deletes older ones.
|
||||
- Final summary always logs:
|
||||
- `processed`
|
||||
- `mapped`
|
||||
|
||||
Reference in New Issue
Block a user