# K8s Script Pack

Kubernetes-focused shell scripts intended for cronjobs and operational utilities.

## Scripts

### `automated-nfs-backup.sh`

Backs up each top-level folder found in `NFS_SOURCE_PATH` into a single encrypted `.7z` archive per run, with optional Kubernetes workload quiescing when a folder name exactly matches a namespace name.

Behavior:
- Exact folder-to-namespace mapping only.
- `kube-system`, `kube-public`, `kube-node-lease`, and `default` are excluded from scale actions by default.
- Unmapped folder: backup still runs, Kubernetes scale actions are skipped.
- Mapped folder: saves replicas, scales selected workloads down, waits, adds that folder to the shared run archive, restores replicas, waits again.
- When `BACKUP_STAGING_PATH` is configured, a live rsync pre-copy is followed by a final offline delta; the namespace is restored before the staged copy is compressed.
- Scale-down issues are warnings by policy (backup still runs).
- Restore issues are warnings by policy (run can still complete successfully).
- Cleanup retains the `N` most recent matching archives by modification time and deletes older ones.

### `backup-k3s-control-plane-config.sh`

Creates and validates a standard `.zip` containing the iris k3s control-plane configuration and credential state, then atomically publishes it to the NAS. It never stops or restarts k3s, and deliberately excludes `/var/lib/rancher/k3s/server/db` and `kine.sock`; this is configuration/credential backup, not a live datastore snapshot.

The archive includes all of `K3S_CONFIG_DIR`, any existing `K3S_SERVICE_FILES`, and existing `cred`, `etc`, `manifests`, `static`, `tls`, `token`, `agent-token`, and `node-token` entries beneath `K3S_SERVER_DIR`.

Defaults mirror the existing `k8s-config-backup` schedule: `ivanch@nas.haven:/export/Backup/k8s`, archive names such as `k3s-control-plane-config_2026-07-21_12-00-00.zip`, and remote retention of the four newest matching archives. Other prefixes, including `k8s-backup_*.7z`, are never considered for retention.

The script uses a non-blocking lock by default (`/var/lock/k3s-control-plane-config-backup.lock`), so overlapping cron and manual invocations fail safely. It builds in a private local temporary directory, validates with `unzip -t`, uploads to a remote temporary filename, and only then renames it into place.

Useful overrides:

- Source paths: `K3S_CONFIG_DIR`, `K3S_SERVER_DIR`, `K3S_SERVICE_FILES` (colon-separated service configuration files).
- Destination and identity: `REMOTE_USER`, `REMOTE_HOST`, `REMOTE_DIR`, `REMOTE_IDENTITY_FILE`, `SSH_PORT`.
- Archive and locking: `ARCHIVE_PREFIX`, `ARCHIVE_TS_FORMAT`, `LOCAL_TMP_PARENT`, `LOCK_PATH`, `CLEANUP_KEEP_COUNT` (default `4`; `0` disables deletion).
- Binaries: `ZIP_BIN`, `UNZIP_BIN`, `SSH_BIN`, `SCP_BIN`, `CURL_BIN`, `FLOCK_BIN`, `MKTEMP_BIN`, `DATE_BIN`, `REALPATH_BIN`, `REMOTE_BASH_BIN`. `REALPATH_BIN` must provide GNU `realpath` `-e` and `-m` options.
- Notifications: `NOTIFY_SUCCESS_URL` (default `http://notify.haven/template/notify/backup`), `NOTIFY_FAILURE_URL` (default `http://notify.haven/template/notify/error`), `NOTIFY_TITLE`, `NOTIFY_ASSET`, and `NOTIFY_CALLER`. Set either URL to an empty value to disable it. Failed notifications only generate a warning; they do not change an otherwise successful backup result.

## Environment Variables

Required:
- `NFS_SOURCE_PATH`: Root path containing folders to back up.
- `BACKUP_OUTPUT_PATH`: Destination path for generated `.7z` archives.

Optional:
- `BACKUP_PASSWORD` (default: empty): When set, archive uses password protection; when empty, archive is not encrypted.
- `KUBECTL_BIN` (default: `kubectl`)
- `KUBE_CONTEXT` (default: empty)
- `WORKLOAD_KINDS` (default: `deployment,statefulset,replicaset,replicationcontroller`)
- `EXCLUDED_NAMESPACES` (default: `kube-system,kube-public,kube-node-lease,default`): Comma-separated namespace names that are always backed up without scale actions. Set an explicit empty value to disable the default exclusions.
- `BACKUP_STAGING_PATH` (default: empty, disabled): Local staging parent for two-pass rsync. It must be outside `NFS_SOURCE_PATH` and have enough free space for the largest folder being backed up.
- `RSYNC_BIN` (default: `rsync`)
- `ARCHIVE_PREFIX` (default: `nfs-backup`)
- `ARCHIVE_TS_FORMAT` (default: `%Y%m%d_%H%M%S`)
- `SEVENZ_METHOD` (default: `lzma2`)
- `SEVENZ_LEVEL` (default: `9`)
- `SEVENZ_HEADER_ENCRYPT` (default: `on`, only applied when `BACKUP_PASSWORD` is set)
- `SEVENZ_THREADS` (default: `on`)
- `SCALE_RETRY_COUNT` (default: `3`)
- `SCALE_RETRY_DELAY_SECONDS` (default: `5`)
- `SCALE_WAIT_SECONDS` (default: `30`): Fixed delay after scaling down and after restoring a mapped namespace.
- `LOG_LEVEL` (default: `info`)
- `TMP_STATE_DIR` (default: `/tmp/k8s-nfs-backup`): Parent directory for fresh per-run replica-state directories. A run never restores state files belonging to an earlier run.
- `NOTIFY_SUCCESS_URL` (default: empty, disabled)
- `NOTIFY_FAILURE_URL` (default: empty, disabled)
- `NOTIFY_TITLE` (default: `Kubernetes`)
- `NOTIFY_ASSET` (default: `kube config`)
- `CLEANUP_KEEP_COUNT` (default: `4`)

Notification payload (success and failure):

```json
{
  "title": "Kubernetes",
  "asset": "kube config",
  "backupSizeInMB": 123
}
```

## Cronjob Notes

- Script is designed to run sequentially (one folder at a time).
- Provide Kubernetes RBAC allowing `get`, `list`, and `scale` on configured workload kinds in target namespaces.
- Ensure `kubectl` context and credentials are present in the runtime.
- Ensure `7z` is installed in the runtime image/host.
- When `BACKUP_STAGING_PATH` is enabled, ensure `rsync` is installed and place staging on fast local storage for the shortest offline delta.

## Failure Semantics

- Missing required env vars, missing commands, invalid paths, or inability to list namespaces: script exits non-zero immediately.
- Folder backup failures are counted. One or more successful folder backups continues with cleanup and exits zero; zero successful folder backups exits non-zero.
- Scale-down warnings/timeouts: logged and counted, backup continues.
- Restore warnings/timeouts: logged and counted, script does not fail solely because of restore warnings.
- If `NOTIFY_SUCCESS_URL` is set, success notification is sent at the end of a successful run.
- If `NOTIFY_FAILURE_URL` is set, a failure notification is sent when any folder backup failure is detected, including partial failures.
- On runs with at least one successful folder backup, cleanup retains the `CLEANUP_KEEP_COUNT` most recent matching archives and deletes older ones.
- Final summary always logs:
  - `processed`
  - `mapped`
  - `unmapped`
  - `backup_successes`
  - `backup_failures`
  - `scale_warnings`
  - `restore_warnings`