server-scripts/k8s/README.md

# K8s Script Pack

Kubernetes-focused shell scripts intended for cronjobs and operational utilities.

## Scripts

### `automated-nfs-backup.sh`

Copies each top-level folder found in `NFS_SOURCE_PATH` to a per-run local staging directory, then creates one encrypted `.7z` archive from the complete staged data. Kubernetes workloads are quiesced while a folder whose name exactly matches a namespace is copied.

Behavior:
- Exact folder-to-namespace mapping only.
- `kube-system`, `kube-public`, and `kube-node-lease` are excluded from scale actions by default.
- Unmapped or excluded folder: copied to local staging without Kubernetes scale actions.
- Mapped folder: saves replicas, scales selected workloads down, waits, copies the quiesced folder to local staging, restores replicas, and waits again.
- After every folder has been processed, one `7z` invocation compresses the complete run staging directory. Workloads are therefore restored before compression starts.
- Per-run replica state and staged data are removed on exit. The staging parent is retained.
- Scale-down issues are warnings by policy (backup still runs).
- Restore issues are warnings by policy (run can still complete successfully).
- Cleanup retains the `N` most recent matching archives by modification time and deletes older ones.

### `backup-k3s-control-plane-config.sh`

Creates and validates a standard `.zip` containing the iris k3s control-plane configuration and credential state, then atomically publishes it to the NAS. It never stops or restarts k3s, and deliberately excludes `/var/lib/rancher/k3s/server/db` and `kine.sock`; this is configuration/credential backup, not a live datastore snapshot.

The archive includes all of `K3S_CONFIG_DIR`, any existing `K3S_SERVICE_FILES`, and existing `cred`, `etc`, `manifests`, `static`, `tls`, `token`, `agent-token`, and `node-token` entries beneath `K3S_SERVER_DIR`.

Defaults mirror the existing `k8s-config-backup` schedule: `ivanch@nas.haven:/export/Backup/k8s`, archive names such as `k3s-control-plane-config_2026-07-21_12-00-00.zip`, and remote retention of the four newest matching archives. Other prefixes, including `k8s-backup_*.7z`, are never considered for retention.

The script uses a non-blocking lock by default (`/var/lock/k3s-control-plane-config-backup.lock`), so overlapping cron and manual invocations fail safely. It builds in a private local temporary directory, validates with `unzip -t`, uploads to a remote temporary filename, and only then renames it into place.

Useful overrides:

- Source paths: `K3S_CONFIG_DIR`, `K3S_SERVER_DIR`, `K3S_SERVICE_FILES` (colon-separated service configuration files).
- Destination and identity: `REMOTE_USER`, `REMOTE_HOST`, `REMOTE_DIR`, `REMOTE_IDENTITY_FILE`, `SSH_PORT`.
- Archive and locking: `ARCHIVE_PREFIX`, `ARCHIVE_TS_FORMAT`, `LOCAL_TMP_PARENT`, `LOCK_PATH`, `CLEANUP_KEEP_COUNT` (default `4`; `0` disables deletion).
- Binaries: `ZIP_BIN`, `UNZIP_BIN`, `SSH_BIN`, `SCP_BIN`, `CURL_BIN`, `FLOCK_BIN`, `MKTEMP_BIN`, `DATE_BIN`, `REALPATH_BIN`, `REMOTE_BASH_BIN`. `REALPATH_BIN` must provide GNU `realpath` `-e` and `-m` options.
- Notifications: `NOTIFY_SUCCESS_URL` (default `http://notify.haven/template/notify/backup`), `NOTIFY_FAILURE_URL` (default `http://notify.haven/template/notify/error`), `NOTIFY_TITLE`, `NOTIFY_ASSET`, and `NOTIFY_CALLER`. Set either URL to an empty value to disable it. Failed notifications only generate a warning; they do not change an otherwise successful backup result.

## Environment Variables

Required:
- `NFS_SOURCE_PATH`: Root path containing folders to back up.
- `BACKUP_OUTPUT_PATH`: Destination path for generated `.7z` archives.

Optional:
- `BACKUP_PASSWORD` (default: empty): When set, archive uses password protection; when empty, archive is not encrypted.
- `KUBECTL_BIN` (default: `kubectl`)
- `KUBE_CONTEXT` (default: empty)
- `WORKLOAD_KINDS` (default: `deployment,statefulset,replicaset`)
- `EXCLUDED_NAMESPACES` (default: `kube-system,kube-public,kube-node-lease`): Comma-separated namespace names that are always backed up without scale actions. Set an explicit empty value to disable the default exclusions.
- `BACKUP_STAGING_PATH` (default: `/tmp/k8s-nfs-backup-staging`): Local parent for per-run staging directories. It must be outside `NFS_SOURCE_PATH` and have enough free space for all folders included in one backup run.
- `RSYNC_BIN` (default: `rsync`)
- `ARCHIVE_PREFIX` (default: `nfs-backup`)
- `ARCHIVE_TS_FORMAT` (default: `%Y%m%d_%H%M%S`)
- `SEVENZ_METHOD` (default: `lzma2`)
- `SEVENZ_LEVEL` (default: `9`)
- `SEVENZ_HEADER_ENCRYPT` (default: `on`, only applied when `BACKUP_PASSWORD` is set)
- `SEVENZ_THREADS` (default: `on`)
- `SCALE_RETRY_COUNT` (default: `3`)
- `SCALE_RETRY_DELAY_SECONDS` (default: `5`)
- `SCALE_WAIT_SECONDS` (default: `30`): Fixed delay after scaling down and after restoring a mapped namespace.
- `LOG_LEVEL` (default: `info`)
- `TMP_STATE_DIR` (default: `/tmp/k8s-nfs-backup`): Parent directory for fresh per-run replica-state directories. A run never restores state files belonging to an earlier run.
- `NOTIFY_SUCCESS_URL` (default: `http://notify.haven/api/v2/notifications/backup`; set to empty to disable)
- `NOTIFY_FAILURE_URL` (default: `http://notify.haven/api/v2/notifications/error`; set to empty to disable)
- `NOTIFY_TITLE` (default: `Kubernetes`)
- `NOTIFY_ASSET` (default: `kube config`)
- `NOTIFY_SOURCE` (default: `NOTIFY_CALLER`, or `Kubernetes NFS backup`): Haven Notify v2 source identifier. `NOTIFY_CALLER` remains a compatibility fallback.
- `CLEANUP_KEEP_COUNT` (default: `4`)

Notifications use the Haven Notify v2 backup and error endpoints. Success payloads include the final archive size and elapsed time:

```json
{
  "title": "Kubernetes",
  "asset": "kube config",
  "sizeBytes": 128974848,
  "durationSeconds": 125,
  "source": "Kubernetes NFS backup"
}
```

Failures use the v2 error contract:

```json
{
  "source": "Kubernetes NFS backup",
  "message": "Some NFS folders failed to back up",
  "severity": "critical",
  "errorCode": "BACKUP_FAILED"
}
```

## Cronjob Notes

- Script is designed to run sequentially (one folder at a time).
- Provide Kubernetes RBAC allowing `get`, `list`, and `scale` on configured workload kinds in target namespaces.
- Ensure `kubectl` context and credentials are present in the runtime.
- Ensure `7z` is installed in the runtime image/host.
- Ensure `rsync` is installed and keep `BACKUP_STAGING_PATH` on fast local storage. Capacity must accommodate one complete uncompressed backup run in addition to the output archive.

## Failure Semantics

- Missing required env vars, missing commands, invalid paths, or inability to list namespaces: script exits non-zero immediately.
- Folder staging failures are counted and failed partial copies are excluded from the archive. One or more successfully staged folders continues to archive creation; zero successfully staged folders exits non-zero.
- If the single archive operation fails, the run treats every staged folder as failed and exits non-zero.
- Scale-down warnings/timeouts: logged and counted, backup continues.
- Restore warnings/timeouts: logged and counted, script does not fail solely because of restore warnings.
- If `NOTIFY_SUCCESS_URL` is set, success notification is sent at the end of a successful run.
- If `NOTIFY_FAILURE_URL` is set, a failure notification is sent when any folder backup failure is detected, including partial failures.
- On runs with at least one successful folder backup, cleanup retains the `CLEANUP_KEEP_COUNT` most recent matching archives and deletes older ones.
- Final summary always logs:
  - `processed`
  - `mapped`
  - `unmapped`
  - `backup_successes`
  - `backup_failures`
  - `scale_warnings`
  - `restore_warnings`