updating flow and notify endpoint

k8s/README.md

@@ -6,14 +6,15 @@ Kubernetes-focused shell scripts intended for cronjobs and operational utilities
 
 ### `automated-nfs-backup.sh`
 
-Backs up each top-level folder found in `NFS_SOURCE_PATH` into a single encrypted `.7z` archive per run, with optional Kubernetes workload quiescing when a folder name exactly matches a namespace name.
+Copies each top-level folder found in `NFS_SOURCE_PATH` to a per-run local staging directory, then creates one encrypted `.7z` archive from the complete staged data. Kubernetes workloads are quiesced while a folder whose name exactly matches a namespace is copied.
 
 Behavior:
 - Exact folder-to-namespace mapping only.
-- `kube-system`, `kube-public`, `kube-node-lease`, and `default` are excluded from scale actions by default.
-- Unmapped folder: backup still runs, Kubernetes scale actions are skipped.
-- Mapped folder: saves replicas, scales selected workloads down, waits, adds that folder to the shared run archive, restores replicas, waits again.
-- When `BACKUP_STAGING_PATH` is configured, a live rsync pre-copy is followed by a final offline delta; the namespace is restored before the staged copy is compressed.
+- `kube-system`, `kube-public`, and `kube-node-lease` are excluded from scale actions by default.
+- Unmapped or excluded folder: copied to local staging without Kubernetes scale actions.
+- Mapped folder: saves replicas, scales selected workloads down, waits, copies the quiesced folder to local staging, restores replicas, and waits again.
+- After every folder has been processed, one `7z` invocation compresses the complete run staging directory. Workloads are therefore restored before compression starts.
+- Per-run replica state and staged data are removed on exit. The staging parent is retained.
 - Scale-down issues are warnings by policy (backup still runs).
 - Restore issues are warnings by policy (run can still complete successfully).
 - Cleanup retains the `N` most recent matching archives by modification time and deletes older ones.
@@ -46,9 +47,9 @@ Optional:
 - `BACKUP_PASSWORD` (default: empty): When set, archive uses password protection; when empty, archive is not encrypted.
 - `KUBECTL_BIN` (default: `kubectl`)
 - `KUBE_CONTEXT` (default: empty)
-- `WORKLOAD_KINDS` (default: `deployment,statefulset,replicaset,replicationcontroller`)
-- `EXCLUDED_NAMESPACES` (default: `kube-system,kube-public,kube-node-lease,default`): Comma-separated namespace names that are always backed up without scale actions. Set an explicit empty value to disable the default exclusions.
-- `BACKUP_STAGING_PATH` (default: empty, disabled): Local staging parent for two-pass rsync. It must be outside `NFS_SOURCE_PATH` and have enough free space for the largest folder being backed up.
+- `WORKLOAD_KINDS` (default: `deployment,statefulset,replicaset`)
+- `EXCLUDED_NAMESPACES` (default: `kube-system,kube-public,kube-node-lease`): Comma-separated namespace names that are always backed up without scale actions. Set an explicit empty value to disable the default exclusions.
+- `BACKUP_STAGING_PATH` (default: `/tmp/k8s-nfs-backup-staging`): Local parent for per-run staging directories. It must be outside `NFS_SOURCE_PATH` and have enough free space for all folders included in one backup run.
 - `RSYNC_BIN` (default: `rsync`)
 - `ARCHIVE_PREFIX` (default: `nfs-backup`)
 - `ARCHIVE_TS_FORMAT` (default: `%Y%m%d_%H%M%S`)
@@ -61,19 +62,33 @@ Optional:
 - `SCALE_WAIT_SECONDS` (default: `30`): Fixed delay after scaling down and after restoring a mapped namespace.
 - `LOG_LEVEL` (default: `info`)
 - `TMP_STATE_DIR` (default: `/tmp/k8s-nfs-backup`): Parent directory for fresh per-run replica-state directories. A run never restores state files belonging to an earlier run.
-- `NOTIFY_SUCCESS_URL` (default: empty, disabled)
-- `NOTIFY_FAILURE_URL` (default: empty, disabled)
+- `NOTIFY_SUCCESS_URL` (default: `http://notify.haven/api/v2/notifications/backup`; set to empty to disable)
+- `NOTIFY_FAILURE_URL` (default: `http://notify.haven/api/v2/notifications/error`; set to empty to disable)
 - `NOTIFY_TITLE` (default: `Kubernetes`)
 - `NOTIFY_ASSET` (default: `kube config`)
+- `NOTIFY_SOURCE` (default: `NOTIFY_CALLER`, or `Kubernetes NFS backup`): Haven Notify v2 source identifier. `NOTIFY_CALLER` remains a compatibility fallback.
 - `CLEANUP_KEEP_COUNT` (default: `4`)
 
-Notification payload (success and failure):
+Notifications use the Haven Notify v2 backup and error endpoints. Success payloads include the final archive size and elapsed time:
 
 ```json
 {
   "title": "Kubernetes",
   "asset": "kube config",
-  "backupSizeInMB": 123
+  "sizeBytes": 128974848,
+  "durationSeconds": 125,
+  "source": "Kubernetes NFS backup"
+}
+```
+
+Failures use the v2 error contract:
+
+```json
+{
+  "source": "Kubernetes NFS backup",
+  "message": "Some NFS folders failed to back up",
+  "severity": "critical",
+  "errorCode": "BACKUP_FAILED"
 }
 ```
 
@@ -83,12 +98,13 @@ Notification payload (success and failure):
 - Provide Kubernetes RBAC allowing `get`, `list`, and `scale` on configured workload kinds in target namespaces.
 - Ensure `kubectl` context and credentials are present in the runtime.
 - Ensure `7z` is installed in the runtime image/host.
-- When `BACKUP_STAGING_PATH` is enabled, ensure `rsync` is installed and place staging on fast local storage for the shortest offline delta.
+- Ensure `rsync` is installed and keep `BACKUP_STAGING_PATH` on fast local storage. Capacity must accommodate one complete uncompressed backup run in addition to the output archive.
 
 ## Failure Semantics
 
 - Missing required env vars, missing commands, invalid paths, or inability to list namespaces: script exits non-zero immediately.
-- Folder backup failures are counted. One or more successful folder backups continues with cleanup and exits zero; zero successful folder backups exits non-zero.
+- Folder staging failures are counted and failed partial copies are excluded from the archive. One or more successfully staged folders continues to archive creation; zero successfully staged folders exits non-zero.
+- If the single archive operation fails, the run treats every staged folder as failed and exits non-zero.
 - Scale-down warnings/timeouts: logged and counted, backup continues.
 - Restore warnings/timeouts: logged and counted, script does not fail solely because of restore warnings.
 - If `NOTIFY_SUCCESS_URL` is set, success notification is sent at the end of a successful run.