Root cause: backend deployment.yaml referenced secretKeyRef key: DB_USER and
key: DB_PASSWORD, but the live secret only has MYSQL_USER and MYSQL_PASSWORD.
kubectl apply reported secret/mysql-secret as "unchanged" (last-applied matched
desired) so the drift was never caught — new pods got CreateContainerConfigError.
Changes:
- backend/deployment.yaml: DB_USER → key: MYSQL_USER, DB_PASSWORD → key: MYSQL_PASSWORD
- mysql/deployment.yaml: add MYSQL_USER/MYSQL_PASSWORD env vars so the app user
(scrumapp) is created if MySQL ever reinitializes from a fresh PVC
- mysql/secret.yaml: remove stale commented-out block with old key names
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Increasing the request to 256Mi caused backend pods to be Pending with no
node assignment — the scheduler couldn't fit them alongside MySQL (512Mi
request) and existing pods on the on-premise nodes.
Memory REQUEST drives scheduling (how much the node reserves).
Memory LIMIT drives OOMKill (the actual cap at runtime).
Keep request at 128Mi so pods schedule, limit at 512Mi so Node.js +
Socket.io + MySQL pool don't get OOMKilled on startup.
Also add terminationGracePeriodSeconds: 15 so pods from failed/previous
builds release their node slot quickly instead of blocking new pod scheduling.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backend was OOMKilled during rolling update startup (Node.js + Socket.io +
MySQL pool exceeds 256Mi). Raised limit to 512Mi and request to 256Mi.
Jenkinsfile: show kubectl get pods immediately after apply so pod state
is visible in build logs. Added full diagnostics (describe + logs) in
post.failure block so the root cause of any future rollout failure is
visible without needing to SSH into the cluster.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
kubectl apply computes a 3-way merge. The base PVC has no storageClassName
(nil), but the already-bound PVC in the cluster has storageClassName=local-path.
This diff caused apply to attempt a mutation on a bound PVC — forbidden by k8s.
Fix: patch the PVC with storageClassName=local-path so desired state matches
live state and apply produces no diff on the PVC.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The mysql-data-pvc was already dynamically provisioned by the cluster's
'local-path' StorageClass. The overlay patch tried to change storageClassName
to 'manual' and volumeName on an already-bound PVC, which Kubernetes forbids:
"spec is immutable after creation except resources.requests"
Fixes:
- Remove mysql-pvc-patch from kustomization.yaml (PVC left as-is)
- Remove mysql-pv.yaml resource (not needed with dynamic provisioner)
- Add comment explaining when manual PV/PVC is needed vs not
Jenkinsfile: add --timeout and FQDN to smoke test curl; add comments
explaining MySQL Recreate strategy startup timing expectations.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Database fixes:
- Add hostPath.type=DirectoryOrCreate so kubelet auto-creates /mnt/data/mysql
- Add fsGroup=999 so MySQL process can write to the hostPath volume
- Add MYSQL_ROOT_HOST=% to allow backend pods to authenticate as root
- Fix liveness/readiness probes to include credentials (-p$MYSQL_ROOT_PASSWORD)
- Increase probe initialDelaySeconds (30/60s) for slow first-run init
- Add 15s grace sleep in backend initContainer after MySQL TCP is up
- Add persistentVolumeReclaimPolicy=Retain to prevent accidental data loss
- Explicit accessModes+resources in PVC patch to avoid list merge ambiguity
- Add nodeAffinity comment in PV for multi-node cluster guidance
Ingress/nginx fixes:
- Remove broken rewrite-target=/ that was rewriting all paths (incl /api) to /
- Route /socket.io directly to backend for WebSocket support
- Add /socket.io/ proxy location to both nginx.conf and K8s ConfigMap
Frontend fix:
- Persist currentUser to localStorage on login so page refresh no longer
clears session and redirects users back to the login page
Tooling:
- Add k8s/overlays/on-premise/deploy.sh for one-command deployment
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
