During rolling updates with the default maxSurge=1, an extra surge pod was
created temporarily (3 pods instead of 2), causing all 3 nodes to report
"Insufficient CPU" and delaying scheduling past the Jenkins rollout timeout.
With maxSurge=0 / maxUnavailable=1, one old pod terminates first before a
new one starts — pod count stays at 2 throughout, no extra CPU needed.
Also increase Jenkins rollout timeout from 300s to 600s as a safety net
for CPU-constrained nodes that may still need extra scheduling time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause: backend deployment.yaml referenced secretKeyRef key: DB_USER and
key: DB_PASSWORD, but the live secret only has MYSQL_USER and MYSQL_PASSWORD.
kubectl apply reported secret/mysql-secret as "unchanged" (last-applied matched
desired) so the drift was never caught — new pods got CreateContainerConfigError.
Changes:
- backend/deployment.yaml: DB_USER → key: MYSQL_USER, DB_PASSWORD → key: MYSQL_PASSWORD
- mysql/deployment.yaml: add MYSQL_USER/MYSQL_PASSWORD env vars so the app user
(scrumapp) is created if MySQL ever reinitializes from a fresh PVC
- mysql/secret.yaml: remove stale commented-out block with old key names
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Increasing the request to 256Mi caused backend pods to be Pending with no
node assignment — the scheduler couldn't fit them alongside MySQL (512Mi
request) and existing pods on the on-premise nodes.
Memory REQUEST drives scheduling (how much the node reserves).
Memory LIMIT drives OOMKill (the actual cap at runtime).
Keep request at 128Mi so pods schedule, limit at 512Mi so Node.js +
Socket.io + MySQL pool don't get OOMKilled on startup.
Also add terminationGracePeriodSeconds: 15 so pods from failed/previous
builds release their node slot quickly instead of blocking new pod scheduling.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Backend was OOMKilled during rolling update startup (Node.js + Socket.io +
MySQL pool exceeds 256Mi). Raised limit to 512Mi and request to 256Mi.
Jenkinsfile: show kubectl get pods immediately after apply so pod state
is visible in build logs. Added full diagnostics (describe + logs) in
post.failure block so the root cause of any future rollout failure is
visible without needing to SSH into the cluster.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Database fixes:
- Add hostPath.type=DirectoryOrCreate so kubelet auto-creates /mnt/data/mysql
- Add fsGroup=999 so MySQL process can write to the hostPath volume
- Add MYSQL_ROOT_HOST=% to allow backend pods to authenticate as root
- Fix liveness/readiness probes to include credentials (-p$MYSQL_ROOT_PASSWORD)
- Increase probe initialDelaySeconds (30/60s) for slow first-run init
- Add 15s grace sleep in backend initContainer after MySQL TCP is up
- Add persistentVolumeReclaimPolicy=Retain to prevent accidental data loss
- Explicit accessModes+resources in PVC patch to avoid list merge ambiguity
- Add nodeAffinity comment in PV for multi-node cluster guidance
Ingress/nginx fixes:
- Remove broken rewrite-target=/ that was rewriting all paths (incl /api) to /
- Route /socket.io directly to backend for WebSocket support
- Add /socket.io/ proxy location to both nginx.conf and K8s ConfigMap
Frontend fix:
- Persist currentUser to localStorage on login so page refresh no longer
clears session and redirects users back to the login page
Tooling:
- Add k8s/overlays/on-premise/deploy.sh for one-command deployment
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
