react-mysql/DEPLOY.md

# On-Premise Kubernetes Deployment Guide

## React + Node.js + MySQL — Production Deployment

---

## Table of Contents

1. [Prerequisites](#1-prerequisites)
2. [Project Structure](#2-project-structure)
3. [Pre-Deployment Checklist](#3-pre-deployment-checklist)
4. [Step-by-Step Deployment](#4-step-by-step-deployment)
5. [Verify the Deployment](#5-verify-the-deployment)
6. [Troubleshooting Guide](#6-troubleshooting-guide)
7. [Updating the Application](#7-updating-the-application)
8. [Teardown](#8-teardown)

---

## 1. Prerequisites

### Required Tools (on your workstation)

| Tool | Minimum Version | Check Command |
|------|----------------|---------------|
| kubectl | v1.24+ | `kubectl version --client` |
| kustomize | v5.0+ (or use kubectl built-in) | `kubectl kustomize --help` |
| docker | v20+ | `docker --version` |
| helm | v3.0+ | `helm version` |

### Required on the Kubernetes Cluster

| Component | Purpose | Install Command |
|-----------|---------|-----------------|
| NGINX Ingress Controller | Routes external traffic | See step 4.1 |
| local-path Provisioner | Provides PersistentVolumes | See step 4.2 |
| `myapp.local` DNS or hosts entry | Browser access | See step 4.6 |

### Cluster Requirements

- Kubernetes **v1.24+**
- At least **1 worker node** with:
  - 1 CPU core free
  - 2 GB RAM free
  - 5 GB disk free (2 GB for MySQL PVC + OS overhead)
- `kubectl` configured with cluster access (`kubectl get nodes` returns Ready)

---

## 2. Project Structure

```
k8s/
├── base/                          # Shared manifests (all environments)
│   ├── kustomization.yaml         # Generates ConfigMap + Secret from property files
│   ├── config.properties          # Non-sensitive env vars (DB_HOST, DB_PORT, etc.)
│   ├── secret.properties          # Sensitive env vars (passwords) — gitignored
│   ├── namespace.yaml             # Namespace: react-mysql
│   ├── mysql/
│   │   ├── configmap-sql.yaml     # Init SQL mounted at /docker-entrypoint-initdb.d/
│   │   ├── statefulset.yaml       # MySQL 8.0 StatefulSet with PVC + probes
│   │   └── service.yaml           # Headless ClusterIP service for StatefulSet DNS
│   ├── backend/
│   │   ├── deployment.yaml        # Node.js API with init container (waits for MySQL)
│   │   └── service.yaml           # ClusterIP :3000
│   └── frontend/
│       ├── deployment.yaml        # React static site served by `serve`
│       └── service.yaml           # ClusterIP :3000
└── overlays/
    └── onpremise/
        ├── kustomization.yaml     # Extends base, adds ingress
        ├── ingress.yaml           # Two Ingress objects (API rewrite + frontend)
        └── patch-storageclass.yaml # Patches MySQL PVC → storageClassName: local-path
```

### How Traffic Flows

```
Browser
  │
  ▼
[myapp.local:80]
  │
  ▼
NGINX Ingress Controller
  ├── /api/* ──rewrite /api→/──► backend Service :3000 ──► Node.js Pod
  │                                                            │
  │                                                            ▼
  │                                                        MySQL Service
  │                                                            │
  │                                                            ▼
  │                                                        mysql-0 Pod
  │                                                            │
  │                                                            ▼
  │                                                        PVC (local-path 2Gi)
  │
  └── /    ──────────────────────► frontend Service :3000 ──► React Pod
```

---

## 3. Pre-Deployment Checklist

Run through this before deploying:

```bash
# 1. Confirm cluster is reachable
kubectl get nodes

# 2. Confirm you have cluster-admin rights
kubectl auth can-i create namespace --all-namespaces

# 3. Confirm Docker Hub images exist
docker manifest inspect subkamble/react-mysql-backend:latest
docker manifest inspect subkamble/react-mysql-frontend:latest

# 4. Confirm secret.properties exists (it's gitignored, must be created manually)
cat k8s/base/secret.properties
```

Expected output for step 4:
```
DB_PASSWORD=pass123
MYSQL_ROOT_PASSWORD=pass123
MYSQL_PASSWORD=pass123
```

> **If secret.properties is missing**, create it:
> ```bash
> cat > k8s/base/secret.properties <<EOF
> DB_PASSWORD=pass123
> MYSQL_ROOT_PASSWORD=pass123
> MYSQL_PASSWORD=pass123
> EOF
> ```
> Replace `pass123` with a strong password in production.

---

## 4. Step-by-Step Deployment

### Step 4.1 — Install NGINX Ingress Controller

```bash
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.replicaCount=1

# Wait for it to be ready (takes 1-2 min)
kubectl wait deployment/ingress-nginx-controller \
  -n ingress-nginx \
  --for=condition=Available \
  --timeout=120s

# Verify
kubectl get pods -n ingress-nginx
```

Expected output:
```
NAME                                        READY   STATUS    RESTARTS
ingress-nginx-controller-xxxxxxxxx-xxxxx    1/1     Running   0
```

---

### Step 4.2 — Install local-path Storage Provisioner

```bash
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

# Wait for it to be ready
kubectl wait deployment/local-path-provisioner \
  -n local-path-storage \
  --for=condition=Available \
  --timeout=60s

# Verify the StorageClass exists
kubectl get storageclass
```

Expected output includes:
```
NAME         PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE
local-path   rancher.io/local-path      Delete          WaitForFirstConsumer
```

---

### Step 4.3 — Verify Kustomize Output (Dry Run)

Always preview what will be applied before deploying:

```bash
kubectl kustomize k8s/overlays/onpremise
```

Verify the output includes:
- `storageClassName: local-path` in the StatefulSet volumeClaimTemplate
- Both Ingress objects (`react-mysql-api-ingress` and `react-mysql-frontend-ingress`)
- ConfigMap `app-config-<hash>` and Secret `app-secret-<hash>`

---

### Step 4.4 — Deploy

```bash
kubectl apply -k k8s/overlays/onpremise
```

Expected output:
```
namespace/react-mysql created
configmap/app-config-7fm29c526g created
configmap/mysql-init-sql created
secret/app-secret-7ht4dgtbc6 created
service/backend created
service/frontend created
service/mysql created
deployment.apps/backend created
deployment.apps/frontend created
statefulset.apps/mysql created
ingress.networking.k8s.io/react-mysql-api-ingress created
ingress.networking.k8s.io/react-mysql-frontend-ingress created
```

---

### Step 4.5 — Watch Pods Come Up

```bash
kubectl get pods -n react-mysql -w
```

Expected sequence:

| Pod | First status | Final status | Time |
|-----|-------------|-------------|------|
| `mysql-0` | `ContainerCreating` | `1/1 Running` | ~30s |
| `backend-xxx` | `Init:0/1` (waiting for MySQL) | `1/1 Running` | ~60s |
| `frontend-xxx` | `Running` | `1/1 Running` | ~20s |

> **Note:** `backend` staying in `Init:0/1` is normal — the init container runs
> `nc -z mysql 3306` in a loop until MySQL's readiness probe passes.
> This prevents backend crash-loops due to MySQL not being ready.

---

### Step 4.6 — Configure DNS / hosts

Get the external IP of the Ingress controller:

```bash
kubectl get svc -n ingress-nginx ingress-nginx-controller
```

Look for the `EXTERNAL-IP` column. Then:

**Option A — /etc/hosts (single machine, testing):**
```bash
echo "<EXTERNAL-IP> myapp.local" | sudo tee -a /etc/hosts
```

**Option B — Internal DNS (production):**
Create an A record in your internal DNS server:
```
myapp.local → <EXTERNAL-IP>
```

---

### Step 4.7 — Verify Ingress Has an Address

```bash
kubectl get ingress -n react-mysql
```

Expected output:
```
NAME                           CLASS   HOSTS         ADDRESS        PORTS
react-mysql-api-ingress        nginx   myapp.local   <EXTERNAL-IP>  80
react-mysql-frontend-ingress   nginx   myapp.local   <EXTERNAL-IP>  80
```

> If `ADDRESS` is empty after 2 minutes, see [Ingress has no ADDRESS](#error-ingress-has-no-address).

---

## 5. Verify the Deployment

Run these checks in order:

```bash
# 1. All pods Running
kubectl get pods -n react-mysql

# 2. Database table exists
kubectl exec mysql-0 -n react-mysql -- \
  mysql -u root -ppass123 -e "SHOW TABLES FROM appdb;"

# 3. Backend API responds
kubectl port-forward svc/backend 3000:3000 -n react-mysql &
curl localhost:3000/user
# Expected: []
kill %1

# 4. Ingress routes API correctly
curl http://myapp.local/api/user
# Expected: []

# 5. Post a user
curl -X POST http://myapp.local/api/user \
  -H "Content-Type: application/json" \
  -d '{"data":"Alice"}'
# Expected: {"affectedRows":1,...}

# 6. User persists
curl http://myapp.local/api/user
# Expected: [{"id":1,"name":"Alice"}]

# 7. Frontend loads (check Content-Type is text/html)
curl -si http://myapp.local/ | head -5

# 8. Static JS asset loads (must be application/javascript, NOT text/html)
curl -si http://myapp.local/static/js/main.ff70bc14.js | head -3

# 9. PVC persistence — delete MySQL pod and verify data survives
kubectl delete pod mysql-0 -n react-mysql
kubectl wait pod/mysql-0 -n react-mysql --for=condition=Ready --timeout=90s
sleep 10
curl http://myapp.local/api/user
# Expected: [{"id":1,"name":"Alice"}]  ← Alice survived pod restart
```

Open `http://myapp.local` in a browser — submit a name, it should appear in the table.

---

## 6. Troubleshooting Guide

---

### ERROR: `ImagePullBackOff` or `ErrImagePull`

**Symptom:**
```
NAME          READY   STATUS             RESTARTS
backend-xxx   0/1     ImagePullBackOff   0
```

**Diagnose:**
```bash
kubectl describe pod -n react-mysql -l app=backend | grep -A10 "Events:"
```

**Cause A — Image does not exist on Docker Hub:**
```
Failed to pull image: pull access denied, repository does not exist
```
Fix: Build and push the images.
```bash
docker build -t subkamble/react-mysql-backend:latest ./backend
docker push subkamble/react-mysql-backend:latest
docker build -t subkamble/react-mysql-frontend:latest ./frontend
docker push subkamble/react-mysql-frontend:latest
```

**Cause B — Docker Hub rate limit:**
```
toomanyrequests: You have reached your pull rate limit
```
Fix: Pre-load images onto each node, then set `imagePullPolicy: Never`.
```bash
# On each cluster node (SSH in):
docker pull subkamble/react-mysql-backend:latest
docker pull subkamble/react-mysql-frontend:latest
```
Or use a private registry and update image references in the deployments.

**Cause C — Private registry, no credentials:**
Create an `imagePullSecret`:
```bash
kubectl create secret docker-registry regcred \
  --docker-server=docker.io \
  --docker-username=<username> \
  --docker-password=<token> \
  -n react-mysql
```
Then add to both deployments:
```yaml
spec:
  imagePullSecrets:
    - name: regcred
```

---

### ERROR: `mysql-0` stays `Pending`

**Symptom:**
```
NAME      READY   STATUS    RESTARTS
mysql-0   0/1     Pending   0
```

**Diagnose:**
```bash
kubectl describe pod mysql-0 -n react-mysql | grep -A10 "Events:"
kubectl get pvc -n react-mysql
```

**Cause A — No StorageClass named `local-path`:**
```
0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims
```
Fix: Install the local-path provisioner (Step 4.2).
```bash
kubectl get storageclass   # local-path must be present
```

**Cause B — `WaitForFirstConsumer` binding (normal behavior):**
The `local-path` StorageClass uses `WaitForFirstConsumer` — the PVC stays `Pending`
until the pod is scheduled. This resolves automatically. Wait 30 seconds and check again.

**Cause C — Node has insufficient disk space:**
```
Insufficient disk space on node
```
Fix: Free up disk on the node or add a new node with enough space.
```bash
# Check node disk usage
kubectl describe node <node-name> | grep -A5 "Allocatable"
```

---

### ERROR: Frontend crash-loops (`CrashLoopBackOff`)

**Symptom:**
```
NAME            READY   STATUS             RESTARTS
frontend-xxx    0/1     CrashLoopBackOff   4
```

**Diagnose:**
```bash
kubectl logs -n react-mysql -l app=frontend --previous
```

**Cause A — Old image cached on node (stale `npx serve`):**
```
npm warn exec The following package was not found and will be installed: serve@14.2.6
npm error signal SIGTERM
```
The node has the old Docker image (CMD: `npx serve`) instead of the fixed one
(CMD: `serve`). The liveness probe kills the container before `npx` finishes
downloading `serve`.

Fix:
```bash
# Check which CMD the image on the node has
# SSH into the affected node, then:
docker inspect subkamble/react-mysql-frontend:latest \
  --format='{{json .Config.Cmd}}'
# If output is ["npx","serve","-s","build"] → stale image

# Force remove and re-pull
docker rmi -f subkamble/react-mysql-frontend:latest
docker pull subkamble/react-mysql-frontend:latest

# Verify fix — output must be ["serve","-s","build"]
docker inspect subkamble/react-mysql-frontend:latest \
  --format='{{json .Config.Cmd}}'

# Then restart the pod
kubectl rollout restart deployment/frontend -n react-mysql
```

**Cause B — Liveness probe firing too early:**
The container starts but port 3000 isn't bound yet when the probe fires.
```
Liveness probe failed: Get "http://10.x.x.x:3000/": connection refused
```
Fix: Increase `initialDelaySeconds` in `k8s/base/frontend/deployment.yaml`:
```yaml
livenessProbe:
  httpGet:
    path: /
    port: 3000
  initialDelaySeconds: 40    # increase from 20 → 40
  periodSeconds: 20
```
Then re-apply:
```bash
kubectl apply -k k8s/overlays/onpremise
```

---

### ERROR: Backend stays in `Init:0/1`

**Symptom:**
```
NAME          READY   STATUS     RESTARTS
backend-xxx   0/1     Init:0/1   0
```

**Diagnose:**
```bash
kubectl logs -n react-mysql -l app=backend -c wait-for-mysql
```

**Cause A — MySQL readiness probe hasn't passed yet (normal for ~30s):**
Wait 60 seconds. The init container loops `nc -z mysql 3306` until MySQL's
readiness probe (`mysqladmin ping`) passes.

**Cause B — MySQL pod is not Running:**
```bash
kubectl get pods -n react-mysql -l app=mysql
```
Fix the MySQL pod first (see MySQL errors below), then backend will unblock.

**Cause C — MySQL service DNS not resolving:**
```bash
# Test from another pod
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never \
  -n react-mysql -- nslookup mysql
```
Expected: resolves to the MySQL pod IP. If it fails, check the headless service:
```bash
kubectl get svc mysql -n react-mysql
# clusterIP must be "None"
```

---

### ERROR: MySQL pod fails (`CrashLoopBackOff`)

**Diagnose:**
```bash
kubectl logs mysql-0 -n react-mysql
```

**Cause A — Wrong password in secret:**
```
[ERROR] Access denied for user 'root'@'localhost'
```
Fix: Verify `secret.properties` has matching passwords and re-apply:
```bash
cat k8s/base/secret.properties
# DB_PASSWORD, MYSQL_ROOT_PASSWORD, MYSQL_PASSWORD must all match

kubectl apply -k k8s/overlays/onpremise
```

**Cause B — Corrupted PVC data (e.g. previous failed init):**
```
[ERROR] InnoDB: Cannot open datafile for read-write
```
Fix: Delete the PVC to wipe and re-initialize:
```bash
kubectl delete statefulset mysql -n react-mysql
kubectl delete pvc mysql-data-mysql-0 -n react-mysql
kubectl apply -k k8s/overlays/onpremise
```
> **Warning:** This deletes all database data.

---

### ERROR: Ingress has no `ADDRESS`

**Symptom:**
```
NAME                      CLASS   HOSTS         ADDRESS   PORTS
react-mysql-api-ingress   nginx   myapp.local             80
```
ADDRESS column is empty after 2+ minutes.

**Diagnose:**
```bash
kubectl get pods -n ingress-nginx
kubectl describe ingress react-mysql-api-ingress -n react-mysql
```

**Cause A — NGINX Ingress Controller not installed:**
```bash
kubectl get pods -n ingress-nginx
# No resources found
```
Fix: Install it (Step 4.1).

**Cause B — Ingress controller pod not Ready:**
```bash
kubectl get pods -n ingress-nginx
# STATUS = Pending or CrashLoopBackOff
```
Fix:
```bash
kubectl describe pod -n ingress-nginx -l app.kubernetes.io/component=controller \
  | grep -A10 "Events:"
```
Common sub-cause: node port conflict. Check if port 80/443 is in use on the node.

**Cause C — Wrong `ingressClassName`:**
The Ingress specifies `ingressClassName: nginx` but the controller was installed
with a different class name.
```bash
kubectl get ingressclass
# NAME     CONTROLLER
# nginx    k8s.io/ingress-nginx   ← must match
```
If the class name differs, patch the ingress YAML or reinstall the controller.

---

### ERROR: White/blank screen in browser

**Symptom:** Page loads but shows only a blank white screen. No visible errors.

**Diagnose in browser:** Open DevTools → Console. You'll see:
```
Failed to load resource: the server responded with a status of 200 (OK)
  /static/js/main.ff70bc14.js
```
The JS file is returning HTML instead of JavaScript.

**Cause — Single Ingress with global rewrite-target:**
If both frontend and API paths are in one Ingress object with
`rewrite-target: /$2`, the rewrite applies to all paths — static assets
(`/static/js/...`) get rewritten to `/`, returning `index.html`.

**Verify:**
```bash
curl -si http://myapp.local/static/js/main.ff70bc14.js | head -3
# BAD:  Content-Type: text/html   ← returning index.html
# GOOD: Content-Type: application/javascript
```

**Fix:** Ensure two separate Ingress objects are used — one for `/api` with
the rewrite annotation, one for `/` without:
```bash
kubectl get ingress -n react-mysql
# Must show TWO ingress objects:
# react-mysql-api-ingress      ← has rewrite-target annotation
# react-mysql-frontend-ingress ← no rewrite annotation
```
If only one exists, re-apply the overlay:
```bash
kubectl delete ingress -n react-mysql --all
kubectl apply -k k8s/overlays/onpremise
```

---

### ERROR: `curl http://myapp.local/api/user` returns 404

**Diagnose:**
```bash
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller | tail -20
```

**Cause A — `/api` path not matching:**
Test the rewrite directly:
```bash
curl -si http://myapp.local/api/user
# Check X-Original-URI header in nginx logs
```

**Cause B — Backend service unreachable:**
```bash
kubectl exec -n react-mysql -l app=backend -- \
  wget -qO- localhost:3000/user
```

**Cause C — Webhook admission error on Ingress creation:**
```
Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"
```
The NGINX ingress admission webhook wasn't ready when you applied. Fix:
```bash
# Wait for ingress controller to be fully ready first
kubectl wait deployment/ingress-nginx-controller \
  -n ingress-nginx --for=condition=Available --timeout=120s

# Then re-apply
kubectl apply -k k8s/overlays/onpremise
```

---

### ERROR: Data lost after MySQL pod restart

**Symptom:** After `kubectl delete pod mysql-0`, all rows are gone.

**Diagnose:**
```bash
kubectl get pvc -n react-mysql
# STATUS must be Bound, not Pending or Lost
```

**Cause A — PVC is in `Lost` state:**
The underlying PersistentVolume was deleted or the node was replaced.
```bash
kubectl describe pvc mysql-data-mysql-0 -n react-mysql
```
This is unrecoverable without a backup. Re-initialize:
```bash
kubectl delete pvc mysql-data-mysql-0 -n react-mysql
kubectl delete pod mysql-0 -n react-mysql
# StatefulSet recreates the pod + new PVC
```

**Cause B — Wrong StorageClass (data not actually persisted):**
```bash
kubectl get pvc mysql-data-mysql-0 -n react-mysql -o jsonpath='{.spec.storageClassName}'
# Must output: local-path
```
If it shows `standard` (minikube default), the on-prem patch wasn't applied.
Check the overlay is being used:
```bash
kubectl kustomize k8s/overlays/onpremise | grep storageClassName
# Must output: storageClassName: local-path
```

---

## 7. Updating the Application

### Update backend or frontend code

```bash
# Rebuild image
docker build -t subkamble/react-mysql-backend:latest ./backend
docker push subkamble/react-mysql-backend:latest

# If nodes cache images locally, SSH into each node and force re-pull:
# docker rmi -f subkamble/react-mysql-backend:latest
# docker pull subkamble/react-mysql-backend:latest

# Rolling restart (zero downtime)
kubectl rollout restart deployment/backend -n react-mysql

# Watch rollout
kubectl rollout status deployment/backend -n react-mysql
```

### Update ConfigMap values (non-sensitive config)

```bash
# Edit k8s/base/config.properties, then re-apply
kubectl apply -k k8s/overlays/onpremise

# Restart pods to pick up new ConfigMap
kubectl rollout restart deployment/backend deployment/frontend -n react-mysql
```

### Update Secret values (passwords)

```bash
# Edit k8s/base/secret.properties, then re-apply
kubectl apply -k k8s/overlays/onpremise
kubectl rollout restart deployment/backend -n react-mysql
```

> **Note:** Changing `MYSQL_ROOT_PASSWORD` after MySQL has initialized will NOT
> change the database password — MySQL stores it internally in the PVC.
> To change the DB password: exec into `mysql-0` and run `ALTER USER`.

---

## 8. Teardown

### Remove the application only (keep cluster intact)

```bash
kubectl delete namespace react-mysql
```

> **Warning:** This deletes the PVC and all MySQL data permanently.

### Remove NGINX Ingress Controller

```bash
helm uninstall ingress-nginx -n ingress-nginx
kubectl delete namespace ingress-nginx
```

### Remove local-path Provisioner

```bash
kubectl delete -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
```

---

## Quick Reference Card

```bash
# Deploy
kubectl apply -k k8s/overlays/onpremise

# Check status
kubectl get pods,svc,ingress,pvc -n react-mysql

# Stream all logs
kubectl logs -n react-mysql -l app=mysql   -f
kubectl logs -n react-mysql -l app=backend -f
kubectl logs -n react-mysql -l app=frontend -f

# Debug a crashing pod
kubectl describe pod <pod-name> -n react-mysql
kubectl logs <pod-name> -n react-mysql --previous

# Test API directly (bypass ingress)
kubectl port-forward svc/backend 3000:3000 -n react-mysql
curl localhost:3000/user

# MySQL shell
kubectl exec -it mysql-0 -n react-mysql -- mysql -u root -ppass123 appdb

# Re-apply after manifest changes
kubectl apply -k k8s/overlays/onpremise

# Full teardown
kubectl delete namespace react-mysql
```