react-mysql/DEPLOY.md

# On-Premise Kubernetes Deployment Guide

## React + Node.js + MySQL — Production Deployment

---

## Table of Contents

1. [Prerequisites](#1-prerequisites)
2. [Project Structure](#2-project-structure)
3. [Pre-Deployment Checklist](#3-pre-deployment-checklist)
4. [Step-by-Step Deployment](#4-step-by-step-deployment)
5. [Verify the Deployment](#5-verify-the-deployment)
6. [Troubleshooting Guide](#6-troubleshooting-guide)
7. [Updating the Application](#7-updating-the-application)
8. [Teardown](#8-teardown)
9. [CI/CD — Jenkins Multibranch Pipeline](#9-cicd--jenkins-multibranch-pipeline)
10. [GitOps — ArgoCD (Reference)](#10-gitops--argocd-reference)

---

## 1. Prerequisites

### Required Tools (on your workstation)

| Tool | Minimum Version | Check Command |
|------|----------------|---------------|
| kubectl | v1.24+ | `kubectl version --client` |
| kustomize | v5.0+ (or use kubectl built-in) | `kubectl kustomize --help` |
| docker | v20+ | `docker --version` |
| helm | v3.0+ | `helm version` |

### Required on the Kubernetes Cluster

| Component | Purpose | Install Command |
|-----------|---------|-----------------|
| NGINX Ingress Controller | Routes external traffic | See step 4.1 |
| local-path Provisioner | Provides PersistentVolumes | See step 4.2 |
| `myapp.local` DNS or hosts entry | Browser access | See step 4.6 |

### Cluster Requirements

- Kubernetes **v1.24+**
- At least **1 worker node** with:
  - 1 CPU core free
  - 2 GB RAM free
  - 5 GB disk free (2 GB for MySQL PVC + OS overhead)
- `kubectl` configured with cluster access (`kubectl get nodes` returns Ready)

---

## 2. Project Structure

```
k8s/
├── base/                          # Shared manifests (all environments)
│   ├── kustomization.yaml         # Generates ConfigMap + Secret from property files
│   ├── config.properties          # Non-sensitive env vars (DB_HOST, DB_PORT, etc.)
│   ├── secret.properties          # Sensitive env vars (passwords) — gitignored
│   ├── namespace.yaml             # Namespace: react-mysql
│   ├── mysql/
│   │   ├── configmap-sql.yaml     # Init SQL mounted at /docker-entrypoint-initdb.d/
│   │   ├── statefulset.yaml       # MySQL 8.0 StatefulSet with PVC + probes
│   │   └── service.yaml           # Headless ClusterIP service for StatefulSet DNS
│   ├── backend/
│   │   ├── deployment.yaml        # Node.js API with init container (waits for MySQL)
│   │   └── service.yaml           # ClusterIP :3000
│   └── frontend/
│       ├── deployment.yaml        # React static site served by `serve`
│       └── service.yaml           # ClusterIP :3000
└── overlays/
    └── onpremise/
        ├── kustomization.yaml     # Extends base, adds ingress
        ├── ingress.yaml           # Two Ingress objects (API rewrite + frontend)
        └── patch-storageclass.yaml # Patches MySQL PVC → storageClassName: local-path
```

### How Traffic Flows

```
Browser
  │
  ▼
[myapp.local:80]
  │
  ▼
NGINX Ingress Controller
  ├── /api/* ──rewrite /api→/──► backend Service :3000 ──► Node.js Pod
  │                                                            │
  │                                                            ▼
  │                                                        MySQL Service
  │                                                            │
  │                                                            ▼
  │                                                        mysql-0 Pod
  │                                                            │
  │                                                            ▼
  │                                                        PVC (local-path 2Gi)
  │
  └── /    ──────────────────────► frontend Service :3000 ──► React Pod
```

---

## 3. Pre-Deployment Checklist

Run through this before deploying:

```bash
# 1. Confirm cluster is reachable
kubectl get nodes

# 2. Confirm you have cluster-admin rights
kubectl auth can-i create namespace --all-namespaces

# 3. Confirm Docker Hub images exist
docker manifest inspect subkamble/react-mysql-backend:latest
docker manifest inspect subkamble/react-mysql-frontend:latest

# 4. Confirm secret.properties exists (it's gitignored, must be created manually)
cat k8s/base/secret.properties
```

Expected output for step 4:
```
DB_PASSWORD=pass123
MYSQL_ROOT_PASSWORD=pass123
MYSQL_PASSWORD=pass123
```

> **If secret.properties is missing**, create it:
> ```bash
> cat > k8s/base/secret.properties <<EOF
> DB_PASSWORD=pass123
> MYSQL_ROOT_PASSWORD=pass123
> MYSQL_PASSWORD=pass123
> EOF
> ```
> Replace `pass123` with a strong password in production.

---

## 4. Step-by-Step Deployment

### Step 4.1 — Install NGINX Ingress Controller

```bash
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace \
  --set controller.replicaCount=1

# Wait for it to be ready (takes 1-2 min)
kubectl wait deployment/ingress-nginx-controller \
  -n ingress-nginx \
  --for=condition=Available \
  --timeout=120s

# Verify
kubectl get pods -n ingress-nginx
```

Expected output:
```
NAME                                        READY   STATUS    RESTARTS
ingress-nginx-controller-xxxxxxxxx-xxxxx    1/1     Running   0
```

---

### Step 4.2 — Install local-path Storage Provisioner

```bash
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

# Wait for it to be ready
kubectl wait deployment/local-path-provisioner \
  -n local-path-storage \
  --for=condition=Available \
  --timeout=60s

# Verify the StorageClass exists
kubectl get storageclass
```

Expected output includes:
```
NAME         PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE
local-path   rancher.io/local-path      Delete          WaitForFirstConsumer
```

---

### Step 4.3 — Verify Kustomize Output (Dry Run)

Always preview what will be applied before deploying:

```bash
kubectl kustomize k8s/overlays/onpremise
```

Verify the output includes:
- `storageClassName: local-path` in the StatefulSet volumeClaimTemplate
- Both Ingress objects (`react-mysql-api-ingress` and `react-mysql-frontend-ingress`)
- ConfigMap `app-config-<hash>` and Secret `app-secret-<hash>`

---

### Step 4.4 — Deploy

```bash
kubectl apply -k k8s/overlays/onpremise
```

Expected output:
```
namespace/react-mysql created
configmap/app-config-7fm29c526g created
configmap/mysql-init-sql created
secret/app-secret-7ht4dgtbc6 created
service/backend created
service/frontend created
service/mysql created
deployment.apps/backend created
deployment.apps/frontend created
statefulset.apps/mysql created
ingress.networking.k8s.io/react-mysql-api-ingress created
ingress.networking.k8s.io/react-mysql-frontend-ingress created
```

---

### Step 4.5 — Watch Pods Come Up

```bash
kubectl get pods -n react-mysql -w
```

Expected sequence:

| Pod | First status | Final status | Time |
|-----|-------------|-------------|------|
| `mysql-0` | `ContainerCreating` | `1/1 Running` | ~30s |
| `backend-xxx` | `Init:0/1` (waiting for MySQL) | `1/1 Running` | ~60s |
| `frontend-xxx` | `Running` | `1/1 Running` | ~20s |

> **Note:** `backend` staying in `Init:0/1` is normal — the init container runs
> `nc -z mysql 3306` in a loop until MySQL's readiness probe passes.
> This prevents backend crash-loops due to MySQL not being ready.

---

### Step 4.6 — Configure DNS / hosts

Get the external IP of the Ingress controller:

```bash
kubectl get svc -n ingress-nginx ingress-nginx-controller
```

Look for the `EXTERNAL-IP` column. Then:

**Option A — /etc/hosts (single machine, testing):**
```bash
echo "<EXTERNAL-IP> myapp.local" | sudo tee -a /etc/hosts
```

**Option B — Internal DNS (production):**
Create an A record in your internal DNS server:
```
myapp.local → <EXTERNAL-IP>
```

---

### Step 4.7 — Verify Ingress Has an Address

```bash
kubectl get ingress -n react-mysql
```

Expected output:
```
NAME                           CLASS   HOSTS         ADDRESS        PORTS
react-mysql-api-ingress        nginx   myapp.local   <EXTERNAL-IP>  80
react-mysql-frontend-ingress   nginx   myapp.local   <EXTERNAL-IP>  80
```

> If `ADDRESS` is empty after 2 minutes, see [Ingress has no ADDRESS](#error-ingress-has-no-address).

---

## 5. Verify the Deployment

Run these checks in order:

```bash
# 1. All pods Running
kubectl get pods -n react-mysql

# 2. Database table exists
kubectl exec mysql-0 -n react-mysql -- \
  mysql -u root -ppass123 -e "SHOW TABLES FROM appdb;"

# 3. Backend API responds
kubectl port-forward svc/backend 3000:3000 -n react-mysql &
curl localhost:3000/user
# Expected: []
kill %1

# 4. Ingress routes API correctly
curl http://myapp.local/api/user
# Expected: []

# 5. Post a user
curl -X POST http://myapp.local/api/user \
  -H "Content-Type: application/json" \
  -d '{"data":"Alice"}'
# Expected: {"affectedRows":1,...}

# 6. User persists
curl http://myapp.local/api/user
# Expected: [{"id":1,"name":"Alice"}]

# 7. Frontend loads (check Content-Type is text/html)
curl -si http://myapp.local/ | head -5

# 8. Static JS asset loads (must be application/javascript, NOT text/html)
curl -si http://myapp.local/static/js/main.ff70bc14.js | head -3

# 9. PVC persistence — delete MySQL pod and verify data survives
kubectl delete pod mysql-0 -n react-mysql
kubectl wait pod/mysql-0 -n react-mysql --for=condition=Ready --timeout=90s
sleep 10
curl http://myapp.local/api/user
# Expected: [{"id":1,"name":"Alice"}]  ← Alice survived pod restart
```

Open `http://myapp.local` in a browser — submit a name, it should appear in the table.

---

## 6. Troubleshooting Guide

---

### ERROR: `ImagePullBackOff` or `ErrImagePull`

**Symptom:**
```
NAME          READY   STATUS             RESTARTS
backend-xxx   0/1     ImagePullBackOff   0
```

**Diagnose:**
```bash
kubectl describe pod -n react-mysql -l app=backend | grep -A10 "Events:"
```

**Cause A — Image does not exist on Docker Hub:**
```
Failed to pull image: pull access denied, repository does not exist
```
Fix: Build and push the images.
```bash
docker build -t subkamble/react-mysql-backend:latest ./backend
docker push subkamble/react-mysql-backend:latest
docker build -t subkamble/react-mysql-frontend:latest ./frontend
docker push subkamble/react-mysql-frontend:latest
```

**Cause B — Docker Hub rate limit:**
```
toomanyrequests: You have reached your pull rate limit
```
Fix: Pre-load images onto each node, then set `imagePullPolicy: Never`.
```bash
# On each cluster node (SSH in):
docker pull subkamble/react-mysql-backend:latest
docker pull subkamble/react-mysql-frontend:latest
```
Or use a private registry and update image references in the deployments.

**Cause C — Private registry, no credentials:**
Create an `imagePullSecret`:
```bash
kubectl create secret docker-registry regcred \
  --docker-server=docker.io \
  --docker-username=<username> \
  --docker-password=<token> \
  -n react-mysql
```
Then add to both deployments:
```yaml
spec:
  imagePullSecrets:
    - name: regcred
```

---

### ERROR: `mysql-0` stays `Pending`

**Symptom:**
```
NAME      READY   STATUS    RESTARTS
mysql-0   0/1     Pending   0
```

**Diagnose:**
```bash
kubectl describe pod mysql-0 -n react-mysql | grep -A10 "Events:"
kubectl get pvc -n react-mysql
```

**Cause A — No StorageClass named `local-path`:**
```
0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims
```
Fix: Install the local-path provisioner (Step 4.2).
```bash
kubectl get storageclass   # local-path must be present
```

**Cause B — `WaitForFirstConsumer` binding (normal behavior):**
The `local-path` StorageClass uses `WaitForFirstConsumer` — the PVC stays `Pending`
until the pod is scheduled. This resolves automatically. Wait 30 seconds and check again.

**Cause C — Node has insufficient disk space:**
```
Insufficient disk space on node
```
Fix: Free up disk on the node or add a new node with enough space.
```bash
# Check node disk usage
kubectl describe node <node-name> | grep -A5 "Allocatable"
```

---

### ERROR: Frontend crash-loops (`CrashLoopBackOff`)

**Symptom:**
```
NAME            READY   STATUS             RESTARTS
frontend-xxx    0/1     CrashLoopBackOff   4
```

**Diagnose:**
```bash
kubectl logs -n react-mysql -l app=frontend --previous
```

**Cause A — Old image cached on node (stale `npx serve`):**
```
npm warn exec The following package was not found and will be installed: serve@14.2.6
npm error signal SIGTERM
```
The node has the old Docker image (CMD: `npx serve`) instead of the fixed one
(CMD: `serve`). The liveness probe kills the container before `npx` finishes
downloading `serve`.

Fix:
```bash
# Check which CMD the image on the node has
# SSH into the affected node, then:
docker inspect subkamble/react-mysql-frontend:latest \
  --format='{{json .Config.Cmd}}'
# If output is ["npx","serve","-s","build"] → stale image

# Force remove and re-pull
docker rmi -f subkamble/react-mysql-frontend:latest
docker pull subkamble/react-mysql-frontend:latest

# Verify fix — output must be ["serve","-s","build"]
docker inspect subkamble/react-mysql-frontend:latest \
  --format='{{json .Config.Cmd}}'

# Then restart the pod
kubectl rollout restart deployment/frontend -n react-mysql
```

**Cause B — Liveness probe firing too early:**
The container starts but port 3000 isn't bound yet when the probe fires.
```
Liveness probe failed: Get "http://10.x.x.x:3000/": connection refused
```
Fix: Increase `initialDelaySeconds` in `k8s/base/frontend/deployment.yaml`:
```yaml
livenessProbe:
  httpGet:
    path: /
    port: 3000
  initialDelaySeconds: 40    # increase from 20 → 40
  periodSeconds: 20
```
Then re-apply:
```bash
kubectl apply -k k8s/overlays/onpremise
```

---

### ERROR: Backend stays in `Init:0/1`

**Symptom:**
```
NAME          READY   STATUS     RESTARTS
backend-xxx   0/1     Init:0/1   0
```

**Diagnose:**
```bash
kubectl logs -n react-mysql -l app=backend -c wait-for-mysql
```

**Cause A — MySQL readiness probe hasn't passed yet (normal for ~30s):**
Wait 60 seconds. The init container loops `nc -z mysql 3306` until MySQL's
readiness probe (`mysqladmin ping`) passes.

**Cause B — MySQL pod is not Running:**
```bash
kubectl get pods -n react-mysql -l app=mysql
```
Fix the MySQL pod first (see MySQL errors below), then backend will unblock.

**Cause C — MySQL service DNS not resolving:**
```bash
# Test from another pod
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never \
  -n react-mysql -- nslookup mysql
```
Expected: resolves to the MySQL pod IP. If it fails, check the headless service:
```bash
kubectl get svc mysql -n react-mysql
# clusterIP must be "None"
```

---

### ERROR: MySQL pod fails (`CrashLoopBackOff`)

**Diagnose:**
```bash
kubectl logs mysql-0 -n react-mysql
```

**Cause A — Wrong password in secret:**
```
[ERROR] Access denied for user 'root'@'localhost'
```
Fix: Verify `secret.properties` has matching passwords and re-apply:
```bash
cat k8s/base/secret.properties
# DB_PASSWORD, MYSQL_ROOT_PASSWORD, MYSQL_PASSWORD must all match

kubectl apply -k k8s/overlays/onpremise
```

**Cause B — Corrupted PVC data (e.g. previous failed init):**
```
[ERROR] InnoDB: Cannot open datafile for read-write
```
Fix: Delete the PVC to wipe and re-initialize:
```bash
kubectl delete statefulset mysql -n react-mysql
kubectl delete pvc mysql-data-mysql-0 -n react-mysql
kubectl apply -k k8s/overlays/onpremise
```
> **Warning:** This deletes all database data.

---

### ERROR: Ingress has no `ADDRESS`

**Symptom:**
```
NAME                      CLASS   HOSTS         ADDRESS   PORTS
react-mysql-api-ingress   nginx   myapp.local             80
```
ADDRESS column is empty after 2+ minutes.

**Diagnose:**
```bash
kubectl get pods -n ingress-nginx
kubectl describe ingress react-mysql-api-ingress -n react-mysql
```

**Cause A — NGINX Ingress Controller not installed:**
```bash
kubectl get pods -n ingress-nginx
# No resources found
```
Fix: Install it (Step 4.1).

**Cause B — Ingress controller pod not Ready:**
```bash
kubectl get pods -n ingress-nginx
# STATUS = Pending or CrashLoopBackOff
```
Fix:
```bash
kubectl describe pod -n ingress-nginx -l app.kubernetes.io/component=controller \
  | grep -A10 "Events:"
```
Common sub-cause: node port conflict. Check if port 80/443 is in use on the node.

**Cause C — Wrong `ingressClassName`:**
The Ingress specifies `ingressClassName: nginx` but the controller was installed
with a different class name.
```bash
kubectl get ingressclass
# NAME     CONTROLLER
# nginx    k8s.io/ingress-nginx   ← must match
```
If the class name differs, patch the ingress YAML or reinstall the controller.

---

### ERROR: White/blank screen in browser

**Symptom:** Page loads but shows only a blank white screen. No visible errors.

**Diagnose in browser:** Open DevTools → Console. You'll see:
```
Failed to load resource: the server responded with a status of 200 (OK)
  /static/js/main.ff70bc14.js
```
The JS file is returning HTML instead of JavaScript.

**Cause — Single Ingress with global rewrite-target:**
If both frontend and API paths are in one Ingress object with
`rewrite-target: /$2`, the rewrite applies to all paths — static assets
(`/static/js/...`) get rewritten to `/`, returning `index.html`.

**Verify:**
```bash
curl -si http://myapp.local/static/js/main.ff70bc14.js | head -3
# BAD:  Content-Type: text/html   ← returning index.html
# GOOD: Content-Type: application/javascript
```

**Fix:** Ensure two separate Ingress objects are used — one for `/api` with
the rewrite annotation, one for `/` without:
```bash
kubectl get ingress -n react-mysql
# Must show TWO ingress objects:
# react-mysql-api-ingress      ← has rewrite-target annotation
# react-mysql-frontend-ingress ← no rewrite annotation
```
If only one exists, re-apply the overlay:
```bash
kubectl delete ingress -n react-mysql --all
kubectl apply -k k8s/overlays/onpremise
```

---

### ERROR: `curl http://myapp.local/api/user` returns 404

**Diagnose:**
```bash
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller | tail -20
```

**Cause A — `/api` path not matching:**
Test the rewrite directly:
```bash
curl -si http://myapp.local/api/user
# Check X-Original-URI header in nginx logs
```

**Cause B — Backend service unreachable:**
```bash
kubectl exec -n react-mysql -l app=backend -- \
  wget -qO- localhost:3000/user
```

**Cause C — Webhook admission error on Ingress creation:**
```
Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"
```
The NGINX ingress admission webhook wasn't ready when you applied. Fix:
```bash
# Wait for ingress controller to be fully ready first
kubectl wait deployment/ingress-nginx-controller \
  -n ingress-nginx --for=condition=Available --timeout=120s

# Then re-apply
kubectl apply -k k8s/overlays/onpremise
```

---

### ERROR: Data lost after MySQL pod restart

**Symptom:** After `kubectl delete pod mysql-0`, all rows are gone.

**Diagnose:**
```bash
kubectl get pvc -n react-mysql
# STATUS must be Bound, not Pending or Lost
```

**Cause A — PVC is in `Lost` state:**
The underlying PersistentVolume was deleted or the node was replaced.
```bash
kubectl describe pvc mysql-data-mysql-0 -n react-mysql
```
This is unrecoverable without a backup. Re-initialize:
```bash
kubectl delete pvc mysql-data-mysql-0 -n react-mysql
kubectl delete pod mysql-0 -n react-mysql
# StatefulSet recreates the pod + new PVC
```

**Cause B — Wrong StorageClass (data not actually persisted):**
```bash
kubectl get pvc mysql-data-mysql-0 -n react-mysql -o jsonpath='{.spec.storageClassName}'
# Must output: local-path
```
If it shows `standard` (minikube default), the on-prem patch wasn't applied.
Check the overlay is being used:
```bash
kubectl kustomize k8s/overlays/onpremise | grep storageClassName
# Must output: storageClassName: local-path
```

---

## 7. Updating the Application

### Update backend or frontend code

```bash
# Rebuild image
docker build -t subkamble/react-mysql-backend:latest ./backend
docker push subkamble/react-mysql-backend:latest

# If nodes cache images locally, SSH into each node and force re-pull:
# docker rmi -f subkamble/react-mysql-backend:latest
# docker pull subkamble/react-mysql-backend:latest

# Rolling restart (zero downtime)
kubectl rollout restart deployment/backend -n react-mysql

# Watch rollout
kubectl rollout status deployment/backend -n react-mysql
```

### Update ConfigMap values (non-sensitive config)

```bash
# Edit k8s/base/config.properties, then re-apply
kubectl apply -k k8s/overlays/onpremise

# Restart pods to pick up new ConfigMap
kubectl rollout restart deployment/backend deployment/frontend -n react-mysql
```

### Update Secret values (passwords)

```bash
# Edit k8s/base/secret.properties, then re-apply
kubectl apply -k k8s/overlays/onpremise
kubectl rollout restart deployment/backend -n react-mysql
```

> **Note:** Changing `MYSQL_ROOT_PASSWORD` after MySQL has initialized will NOT
> change the database password — MySQL stores it internally in the PVC.
> To change the DB password: exec into `mysql-0` and run `ALTER USER`.

---

## 8. Teardown

### Remove the application only (keep cluster intact)

```bash
kubectl delete namespace react-mysql
```

> **Warning:** This deletes the PVC and all MySQL data permanently.

### Remove NGINX Ingress Controller

```bash
helm uninstall ingress-nginx -n ingress-nginx
kubectl delete namespace ingress-nginx
```

### Remove local-path Provisioner

```bash
kubectl delete -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
```

---

---

## 9. CI/CD — Jenkins Multibranch Pipeline

This project ships with a `Jenkinsfile` at the repo root that automates the full
build → test → push → deploy lifecycle.

### 9.1 Prerequisites on the Jenkins Server

| Tool | Purpose | Install |
|------|---------|---------|
| Docker | Build & push images | `apt install docker.io` + add `jenkins` user to `docker` group |
| kubectl | Apply k8s manifests | [Official install guide](https://kubernetes.io/docs/tasks/tools/) |
| kustomize | Render overlays | `curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash` |
| NodeJS (v18+) | `npm ci` / `npm test` | Jenkins NodeJS plugin or system package |

```bash
# Add jenkins user to docker group (run on Jenkins host, then restart Jenkins)
sudo usermod -aG docker jenkins
sudo systemctl restart jenkins
```

### 9.2 Jenkins Plugins Required

Install via **Manage Jenkins → Plugins**:

| Plugin | Purpose |
|--------|---------|
| **Multibranch Pipeline** | Detects branches + PRs automatically |
| **Docker Pipeline** | Docker credential binding |
| **Credentials Binding** | Injects secrets into pipeline steps |
| **NodeJS** | Managed Node.js installations |
| **Git** | SCM checkout |

### 9.3 Configure Jenkins Credentials

Go to **Manage Jenkins → Credentials → System → Global credentials → Add**:

| Credential ID | Kind | Fields | Used for |
|---------------|------|--------|---------|
| `dockerhub-credentials` | Username with password | Username: `subkamble` / Password: Docker Hub PAT | `docker push` |
| `kubeconfig-secret` | Secret file | Upload your `~/.kube/config` | `kubectl` cluster access |

> **Docker Hub PAT:** Generate at hub.docker.com → Account Settings → Security → New Access Token (Read/Write scope).

> **kubeconfig tip:** If Jenkins runs inside the same cluster, use a ServiceAccount + RBAC instead of a kubeconfig file.

### 9.4 Create the Multibranch Pipeline Job

1. **Jenkins Dashboard → New Item**
2. Name: `react-mysql` → select **Multibranch Pipeline** → OK
3. Under **Branch Sources** → Add source → **Git**
   - Repository URL: your repo URL
   - Credentials: add repo access if private
4. Under **Build Configuration** → Mode: **by Jenkinsfile** → Script Path: `Jenkinsfile`
5. Under **Scan Multibranch Pipeline Triggers** → enable **Periodically if not otherwise run** (e.g. every 5 min), or configure a webhook (preferred)
6. Save → **Scan Multibranch Pipeline Now**

Jenkins will discover all branches and create sub-jobs automatically.

### 9.5 Webhook Setup (recommended — avoids polling)

**GitHub:**
1. Repo → Settings → Webhooks → Add webhook
2. Payload URL: `http://<JENKINS_URL>/multibranch-webhook-trigger/invoke?token=react-mysql`
3. Content type: `application/json`
4. Events: **Push** + **Pull requests**

Install the **Multibranch Scan Webhook Trigger** plugin on Jenkins to handle the token.

### 9.6 Branch Behavior Summary

| Branch | Build | Test | Push Image | Deploy |
|--------|-------|------|-----------|--------|
| `main` | Yes | Yes | Yes (`latest` + SHA tag) | `k8s/overlays/onpremise` (production) |
| `staging` | Yes | Yes | Yes (branch-SHA tag) | `k8s/overlays/minikube` (staging) |
| `develop` | Yes | Yes | No | No |
| `feature/**` | Yes | Yes | No | No |
| PR branches | Yes | Yes | No | No |

### 9.7 Image Tagging Convention

| Branch | Tag format | Example |
|--------|-----------|---------|
| `main` | `latest` + `<sha8>` | `latest`, `a1b2c3d4` |
| `staging` | `staging-<sha8>` | `staging-a1b2c3d4` |
| `feature/login` | `feature-login-<sha8>` | `feature-login-a1b2c3d4` |

### 9.8 First-Run Checklist

```bash
# Verify Jenkins can reach Docker Hub
docker login -u subkamble

# Verify Jenkins can reach the cluster
kubectl get nodes

# Verify kustomize is on PATH
kustomize version

# Trigger a manual build for main branch from Jenkins UI
# then watch the Console Output for each stage
```

### 9.9 Rollback

```bash
# List available image tags (via Docker Hub API or local docker images)
docker images subkamble/react-mysql-backend

# Pin a specific tag manually and re-deploy
cd k8s/base
kustomize edit set image \
    subkamble/react-mysql-backend=subkamble/react-mysql-backend:<previous-sha> \
    subkamble/react-mysql-frontend=subkamble/react-mysql-frontend:<previous-sha>
cd -
kubectl apply -k k8s/overlays/onpremise
kubectl rollout status deployment/backend  -n react-mysql
kubectl rollout status deployment/frontend -n react-mysql
```

---

## 10. GitOps — ArgoCD (Reference)

> The Jenkinsfile contains fully commented-out ArgoCD pipeline stages.
> Follow this section to activate GitOps mode.

### 10.1 Install ArgoCD

```bash
kubectl create namespace argocd
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all ArgoCD pods to be ready
kubectl wait --for=condition=Ready pods --all -n argocd --timeout=180s

# Expose the ArgoCD API server (choose one):

# Option A — Port-forward (local access only)
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Option B — LoadBalancer (on-prem with MetalLB or cloud)
kubectl patch svc argocd-server -n argocd \
  -p '{"spec":{"type":"LoadBalancer"}}'

# Option C — Ingress (add argocd.myapp.local to your Ingress controller)
```

### 10.2 Get Initial Admin Password

```bash
kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath="{.data.password}" | base64 -d && echo

# Login
argocd login localhost:8080 --username admin --password <above-password> --insecure

# Change password immediately
argocd account update-password
```

### 10.3 Apply ArgoCD Application Manifests

Create the two Application objects (one per environment).
These are documented in full inside the `Jenkinsfile` comments.
Here is the summary of what they do:

| Application | Watches branch | Overlay | Sync policy |
|-------------|---------------|---------|------------|
| `react-mysql-production` | `main` | `k8s/overlays/onpremise` | Automated (prune + selfHeal) |
| `react-mysql-staging` | `staging` | `k8s/overlays/minikube` | Automated (prune + selfHeal) |

```bash
# Copy the YAML blocks from the Jenkinsfile comments into files, then:
kubectl apply -f argocd/application-production.yaml
kubectl apply -f argocd/application-staging.yaml

# Verify apps are registered
argocd app list
```

### 10.4 Jenkins Credentials for GitOps Mode

Add these alongside the existing credentials:

| Credential ID | Kind | Purpose |
|---------------|------|---------|
| `argocd-auth-token` | Secret text | ArgoCD API token for `argocd app sync` |
| `gitops-ssh-key` | SSH username with private key | Push image tag updates back to Git |

**Generate an ArgoCD API token:**
```bash
argocd account generate-token --account admin
# Copy the output → paste as the 'argocd-auth-token' secret text
```

### 10.5 Activate GitOps in the Jenkinsfile

1. Open `Jenkinsfile`
2. Remove the active **Deploy** and **Smoke Test** stages (or leave them as fallback)
3. Uncomment the `Update GitOps Manifest` and `ArgoCD Sync` stage blocks in the GitOps section at the bottom
4. Replace `YOUR_ARGOCD_SERVER` with your ArgoCD server hostname/IP
5. Replace `YOUR_ORG` in the git clone URL with your GitHub org/username
6. Commit and push — Jenkins picks up the change automatically

### 10.6 GitOps Flow Diagram

```
Developer pushes code
        │
        ▼
   Jenkins CI
   ┌─────────────────────────────────────┐
   │ 1. npm ci → test                    │
   │ 2. docker build + push :<sha-tag>   │
   │ 3. kustomize edit set image         │
   │ 4. git commit + push [skip ci]      │
   └─────────────────┬───────────────────┘
                     │ Git push (image tag bump)
                     ▼
             Git Repository
             (k8s/base/kustomization.yaml updated)
                     │
                     │ ArgoCD polls every 3 min
                     │ (or Jenkins triggers sync via argocd CLI)
                     ▼
              ArgoCD detects drift
              ┌────────────────────────────┐
              │ kubectl apply -k overlay/  │
              │ prune removed resources    │
              │ self-heal manual changes   │
              └────────────┬───────────────┘
                           │
                           ▼
                    Kubernetes Cluster
                    (react-mysql namespace updated)
```

### 10.7 ArgoCD Quick Reference

```bash
# View app status
argocd app get react-mysql-production

# Force immediate sync (skip waiting for poll interval)
argocd app sync react-mysql-production

# Rollback to previous revision
argocd app history react-mysql-production
argocd app rollback react-mysql-production <revision-id>

# Pause auto-sync (for maintenance)
argocd app set react-mysql-production --sync-policy none

# Resume auto-sync
argocd app set react-mysql-production --sync-policy automated

# Delete app (does NOT delete k8s resources by default)
argocd app delete react-mysql-production

# Delete app AND k8s resources
argocd app delete react-mysql-production --cascade
```

---

## Quick Reference Card

```bash
# Deploy
kubectl apply -k k8s/overlays/onpremise

# Check status
kubectl get pods,svc,ingress,pvc -n react-mysql

# Stream all logs
kubectl logs -n react-mysql -l app=mysql   -f
kubectl logs -n react-mysql -l app=backend -f
kubectl logs -n react-mysql -l app=frontend -f

# Debug a crashing pod
kubectl describe pod <pod-name> -n react-mysql
kubectl logs <pod-name> -n react-mysql --previous

# Test API directly (bypass ingress)
kubectl port-forward svc/backend 3000:3000 -n react-mysql
curl localhost:3000/user

# MySQL shell
kubectl exec -it mysql-0 -n react-mysql -- mysql -u root -ppass123 appdb

# Re-apply after manifest changes
kubectl apply -k k8s/overlays/onpremise

# Full teardown
kubectl delete namespace react-mysql
```