test-project
This commit is contained in:
tusuii
830
DEPLOY.md
Normal file
830
DEPLOY.md
Normal file
@@ -0,0 +1,830 @@
|
||||
# On-Premise Kubernetes Deployment Guide
|
||||
|
||||
## React + Node.js + MySQL — Production Deployment
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Prerequisites](#1-prerequisites)
|
||||
2. [Project Structure](#2-project-structure)
|
||||
3. [Pre-Deployment Checklist](#3-pre-deployment-checklist)
|
||||
4. [Step-by-Step Deployment](#4-step-by-step-deployment)
|
||||
5. [Verify the Deployment](#5-verify-the-deployment)
|
||||
6. [Troubleshooting Guide](#6-troubleshooting-guide)
|
||||
7. [Updating the Application](#7-updating-the-application)
|
||||
8. [Teardown](#8-teardown)
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
### Required Tools (on your workstation)
|
||||
|
||||
| Tool | Minimum Version | Check Command |
|
||||
|------|----------------|---------------|
|
||||
| kubectl | v1.24+ | `kubectl version --client` |
|
||||
| kustomize | v5.0+ (or use kubectl built-in) | `kubectl kustomize --help` |
|
||||
| docker | v20+ | `docker --version` |
|
||||
| helm | v3.0+ | `helm version` |
|
||||
|
||||
### Required on the Kubernetes Cluster
|
||||
|
||||
| Component | Purpose | Install Command |
|
||||
|-----------|---------|-----------------|
|
||||
| NGINX Ingress Controller | Routes external traffic | See step 4.1 |
|
||||
| local-path Provisioner | Provides PersistentVolumes | See step 4.2 |
|
||||
| `myapp.local` DNS or hosts entry | Browser access | See step 4.6 |
|
||||
|
||||
### Cluster Requirements
|
||||
|
||||
- Kubernetes **v1.24+**
|
||||
- At least **1 worker node** with:
|
||||
- 1 CPU core free
|
||||
- 2 GB RAM free
|
||||
- 5 GB disk free (2 GB for MySQL PVC + OS overhead)
|
||||
- `kubectl` configured with cluster access (`kubectl get nodes` returns Ready)
|
||||
|
||||
---
|
||||
|
||||
## 2. Project Structure
|
||||
|
||||
```
|
||||
k8s/
|
||||
├── base/ # Shared manifests (all environments)
|
||||
│ ├── kustomization.yaml # Generates ConfigMap + Secret from property files
|
||||
│ ├── config.properties # Non-sensitive env vars (DB_HOST, DB_PORT, etc.)
|
||||
│ ├── secret.properties # Sensitive env vars (passwords) — gitignored
|
||||
│ ├── namespace.yaml # Namespace: react-mysql
|
||||
│ ├── mysql/
|
||||
│ │ ├── configmap-sql.yaml # Init SQL mounted at /docker-entrypoint-initdb.d/
|
||||
│ │ ├── statefulset.yaml # MySQL 8.0 StatefulSet with PVC + probes
|
||||
│ │ └── service.yaml # Headless ClusterIP service for StatefulSet DNS
|
||||
│ ├── backend/
|
||||
│ │ ├── deployment.yaml # Node.js API with init container (waits for MySQL)
|
||||
│ │ └── service.yaml # ClusterIP :3000
|
||||
│ └── frontend/
|
||||
│ ├── deployment.yaml # React static site served by `serve`
|
||||
│ └── service.yaml # ClusterIP :3000
|
||||
└── overlays/
|
||||
└── onpremise/
|
||||
├── kustomization.yaml # Extends base, adds ingress
|
||||
├── ingress.yaml # Two Ingress objects (API rewrite + frontend)
|
||||
└── patch-storageclass.yaml # Patches MySQL PVC → storageClassName: local-path
|
||||
```
|
||||
|
||||
### How Traffic Flows
|
||||
|
||||
```
|
||||
Browser
|
||||
│
|
||||
▼
|
||||
[myapp.local:80]
|
||||
│
|
||||
▼
|
||||
NGINX Ingress Controller
|
||||
├── /api/* ──rewrite /api→/──► backend Service :3000 ──► Node.js Pod
|
||||
│ │
|
||||
│ ▼
|
||||
│ MySQL Service
|
||||
│ │
|
||||
│ ▼
|
||||
│ mysql-0 Pod
|
||||
│ │
|
||||
│ ▼
|
||||
│ PVC (local-path 2Gi)
|
||||
│
|
||||
└── / ──────────────────────► frontend Service :3000 ──► React Pod
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Pre-Deployment Checklist
|
||||
|
||||
Run through this before deploying:
|
||||
|
||||
```bash
|
||||
# 1. Confirm cluster is reachable
|
||||
kubectl get nodes
|
||||
|
||||
# 2. Confirm you have cluster-admin rights
|
||||
kubectl auth can-i create namespace --all-namespaces
|
||||
|
||||
# 3. Confirm Docker Hub images exist
|
||||
docker manifest inspect subkamble/react-mysql-backend:latest
|
||||
docker manifest inspect subkamble/react-mysql-frontend:latest
|
||||
|
||||
# 4. Confirm secret.properties exists (it's gitignored, must be created manually)
|
||||
cat k8s/base/secret.properties
|
||||
```
|
||||
|
||||
Expected output for step 4:
|
||||
```
|
||||
DB_PASSWORD=pass123
|
||||
MYSQL_ROOT_PASSWORD=pass123
|
||||
MYSQL_PASSWORD=pass123
|
||||
```
|
||||
|
||||
> **If secret.properties is missing**, create it:
|
||||
> ```bash
|
||||
> cat > k8s/base/secret.properties <<EOF
|
||||
> DB_PASSWORD=pass123
|
||||
> MYSQL_ROOT_PASSWORD=pass123
|
||||
> MYSQL_PASSWORD=pass123
|
||||
> EOF
|
||||
> ```
|
||||
> Replace `pass123` with a strong password in production.
|
||||
|
||||
---
|
||||
|
||||
## 4. Step-by-Step Deployment
|
||||
|
||||
### Step 4.1 — Install NGINX Ingress Controller
|
||||
|
||||
```bash
|
||||
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
|
||||
helm repo update
|
||||
|
||||
helm install ingress-nginx ingress-nginx/ingress-nginx \
|
||||
--namespace ingress-nginx \
|
||||
--create-namespace \
|
||||
--set controller.replicaCount=1
|
||||
|
||||
# Wait for it to be ready (takes 1-2 min)
|
||||
kubectl wait deployment/ingress-nginx-controller \
|
||||
-n ingress-nginx \
|
||||
--for=condition=Available \
|
||||
--timeout=120s
|
||||
|
||||
# Verify
|
||||
kubectl get pods -n ingress-nginx
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
NAME READY STATUS RESTARTS
|
||||
ingress-nginx-controller-xxxxxxxxx-xxxxx 1/1 Running 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4.2 — Install local-path Storage Provisioner
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
|
||||
|
||||
# Wait for it to be ready
|
||||
kubectl wait deployment/local-path-provisioner \
|
||||
-n local-path-storage \
|
||||
--for=condition=Available \
|
||||
--timeout=60s
|
||||
|
||||
# Verify the StorageClass exists
|
||||
kubectl get storageclass
|
||||
```
|
||||
|
||||
Expected output includes:
|
||||
```
|
||||
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE
|
||||
local-path rancher.io/local-path Delete WaitForFirstConsumer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4.3 — Verify Kustomize Output (Dry Run)
|
||||
|
||||
Always preview what will be applied before deploying:
|
||||
|
||||
```bash
|
||||
kubectl kustomize k8s/overlays/onpremise
|
||||
```
|
||||
|
||||
Verify the output includes:
|
||||
- `storageClassName: local-path` in the StatefulSet volumeClaimTemplate
|
||||
- Both Ingress objects (`react-mysql-api-ingress` and `react-mysql-frontend-ingress`)
|
||||
- ConfigMap `app-config-<hash>` and Secret `app-secret-<hash>`
|
||||
|
||||
---
|
||||
|
||||
### Step 4.4 — Deploy
|
||||
|
||||
```bash
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
namespace/react-mysql created
|
||||
configmap/app-config-7fm29c526g created
|
||||
configmap/mysql-init-sql created
|
||||
secret/app-secret-7ht4dgtbc6 created
|
||||
service/backend created
|
||||
service/frontend created
|
||||
service/mysql created
|
||||
deployment.apps/backend created
|
||||
deployment.apps/frontend created
|
||||
statefulset.apps/mysql created
|
||||
ingress.networking.k8s.io/react-mysql-api-ingress created
|
||||
ingress.networking.k8s.io/react-mysql-frontend-ingress created
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4.5 — Watch Pods Come Up
|
||||
|
||||
```bash
|
||||
kubectl get pods -n react-mysql -w
|
||||
```
|
||||
|
||||
Expected sequence:
|
||||
|
||||
| Pod | First status | Final status | Time |
|
||||
|-----|-------------|-------------|------|
|
||||
| `mysql-0` | `ContainerCreating` | `1/1 Running` | ~30s |
|
||||
| `backend-xxx` | `Init:0/1` (waiting for MySQL) | `1/1 Running` | ~60s |
|
||||
| `frontend-xxx` | `Running` | `1/1 Running` | ~20s |
|
||||
|
||||
> **Note:** `backend` staying in `Init:0/1` is normal — the init container runs
|
||||
> `nc -z mysql 3306` in a loop until MySQL's readiness probe passes.
|
||||
> This prevents backend crash-loops due to MySQL not being ready.
|
||||
|
||||
---
|
||||
|
||||
### Step 4.6 — Configure DNS / hosts
|
||||
|
||||
Get the external IP of the Ingress controller:
|
||||
|
||||
```bash
|
||||
kubectl get svc -n ingress-nginx ingress-nginx-controller
|
||||
```
|
||||
|
||||
Look for the `EXTERNAL-IP` column. Then:
|
||||
|
||||
**Option A — /etc/hosts (single machine, testing):**
|
||||
```bash
|
||||
echo "<EXTERNAL-IP> myapp.local" | sudo tee -a /etc/hosts
|
||||
```
|
||||
|
||||
**Option B — Internal DNS (production):**
|
||||
Create an A record in your internal DNS server:
|
||||
```
|
||||
myapp.local → <EXTERNAL-IP>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Step 4.7 — Verify Ingress Has an Address
|
||||
|
||||
```bash
|
||||
kubectl get ingress -n react-mysql
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
NAME CLASS HOSTS ADDRESS PORTS
|
||||
react-mysql-api-ingress nginx myapp.local <EXTERNAL-IP> 80
|
||||
react-mysql-frontend-ingress nginx myapp.local <EXTERNAL-IP> 80
|
||||
```
|
||||
|
||||
> If `ADDRESS` is empty after 2 minutes, see [Ingress has no ADDRESS](#error-ingress-has-no-address).
|
||||
|
||||
---
|
||||
|
||||
## 5. Verify the Deployment
|
||||
|
||||
Run these checks in order:
|
||||
|
||||
```bash
|
||||
# 1. All pods Running
|
||||
kubectl get pods -n react-mysql
|
||||
|
||||
# 2. Database table exists
|
||||
kubectl exec mysql-0 -n react-mysql -- \
|
||||
mysql -u root -ppass123 -e "SHOW TABLES FROM appdb;"
|
||||
|
||||
# 3. Backend API responds
|
||||
kubectl port-forward svc/backend 3000:3000 -n react-mysql &
|
||||
curl localhost:3000/user
|
||||
# Expected: []
|
||||
kill %1
|
||||
|
||||
# 4. Ingress routes API correctly
|
||||
curl http://myapp.local/api/user
|
||||
# Expected: []
|
||||
|
||||
# 5. Post a user
|
||||
curl -X POST http://myapp.local/api/user \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"data":"Alice"}'
|
||||
# Expected: {"affectedRows":1,...}
|
||||
|
||||
# 6. User persists
|
||||
curl http://myapp.local/api/user
|
||||
# Expected: [{"id":1,"name":"Alice"}]
|
||||
|
||||
# 7. Frontend loads (check Content-Type is text/html)
|
||||
curl -si http://myapp.local/ | head -5
|
||||
|
||||
# 8. Static JS asset loads (must be application/javascript, NOT text/html)
|
||||
curl -si http://myapp.local/static/js/main.ff70bc14.js | head -3
|
||||
|
||||
# 9. PVC persistence — delete MySQL pod and verify data survives
|
||||
kubectl delete pod mysql-0 -n react-mysql
|
||||
kubectl wait pod/mysql-0 -n react-mysql --for=condition=Ready --timeout=90s
|
||||
sleep 10
|
||||
curl http://myapp.local/api/user
|
||||
# Expected: [{"id":1,"name":"Alice"}] ← Alice survived pod restart
|
||||
```
|
||||
|
||||
Open `http://myapp.local` in a browser — submit a name, it should appear in the table.
|
||||
|
||||
---
|
||||
|
||||
## 6. Troubleshooting Guide
|
||||
|
||||
---
|
||||
|
||||
### ERROR: `ImagePullBackOff` or `ErrImagePull`
|
||||
|
||||
**Symptom:**
|
||||
```
|
||||
NAME READY STATUS RESTARTS
|
||||
backend-xxx 0/1 ImagePullBackOff 0
|
||||
```
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl describe pod -n react-mysql -l app=backend | grep -A10 "Events:"
|
||||
```
|
||||
|
||||
**Cause A — Image does not exist on Docker Hub:**
|
||||
```
|
||||
Failed to pull image: pull access denied, repository does not exist
|
||||
```
|
||||
Fix: Build and push the images.
|
||||
```bash
|
||||
docker build -t subkamble/react-mysql-backend:latest ./backend
|
||||
docker push subkamble/react-mysql-backend:latest
|
||||
docker build -t subkamble/react-mysql-frontend:latest ./frontend
|
||||
docker push subkamble/react-mysql-frontend:latest
|
||||
```
|
||||
|
||||
**Cause B — Docker Hub rate limit:**
|
||||
```
|
||||
toomanyrequests: You have reached your pull rate limit
|
||||
```
|
||||
Fix: Pre-load images onto each node, then set `imagePullPolicy: Never`.
|
||||
```bash
|
||||
# On each cluster node (SSH in):
|
||||
docker pull subkamble/react-mysql-backend:latest
|
||||
docker pull subkamble/react-mysql-frontend:latest
|
||||
```
|
||||
Or use a private registry and update image references in the deployments.
|
||||
|
||||
**Cause C — Private registry, no credentials:**
|
||||
Create an `imagePullSecret`:
|
||||
```bash
|
||||
kubectl create secret docker-registry regcred \
|
||||
--docker-server=docker.io \
|
||||
--docker-username=<username> \
|
||||
--docker-password=<token> \
|
||||
-n react-mysql
|
||||
```
|
||||
Then add to both deployments:
|
||||
```yaml
|
||||
spec:
|
||||
imagePullSecrets:
|
||||
- name: regcred
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ERROR: `mysql-0` stays `Pending`
|
||||
|
||||
**Symptom:**
|
||||
```
|
||||
NAME READY STATUS RESTARTS
|
||||
mysql-0 0/1 Pending 0
|
||||
```
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl describe pod mysql-0 -n react-mysql | grep -A10 "Events:"
|
||||
kubectl get pvc -n react-mysql
|
||||
```
|
||||
|
||||
**Cause A — No StorageClass named `local-path`:**
|
||||
```
|
||||
0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims
|
||||
```
|
||||
Fix: Install the local-path provisioner (Step 4.2).
|
||||
```bash
|
||||
kubectl get storageclass # local-path must be present
|
||||
```
|
||||
|
||||
**Cause B — `WaitForFirstConsumer` binding (normal behavior):**
|
||||
The `local-path` StorageClass uses `WaitForFirstConsumer` — the PVC stays `Pending`
|
||||
until the pod is scheduled. This resolves automatically. Wait 30 seconds and check again.
|
||||
|
||||
**Cause C — Node has insufficient disk space:**
|
||||
```
|
||||
Insufficient disk space on node
|
||||
```
|
||||
Fix: Free up disk on the node or add a new node with enough space.
|
||||
```bash
|
||||
# Check node disk usage
|
||||
kubectl describe node <node-name> | grep -A5 "Allocatable"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ERROR: Frontend crash-loops (`CrashLoopBackOff`)
|
||||
|
||||
**Symptom:**
|
||||
```
|
||||
NAME READY STATUS RESTARTS
|
||||
frontend-xxx 0/1 CrashLoopBackOff 4
|
||||
```
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl logs -n react-mysql -l app=frontend --previous
|
||||
```
|
||||
|
||||
**Cause A — Old image cached on node (stale `npx serve`):**
|
||||
```
|
||||
npm warn exec The following package was not found and will be installed: serve@14.2.6
|
||||
npm error signal SIGTERM
|
||||
```
|
||||
The node has the old Docker image (CMD: `npx serve`) instead of the fixed one
|
||||
(CMD: `serve`). The liveness probe kills the container before `npx` finishes
|
||||
downloading `serve`.
|
||||
|
||||
Fix:
|
||||
```bash
|
||||
# Check which CMD the image on the node has
|
||||
# SSH into the affected node, then:
|
||||
docker inspect subkamble/react-mysql-frontend:latest \
|
||||
--format='{{json .Config.Cmd}}'
|
||||
# If output is ["npx","serve","-s","build"] → stale image
|
||||
|
||||
# Force remove and re-pull
|
||||
docker rmi -f subkamble/react-mysql-frontend:latest
|
||||
docker pull subkamble/react-mysql-frontend:latest
|
||||
|
||||
# Verify fix — output must be ["serve","-s","build"]
|
||||
docker inspect subkamble/react-mysql-frontend:latest \
|
||||
--format='{{json .Config.Cmd}}'
|
||||
|
||||
# Then restart the pod
|
||||
kubectl rollout restart deployment/frontend -n react-mysql
|
||||
```
|
||||
|
||||
**Cause B — Liveness probe firing too early:**
|
||||
The container starts but port 3000 isn't bound yet when the probe fires.
|
||||
```
|
||||
Liveness probe failed: Get "http://10.x.x.x:3000/": connection refused
|
||||
```
|
||||
Fix: Increase `initialDelaySeconds` in `k8s/base/frontend/deployment.yaml`:
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
port: 3000
|
||||
initialDelaySeconds: 40 # increase from 20 → 40
|
||||
periodSeconds: 20
|
||||
```
|
||||
Then re-apply:
|
||||
```bash
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ERROR: Backend stays in `Init:0/1`
|
||||
|
||||
**Symptom:**
|
||||
```
|
||||
NAME READY STATUS RESTARTS
|
||||
backend-xxx 0/1 Init:0/1 0
|
||||
```
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl logs -n react-mysql -l app=backend -c wait-for-mysql
|
||||
```
|
||||
|
||||
**Cause A — MySQL readiness probe hasn't passed yet (normal for ~30s):**
|
||||
Wait 60 seconds. The init container loops `nc -z mysql 3306` until MySQL's
|
||||
readiness probe (`mysqladmin ping`) passes.
|
||||
|
||||
**Cause B — MySQL pod is not Running:**
|
||||
```bash
|
||||
kubectl get pods -n react-mysql -l app=mysql
|
||||
```
|
||||
Fix the MySQL pod first (see MySQL errors below), then backend will unblock.
|
||||
|
||||
**Cause C — MySQL service DNS not resolving:**
|
||||
```bash
|
||||
# Test from another pod
|
||||
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never \
|
||||
-n react-mysql -- nslookup mysql
|
||||
```
|
||||
Expected: resolves to the MySQL pod IP. If it fails, check the headless service:
|
||||
```bash
|
||||
kubectl get svc mysql -n react-mysql
|
||||
# clusterIP must be "None"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ERROR: MySQL pod fails (`CrashLoopBackOff`)
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl logs mysql-0 -n react-mysql
|
||||
```
|
||||
|
||||
**Cause A — Wrong password in secret:**
|
||||
```
|
||||
[ERROR] Access denied for user 'root'@'localhost'
|
||||
```
|
||||
Fix: Verify `secret.properties` has matching passwords and re-apply:
|
||||
```bash
|
||||
cat k8s/base/secret.properties
|
||||
# DB_PASSWORD, MYSQL_ROOT_PASSWORD, MYSQL_PASSWORD must all match
|
||||
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
```
|
||||
|
||||
**Cause B — Corrupted PVC data (e.g. previous failed init):**
|
||||
```
|
||||
[ERROR] InnoDB: Cannot open datafile for read-write
|
||||
```
|
||||
Fix: Delete the PVC to wipe and re-initialize:
|
||||
```bash
|
||||
kubectl delete statefulset mysql -n react-mysql
|
||||
kubectl delete pvc mysql-data-mysql-0 -n react-mysql
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
```
|
||||
> **Warning:** This deletes all database data.
|
||||
|
||||
---
|
||||
|
||||
### ERROR: Ingress has no `ADDRESS`
|
||||
|
||||
**Symptom:**
|
||||
```
|
||||
NAME CLASS HOSTS ADDRESS PORTS
|
||||
react-mysql-api-ingress nginx myapp.local 80
|
||||
```
|
||||
ADDRESS column is empty after 2+ minutes.
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl get pods -n ingress-nginx
|
||||
kubectl describe ingress react-mysql-api-ingress -n react-mysql
|
||||
```
|
||||
|
||||
**Cause A — NGINX Ingress Controller not installed:**
|
||||
```bash
|
||||
kubectl get pods -n ingress-nginx
|
||||
# No resources found
|
||||
```
|
||||
Fix: Install it (Step 4.1).
|
||||
|
||||
**Cause B — Ingress controller pod not Ready:**
|
||||
```bash
|
||||
kubectl get pods -n ingress-nginx
|
||||
# STATUS = Pending or CrashLoopBackOff
|
||||
```
|
||||
Fix:
|
||||
```bash
|
||||
kubectl describe pod -n ingress-nginx -l app.kubernetes.io/component=controller \
|
||||
| grep -A10 "Events:"
|
||||
```
|
||||
Common sub-cause: node port conflict. Check if port 80/443 is in use on the node.
|
||||
|
||||
**Cause C — Wrong `ingressClassName`:**
|
||||
The Ingress specifies `ingressClassName: nginx` but the controller was installed
|
||||
with a different class name.
|
||||
```bash
|
||||
kubectl get ingressclass
|
||||
# NAME CONTROLLER
|
||||
# nginx k8s.io/ingress-nginx ← must match
|
||||
```
|
||||
If the class name differs, patch the ingress YAML or reinstall the controller.
|
||||
|
||||
---
|
||||
|
||||
### ERROR: White/blank screen in browser
|
||||
|
||||
**Symptom:** Page loads but shows only a blank white screen. No visible errors.
|
||||
|
||||
**Diagnose in browser:** Open DevTools → Console. You'll see:
|
||||
```
|
||||
Failed to load resource: the server responded with a status of 200 (OK)
|
||||
/static/js/main.ff70bc14.js
|
||||
```
|
||||
The JS file is returning HTML instead of JavaScript.
|
||||
|
||||
**Cause — Single Ingress with global rewrite-target:**
|
||||
If both frontend and API paths are in one Ingress object with
|
||||
`rewrite-target: /$2`, the rewrite applies to all paths — static assets
|
||||
(`/static/js/...`) get rewritten to `/`, returning `index.html`.
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
curl -si http://myapp.local/static/js/main.ff70bc14.js | head -3
|
||||
# BAD: Content-Type: text/html ← returning index.html
|
||||
# GOOD: Content-Type: application/javascript
|
||||
```
|
||||
|
||||
**Fix:** Ensure two separate Ingress objects are used — one for `/api` with
|
||||
the rewrite annotation, one for `/` without:
|
||||
```bash
|
||||
kubectl get ingress -n react-mysql
|
||||
# Must show TWO ingress objects:
|
||||
# react-mysql-api-ingress ← has rewrite-target annotation
|
||||
# react-mysql-frontend-ingress ← no rewrite annotation
|
||||
```
|
||||
If only one exists, re-apply the overlay:
|
||||
```bash
|
||||
kubectl delete ingress -n react-mysql --all
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ERROR: `curl http://myapp.local/api/user` returns 404
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller | tail -20
|
||||
```
|
||||
|
||||
**Cause A — `/api` path not matching:**
|
||||
Test the rewrite directly:
|
||||
```bash
|
||||
curl -si http://myapp.local/api/user
|
||||
# Check X-Original-URI header in nginx logs
|
||||
```
|
||||
|
||||
**Cause B — Backend service unreachable:**
|
||||
```bash
|
||||
kubectl exec -n react-mysql -l app=backend -- \
|
||||
wget -qO- localhost:3000/user
|
||||
```
|
||||
|
||||
**Cause C — Webhook admission error on Ingress creation:**
|
||||
```
|
||||
Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"
|
||||
```
|
||||
The NGINX ingress admission webhook wasn't ready when you applied. Fix:
|
||||
```bash
|
||||
# Wait for ingress controller to be fully ready first
|
||||
kubectl wait deployment/ingress-nginx-controller \
|
||||
-n ingress-nginx --for=condition=Available --timeout=120s
|
||||
|
||||
# Then re-apply
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ERROR: Data lost after MySQL pod restart
|
||||
|
||||
**Symptom:** After `kubectl delete pod mysql-0`, all rows are gone.
|
||||
|
||||
**Diagnose:**
|
||||
```bash
|
||||
kubectl get pvc -n react-mysql
|
||||
# STATUS must be Bound, not Pending or Lost
|
||||
```
|
||||
|
||||
**Cause A — PVC is in `Lost` state:**
|
||||
The underlying PersistentVolume was deleted or the node was replaced.
|
||||
```bash
|
||||
kubectl describe pvc mysql-data-mysql-0 -n react-mysql
|
||||
```
|
||||
This is unrecoverable without a backup. Re-initialize:
|
||||
```bash
|
||||
kubectl delete pvc mysql-data-mysql-0 -n react-mysql
|
||||
kubectl delete pod mysql-0 -n react-mysql
|
||||
# StatefulSet recreates the pod + new PVC
|
||||
```
|
||||
|
||||
**Cause B — Wrong StorageClass (data not actually persisted):**
|
||||
```bash
|
||||
kubectl get pvc mysql-data-mysql-0 -n react-mysql -o jsonpath='{.spec.storageClassName}'
|
||||
# Must output: local-path
|
||||
```
|
||||
If it shows `standard` (minikube default), the on-prem patch wasn't applied.
|
||||
Check the overlay is being used:
|
||||
```bash
|
||||
kubectl kustomize k8s/overlays/onpremise | grep storageClassName
|
||||
# Must output: storageClassName: local-path
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Updating the Application
|
||||
|
||||
### Update backend or frontend code
|
||||
|
||||
```bash
|
||||
# Rebuild image
|
||||
docker build -t subkamble/react-mysql-backend:latest ./backend
|
||||
docker push subkamble/react-mysql-backend:latest
|
||||
|
||||
# If nodes cache images locally, SSH into each node and force re-pull:
|
||||
# docker rmi -f subkamble/react-mysql-backend:latest
|
||||
# docker pull subkamble/react-mysql-backend:latest
|
||||
|
||||
# Rolling restart (zero downtime)
|
||||
kubectl rollout restart deployment/backend -n react-mysql
|
||||
|
||||
# Watch rollout
|
||||
kubectl rollout status deployment/backend -n react-mysql
|
||||
```
|
||||
|
||||
### Update ConfigMap values (non-sensitive config)
|
||||
|
||||
```bash
|
||||
# Edit k8s/base/config.properties, then re-apply
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
|
||||
# Restart pods to pick up new ConfigMap
|
||||
kubectl rollout restart deployment/backend deployment/frontend -n react-mysql
|
||||
```
|
||||
|
||||
### Update Secret values (passwords)
|
||||
|
||||
```bash
|
||||
# Edit k8s/base/secret.properties, then re-apply
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
kubectl rollout restart deployment/backend -n react-mysql
|
||||
```
|
||||
|
||||
> **Note:** Changing `MYSQL_ROOT_PASSWORD` after MySQL has initialized will NOT
|
||||
> change the database password — MySQL stores it internally in the PVC.
|
||||
> To change the DB password: exec into `mysql-0` and run `ALTER USER`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Teardown
|
||||
|
||||
### Remove the application only (keep cluster intact)
|
||||
|
||||
```bash
|
||||
kubectl delete namespace react-mysql
|
||||
```
|
||||
|
||||
> **Warning:** This deletes the PVC and all MySQL data permanently.
|
||||
|
||||
### Remove NGINX Ingress Controller
|
||||
|
||||
```bash
|
||||
helm uninstall ingress-nginx -n ingress-nginx
|
||||
kubectl delete namespace ingress-nginx
|
||||
```
|
||||
|
||||
### Remove local-path Provisioner
|
||||
|
||||
```bash
|
||||
kubectl delete -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference Card
|
||||
|
||||
```bash
|
||||
# Deploy
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
|
||||
# Check status
|
||||
kubectl get pods,svc,ingress,pvc -n react-mysql
|
||||
|
||||
# Stream all logs
|
||||
kubectl logs -n react-mysql -l app=mysql -f
|
||||
kubectl logs -n react-mysql -l app=backend -f
|
||||
kubectl logs -n react-mysql -l app=frontend -f
|
||||
|
||||
# Debug a crashing pod
|
||||
kubectl describe pod <pod-name> -n react-mysql
|
||||
kubectl logs <pod-name> -n react-mysql --previous
|
||||
|
||||
# Test API directly (bypass ingress)
|
||||
kubectl port-forward svc/backend 3000:3000 -n react-mysql
|
||||
curl localhost:3000/user
|
||||
|
||||
# MySQL shell
|
||||
kubectl exec -it mysql-0 -n react-mysql -- mysql -u root -ppass123 appdb
|
||||
|
||||
# Re-apply after manifest changes
|
||||
kubectl apply -k k8s/overlays/onpremise
|
||||
|
||||
# Full teardown
|
||||
kubectl delete namespace react-mysql
|
||||
```
|
||||
Reference in New Issue
Block a user