GKE Reliability

This reference covers high availability and reliability configuration for GKE clusters and workloads.
MCP Tools: get_cluster, get_k8s_resource, describe_k8s_resource, apply_k8s_manifest, list_k8s_events

Golden Path Reliability Defaults

SettingGolden Path ValueNotes
Cluster typeRegional (4 zones:Control plane replicated across
: : us-central1-a/b/c/f) : zones :
Upgrade strategySURGE (maxSurge: 1)Rolling upgrades with extra
: : : capacity :
Auto-repairtrueUnhealthy nodes replaced
: : : automatically :
Auto-upgradetrueNodes follow control plane
: : : version :
Release channelREGULARBalanced freshness and stability
Stateful HAEnabledLeader election for stateful
: : : workloads :

Workflows

1. Verify Cluster High Availability

text
# MCP (preferred)
get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>",
  readMask="location,locations,nodePools.locations")

# gcloud fallback
gcloud container clusters describe <CLUSTER> --region <REGION> \
  --format="json(location, locations)" \
  --quiet
  • If location is a region (e.g., us-central1), the control plane is regional
  • If locations has multiple entries, nodes span multiple zones

2. Pod Disruption Budgets (PDBs)

PDBs ensure minimum pod availability during voluntary disruptions (node upgrades, autoscaler scale-down).
Check existing PDBs:
text
# MCP (preferred)
get_k8s_resource(parent="...", resourceType="poddisruptionbudget")

# kubectl fallback
kubectl get pdb --all-namespaces
Create PDB:
yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
  namespace: default
spec:
  minAvailable: 2       # Or use maxUnavailable: 1
  selector:
    matchLabels:
      app: my-app
Every production Deployment with 2+ replicas should have a PDB.

3. Health Probes

Every production container should have liveness and readiness probes. Startup probes are recommended for slow-starting apps.
Check existing probes:
text
# MCP (preferred)
describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>")

# kubectl fallback
kubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"
Recommended probe configuration:
yaml
spec:
  containers:
  - name: app
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
    startupProbe:             # For slow-starting apps
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 30    # 30 * 5s = 150s max startup time
  • Readiness: Determines when a pod can accept traffic
  • Liveness: Determines when to restart a container
  • Startup: Disables liveness/readiness until the app is ready (prevents premature restarts)

4. Graceful Shutdown

Ensure applications handle SIGTERM and drain in-flight requests:
yaml
spec:
  terminationGracePeriodSeconds: 30    # Default; increase for long-running requests
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]  # Allow LB to deregister

5. Topology Spread Constraints

Distribute pods across zones and nodes to survive failures:
yaml
spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: my-app
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: my-app
  • Zone spread (DoNotSchedule): Hard requirement -- pods must be balanced across zones
  • Node spread (ScheduleAnyway): Best-effort -- prefer distribution but don't block scheduling

6. Replicas

Workload TypeMinimum ReplicasReason
Stateless web/API2Survive single pod/node
: : : failure :
Critical services3Survive zone failure with zone
: : : spread :
Stateful (databases)3 (with replication)Application-level quorum
Batch/jobs1Ephemeral by nature

Best Practices & Production Guidelines

  1. Regional clusters for production: Always use regional clusters to survive zone failures.
  2. PDBs for everything: Every production workload with 2+ replicas needs a PodDisruptionBudget (PDB) to protect against voluntary disruptions.
  3. Probes with Explicit Timeouts: Every production container must have both liveness and readiness probes defined. Always explicitly define initialDelaySeconds, periodSeconds, and timeoutSeconds for all probes. Never rely on the Kubernetes default timeout of 1 second if your application requires more, but always set a strict limit to prevent hanging connections.
  4. Zone spreading: Use topology spread constraints to distribute pods across failure domains (zones and nodes).
  5. Graceful shutdown: Handle SIGTERM and set appropriate terminationGracePeriodSeconds with a preStop sleep hook to allow load balancer deregistration.
  6. Maintenance windows: Schedule upgrades during low-traffic periods (see the gke-upgrades skill).

Run your whole company from one repo you own.

Start with one job and grow from there.