# SkillPatch skill: gke-observability This skill configures observability for Google Kubernetes Engine (GKE), covering Cloud Logging, Cloud Monitoring, and managed Prometheus setup. It provides a "golden path" of recommended defaults, including control-plane metrics (API server, scheduler, controller manager), and includes ready-to-use gcloud CLI commands for enabling the full monitoring suite. It is scoped specifically to GKE-native observability tooling. You (the agent) were given this URL and asked to use this skill. This is a **public** skill — no SkillPatch account, API key, or prior setup is required. Two ways to use it: **1. Use it right now** — the skill's complete file tree (SKILL.md + all reference files) is inlined below; read `SKILL.md` first, then follow it, consulting the other files as it directs. **2. Install the exact package onto disk** (recommended if you can run a shell — this reproduces the skill byte-for-byte, including any binary assets that can't be inlined): ```bash mkdir -p .claude/skills/gke-observability curl -sSL https://skillpatch.dev/install_skill/gke-observability | tar -xz -C .claude/skills/ ``` (`.claude/skills/` is Claude Code's convention; use whatever directory your agent loads skills from.) --- ## Skill files (1) - `SKILL.md` ### `SKILL.md` ````markdown --- name: gke-observability description: >- Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collection. Don't use to configure local application logging frameworks or external APMs outside GKE. metadata: category: CloudObservabilityAndMonitoring --- # GKE Observability This reference covers monitoring, logging, and metrics configuration for GKE. The golden path enables comprehensive observability including control-plane metrics. > **MCP Tools:** `gke:get_cluster`, `gke:list_k8s_events`, `gke:get_k8s_logs`, > `gke:get_k8s_cluster_info`, `gke:describe_k8s_resource`. **CLI-only:** `gcloud > container clusters update --monitoring=...`, `gcloud logging read` ## Golden Path Observability Defaults Setting | Golden Path Value | Notes --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ----- `loggingConfig` components | SYSTEM_COMPONENTS, WORKLOADS | Full workload logging `monitoringConfig` components | SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER | Full suite including control-plane `managedPrometheusConfig.enabled` | `true` | Google-managed Prometheus `advancedDatapathObservabilityConfig.enableMetrics` | `true` | Dataplane V2 flow metrics `loggingService` | `logging.googleapis.com/kubernetes` | Cloud Logging `monitoringService` | `monitoring.googleapis.com/kubernetes` | Cloud Monitoring ### Control-Plane Metrics (Golden Path Addition) The golden path adds three control-plane monitoring components not present in default clusters: | Component | What It Monitors | | -------------------- | ----------------------------------------------------- | | `APISERVER` | API server request latency, error rates, admission | : : webhook performance : | `SCHEDULER` | Scheduling latency, pending pods, scheduling failures | | `CONTROLLER_MANAGER` | Controller work queue depth, reconciliation latency | These are critical for diagnosing cluster-level issues (slow API responses, scheduling delays, stuck controllers). ## Enabling Full Monitoring ```bash # Enable golden path monitoring suite gcloud container clusters update --region \ --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,CADVISOR,KUBELET,DCGM \ --quiet # Enable Managed Prometheus gcloud container clusters update --region \ --enable-managed-prometheus \ --quiet # Enable Dataplane V2 observability metrics gcloud container clusters update --region \ --enable-dataplane-v2-flow-observability \ --quiet ``` ## Managed Prometheus Golden path enables Google Managed Prometheus for metrics collection and querying. **Querying metrics:** - Use Cloud Monitoring Metrics Explorer in the console - Use PromQL via the Prometheus UI or API - Grafana dashboards via Managed Grafana **Key GKE metrics:** | Metric | Source | Use | | --------------------------------------- | ------------------ | ------------- | | `container_cpu_usage_seconds_total` | cAdvisor | Pod CPU usage | | `container_memory_working_set_bytes` | cAdvisor | Pod memory | : : : usage : | `kube_pod_status_phase` | kube-state-metrics | Pod lifecycle | | `apiserver_request_duration_seconds` | API Server | Control plane | : : : latency : | `scheduler_scheduling_duration_seconds` | Scheduler | Scheduling | : : : performance : | `node_cpu_seconds_total` | Kubelet | Node CPU | | `DCGM_FI_DEV_GPU_UTIL` | DCGM | GPU | : : : utilization : ## Live Resource Usage (kubectl-only) No MCP or gcloud equivalent exists for live resource usage. Use `kubectl top`: ```bash kubectl top pods --all-namespaces --sort-by=cpu kubectl top nodes kubectl top pods --containers -n # per-container breakdown ``` ## Cloud Logging (gcloud-only) **Querying cluster logs** (no MCP equivalent — use `gcloud logging read`): ```bash # System component logs gcloud logging read \ 'resource.type="k8s_cluster" AND resource.labels.cluster_name=""' \ --project --limit 50 \ --quiet # Workload logs for a specific namespace gcloud logging read \ 'resource.type="k8s_container" AND resource.labels.cluster_name="" AND resource.labels.namespace_name=""' \ --project --limit 50 \ --quiet # Audit logs (who did what) gcloud logging read \ 'resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"' \ --project --limit 50 \ --quiet ``` ## Diagnostic Settings For security monitoring and troubleshooting, enable control-plane audit logs: ```bash # View current logging config gcloud container clusters describe --region \ --format="yaml(loggingConfig)" \ --quiet ``` ## Alerting Set up alerts for critical conditions: Condition | Metric | Threshold ----------------------- | --------------------------------------------------- | --------- High API server latency | `apiserver_request_duration_seconds` | P99 > 5s Pod crash loops | `kube_pod_container_status_restarts_total` | > 5 in 10min Node not ready | `kube_node_status_condition` | condition=Ready, status!=True High GPU utilization | `DCGM_FI_DEV_GPU_UTIL` | > 95% sustained PVC near capacity | `kubelet_volume_stats_used_bytes / capacity` | > 85% Scheduling failures | `scheduler_schedule_attempts_total{result="error"}` | > 0 ### Proposing Dashboards & Alerts (Production Rules) When designing or proposing alerting and dashboard strategies for GKE: 1. **Always explicitly name Google Cloud Monitoring** as the platform to implement these alerts and dashboards. 2. **Always include API server latency** (via `apiserver_request_duration_seconds` metric) on the dashboard as a critical indicator of control plane health, alongside node CPU/Memory and pod crash loops. ### Node Health (Production Rules) A comprehensive assessment of node health relies on analyzing these two metrics together: 1. **`kubernetes.io/node/status_condition`** (filtered by `status_condition="Ready"`): Use this to track healthy nodes. Note that it will only report values for nodes that have successfully bootstrapped. 2. **`compute.googleapis.com/instance_group/size`** (filtered by `instance_group_name="gke--.*"`): Use this to track the total number of nodes in a specific cluster. Note that it does not differentiate between healthy and unhealthy nodes. ## Cost Considerations Monitoring and logging have associated costs: - **Cloud Logging**: Charged per GiB ingested beyond free tier (50 GiB/project/month) - **Cloud Monitoring**: Free for GKE system metrics; custom metrics charged per time series - **Managed Prometheus**: Charged per samples ingested To reduce costs in non-production: ```bash # Reduce to system-only monitoring gcloud container clusters update --region \ --monitoring=SYSTEM \ --quiet ``` ## Distributed Tracing & Continuous Profiling (Recommended) **Not golden path defaults** — recommended for production microservice architectures and performance-sensitive workloads. - **Cloud Trace**: Add OpenTelemetry SDK to your app with the `opentelemetry-operations-go` (or equivalent) exporter. Traces appear in Cloud Trace console. Identifies cross-service latency bottlenecks. - **Cloud Profiler**: Add the Cloud Profiler agent to your app. Profiles CPU and memory usage in production with low overhead. Identifies hotspots and compares across versions. ## LQL Query Examples Common Logging Query Language patterns for GKE troubleshooting: ``` # Error logs for a specific container resource.type="k8s_container" AND resource.labels.container_name="my-app" AND severity>=ERROR # OOMKilled events resource.type="k8s_event" AND jsonPayload.reason="OOMKilling" # Pod scheduling failures resource.type="k8s_event" AND jsonPayload.reason="FailedScheduling" # Audit logs (who did what) resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com" ``` ## Supporting Links - [GKE system metrics](https://docs.cloud.google.com/monitoring/api/metrics_kubernetes) - [GKE Observability Documentation](https://cloud.google.com/kubernetes-engine/docs/concepts/observability) - [Google Cloud Managed Service for Prometheus](https://cloud.google.com/stackdriver/docs/managed-prometheus) - [Cloud Logging Query Language (LQL)](https://cloud.google.com/logging/docs/view/logging-query-language) - [Google Cloud Monitoring Alerts](https://cloud.google.com/monitoring/alerts) ````