check-metrics

# Check Metrics Skill

This skill helps you query Prometheus metrics and analyze platform performance.

## When to Use

- User asks about resource usage (CPU, memory, disk)
- Investigating performance issues
- Checking service health metrics
- After deployments to verify metrics collection
- Analyzing platform capacity and scaling needs

## What This Skill Does

1. **Query Metrics**: Execute PromQL queries against Prometheus
2. **Resource Usage**: Check CPU, memory, disk usage
3. **Service Health**: Verify service metrics and availability
4. **Performance Analysis**: Analyze request rates, latency, errors
5. **Capacity Planning**: Review resource trends

## Examples

### Access Prometheus UI

**Prometheus UI**: Port-forward to access locally
```bash
kubectl port-forward -n observability svc/prometheus 9090:9090 &
# Open http://localhost:9090
```

**Grafana Explore**: https://grafana.localtest.me:9443/explore
- Select **Prometheus** datasource
- Enter PromQL queries

### Query Metrics via CLI

```bash
# Basic query
kubectl exec -n observability deployment/grafana -- \
  curl -s -G 'http://prometheus.observability.svc:9090/api/v1/query' \
  --data-urlencode 'query=up' | python3 -m json.tool

# Query with time range
kubectl exec -n observability deployment/grafana -- \
  curl -s -G 'http://prometheus.observability.svc:9090/api/v1/query_range' \
  --data-urlencode 'query=rate(container_cpu_usage_seconds_total[5m])' \
  --data-urlencode 'start='$(date -u -v-1H +%s) \
  --data-urlencode 'end='$(date -u +%s) \
  --data-urlencode 'step=60' | python3 -m json.tool
```

## Common PromQL Queries

### Service Health

```promql
# Check if services are up
up{job="kubernetes-pods"}

# Count running pods by namespace
count by (kubernetes_namespace) (up == 1)

# Check deployment replicas
kube_deployment_status_replicas_available

# Check StatefulSet replicas
kube_statefulset_status_replicas_ready
```

### CPU Usage

```promql
# Pod CPU usage (percentage of limit)
sum(rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[5m])) by (namespace, pod, container)
/ sum(container_spec_cpu_quota{container!="",container!="POD"} / container_spec_cpu_period{container!="",container!="POD"}) by (namespace, pod, container) * 100

# Node CPU usage
100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Top CPU consuming pods
topk(10,
  sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (namespace, pod)
)
```

### Memory Usage

```promql
# Pod memory usage (percentage of limit)
sum(container_memory_working_set_bytes{container!="",container!="POD"}) by (namespace, pod, container)
/ sum(container_spec_memory_limit_bytes{container!="",container!="POD"}) by (namespace, pod, container) * 100

# Pod memory usage in bytes
container_memory_working_set_bytes{container!="",container!="POD"}

# Top memory consuming pods
topk(10,
  sum(container_memory_working_set_bytes{container!=""}) by (namespace, pod)
)
```

### Network Traffic

```promql
# Network receive rate
rate(container_network_receive_bytes_total[5m])

# Network transmit rate
rate(container_network_transmit_bytes_total[5m])

# Total network I/O by pod
sum by (pod) (
  rate(container_network_receive_bytes_total[5m]) +
  rate(container_network_transmit_bytes_total[5m])
)
```

### Disk Usage

```promql
# Filesystem usage percentage
(kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100

# PVC usage by namespace
sum by (namespace, persistentvolumeclaim) (
  kubelet_volume_stats_used_bytes
)

# Disk I/O rate
rate(container_fs_writes_bytes_total[5m])
```

### Pod Status

```promql
# Pods not running
kube_pod_status_phase{phase!="Running"}

# Pod restart count
kube_pod_container_status_restarts_total

# Pods waiting (pending)
kube_pod_status_phase{phase="Pending"}

# Pods in crash loop
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}
```

### Request Metrics (if instrumented)

```promql
# Request rate
rate(http_requests_total[5m])

# Error rate
rate(http_requests_total{status=~"5.."}[5m])

# Request latency (p95)
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
)
```

## Check Specific Components

### Prometheus Metrics
```bash
# Check Prometheus scrape targets
kubectl exec -n observability deployment/grafana -- \
  curl -s 'http://prometheus.observability.svc:9090/api/v1/targets' | python3 -m json.tool

# Prometheus storage size
kubectl exec -n observability deployment/grafana -- \
  curl -s 'http://prometheus.observability.svc:9090/api/v1/status/tsdb' | python3 -m json.tool
```

### Grafana Metrics
```promql
# Grafana datasource queries
grafana_datasource_request_total

# Grafana dashboard loads
grafana_page_response_status_total
```

### Keycloak Metrics (if exposed)
```promql
# Keycloak sessions
keycloak_sessions

# Keycloak login failures
keycloak_failed_login_attempts
```

### Istio Metrics
```promql
# Istio requests
istio_requests_total

# Istio request duration
histogram_quantile(0.95,
  rate(istio_request_duration_milliseconds_bucket[5m])
)

# Istio error rate
rate(istio_requests_total{response_code=~"5.."}[5m])
```

## Resource Monitoring via kubectl

### Quick Resource Check
```bash
# Node resources
kubectl top nodes

# Pod resources (all namespaces)
kubectl top pods -A --sort-by=memory

# Pod resources (specific namespace)
kubectl top pods -n observability --sort-by=cpu

# Container resources in pod
kubectl top pod <pod-name> -n <namespace> --containers
```

### Resource Limits and Requests
```bash
# Show resource requests/limits for deployment
kubectl describe deployment <name> -n <namespace> | grep -A 5 "Limits\|Requests"

# Show all pod resource requests
kubectl get pods -n <namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].resources}{"\n"}{end}'
```

## Grafana Dashboards

**Access**: https://grafana.localtest.me:9443/dashboards

**Key Dashboards**:
1. **Kubernetes / Compute Resources / Cluster** - Overall cluster metrics
2. **Kubernetes / Compute Resources / Namespace (Pods)** - Per-namespace pod resources
3. **Kubernetes / Compute Resources / Pod** - Individual pod metrics
4. **Prometheus** - Prometheus self-monitoring
5. **Loki Logs** - Log volume and patterns
6. **Istio Mesh** - Service mesh metrics

### Create Custom Queries in Grafana

1. Navigate to **Explore** (compass icon in sidebar)
2. Select **Prometheus** datasource
3. Enter PromQL query
4. Click "Run query"
5. Optionally save to dashboard

## Troubleshooting with Metrics

### Issue: High CPU Usage
```promql
# Find pods using >80% CPU
sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace, pod, container)
/ sum(container_spec_cpu_quota / container_spec_cpu_period) by (namespace, pod, container) * 100 > 80
```

### Issue: High Memory Usage
```promql
# Find pods using >80% memory
sum(container_memory_working_set_bytes) by (namespace, pod, container)
/ sum(container_spec_memory_limit_bytes) by (namespace, pod, container) * 100 > 80
```

### Issue: Service Not Responding
```promql
# Check if service endpoints are up
up{job="kubernetes-service-endpoints"}

# Check scrape failures
up == 0
```

### Issue: Disk Full
```promql
# Find PVCs >80% full
(kubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytes) * 100 > 80
```

## Alert Query Testing

When investigating alerts, test the PromQL query:

```bash
# Get alert query from Grafana
kubectl exec -n observability deployment/grafana -- \
  curl -s 'http://localhost:3000/api/v1/provisioning/alert-rules' \
  -u admin:admin123 | python3 -c "
import sys, json
rules = json.load(sys.stdin)
alert_uid = 'prometheus-down'  # Change this
rule = next((r for r in rules if r.get('uid') == alert_uid), None)
if rule:
    query = rule['data'][0]['model']['expr']
    print(f'Query: {query}')
"

# Test the query
kubectl exec -n observability deployment/grafana -- \
  curl -s -G 'http://prometheus.observability.svc:9090/api/v1/query' \
  --data-urlencode "query=<QUERY_FROM_ABOVE>" | python3 -m json.tool
```

## Metrics Collection Issues

### Check if Metrics Are Being Scraped
```promql
# Check last scrape time
time() - timestamp(up)

# Check scrape duration
scrape_duration_seconds
```

### Verify Metric Exists
```bash
# List all metrics
kubectl exec -n observability deployment/grafana -- \
  curl -s 'http://prometheus.observability.svc:9090/api/v1/label/__name__/values' | python3 -m json.tool

# Search for specific metric
kubectl exec -n observability deployment/grafana -- \
  curl -s 'http://prometheus.observability.svc:9090/api/v1/label/__name__/values' | grep "your_metric"
```

## Related Documentation

- [PromQL Documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/)
- [Grafana Explore](https://grafana.com/docs/grafana/latest/explore/)
- [Alert Testing Guide](../../../docs/04-observability/ALERT_TESTING_GUIDE.md)
- [CLAUDE.md Monitoring](../../../CLAUDE.md#monitoring--access)

## Pro Tips

1. **Use rate() for counters**: `rate(metric[5m])` instead of raw counter values
2. **Aggregate with by/without**: `sum by (namespace) (metric)` to group metrics
3. **Use recording rules**: For frequently used complex queries
4. **Set appropriate time ranges**: Use `[5m]` for rate calculations
5. **Test queries in Explore first**: Before adding to dashboards or alerts

🤖 Generated with [Claude Code](https://claude.com/claude-code)