How to Monitor GPU Server Performance on Linux

AI workloads demand careful GPU monitoring — VRAM exhaustion causes model crashes, overheating causes throttling and poor utilisation means wasted capacity. This guide covers the full monitoring stack.

Step 1 — nvidia-smi Basics

# Current GPU status
nvidia-smi

# Continuous monitoring (refresh every 1 second)
watch -n 1 nvidia-smi

# Specific metrics
nvidia-smi --query-gpu=name,temperature.gpu,utilization.gpu,utilization.memory,memory.used,memory.free,power.draw --format=csv

Step 2 — Log GPU Metrics

# Log every 5 seconds to file
nvidia-smi dmon -s pucvmet -d 5 -o DT >> /var/log/gpu-metrics.log &

# Format: date time power utilisation memory temperature

Step 3 — Install DCGM Exporter for Prometheus

# NVIDIA Data Center GPU Manager Exporter
docker run -d   --gpus all   --cap-add SYS_ADMIN   -p 9400:9400   --name dcgm-exporter   --restart always   nvcr.io/nvidia/k8s/dcgm-exporter:latest

Step 4 — Add to Prometheus

sudo nano /etc/prometheus/prometheus.yml

scrape_configs:
  - job_name: 'gpu'
    static_configs:
      - targets: ['localhost:9400']

Step 5 — Grafana Dashboard

Grafana → Import → Dashboard ID: 12239 (NVIDIA DCGM Exporter Dashboard)

Step 6 — Alert on GPU Issues

# Grafana alert — GPU temperature > 85°C
Query: DCGM_FI_DEV_GPU_TEMP
Condition: IS ABOVE 85
Alert: "GPU overheating on $(hostname)"

# Alert — VRAM > 90% used
Query: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE * 100
Condition: IS ABOVE 90

Step 7 — Monitor Ollama Specifically

# Check which models are loaded in VRAM
curl http://localhost:11434/api/ps

# Ollama memory usage
curl http://localhost:11434/api/ps | python3 -m json.tool

Conclusion

GPU monitoring prevents costly AI workload failures. Our team configures complete GPU server monitoring for AI infrastructure.