Advanced monitoring
One-day workshop on monitoring as code, telemetry data analysis, troubleshooting with runbooks and on-call, and instrumentation of metrics, logs and traces.
Introduction
Start here - what the day covers and what to prepare beforehand
Workshop introduction
What the workshop covers, how the day is structured, which repositories and tools to have ready before it starts, and how the lessons work.
Monitoring as code
Mixins, observability libraries and Alloy configurations as versioned code
Monitoring mixins and observability libraries
How monitoring mixins and reusable observability libraries package dashboards, rules and alerts as versioned, testable code.
Building a mixin and Alloy configuration
Hands-on walkthrough of authoring a monitoring mixin and the matching Grafana Alloy collection pipeline for one component - the course's online store.
Testing and deploying configuration artifacts
Lint, test and deploy rendered dashboards, rules and collector configurations through a pipeline instead of by hand.
Generating configurations with AI
Use LLM assistants and coding agents to draft mixins, alert rules, dashboards and Alloy pipelines while keeping the result trustworthy.
Telemetry data analysis
Anomalies, RED and USE, cost and capacity
Anomaly detection in time series
General methods for spotting anomalies in monitoring time series, from thresholds and rolling statistics to seasonal and forecast-based detection.
RED and USE analysis
Apply the RED (rate, errors, duration) and USE (utilization, saturation, errors) methods to analyse services and resources.
Cost management from collection to processing
Understand and control the cost of telemetry — what is collected, how much is stored and what each query and dashboard costs.
Capacity management from collection to processing
Plan capacity of workloads and of the monitoring pipeline itself using utilization, saturation and growth trends.
Troubleshooting
Runbooks and automated remediation
Monitoring runbooks
What an alert runbook must contain so the on-call engineer can act without guessing, and how to keep runbooks next to the alerts.
AI agents for automatic remediation
Use an LLM agent to triage alerts, follow the runbook and apply safe automatic fixes with guardrails and human approval.
Monitoring instrumentation
Metrics, logs, traces and Alloy
Metrics instrumentation with prometheus_client
Instrument application code with the prometheus_client library — metric types, labels, exposition and exporter patterns.
Log instrumentation with native libraries
Emit structured, correlated logs from applications with standard logging libraries and ship them to Loki with Alloy.
Tracing instrumentation with native libraries
Add distributed tracing to services with OpenTelemetry SDKs and auto-instrumentation and ship spans through Alloy to Tempo.
Advanced Grafana Alloy configuration
Structure large Alloy deployments with modules, remote configuration, clustering and per-environment scenarios.