How Grafana Security is using Dapr to improve vulnerability scanning

Grafana open source software empowers users to query, visualize, alert on, and explore metrics, logs, and traces, regardless of their storage location.

Challenge

Grafana Labs needed a continuous scanning solution for container images from diverse sources and repositories, both public and private on GitHub. Rather than relying on costly SaaS vulnerability detection services with proprietary lock-in, Grafana developed an internal light scanning solution to expose vulnerability data within Grafana itself—a familiar tool for all engineers.

Solution

Grafana implemented an event-driven architecture running on AWS managed Kubernetes and Dapr. The solution divides into four principal areas:

Publishers — Pods and cron jobs send events for scans running daily, on-demand from developers, or via webhooks. A Kubernetes cronjob with a Dapr sidecar pulls container configurations and publishes image lists to different pub/sub topics.

Subscribers and Publishers — Scanner pods run container scanner binaries wrapped in Flask applications. Each pod has a Dapr sidecar for consuming events from Redis queues efficiently. Failed events are retried using Dapr resiliency policies and failed scans are moved to a dead letter queue. Dapr guarantees at-least-once delivery, and the Dapr statestore prevents redundant processing by storing successful request IDs as hashes in Redis.

Subscribers — Two pods listen to scan-results topics. One stores results in S3 for compliance and debugging. The other ingests results into a relational database to calculate SLO violations and aggregate issues from multiple scanners.

Visualization and Notifications — A database-connected pod exposes a GraphQL endpoint feeding an internally developed Grafana plugin, allowing developers to view and manage CVEs directly from Grafana. A notifier pod alerts developer teams via Slack or GitHub based on preferences.

Impact

• Reliability: Dapr's resiliency features ensure no scans are missed; failed scans are automatically retried
• Efficiency: Idempotency checks prevent redundant processing, saving computational resources
• Integration: Developers view and address CVEs within existing monitoring dashboards for faster response times
• Flexibility: The architecture enables adding or modifying scanners without disrupting the overall system
• Scalability: The solution seamlessly handles evolving requirements