Optimizing Dapr HTTP metrics with path matching

Introduction

Contributing to open-source software is both exciting and challenging. Recently, I had the opportunity to make my first contribution to the Dapr project—a popular open-source runtime that simplifies microservices development. In this article, I’ll walk you through the new 'HTTP Metrics path matching' feature introduced in Dapr 1.14 and the process of proposing a new Dapr feature.

These new features are brought to life in Conductor, a powerful platform from Diagrid that automates Dapr operations, enforces security best practices, and optimizes resource usage. Conductor not only streamlines the management of Daprized applications but also provides real-time insights and collaboration tools, enabling you to monitor multiple clusters with ease, improve uptime, and cut infrastructure costs significantly. It’s designed to elevate your Dapr experience by putting your operations on autopilot, ensuring security and reliability, and giving you the clarity you need to scale your microservices effectively.

This article assumes you have some familiarity with Dapr. If you're new to Dapr or need a refresher, you might want to explore the official documentation first. That way, you'll have the context needed to follow along with the details discussed here.

My contribution involved implementing the path matching API, a feature that enhances observability, lowers resource consumption, and improves security by reducing cardinality in metrics. Path matching balances maintaining relevant metric information and minimizing resource usage. For those unfamiliar, cardinality refers to the uniqueness of data points in observability systems. High cardinality can lead to performance bottlenecks, increased costs, and challenges in managing and analyzing metrics.

By the end of this article, you’ll have a better understanding of Dapr, cardinality, the Dapr contribution proposal process, and how Conductor can make you more effective in managing your Dapr workloads. I hope my experience provides insights and inspiration for you to start contributing to Dapr and other open-source projects.

Understanding cardinality

In observability, cardinality refers to the uniqueness and variety of data points within a collection. When it comes to metrics, high cardinality means having numerous unique values for a given label. For example, consider the following high cardinality scenario in Dapr:

‍

Here, each request to the order-service is logged with a unique order ID in the path, resulting in high cardinality.

In contrast, low cardinality groups these requests under a single pattern, reducing the number of unique data points:

‍

This approach reduces your observability systems' complexity and resource demands while still capturing the necessary metrics.

High cardinality can impact your observability systems in several ways:

• Performance Issues: Managing and querying a large volume of unique data points can slow down monitoring systems, causing delays and inefficiencies.
• Increased Costs: The resources needed to store and process high-cardinality data can drive up infrastructure and storage costs.
• Data Management Challenges: Handling a wide range of unique data points can complicate data analysis, making it harder to extract actionable insights and maintain clean, useful data.
• ‍Security Risks: Unconstrained cardinality can expose your system to security vulnerabilities, such as potential denial-of-service (DDOS) attacks, where an overload of unique data points can overwhelm and disrupt your systems.

In essence, high cardinality can undermine the effectiveness of your monitoring tools, turning straightforward tasks into resource-intensive and complex processes. Addressing high cardinality is crucial for maintaining an efficient and secure observability setup—it’s a ticking time bomb if left unchecked!

Introducing path matching

In Dapr 1.14, we introduced a new metrics configuration called "path matching". This setting is designed to tackle the challenges of high cardinality in observability metrics by allowing you to specify which paths are most important for your monitoring needs.

Key Benefits:

• Group What Matters: By using patterns to group similar requests (e.g., /users/{userID}), the API reduces metrics cardinality.
• Enhanced Efficiency: The API lowers memory and storage usage, improves performance, enhances security, and simplifies metric analysis.
• ‍Scalable Performance: Leveraging Go’s standard library routing, path matching ensures efficient and scalable path matching.

Since path matching occurs within Dapr’s request handler, the implementation utilizes Go’s robust standard library router. This design choice ensures that the path matching API not only streamlines your metrics configuration but also delivers high performance even under heavy loads, making it a reliable solution for high-throughput environments.

How does Dapr manage high cardinality until now?

Previously, Dapr’s Rules API allowed for defining regex patterns to manage cardinality. However, regex patterns can be difficult to maintain and often CPU-intensive, as each metric requires its own regex. Path matching simplifies this process by letting you define important paths only once and applying the rule to all relevant metrics. For optimal results, use Path matching with the increasedCardinality flag set to false.

In the configuration shown below, regex patterns are used to match HTTP paths for various metrics. While functional, this approach can become cumbersome and CPU-intensive, as each metric must have its own regex pattern:

Simplifying the configuration with path matching

With path matching, you can simplify this setup by defining a single path pattern for relevant metrics. This approach reduces complexity and enhances performance by grouping similar paths together:

This is an example of one of the path matching benefits: you only need to specify paths once.

Let’s see path matching in action

Conductor is a powerful tool from Diagrid designed to simplify the management and observability of Dapr applications. It offers an intuitive interface for monitoring and controlling your Dapr workloads, making it easier to visualize and analyze metrics.

You can sign up for Conductor Free and explore its features yourself. Start your trial and see how Conductor can make managing your Dapr applications easier and more efficient!

To demonstrate the benefits of path matching, I deployed two Daprized applications in separate Kubernetes clusters. Both applications consist of a UserService that interacts with an OrderService through three endpoints: /orders, /baskets, and /payments using Service Invocation. Each call between services generates a unique ID, leading to high cardinality.

High cardinality cluster

In the first cluster, I configured Dapr with default high cardinality settings.

Here’s what we can observe using Conductor.

1. ‍Increased Memory Usage: The system showed significantly higher memory consumption due to the large number of unique metric data points generated by individual IDs. The sheer volume of data made it challenging to manage and monitor effectively.

2. Monitoring Challenges: The dashboard was cluttered with numerous data points, making it difficult to extract meaningful insights. For instance, each unique basket ID was displayed as a separate path in the graph, significantly increasing the number of data points and creating a lot of "noise." This high cardinality obscured the visibility into the actual performance and behavior of the services, making it harder to monitor and analyze effectively.

Path matching cluster

The second cluster configuration uses path matching for the orders and payments endpoints, while the baskets endpoint is ignored since it’s not a priority from a business perspective. This is another benefit of path matching: it allows you to focus on what’s important, avoiding unnecessary resource usage.

The first noticeable difference is the significant reduction in resource usage: in the high cardinality cluster, the UserService consumed 180MB and the OrderService 100MB. In contrast, the path matching cluster reduced these to 110MB (a 39% reduction) and 57MB (a 43% reduction), respectively, demonstrating the efficiency gains from reducing unnecessary metric data.

The second key difference lies in observability: with path matching, the reduced cardinality results in much cleaner and more understandable graphs. This reduction in noise allows for a clearer view of the system’s performance, making it easier to spot trends and identify issues.

Comparing these two clusters and the data displayed in Conductor, it’s clear that path matching not only simplifies metric configuration but also enhances observability by reducing cardinality and significantly lowering resource usage. The visual clarity and performance improvements observed in Conductor, along with the reduced memory consumption, underscore the practical benefits of adopting path matching for managing high-throughput applications.

The process of contributing to Dapr

The process of contributing to Dapr involves several steps to ensure that new features or changes align with the project's goals and address significant problems while maintaining high quality. Here’s a simplified overview of how it works:

Submit a proposal

• Open a PR with your proposal in the Dapr proposals repository. This is where the community and maintainers can review and provide feedback on new proposals.
• HTTP path matching proposal: https://github.com/dapr/proposals/pull/58

Proposal review

• The Dapr community and project maintainers will review your proposal. Be prepared to discuss any feedback and make necessary revisions to it.

Proposal implementation

• Once your proposal is approved, implement the changes according to the approved design. Open a PR in the Dapr repository. Your PR will be reviewed by Dapr maintainers and other contributors. They will check for code quality, adherence to project guidelines, and overall impact.
• HTTP path matching implementation: https://github.com/dapr/dapr/pull/7746

Documentation

• Update any relevant documentation to reflect the new API or feature.
• HTTP path matching docs: https://github.com/dapr/docs/pull/4155

Contributing to Dapr allows you to solve real-world challenges and immerse yourself in the open-source culture. If you're interested in learning more about how we approach open source at Diagrid, I recommend checking out this blog post on Diagrid's open-source culture. It provides deeper insights into our values, community involvement, and commitment to fostering a collaborative environment.

These are the steps you need to follow to contribute to Dapr and help improve the project for everyone!

Conclusion

The introduction of path matching in Dapr 1.14 marks a significant step forward in managing observability metrics. By enabling you to define key paths once and apply them across all metrics, Path Matching simplifies configuration, reduces cardinality, and enhances overall system performance and security.

As demonstrated, this API not only lowers memory and storage usage but also improves performance and data management, making it a powerful tool for high-throughput environments. For more detailed information on how to implement and configure path matching, I encourage you to check out Dapr metrics documentation.

Contributing to open-source projects like Dapr can be a rewarding experience, offering opportunities to solve real-world challenges and improve essential tools for the community. I hope this article provides you with a clear understanding of path matching and inspires you to explore Dapr further or consider making your own contributions to open-source projects.

Conductor can assist with HTTP metrics and beyond. It helps detect optimization opportunities, saving resources, and shows usage while raising alerts when thresholds are exceeded. Try Conductor to optimize your applications with ease.

Thank you for reading, and feel free to reach out with any questions or feedback!