No Workflow is an Island

Establishing business processes using workflows emerged in computing decades ago and has served as the cornerstone of business automation. Fast-forward to today, and modern applications have grown incredibly complex. They incorporate various types of computation, rely on event-driven designs, communicate with multiple services, are built to handle failures and are expected to maintain high levels of security.

Amid this complexity, one element remains consistent: the role of business workflows. However, the methods by which developers implement these workflows have diversified, reflecting the variety of choices available.

Let’s dig into developer-friendly code-based workflow engines, which have gained prominence in distributed applications, microservices or cloud native architectures. Our focus here is on the need to integrate these workflow engines with event-driven messaging, synchronous communication, state storage and other developer patterns.

Workflow Orchestration and Automation

In the era of software development, workflow engines (or runtimes) have undergone significant evolution to address the complexities of enterprise workflow orchestration and automation scenarios. They must integrate with a diverse array of systems and services, providing functionality such as conditional branching, parallel execution and handling external system interactions. Furthermore, their capabilities extend to orchestrating microservices and facilitating event-driven architectures.

Workflow engines typically manifest in two distinct flavors: code-based programmatic engines and domain-specific language-based (DSL) engines. While code-based engines cater to developers, DSL-based ones are primarily tailored for business users.

Although DSL engines offer advantages, including visual design capabilities and integrations, they often fall short in terms of developer-centric tooling — lacking features like debugging, SDK integration and the ability to harness test suites, generally giving code-based workflows an advantage. But before we get into a practical application example, let’s explore the concept of bounded contexts.

Bounded Contexts

A bounded context is a concept within the context of domain-driven design (DDD), an approach that focuses on creating a shared understanding of complex business domains between developers and domain experts. A bounded context is one of the building blocks of DDD and manages the complexity of large-scale software systems by dividing them into manageable, isolated business components.

It is common to have a workflow at the center of each bounded context. However, no workflow is an island, since typically it has to communicate with other bounded contexts and, importantly, also within the bounded context.

Bounded contexts have clear and explicit boundaries. These boundaries are established to prevent ambiguity and conflicts when different parts of the software interact. Communication and data exchange between bounded contexts are managed through well-defined interfaces, APIs and messaging.

For example, imagine an e-commerce platform with bounded contexts for “order management”, “inventory management” and “customer engagement.” Each of these contexts has its own domain model, business rules and terminology related to their specific responsibilities. The order management context might define what an order is and how it should be processed, while the inventory management context focuses on how to track and manage product stock.

Applying Bounded Contexts

Let’s work through an example of what this means when building an e-commerce-processing system that includes the bounded contexts mentioned above and shown in the diagram below.

Diving into the order management bounded context, it would contain a workflow that orchestrates activities, taking full advantage of the workflow patterns available such as task chaining, branching fan-out/fan-in and async calls to other services. For instance, the business process for this order management workflow may resemble the following state machine.

Looking at the implementation, the order-processing service needs to communicate with other services in its context, for example a payment service to confirm payments. However, the order-processing service also saves its state and customer data to a database, and to do that it needs credentials from a key store. This is where the workflow requires supporting services and the ability to communicate within and across bounded contexts. The diagram below illustrates an example implementation of the order management bounded context.

Communication across bounded contexts is essential, both synchronously and asynchronously. In this case, synchronous communication is necessary to reserve items in the inventory management system, while messaging is used to inform the customer application about the successful order fulfillment.

To improve the customer experience, it’s important to ensure that no message is sent without ensuring the order was placed successfully. This is where the outbox pattern is useful between the order management and customer engagement contexts.

Developer Challenges with Distributed Apps

This design approach enables separation of concerns among different developer teams and the flexibility to independently scale and deploy bounded contexts. To fully leverage the benefits of distributed applications and microservice architectures, a combination of technologies that implement common software patterns is required.

For instance, sagas for workflows, service discovery with request/reply, and publish/subscribe for event-driven messaging. The challenge for developers lies in understanding which patterns to employ and then integrating numerous libraries, runtimes and SDKs.

Typically, most backend developers end up being constrained to a specific language and often a particular programming framework within that language. For example, Spring, which is used by 80% of backend developers, dominates the Java ecosystem. Node.js (42%) along with Express (20%) and Nest.js (5%) cover the majority of JavaScript development. C# has all its .NET flavors and Python has too many to count.

As a developer, you then either depend on the framework providing the patterns you need and, if it does not, then you either have to integrate a library that works with your framework or write the code yourself. All the while you are not writing business code.

As we saw in our example, distributed applications need to combine patterns to create more complex business scenarios. This includes using workflows that discover and call other services, sending pub/sub messages for event handling and maintaining data isolation.

Additionally, you need to incorporate cross-cutting concerns such as reliability patterns for handling timeouts, retries and circuit-breakers to enhance your application’s resilience against network and process failures. Security patterns include identity to establish trust, authentication and message encryption. Last, observability patterns for end-to-end tracing and reporting usage metrics are indispensable for monitoring and optimizing your application’s performance.

Now Implement It

Returning to our application example, let’s explore a few technologies that we can combine to achieve the benefits of a distributed application architecture. It’s worth noting that compiling an exhaustive list of these technologies is an article in itself.

For code-based workflow engines, choices include Apache Airflow, Cadence or Temporal, and some DSL based workflows like Netflix Conductor are developer-friendly when using system tasks for communication. When it comes to service discovery and synchronous communication, popular choices include service meshes operating at the network layer, NATs or Consul along with gRPC and HTTP/REST. For messaging, you can consider Apache Kafka, RabbitMQ, Redis Streams or public cloud pub/sub services.

When layering resiliency across all communication, you have tools like Hystrix, Resilience4j and Polly if you work in Java or .NET, however with languages such as Python, you are mostly on your own. With security and identity there are a multitude of technologies for secure communication, along with authentication and authorization, including the use of API gateways and service meshes. The choice here depends on your specific use case, technology stack and security requirements.

Finally, for observability, you can instrument your code with tools like OpenTelemetry, Log4j, Fluentd or select from various observability tool client libraries such as Jaeger, Prometheus or Zipkin.

Just looking at this limited set of technology examples, developers are faced with the daunting task of building a platform of technologies in the languages of their choice. On top of this, you need to add deployment, policy management and the overall application platform maintenance that is required by the operations team.

The overarching question emerges: Can there be a more comprehensive set of patterns and APIs that combine and operate together to bring together developers from across all languages and frameworks instead of the fragmented landscape that exists today? How do you stop reinventing patterns?

Dapr — Unified APIs and Patterns for Distributed Application Development

The Dapr project has introduced a set of unified APIs that enable any developer working with any framework to build distributed, microservice applications using HTTP or gRPC calls. The APIs encapsulate common software patterns, and although each API can be used independently, their strength lies in their ability to be combined. At the same time, required cross-cutting concerns are applied consistently. Just as Kubernetes and containers have transformed the compute landscape by providing a unified set of platform APIs, Dapr delivers unified APIs for distributed application development.

The diagram below illustrates how you can invoke the Dapr APIs from your preferred language framework, and if necessary, combine Dapr with any language-specific APIs that cater to your application requirements. Furthermore, Dapr APIs can be connected to a broad range of infrastructure services through its component model, creating portable code that is independent of the underlying cloud platform.

Dapr’s code-based workflow engine and its patterns is designed to work with the Dapr unified service invocation, pub/sub, secrets, key/value and binding APIs. And there are other Dapr APIs not covered in the example here, including external configuration, cryptography, actors and locking. Each of these APIs aligns with distributed system patterns.

Applying this to our order management application, the diagram below shows how the unified Dapr APIs can be used within and across the bounded contexts.

In the evolving software landscape, where complexity has become the norm, workflow engines remain essential for many business applications. As we have seen in our example, workflows need to communicate with, and be supported by, other services. With Dapr, developers have access to a unified set of APIs and patterns, including workflow, that span language barriers and frameworks and frees them from the challenges of fragmented technologies.

Want to know more about Dapr or have questions about the project? Join the Dapr Discord server and engage with thousands of Dapr developers.