How FICO runs the mission-critical Northstar platform, backed by Diagrid

At a glance

• Company: FICO, analytics and decisioning software, best known for the FICO Score
• Platform: Northstar, FICO's next-generation decisioning platform on Amazon EKS
• Event volume: in fraud detection, 3,000 customer transactions per second can become 30,000 internal events
• Dapr APIs: pub/sub (Pulsar), resiliency policies
• Diagrid: Premium Dapr support (1-hour critical response, 24x7), upstream Pulsar feature development, resiliency design
• Engagement: over 50 issues yearly handled since early 2024, supported across seven Dapr minor releases

FICO runs the event service for Northstar, its next-generation platform, on open source Dapr pub/sub and StreamNative Pulsar, running on Amazon EKS in regions worldwide.

Diagrid backs that open source stack. FICO has Premium Dapr support with a 1-hour, 24x7 response SLA for critical issues. Diagrid builds the Pulsar features FICO needs into upstream Dapr, and its engineers designed much of the layered resiliency model that keeps FICO's messages from being dropped.

The challenge

FICO is breaking its long-standing products, such as fraud and portfolio optimization, into smaller building blocks that customers and partners can combine in new ways. That decomposition moves work onto the network: every business capability now depends on events passing reliably between components. Northstar's event service carries those events, including metering events that FICO uses to bill its customers, so a dropped message can mean lost revenue.

Event volumes are large and change fast. In fraud detection, which Hugo Smitter, FICO's Principal Platform Architect, compares to an arms race, 3,000 customer transactions per second can become 30,000 internal events as FICO captures features across more dimensions. Every new fraud model adds variables and more events. On Kafka, where storage and brokers scale together, absorbing a new use case meant reshuffling clusters.

FICO set four requirements for the new event service:

• Reliable pub/sub. A pub/sub API over gRPC and REST, with end-to-end delivery guarantees rather than best effort.
• Security. OIDC authentication, pluggable authorization, encryption in motion and at rest, and schema enforcement that cybersecurity controls.
• No lock-in. FICO had chosen StreamNative's private cloud Pulsar after a broker evaluation, but stakeholders didn't want applications tied to Pulsar's API any more than to Kafka's. FICO wanted to switch brokers through configuration, not code, and to give one API to teams writing Java, Python, Go, C++ and Rust.
• Low overhead. Any abstraction layer had to stay light on CPU and memory next to latency-sensitive workloads like fraud detection.

FICO weighed building its own sidecar framework in Java and Spring, but a performant abstraction with operators, upgrades and broker switching was more than the team wanted to own. Dapr met the design.

Open source on a critical path: who do you call?

Metering events feed customer billing, and fraud detection runs in real time, on about 50 Dapr clusters worldwide. Running open source software on that path raises the question every platform team asks: who do you call when it breaks?

FICO calls Diagrid, founded by Dapr's creators and the lead maintainer of the project. Under Premium Dapr support, critical issues get a response within one hour, 24x7. Diagrid engineers also meet FICO's team every two weeks to review issues and feature requests. Since early 2024, Diagrid has handled more than 50 issues per year and supported FICO across seven Dapr minor releases.

When a deadlock hit under load

In April 2026, a Pulsar subscriber on FICO's platform deadlocked under load. At about 18 messages a second, Dapr stopped processing events after 21 failed delivery attempts, and pods had to be restarted by hand.

FICO raised the issue on April 13. By the next day, Diagrid engineers had traced it to two bugs in Dapr 1.16: the sidecar ignored the app's maximum concurrency and its synchronous processing mode, so a single-threaded handler was flooded with parallel deliveries and retries.

Diagrid gave FICO a two-step plan. First, raise the resiliency policy's retry count from 20 to 40 to stabilize the environment. Then upgrade to Dapr 1.17, which already contained the fixes. Diagrid also started a backport to 1.16 in case the upgrade had to wait. FICO upgraded to 1.17, and the fix held.

Upstream feature development

When FICO adopted Dapr, the Pulsar component supported only shared subscriptions and Pulsar's native token authentication. Within six months, Diagrid added:

• Schema enforcement for JSON, Avro and Protobuf
• OIDC authentication
• End-to-end encryption of message payloads
• Support for more Pulsar subscription types

More recent work includes Pulsar listener name support in Dapr 1.18, which lets FICO's sidecars connect to brokers directly instead of through a proxy. Support for Apache Pulsar 5.0 scalable topics is in progress. Diagrid builds every one of these features into open source Dapr ensuring FICO runs no private fork, takes fixes and features through the normal Dapr release train, and isn't locked into Diagrid.

Event service architecture

Each Northstar service runs with a Dapr sidecar on Amazon EKS. Applications publish and subscribe through the Dapr pub/sub API, and the sidecar holds the Pulsar client, authentication, schema enforcement and encryption. Pulsar runs as StreamNative's private cloud offering in multiple regions, so the brokers stay under FICO's control.

Because the broker sits behind a Dapr component, moving a topic between Pulsar and Kafka is a configuration change.

No dropped messages: layered resiliency designed with Diagrid

FICO can't afford to lose events, especially metering events that feed customer billing. Diagrid engineers designed much of FICO's answer, including the layered retry model, the ack-timeout guidance and the batching guidance, and worked through it with FICO and StreamNative in a joint session. FICO wrote the result into an internal resiliency standard for every team on Northstar.

Pulsar handles short, transient failures inside the client, and Dapr resiliency policies take over for anything longer. Each failure has a defined owner:

Two settings make the layers work together. FICO turns off Pulsar's consumer ack timeout, so redelivery is driven only by Dapr's NACKs and isn't duplicated by a second retry loop. And because Dapr retries stack on top of Pulsar's, the team sizes its retry counts and intervals for the total time a message can be in flight, with exponential backoff for slow brokers.

FICO enforces the standard with Kyverno policies on its Dapr pub/sub components, so every team on the platform runs the same pub/sub configuration.

Next: Northstar launch and client onboarding

FICO has launched Northstar and is onboarding customers onto it. Each new customer adds decisioning traffic and metering events to the same Dapr and Pulsar event service, and the resiliency standard applies to every service that goes live.

Results

Additional Resources

• FICO's event service with StreamNative Pulsar and Dapr, Hugo Smitter
• Harnessing Crossplane and Dapr for DevOps: FICO's Platform Engineering Journey, Hugo Smitter, Diagrid, November 2024