How should I test how each platform handles non-deterministic LLM calls during workflow replay in a side-by-side evaluation?

To objectively test replay with LLM calls, employ a mock LLM with a fixed seed to produce repeatable outputs. Catalyst records inputs in history and re-sends them during replay. DBOS requires wrapping nondeterministic calls as stored steps for determinism. Inngest replays steps with recorded arguments. Validate each platform’s behavior when replay diverges from live execution in controlled test scenarios.