Circuit breaker protection
(active-standby mode)
If you configure the Fanout service in active-standby delivery mode, Pismo provides an additional layer of resilience through an automatic circuit breaker.
The circuit breaker helps prevent repeated delivery failures when a destination becomes unavailable. During an outage, the circuit breaker temporarily pauses normal delivery attempts, monitors endpoint health and automatically resumes event delivery when the destination recovers.
Figure 4. Circuit breaker auto-recovery process flow
How the circuit breaker works
Under normal conditions, events are delivered to your configured destination and each delivery attempt is recorded in the Delivery Audit Log.
When event delivery fails, the affected events are moved to a dedicated tenant-specific error queue and are retried according to the configured retry policy.
If repeated retry attempts continue to fail, the circuit breaker opens. While the circuit remains open:
New events are held in the error queue. Additional delivery attempts are paused. Health checks are periodically sent to the destination to verify availability.
When a health check succeeds, the circuit breaker automatically closes and event delivery resumes.
This recovery process does not require manual intervention.
Circuit breaker lifecycle
| State | Description |
|---|---|
| Closed | Events are delivered normally. |
| Failure detected | Failed events are moved to the error queue and retry processing begins. |
| Open | Delivery attempts are paused while endpoint recovery is monitored. |
| Health check | The platform sends lightweight requests to verify endpoint availability. |
| Recovery | A successful health check closes the circuit and event delivery resumes. |
Health check probes that originate from the error queue are never re-queued to the error queue. This prevents infinite retry loops.
What to expect during an outage
When the circuit breaker opens:
- Events are retained in the error queue rather than discarded.
- The platform continues monitoring destination availability.
- Event delivery resumes automatically after the destination becomes available.
- No manual reset or recovery action is required.
Because the Fanout service uses an at-least-once delivery model, events that were delayed during the outage are delivered after recovery is detected.
Updated about 13 hours ago