OpenTelemetry Collector with Failover Connector, Sending_Queue and File_Storage

The failover connector is an internal component that acts as an exporter for the ingress pipeline and a receiver for the downstream pipelines. It maintains a prioritized list of destinations, routing telemetry to the first healthy pipeline. It periodically retries higher-priority pipelines to return healthy pipelines when they become healthy again.

• The connector does not buffer or persist data — it only routes based on health.
• Buffering and persistence are provided by the exporters (sending_queue and file_storage) in each pipeline.
• Both pipelines send to the same aggregator endpoint.
• After recovery, the connector routes new traffic to Primary while Failover drains backlog.

• Connector routes new incoming telemetry to the Primary Pipeline (highest priority, healthy).
• Primary exporter sends data directly to the Otel Aggregator.
• Failover pipeline is idle but ready.

Result: Low latency, normal flow.

• Primary exporter's sending_queue starts buffering in memory. When it reaches limits or errors surface, the pipeline returns failure to the connector.
• Connector marks Primary as unhealthy and switches traffic to the Failover pipeline.
• Failover exporter's sending_queue + file_storage buffer and persist telemetry safely for the duration of the outage.

Result: No data loss (within capacity). Live traffic goes to failover; data is durable on disk.

• Connector retries Primary on retry_interval (e.g., 30s).
• When Primary is healthy again, connector routes new incoming telemetry back to Primary.
• Failover pipeline continues sending its buffered backlog (from memory and disk) until fully drained.
• Both pipelines may send simultaneously for a period:
• – Primary sends fresh/new telemetry
• – Failover sends older/buffered telemetry

Result: System returns to normal while ensuring no data loss during the outage window.