This incident is now considered resolved. Below please find the Root Cause Analysis (RCA) from Early Warning Services.
Again, thank you for your patience and support, as Early Warning Services addressed their issue.
Incident Root Cause Analysis (RCA)
Service(s) Impacted: Multiple Services: • Real-time Deposit Chek® and Payment Chek® Services with Account Owner Authentication (AOA) • AOA Graphical User Interface (GUI) • Asset Search and VerificationSM Service • Identity Chek® Service Alerts • Real-time Identity Chek® Service • Real-time Deposit Chek® Service • Batch Identity Chek® Service • Batch Payment Chek® Service • Shared Directory • Zelle®
Issue: On February 4, 2021 at 1:22 AM CT, network traffic from DCN was interrupted due to equipment failure.
Root Cause: A router failed during a necessary firmware upgrade and could not be recovered or rolled back. This failure required the entire device to be physically replaced. The replacement router failed so a second replacement had to be sourced. To avoid additional potential customer impacts/service interruption, Early Waring scheduled the replacement for the router for the evening of Feb 13, 2021.
Impact: VPN and Direct Connect services through Early Warning North Data Center (DCN). Customers connecting through other means, such as MPLS, were not affected.
Resolution: The failed equipment was configured and replaced on February 13, 2021.
Preventative Measures: Early Warning is working with appropriate vendors to ensure that redundancy and resiliency measures are strengthened.
Posted Mar 05, 2021 - 11:55 EST
Update
AFS is pleased to relay that all Early Warning Services' functionalities, impacted by this incident, are back to processing through Early Warning Services' primary data center.
Last night, as part of a regularly scheduled Production release, the last remaining services, Batch Deposit Chek and Contribution, were pointed back to Early Warning Services' primary data center.
This completes this incident's technical effort. However, the incident will remain open until all root cause analysis information can be received and relayed.
Thank you for your continued patience.
Posted Feb 24, 2021 - 16:48 EST
Update
AFS is happy to report the following... - Real Time Deposit Chek is working as expected, within Early Warning Service's primary data center - The CAT environment is again accessible and working as expected within Early Warning Service's primary data center - The 3rd party information services component status has been set back to operational
Batch Deposit Chek is still operating within Early Warning Service's secondary data center, so AFS will continue to keep the incident open with a status of monitoring.
Thank you for your continued patience as Early Warning Services continues their efforts.
Posted Feb 17, 2021 - 14:59 EST
Update
AFS' effort to point Real Time Deposit Chek processing back to Early Warning Services primary data center completed successfully. Monitoring continues and further status will relayed tomorrow morning (2/17/2021).
Posted Feb 16, 2021 - 21:26 EST
Update
Early Warning Services has made additional progress on their efforts to restore their primary data center and have requested that AFS begin using their primary data center again, for Real Time Deposit Chek. In support of this, AFS will modify its systems at 9 PM eastern, tonight (2/16/2021). No outage is expected.
Batch Deposit Chek is still being provided by Early Warning Services' secondary data center and their CAT environment is still unavailable.
AFS will continue to pass along further updates, as they are available. Thanks so much for your continued patience.
Posted Feb 16, 2021 - 16:53 EST
Update
Early Warning Services is continuing their efforts to restore connectivity to their primary data center. Their impacted production services, utilized by AFS, continue to be supported from their their secondary data center.
The Early Warning Services Customer Acceptance Test (CAT) environment still remains down. The current ETA for its availability is Monday, 2/15/2021.
AFS will continue to pass along updates, as they are available. Thanks so much for your continued patience.
Posted Feb 11, 2021 - 16:59 EST
Monitoring
As Early Warning Services continues to work on their primary data center, AFS has failed over to their secondary data center. Connectivity to their secondary data center is operational at this time. The Early Warning Services Customer Acceptance Test (CAT) environment still remains down and AFS is working on the go ahead to fail over to the CAT secondary environment. As AFS continues to receive updates, communication with AFS' customers will follow.
Posted Feb 05, 2021 - 16:56 EST
Identified
One of AFS' 3rd Party resources, Early Warning Services, had a communication outage, earlier this morning. A work around is in place, but intermittent failures are still occurring. EWS is currently working toward a resolution. AFS will continue to provide updates and will notify you once a resolution is put in place. We understand how troublesome this is for those affected. Thanks so much for your patience.
Posted Feb 04, 2021 - 14:27 EST
This incident affected: 3rd party information services.