September 11 - 2023 - SmartLink Outage

Flexers,

In the 7AM CDT hour this morning, the major remaining issue causing SmartLink disruption was addressed. This boiled down to some problems in the SmartLink server code dealing with the additional scaling we had enabled to help address the load on the server. We have had a number of confirmations from the field that the system has been stable since our latest server side change.

We will continue to monitor things and let you know if there are additional developments, but I believe the immediate issue's root cause and several complicating factors have all been addressed. We have several key lessons learned and will be working hard to ensure that we don't face another outage like this moving forward.

I am both incredibly sorry for the frustration that our Flexers have experienced during this outage -AND- incredibly proud of our teams ability to stand in the face of trials and endeavor until things are made right. As several have pointed out, there are areas of improvement and defects that were discovered along the way. What makes it all worthwhile during the hard times are the fantastic people we get to work alongside and the customers that encourage us even while experiencing frustration.

Onward and upward.

Timeline

2023-09-11 0000Z: SSL.com Root CA certificate expired

2023-09-11 1700Z: Helpdesk and internal reports of issues accessing radios through SmartLink

2023-09-12 0800Z: Expired certificate discovered

2023-09-12 0900Z: smartlink.flexradio.com certificate (and supporting CA chain) updated on SmartLink server

2023-09-13 1930Z: SmartSDR software (v3.5.9 and v2.10.1) was released to add the new SSL.com Root CA to the FLEX-6000 radio firmware

2023-09-15 1830Z: Server scaling helps with load, but introduces new server-side issues

2023-09-17 1300Z: Diminishing returns on existing server optimization led to life support and monitoring while pouring more resources into New SmartLink Server project

New SmartLink Server

2023-09-14: Updated infrastructure design began

2023-09-22: Internal pre-Alpha test

2023-09-25: Release to Alpha team

2023-10-03 2000Z: Public rollout

Root Cause

The SmartLink server became overwhelmed with unsuccessful connection attempts when the SSL.com Root CA certificate on which our smartlink.flexradio.com certificate depended expired.  This led to server instability and timeout errors that meant only a small fraction of users were being serviced, and those that were serviced had extremely long, unacceptable latency.

Lessons Learned

While it is not typical, Root CA expirations in SSL certificate chains should be checked in addition to the expiration date of the certificate itself.  Replacing these certificates well ahead of the expiration (30 days) will ensure service outages due to certificate expirations do not occur.

Keeping the list of Root CA certificates up to date in the radio firmware can be automated to ensure that certificate updates can be performed on the server side without requiring a corresponding software update.  This has been implemented (SMART-9725) and all future SmartSDR builds will include a best known Root CA list.

The SmartLink server that had served us well since launch became overwhelmed. This was a combination of a relatively under-resourced server dealing with many thousands of radios and clients all attempting the most expensive part of the connection lifecycle (initial authentication with crypto validation). While the server had performed well in ideal circumstances, it struggled under the load in this less typical scenario.  Having enough performance margin (or scalability) to overcome unusual circumstances is something to be considered when designing enterprise level systems.

Corrective Actions

Our new SmartLink infrastructure includes automatic certificate renewal and installation.  Any changes such as expired Root CAs will generate certificate updates that will automatically be integrated into the server moving forward.

We also integrated a process in our build system to pull the latest Root CA list into our radio firmware on every build moving forward.  This will mean that when server side certificates do have to change, they should work with recent software without having to release an emergency update.

We have separated the CPU intensive crypto functions to our cloud provider’s load balancer.  This will reduce the load on the application server and provide better application throughput increasing responsiveness.  We can also scale the application containers CPU and memory as necessary.  We significantly rewrote the application server software to be more efficient and increase its capacity for concurrent connections.

While carefully analyzing code paths and network activity during the recent SmartLink outage, it was apparent that even radios that were not registered for SmartLink were making an initial connection to the SmartLink server. This was not intentional and is tracked as SMART-9754.

When the radio failed to connect to the SmartLink server (in this case, as a result of an expired certificate), this resulted in subsequent attempts every 5 seconds. This is a non-standard design where a backoff is more typical (try again in 5 seconds, then ~20, then ~120, etc). This would help keep the traffic of the connection process (which involves crypto that can be expensive computationally at scale) from becoming a burden to the server if a systemic connection issue is encountered. This is tracked as SMART-9744.

As noted by several users in our community, it was also found that the thread priority of the SmartLink connection thread in the radio was such that these failed attempts ended up pre-empting other core functions of the radio (audio being the most obvious). The slow response of the server on top of the failed connection compounded this issue (high priority thread waiting on a slow server response).  This is a poor design and is tracked as SMART-9755.

We have a similar non-standard design observed in FlexLib, (used in SmartSDR for Windows, SmartSDR for Maestro, and SmartSDR for M models).  In this case, we have backoff logic when a connection doesn’t work out, but the design pattern falls apart with certain connection failures such as certificate authentication exceptions.  This results in a disconnect call within a tight loop that initiates another connection immediately.  The result is a barrage of connection attempts from these clients.  This bug is tracked as SMART-9772.