SmartLink Outage Update - June 17, 2025

Mike-VA3MW

Thanks @Annaliese - NH6Z for writing this up.

SmartLink Post-mortem for Server Instability

Annaliese C. McDermond, Senior DevOps Engineer

Incident Description

On the morning of June 17, 2025, Software Engineering was alerted to multiple SmartLink outages by our Uptime Robot and CloudWatch monitoring systems.  Upon investigation it was discovered that the SmartLink server software was periodically restarting, which disconnects users and can cause some issues when radios and clients attempt to reconnect.  Further analysis of the logs of these crashes revealed a bug in the SmartLink server software that was causing crashes on certain code paths. 

Software Engineering drafted code changes to the SmartLink server software in an expedited process and redeployed the software to the production SmartLink environment at approximately 18:22 UTC on June 17.  This appears to have solved the issue.

Remedial Actions

• Software Engineering will continue to monitor the service for crashes and evaluate the condition of the service periodically to identify any additional necessary fixes.
• Software Engineering will explore other ways to make the client and radio code in the SmartLink system more resilient to disconnections from the server. (SMART-11376)
• Software Engineering will explore ways to more effectively communicate a radio or client’s SmartLink status to the user. (SMART-11380)