SmartSDR v3.8.20 and the SmartSDR v3.8.20 Release Notes
SmartSDR v2.12.1 and the SmartSDR v2.12.1 Release Notes
Power Genius XL Utility v3.8.9 and the Power Genius XL Release Notes v3.8.9
Tuner Genius XL Utility v1.2.11 and the Tuner Genius XL Release Notes v1.2.11
Antenna Genius Utility v4.1.8
Need technical support from FlexRadio? It's as simple as Creating a HelpDesk ticket.
Publish Blameless Post-Mortems
With SmartLink service restored, it would be good to put to rest community speculation (some clear-eyed, some wild-eyed) regarding the root causes and aggravating factors of the outage, and the process changes FlexRadio has put in place to both restore service and reduce severity and frequency of future outages. See for example https://www.etsy.com/codeascraft/blameless-postmortems/
These post-mortems should be:
- blameless (no personnel at-fault / at-risk/ identified publicly),
- specific (i.e. not 'a cert expired' but "The intermediate CA cert responsible for authenticating radio clients and software clients to the SmartLink service expired before it was replaced")
- detailed in
- the problem space (see above)
- the restoration space ("A new intermediate CA was signed by our internal root CA and installed on the SmartLink servers, which restored service.") and
- the preventative space ("To prevent this recurring, we have put in place two independent internal processes -- one which automatically renews the cert 60 days before the old cert's expiration, and a separate one which checks the deployed cert and alerts engineering if the cert will expire within 45 days, indicating a failure of the primary system. We have further added an automated process to generate a 'cert-update-only' point release for all current releases, so that customers can avoid service issues without changing the current functionality of their devices.")
- Comprehensive, in providing a full picture of the incident, the root and aggravating causes, and other related/confounding problems (e.g. addressing in a similar manner the SmartLink backoff timer challenges, implementing some kind of source-IP-based rate limiting at the Smartlink service front end, etc)
Having read a bunch of the Facebook group posts, I'm sure that, to put it concisely, haters gonna hate. However, I hope that doesn't dissuade Flex from speaking to customers who-- having put in the hard work and busted knuckles to learn their respective professions and hobbies, and having made their own mistakes along the way, can respect Flex's transparency and efforts to improve their products and services.
Comments
-
Well laid out and well said.
Constructive problem identification and resolution.
Alan. WA9WUD
0 -
I expect a company like FRS to be more proactive and less reactive. They have been caught with their pants down too many times recently.
Pete K1OYQ
2 -
I wish I could vote for this 100 times.0
-
Yes, absolutely.0
-
This should also apply to folks who post a question with a problem on this forum.
Instead of just saying that the problem is solved, and leaving us all hanging, tell us the solution.
Sorry...one of my pet peeves.
4 -
This is such a great suggestion both in the way that it was posted (constructive) and with the excellent link that explains why this is a good practice (driving the right behaviors). I can't make any promises, but I'll do what I can from my end to make sure that we publish something like this. Thanks for the suggestion.
3 -
-
Thanks for this Eric. It helps to know what happened and what is being done. I appreciate the effort that goes into a report like this (been there...).
1 -
Eric, if I understand correctly from the information in the link you posted, even radios not setup and using SmartLink were pinging the SmartLink servers? Why would that be the case? I run v3.x software but don't use SmartLink. I can understand my radio checking for firmware and software updates on startup ( if an internet connection is available) . But, otherwise, my radio and others not using SmartLink, should have no effect on the SmartLink system.
I have monitored my radio with Wireshark in the past and have only seen the initial update ping and nothing else. Has something changed? I understand about the certificates needed for accessing the SmartLink Authentication servers. But, why would this even be needed for someone not using SmartLink?
As an aside, I think that as some have requested, there should be an explicit way to log off of the SmartLink system and, until the user wants to use SmartLink again, no pings or attempted connection between the radio and the SmartLink server. ( And just the firmware/software update check on startup if there is an internet connection available)
Just my thoughts.
James
WD5GWY
0 -
The intention is that if your radio isn't registered for SmartLink that it wouldn't contact the SmartLink server (unless initiating the registration process). That is the on/off switch as far as our intended design. As stated in the postmortem:
...radios that were not registered for SmartLink were making an initial connection to the SmartLink server.
This is a bug that will be addressed.
2 -
Eric, I think I understand better. One other question. You mentioned registering for SmartLink as the On/Off , method. Does that mean that once a SmartLink user is finished connecting to their remote pc, that the user is no longer logged into SmartLink for Authentication until the next session? And, if they connect locally over their home network, SmartLink is not involved in the process of connecting to the radio over the local network. And there is no attempt by SmartSDR to look for the connection to SmartLink?
Thanks for the information. ( just trying to understand the process)
James
WD5GWY
0 -
There are multiple pieces of the puzzle here. When you "log in", you are simply getting an authentication token. That token is then used when connecting to the SmartLink server. Once you are connected to the radio, the SmartLink server is no longer in the mix with regard to the connection to the radio.
For local connections, the SmartLink server is not involved at all. If you are logged into SmartLink, the radio chooser may show SmartLink connected radios if any are available outside of your local network. But connecting to a local radio doesn't involve SmartLink at all. As a broker, it isn't necessary on a LAN as the radio is broadcasting "here I am" with discovery packets which helps the client know about the available radios.
1 -
Cheers to Flex Team for this Transparency. Very good.
0 -
Thank you Eric for clarifying how the system works. I have had people argue with me that thought the system had to phone home for every single setup, no matter if the user was using SmartLink or not. I always thought that was wrong as I have used Wireshark several times in the past and never could verify any activity beyond the check for new updates on startup. ( outside of my LAN)
James
WD5GWY
0 -
@Eric-KE5DTO -- I want to take a moment to thank you and the team for the linked writeup. I for one found it very informative, and felt significantly more confidence in the fixes with an understanding of what they are. Please excuse the lateness of my reply; it's in no way reflective of my interest in the topic or your reply. :)
Good post-mortems can be hard to write, especially the first one. Kudos to you and the team for both doing the hard work and for taking the suggestion to heart. Nobody wants a service outage, but if I had to take one, this is the kind of thing I want to read after it happens.
Thanks,
--John KD3H
0
Leave a Comment
Categories
- All Categories
- 260 Community Topics
- 2.1K New Ideas
- 538 The Flea Market
- 7.6K Software
- 5.9K SmartSDR for Windows
- 139 SmartSDR for Maestro and M models
- 368 SmartSDR for Mac
- 251 SmartSDR for iOS
- 226 SmartSDR CAT
- 175 DAX
- 345 SmartSDR API
- 8.8K Radios and Accessories
- 6.9K FLEX-6000 Signature Series
- 45 FLEX-8000 Signature Series
- 803 Maestro
- 43 FlexControl
- 838 FLEX Series (Legacy) Radios
- 808 Genius Products
- 424 Power Genius XL Amplifier
- 280 Tuner Genius XL
- 104 Antenna Genius
- 246 Shack Infrastructure
- 168 Networking
- 410 Remote Operation (SmartLink)
- 119 Contesting
- 642 Peripherals & Station Integration
- 116 Amateur Radio Interests
- 880 Third-Party Software