SSB Distortion when SmartLink Unavailable

Jeff_W4DD

FYI,

  It appears the Flex SmartLink server is still unavailable (Tuesday 8:00AM Eastern). As reported last fall, it appears when the server is unavailable, it causes audio distortion on SSB for Flex radios. Of the 4 Flex users on this morning's Net, the one remote user could not access his radio (SmartLInk unavailable) and the other 3 Flex radio owners (local access) had random SSB distortion. Owners of other radio types could hear the distortion, so it is Tx problem.

When we had this same issue last fall (when SmartLink was unavailable), all 3 Flex radios on the Net had distortion, and I received an email from another group of Flex users confirming they had the same issue within an hour of when we had the issue. 

I suspected that this distortion might be related to the SmartLink server issue, so last fall I disconnected my 6600 from the Internet and went Layer 2 connectivity through my local L2 switch. My audio distortion ceased, while the other two Flex stations on the Net continued to have the audio distortion. The distortion sounds like random clicking or loss of audio. It may occur 2-3 times in one second, or only once during a transmission. Some transmissions have no distortion.

Others may want to try the Layer 2 solution so we can verify it is indeed a problem related to the Flex radio registration process with the SmartLink server.

Jeff, W4DD

Pete La

We have a morning net with 4 Flex users in it, and all of us are experiencing the same thing. Does anyone know why a Smartlink outage will effect our audio?

Pete K1OYQ

Mike-VA3MW

On a SmartLink connection, the audio is managed differently to ensure that it is able to transport via smaller 'pipes'. The audio stream is compressed a bit and it is much smaller.

If you are on a local lan or a VPN to the radio, then the audio stream is much bigger and not compressed at all.

If you are experiencing distortion, it is due to local LAN transport, not the SmartLink server. All the SmartLink server does is broker the initial part of the connection and from there on, it is a Peer to Peer connection.

The clicking is actually UDP packets being dropped. Likely by your switch not being able to handle all the traffic at the same time. If you have a managed switch, there will be a login/maintenance window that will show you this information.

73

Pete La

For those who understand all that gobbly **** about pipes and VPNs and LANs that is a great explaination, but how do we stop it? This doesn't happen once in a while it happens frequently , but inconsistently. None of us are operating remotely, so I don't see the connection. I'm not using a switch that I know of, just a lan cable direct into my router. Is it because of the connection to our PC while using SSDR? Should we only operate from the front panel when this occurs? Should we disconnect from the LAN when this happens. We're looking for a fix Mike, not just an explaination. It also causes the VOX to drop out sometimes which is very annoying. All four of us on our net this morning experienced the same audio difficulties this morning. This is another one of those annoying nuesance type problems with Flex Radios. Is FRS doing anything to address it? It has been going on for a long time.

Pete K1OYQ

Jeff_W4DD

My connection is directly into the rear of the Flex radio. No LAN transport of audio is involved. Others having the issue have the same configuration. Audio goes directly into the back of their radio where it modulates the RF and gets send out over the air.

Two of us just completed an experiment. The other person has audio into the back of his radio also. He unplugged his L2 switch from his router. His Tx audio clicking cleared up and his transmissions were perfect.

There may be other causes of audio clicking, like packet loss on a network (when audio packets are routed on the Network), but when the SmartLink Server is unavailable, the clicking seems to occur for all Flex's having the connection to the Internet (or maybe just the ones enabled for SmartLink). When the SmartLink server returns to service, this particular cause will cease.

Mike-VA3MW

@Pete La

You might have to have someone help you out from a local club. They are a great resource for networking resources. And, you are using the Switch that is in your Router. Those can be of sub standard quality since the routers are made just good enough for surfing. You may want to purchase a different switch such as one from TP-LINK and see if that helps. We often find that switches in metal cases seem to be high quality.

@Jeff_W4DD

The transport is still LAN based even if you plug the cable from the Radio into the back of a computer. Just like Pete's concerns, the onboard NIC cards can be sub standard on your PC. You do want to make sure your drivers are up to date.

If the ticking cleared up. than the Router (which can be a Router, Switch and Wifi Access point all in one) could be the issue. Even cables have to be considered (this is well discussed in this community).

The radio only does LAN connections and uses a network. It is all Layer 3 TCP/IP using both TCP and UDP protocols. For local LAN use, none of the data is shared on the internet. It is all on your local LAN.

It is hard to tell from your description exactly the topology. Even when SmartLink is up, SmartLink has nothing to do with routing audio packets, not even a little bit.

Let us know what you find out (I tried to recreate it, but I was not able to).

Pete La

Or, One could buy a something other than a Flex. I must admit I am getting quite tired of these type of issues. It's always something wrong with my setup and never FRS's fault. Great concept radio, but it leaves a lot to be desired in some areas. If I had more time on this planet (I'm 78 and fighting pancreatic cancer) I would sell the Flex and purchase something less troublesome. I didn't bargan for all this when I purchased the rig in 2018. A lot of hams are weary of FRS and say they don't want a radio that is dependent on the internet. Now I know why.

Pete K1OYQ

Geoff AB6BT

Unless I'm missing something, the radio is not dependent on the Internet, only the LAN...

Pete La

Who knows? I'm upset that we have to put up with these audio problems (popping, cracking, and VOX dropouts) from our Flex radios because Smartlink goes on the blink. Seven Flex owners have experienced this today. To me that relates the issue to the internet not just the LAN. This is a Flex problem and they are not going to put it off on the owners. I think Mike is out of line suggesting we buy new, PCs or routers or switches because our equipment is old. If Flex knows the owners are experiencing these issues, and I feel it is up to them to work it out and improve the system, not tell the owners to go buy new stuff. That one really upset me. Maybe I had higher expectations from FRS.

Pete K1OYQ

ka9ees

I experienced this, this morning. I'm not using Smartlink. 3.5.8 running Win10. How could something external to my LAN effect the performance of my 6600M?

I switched to the front panel. I still have the problem. This is quite frustrating.

I'm sure the software engineers are hard at work solving the problem. Hopefully it will be quickly.

WX7Y

Power off the radio, Turn off the DC and wait 15 seconds and turn it back on and start the radio and see if it goes away.

This may help

K0FLY

First of all Geoff is correct about radios not needing to be connected to the internet. None of my Flex radios are connected to the internet, except to update the license. I do not use Smart Link, I'm not a remote operator.

As far as problems with Smart link being down and the clicking, I'm SPECULATING that there is a sub routine in Smart SDR which is trying to establish contact with the server. Each time Smart SDR knocks on the server's door, it causes some of the behavior being described in this thread.

Assuming that I'm correct , what needs to happen short term is a procedure to disable Smart link feature when the server is down. I noticed that SDR CAT has a check box to enable auto connect to the Smart Link server. Is this feature enabled, if so will this behavior go away if that box is unchecked?

Only Guessing.flames great fully accepted

Gayle K0FLY

Jeff_W4DD

Correct Gayle,

That is the current theory. Something in the radio registration process is causing the clicking when the registration process does not go smoothly. It is only a theory, so as yet unproven, but I confirmed when the SmartLink service was unavailable, the clicking started on multiple radios in different locations. If the radio was disconnected from the Internet, the clicking instantly stopped and SSB audio was clean.

I have a ticket open on the issue and once things settle back to normal, perhaps they can take a look at why the two events (SmartLink Unavailable and SSB Tx audio clicking) are correlated.

Jeff, W4DD

AndyB

Had SSB RX audio clicks last night, all bands. Also received reports of clicking or distorted SSB tx audio. This was between 1900 and 2130 UTC 12 September 2023

Reboot of flex and PC no effect.
NOT using smartlink
HARDWIRED audio (rx and tx) into sockets on back of the flex.

Nothing done to remedy this yet, be interesting to see if it’s changed when I come to use the radio tomorrow night.

If we know the ports/IP address involved in smartlink might be an idea to firewall them and see what happens.

Andy
M5TKA

ka1jy

The audio popping is on all modes, transmit and receive, I tried everything I could think of. Dropped the video to windows 2 quality, turned off the dax app, dropped to one slice, lowered refresh rates, disabled all other sound devices, reduced the SmartSdr window to 2"x2". closed any background task I could find. Still popping. My I7-8700K, 32gig ram, M2 drives, 500M up and down ISP, gig-lan, CPU min duty 100%, win10 pro, Nvidea 1660S, fixed IP addresses, etc etc was enough PC to keep up but apparently not anymore. You can pick out the Flex users on FT8 now because their bandwidth kicks up when this happens,. I can spot the flex users on SSB or digital or any mode. This has gone over the top in recent weeks. No amount of rebooting or restarting or reducing PC loading seems to solve this. I will bet my flex this one is brought to us by MS.

AndyB

14 September 1900 to 2130 UTC nothing done to radio or PC. No unusual noise on RX audio, nor any reports of poor TX audio.

k0oks

There needs to be a setting to COMPLETELY disable SmartLink. Many of us want to use the radio locally ONLY, and couldn't care less about SmartLink. The radio should NEVER connect or attempt to connect to Flex or any remote server unless the user MANUALLY requests to do so for an update.

ka1jy

Man, I TOTALLY agree with this comment. It has been a never-ending battle to stop the popping and distortion on the flex. I talk with a diverse group of flex users for a couple of hours every day and not one of them is immune from this issue. Doesn't matter what the flex configuration, maestro, no maestro, "m" model, non-M, doesn't matter. The radios all seem to pop and distort and have audio problems. It comes and goes. It ebbs and flows. We might go an entire 3 hour session without any issues. Other days it hammers every transmission. If smartlink is even a minor contributor to this problem the quickest way to make a lot of users happy is to give us a version that allows us to turn that connection off. Seems like a no-brainer. I know Flex's bread and butter is the remote crowd. I admit I bought the Flex with the intent of putting it in a remote location and eventually I will but until then I would be extremely happy to get ANY improvement on the distortion and audio problems. So put me as a "Yes" vote on the question of a smartlink detached option. Please.

ka1jy

While I am snarling. This comment was interesting. "On a SmartLink connection, the audio is managed differently to ensure that it is able to transport via smaller 'pipes'. The audio stream is compressed a bit and it is much smaller."

Okay - I get that. I wondered how the remote operation would work considering how often the local lan configuration suffers from these audio stream problems. While we are rolling out a version that drops the smartlink connection how about an option to use those skinny pipes on the stand-alone configuration? I took a glance at the audio-setting for the pipes I can see in windows and those settings were all 48K 24 bit sample rates and I didn't see any compression options. Obvious question is why? For voice? Even for low signal rate digital like FT8 it seems like we could squeeze up the bandwidth on these audio pipes by a significant amount. Would that help the popping and buffer dropouts? Someone much smarter than me might answer that question but if the answer is even "maybe" I would be willing to give it a try.

Jeff_W4DD

9/28/23 7:55AM Eastern:

I started the Flex 6600 (v3.5.9) this morning to join the net that starts at 8:00AM (local Flex access). I recently started running Wireshark to monitor the radio to SmartLink activity. I noticed the radio to SmartLink packet exchange was not the normal 2 or 3 packet exchange but the abnormal group of 20-25 packets (to/from the SmartLink 23.x.x.x address). This indicates to me the radio is having trouble registering with the SmartLink server. (I have a SmartLink account, but my radio is "unregistered")

I joined the net and one of the other members said he received a text message from a member that, despite many attempts, remote Flex access to his home radio was not working so he could not join (different ISP's than I have). Other Flex users (all local to their radio) joined the net without issues. At 8:15AM I noticed a flurry of packet transmissions from my radio to the SmartLink server. The packet exchange between my Flex radio and the SmartLink server then resumed the normal 2-3 packet exchange. Within 30 seconds of the normal exchange, I texted the other member and asked that he try connecting again. He connected instantly and all was normal.

The experience this morning confirms there is an association between the radio not being able to register with SmartLink and the ability of users running remote to access their radios. Because of the large number of static crashes today from storms in the distance, I could not test the association with RX popping.

For those wishing to monitor their radio activity with the SmartLink server, your "tap" device must be between the radio and your router in order to see the radio/SmartLink packet flow. The Wireshark web site has lots of information on how to do this (or drop me a direct email). It is simplest to use a separate PC/laptop running Wireshark, not the PC used to run SmartSDR.

An option to not have the radio register will be a step in the right direction. That would eliminate many attempts on the SmartLink server. I am looking forward to that feature.

Jeff_W4DD

For those following the email threads on SSB Tx clicking and Rx popping (and likely FT8/CW issues), Eric from Flex posted an excellent Root Cause Analysis on the recent SmartLink outages. I appreciate Flex's diligence in "connecting the dots" and look forward to seeing the the performance improvements in a future release. Thanks to other users who sent me emails confirming what I was seeing on Tx and Rx and on the association with registration packets from the radio to the SmartLink server. Special Thanks to the Flex Engineering team for their hard work.

Excerpt from Eric's RCA:

While carefully analyzing code paths and network activity during the recent SmartLink outage, it was apparent that even radios that were not registered for SmartLink were making an initial connection to the SmartLink server. This was not intentional and is tracked as SMART-9754.

When the radio failed to connect to the SmartLink server (in this case, as a result of an expired certificate), this resulted in subsequent attempts every 5 seconds. This is a non-standard design where a backoff is more typical (try again in 5 seconds, then ~20, then ~120, etc). This would help keep the traffic of the connection process (which involves crypto that can be expensive computationally at scale) from becoming a burden to the server if a systemic connection issue is encountered. This is tracked as SMART-9744.

As noted by several users in our community, it was also found that the thread priority of the SmartLink connection thread in the radio was such that these failed attempts ended up pre-empting other core functions of the radio (audio being the most obvious). The slow response of the server on top of the failed connection compounded this issue (high priority thread waiting on a slow server response). This is a poor design and is tracked as SMART-9755.