Help with periodic SmartSDR Network Freezes

Jeremy Gilbert KC1JZE

Every 5-15 minutes or so, my MacOS SmartSDR session (audio and waterfall) appear to freeze and I have to restart the connection. This makes a pretty bad experience. Even if I give it a solid 2-3 minutes, the connection doesn't resume but comes back instantly if I disconnect and reconnect. During these freeze situations, ssh and RDP connections to hosts on the radio's same network exhibit stutter but remain active. This freezing behavior only occurs from one of my operating locations which is served by a broadband connection.

I am using 3.4.23 on the radio, and SmartLink for orchestration. The connection between the endpoint and the radio has variable latency (usually 25ms but sometimes 100-200ms) and some small ongoing (~1%) packet loss. Per suggestions, have checked for buffer bloat on waveform.com (grade B on the client side, grade A on the radio side). Bandwidth between the two locations appears to be about 10 MBit in each direction according to iperf3 with TCP.

Operating from other broadband locations does not trigger the same behavior, so I think I can rule out a problem with the radio, the Mac or the SmartSDR version.

Is there some reason to think the SmartSDR protocol can't recover from momentary packet loss or slightly variable network conditions? How does one debug a problem like this?

Alan

From your description, and the fact that other locations are OK, the problem most likely is contained in your broadband (WAN), and/or your LAN components.

Does your router have an event log? If so, that is a good source of troubleshooting data.

Alan. WA9WUD

Jeremy Gilbert KC1JZE

Thanks Alan.. As a further troubleshooting step, I brought my Maestro over here. It is also exhibiting the same problem. Curiously, during the hang events, the network RTT and is low and the "network status" shows as "excellent" within the Maestro interface. I think this is telling me that the TCP side of the connection is wedged, while the UDP side has been continuing fine (or at least recovers). I wonder if somehow some sporadic bits of packet loss are gumming up the TCP side and it finds itself too much in arrears to fix itself?

I ran a TCP based iperf3 TCP test, this is what I see during the "outage"

j.gilbert@marshmallow ~ % iperf3 -c qth.local -t 10000 -R

pi@qth:~ $ iperf3 -s -V

[ ID] Interval Transfer Bitrate Retr Cwnd

<SNIP>

[ 5] 480.00-481.00 sec 926 KBytes 7.58 Mbits/sec 0 75.1 KBytes
[ 5] 481.00-482.00 sec 1.21 MBytes 10.1 Mbits/sec 2 48.3 KBytes
[ 5] 482.00-483.00 sec 1.21 MBytes 10.1 Mbits/sec 0 75.1 KBytes
[ 5] 483.00-484.00 sec 1.21 MBytes 10.1 Mbits/sec 2 67.1 KBytes
[ 5] 484.00-485.00 sec 1.21 MBytes 10.1 Mbits/sec 1 67.1 KBytes
[ 5] 485.00-486.00 sec 1.21 MBytes 10.1 Mbits/sec 2 40.3 KBytes
[ 5] 486.00-487.00 sec 926 KBytes 7.58 Mbits/sec 1 45.6 KBytes
[ 5] 487.00-488.00 sec 926 KBytes 7.58 Mbits/sec 0 67.1 KBytes
[ 5] 488.00-489.00 sec 1.21 MBytes 10.1 Mbits/sec 1 67.1 KBytes

<MAESTRO HANG OCCURS HERE>

[ 5] 489.00-490.00 sec 926 KBytes 7.58 Mbits/sec 23 59.0 KBytes
[ 5] 490.00-491.00 sec 1.21 MBytes 10.1 Mbits/sec 0 85.9 KBytes
[ 5] 491.00-492.00 sec 1.21 MBytes 10.1 Mbits/sec 3 53.7 KBytes
[ 5] 492.00-493.00 sec 926 KBytes 7.58 Mbits/sec 1 56.4 KBytes
[ 5] 493.00-494.00 sec 1.21 MBytes 10.1 Mbits/sec 0 83.2 KBytes
[ 5] 494.00-495.00 sec 1.21 MBytes 10.1 Mbits/sec 2 51.0 KBytes
[ 5] 495.00-496.00 sec 926 KBytes 7.58 Mbits/sec 0 75.1 KBytes
[ 5] 496.00-497.00 sec 1.21 MBytes 10.1 Mbits/sec 1 72.5 KBytes
[ 5] 497.00-498.00 sec 1.21 MBytes 10.1 Mbits/sec 1 69.8 KBytes
[ 5] 498.00-499.00 sec 1.27 MBytes 10.6 Mbits/sec 2 67.1 KBytes
[ 5] 499.00-500.00 sec 1.21 MBytes 10.1 Mbits/sec 1 67.1 KBytes

<MAESTRO HANG STILL NOT RECOVERED>

Notably, there were 23 TCP retries when the hang occurs, suggesting there was a brief window of packet loss at that exact interval. The TCP testing tool continues on just fine through this, but the Flexradio / Maestro connection does not.

I have just pulled the logs from the WAN/LAN router. Aside from DHCP renewal errors, they are clean.

Much appreciated for the help/windows/troubleshooting ideas.

Jeremy

KE8ROY891

Hi:

Were you able to resolve this issue. I have a similar issue Mac OS SmartSDR. In my case, I am local to the Radio (direct connection via my router at my QTH). When I transmit (any mode, CW, SSB or even tune) after changing from one band (e.g. 20m to 10m) to another, the application freezes. The radio (since I connected to the radio via headphones, and PTT pedal, and CW key) will actually continue to function. I have been in the middle of CW QSO and I was able to finish it, I just couldn't see it in the software.

I have tried restarting the router, removing items on my Mac that start on startup, and cleaning hard disk space. I have an Intel MacBook Pro 2016. Still works great. I have updated to the latest firmware on the radio and I am running Mac OS Sanoma (both recommended by Marcus) running a 1 year old 6600. If anyone has ideas, that would be great. BTW, up until Month or so ago, this all worked great.

Again, any help would be appreciated.

Britt (KE8ROY)

KD0RC

I am surprised that 3.4.23 works at all over SmartLink. I would upgrade everything to 3.5.9 and see if that fixes it.

I had a similar situation where SmartSDR froze, then closed, but the radio kept transmitting for a bit. Turned out to be RF getting into the router and/or switch and/or computer. Ferrites on all my Ethernet cables (including non-radio related ones) cured the issue.

KE8ROY891

Thanks, Len:

I did upgrade to 3.5.9 on Monday this week. No help there. All of this worked before but maybe something shifted. I'll work on adding Ferrites and see if that resolves it.

Thanks much

73 Britt

Mike-VA3MW

My 2 cents --- I quickly scanned your finding good work so far. You have eliminated a few things.

My guess is that your Switch is 'flapping'. Can you try a different switch, even for a test? Even a simple 100mb/sec switch.

This is definitely a transport layer issue.

KE8ROY891

Thanks for jumping in @Mike-VA3MW. I have an old switch here someplace. :) I'll try to find something. I thought about connecting directly to the radio too. But I see there are some hoops to jump through there.

Thanks much, more to follow after I test this.

Britt (KE8ROY)

KE8ROY891

@Mike-VA3MW , and team

I found an old router (not that old but older) and placed it in my network. Unfortunately the results were the same. That is, change bands, transmit, app freezes. I did place Ferrite beads on all my Ethernet cables (quality of them is suspect I admit). None of this has mitigated the problem. Here are other things I have tried.

I have the iOS app both on my phone and iPad and that works to transmit reliably (at least in testing)

I have SmartSDR for Windows 10 on one of my Macs, in a separate partition (so, not in Parallels though I have nothing against Parallels). This too has been working, reliably. In fact I can't say that I have ever crashed it (though as a "Mac guy", it hurts just a little to say that). It too is updated to 3.5.9.

In short. Everything works but the Mac App. I have attempted to clear the cache on the Mac as best as I can without blowing away part of the OS.

Thanks everyone for the help. Any other ideas are welcome....I am running low on them.

73, Britt (KE8ROY)

Mike-VA3MW

Turn off Location Services on the MAC. See if that helps.

(Yes, it makes no sense at all-but give it a shot)

KE8ROY891

@Mike-VA3MW and @KD0RC

Thanks very much for all the help.

My issue is resolved. In the end, I deleted the app, and reinstalled. The clue was that the App had grown to 400MB, from the original install of 64MB. The app package/bundle does save back data and settings. I would expect some growth but something was wrong. I deleted it, downloaded a new app from the App store and bingo, it is fixed. I have tested both CW and SSB and I am doing JS8Call heartbeats and responses as I type. I am geeked to have things working again. I wanted to post this here to be sure a solution is documented. I'll also let Marcus know. BTW, I don't blame him. He is brilliant. I couldn't do what he does. I am thankful for the app. STUFF happens and it isn't always the developers fault.

Anyway, thanks for your help and I hope this helps someone else. Oh and thanks to my Elmer for the encouragement to try my last idea.

73!

Britt - KE8ROY

PS and I'll turn off location services...I kinda don't like it anyway :)