SSDR Waterfall stutter on startup

Steve G1XOW

Just started to notice a network problem over passed couple of weeks. Not tried any older versions of SSD yet due to lack of time and work pressures.

The F6k is connected directly via Ethernet cat6 shielded cable to the PC (no switch, no router, no DHCP etc).

It looks like a network packet timing issue. The problem is that the waterfall stutters (almost stops) when SSDR is first loaded. The only thing that fixes it is to disable and then re-enable the Ethernet port on the PC. Then it is totally fine until the next load of SSDR when it again does it pretty predictably.

I'm almost certain this is an initial packet sampling issue as the network pop-up reporters says 1 bar (network poor, Latency <1ms) when the problem manifests itself, along with a small percentage of lost packets. However, even when it is stuttering like crazy on the waterfall, the RX and I think also the TX audio is fine. Also driving the radio is also fine, and other apps are fine too. So it appears that only the waterfall network packets are being dropped / affected by the problem.

Normally my network link is perfect at all other times: for example just done a very intensive 24 hour run contest without needing to resolve any network problem, or do a reboot etc. The result was a lost packet count after 24 hrs = zero!

Hardware: F6k, SSDR on W10 (all updates applied), Core i7-4790k @ 4GHz, 32GB DDR3, SSD,GTX 750ti running 4k to single monitor.

 73 de Steve G1XOW

EA4GLI

I remember running into similar behavior when connecting by VPN two networks with the same gateway address. Both of the networks were using 192.168.1.1 as the gateway and that produced dropped udp packets. I would look at your lan configuration and even for the chance of having a second dhcp server in the network that might be conflicting without you knowing it.
The next thing might be firewall and Antivirus settings. Seeing that Windows 10 upgrades on its own maybe it reset some firewall setting without your knowledge and that is producing an issue with those packets.

Also test with a different ethernet cable just to eliminate that as a factor.

Good luck finding the problem.

Steve G1XOW

The IP for the F6k is local only and on a 169..... address, but the other side of the network is via a WIFI Pcie network card. I'll check it just in case the Ethernet has somehow gained a default gateway to the router.

Bill W2PKY

Had this issue when upgrading to Win10 turned out to be 256 buffer size on the Ethernet card. Bumped up to 2048 and problem was solved.

Mike va3mw

A few things I would consider. Since not everyone is reporting this, then we have to look outside SmartSDR and into the environment. This has to do with bursty network communication. As an example, I don't see it on my remote operation on a VPN where my maximum bandwidth is under 500kb/s. Can you try the following. This may not tell us the answer, but help to find the problem. Ensure the drivers on your network card are up to date Hard code the speed on your network card to 100mb/s and not autonegotiate Having a switch/dhcp network in the middle may help the packet routing and buffering rather than using the 169 address (this may be part of your negotiation issue and actually having a proper internet assignment may speed things up). Windows will do its DHCP dance on ports that are coming active first before defaulting to a 169. address. What if you test using this setup (regular network/DHCP) instead of a direct connection, do you have the same problem? While SSDR is starting, do an cmd line 'ipconfig' over and over and see what you see as an assigned address The 6k will never move enough data to saturate a 100mb/s network. You can watch your network load on windows to see how much data is actually moving. I think you will be surprised how little actual traffic there is. Just some ideas anyway, Mike va3mw

EA4GLI

What a great find Bill!!! Not sure if it is the problem for Steve, but it is a good piece of info.

W0JD

The network heath in SmartSDR is from what I have observed based on the conversational round trip time (RTT).

First step in any problem resolution is to measure the variables. If for example the PC is highly loaded, the SmartSDR application will starve for resources and things will stutter. Initial places to measure would be as follows.

1) Validate the plumbing with the available O/S tools.

Ping the radio using big payloads let this run for hundreds of pings.

ping -t -l 20000 <radio ip>

Ping the network gateway normally your ISP access point or perhaps a router you have installed. Running ipconfig /all will identify the default gateway.

ping -t -l 20000 <default gateway>

Examine the network re-trans and failure statistics

netstat -se

Per your note you have already done this as observed packet loss was zero. Where did you measure?


2) Observe overall system load.

While you should have plenty of horse power on the PC side, worth watching the resource monitor on startup for available CPU headroom, not swapping heavely and no or little wait depth on disk.

Perhaps the severing of the network (logically) is dropping connections which are consuming critical resources. Are you running AV? is the system clean is it doing a scan and causing resource issues?

Just some thoughts

73, Jay


 

Mike va3mw

Excellent!

I will try this over my vpn to see how bad it has to be before it is bad for operating.

Steve G1XOW

Thanks for the ideas guys. I should have said I'm an MCSE to save some of the more obvious try-this-try-that replies, but thanks anyway.

I knew it couldn't be a cable issue, or IP/DHCP issue as it works fine except for these random intermittent examples where the waterfall data stream fails to sync...and never catches up. Even when the fault state is happening (and is allowed to continue) the radio remains entirely usable. So, to reiterate the radio works fine in every other way - clearly Ethernet data is in fact flying back and forth with ease all the time, only the waterfall stream is being affected. So the question is what is so special about the waterfall data stream?

Anyway, I had a quick play with Ethernet adapter hardware config and think I sorted it - time will tell.

The setting changed were: enlarged RX buffer from 265 to 2048, disabled jumbo frames, and fixed the speed at 1Gbps (was auto detect). No change to DHCP, or AAIA IP allocations which were already fixed at both ends.

Not had time to slice and dice it any further but one of these settings has made the difference, my guess is the RX buffer size, buy why only waterfall data being affected?

As the Ethernet connection to the F6k is isolated from the rest of my network, its easy for me to see what traffic is being thrown around. In total the Ethernet has 12Mbps of pretty much constant RX loading so even  an old-school 100Mbps would be fine, no real need for 1Gbps. Of the 12Mbps about 66% of it is the DAX stream, and the rest is the main SSDR control stream.

An interesting side note: even if you terminate the DAX application process, the DAX network data stream continues banging data at you even when you don't need it. You have to make sure the DAXIQ and DAX setting are off in SSDDR otherwise the radio sends it anyway. This maybe important when connecting over slower VPN, RDP or WAN links).

 73 de Steve G1XOW

Ken - NM9P

Steve, Do you mean turning the DAX and DAXIQ streams off in the Slice Flag and Left-Hand pull-out menus?  

I have also noticed that in a couple of bandwidth tests about 66% of my data rate was removed when I turned the DAX click-boxes off in the DAX Control panel and turned DAX off in the slice and Panadapter windows.  I didn't check it comparing whether the DAX control panel was closed or not.

Ken - NM9P

Steve G1XOW

Hi Ken.

Yes. I normally have the DAXIQ on channel 2 and the slice DAX on channel 1.
It looks like the settings work entirely independently of each other, which is fine.

The observation is not really a problem for most people, only that it suggests that the radio is sending out DAX data "in the dark" (blind), i.e without any ack/control or handshaking going on from the PC-side.

Cheers, 73 de Steve G1XOW

Tim - W4TME

The audio coming from the radio is UDP.  The protocol does not use ack/control or handshaking.

Steve G1XOW

Tim,

I was actually referring to inter-process coordination, not UDP datagram flow control as such.

However, having tried it again and this time preventing the DAX control panel from auto starting, I can see there is some kind of basic awareness going on because the radio does not actually send DAX streams if the DAX control panel is not also running.

Going back to the original problem: assuming that the TX and RX audio streams are also UDP datagrams then why should only the waterfall be affected in this way?  It suggests that the waterfall read-and-display routine is not very tightly written if it has to rely on very large buffers.

73 de Steve G1XOW

Tim - W4TME

I suspect something localized is causing the problem.  SmartSDR is a WPF application and relies on .NET and DirectX for doing the display rendering.  For issues like this, I would normally recommend that the user make sure that the most current video driver is loaded.  Also, if the PC is using embedded graphics (as opposed to a PCIe x16 video card), make sure that the BIOS is configured to allow the video use as much RAM as possible.

I also noticed that you have the RF Preamp on.  This is usually not necessary on 20m (https://helpdesk.flexradio.com/hc/en-us/articles/204923669-How-to-determine-the-amount-of-RF-Preamp-...).  I'd turn it off and see if that makes a difference.

Steve G1XOW

Tim,

The 20m preamp was on because I am presently chasing some VDSL wideband noise sources, however, the problem has been seen on any band, even those where the preamp is not available such as 80/160m.

The GPU is actually a Pcie Nvidia 750ti GT with 2GB of dedicated DDR5 ram. The driver is the very latest signed one from the manufacturer, had also previously tried an older driver from about 8 months ago too. Also, tried lower res and full-screen modes. same outcome. No other GPU oddities have been observed - even when doing 3D rendering in apps like Revit or AutoCad.

I don't think this is anything to do with rendering as the change of NIC buffering appears to have solved it. I will do some more testing this evening to prove it. If I can make to problem reappear and then be solved again by a single change of the NIC buffers I think that would be pretty conclusive.

Tim - W4TME

Yeah, that video card should render the displays at any frame rate just fine.

"If I can make to problem reappear and then be solved again by a single change of the NIC buffers I think that would be pretty conclusive."

Agreed.

Walt - KZ1F

32GB? Steve, are you renting time on that thing?

Steve G1XOW

Walt,

No way, get your own!  

I run a very busy IT support centre, with 5 engineers, so my home machine is also my mission control fall-back station. As well as all the ham stuff I have 6 other partitions with everything from Ubunto, MS Server 2010 etc. etc. on it. It actually has 4 x 1TB SSD drive is raid-10 array if you want to get down and dirty with then spec...

Also have a 4 CPU Xeon mobo with 64GB ram, and about 10TB of drives sitting in the shack waiting for time to kick it back to purpose.

Walt - KZ1F

Nah, I have an OpenStack cloud running in the basement. Now the. XPS 8910 w/16GB is set up it's predecessor, a 3847 w/16GB will be yet another compute node. That'll run a boatload of containers and VM s.

K5CG

"Also have a 4 CPU Xeon mobo with 64GB ram, and about 10TB of drives sitting in the shack waiting for time to kick it back to purpose."

FREENAS!!!

Ken - NM9P

I played with FREENAS a few years ago just for fun, but didn't really have any practical use for it, but it was educational.  I really should dust off one of my 6 old computers that are gathering dust in the corner of my shack and set it up again.

I also used a Dlink DNS 323 with two 500 GB drives in Raid as a networked backup device until one of the drives failed.  Then while trying to install a replacement drive I accidentally formatted the wrong one!  Yuk!  Kind of took the steam out of my operation.  I have had two new drives for a couple of years and have never gotten around to installing them.

K5CG

My FreeNAS (10TB - 8 usable) is my main CIFS share for my home, it hosts media files for Plex, block devices mounted over iSCSI for Xenserver VMs. You don't need a RAID controller if you use ZFS, but you'll want 8GB RAM (or more). Pretty hard to beat for free. Steve's "extra"machine sounds better than perfect.

Ken - NM9P

That reminds me why I stopped using it...

Several years ago, when they introduced ZFS, I didn't have enough RAM in my old computer to do it and lost interest because I didn't want to put any more money into an old computer.  I already had put an IDE-to-memory card adapter for a 2GB memory card (the one that was about four times larger than an SD card) to use as a makeshift SSD drive to hold the FREENAS operating system, and I had a HUGE (tongue in cheek) HD of about 40 GB or so....I think I only had about 2 GB of RAM, if that much.  I never put anything critical on it, just some pictures, iTunes music, and a few videos that apparently were in a format that the video server at the time wouldn't serve.

As I said, I was just playing.  I had just started messing around with various Linux Distros, primarily Mepis, SUSE, Knoppix, and DSL back then.  I play around mostly with Mint right now.

Anyway, to get back to the topic of Flex and Networking....and probably should be on another thread...

Is it possible to run VPN software such as OpenVPN or Softether out of the FREENAS operating system in order to have the single machine doing double duty?  If so, it might entice me to fire up an old computer to do the task.  Or I could just start playing with my Raspbery Pi 3.....

Ken - NM9P

K5CG

Yes, off topic. Email me about this if you want to continue. My email address is listed on qrz.com.

Steve G1XOW

Tim,

just to conclude this. After many nights of combination testing of ALL of the vast settings in my NIC drivers, I can positively confirm that this problem is caused (and goes away) by changing from 1Gbps full duplex to 100Mbps full duplex on the NIC.

None of the other NIC settings affect the outcome in anyway. For now 100Mbps is okay but this needs listing to get looked at. I think it might only apply where the NIC is directly connected to the PC (i.e. no switch, router etc).

For confirmation the TX and RX audio remains fine and the radio is usable, it is just the waterfall that becomes totally destroyed and does not recovery.

73 de Steve G1XOW

Michael - OZ1MDF

@Steve - G1XOW,
Please check your mailbox spam-filter .. I am trying to reach you !