SSDR Waterfall stutter on startup

  • 1
  • Problem
  • Updated 2 years ago
  • (Edited)

Just started to notice a network problem over passed couple of weeks. Not tried any older versions of SSD yet due to lack of time and work pressures.

The F6k is connected directly via Ethernet cat6 shielded cable to the PC (no switch, no router, no DHCP etc).

It looks like a network packet timing issue. The problem is that the waterfall stutters (almost stops) when SSDR is first loaded. The only thing that fixes it is to disable and then re-enable the Ethernet port on the PC. Then it is totally fine until the next load of SSDR when it again does it pretty predictably.

I'm almost certain this is an initial packet sampling issue as the network pop-up reporters says 1 bar (network poor, Latency <1ms) when the problem manifests itself, along with a small percentage of lost packets. However, even when it is stuttering like crazy on the waterfall, the RX and I think also the TX audio is fine. Also driving the radio is also fine, and other apps are fine too. So it appears that only the waterfall network packets are being dropped / affected by the problem.

Normally my network link is perfect at all other times: for example just done a very intensive 24 hour run contest without needing to resolve any network problem, or do a reboot etc. The result was a lost packet count after 24 hrs = zero!

Hardware: F6k, SSDR on W10 (all updates applied), Core i7-4790k @ 4GHz, 32GB DDR3, SSD,GTX 750ti running 4k to single monitor.

 73 de Steve G1XOW

Photo of Steven G1XOW

Steven G1XOW

  • 284 Posts
  • 119 Reply Likes

Posted 2 years ago

  • 1
Photo of EA4GLI - 8P9EH - Salvador

EA4GLI - 8P9EH - Salvador

  • 1758 Posts
  • 536 Reply Likes
I remember running into similar behavior when connecting by VPN two networks with the same gateway address. Both of the networks were using 192.168.1.1 as the gateway and that produced dropped udp packets. I would look at your lan configuration and even for the chance of having a second dhcp server in the network that might be conflicting without you knowing it.

The next thing might be firewall and Antivirus settings. Seeing that Windows 10 upgrades on its own maybe it reset some firewall setting without your knowledge and that is producing an issue with those packets.


Also test with a different ethernet cable just to eliminate that as a factor.


Good luck finding the problem.
(Edited)
Photo of Danny K5CG

Danny K5CG

  • 307 Posts
  • 45 Reply Likes
"Also have a 4 CPU Xeon mobo with 64GB ram, and about 10TB of drives sitting in the shack waiting for time to kick it back to purpose."

FREENAS!!!
Photo of Ken - NM9P

Ken - NM9P, Elmer

  • 4025 Posts
  • 1248 Reply Likes
I played with FREENAS a few years ago just for fun, but didn't really have any practical use for it, but it was educational.  I really should dust off one of my 6 old computers that are gathering dust in the corner of my shack and set it up again.

I also used a Dlink DNS 323 with two 500 GB drives in Raid as a networked backup device until one of the drives failed.  Then while trying to install a replacement drive I accidentally formatted the wrong one!  Yuk!  Kind of took the steam out of my operation.  I have had two new drives for a couple of years and have never gotten around to installing them.
Photo of Danny K5CG

Danny K5CG

  • 307 Posts
  • 45 Reply Likes
My FreeNAS (10TB - 8 usable) is my main CIFS share for my home, it hosts media files for Plex, block devices mounted over iSCSI for Xenserver VMs. You don't need a RAID controller if you use ZFS, but you'll want 8GB RAM (or more). Pretty hard to beat for free. Steve's "extra"machine sounds better than perfect.
Photo of Ken - NM9P

Ken - NM9P, Elmer

  • 4025 Posts
  • 1248 Reply Likes
That reminds me why I stopped using it...

Several years ago, when they introduced ZFS, I didn't have enough RAM in my old computer to do it and lost interest because I didn't want to put any more money into an old computer.  I already had put an IDE-to-memory card adapter for a 2GB memory card (the one that was about four times larger than an SD card) to use as a makeshift SSD drive to hold the FREENAS operating system, and I had a HUGE (tongue in cheek) HD of about 40 GB or so....I think I only had about 2 GB of RAM, if that much.  I never put anything critical on it, just some pictures, iTunes music, and a few videos that apparently were in a format that the video server at the time wouldn't serve.

As I said, I was just playing.  I had just started messing around with various Linux Distros, primarily Mepis, SUSE, Knoppix, and DSL back then.  I play around mostly with Mint right now.

Anyway, to get back to the topic of Flex and Networking....and probably should be on another thread...

Is it possible to run VPN software such as OpenVPN or Softether out of the FREENAS operating system in order to have the single machine doing double duty?  If so, it might entice me to fire up an old computer to do the task.  Or I could just start playing with my Raspbery Pi 3.....

Ken - NM9P
Photo of Danny K5CG

Danny K5CG

  • 307 Posts
  • 45 Reply Likes
Yes, off topic. Email me about this if you want to continue. My email address is listed on qrz.com.
Photo of Mike va3mw

Mike va3mw

  • 824 Posts
  • 196 Reply Likes
A few things I would consider. Since not everyone is reporting this, then we have to look outside SmartSDR and into the environment. This has to do with bursty network communication. As an example, I don't see it on my remote operation on a VPN where my maximum bandwidth is under 500kb/s.

Can you try the following. This may not tell us the answer, but help to find the problem.

Ensure the drivers on your network card are up to date
Hard code the speed on your network card to 100mb/s and not autonegotiate
Having a switch/dhcp network in the middle may help the packet routing and buffering rather than using the 169 address (this may be part of your negotiation issue and actually having a proper internet assignment may speed things up). Windows will do its DHCP dance on ports that are coming active first before defaulting to a 169. address.
What if you test using this setup (regular network/DHCP) instead of a direct connection, do you have the same problem?
While SSDR is starting, do an cmd line 'ipconfig' over and over and see what you see as an assigned address

The 6k will never move enough data to saturate a 100mb/s network. You can watch your network load on windows to see how much data is actually moving. I think you will be surprised how little actual traffic there is.

Just some ideas anyway, Mike va3mw
Photo of KE0U

KE0U

  • 29 Posts
  • 9 Reply Likes
The network heath in SmartSDR is from what I have observed based on the conversational round trip time (RTT).

First step in any problem resolution is to measure the variables. If for example the PC is highly loaded, the SmartSDR application will starve for resources and things will stutter. Initial places to measure would be as follows.

1) Validate the plumbing with the available O/S tools.

Ping the radio using big payloads let this run for hundreds of pings.

ping -t -l 20000 <radio ip>

Ping the network gateway normally your ISP access point or perhaps a router you have installed. Running ipconfig /all will identify the default gateway.

ping -t -l 20000 <default gateway>

Examine the network re-trans and failure statistics

netstat -se

Per your note you have already done this as observed packet loss was zero. Where did you measure?


2) Observe overall system load.

While you should have plenty of horse power on the PC side, worth watching the resource monitor on startup for available CPU headroom, not swapping heavely and no or little wait depth on disk.

Perhaps the severing of the network (logically) is dropping connections which are consuming critical resources. Are you running AV? is the system clean is it doing a scan and causing resource issues?

Just some thoughts

73, Jay


 
Photo of Mike va3mw

Mike va3mw

  • 824 Posts
  • 196 Reply Likes
Excellent!

I will try this over my vpn to see how bad it has to be before it is bad for operating.
(Edited)
Photo of Steven G1XOW

Steven G1XOW

  • 284 Posts
  • 119 Reply Likes
Thanks for the ideas guys. I should have said I'm an MCSE to save some of the more obvious try-this-try-that replies, but thanks anyway.

I knew it couldn't be a cable issue, or IP/DHCP issue as it works fine except for these random intermittent examples where the waterfall data stream fails to sync...and never catches up. Even when the fault state is happening (and is allowed to continue) the radio remains entirely usable. So, to reiterate the radio works fine in every other way - clearly Ethernet data is in fact flying back and forth with ease all the time, only the waterfall stream is being affected. So the question is what is so special about the waterfall data stream?

Anyway, I had a quick play with Ethernet adapter hardware config and think I sorted it - time will tell.

The setting changed were: enlarged RX buffer from 265 to 2048, disabled jumbo frames, and fixed the speed at 1Gbps (was auto detect). No change to DHCP, or AAIA IP allocations which were already fixed at both ends.

Not had time to slice and dice it any further but one of these settings has made the difference, my guess is the RX buffer size, buy why only waterfall data being affected?

As the Ethernet connection to the F6k is isolated from the rest of my network, its easy for me to see what traffic is being thrown around. In total the Ethernet has 12Mbps of pretty much constant RX loading so even  an old-school 100Mbps would be fine, no real need for 1Gbps. Of the 12Mbps about 66% of it is the DAX stream, and the rest is the main SSDR control stream.

An interesting side note: even if you terminate the DAX application process, the DAX network data stream continues banging data at you even when you don't need it. You have to make sure the DAXIQ and DAX setting are off in SSDDR otherwise the radio sends it anyway. This maybe important when connecting over slower VPN, RDP or WAN links).

 73 de Steve G1XOW
Photo of Tim - W4TME

Tim - W4TME, Customer Experience Manager

  • 9152 Posts
  • 3478 Reply Likes
I suspect something localized is causing the problem.  SmartSDR is a WPF application and relies on .NET and DirectX for doing the display rendering.  For issues like this, I would normally recommend that the user make sure that the most current video driver is loaded.  Also, if the PC is using embedded graphics (as opposed to a PCIe x16 video card), make sure that the BIOS is configured to allow the video use as much RAM as possible.

I also noticed that you have the RF Preamp on.  This is usually not necessary on 20m (https://helpdesk.flexradio.com/hc/en-us/articles/204923669-How-to-determine-the-amount-of-RF-Preamp-...).  I'd turn it off and see if that makes a difference.
(Edited)
Photo of Steven G1XOW

Steven G1XOW

  • 284 Posts
  • 119 Reply Likes
Tim,

The 20m preamp was on because I am presently chasing some VDSL wideband noise sources, however, the problem has been seen on any band, even those where the preamp is not available such as 80/160m.

The GPU is actually a Pcie Nvidia 750ti GT with 2GB of dedicated DDR5 ram. The driver is the very latest signed one from the manufacturer, had also previously tried an older driver from about 8 months ago too. Also, tried lower res and full-screen modes. same outcome. No other GPU oddities have been observed - even when doing 3D rendering in apps like Revit or AutoCad.

I don't think this is anything to do with rendering as the change of NIC buffering appears to have solved it. I will do some more testing this evening to prove it. If I can make to problem reappear and then be solved again by a single change of the NIC buffers I think that would be pretty conclusive.
Photo of Tim - W4TME

Tim - W4TME, Customer Experience Manager

  • 9152 Posts
  • 3478 Reply Likes
Yeah, that video card should render the displays at any frame rate just fine.

"If I can make to problem reappear and then be solved again by a single change of the NIC buffers I think that would be pretty conclusive."

Agreed.
Photo of Steven G1XOW

Steven G1XOW

  • 284 Posts
  • 119 Reply Likes
Tim,

just to conclude this. After many nights of combination testing of ALL of the vast settings in my NIC drivers, I can positively confirm that this problem is caused (and goes away) by changing from 1Gbps full duplex to 100Mbps full duplex on the NIC.

None of the other NIC settings affect the outcome in anyway. For now 100Mbps is okay but this needs listing to get looked at. I think it might only apply where the NIC is directly connected to the PC (i.e. no switch, router etc).

For confirmation the TX and RX audio remains fine and the radio is usable, it is just the waterfall that becomes totally destroyed and does not recovery.

73 de Steve G1XOW
Photo of Michael - OZ1MDF

Michael - OZ1MDF

  • 40 Posts
  • 14 Reply Likes
@Steve - G1XOW,
Please check your mailbox spam-filter .. I am trying to reach you !
(Edited)