dropped packets when client connected from same machine

  • 1
  • Problem
  • Updated 3 years ago
  • Solved
  • (Edited)
Flex 6300
SmartSDR 1.5.1

Running SmartSDR from a laptop directly wired (or through a GigE switch) I see Network quality is good and ping time < 1ms and ZERO dropped packets for long periods of time. Very solid network link.

If I run another client app on the same laptop. as soon as the additional client app connects I see packet loss and ORANGE/RED status on the network graph in SmartSDR. Ping time is sill < 1ms.

If I run the same application on another laptop (a peer on the same network) there are no lost packets.

This happens with the Behringer CMD Micro DJ panel application William Hemmingsen created and the HRI Agent application that Mark WS7M has started.

I've disabled Windows Firewall completely and it made no difference.

Dissbled WiFi, the only NIC on the laptop is the wired NIC to the radio/switch. No difference.

SmartSDR may be reporting the packets sent to the add-on application as lost but I don't think it should do that.

Update: I also moved off the 169.254.0.0/16 default network and placed the laptop and the 6300 onto a router where they ran on the 192.169.1.0/24 subnet (DHCP from the router) and it behaves exactly the same.

Has anybody else experienced this?

Danny
K2DMS
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes

Posted 4 years ago

  • 1
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Here is a video capture of the behavior. This was taken using Mark's new HRI Agent application but it does exactly the same thing with the Berhinger CMD Micro DJ app. It isn't anything specific to one app or another, but all of them.

https://www.dropbox.com/s/if3xt87nts2vabu/2016-01-02%20at%2000-14-42.mp4?dl=0

Danny
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
I have seen this too with William's CMD Micro App. I can provide wireshark captures if anyone from Flex would like to see them.
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
Actually now that I look at it closer, performance doesn't really seem to suffer. It looks like the client just reports the packets as dropped, like Danny noted.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
I neglected to mention that my OS is WIndows 10 Home X64. The NIC is a Realtek PCIe GigE controller (in an HP ENVY 17T laptop).
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
Odd behavior for me now. This morning when I woke up I saw Danny's last post and it gave me the idea to try the other NIC in my Machine. (I am on Win8.1 x64) as I was using a Realtek before and the other is Intel.

Tried the other (Intel) NIC and everything appeared to be working as normal, other than a little lag for a few seconds at first. Diagnostics indicated no dropped packets.

So I shut down Smart SDR and the CMD Micro program and switched back to the Realtek, and it appears to be working fine again. No dropped packets or anything. Very strange. So it is intermittent, at least for me.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
I can reproduce this at will on my HP Envy 17t (Windows 10 Home X64) but it does not happen on my HP Elitebook with Windows 7 with the same radio and same version of HRI Agent. I'll try a USB NIC on the HP Envy and try to separate the NIC from Windows 10.
(Edited)
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Update: The same thing happens with the built in WiFi NIC so it isn't isolated to the RealTek GigE wired NIC. I also uninstalled VMware Workstation 11 thinking perhaps the extra internal virtual networks were interfering with the native networking but that made no difference.

Next: I'll install the SmartSDR and HRI Agent on my wife'd Dell laptop (windows 8.1). I suspect that will work fine so then I'll install Windows 8.1 on the HP Envy 17t and see how it goes.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Update: I was surprised to find that the dropped packets condition also happens on my wife's Dell 17R (Core i5 - Windows 8.1 X64) using the WiFi connection.

I will install Windows 7 on my HP Envy 17t and see what happens there. I think it's also time I took some wireshark captures to grab some traffic.

Danny
K2DMS
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Update: I have found some information that may be useful.

Windows 7 X64 Ultimate on a Dell Core i5 desktop.

If I start HRI Agent (or the agent for the Behringer CMD Micro DJ) first, and start SmartSDR last, then the issue does not occur. If I start SmartSDR first, followed by either program, then I see dropped packet indication.

"Dropped packet" isn't really the correct term but this is how SmartSDR sees it apparently as it de-duplicates the data. Looking at Wireshark when this happens I see duplicate stream data in the VITA 49 protocol packets (different  VRT Header and Sequence numbers) but identical stream ID and data payloads. It appears as though stream data is being sent from the radio to what it thinks are 2 different clients but at the same IP address. This is purely conjecture on my part of course because I don't know anything about this traffic other than I can confirm that the duplicate packets are not there when dropped packets are not indicated.

Wireshark capture file. 

https://www.dropbox.com/s/jd1weholzmsyr3i/SMartSDR%20then%20HRI%20Agent%20start.pcapng?dl=0

The link above is a capture from Wireshark 2.0.1 with filter as "host 192.168.1.242 and not broadcast and not arp and not icmp". 192.168.1.242 is the IP of the Flex 6300 on my local subnet.

This captures the initial connection from SmartSDR to the radio. You can clearly see the UDP fragmented packets followed by VITA 49 protocol packets and in the VITA 49 packets there are two stream numbers, usually Stream ID: 0x40000000 and Stream ID: 0x00000700, sometimes Stream ID: 0x42000000.

I started up HRI Agent and you can see it connect at packet 771.

Immediately as of packet number 782 you can see that its data payload is identical to the packet before 781. Different VRT Headers but the exact same Stream ID and Data fields.

Something is not right here. We shouldn't have to start 3rd party apps before SmartSDR, it shouldn't matter what order you start them in. The radio shouldn't be sending duplicated VITA 49 data packets only to have SmartSDR claim them as dropped (IMHO).

I believe this will require some research on the part of smarter people than myself.

Help?

Danny
K2DMS
(Edited)
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Digging a little more... I had the idea to use Wireshark to capture data on 2 machines, one with SmartSDR running and another with HRI Agent running to see what data was streaming.

If I start SmartSDR first on one machine, and then HRI Agent on the other, I see the VITA 49 protocol using Stream ID 700 on both machines in Wireshark.

The stream ID 700 data doesn't stop even if the disconnect button is pressed on HRI Agent, not until the application is closed does it stop.

This would appear to be why the duplicate/dropped packets are not reported, when the HRI Agent is on a different machine. The stream is present but not counted by SmartSDR.

I captured a change to a slice and I saw TCP packets coming from the radio to the HRI Agent app which included the new frequency in the data field (intermixed with the VITA 49 data).

The other test, to confirm, I started Wireshark on the laptop, started HRI Agent and capture the TCP connect session. I started SmartSDR on the desktop and I see no VITA 49 packets at all on the Laptop. This is also what I was seeing when SmartSDR and HRI Agent were on the same machine if I started HRI Agent first (no duplicate packets).

Can somebody do a similar test and see if there are VITA 49 packets on the machine where HRI Agent is running as a standalone (SmartSDR on another machine)?

I can see no reason why the HRI Agent machine would be getting this VITA 49 protocol data. There is something in the radio firmware or SmartSDR that is causing the duplicate stream to be initiated to the HRI Agent (wherever it is) for no apparent reason.

I think I've proven here that it is not HRI Agent that is the culprit.

Fun fun...

Danny
K2DMS
(Edited)
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Additional Info: CAT and DAX are not running on either machine. 
Photo of Mark - WS7M

Mark - WS7M

  • 1350 Posts
  • 503 Reply Likes
Danny,

This could be that the HRI Agent is not unsubscribing to the slices.  I'll check into that.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Next I am going to capture the TCP packets for HRI Agent and compare them based on the startup sequence (HRI Agent first vs. SmartSDR first) and see if there are any differences.

Danny
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
When SmartSDR is not running there are some packets sent from the radio to the HRI Agent indicated as H1 protocol instead of TCP protocol when SmartSDR is running and 2 packets with the slice information is also sent from the radio to HRI Agent when SmartSDR is running which are replaced by 1 - H1 packet when it is not. Otherwise the data sent from HRI Agent appears identical.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Official Response
This behavior doesn't appear to be occurring in v1.6.
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
I upgraded to 1.6.17 and am still having this problem with the CMD micro program. The network stats show about 50% lost packets...

I do seem to have found a workaround though...  If I start the CMD Micro program first, then SmartSDR, then DXLabs, the network stats seem to be happy.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Yes, me too. Although I reported that it wasn't happening with one desktop, it did with my laptop so I think there is still something to it. I need to capture some more data.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
The problem remains with v1.6.17.

I start SmartSDR, I see VITA 49 protocol packets coming from the radio to the SmarSDR app. I start HRI Agent, I see duplicate VITA 49 protocol packets. SmartSDR reports it as dropped with bad health.

HRI Agent connects at packet 6932, duplicate data can be seen starting at packet 6944. Stream ID 700 in both with different VRT headers in each copy. Wireshark capture file here

https://www.dropbox.com/s/5czy561p7eg2t6z/2015-01-13-VITA49_dupes.pcapng?dl=0

Interestingly, when I was not seeing this condition, there were no VITA 49 packets when SmartSDR connected, and only one set when HRI Agent connected. Observation only.
(Edited)
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Even if apps claimed a different UDP port wouldn't the data still be duplicated on the subnet? That doubles the utilization for the same information. Wouldn't a multicast protocol be better at distributing just a single copy of the data stream? Just thinking out loud (in the dark).

Danny 
Photo of Steve - N5AC

Steve - N5AC, VP Engineering / CTO

  • 1057 Posts
  • 1097 Reply Likes
Yes, but multicast has it's own problems -- specifically that it puts a load on every computer on the subnet.  Generally, for high-volume data applications multicast goes hand-in-hand with VLAN (or segmented networks).  For now, we've assumed that this solution is too much trouble for the typical amateur radio enthusiast so we have not implemented it.  Incidentally, our government products do multicast over VLAN.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Very Good Steve, I'm just grasping at straws here. I was just imagining that with all the 3rd party apps that hams are connecting (DDUtil, HRI Agent, DVKs, S-Meters, CMD Micro, DX Lab Suite, FRStack)... The list is getting long. "If" every app opens a new UDP stream... is there a tipping point where multicast makes sense? Don't feel you need to answer, I'm just thinking out loud here again and learning as I go.

Danny
Photo of Steve - N5AC

Steve - N5AC, VP Engineering / CTO

  • 1057 Posts
  • 1097 Reply Likes
Yes, there's certainly a place for it.  We could also build an aggregation layer for the PC where it keeps track of everything that PC has subscribed to, asks for it once, and distributes to the clients that asked for it.  Lots of things we could do.  Really the panadapter, waterfall and DAX are the big bandwidth users and they are not typically shared today.  Metering is just not a big enough deal to worry a lot about, but someday we may look into this.
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 915 Posts
  • 343 Reply Likes
Note that while there is duplicate data being sent in these situations, the actual "failure" here (supposed packet loss) would be fixed by not duplicating the data on the same port.
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 340 Posts
  • 70 Reply Likes
I run HRI almost 100% of the time radio is on... I have checked for dropped packets after these posts started... None here.  Although the radio and operating computer are on the same switch and the system running HRI is connected directly to the home router. 
I can assist in testing if you wish.
Dan
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 915 Posts
  • 343 Reply Likes
Dan -- it would be interesting if you could run a test for me.  Close all the Flex related clients and open them in this order:
1. SmartSDR
2. CAT (optional)
3. DAX (optional)
4. HRI

Then confirm whether you get the reported packet loss.  I suspect you will see it when starting things in this order.
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 340 Posts
  • 70 Reply Likes


After less than a min. Now 9197 ..
Dan -- KC4GO
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 915 Posts
  • 343 Reply Likes
This confirms the theory.  Thanks Dan.
Photo of Mark - WS7M

Mark - WS7M

  • 1350 Posts
  • 503 Reply Likes
I just tried that order.  0 dropped packets and this is with HRI still not connecting to UDP.
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
I did the same test sequence, I opened HRI Agent at about 3300 total packets logged (none dropped). By the time I got to 6600 logged (doubled) I had 2130 dropped.
Photo of Mark - WS7M

Mark - WS7M

  • 1350 Posts
  • 503 Reply Likes
So I guess I'm confused.  Should I modify HRI to connect to or specify then connect to a UDP port to solve this issue from my side?
Photo of Danny K5CG

Danny K5CG

  • 385 Posts
  • 61 Reply Likes
Please check #2849, if it is related or a duplicate. Steve opened it from earlier info.
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 915 Posts
  • 343 Reply Likes
Thanks for the heads up.  I related them and closed #2849 as I believe we have nailed the root cause and a work around for now.
Photo of John G3WGV

John G3WGV

  • 199 Posts
  • 37 Reply Likes
Hi Eric,
Sorry to dredge this one back up again but did #2888 get resolved? I am getting the same thing running WinSSDR plus my logging client, which is connected to the TCP stream on 4992. The logging client doesn't use streaming UDP data, so I don't set the port or listen for it. The only thing I'm subscribing to on the TCP port is "slice all". I'm seeing 56% dropped packets on the network diagnostic monitor. Using v1.8.4.84.

I'm guessing the dropped packets relates to UDP so I'm wondering if the radio is streaming stuff I haven't asked for? It's as if meter or FFT data are being streamed to my logging client.

Edit: I changed my logging program to set a different streaming UDP port and that completely fixed the problem. So I am guessing that the radio is streaming a pile of stuff even though I've only subscribed to slices.

John.
(Edited)
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 915 Posts
  • 343 Reply Likes
John,

No problem.  I show that #2888 is still open, but it is scheduled to be worked on.
Photo of John G3WGV

John G3WGV

  • 199 Posts
  • 37 Reply Likes
Thanks Eric. I'm starting to get the hang of this API now :)