dropped packets when client connected from same machine

K5CG

Flex 6300
SmartSDR 1.5.1

Running SmartSDR from a laptop directly wired (or through a GigE switch) I see Network quality is good and ping time < 1ms and ZERO dropped packets for long periods of time. Very solid network link.

If I run another client app on the same laptop. as soon as the additional client app connects I see packet loss and ORANGE/RED status on the network graph in SmartSDR. Ping time is sill < 1ms.

If I run the same application on another laptop (a peer on the same network) there are no lost packets.

This happens with the Behringer CMD Micro DJ panel application William Hemmingsen created and the HRI Agent application that Mark WS7M has started.

I've disabled Windows Firewall completely and it made no difference.

Dissbled WiFi, the only NIC on the laptop is the wired NIC to the radio/switch. No difference.

SmartSDR may be reporting the packets sent to the add-on application as lost but I don't think it should do that.

Update: I also moved off the 169.254.0.0/16 default network and placed the laptop and the 6300 onto a router where they ran on the 192.169.1.0/24 subnet (DHCP from the router) and it behaves exactly the same.

Has anybody else experienced this?

Danny
K2DMS

K5CG

Here is a video capture of the behavior. This was taken using Mark's new HRI Agent application but it does exactly the same thing with the Berhinger CMD Micro DJ app. It isn't anything specific to one app or another, but all of them.

https://www.dropbox.com/s/if3xt87nts2vabu/2016-01-02%20at%2000-14-42.mp4?dl=0

Danny

James - K4JK

I have seen this too with William's CMD Micro App. I can provide wireshark captures if anyone from Flex would like to see them.

James - K4JK

Actually now that I look at it closer, performance doesn't really seem to suffer. It looks like the client just reports the packets as dropped, like Danny noted.

K5CG

I neglected to mention that my OS is WIndows 10 Home X64. The NIC is a Realtek PCIe GigE controller (in an HP ENVY 17T laptop).

James - K4JK

Odd behavior for me now. This morning when I woke up I saw Danny's last post and it gave me the idea to try the other NIC in my Machine. (I am on Win8.1 x64) as I was using a Realtek before and the other is Intel.

Tried the other (Intel) NIC and everything appeared to be working as normal, other than a little lag for a few seconds at first. Diagnostics indicated no dropped packets.

So I shut down Smart SDR and the CMD Micro program and switched back to the Realtek, and it appears to be working fine again. No dropped packets or anything. Very strange. So it is intermittent, at least for me.

K5CG

I can reproduce this at will on my HP Envy 17t (Windows 10 Home X64) but it does not happen on my HP Elitebook with Windows 7 with the same radio and same version of HRI Agent. I'll try a USB NIC on the HP Envy and try to separate the NIC from Windows 10.

K5CG

Update: The same thing happens with the built in WiFi NIC so it isn't isolated to the RealTek GigE wired NIC. I also uninstalled VMware Workstation 11 thinking perhaps the extra internal virtual networks were interfering with the native networking but that made no difference.

Next: I'll install the SmartSDR and HRI Agent on my wife'd Dell laptop (windows 8.1). I suspect that will work fine so then I'll install Windows 8.1 on the HP Envy 17t and see how it goes.

K5CG

Update: I was surprised to find that the dropped packets condition also happens on my wife's Dell 17R (Core i5 - Windows 8.1 X64) using the WiFi connection.

I will install Windows 7 on my HP Envy 17t and see what happens there. I think it's also time I took some wireshark captures to grab some traffic.

Danny
K2DMS

K5CG

Update: I have found some information that may be useful.

Windows 7 X64 Ultimate on a Dell Core i5 desktop.

If I start HRI Agent (or the agent for the Behringer CMD Micro DJ) first, and start SmartSDR last, then the issue does not occur. If I start SmartSDR first, followed by either program, then I see dropped packet indication.

"Dropped packet" isn't really the correct term but this is how SmartSDR sees it apparently as it de-duplicates the data. Looking at Wireshark when this happens I see duplicate stream data in the VITA 49 protocol packets (different  VRT Header and Sequence numbers) but identical stream ID and data payloads. It appears as though stream data is being sent from the radio to what it thinks are 2 different clients but at the same IP address. This is purely conjecture on my part of course because I don't know anything about this traffic other than I can confirm that the duplicate packets are not there when dropped packets are not indicated.

Wireshark capture file. 

https://www.dropbox.com/s/jd1weholzmsyr3i/SMartSDR%20then%20HRI%20Agent%20start.pcapng?dl=0

The link above is a capture from Wireshark 2.0.1 with filter as "host 192.168.1.242 and not broadcast and not arp and not icmp". 192.168.1.242 is the IP of the Flex 6300 on my local subnet.

This captures the initial connection from SmartSDR to the radio. You can clearly see the UDP fragmented packets followed by VITA 49 protocol packets and in the VITA 49 packets there are two stream numbers, usually Stream ID: 0x40000000 and Stream ID: 0x00000700, sometimes Stream ID: 0x42000000.

I started up HRI Agent and you can see it connect at packet 771.

Immediately as of packet number 782 you can see that its data payload is identical to the packet before 781. Different VRT Headers but the exact same Stream ID and Data fields.

Something is not right here. We shouldn't have to start 3rd party apps before SmartSDR, it shouldn't matter what order you start them in. The radio shouldn't be sending duplicated VITA 49 data packets only to have SmartSDR claim them as dropped (IMHO).

I believe this will require some research on the part of smarter people than myself.

Help?

Danny
K2DMS

Mark_WS7M

Hi Danny,

Good detective work.  I will try the same sequences myself and see what I see.

mark

K5CG

Mark,

I've discovered something else that is probably significant for root cause analysis.

If I start HRI Agent first, then SmartSDR second... I can close and restart HRI Agent as many times as desired and it doesn't trigger the duplicate(dropped) packet issue.

However, if I start SMartSDR first, every time I open HRI Agent I get the duplicate(dropped) packet issue.

So it only happens when SmartSDR is started first and then HRI Agent afterward.

What does that tell us?

Danny

Mark_WS7M

Danny,

I'm really not sure what it tells us.

For me the unknown is what Flex is doing as part of their packet counting and roundtrip mechanism.  

What I can tell you is what HRI Agent does:

When first run it opens a UDP socket listing for any address for UDP datagrams on for 4992.

By nature UDP transmission is stateless meaning there is no record of connect, disconnect etc.  It is a way of broadcasting data.

HRI Agent when the UDP socket is opened receives the broadcast datagrams and decodes them.  The datagrams are kind of large.  They consist of like 28 bytes of header plus a bunch of space delimited strings like  model=<model> serial=<serial> etc.  HRI Agent pretty much ignores the 28 byte header at the time being and decodes the strings.

It then looks in a list it keeps to see if the radio is a new radio.  If so it adds it to the list and refreshes the listbox you see on screen.

If you leave HRI agent in this state this is literally all it is doing.  So the first thing I'd like to know is in your testing where you have run agent first or SmartSDR first what state have you left the agent in.  If you left it in the unconnected state and SmartSDR is still reporting lost packets then I venture to say there is a problem with their packet counting algorithm because UDP packets sort of really should not count.  IE they are just sent off into the void.

If you take HRI Agent to the next level and connect it to your radio with a double click it then opens a TCP socket to the radio and almost immediately issues the API command to subscribe to all slices.   Slices are the only thing right now that HRI Agent subscribes to.

This means HRI Agent receives radio information and information about all slices in one big blast upon connection.  I could see this causing a minor hit on the packet loss but not much.

Beyond that HRI Agent is totally in a listening mode.  IE it is not commanding the radio in any way.  It is simply receiving and decoding the various data sent out by the slice subscription command.

So if this causes the packet loss to go up then I still must suggest there is a problem in their algorithm. 

I have not had a chance to look at your wire shark capture.  I will tomorrow.  But I don't think that HRI Agent is doing anything illegal to cause the packet loss.  The API clearly states that external clients can subscribe to slices, meters, pan adapters, audio, etc.  HRI Agent is only subscribing to slices and nothing else.

So I will help any way I can and my first task is going to be to use my laptop and setup the scenarios you mentioned and see if I can duplicate packet loss.  Before you had the before/after cases I did try running my laptop on Win10 and connection both HRI Agent and SmartSDR and I saw only 1 dropped packet but I cannot remember the order it was done in.  So let me try to reproduce your results.

Mark

K5CG

Mark,

There are no duplicated packets (supposedly dropped) until the double click on the radio entry in the HRI Agent dialog (when SmartSDR is already running).

I can certainly live with starting HRI Agent first as a SOP, but I think there is some value in understanding what is going on here so that it isn't a requirement.

Thanks for your help.

Danny

Dan -- KC4GO

I have a Desktop W10, i7, Realtek  PCI GBE  and a Dell XPS-13 W10, i7, Dell GBE on the same network I tried them both with no dropped packets. I also tried running HRI on my server "Odd Job" Realtek PCI GBE and could not reproduce the network issue.  I also have a switch between the radio and computers.  Between the server and the radio is the switch and the router. 
Wire Shark produced no abnormal packets as well. 
Dan--- KC4GO


K5CG

Thanks for trying Dan.

K5CG

Digging a little more... I had the idea to use Wireshark to capture data on 2 machines, one with SmartSDR running and another with HRI Agent running to see what data was streaming.

If I start SmartSDR first on one machine, and then HRI Agent on the other, I see the VITA 49 protocol using Stream ID 700 on both machines in Wireshark.

The stream ID 700 data doesn't stop even if the disconnect button is pressed on HRI Agent, not until the application is closed does it stop.

This would appear to be why the duplicate/dropped packets are not reported, when the HRI Agent is on a different machine. The stream is present but not counted by SmartSDR.

I captured a change to a slice and I saw TCP packets coming from the radio to the HRI Agent app which included the new frequency in the data field (intermixed with the VITA 49 data).

The other test, to confirm, I started Wireshark on the laptop, started HRI Agent and capture the TCP connect session. I started SmartSDR on the desktop and I see no VITA 49 packets at all on the Laptop. This is also what I was seeing when SmartSDR and HRI Agent were on the same machine if I started HRI Agent first (no duplicate packets).

Can somebody do a similar test and see if there are VITA 49 packets on the machine where HRI Agent is running as a standalone (SmartSDR on another machine)?

I can see no reason why the HRI Agent machine would be getting this VITA 49 protocol data. There is something in the radio firmware or SmartSDR that is causing the duplicate stream to be initiated to the HRI Agent (wherever it is) for no apparent reason.

I think I've proven here that it is not HRI Agent that is the culprit.

Fun fun...

Danny
K2DMS

K5CG

Additional Info: CAT and DAX are not running on either machine. 

Mark_WS7M

Danny,

This could be that the HRI Agent is not unsubscribing to the slices.  I'll check into that.

K5CG

Next I am going to capture the TCP packets for HRI Agent and compare them based on the startup sequence (HRI Agent first vs. SmartSDR first) and see if there are any differences.

Danny

K5CG

When SmartSDR is not running there are some packets sent from the radio to the HRI Agent indicated as H1 protocol instead of TCP protocol when SmartSDR is running and 2 packets with the slice information is also sent from the radio to HRI Agent when SmartSDR is running which are replaced by 1 - H1 packet when it is not. Otherwise the data sent from HRI Agent appears identical.

K5CG

This behavior doesn't appear to be occurring in v1.6.

James - K4JK

I upgraded to 1.6.17 and am still having this problem with the CMD micro program. The network stats show about 50% lost packets...

I do seem to have found a workaround though...  If I start the CMD Micro program first, then SmartSDR, then DXLabs, the network stats seem to be happy.

K5CG

Yes, me too. Although I reported that it wasn't happening with one desktop, it did with my laptop so I think there is still something to it. I need to capture some more data.

K5CG

The problem remains with v1.6.17.

I start SmartSDR, I see VITA 49 protocol packets coming from the radio to the SmarSDR app. I start HRI Agent, I see duplicate VITA 49 protocol packets. SmartSDR reports it as dropped with bad health.

HRI Agent connects at packet 6932, duplicate data can be seen starting at packet 6944. Stream ID 700 in both with different VRT headers in each copy. Wireshark capture file here

https://www.dropbox.com/s/5czy561p7eg2t6z/2015-01-13-VITA49_dupes.pcapng?dl=0

Interestingly, when I was not seeing this condition, there were no VITA 49 packets when SmartSDR connected, and only one set when HRI Agent connected. Observation only.

Eric-KE5DTO

As best as I can tell from the wireshark, the HRI Agent is not setting it's own port for VITA UDP traffic.  As such, it is likely trying to share the default port (4991) with the other clients.  I thought we had our port initialization set to keep this from happening, but this may not be something we can control (sharing of the open port we are using).  The way to solve this is to have each client try to open the UDP port for listening and then bump up to another port if it is already in use.  Then this gets reported to the radio in a "client udpport <portnumber>" command.  This should resolve having to start the clients in a particular order to make them work.

Mark_WS7M

Hi Eric,

I wrote the HRI Agent and I was not aware that when subscribing to slice only that any UDP data would be sent beyond the radio discovery data.  I don't recall seeing mention of this in the Wiki either.

I can try a mod to open the UDP port and see if it helps.

Dan -- KC4GO

I run HRI almost 100% of the time radio is on... I have checked for dropped packets after these posts started... None here.  Although the radio and operating computer are on the same switch and the system running HRI is connected directly to the home router. 
I can assist in testing if you wish.
Dan

K5CG

Hi Dan,

The capture file I created was on a Win7 X64 HP laptop running 1.6.17 (also did it with 1.5) connected over Wireless-N AP to a Procurve 2824 switch connected to the Flex 6300 (pretty simple network). It never occurs when I start HRI Agent first, only if I start SmartSDR first. There are also reports of this happening with the CMD Micro controller app in the same fashion.

Although I hadn't initially been able to trigger this condition on my Win7 Desktop (wired to the Procurve switch),  I'm going to re-install everything again and try once more. Also with the HP Laptop directly connected.

Eric-KE5DTO

This is a good question -- what actually generates UDP data?

We send out meter, DAX, Panadapter, Waterfall via UDP.  When you do a "sub slice all", I believe this will generate meter UDP traffic associated with that Slice.

After talking with the engineers here, I think I understand what is going on here.  When SmartSDR is started up first, it will claim the 4991 UDP port.  When other clients start up but do not set their UDP port with the command mentioned above, they will use the default 4991 port as well.  Since this will generate traffic on the same port, this confuses the packet statistics as we have packet counts and expect them to be sequential.  With the radio sending Slice meter packets to "both" clients (really, to the same port), the SmartSDR client is going to show heavy loss.

I would bet that those that aren't seeing this problem probably start CAT and/or DAX and leave them running so one of those applications is using 4991.  They are having the same problem, there just is not visibility into the issue in those applications.

The solution here is likely to prevent UDP output from the radio until a UDP port is set for each client using the "client udpport <portnumber>" command.  In the interim, just specifying a unique UDP port should also work to keep SmartSDR from giving false positive packet loss stats.

K5CG

Even if apps claimed a different UDP port wouldn't the data still be duplicated on the subnet? That doubles the utilization for the same information. Wouldn't a multicast protocol be better at distributing just a single copy of the data stream? Just thinking out loud (in the dark).

Danny