dropped packets when client connected from same machine

  • 1
  • Problem
  • Updated 3 years ago
  • Solved
  • (Edited)
Flex 6300
SmartSDR 1.5.1

Running SmartSDR from a laptop directly wired (or through a GigE switch) I see Network quality is good and ping time < 1ms and ZERO dropped packets for long periods of time. Very solid network link.

If I run another client app on the same laptop. as soon as the additional client app connects I see packet loss and ORANGE/RED status on the network graph in SmartSDR. Ping time is sill < 1ms.

If I run the same application on another laptop (a peer on the same network) there are no lost packets.

This happens with the Behringer CMD Micro DJ panel application William Hemmingsen created and the HRI Agent application that Mark WS7M has started.

I've disabled Windows Firewall completely and it made no difference.

Dissbled WiFi, the only NIC on the laptop is the wired NIC to the radio/switch. No difference.

SmartSDR may be reporting the packets sent to the add-on application as lost but I don't think it should do that.

Update: I also moved off the 169.254.0.0/16 default network and placed the laptop and the 6300 onto a router where they ran on the 192.169.1.0/24 subnet (DHCP from the router) and it behaves exactly the same.

Has anybody else experienced this?

Danny
K2DMS
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes

Posted 4 years ago

  • 1
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Here is a video capture of the behavior. This was taken using Mark's new HRI Agent application but it does exactly the same thing with the Berhinger CMD Micro DJ app. It isn't anything specific to one app or another, but all of them.

https://www.dropbox.com/s/if3xt87nts2vabu/2016-01-02%20at%2000-14-42.mp4?dl=0

Danny
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
I have seen this too with William's CMD Micro App. I can provide wireshark captures if anyone from Flex would like to see them.
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
Actually now that I look at it closer, performance doesn't really seem to suffer. It looks like the client just reports the packets as dropped, like Danny noted.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
I neglected to mention that my OS is WIndows 10 Home X64. The NIC is a Realtek PCIe GigE controller (in an HP ENVY 17T laptop).
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
Odd behavior for me now. This morning when I woke up I saw Danny's last post and it gave me the idea to try the other NIC in my Machine. (I am on Win8.1 x64) as I was using a Realtek before and the other is Intel.

Tried the other (Intel) NIC and everything appeared to be working as normal, other than a little lag for a few seconds at first. Diagnostics indicated no dropped packets.

So I shut down Smart SDR and the CMD Micro program and switched back to the Realtek, and it appears to be working fine again. No dropped packets or anything. Very strange. So it is intermittent, at least for me.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
I can reproduce this at will on my HP Envy 17t (Windows 10 Home X64) but it does not happen on my HP Elitebook with Windows 7 with the same radio and same version of HRI Agent. I'll try a USB NIC on the HP Envy and try to separate the NIC from Windows 10.
(Edited)
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Update: The same thing happens with the built in WiFi NIC so it isn't isolated to the RealTek GigE wired NIC. I also uninstalled VMware Workstation 11 thinking perhaps the extra internal virtual networks were interfering with the native networking but that made no difference.

Next: I'll install the SmartSDR and HRI Agent on my wife'd Dell laptop (windows 8.1). I suspect that will work fine so then I'll install Windows 8.1 on the HP Envy 17t and see how it goes.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Update: I was surprised to find that the dropped packets condition also happens on my wife's Dell 17R (Core i5 - Windows 8.1 X64) using the WiFi connection.

I will install Windows 7 on my HP Envy 17t and see what happens there. I think it's also time I took some wireshark captures to grab some traffic.

Danny
K2DMS
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Update: I have found some information that may be useful.

Windows 7 X64 Ultimate on a Dell Core i5 desktop.

If I start HRI Agent (or the agent for the Behringer CMD Micro DJ) first, and start SmartSDR last, then the issue does not occur. If I start SmartSDR first, followed by either program, then I see dropped packet indication.

"Dropped packet" isn't really the correct term but this is how SmartSDR sees it apparently as it de-duplicates the data. Looking at Wireshark when this happens I see duplicate stream data in the VITA 49 protocol packets (different  VRT Header and Sequence numbers) but identical stream ID and data payloads. It appears as though stream data is being sent from the radio to what it thinks are 2 different clients but at the same IP address. This is purely conjecture on my part of course because I don't know anything about this traffic other than I can confirm that the duplicate packets are not there when dropped packets are not indicated.

Wireshark capture file. 

https://www.dropbox.com/s/jd1weholzmsyr3i/SMartSDR%20then%20HRI%20Agent%20start.pcapng?dl=0

The link above is a capture from Wireshark 2.0.1 with filter as "host 192.168.1.242 and not broadcast and not arp and not icmp". 192.168.1.242 is the IP of the Flex 6300 on my local subnet.

This captures the initial connection from SmartSDR to the radio. You can clearly see the UDP fragmented packets followed by VITA 49 protocol packets and in the VITA 49 packets there are two stream numbers, usually Stream ID: 0x40000000 and Stream ID: 0x00000700, sometimes Stream ID: 0x42000000.

I started up HRI Agent and you can see it connect at packet 771.

Immediately as of packet number 782 you can see that its data payload is identical to the packet before 781. Different VRT Headers but the exact same Stream ID and Data fields.

Something is not right here. We shouldn't have to start 3rd party apps before SmartSDR, it shouldn't matter what order you start them in. The radio shouldn't be sending duplicated VITA 49 data packets only to have SmartSDR claim them as dropped (IMHO).

I believe this will require some research on the part of smarter people than myself.

Help?

Danny
K2DMS
(Edited)
Photo of Mark - WS7M

Mark - WS7M

  • 1335 Posts
  • 501 Reply Likes
Hi Danny,

Good detective work.  I will try the same sequences myself and see what I see.

mark
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Mark,

I've discovered something else that is probably significant for root cause analysis.

If I start HRI Agent first, then SmartSDR second... I can close and restart HRI Agent as many times as desired and it doesn't trigger the duplicate(dropped) packet issue.

However, if I start SMartSDR first, every time I open HRI Agent I get the duplicate(dropped) packet issue.

So it only happens when SmartSDR is started first and then HRI Agent afterward.

What does that tell us?

Danny
Photo of Mark - WS7M

Mark - WS7M

  • 1335 Posts
  • 501 Reply Likes
Danny,

I'm really not sure what it tells us.

For me the unknown is what Flex is doing as part of their packet counting and roundtrip mechanism.  

What I can tell you is what HRI Agent does:

When first run it opens a UDP socket listing for any address for UDP datagrams on for 4992.

By nature UDP transmission is stateless meaning there is no record of connect, disconnect etc.  It is a way of broadcasting data.

HRI Agent when the UDP socket is opened receives the broadcast datagrams and decodes them.  The datagrams are kind of large.  They consist of like 28 bytes of header plus a bunch of space delimited strings like  model=<model> serial=<serial> etc.  HRI Agent pretty much ignores the 28 byte header at the time being and decodes the strings.

It then looks in a list it keeps to see if the radio is a new radio.  If so it adds it to the list and refreshes the listbox you see on screen.

If you leave HRI agent in this state this is literally all it is doing.  So the first thing I'd like to know is in your testing where you have run agent first or SmartSDR first what state have you left the agent in.  If you left it in the unconnected state and SmartSDR is still reporting lost packets then I venture to say there is a problem with their packet counting algorithm because UDP packets sort of really should not count.  IE they are just sent off into the void.

If you take HRI Agent to the next level and connect it to your radio with a double click it then opens a TCP socket to the radio and almost immediately issues the API command to subscribe to all slices.   Slices are the only thing right now that HRI Agent subscribes to.

This means HRI Agent receives radio information and information about all slices in one big blast upon connection.  I could see this causing a minor hit on the packet loss but not much.

Beyond that HRI Agent is totally in a listening mode.  IE it is not commanding the radio in any way.  It is simply receiving and decoding the various data sent out by the slice subscription command.

So if this causes the packet loss to go up then I still must suggest there is a problem in their algorithm. 

I have not had a chance to look at your wire shark capture.  I will tomorrow.  But I don't think that HRI Agent is doing anything illegal to cause the packet loss.  The API clearly states that external clients can subscribe to slices, meters, pan adapters, audio, etc.  HRI Agent is only subscribing to slices and nothing else.

So I will help any way I can and my first task is going to be to use my laptop and setup the scenarios you mentioned and see if I can duplicate packet loss.  Before you had the before/after cases I did try running my laptop on Win10 and connection both HRI Agent and SmartSDR and I saw only 1 dropped packet but I cannot remember the order it was done in.  So let me try to reproduce your results.

Mark
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Mark,

There are no duplicated packets (supposedly dropped) until the double click on the radio entry in the HRI Agent dialog (when SmartSDR is already running).

I can certainly live with starting HRI Agent first as a SOP, but I think there is some value in understanding what is going on here so that it isn't a requirement.

Thanks for your help.

Danny
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 340 Posts
  • 70 Reply Likes
I have a Desktop W10, i7, Realtek  PCI GBE  and a Dell XPS-13 W10, i7, Dell GBE on the same network I tried them both with no dropped packets. I also tried running HRI on my server "Odd Job" Realtek PCI GBE and could not reproduce the network issue.  I also have a switch between the radio and computers.  Between the server and the radio is the switch and the router. 
Wire Shark produced no abnormal packets as well. 
Dan--- KC4GO

Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Thanks for trying Dan.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Digging a little more... I had the idea to use Wireshark to capture data on 2 machines, one with SmartSDR running and another with HRI Agent running to see what data was streaming.

If I start SmartSDR first on one machine, and then HRI Agent on the other, I see the VITA 49 protocol using Stream ID 700 on both machines in Wireshark.

The stream ID 700 data doesn't stop even if the disconnect button is pressed on HRI Agent, not until the application is closed does it stop.

This would appear to be why the duplicate/dropped packets are not reported, when the HRI Agent is on a different machine. The stream is present but not counted by SmartSDR.

I captured a change to a slice and I saw TCP packets coming from the radio to the HRI Agent app which included the new frequency in the data field (intermixed with the VITA 49 data).

The other test, to confirm, I started Wireshark on the laptop, started HRI Agent and capture the TCP connect session. I started SmartSDR on the desktop and I see no VITA 49 packets at all on the Laptop. This is also what I was seeing when SmartSDR and HRI Agent were on the same machine if I started HRI Agent first (no duplicate packets).

Can somebody do a similar test and see if there are VITA 49 packets on the machine where HRI Agent is running as a standalone (SmartSDR on another machine)?

I can see no reason why the HRI Agent machine would be getting this VITA 49 protocol data. There is something in the radio firmware or SmartSDR that is causing the duplicate stream to be initiated to the HRI Agent (wherever it is) for no apparent reason.

I think I've proven here that it is not HRI Agent that is the culprit.

Fun fun...

Danny
K2DMS
(Edited)
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Additional Info: CAT and DAX are not running on either machine. 
Photo of Mark - WS7M

Mark - WS7M

  • 1335 Posts
  • 501 Reply Likes
Danny,

This could be that the HRI Agent is not unsubscribing to the slices.  I'll check into that.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Next I am going to capture the TCP packets for HRI Agent and compare them based on the startup sequence (HRI Agent first vs. SmartSDR first) and see if there are any differences.

Danny
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
When SmartSDR is not running there are some packets sent from the radio to the HRI Agent indicated as H1 protocol instead of TCP protocol when SmartSDR is running and 2 packets with the slice information is also sent from the radio to HRI Agent when SmartSDR is running which are replaced by 1 - H1 packet when it is not. Otherwise the data sent from HRI Agent appears identical.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Official Response
This behavior doesn't appear to be occurring in v1.6.
Photo of James - K4JK

James - K4JK

  • 14 Posts
  • 1 Reply Like
I upgraded to 1.6.17 and am still having this problem with the CMD micro program. The network stats show about 50% lost packets...

I do seem to have found a workaround though...  If I start the CMD Micro program first, then SmartSDR, then DXLabs, the network stats seem to be happy.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Yes, me too. Although I reported that it wasn't happening with one desktop, it did with my laptop so I think there is still something to it. I need to capture some more data.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
The problem remains with v1.6.17.

I start SmartSDR, I see VITA 49 protocol packets coming from the radio to the SmarSDR app. I start HRI Agent, I see duplicate VITA 49 protocol packets. SmartSDR reports it as dropped with bad health.

HRI Agent connects at packet 6932, duplicate data can be seen starting at packet 6944. Stream ID 700 in both with different VRT headers in each copy. Wireshark capture file here

https://www.dropbox.com/s/5czy561p7eg2t6z/2015-01-13-VITA49_dupes.pcapng?dl=0

Interestingly, when I was not seeing this condition, there were no VITA 49 packets when SmartSDR connected, and only one set when HRI Agent connected. Observation only.
(Edited)
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
As best as I can tell from the wireshark, the HRI Agent is not setting it's own port for VITA UDP traffic.  As such, it is likely trying to share the default port (4991) with the other clients.  I thought we had our port initialization set to keep this from happening, but this may not be something we can control (sharing of the open port we are using).  The way to solve this is to have each client try to open the UDP port for listening and then bump up to another port if it is already in use.  Then this gets reported to the radio in a "client udpport <portnumber>" command.  This should resolve having to start the clients in a particular order to make them work.
Photo of Mark - WS7M

Mark - WS7M

  • 1335 Posts
  • 501 Reply Likes
Hi Eric,

I wrote the HRI Agent and I was not aware that when subscribing to slice only that any UDP data would be sent beyond the radio discovery data.  I don't recall seeing mention of this in the Wiki either.

I can try a mod to open the UDP port and see if it helps.
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
This is a good question -- what actually generates UDP data?

We send out meter, DAX, Panadapter, Waterfall via UDP.  When you do a "sub slice all", I believe this will generate meter UDP traffic associated with that Slice.

After talking with the engineers here, I think I understand what is going on here.  When SmartSDR is started up first, it will claim the 4991 UDP port.  When other clients start up but do not set their UDP port with the command mentioned above, they will use the default 4991 port as well.  Since this will generate traffic on the same port, this confuses the packet statistics as we have packet counts and expect them to be sequential.  With the radio sending Slice meter packets to "both" clients (really, to the same port), the SmartSDR client is going to show heavy loss.

I would bet that those that aren't seeing this problem probably start CAT and/or DAX and leave them running so one of those applications is using 4991.  They are having the same problem, there just is not visibility into the issue in those applications.

The solution here is likely to prevent UDP output from the radio until a UDP port is set for each client using the "client udpport <portnumber>" command.  In the interim, just specifying a unique UDP port should also work to keep SmartSDR from giving false positive packet loss stats.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Even if apps claimed a different UDP port wouldn't the data still be duplicated on the subnet? That doubles the utilization for the same information. Wouldn't a multicast protocol be better at distributing just a single copy of the data stream? Just thinking out loud (in the dark).

Danny 
Photo of Steve - N5AC

Steve - N5AC, VP Engineering / CTO

  • 1057 Posts
  • 1097 Reply Likes
Yes, but multicast has it's own problems -- specifically that it puts a load on every computer on the subnet.  Generally, for high-volume data applications multicast goes hand-in-hand with VLAN (or segmented networks).  For now, we've assumed that this solution is too much trouble for the typical amateur radio enthusiast so we have not implemented it.  Incidentally, our government products do multicast over VLAN.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Very Good Steve, I'm just grasping at straws here. I was just imagining that with all the 3rd party apps that hams are connecting (DDUtil, HRI Agent, DVKs, S-Meters, CMD Micro, DX Lab Suite, FRStack)... The list is getting long. "If" every app opens a new UDP stream... is there a tipping point where multicast makes sense? Don't feel you need to answer, I'm just thinking out loud here again and learning as I go.

Danny
Photo of Steve - N5AC

Steve - N5AC, VP Engineering / CTO

  • 1057 Posts
  • 1097 Reply Likes
Yes, there's certainly a place for it.  We could also build an aggregation layer for the PC where it keeps track of everything that PC has subscribed to, asks for it once, and distributes to the clients that asked for it.  Lots of things we could do.  Really the panadapter, waterfall and DAX are the big bandwidth users and they are not typically shared today.  Metering is just not a big enough deal to worry a lot about, but someday we may look into this.
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
Note that while there is duplicate data being sent in these situations, the actual "failure" here (supposed packet loss) would be fixed by not duplicating the data on the same port.
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 340 Posts
  • 70 Reply Likes
I run HRI almost 100% of the time radio is on... I have checked for dropped packets after these posts started... None here.  Although the radio and operating computer are on the same switch and the system running HRI is connected directly to the home router. 
I can assist in testing if you wish.
Dan
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Hi Dan,

The capture file I created was on a Win7 X64 HP laptop running 1.6.17 (also did it with 1.5) connected over Wireless-N AP to a Procurve 2824 switch connected to the Flex 6300 (pretty simple network). It never occurs when I start HRI Agent first, only if I start SmartSDR first. There are also reports of this happening with the CMD Micro controller app in the same fashion.

Although I hadn't initially been able to trigger this condition on my Win7 Desktop (wired to the Procurve switch),  I'm going to re-install everything again and try once more. Also with the HP Laptop directly connected.
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 340 Posts
  • 70 Reply Likes
Ah ha, I use a gigabit net wired. And my wireless is AC1900. I have a spare router (older N  Linksys E1000 300 Mbps I can test with using my laptop 
@Mark and Eric 
           Let me know if you want me to configure this network and test. 
Dan -- KC4GO
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
Dan -- it would be interesting if you could run a test for me.  Close all the Flex related clients and open them in this order:
1. SmartSDR
2. CAT (optional)
3. DAX (optional)
4. HRI

Then confirm whether you get the reported packet loss.  I suspect you will see it when starting things in this order.
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 340 Posts
  • 70 Reply Likes


After less than a min. Now 9197 ..
Dan -- KC4GO
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
This confirms the theory.  Thanks Dan.
Photo of Mark - WS7M

Mark - WS7M

  • 1335 Posts
  • 501 Reply Likes
I just tried that order.  0 dropped packets and this is with HRI still not connecting to UDP.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
I did the same test sequence, I opened HRI Agent at about 3300 total packets logged (none dropped). By the time I got to 6600 logged (doubled) I had 2130 dropped.
Photo of Mark - WS7M

Mark - WS7M

  • 1335 Posts
  • 501 Reply Likes
So I guess I'm confused.  Should I modify HRI to connect to or specify then connect to a UDP port to solve this issue from my side?
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
If you aren't going to use the UDP data, there is no reason to have a UDP listener.  But I would recommend moving the port the radio uses off of the default 4991 using the client udpport command so as not to confuse the SmartSDR packet count.
Photo of Steve - N5AC

Steve - N5AC, VP Engineering / CTO

  • 1057 Posts
  • 1097 Reply Likes
Mark, you can look in our code, but as I recall we open ports starting with 4991 NOT in a shared mode until we get one that is ours exclusively.  Then we tell the radio it can send the data to that port.  (this is from memory, but I think it's correct)
Photo of Mark - WS7M

Mark - WS7M

  • 1335 Posts
  • 501 Reply Likes
Hi Steve.  I built a test version that does what Eric recommended.  After connecting I sent out client udpport 32769.  This by Wireshare shows that the radio begins to send UDP packets to 32769.  However I never open that port up and listen since I don't need the UDP data for this application.

In a couple tests this fixed the dropped packets issue.  I'm planning to turn this into a release today for those using it.

The issue is a strange one because when Danny first reported it I started to snoop around and never could see dropped packets on my computer.  Didn't matter the order or what I did.  I might see 10 or so but not continuous and never did my packet latency get bad.

Anyway my question to Eric was simply that if I'm subscribing to slices only why do I even get UDP data?  It might just be a misunderstanding on my part but I'm not trying to get audio or pan/waterfall.  I just want to know what the slice is doing.
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
The short answer is that you probably shouldn't be getting UDP packets from just subscribing to the Slice data.  Definitely should not be receiving UDP packets until you specify a UDP port.

The data you are receiving is meter packets related to the Slices.  This should probably be tied to a "sub meter all" or "sub meter slc all" or something like this though.  I entered this as #2888.

Thanks for helping us sort through this.
Photo of Danny K5CG

Danny K5CG

  • 356 Posts
  • 60 Reply Likes
Please check #2849, if it is related or a duplicate. Steve opened it from earlier info.
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
Thanks for the heads up.  I related them and closed #2849 as I believe we have nailed the root cause and a work around for now.
Photo of John G3WGV

John G3WGV

  • 199 Posts
  • 37 Reply Likes
Hi Eric,
Sorry to dredge this one back up again but did #2888 get resolved? I am getting the same thing running WinSSDR plus my logging client, which is connected to the TCP stream on 4992. The logging client doesn't use streaming UDP data, so I don't set the port or listen for it. The only thing I'm subscribing to on the TCP port is "slice all". I'm seeing 56% dropped packets on the network diagnostic monitor. Using v1.8.4.84.

I'm guessing the dropped packets relates to UDP so I'm wondering if the radio is streaming stuff I haven't asked for? It's as if meter or FFT data are being streamed to my logging client.

Edit: I changed my logging program to set a different streaming UDP port and that completely fixed the problem. So I am guessing that the radio is streaming a pile of stuff even though I've only subscribed to slices.

John.
(Edited)
Photo of Eric - KE5DTO

Eric - KE5DTO, Official Rep

  • 887 Posts
  • 338 Reply Likes
John,

No problem.  I show that #2888 is still open, but it is scheduled to be worked on.
Photo of John G3WGV

John G3WGV

  • 199 Posts
  • 37 Reply Likes
Thanks Eric. I'm starting to get the hang of this API now :)