Maestro LAN radio discovery delayed

Paul - KB5MU

When I power up my Maestro, it says "There are currently no radios available", even though my 6600M is turned on and connected to the LAN. Then if I leave everything alone for a while (typical: half an hour!) the Maestro eventually discovers the 6600M.

The Maestro is on 5GHz WiFi. Other SmartSDR devices in the house (including a MacBook Air, an iPad, and computers running Windows 8.5, 10, and 11) have no such problems, on wifi or on ethernet. A packet trace using Wireshark on the MacBook Air sitting right next to the Maestro and on the same 5GHz WiFi shows that the discovery packets are coming out like clockwork every second on UDP port 4992 addressed to 255.255.255.255.

The 6600M is connected to the LAN via ethernet through two switches. Devices anywhere on the ethernet and those on the wifi all share the same subnet. The router is an AmpliFi Router HD (firmware 3.6.2) with one AmpliFi MeshPoint HD.

The Maestro has a charged battery inside, and is also connected continuously to the AC adapter.

SmartLink is not involved here, this is a local radio. If the Maestro is logged in to SmartLink, it shows the radio available through SmartLink right away (instead of saying "There are currently no radios available"), but still doesn't discover the radio locally right away. When it does eventually discover the local radio, the local entry replaces the SmartLink entry. (Didn't it always show both before? The iPad version does.)

This is new behavior. With the same setup, the Maestro used to reliably discover the radio. Occasionally, it still does, but usually I see the behavior described above. I am not aware of anything significant that has changed in the environment, though the LAN is always changing. The Maestro and 6600M are running 3.2.39; the behavior change did not coincide with the version update.

Any ideas?

KD0RC

Hi Paul, I recently helped a guy with a similar problem. In his case he had a switch plugged into his router, then another switch plugged into the first switch, then the radio and Maestro plugged into the second switch.

To fix his problems, we plugged his 6400 and Maestro into the first switch and got rid of the second switch. It meant some creative cable-routing, but in the end, it worked great.

When you mention two switches, do you mean one plugged into the other like I described above? If so, I think that is your issue. One test to try is to plug the Maestro directly into the radio. It should hook up right away, indicating that the multiple switches are the likely culprit.

Paul - KB5MU

Thanks Len for your insight. I do have two switches cascaded as you describe, and the 6600M is at the end of the chain, but the Maestro isn’t connected to any of the switches. It’s connected over WiFi to the router.

I need to test more different configurations to be sure, but it seems like my problem doesn’t occur at all if the Maestro is connected via Ethernet. It seems to be specific to the Maestro being on WiFi. Running the Maestro on WiFi is an essential use case. This is doubly true since it won’t see the Ethernet when running on batteries.

Even if I find a network configuration that works, I want to understand why this configuration doesn’t work. It’s my understanding that the broadcast UDP discovery packet is the only thing required for the Maestro to list the radio as available. Those packets are available on the WiFi network, so why doesn’t that work? Even weirder, why does it fail to work for hundreds of packets and then suddenly work on (at least) one, identical to all the packets it just ignored?

-Paul

KD0RC

Hi Paul, yep, you have a very similar setup and issue as the one I talked about. I think that the issue is the cascaded switches anywhere along the way. What we noticed in the case of the ham I helped was that the network quality indicator would cycle from green down through red and back again. Sometimes it cycled through in a few minutes and sometimes it was a matter of hours. It gave the impression of an unstable network.

I can't be sure what was happening, but my thought is that something gets out of sync between the two switches over time. It is almost like the clocks are not forced into sync. That is probably not technically correct, but it kind of describes the symptoms. Maybe a network engineer could chime in and describe why cascaded switches don't work properly.

If you have a way to get the 6600 connected to the router with only one switch (or, better yet - directly to the router), I think that you will have it solved.

Mike-VA3MW

In order for the Maestro to see the Radio, it must hear the network broadcast from the radio.

More and more switches are 'green smart' and may delay the network broadcasts to save power. You will have to check the specs on the switches you are using.

The same is true for WiFi as network broadcasts can actually cause for more battery drain on WiFi devices such as phones, so they limit broadcast data. All of these play into the Maestro not being able to hear the radio.

Your test with the LAN cable did confirm that it is related to the network in part.

John KB4DU

If it helps to alleviate the need for creative lan wiring, I use a pair of TP-Link power wiring Ethernet adapters, as suggested by Mike. In my case one is at the router/WAP in the middle of the house to the shack at one end. Works well. When at home they support spots/logging/etc. when remote they handle the outbound data for SSDR on the iPad.

Paul - KB5MU

Mike,

My test with Wireshark on the MacBook Air side-by-side with the Maestro on the same WiFi proves that the network broadcasts are getting through the switches and router just fine, they are not being delayed or filtered out or thinned out or anything. Literally the only part of the network that remains suspect here is the WiFi reception inside the Maestro.

This is confirmed by the fact that other SmartSDR devices on WiFi (the iPad and the MacBook Air) do not exhibit the delayed discovery problem.

Since other aspects of WiFi reception in the Maestro seem fine (as evidenced by 0.00% packet loss rates while connected), that leaves some kind of software issue inside the Maestro that causes it to disregard nearly all broadcast discovery packets coming in over WiFi.

Unless there is more to the discovery broadcast than those UDP packets broadcast to 255.255.255.255 on port 4992?

Len,

I don't know what the "network quality indicator" bar graph is really measuring, especially in a wired network like the one you helped debug. I would have assumed that it shows packet loss rates.

The Maestro displays packet statistics on the Menu->Network screen. Mine has been connected to the radio for many hours at the moment, and it reports that it has dropped 14 out of 6,350,000 packets, or 0.00%. That screen also shows something called "network status", which takes on subjective values like "Excellent". I am guessing that this is the same value as the bar graph displayed in the GUI. The "network status" does vary sometimes, even as the packets continue to flow without any packets dropped. So apparently "network status" is not packet loss rate. What does that leave? Maybe timing jitter, which is to say packets being delayed by varying amounts. Timing jitter would have no effect on broadcast packets.

The extremely low dropped packet count shows that there is no overall problem whatsoever with the network connection. Whatever is going wrong must be specific to the broadcast discovery packets (which cannot be counted on that display) or the processing thereof.

All,

I (temporarily) eliminated the second switch from the network. So now the radio is connected to just one switch which is connected to the router. This does not cure the problem.

Then, I eliminated the first switch. Now the radio is connected directly to the router. This also does not cure the problem. Nor does it change the packets captured by Wireshark over the WiFi.

Wiring up the Maestro to the ethernet does cure the problem, but it's not a solution because I need to be able to use WiFi with the Maestro.

Next theory? Is there something I can do to hard-reset the WiFi subsystem inside the Maestro? Or capture packets received inside the Maestro?

-Paul

KD0RC

Hi Paul, it might be time to submit a helpdesk ticket.

Paul - KB5MU

I did submit a Helpdesk ticket, and this led to the solution.

Helpdesk walked me through resetting the Maestro (key step: force a hard poweroff by holding the power button down 15+ seconds until the screen goes black, repeat until it powers on cleanly), and that didn't lead anywhere. The next step they proposed was to reset everything else on the network (router, modem). Before we got to trying that, though, the Helpdesk posed a number of questions about my WiFi configuration. This led to a solution.

Their idea was that the Maestro was not really connected to the same WiFi router as the radio. I was sure that it was. The Maestro's IP address was in the right subnetwork, and the WiFi password the Maestro was using would only grant access to the right WiFi network. In order to confirm/prove this belief, I logged into the WiFi router and looked at the list of connected devices. Sure enough, the Maestro was listed, at the expected IP address. Theory confirmed ... but ... I also noticed that the Maestro was connected to the WiFi network through a meshpoint. That "should not" matter, but I had not considered it, even though I listed the AmpliFi MeshPoint HD as a component in my WiFi network at the top of this thread. When I captured a packet trace with Wireshark, I did not check to see whether a meshpoint was involved.

Resetting (unplugging and plugging back in) the meshpoint made the problem go away. Even with the Maestro using the meshpoint, it was once again able to discover the radio promptly. The meshpoint must have been in a very peculiar weird state that allowed it to pass unicast traffic normally but not broadcast traffic. We don't know exactly how the eventual successful broadcast packet made it through to the Maestro. Perhaps the meshpoint would pass an occasional broadcast packet. Or perhaps RF fading or interference would occasionally cause the Maestro to stray off of the meshpoint for connectivity, and find a broadcast packet direct from the primary WiFi router.

In summary:

Multiple cascaded Ethernet switches -- not a problem

Switches and routers behaving as designed -- not a problem

WiFi network with meshpoint(s) -- not as simple as without meshpoints. Failure modes can be local to a meshpoint. Reset everything in the network, including meshpoints, to cure anomalous behavior.

-Paul