Maestro performance problem

Dave K5OZ

Mike,

If y’all would hurry up and get my Maestro order delivered I could check it out on my system....LOL. By the way, thanks for the insightful explanation above about what could be happening and I do know that one of your Maestro part suppliers is having problems with COVID-19.

Anxiously awaiting my unit.

73,

Dave

K5OZ

Ignacy

I spent lots of time trying to figure out how to operate the Flex without excessive the latency. By latency I mean delay in any operation, audio, tuning, key presses. Excessive means 0.5 second or longer. Below is the summary.

Ignacy NO9E

Many users of Maestro and users of Flex via PCs experience excessive latencies, usually > 0.7 s. OK with digital modes, irritating in casual CW and SSB, debilitating in contesting. Some don't experience long latencies, just a regular latency (~ 0.2 s). 

Lots of ideas how to fix long latencies that fill weeks to try, with none of them working. Based on observations, the problem is with a specific driver/ process in Windows. Which one I don't know. Here is the rationale and an idea to troubleshoot. 

Long latency by Maestro, either connected directly or WiFi. Means network is a not a source of problem.

With W10, long latency with new I7/16Gb, small latency with old I5/16 Gb. Means processor speed is not an issue. Means Windows in general is not an issue, just its detail. Means DPR (deferred procedure call) timing is not a problem. 

Latencies above do not change with connection by Ethernet or WiFi, again meaning that the network is not a problem. 

Latency is present in sound and in button response. So if a sound is delayed by 0.5 second, response to changing BW from 2.1 to 6 kHz is delayed by more less the same. Means that the latency of the sound card (or sound driver) does not cause the issue. 

Small latency with DogSDR under Mac. Also reported small latency with SmartSDR for Mac. Means the procedure that causes a problem with Windows is not a problem with Mac. 

One way to troubleshoot is to find two PCs where one shows an issue and another one does not. If one PC shows long latency and another does not, one way to debug is to list all the processes under both systems, and examine the differences. May not help if the problem is in parameters, not processes. Since Maestro is Windows 8, it means the problem is associated with a process already present in Windows 8.

DL4RCE

Hi Ignacy, thanks for the info. Just to add on this, I updated the Maestro tablet with latest drivers since any updates are disabled by default. It seemed to be better but in the end same performance issues. Nevertheless load on the Maestro tablet is quite high, maybe this is one part of the problem.

Volker

Ignacy

How did you update the tablet?

If the system information in Maestro is on a SD, perhaps one can copy SD from Maestro that does not show long latency....

My old Dell runs SmartSDR nicely but fell short of CW Skimmer. Sometimes was able to run 3 instances but most of the time even 1 is unstable. So I bought new PC (NUC) with current I7 processor. Imagine my surprise that this PC showed > 0.5 s latency. Despite being 95% idle and having 1 ms ping.

Ignacy, NO9E

DL4RCE

Hi Ignacy, there is an exploit which I can/will not share here. I was able to access the OS.

Also have a NUCi7 which runs pretty well, but no direct comparison.

Regards

Volker

Mike-VA3MW

Tim, Ken, Dan and I were on a call this morning about latency and they all reminded me that there is a LOT of latency in the PC sound card meaning that from the time the signal gets to the radio, the PC (not SmartSDR) takes time moving the data from the LAN connector to the Speaker port.

I did some research on this (just getting started) and if you start looking into the audio professionals who record live shows, they are challenged with it as well. They do actual sound card Windows tuning. I don't have much more to offer on this, but I am starting to look into it.

Ignacy

Check whether you have delays changing antennas, bandwidth or tuning. If you do, the problem is not with audio drivers.

I spent days focusing on solving audio delays with recommendations by "audio professionals". Nothing gained except my wife was ready to divorce...

Ignacy NO9E

Dan-N7HQ

@Ignacy

The Maestro does not use an SD card. It has an integral SSD which is not field replaceable.

@DL4RCE

Exploits of any kind on our products void the warranty, and our service department policy is to not repair products where an exploit has been used.

As Mike mentioned, the Maestro performance issue does not occur in every Maestro, it appears to be environmental and is not among the "usual suspects". Our engineering team is working on the problem and adding additional instrumentation to help isolate a root cause.

73,

Dan, N7HQ

Carl

I just want to point out that the topic of of the original post was "Maestro Performance Problems." The discussion on router induced latency is interesting as is the idea of sound card latency, however, I'd like to reiterate that the tuning latency and erratic UI behaviors that I experienced on the Maestro, were easily reproduced with a direct wire connection between the Maestro and the 6400.

I reproduced this behavior countless times on two different Maestros and two different 6400s. (in every combination) I also ruled out other variables by testing with different cables, different power supplies, and almost every version of software from 2.6 to 3.1. I tested with "reboot" sequences and complete power cycles (cold boot w/ power supply disconnect) This behavior returned consistently every time, with the only difference being how long it took to re-occur.

I recorded the time it took to degrade with each test, and saw times from 6 Minutes to 1.5 hours, but the problem ALWAYS Returned. I worked on this project for weeks to help the FLEX team find a root cause or a solution to this issue, because I really liked the Maestro when it it was working. Sadly I could not find a configuration (wired or wireless) that would run reliably for much more than an hour.

The worst offending Maestro/6400 pair was sent back to FLEX right around the beginning of the Covid Outbreak. I hope they are looking carefully at these units to dig deeper into the root cause. I believe they will ultimately find issues with the network interface or drivers on the Maestro itself. I was unable to troubleshoot the embedded Windows tablet because it's all locked down.

You can see the behavior I was experiencing in this video below:

https://youtu.be/elb6prJfYU0

In my experience the Maestro performance issues illustrated in the video are still unresolved. However, my SmartSDR on a Windows PC continues to work well even over my wireless LAN.

-Carl

W1NDS

Mike-N1VE

This morning I encountered a very frustrating problem with the Maestro. I had been transmitting at 30 watts, so there wasn't much if any stray RF in the shack. Without warning, most of the display went black, leaving only the meters, including those for A and B sections on the display. No sound and no receive information on the display. I rebooted the Maestro and nothing improved after rebooting. I rebooted again and again the display was mostly blank and there was no sound or visible signals. After a minute or two, the Maestro went back to normal.

DL4RCE

@Dan-N7HQ this is understood and I do not propose this as a solution nor do I want to share any further information on that.

I do experience similar performance drops as @Carl but not that hard. Nevertheless contesting is not possible. Maybe it is related with an upgrade of software since I use it for 2+ years now. Maybe related to (old) profiles?

I did have a ticket and sent the unit to Norbert which did not experience the problem which made me worry a little.

@Dan-N7HQ it is good that you are taking a close look, maybe there will be a fix, much interested in that.

Thanks and regards

Volker

Mike-VA3MW

@Carl I talked to Dan about this and unfortunately they were not able to recreate your problem in the lab which is unfortunate.

Also, I know that they take these types of experiences pretty seriously and they spend lots of time trying to recreate problems like yours.

They want to make it fail since the data they find can be used to future work and to prevent others from experiencing it. This is why we always ask so many detailed questions so we can recreate it. I have seen them 'cook' customers radios for an entire weekend or more.

If they can get it to fail they have a better chance of fixing.

Mike

Ignacy

If the Flex staff cannot find problems with the Maestro that showed a problem elsewhere, the problem MUST be an interaction of specific Flex radio with Maestro. I am wondering whether the reported problems occur only with 6400 and 6600, or do they occur with all 6000 series radios. If they occur with 6400/6600 only, this is an important information.

Perhaps a good way to find the problem is to send both the radio and the Maestro to Flex HQ.

Ignacy, NO9E

DL4RCE

@Ignacy that is a good question if this ever happened with a 6700 6500 6300?! maybe someone knows.

Volker

N8AUM

Isn't there a way to access some error log that can than just be sent to FRS and have them look at it ? My Maestro will run for couple days SOMETIMES just monitoring but usually after couple hours will just go "blank" or totally drop it's connectivity requiring rebooting. Made no difference which version of firmware and currently using the .10 version.

Vidas N8AUM

Ignacy

I created a poll on groups.io about who experiences long delays. Few answers but picture emerging.

Nobody using 6300/6500/6700 reported a problem.

Of 6400/6600 users, 50% of Maestro users and 30% of PC users reported the problem.

Assuming that this is real, Flex HQ would not find a problem with Maestro if they used 6300/6500/6700.

From few reports , we know that the Mac version do not show the long delay. This points to a problem being interaction of something specific in the Windows version of SMartSDR and some feature of 6400/6600.

Ignacy NO9E

LZ2BE

Hi Ignacy, I have 6600 and Maestro, v.3.1.12. There is no RX audio delay only if phones are connected to the radio.

All other options like Maestro Audio Out, SSDR PC audio or DAX RXn in Digi modes have more than 0.5 sec delay.

It is easy to replicate.

Open a slice on Maestro - for example 14020kHz CW, set Main Audio Out to Maestro and plug earphones to Maestro phones jack.

Open a slice on SSDR, same freq and plug another earphones to Radio jack.

Find and listen to a station with SSDR and Maestro phones at the same time and you can notice the delay. You can try opposite - Maestro with audio from radio output and SSDR with PC audio, the same effect.

You may see on waterfall signal peaks disappear while you still hear CW.

The same is on RTTY. Signal traces on SSDR waterfall disappear, while on MMTTY waterfall they still exist and delay in my case was 0.7 sec.

73 Boyan, LZ2BE

Ignacy

Boyan,

The weird stuff with Maestro is that some people have long delays and some don't. My Maestro does.

No delay on my 2012 old computer but delay with new 2020 computer. Must be some weird stuff.

It seems that only Flex can fix the issue with Maestro. With PC, we can try a few computers and find out which has minimal delay. Or use Mac for Windows. But then we must use Mac logging/contesting program.

Ignacy, NO9E

AWe

I´m using Maestro and 6400 Combo, latest Hard- and Software. As i said in other threads, it´s very frustrating with the long delay in all windows-based things like the Maestro or SmartSDR for Windows. I can also confirm the lag from Carl´s video, sometime the frequency "freezes" and restarting helps a while.

The most time i use the Flex on a Mac system or Ipad, the Maestro and SmartSDR for Windows and all Apps for Windows are for me unusable because of the lags and high delay.

73 Al

Ignacy

Remote Ham Radio (RHR) updated most of their stations to Flex. I logged to Flexes 6300/6500/6700, and the lag was small, barely higher than K3. They also have one stations with 6600 but I don't have access to it.

Will try my Maestro with their stations over the weekend.

Ignacy, NO9E

Paul_M0AIA

Hi Flex Community,

I too have suffered with Maestro slow performance from new. I purchased my Maestro back in July 2019 and it was running the new software version 3 at the time when I un-boxed it.

I felt the benefit of the Maestro right from first power on but was often very disappointed with the slow laggy performance only a few minutes after it had been powered on that I could never quite put my finger on the root cause of the performance degrading. Because of this my Maestro over the last 14 months has been used around once a month for a day or so and then put back on the shelf sadly. I even advertised it on the Flex Community sadly as I preferred to use my Laptop over the Maestro.

Recently I have put more effort into testing differed software config's and also power voltage/internal battery pack and also network connections wired/Wi-Fi.

To cut a long story very short I believe on my Maestro unit I have fixed the slow performance issue that a lot of us in the Flex community have experienced.

To give you a understanding of how I operate and the workload the Maestro has to deal with, which I believe has had a direct impact on performance. I am a band hopper and also use two slices from time to time so one can be used for digital. I'm not the type of operator who powers on and calls CQ on a single frequency or runs a Net and then powers off after their operating session. I'm constantly on the move seeking DX right across the HF bands 80 through to 6

So here is the work around that I have implemented that has proved 100% successfully after days of hard use over the last few days with the propagation improvements where all seeing on 17m & 15m with contacts all over Asia and Oceania using my Maestro and finally enjoying the benifits the Maestro brings to the operating and Yes I'm loving my Maestro now :-)

I have connected my Maestro via an ethernet cable that I have modified to use 4 wires only. Todays ethernet cables have 8 wires that are used to achieve 1000Mbps (the ethernet cable delivered with your Maestro from new has 8 wires). What I have done is removed 4 of the wires forcing the network adapter in the Maestro to establish a link speed of 100Mbs as four wires cannot run at 1000Mbs. The pin wiring of my modified ethernet RJ45 is Pin 1,2,3,6

Here is a short description of the pairs used by the 10Base-T (10 Mbps) and 100Base-TX (100 Mbps) ethernet are Pins 1-2 and 3-6. The 1000Base-T (1 Gbps) standard requires all four pairs (wired as Pin 1-2, 3-6, 4-5, 7-8).

My home shack network setup is:

. Netgear Gigabit 5-Port Ethernet GS105v5 (metal box and is RF quite)

. Netgear Switch linked to Broadband router for internet access - connected at 1000Mbs

. Flex 6600 - connected at 1000Mbs

. Dell Laptop - connected at 100Mbs (interesting as this PC running Smart SDR has been faultless)

. Maestro - now connected at 100Mbs ((RJ45 4 pins only) (performance not degraded)) performance is as you would expect/require

. Flex Software v.3.1.12

Now here is the disclaimer:

I'm not telling you to do this and if you want to test this specific ethernet pin wiring then its at your own risk and you cannot hold me responsible for any damage to your equipment.

That leaves me to wish you all the best of luck and I hope you gain the same performance improvements that I have seen that now make operating the Flex 6000 series radio with a Maestro a complete joy.

Kind Regards

Paul M0AIA

Ignacy

Boyan wrote that PCs show delays only with PC audio. I checked my NIC I7 and it was the case. So it seems that routing audio through PC in Windows 10 causes delays in everything else. Not on Mac.

Wonder whether the same happens in Maestro. If one redirects audio to Flex, does it still show a delay?

I am away from my remote station and will test it myself later.

Following Pauls' s idea of a 100 Mb cable for Maestro, I will check it as well.

Ignacy, NO9E

LZ2BE

Hi Paul,

according to this picture

CAT cables are already wired as you suggest.

I don't know if other wires than 1-2 and 3-6 are used in Maestro.

73 Boyan, LZ2BE