RX Audio Latency

Steve London

I have been testing SmartSDR 3.1.12 with a remote Flex 6700. The network latency seems acceptable - consistently about 150 msec RTT. However, I am hearing a very long delay in the received audio path. I have been listening to CW signals locally at the same time as through SmartSDR. The audio delay is in the order of 1000 msec. Not good for serious contesting. It's interesting that the waterfall will display the signal several hundred msec before hearing it. So, it seems that most the delay is in the SmartSDR->Sound card path. I am running with a 4 core, fast computer. CPU is running under 10 %. I have changed the CW filters to low latency. Using the motherboard sound card with the standard Windows 10 HD driver.

What else can I try to reduce the latency to acceptable levels ?

Thanks.

AWe

I can confirm this also, no fix and no response from FRS. I can´t believe that there´s anybody in this world who can accept this problem and has fun with this bug. Maestro is also not workable.

The only solution is to use a MAC and the Mac version of SmartSDR or wait and wait and wait until FRS wakes up.

73

John KB4DU

I don’t have that much delay, mine is more like.5 sec. some of this is inherent in the sir design, like propagation delay in cascade logic circuits. There is also some due to the filters chosen. I have mine set for minimum latency.

George-SV9DJO

[{"insert":"Hi Steve, In order to avoid it you should use sound card which supports ASIO. You should find big difference.Same story with Maestro,big latency even on same network.Only ipad which have better hardware has minimum latency.\n"}]

Ignacy

I post a message that I also posted in Maestro category.

Many users of Maestro and users of Flex via PCs experience excessive latencies, usually > 0.7 s. OK with digital modes, irritating in casual CW and SSB, debilitating in contesting. Some don't experience long latencies, just a regular latency (~ 0.2 s). 

Lots of ideas how to fix long latencies that fill weeks to try, with none of them working. Based on observations, the problem is with a specific driver/ process in Windows. Which one I don't know. Here is the rationale and an idea to troubleshoot. 

Long latency by Maestro, either connected directly or WiFi. Means network is a not a source of problem.

With W10 and current updates, long latency with new I7/16Gb, small latency with old I5/16 Gb. Means processor speed is not an issue. Means Windows in general is not an issue, just its detail. Means DPR (deferred procedure call) timing is not a problem. 

Latencies above do not change with connection by Ethernet or WiFi, again meaning that the network is not a problem. 

Latency is present in sound and in button response. So if a sound is delayed by 0.5 second, response to changing BW from 2.1 to 6 kHz is delayed by more less the same. Means that the latency of the sound card (or sound driver) does not cause the issue. 

Small latency with DogSDR under Mac. Means the procedure that causes a problem with Windows is not a problem with Mac. 

One way to troubleshoot is to find two PCs where one shows an issue and another one does not. If one PC shows long latency and another does not, one way to debug is to list all the processes under both systems, and examine the differences. May not help if the problem is in parameters, not processes. 

Dale - AE6DS

Has anyone with the higher latency tried forcing the buffers to minimal size?

https://github.com/miniant-git/REAL

Mike-VA3MW

Grab an superhet radio and a Flex radio and listen to the same signal.

Make 100% sure that you are NOT using a PC to listen. You want to listen to the audio from the back of a 6000 series radio. Either the headphone or powered speaker port.

I will tell you the answer. There is a delay, but it is subtle, very subtle. I just did this test last night with a KX2 and my 6600 on 80M while I listened to my American friends discuss the debate. That in itself was entertaining. I urge you to do the test on your own. I was blown away with how close it was.  

Now, listen to the PC audio with the audio coming out the PC speakers and, yes the delay is much longer. I then called engineering to discuss the AF path from the radio, over the LAN through the radio to the PC speakers.  

The AF data is sent to the remote PC to a small program that is part of SmartSDR where it is immediately handed over to the Windows sound subsystem that then passes it to the speakers. The delay on this was incredibly short.

Want to know what it looks like once Windows gets its hands on it?  it looks like this.  

Look at all the stuff that wants to play with your audio before it gives you your audio. This is not part of SmartSDR but is part of Windows.

Windows is not a real-time operating system. All requests to the operating system are delivered on a best-effort basis. There are no guarantees whatsoever that requests are delivered within a certain time frame, which are the characteristics of a real-time operating system. That is not a problem for most devices and tasks but this is bad news for audio applications (which are considered soft real-time) because they need to deliver data to the subsystem and the hardware in buffers several times per second. If one or more buffers miss their deadlines and are not delivered in the time it has audible consequences.

What about DPCs and ISRs?

The Windows thread dispatcher (also known as scheduler) which is part of the kernel executes threads based on a priority scheme. Threads with higher priority will be given a longer execution time (also known as quantum or time slice) than threads with a lower priority. However, the kernel also knows other types of units of execution known as interrupt service routines (ISRs). Devices connected to the system may interrupt on a connected CPU and cause their interrupt service routines to execute. An interrupt can occur on the same processor that an audio program is running on. Any thread that was running on the processor on which an interrupt occurred will be temporarily halted regardless of its priority. The interrupt service routine (ISR) is executed and may schedule a DPC (Deferred Procedure Call) to offload an amount of work. The DPC will most likely run immediately on the same processor which means the audio application will halt until both the ISR and the DPC routines have finished execution. That is because ISRs and DPCs run at elevated IRQL which means they cannot become preempted by the thread dispatcher (scheduler). Therefore to guarantee responsiveness of the system, ISR and DPC routines should execute as fast as possible. Guidelines say that they should not spend more than 100 µs of execution time however this is often not reached due to hardware factors beyond the control of the driver developer. If execution time gets too high, the audio program may be unable to deliver audio buffers to the hardware in a timely manner.

So, while using SmartSDR on LAN remote exposes the problem, you won't fix it there. That is just the symptom. (this is also why a remote rig to remote rig tunnel is faster--there is no Windows subsystem to deal with). The delay is in the Windows audio subsystem.  You may be able to tune this, but I haven't found any tuning I can do on my sound card. I did want to start at buffer size. In talking to some audio friends in the broadcast industry they wished me good luck. LOL

The same happens on a Maestro as the audio is routed through the Windows Embedded tablet that is the core part of the Maestro. The delays on a local lan I experienced are maybe 200-300ms (guessing at the timing). 

It sounded LONG but it would not impact me in a contest because the receiving station does not always take the first station he hears, he takes the first complete callsign he copies since he can complete the contact in the shortest amount of time. Again, the data gets to the tablet in a very short order. Windows does a bad job of processing it from our perspective. Well, it isn't that bad. Only the OCD types get fussed by it. LOL

If audio latency is that critical to you, then grab the audio from the back of the radio and do not let Windows touch it. Oh, the same is true for TX audio. Problem solved.  

If you have a way to speed it up through some Windows changes/tuning, I would love to hear it.  

Mike va3mw

John KB4DU

Yup, this is the reason I use the radio audio, not pc audio. Windows is promoted as multi-tasking, but not really, it’s time slicing.

Nick RN3KK

If connect to radio from two clients (ipad and windows), sound on ipad is fastly!

Mike-VA3MW

@Nick RN3KK Yes, not surprisingly. It does not have the same overhead that the windows operating system has.

AWe

Ok Mike, thanks for information. BUT, i can tell you no other manufacturer of SDR´s has this - i call it again - problem.

For example, when you use a SunSDR you hear what you see "live" with "no" delay. Same on the Apache Labs, everything smooth, visual pan and audio is sync. All running on Windows 10.

I think it´s time to ask a programmer for a solution... Maybe it´s enough to let us choose the audiobuffer in SmartSDR?

73

Mike-VA3MW

@AWe

I have no experience with either radio, so I can't comment, but a demo showing the sound coming from Windows and not directly from the hardware would be interesting to see and confirm we are comparing the same things. As I mentioned, if you use the audio directly from the radio, the latency is awesomely low.

My discussion here is only regarding the audio and not regarding the panadapter at all. For me personally, that is secondary information and not as time critical for making a QSO in either a contesting on non-contesting environment. I can accept that being slightly latent as it won't affect my run rate at all. Again, it is all about trade offs.

In reviewing the physical connections for both the SunSDR and the Anan, they have physical speaker connections on the back of the radio. That is not what I am comparing this to.

Programmers are smart/efficent and they will make use of the API to send the data to the place it needs to be and they may or may not have the experience when it comes to what happens between the application and the end user (in this case, the speaker). They may or may not have in depth knowledge of how the data gets from point A to point B other then it does or does not.

BTW, I did discuss this with one of our VPs of engineering who is a programmer before posting it who was able to confirm my findings. There is some buffer management you can do, but it is a balance between quality and performance. Also, the more signal processing you do, the more time it takes to do it.

Thanks for sharing your findings.

Mike

Asher - K0AU

Does Flex use the Windows Low Latency Audio feature?

Windows10 has options to reduce the processing in the audio stack. https://docs.microsoft.com/en-us/windows-hardware/drivers/audio/low-latency-audio

Windows 7 is past end-of-support. I don't know the official Flex stance, but that's a strong signal that backward compatibility should no longer be an issue. https://www.microsoft.com/en-us/microsoft-365/windows/end-of-windows-7-support

Jon_EA2W

Hi!

I shared this already in another post, but this is the post where it should be so here it is again with some additional details.

I recorded a video comparing LATENCY of Flex (PC Audio) vs backpanel audio sent through PC Line IN and RemAud by DF3CB. A huge difference and definitely not a Win10 issue.

https://youtu.be/qqZFFDEDvnY

Setup:

Flex 6600 Remote connection is done via WAN with 300MB Fiber links both sites, using Softether VPN.

WAN latency is around 10-15ms according to Flex network monitor.

On the local side, PC with Intel i5 + 32GB RAM with Win10 Pro.

For testing the Flex 6600 audio without any other device, simply click "PC AUDIO" button in SmartSDR.

For testing RemAud by DF3CB, the audio is taken from the Flex 6600 back panel and connected to the sound card´s line IN of a PC with win 10 located in the remote side, with RemAud Server.

RemAud client is open in the PC on the local side. Clicking on "PC AUDIO" in SmartSDR switches between audio coming through the backpanel ("PC AUDIO" unselected) and coming through SmarSDR ("PC AUDIO" selected).

In my opinion there is a lot of room for improvement...

73! Jon

Mike-VA3MW

Hi Jon

Yes, we are 100% aware of this and it will be looked into at some point, but no date at this time.

As you have found alternate audio solutions, you may want to explore using those if you can't connect to the back of the radio directly.

73

Ignacy

Jon,

Impressive work.

If Windows audio was slowed by DPC, then a faster computer would handle DPC faster. But more less the same lag with slower and faster computers.

I tested PC and Mac with Zoom, which is definitely very well optimized. Very small lag. This shows like you did with RemAUD that Windows is at most a partial problem. Perhaps it is the compression mechanism. Can you switch between PC Audio and RemAUD and compare the transfer rates?

It seems that for 2 years there is no software development for Flex except for bug fixes. K4 is coming however its software is still incomplete and no duplex.

Ignacy NO9E

Marios Nicolaou

Hi

Having spent a significant amount of time comparing different methods of audio transmission (using KU7T published method), I found that actually using ipad/iphone audio or mac was up to 7 times better than using smartSDR for the PC. My current latency is 65-70ms at 3500km.

If you would like real numbers see report here:

http://5b4wn.com/main/a2203.html

Hope this helps

Marios

5B4WN/G0WWW

C4W

DL4RCE

I do the same, using latest iPad Air and it works super great.

Volker