Distorted TX DAX audio

James Skala

Radio Flex 6300 / SmartSDR 2.0.17 / OS Windows 10.  I have a remotehams remote that I provide to other hams.  Every couple days I have been experiencing distortion in the TX DAX audio.  A simple close of all the programs and relaunch fixes the issue.  This has been occurring with 1.9 / 1.10 and now 2.0.  Was hoping that it would washout with the newer releases.  Audio into and out of remotehams is clean.  I have just closed and launched the DAX program and the problem goes away.  See below Link with Audio of a TX with no problem and a less then a minute later it starts.

https://drive.google.com/drive/folders/0ByikFTG2iMyWcl9vMmFaMURTRGc?usp=sharing

K9DUR

James, This is a known issue that has been around for some time. All you have to do is stop & re-start DAX. I hope that Flex can get a handle on it soon. 73, Ray, K9DUR

James Skala

UGG! Tim when will this be fixed?

Ian1

I am having the exact same problem as well.

Ian

Tim - W4TME

This issue is in our bug tracker as issue #2391.  I have no ETA data at this time.

Mike NA5U/K5D/K1N

I was told this is memory leak. There was a stab at fixing it but looks like it didn't work. It was better after the attempt but 2.0 has it back as bad as before.
73,
Mike
NA5U

Tim - W4TME

We suspect buffer corruption, not a memory leak.

Lee - N2LEE

makes more sense than a memory leak now that you mention it

Sergey R5AU

James !

I am operate on Win10 virtual machine and at time of switching between Host and VM some time buffer corrupted  or in case network time out it can be happened. My suggestion is to change priority of the DAX task via command    START /REALTIME DAX.EXE   with admin privileges. of course you can create a BAT file and out into the Start Menu, BTW do not forget disable initial autostart  of dax.exe

Tim - W4TME

This buffer corruption problem has been linked to long duration DPCs on the PC running DAX.  If this is your issue, you can try to take mitigating action to resolve the condition.

Here are some relevant HelpDesk articles on the subject.

https://helpdesk.flexradio.com/hc/en-us/articles/202118398-What-are-DPCs-and-why-do-they-matter-

https://helpdesk.flexradio.com/hc/en-us/articles/202118698-Using-LatencyMon-to-collect-DPC-Latency-D...

Bill W2PKY

I could never get DAX stable under Win10 on my ASUS Z97-A motherboard, no matter what I did with Affinity,Privileges,Priority etc. Went back to Win7 Pro and DAX is stable on the receive side but the transmit buffer becomes corrupted if SSDR is run for long periods of time, IE. overnight.
However DAX is stable on my Surface Pro 3 & 4. So it seems that Win10 has issues with certain hardware. Just my observation.

James Skala

Tim, thanks for that information.  I will run this and see what we get.  So no matter how powerful the PC or CPU usage this DPC ISR stuff could be happening in the background even thought CPU and RAM is in great shape? 

Is FlexRadio saying this is not a DAX issue but a consumer configuration / performance issue?

What about the external programs like remotehams that process audio and their audio chain has no problem while DAX is?

thanks for your hard work, really enjoying 2.0

K9DUR

Tim,

I just ran LatencyMon for about 12 minutes.  "Highest  measured interrupt to process latency" was initially 864 us and never went higher than that.  The "Highest reported DPC routine execution tims" was 442 us & came from ndis.sys.

My PC runs 24/7 & problem occurs about 24-72 hours after DAX is started, whether I am actively using DAX or not.  I have not been using it recently in order to monitor the problem, but I have created a script file that kills & re-starts both CAT & DAX when I start the SamrtSDR client.

73, Ray, K9DUR

James Skala

Network Driver Interface,
See link https://answers.microsoft.com/en-us/windows/forum/windows_10-other_settings/windows-10-high-dpc-late...

Very Good read look like TCP Optimization settings fixed some similar problems

See another good read
https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/example-15--measuring-dpc-isr-time

Tim - W4TME

James, DPCs are not related to how powerful the CPU is or the speed and amount of RAM.  It is about the hardware interaction (primarily drivers and the BIOS) with the operating system.

What we are saying is that the current DAX driver appears to be susceptible to issues with PCs that experience long duration DPCs.  We are investigating using a different driver technology for DAX that should provide relief from this type of situation.  


Ray - I have seen instances where LatencyMon had to run for hours to "catch" an instance of a long duration DPC.

James Skala

Thanks cant wait to get home to hound dog this out.

Sergey R5AU

Bill many DPC/Buffer interrupts issues are belong Power management/ACPI systems, and in some cases - ex.: LapTops can't be solved at all (Motherboard/BIOS things) in all other cases 3 ways of solving are here:
- BOIS
- different OS management: Power settings
- drivers update, BTW with drivers not everything are clear, even native drivers for HW are not always optimal

also useful links:
https://www.sweetwater.com/sweetcare/articles/solving-dpc-latency-issues/
https://answers.microsoft.com/en-us/windows/forum/windows_10-other_settings/windows-10-high-dpc-late...
https://answers.microsoft.com/en-us/windows/forum/windows_10-performance/high-ndissys-dpc-windows-10...
check your drivers - https://sdi-tool.org/

P.S. I have read many times issues related with ROG ASUS MB series 

James Skala

_________________________________________________________________________________________________________
CONCLUSION
_________________________________________________________________________________________________________
Your system appears to be suitable for handling real-time audio and other tasks without dropouts.
LatencyMon has been analyzing your system for  5:29:42  (h:mm:ss) on all processors.

_________________________________________________________________________________________________________
SYSTEM INFORMATION
_________________________________________________________________________________________________________
Computer name:                                        DESKTOP-IT7CH92
OS version:                                           Windows 10 , 10.0, build: 15063 (x64)
Hardware:                                             M32CD_A_F_K20CD_K31CD, ASUSTeK COMPUTER INC.
CPU:                                                  GenuineIntel Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz
Logical processors:                                   4
Processor groups:                                     1
RAM:                                                  16308 MB total

_________________________________________________________________________________________________________
CPU SPEED
_________________________________________________________________________________________________________
Reported CPU speed:                                   3696 MHz
Measured CPU speed:                                   1 MHz (approx.)
Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results.
WARNING: the CPU speed that was measured is only a fraction of the CPU speed reported. Your CPUs may be throttled back due to variable speed settings and thermal issues. It is suggested that you run a utility which reports your actual CPU frequency and temperature.

_________________________________________________________________________________________________________
MEASURED INTERRUPT TO DPC LATENCIES
_________________________________________________________________________________________________________
The interrupt to DPC latency reflects the measured interval in which a DPC could execute in response to a hardware request from the moment the interrupt service routine started execution.
Highest measured interrupt to DPC latency (µs):       527.514713
Average measured interrupt to DPC latency (µs):       2.982746

_________________________________________________________________________________________________________
 REPORTED ISRs
_________________________________________________________________________________________________________
Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.
Highest ISR routine execution time (µs):              43.547619
Driver with highest ISR routine execution time:       HDAudBus.sys - High Definition Audio Bus Driver, Microsoft Corporation
Highest reported total ISR routine time (%):          0.003113
Driver with highest ISR total time:                   HDAudBus.sys - High Definition Audio Bus Driver, Microsoft Corporation
Total time spent in ISRs (%)                          0.006193
ISR count (execution time <250 µs):                   1796816
ISR count (execution time 250-500 µs):                0
ISR count (execution time 500-999 µs):                0
ISR count (execution time 1000-1999 µs):              0
ISR count (execution time 2000-3999 µs):              0
ISR count (execution time >=4000 µs):                 0

_________________________________________________________________________________________________________
REPORTED DPCs
_________________________________________________________________________________________________________
DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.
Highest DPC routine execution time (µs):              375.566017
Driver with highest DPC routine execution time:       ndis.sys - Network Driver Interface Specification (NDIS), Microsoft Corporation
Highest reported total DPC routine time (%):          0.143481
Driver with highest DPC total execution time:         ndis.sys - Network Driver Interface Specification (NDIS), Microsoft Corporation
Total time spent in DPCs (%)                          0.504743
DPC count (execution time <250 µs):                   124171242
DPC count (execution time 250-500 µs):                0
DPC count (execution time 500-999 µs):                28
DPC count (execution time 1000-1999 µs):              0
DPC count (execution time 2000-3999 µs):              0
DPC count (execution time >=4000 µs):                 0

_________________________________________________________________________________________________________
 REPORTED HARD PAGEFAULTS
_________________________________________________________________________________________________________
Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

Process with highest pagefault count:                 none
Total number of hard pagefaults                       0
Hard pagefault count of hardest hit process:          0
Highest hard pagefault resolution time (µs):          0.0
Total time spent in hard pagefaults (%):              0.0
Number of processes hit:                              0

_________________________________________________________________________________________________________
 PER CPU DATA
_________________________________________________________________________________________________________
CPU 0 Interrupt cycle time (s):                       764.454298
CPU 0 ISR highest execution time (µs):                43.547619
CPU 0 ISR total execution time (s):                   4.569197
CPU 0 ISR count:                                      1475071
CPU 0 DPC highest execution time (µs):                375.566017
CPU 0 DPC total execution time (s):                   379.468369
CPU 0 DPC count:                                      117965197
_________________________________________________________________________________________________________
CPU 1 Interrupt cycle time (s):                       29.986554
CPU 1 ISR highest execution time (µs):                5.339827
CPU 1 ISR total execution time (s):                   0.331346
CPU 1 ISR count:                                      321744
CPU 1 DPC highest execution time (µs):                220.524351
CPU 1 DPC total execution time (s):                   4.217557
CPU 1 DPC count:                                      1663268
_________________________________________________________________________________________________________
CPU 2 Interrupt cycle time (s):                       44.540662
CPU 2 ISR highest execution time (µs):                2.205087
CPU 2 ISR total execution time (s):                   0.000002
CPU 2 ISR count:                                      1
CPU 2 DPC highest execution time (µs):                182.601732
CPU 2 DPC total execution time (s):                   8.217507
CPU 2 DPC count:                                      2277722
_________________________________________________________________________________________________________
CPU 3 Interrupt cycle time (s):                       37.633667
CPU 3 ISR highest execution time (µs):                0.0
CPU 3 ISR total execution time (s):                   0.0
CPU 3 ISR count:                                      0
CPU 3 DPC highest execution time (µs):                114.716991
CPU 3 DPC total execution time (s):                   7.506331
CPU 3 DPC count:                                      2265083
_________________________________________________________________________________________________________

Tim - W4TME

You only ran it for 5 minutes.  Normally you have to run it while using the radio and after you experience DAX corruption for the data to be valid.

James Skala

5:29:42 (h:mm:ss) 5 Hours

Tim - W4TME

Whoops, missed that.  Was there an issue with DAX during that time period?

James Skala

Yep, so I kept it up and continue to text into dummy load via remotehams. I will perform capture again in a couple days when it occurs again

Tim - W4TME

RRR.  Thanks

Kevin

Wouldn't a DPC issue be a short term issue? Maybe when the system is busy handling lots of interrupts there's some sort of issue. But then when things calm down again shouldn't DAX start working again?

For instance, I listen to Disturbed in VLC and I hear a pop. Dang. But that's it. It popped and went away and all is back to normal.

Then I transmit a single tone with DAX and I see this:


That's not a pop. It doesn't clear up by itself and it probably sounds strange on the air. In fact, maybe it sounds like something from Disturbed. This is what happens to me every 24+ hours or so.

I restart DAX and get this:


Is this covered by whatever #2391 is about? As 2391 has been around at least six months and I believe the problem has been around longer is it just something that can't be fixed?

I guess there's lots of people very happy with the remote feature in the new software version. But for me, it is the same old software with pretty much all the same old issues. Almost exactly the same old issues.

73,
Kev K4VD

James Skala

Had issue occur multiple times since last post see below output from latencymon on last occurrence 


_________________________________________________________________________________________________________
CONCLUSION
_________________________________________________________________________________________________________
Your system appears to be suitable for handling real-time audio and other tasks without dropouts.
LatencyMon has been analyzing your system for  2:12:08  (h:mm:ss) on all processors.

_________________________________________________________________________________________________________
SYSTEM INFORMATION
_________________________________________________________________________________________________________
Computer name:                                        DESKTOP-IT7CH92
OS version:                                           Windows 10 , 10.0, build: 15063 (x64)
Hardware:                                             M32CD_A_F_K20CD_K31CD, ASUSTeK COMPUTER INC.
CPU:                                                  GenuineIntel Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz
Logical processors:                                   4
Processor groups:                                     1
RAM:                                                  16308 MB total

_________________________________________________________________________________________________________
CPU SPEED
_________________________________________________________________________________________________________
Reported CPU speed:                                   3696 MHz
Measured CPU speed:                                   1 MHz (approx.)
Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results.
WARNING: the CPU speed that was measured is only a fraction of the CPU speed reported. Your CPUs may be throttled back due to variable speed settings and thermal issues. It is suggested that you run a utility which reports your actual CPU frequency and temperature.

_________________________________________________________________________________________________________
MEASURED INTERRUPT TO DPC LATENCIES
_________________________________________________________________________________________________________
The interrupt to DPC latency reflects the measured interval in which a DPC could execute in response to a hardware request from the moment the interrupt service routine started execution.
Highest measured interrupt to DPC latency (µs):       543.306908
Average measured interrupt to DPC latency (µs):       3.310021

_________________________________________________________________________________________________________
 REPORTED ISRs
_________________________________________________________________________________________________________
Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.
Highest ISR routine execution time (µs):              34.524351
Driver with highest ISR routine execution time:       HDAudBus.sys - High Definition Audio Bus Driver, Microsoft Corporation
Highest reported total ISR routine time (%):          0.011174
Driver with highest ISR total time:                   Wdf01000.sys - Kernel Mode Driver Framework Runtime, Microsoft Corporation
Total time spent in ISRs (%)                          0.014343
ISR count (execution time <250 µs):                   3402540
ISR count (execution time 250-500 µs):                0
ISR count (execution time 500-999 µs):                0
ISR count (execution time 1000-1999 µs):              0
ISR count (execution time 2000-3999 µs):              0
ISR count (execution time >=4000 µs):                 0

_________________________________________________________________________________________________________
REPORTED DPCs
_________________________________________________________________________________________________________
DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.
Highest DPC routine execution time (µs):              226.984848
Driver with highest DPC routine execution time:       ndis.sys - Network Driver Interface Specification (NDIS), Microsoft Corporation
Highest reported total DPC routine time (%):          0.231901
Driver with highest DPC total execution time:         ndis.sys - Network Driver Interface Specification (NDIS), Microsoft Corporation
Total time spent in DPCs (%)                          0.723889
DPC count (execution time <250 µs):                   53943768
DPC count (execution time 250-500 µs):                0
DPC count (execution time 500-999 µs):                0
DPC count (execution time 1000-1999 µs):              0
DPC count (execution time 2000-3999 µs):              0
DPC count (execution time >=4000 µs):                 0

_________________________________________________________________________________________________________
 REPORTED HARD PAGEFAULTS
_________________________________________________________________________________________________________
Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

Process with highest pagefault count:                 none
Total number of hard pagefaults                       0
Hard pagefault count of hardest hit process:          0
Highest hard pagefault resolution time (µs):          0.0
Total time spent in hard pagefaults (%):              0.0
Number of processes hit:                              0

_________________________________________________________________________________________________________
 PER CPU DATA
_________________________________________________________________________________________________________
CPU 0 Interrupt cycle time (s):                       387.643007
CPU 0 ISR highest execution time (µs):                34.524351
CPU 0 ISR total execution time (s):                   3.700013
CPU 0 ISR count:                                      2479622
CPU 0 DPC highest execution time (µs):                226.984848
CPU 0 DPC total execution time (s):                   221.033955
CPU 0 DPC count:                                      51501895
_________________________________________________________________________________________________________
CPU 1 Interrupt cycle time (s):                       16.723305
CPU 1 ISR highest execution time (µs):                6.873918
CPU 1 ISR total execution time (s):                   0.848960
CPU 1 ISR count:                                      922913
CPU 1 DPC highest execution time (µs):                73.712121
CPU 1 DPC total execution time (s):                   1.526959
CPU 1 DPC count:                                      643739
_________________________________________________________________________________________________________
CPU 2 Interrupt cycle time (s):                       18.518138
CPU 2 ISR highest execution time (µs):                3.058983
CPU 2 ISR total execution time (s):                   0.000010
CPU 2 ISR count:                                      5
CPU 2 DPC highest execution time (µs):                211.801948
CPU 2 DPC total execution time (s):                   3.711762
CPU 2 DPC count:                                      873541
_________________________________________________________________________________________________________
CPU 3 Interrupt cycle time (s):                       17.157728
CPU 3 ISR highest execution time (µs):                0.0
CPU 3 ISR total execution time (s):                   0.0
CPU 3 ISR count:                                      0
CPU 3 DPC highest execution time (µs):                55.425866
CPU 3 DPC total execution time (s):                   3.307845
CPU 3 DPC count:                                      924593
_________________________________________________________________________________________________________

Mark - K9MQ

Is there a way in Windows 10 that DAX can be restarted every day at a certain time?

Cedric HB9HFN

Yes. I use the windows task scheduler to restart DAX once a day. My "restart" batch looks like as following:

taskkill /f /im DAX.exe

timeout 2

start "" /min /d "C:Program FilesFlexRadio SystemsSmartSDR v2.0.17DAX" DAX.exe

73, Cedric HB9HFN

Peter K1PGV

>Wouldn't a DPC issue be a short term issue Yes. Sustained, predictable, issues are not likely to be DPC related. Some other driver issue? Certainly. Triggered as a result of a few really long instances of ISR to DPC Latency? Maybe. But not a DPC problem as we classically discuss them here on the forum. Peter K1PGV

Patrick

Like others, I have also experienced this problem. But I still use windows 7, so the problem is not directly based on OS version. It usually happens to me when the system has been running for 4 + hours without being used. If I should use the system during that period I don't have the problem. When I do have the problem it is when I am using Dax along the digital apps that depend on Dax for audio input. So what is happening when the system is running for long periods of time not being actively used. Is the streaming audio overloading a buffer due to memory leaks. My own solution is to shut down the computer and radio when not in use. When l do this there is never a problem.

Greg - KØGDI

I get the same problem using my 6700 as an APRS digipeater, stations can decode me fine for like a day until the audio starts getting distorted.  Closing DAX and reopening it fixes it.  Hope they can fix this soon.

Tim - W4TME

Thanks for the info.  I'll add it to the defect report.