SDR waterfall suddenly slow/jerky

Brent Parker

I've been using SDR and my 6500 for about a month. Computer is a dell T5400, 8 core, with GTX 670 display. Everything has been work fine. Very Smooth

Suddenly today, the waterfall display is very slow and jerky. I updated graphics, checked network, even un-insatlled 1.3.8 and re-installed. SDR when I terminate it, doesn't shut down, and I have to **** it in the task manager. 

I then installed 1.3.8 on my wife's computer (right in the shack) and it's fine! So, it's the computer, not the radio.

So finally I un-installed 1.3.8 and downgraded to 1.3.0.

Everyting is nice and smooth again! 

Ideas??

I could try up-grading to 1.3.8 on this machine again, but I'm sort of from the school "if it ain't broke, don't fix it!" 

I welcome all thoughts. I know I've seen some discussions here about display issues, just suprised to see this pop up, when I made no changes on this computer!

Thanks in advance,   Brent  NB4AP

Walt - KZ1F

Brent,
   I don't know that  FRS, considers it a known problem but I can tell you it is a reported problem. Look for my report on hyperactivity. A number of people agreed with my report, said they have the same issue, and some have publicly and other privately report downgrading to 1.3.0. There are related issues that FRS, which is to say Tim, has acknowledged as known problems.

   Is your wife's machine a Dell as well?

   One of the problems on this site is when one person reports an issue or makes a suggestion, others often come out of the woodwork as if it was a personal attack on them, that they have to defend FRS's honor. While I have my own description of that behavior, what it does is derail the initial problem report. The fact someone hasn't seen this issue doesn't mean it isn't broken in the code.

You mentioned specifically the waterfall. Do you also see it in the panadaptor band scope as well? In the waterfall, does it appear to update faster but it has, what appears to be, blank lines interspersed with the data lines?

Walt - kz1f

Brent Parker

Walt,

There is significant overall latency. The waterfall, simply can't keep up. However if I tune, the slice is lagging behind too. The flexcontrol is sluggish. When I startup SDR, it takes 8 seconds or so, for the waterfall to even show up.

It's as if this network on this maching just became about a 2mbps instead of 100 or 1000.

My wife's machine is exactly the same dell, except is a a much inferior video card!  It's smooth.

Mine has been working fine, 'till I came home today and found the issue.

Version 1.3 is fine. Strange issue.

Thanks,    Brent NB4AP

Brent Parker

Problem fixed............not sure why/how?

Installed new video card/driver, no joy. Installed new network patch cable, still the same.

Finally deleted an install of a remote control software (Radmin, it allows me to remote control/update the computer from a different computer on my network), plus killed a bunch of utilities running in the tray, and it came back to life.

I've thinking it's the Radmin software, but will try and replicate to be sure.

The "nice" think was having a spare computer to log into the radio, and see that work properly, to then know it was something with the computer!

Brent  NB4AP

Walt - KZ1F

You are correct Brent, this (or that) is a strange problem. The more you describe what you are witnessing, the less it sounds like what I reported. Did you get the opportunity to look at https://community.flexradio.com/flexradio/topics/1-3-8-hyperactivity? There is a known, acknowledged latency issue. It was reported against the SWR meter or power meter. In my case when I am on SSB when I unkey the mic the display still shows me talking for about 2 secs.  The issue I see with waterfall is not that it goes slower, it goes significantly faster and there are blank lines in the display, so either the event to scroll and paint the top line of pixels is being erroneously fired or it is being properly fired but lacks the pixels to paint, after the scroll. The net result is there are pixel width blank line interspersed with the waterfall data. In both cases, top display and bottom display, it is significantly faster than normal speed. Don, AI6RE put up a YouTube video of it, http://youtu.be/lWMDLix2klQ. If the computer use works fine on 1.3.0 but shows weirdnesses with 1.3.8, it is not the computer. This is why I tried to give the detail I did so that the engineers at FRS could use that to, hopefully, narrow down the area of code that had changed between 1.3.0 and 1.3.8 to identify what triggers the symptoms.

It is my understanding that when FRS acknowledges a known bug Tim changes the community board topic to a helpdesk ticket. In the case of what I reported this was not done. It may be this is sufficiently operlapping other transitioned reports that it will be fixed by virtue of the others being fixed. It is my sincere hope it is fixed in 1.4. I don't believe day one of 1.3.8 presented these issues here. I believe they developed over time. That would indicate a state change, perhaps as a result of some switch being persisted that wasn't present in 1.3.0. If the data isn't persisted there is no problem but if whatever data is, there is an unintended side effect. This could explain why some people see these problems and others don't. But, again, the fact others don't see these issues in no way mean there isn't a software issue.

As one 6300 owner said in my earlier thread, "I see it much worse than the video", actually, so do I on a 6500.

Brent Parker

Walt,   It appears my issues was somewhat different. There was no hyyperactivity. Just extreme slow behavior. The waterfall was running at about 0.25% speed or less, then leaving a blank bar, where it would skip and try and catch up, creating very jerky behavior. Definatly someting with the cpu being delayed getting back to the task.  Yes, I read the link, but again it does sound different. If I re-create this, I'll video it and save.

I have a lot of computers, some at the house (shack) some at the office (which gets to the house with a wireless bridge) This was the first computer to exhibit this behavior!  I installed 1.3.8 on two at the house and three at the office, and all worked great. Mind you, the wireless link is only 17mbps, but is plenty fast!

Then suddenly this wierd behavior on the one machine. I reverted several time to 1.3.0, which cured it and then upgraded to 1.3.8 and it would re-appear.

Finally I terminated a remote driver (it's a remote mirror driver to allow remote control of the computer), plus I killed about every element running in the tray (security, ups utility, icloud, nvidia control panel) and the problem went away. So presumably it was one of those 5 elements, interacting with something not quite right with 1.3.8.

This afternoon, I'm going to try and recreate the problem and eliminate one piece at a time to try and isolate this interacting element.

If I can isolate this issue, the I suspect Tim will elevate this to a "bug", but right now there are probably too many un-knows.

I'll  let you know how my testing goes! 

73  Brent NB4AP

W8TZ Ross

"One of the problems on this site is when one person reports an issue or makes a suggestion, others often come out of the woodwork as if it was a personal attack on them, that they have to defend FRS's honor."

  It is nice to see that I am not the only one who has noticed this "behavior".

Walt - KZ1F

Cool, thank you. Understand what you have is, effectively, a Dell PowerEdge, rather than a Sandy Bridge processor it's a Xeon, can take 128GB of memory. It's dual quad core. It's a business class machine. What you could add are things like do you use profiles, do you use CWX, effectively what feature you use that are new to 1.3.8 that you didn't replicate use of on your wife's machine. The more detail on what features you use the better the likelihood someone at FRS can narrow down the likely culprit. When I turned off the Dell Powersaver profile to not put the monitor to sleep or switch it to 1/2 brightness after non-use, I thought that 'fixed' the problem. No, it appeared initially like it did but, no, it's right back to what it was before with respect to SSDR behavior.It's just now the monitor never goes out. That occurred to me either right before or right after Al, NN4ZZ said he never uses any power saver features (that I believe come turned on). Al, is one of the users that hasn't experienced these behaviors. Al, coincidently, also is not a phone operator so perhaps people who are ardent CW operators never see the issue. Again, what is different between those that see this and those than don't. Anyway, I am interested in anything you could add.

Brent Parker

Walt  - Narrowing down the problem. It's my remote software (Radmin). I have the Radmin "server" on every computer in my network (about 30 computers - 5 at home, the rest at work). All the machines are Dell PowerEdge T5400's, T5500's or T5600 with various forms of dual processor with typical 4-8 cores in each processor, lots of ram, and "big" video cards. My firm practices architecture and it takes significant computer power to do that.

If I remote from say work, to the workstation at the house, the "mirror" driver (Radmin video) is sending all that display info, back over the wireless bridge, and both the local computer (server) and the remote (client) have a fairly jerky display of both the spectrum and waterfall. That doesn't suprise me too much as thats a lot of data to move. However it appears the issues is literally in the local computer (server) haveing to actually display the graphics and then "mirror" it out to the remote!

It was working fine for the last month, and is working fine now, with a re-install.

However what happened yesterday, was the local machine starting haveing the same behavior, even without the remote machine logged in. When I finally deleted the Radmin install, the display returned to normal. I've re-installed now, and it still works fine. If I remote into it, it goes slow, but as soon as I log off, it's back to normal, which is what I had before.

So apparently Radmin even with re-boots, was hung as if something was logged in.

The side story, is my M6400 workstation 17" laptop, I thought was a dog and would not display fast enough . I just deleted Radmin from it and the display is great!  So everything I said bad about it, I have to take back!

It is wierd that 1.3.0 displayed fine, so there is some interaction with 1.3.8 and Radmin. Who knew.

Finally you should know I've installed SDR 1.3.8 at work, accessed the radio at home over the wireless bridge (17mbps) and the display is great. 17mbps for lan/internet seem to be plenty fast enough, which is great news for the future (2.0). 1.3.8 isn't officially sending audio/mic over the lan yet, but there are a couple of un-official techniques. Sometimes they work, sometimes the audio gets choppy. I'm sure that is part of what the boys are working on to get right.

Thought you'd like the update and problem mostly solved.

Regards,   Brent  NB4AP

W1IMD

Brent et al.

I have had similar issues with RealVNC and posted a message just days ago about this. Stu K6TU responded and he believes it is not a SmartSDR issue. His suggestion was to reduce the Frames Per Second rate and dropping it to 10 FPS results in video that does not appear to be delayed either locally or remotely.

I have been using RealVNC for years and this delayed video behavior is a recent development. I cannot say whether it's an issue with SmartSDR or an update in RealVNC or even Windows. I also saw delayed video even when nothing was connected remotely.

For now, I am satisfied with a 10 FPS rate, and I don't use the waterfall. A friend who has an almost identical setup does use the waterfall and he's good with this setting as well.

Can't wait for 1.4

73,

Paul

Walt - KZ1F

That's very interesting. You're right in that the PowerEdge should be more than sufficiency to run anything you are running plus renting time to the Condor Project. Are you familiar with DMA, direct memory access? Your video card has a GPU and has however much video memory local to the GPU. A lot of this alludes me now as DMA was big in the 80's if not 70's when memory on auxiliary cards was mapped directly into the processor's address space. So what if there is a lock on the DMA memory and WPF is blocked on waiting for access to the memory on the video card. This prevents a lot of copying from the video card to processor memory. WPF isn't new to 1.3.8 but perhaps it may be something FRS did or set in a config (ring 3 shouldn't even see any of this) that causes locking on the video. And that doesn't even address what I've seen as I don't use radmin. However, now you've raised that there was some software I installed on the request of Dudley so he could 'dial in'. I think the particular problem was with 1.3.8 though. Plus what I see is uber fast spastic drawing as opposed to a lock contention issue.

I don't understand something. You reinstalled 1.3.8 meaning a complete uninstall then a **** reinstall? No upgrade, correct? So given what you were saying about radmin and how it is involved, does the uninstall/reinstall from scratch still hold relevance?

Thanks for digging into this,

Walt

Walt - KZ1F

Paul, was that while using RealVNC or just having it installed? In other words, if it installs as a service it would always be 'on'. What it was I installed for Dudley was TeamViewer and it did have an automatic and started Windows Service running, which I stopped and made manual. Did you experience this issue using RealVNC to remote into your machine or merely having it run beside SSDR in server mode..vs. client mode.

Brent Parker

Walt - to do a SDR install of 1.3.0, I'd use uninstall SDR and the DAX, and uninstall FlexVSP, then re-boot. I'd then install SDR 1.3.0. Then I'd connect to the radio. It would see the changed software and the radio downgrades from 1.3.8, to 1.3.0 .  When I went back to 1.3.8, repeated the un-install process and re-installed SDR and again the radio would upgrade.

Similar with Radmin, I completly un-installed it, found at that point that SDR 1.3.8 was good, then re-installed Radmin clean. It was at that point it went back to normal, that is if I was remoting in, it went slow, but if I wasn't, it "released" it, and the display worked normally.

So maybe the remote software you temporarly installed behaved similar?? All remote softwere basically has to have a remote "mirror" driver, I would think, and maybe that timing can get messed up.

Brent

Brent Parker

Paul - Apparently from Walt's recent post, he also had a "remote" software installed, chasing a problem with Dudley, so that's three or four instances where if you were remote controlling the computer driving the radio, there was a latency/timing/delay issue.

Yes mine got better if I changed the frame rate, but not great. The FlexControl was also very slugish. Turn waterfall off??? can you do that, or just drag down the middle bar to make is dissappear??

Yea, I believe 1.4 will be great, with the audio officially over the lan. Probably be able to do wan, with vpn too.

73,  Brent

Walt - KZ1F

Dudley never actually did connect in but my problem was hyperactivity in the waterfall and band scope plus a 2 sec lag plus my Flex Control, which I don't use hardly at all when I would was slow and overshot where I was trying to adjust it to. There clearly is overlap between what I saw/am seeing and what Paul and Brent are seeing. Again, the fix still lies with SSDR as this 3rd party software runs/ran fine with 1.3.0. But it does give FRS more information than they had a week ago.

Walt

Walt - KZ1F

maybe it has to do with dropping in and out of GPU mode. in one case setting up timing to use the GPU then dropping out of that mode to non-GPU mode and not readjusting the timing or the other way around. In once case things would go slower in the other faster. That should all be completely transparent though to SSDR as WPF should (something else I don't know is WPF) be taking care of it.  It does appear timing related. That is such a stretch though. Well, it gives FRS more information as the hyperactivity issue was never, apparently, recognized as a bug despite the number of people claiming the identical symptoms. I reverted back to Dell defaults so I will see if the hyperactivity occurs on a screen awake. I sincerely hope it is resolved in 1.4.
Yep, once the monitor wakes from being in sleep mode the timing is all fracked up. After a few moments it does resolve after a few seconds, perhaps 3 - 5.
Walt

Brent Parker

Walt, so are you still running SDR 1.3.0 and is teamviewer still installed?  If that's the case, I'd do the remove software bit in windows of teamviewer and then remover SDR, then re-install 1.3.8 and when you connect to the radio the radio will update too. I'd love to know if this resolves this. 

Brent

Walt - KZ1F

no, I am on 1.3.8. 1.30 worked flawlessly. SSDR definitely reacts to the monitor being woken up. The SSDR code should know or care what the monitor state is. They could have added something akin to OnWakeUp() and that code is **** up.

Oh wow! Look what I found:
WPF Application windows does not timely refreshed after screen awakes from sleep

as well as

<code>
private void SystemEvents_PowerModeChanged(object sender, PowerModeChangedEventArgs e) {  if (e.Mode == PowerModes.Suspend)  {   PrepareToSleep();  } }
private void PrepareToSleep() {  this.TabControlMain.SelectedIndex = 1; } 

</code>

If there is a PowerModes.Suspend state there would be the converse of that

There is a solution given to that from a Microsoft Engineer. I also found:
WPF Application windows does not fully refreshed after screen awakes from sleep

So I believe we've narrowed down at least part of this to SSDR code.

Walt - KZ1F

Brent, there were a bunch of other problems with displays getting slow. Just because someone else has a program that depends upon timing doesn't mean FRS does, but it looks timing related. The other thing I found references to were locks and processor affinity. Those issues are worse than timing issues.

W1IMD

I first noticed the problem while operating remote but while at my site, I saw the delay while operating locally with no one on the remote side. That threw me for a loop. The RealVNC server operates as a service and is always available.

W1IMD

I don't believe there is a way to turn the waterfall off, I do as you say, drag the middle bar down to hide it.

Tim - W4TME

We are aware of these issues with display rendering and that the issue got worse for some users with v1.3.8.  It is one of the reasons that v1.4 is delayed.  The SmartSDR ecosystem is a complex system.   There are numerous reasons for display issues and the factors have to be identified and  characterized to determine how they contribute to our specific situation.  Only then can we make changes to improve the system.  There isn't a "silver bullet" fix and some of the changes may be foundational in nature, so the work we are doing is not trivial.  I cannot guarantee we have found and resolved all the issues, but the situation has improved greatly for v1.4 based on the testing we have been doing.

Brent Parker

Tim - Glad you are aware of the issue and "chasing" it.

My instance was pretty wierd. For a month, it worked fine, the suddenly it started displaying the jerky behavior. When I un-installed Radmin, it cleaned up. I then re-installed Radmin and it's still working fine.

My laptop (dell dual core workstation) that I thought was under-performing display the same issue, from day one. Having discovered the Radmin issue, I installed it there, and it's suddenly rocks!

That's a wierd little problem to chase, when it's only stuttering part of the time!  Good luck with that. If I can offer any other results/testing, let met know! 

Regards,  73  Brent NB4AP

Ken - NM9P

Thanks for the update, Tim.  I look forward to the fixes.