Is a more structured Alpha/Beta program needed?

Gerald-K5SDR

Yes, I am shaking my head at all the free management advice.  ;>)  Unfortunately, there is not much that would apply to this specific problem.

I'm just recovering this weekend from jet lag after 10 days in Germany and the UK for Hamradio 2016 and other meetings.  I have not been in the office to meet with the team since June 20th but here is what I know:

1. There are approximately 50 Maestros in use on the alpha/beta team.
2. The alpha/beta team is made up of a cross section of hams with varied levels of technical experience and operating style.  Off hand, I don't know the current size of the team since I don't manage it but it is about as large as our engineering team wants to manage.
3. There were 0.00% upgrade failures in alpha/beta.
4. If the alpha/beta team were monkeys, there would still be 0.00% failures since this appears to relate to some type of hardware timing issue related to upgrading PIC software on only to a small number of units.  Technical competence of the alpha/beta testers has no bearing on this specific issue.
5. You could probably upgrade/downgrade over and over forever on those 50 units and never see the problem.  
6. When we realized there was a problem in the general release, we immediately removed the download and announced it here.
7. We are working directly with the small number of customers who reported the problem to expedite the fix at our expense.
8. If your Maestro updated to v1.8, you don't have the problem.  Keep using it.
9. We have many quality procedures in place such as full team design sessions and daily code pull requests/peer reviews.  We also do automated scripted testing on all radio models on every build.  However, we do not yet have any type of automated testing on Maestro builds.  That would be more difficult.

In another thread I related that I had a very similar thing happen with on my iPad 2 when I upgraded iOS a couple years ago.  The update temporarily bricked the iPad so that it would not complete the upgrade nor could I downgrade.  Apple has comparatively infinite financial and human resources to do testing but they did not catch this issue until it was in general release.  The problem only occurred on some units, which I learned by Googling the issue.  Fortunately, they were able to clear the problem within a few weeks without sending it in.  

I also recently had a Gateway PC that completely bricked on a Windows 10 update.  This was after it had been running on Windows 10 for weeks.  It gets completely stuck in the upgrade cycle.  I haven't been able to recover it yet.

If it can happen to Apple and Microsoft, it can certainly happen to us.  This is no fun for us either so you can be assured we will do an after action review to see what can be done to improve in the future.  

Happy Independence Day!

Gerald

VE6KWA

Have you considered Decaf Burt ??   

KC9NRN

I'd be more than happy to fix your Windows 10 issue. I can't imagine having an issue crop up that bricks a device and then you get all the "experts" telling you how to fix it. The network I manage runs smoothly most of the time then suddenly one of the profiles on Terminal Server decides not to load the profile fully affecting several users who all have an idea on how to fix the problem even though they know nothing about Terminal Server or servers or heck even their computers but everybody is an expert on telling people what to do when they don't have to back it up. It amazes me to no end and the constant calls slow down the process of fixing the issue. I digress. I'm happy with my 6300 thus far and will be persistent on getting onto the Beta list if possible, not because I think I know everything. The exact opposite is true, hopefully the newness I bring to SDR radios can be a benefit if I know the outline if what FlexRadio is looking for during testing.

Jim Best

Where this thread is concerned, I have enjoyed using the delete button. After a point 
this becomes a waste of bandwidth.

Burt Fisher

Ok Gerald you did all one could reasonably do.
As far as my earlier comments:
https://www.youtube.com/watch?v=V3FnpaWQJO0

KY6LA_Howard

Gerald '' Had 2 iPads bricked so far. Neither ever totally recovered. Still glitchy. Gave them to grandkids. So it happens to the best of them.

KC9NRN

Ok Burt, that had me laughing!!

Burt Fisher

I wasn't the only one that was wrong, why am I the only one admitting it?

Kevin Va3KGS

Yes Burt, your right, this community has the technical base and entertainment.  That's why I (we) keep coming back here ever day. My Radio(s) are always on for stimulation while surfing through the community forum.

Have a great day

Kevin, Va3KGS

Philip KA4KOE

Pull the pin and THEN release the spoon. Then start counting. Remember the Holy Hand Grenade of Antioch??? From Brother Maynard?

https://www.youtube.com/watch?v=xOrgLj9lOwk

Philip KA4KOE

When Neil got wrapped up in politics, etc., I lost all respect for him.

Philip KA4KOE

In my case, they are indeed doing this very thing!

Ernest

Steven,
Why aren't you working at Flex?
Since you and your other collaborators here seem to have the answer to (all) the problems.
Do you honestly feel or think that these bugs you are alleging; were set purposely? Criticising the Company and the Alpha and Beta testers for not finding these problems, clearly show your lack of appreciation for their efforts and time. 
As others have said, ", one has to wonder if a lot of these are not caused by the operators not following directions."
The funny part of all this, is that when I check other manufactures radio sites, (if they bother to have one) you don't hear a peep from these individuals there.

Jay Nation

I was thinking that once I was clear I'd just yell ...
OK! you can put it down now and step back slowly! 
Next thing I know, someone taps me on the back, hands me a grenade, and sez ...
Huh? I couldn't hear what you said. 

Run Away!!!

73, Jay - NO5J

Simon Lewis

just don't count to 5

Paul Christensen, W9AC

To Gerald and Team:

Perhaps its worth re-evaluating the definitions being used for new releases.  I suggest keeping alpha within the 50 or so testers.  Once there's consensus between Flex and the testers, release it to the public only as beta -- and not an official release.  We in the Flex Community then have the option of sticking with an existing, stable release, or try the beta and help move it from beta to the next official release.  IOW, no new release is an official release.  Once beta and its quick fixes are made stable, it then moves to official release status.  This redefines the expectation among users (i.e., "it's beta and still in development...don't expect perfection from this release").  If anyone is in any way hesitant about possible imperfections in the beta release, then wait.  Wait for it to change from beta into the official release.   

The problem as I see it, is the assumption that every new SSDR release is completely stable.  We all download the latest release and then get these knee-**** reactions from users after every single new, public release -- every one of them.   Why not keep the public release experimental, and state it as such by calling it beta? 

What I've just described is Elecraft's release process.  In the early years of the K3, these same issues and annoyances proliferated the messages boards.  It quickly got out of control.  Subsequent to the change, their release process has been in place for many years and it works well (albeit not perfect) among their user base.

Paul, W9AC 

 

Varistor

The question of how many testers are testing the product is not that interesting. The real question is who and how designs the tests plans, how are the communicated to the testers, and how do the testers follow them. If testers are simply told "Go ahead, test" the overall success and quality of the testing process will be very low.

Test and security planning starts the minute you design (not write!) your system. IBM did a study that fixing a bug during the development process is 80 times cheaper than once the software has been released.

KC9NRN

100% agreed Philip.

KC9NRN

A fresh set of eyes is never a bad thing, some of the reasons for not taking on new beta testers make zero sense to me but heck, I'm just a network engineer, what do I know.

John-K3MA

Burt, whom else do you believe was wrong and why?  I am not sure I understand why you even think you were wrong?

Jay Nation

Until I'm given the chance to prove my competency, I accept the fact that I'm judged less competent. No one will know the real level of my competency until I'm given the opportunity to demonstrate just how competent, I really might be. The same classification of "less competent" applies to everyone else being underutilized.  Call me names, or watch me work. I don't understand why so many fingers need to pointed at the alpha/beta testers, before any real cause of the problem is known. I'd rather spend time with "Team Less Competent", "Team Finger Pointing" isn't likely to be any more competent, and might be a lot less likely to volunteer to do any of the work. Who wants to be the one being pointed at the next time something fails to fail during testing. The current crop of alpha/beta testers got us all this far. They might have a clue about what really went wrong, and how to avoid a repeat down the road.

So did everyone enjoy seeing this Firework display here in the Community?

73, Jay - NO5J 

John-K3MA

Gerald thank you for your official response regarding the Maestro issue.  My comments were more based upon the total situation in release 1.8 of which I understand the Maestro is of major concern.  Would you also care to address the other item which has been a issue for many regarding the power foldback problems people are having with the release?  I did not see that mentioned specifically in your response above.

Also, I have personally heard one Beta tester mention that "they were not aware of the power foldback change prior to the release".  I am not sure if that comment applies to only him or if Flex wanted him as a beta tester to test the power foldback functionality or not.  All I can be certain of from his statement is that he is a beta tester and did not beta test the power foldback functionality.  I only mention it here so that if you wish you could cover the breath of the testing of the power foldback in your response.

K7NXT

I think the reason that this thread refuses to die is because there are two diametrically opposed viewpoints about the embedded and network software.  

But first I would ask the group - would it be different if we were discussing software that runs your car?  Or is Ham radio a special case?  So here are the two viewpoints that I see:

Viewpoint 1. Ham as a hobby is all about experimenting, tweaking, and getting involved in technology. Therefore, bring on the incremental releases, play with them, learn about networking, WiFI, VPNs, etc. and revel in the technology including the problems and solutions.

Viewpoint 2. People are willing to pay premium prices for premium products - products that just work. Once manufacturers started putting embedded computers into their products and rely on networks, they also take responsibility for the complexity that results.  

I see that the discussion about the number and types of testers needed, the labels given to people who disagree, etc. as evidence of these two viewpoints.


Many of us waited for months to get a Flex on the premise  that the product would be solid when we received it.  I think most of us still have the patience for Flex to work out problems that exist, and because of that, I concur with the idea of fewer and more robust releases.

73s.

EA4GLI

Paul, that is an excellent suggestion and I hope FRS seriously considers it.

Gerald-K5SDR

Paul, I truly appreciate the suggestion. However, we have been doing multiple software releases per year as a company for over 13 years now. In the early days we did public beta on every release just as you suggested and it was a total support nightmare because people don't properly self select in or out and the world thinks you are perpetually in beta. When we switched to a private beta on PowerSDR believe me our overall quality of life improved. In a perfect world where everyone follows the rules as you proposed it would be great. Public beta did not work well in our experience for the reasons stated. Remember that with all their infinite testing resources and billions of customers, both Apple and Microsoft bricked two of my devices during OS upgrade respectively? The latter is still bricked. If a frog had wings he wouldn't bump his but. ;>)

Gerald-K5SDR

We provide release notes and a shared testing spreadsheet by issue for every single beta release. I don't know who the tester is but he must not have paid attention. If the test team had reported a problem with this feature, we would have held the release or removed the feature. 100% of the testers had the software and it requires no special expertise to know if it works or not.

Burt Fisher

Others were saying Flex needed to diversify their testers and Gerald said they already had (this among other things).

KC9NRN

Yet it works for Elecraft, EVGA, NVIDIA, AMD, Intel, Microsoft, Apple and a large assortment of other companies too long to list here. The thing is Paul is spot on, it's not just Elecraft who does this, heck I have a beta driver for NVIDIA sitting in my system tray right now, I can choose to install or not and I accept the risk for what happens.

I am on the fast ring on the Windows 10 Insider program, it too is prompting me to install the latest Windows 10 build, again I take the risk, not Microsoft even if bad news does affect them, they can say I accepted the risk of using Alpha or Beta software, something I have done for 25 years, I know I'm not alone in this community that is willing to accept the risk, you can't **** if you accept the agreement, maybe you need to change your agreement.

You upgrading two devices and now they're bricked from Apple AND Microsoft is so far from the norm you should play the lottery that's how crazy the odds are and the Windows 10 machine, is NOT bricked. It can be formatted and all is well, the Maestros that have to be returned to be fixed, not so much, maybe you format them but for the user, they are truly bricked.

The thing is when performing an upgrade on top of an OS that has had how many things installed over time, you shouldn't be shocked when bad things happen and by accepting Microsoft's terms them there is the breaks.

When we install new versions of your software are we supposed to perform an upgrade or uninstall the previous version first? 50 people testing software is too few, maybe for Alpha its fine and I'm sure for 13 years maybe it was fine but the hardware you're selling is no longer a client, it's server class and 50 people beta testing for not only the 6000 but also the Maestro, is ridiculously small unless of course your user base is just a few hundred which its not.

I am spending $3,800.00 or so for the 6300 and Maestro, compared to those who bought the 6700 and Maestro I'm small potatoes but to me that $3,800 represents a lot of saving so hearing that the beta user size is 50 people, shocked me, I thought Flex had more resources than that to support a larger beta group.

Perhaps a re-think is in order considering you added a another piece of hardware that has the promise of WAN, is a head unit for your signature line of transceivers and has the same amount of testers that were there before the Maestro.

How many of the 50 testers purchased Maestros? If all 50 do you really think that was enough of a sample size to vet all possibilities? Rhetorical question.

I am not writing this to **** on your parade or to say you're wrong, not even a little but I do think that your response claiming well Apple and Microsoft bricked my stuff so how we're doing things is fine, a public beta works for hundreds of companies, maybe you could re-evaluate? 

Bill -VA3WTB

I like what Gerald said the other day, Each of the testers could have upgraded hundreds of times without seeing any problems. And his point, that this could happen to any software company, and as we see, it does, not only a Flex problem.

KC9NRN

Bill,

The amount of testers using the same hardware and OS and not changing them out means that likely yes, they can upgrade a few hundred times and get similar results.

The facts are, the more who actually test it, on different systems the more results you'll get that are valid. Unless each beta tester changed OS, hardware or heck the network switch or something like that results are easy to duplicate.

Maybe I'm not seeing it the same way as Gerald is, maybe I have been involved in too many beta tests where fewer meant less valid results thus they added testers to get more valid results or at least more results from differing systems to say the results given were valid.

I love the hardware and the software, I just don't agree on the method of using so few people to test and validate them.