Is a more structured Alpha/Beta program needed?

John-K3MA

I must admit I have no idea how the current Flex Alpha and Beta test programs operate.  I do not know it the selection of the testers is done using a criteria, if they go thru performance evaluations, if they are managed actively or even it they are paid for their time.

At the Flex Dinner in Dayton it was announced that a more rapid release of updates was going to be a focus area for Flex.  I believe it was mentioned that the target was to release a software upgrade once every few months.

WIth the recent 1.8 release it seems a large number of customer are having issues with the changes and additionally it appears from the community posts that the issues are in several functional areas. 

I certainly like the idea of Flex releasing more rapid updates but I am thinking that any improvement in more rapid releases will be lost if the releases are issue prone such as it seem is the case with the recent 1.8 release.  Not only does this leave a bad taste in the customers mouth but resources at Flex need to be deployed to fire fight and fix these issue and might end up effecting the timing of other functionality development efforts.  Also, feedback from the user base might be delayed as people decide it is in their best interest to wait for others to load the updates and see if they work properly before doing so themselves.

Hence, my question.  Does the Alpha and Beta testing process need to be more structured with specific criteria, test scripts, time and deliverable requirement in order to support the quicker release focus?

I am not really expecting an answer either agreeing or defending the process only pointing out the need for Flex management to critically look at the root causes of this recent failure and what can be done to improve the process to support the achievement of the goal of quicker releases.

Regards, John K3MA

[Deleted User]

I couldn't agree with you more. I have been saying this all along. Why weren't some of these bugs caught by the Alpha testers and by Flex. They really need to focus on getting these bugs corrected and not on the next feature. Maybe it is time for new Alpha and Beta test program as these issues are not being caught . . . Just my opinion.

Peter

I totally agree!

Mike NN9DD

I agree that releases with bugs do leave a sour taste in ones mouth. Having said that, the fact that these issues Are so random with the exception of the Mastro issue, one has to wonder if a lot of these are not caused by the operators not following directions, system issues on the users end or some other issue in their PC or process. I can't recall ever having an issue with an install or the usage of my equipment afterwards. I will admit to following the directions and always doing a clean install and I may be a bit more tech savvy then some having issues. But the point still stands the randomness of these issues makes me think some of the errors are on our end as operators Mike N9DFD

Chris DL5NAM

yes, i agree. To many claqueure at Beta test team.

Jay Nation

Steven
They were caught by the beta testers.
You may know some of them.
And rather quickly too. Maybe the alpha testers aren't really necessary.

Ok, you hold the grenade, I'll hold the pin, No just stay right here, I'll be right back, 3-1000, 4-1000, 5-1000.

73, Jay - NO5J 

[Deleted User]

So if they were caught why was it released . . . . Or are all of us the BETA testers. . . . If so we caught the bugs

Jay Nation

You figured it out, too!

73, Jay - NO5J

KD7CAO

I agree with you. I just realized I sent a post to the support desk rather than on here, but along those lines I wish the current and Alpha Maestro software matched the current/Alpha Windows Software. I downloaded the newest update from the Flex Radio Website this morning for my Desktop and it is 1.8.3 but the newest version on the Maestro is 1.9.0.81 (Alpha). Guess what I installed the Maestro but now to use the Windows computer I have to downgrade the radio. It is frustrating that the versions never match.

[Deleted User]

Yea . . . LOL

Jay Nation

I never even got the chance to try Maestro with 1.8.3, before it was pulled. My guess is I'll never get to try it. 

SmartSDR is going to stay at 1.7.30, until at least 1.8.4, too.

I dislike UPS, just that much.

73, Jay - NO5J 

Flex_Fan

Nothing more true could be said John.

When it comes to software there can not be too many people testing testing and re testing.  
Put the software in the hands of more beta testers with all types of hardware configurations.  

To have software brick equipment so it has to be sent to the factory to be fixed is no way to do software releases and not cost effective.  

Sorry Flex, I love your products but you dropped the ball on this one. And using excuses that this happens to Microsoft and Apple etc are just excuses. Bad programming is just that no matter who does it!

[Deleted User]

Agree

KY6LA_Howard

The Beta Testing group involved more than 50 testers.  None had the Bricking issue.  None had the Power foldback issue that disabled the tube amps.

BUT even when Microsoft and Apple Beta test in the millions of beta testers some things fall thru the cracks.

The bricking issue seems to be a timing issue with a very small number of Maestro's during the update process  The power foldback issue only seems to happen when people use older tube amps...and it appears to be happening when there is a lot of common mode current due to lack of proper common mode chokes

Perhaps the issue with the Beta group is that we are too competent and our stations mostly are very modern with few glitchy stuff ....

In my professional life by far my best beta tester was a secretary in our office who could break anything she touched...if a device could survive her and still operate then I gave it to my XYL who would then try to break it...

I would suggest that Flex add less competent testers  such as several of the people who have responded above... I am sure that they would be happy to help...

Jay Nation

I'm certainly less competent! 
I'll volunteer.
For all the good it will do me.
Not going to hold my breath waiting though.
It's hard to know that there is a problem, before there is a problem. 
The arrow of time is not adjustable.
73, Jay - NO5J

Flex_Fan

When I worked in hardware/Software development  we did just that. Handed the software and an installation sheet to the most un savy people we could find. They always found the bugs because they did not assume anything.  There is no excuse for buggy software to kit the consumer.   Sorry make all the excuses in the world but the consumer deserves better.

Rick Hadley - W0FG

This thread has me thinking...a dangerous idea, I realize.  I'm not on the Microsoft Windows early release program because I don't like dealing with OS issues that can **** EVERYTHING up, but I wouldn't at all mind being on the Flex Alpha team, because I know that in most cases I can revert to an earlier release and keep operating if a bug crops up in a new version.  Perhaps Flex should consider implementing a Microsoft-style model and make the pre-release versions available to a greater audience, with the understanding that there most likely will be bugs, and not to sign up for it without a waiver of complaint when they do show up.  Something like that might help catch the less obvious problems earlier.

Mike va3mw

All

Take a deep breath ... please.  All software has bugs and rest assured, Flex would not ship out anything at anytime if they thought it would impact the community.  That would not help anyone.  

In the software world, you have software unit testing and software functional testing.  Both are part of the product life cycle.

Howard's comments are spot on.  Flex does not have control over the user environment and we all know (or should know) RF can be a strange beast to manage.  

Flex was very smart to allow the community to be able to change versions on the fly, both with SSDR and the Maestro.  That is why you get to see the last 5 versions.   It makes it easy to roll back should a bug get past the palace guards.

World order will be restored in due time.    Just roll back and carry on.

73 all

Mike va3mw

Jay Nation

I'm guessing, but I bet the Change Log for the next release will include an item stating ...

Corrected: a snippit of code, that bricked a few of the first production Maestro's.

and another.

Optimized: The high SWR power foldback code, and also made it a user selectable option.

Not to bad for a Friday release, before a holiday weekend.

Might also want to refactor the decision to release updates the Friday before a holiday weekend.

Especially holidays, that already feature plenty of other stuff that goes boom. 

73, Jay - NO5J

Martin AA6E

I think we have to let Flex management do its thing here.  They have to decide how to structure the test process to get the best product at a tolerable cost.  An open beta program would probably be awful to manage. 

Users are always the final testers.  The risk averse folks will hold back while the bleeding edge guys find some deep bugs.  That always happens.  And we users are always clamoring for new features and releases.

EA4GLI

Anyone feels like proposing an alternative that will provide bulletproof fault-free releases?

Walt - KZ1F

I made some suggestions to Steve, on this subject, offline last evening. Some things don't readily lend themselves to public discourse.

Joe WD5Y

Could it be a possibility to give a general flex 6000 user the choice of loading a beta version or even alpha version with the total written understanding of the possible undesirable consequences with no guarantees? Possibly with this in the works the community feedback may be of some help resolving issues, might also stop some of the armchair quarterbacking or pointing fingers and I do not mean this in a derogatory manner, it is human nature for us all to stand back and point fingers when we are not part of the solution. 73's Joe WD5Y

Jim Gilliam

I wonder what percentage of the complaints are more to cockpit errors and their lack of experience in dealing with the technology.

Burt Fisher

I wonder if the testers are friends of Flex. If you want a good test give it your critics. If "Mikey likes it" it has to be good.
https://www.bing.com/videos/search?q=mikey+likes+it&view=detail&mid=A0BF4C2E7DB0365A6195A0BF...

Burt Fisher

You have to stop blaming the operators. The goal should be to make it reasonable for the average ham to use it. You should not have to do three spins to the left, two to the right and face Mecca to get it to work.

Burt Fisher

I am less competent I teach 9 year olds.

Burt Fisher

Give it to the complainers not the old boys

Burt Fisher

In general they are not selling to Registered Professional Engineers

SDR Fan

You test it since your heads in the sand, Even if you find problems you will say its not possible that Flex had any problems and then you will give us more of your excuses and how we are the problem. You have no objectivity.

Flex_Fan

More testing and better testers. If we found the problem after the release so can testers find it before release if they know how to test software.