Is a more structured Alpha/Beta program needed?

  • 8
  • Question
  • Updated 3 years ago
  • Answered
I must admit I have no idea how the current Flex Alpha and Beta test programs operate.  I do not know it the selection of the testers is done using a criteria, if they go thru performance evaluations, if they are managed actively or even it they are paid for their time.

At the Flex Dinner in Dayton it was announced that a more rapid release of updates was going to be a focus area for Flex.  I believe it was mentioned that the target was to release a software upgrade once every few months.

WIth the recent 1.8 release it seems a large number of customer are having issues with the changes and additionally it appears from the community posts that the issues are in several functional areas. 

I certainly like the idea of Flex releasing more rapid updates but I am thinking that any improvement in more rapid releases will be lost if the releases are issue prone such as it seem is the case with the recent 1.8 release.  Not only does this leave a bad taste in the customers mouth but resources at Flex need to be deployed to fire fight and fix these issue and might end up effecting the timing of other functionality development efforts.  Also, feedback from the user base might be delayed as people decide it is in their best interest to wait for others to load the updates and see if they work properly before doing so themselves.

Hence, my question.  Does the Alpha and Beta testing process need to be more structured with specific criteria, test scripts, time and deliverable requirement in order to support the quicker release focus?

I am not really expecting an answer either agreeing or defending the process only pointing out the need for Flex management to critically look at the root causes of this recent failure and what can be done to improve the process to support the achievement of the goal of quicker releases.

Regards, John K3MA
Photo of John-K3MA

John-K3MA

  • 102 Posts
  • 29 Reply Likes

Posted 3 years ago

  • 8
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
I couldn't agree with you more. I have been saying this all along. Why weren't some of these bugs caught by the Alpha testers and by Flex. They really need to focus on getting these bugs corrected and not on the next feature. Maybe it is time for new Alpha and Beta test program as these issues are not being caught . . . Just my opinion.
(Edited)
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
I wonder if the testers are friends of Flex. If you want a good test give it your critics. If "Mikey likes it" it has to be good.
https://www.bing.com/videos/search?q=mikey+likes+it&view=detail&mid=A0BF4C2E7DB0365A6195A0BF...
Photo of Jon - KF2E

Jon - KF2E

  • 679 Posts
  • 222 Reply Likes
Good point Burt! 

Jon...kf2e
Photo of Ernest - W4EG

Ernest - W4EG

  • 215 Posts
  • 39 Reply Likes
Steven,
Why aren't you working at Flex?
Since you and your other collaborators here seem to have the answer to (all) the problems.
Do you honestly feel or think that these bugs you are alleging; were set purposely? Criticising the Company and the Alpha and Beta testers for not finding these problems, clearly show your lack of appreciation for their efforts and time. 
As others have said, ", one has to wonder if a lot of these are not caused by the operators not following directions."
The funny part of all this, is that when I check other manufactures radio sites, (if they bother to have one) you don't hear a peep from these individuals there.
Photo of Peter

Peter

  • 79 Posts
  • 5 Reply Likes
I totally agree!
Photo of Mike Hoing

Mike Hoing

  • 277 Posts
  • 43 Reply Likes
I agree that releases with bugs do leave a sour taste in ones mouth. Having said that, the fact that these issues Are so random with the exception of the Mastro issue, one has to wonder if a lot of these are not caused by the operators not following directions, system issues on the users end or some other issue in their PC or process.

I can't recall ever having an issue with an install or the usage of my equipment afterwards. I will admit to following the directions and always doing a clean install and I may be a bit more tech savvy then some having issues. But the point still stands the randomness of these issues makes me think some of the errors are on our end as operators

Mike
N9DFD
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
You have to stop blaming the operators. The goal should be to make it reasonable for the average ham to use it. You should not have to do three spins to the left, two to the right and face Mecca to get it to work.
Photo of K1UO - Larry

K1UO - Larry

  • 866 Posts
  • 135 Reply Likes
...or was that three to the right and two to the left.... ..  Mecca you say?
Photo of Chris DL5NAM

Chris DL5NAM

  • 670 Posts
  • 152 Reply Likes
yes, i agree. To many claqueure at Beta test team.
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
Steven
They were caught by the beta testers.
You may know some of them.
And rather quickly too. Maybe the alpha testers aren't really necessary.

Ok, you hold the grenade, I'll hold the pin, No just stay right here, I'll be right back, 3-1000, 4-1000, 5-1000.

73, Jay - NO5J 
Photo of Philip KA4KOE

Philip KA4KOE

  • 175 Posts
  • 22 Reply Likes
Pull the pin and THEN release the spoon. Then start counting. Remember the Holy Hand Grenade of Antioch??? From Brother Maynard?

https://www.youtube.com/watch?v=xOrgLj9lOwk
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
I was thinking that once I was clear I'd just yell ...
OK! you can put it down now and step back slowly! 
Next thing I know, someone taps me on the back, hands me a grenade, and sez ...
Huh? I couldn't hear what you said. 

Run Away!!!

73, Jay - NO5J
Photo of Simon Lewis

Simon Lewis

  • 495 Posts
  • 118 Reply Likes
just don't count to 5
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
So if they were caught why was it released . . . . Or are all of us the BETA testers. . . . If so we caught the bugs
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
You figured it out, too!

73, Jay - NO5J
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
Yea . . . LOL
Photo of KD7CAO

KD7CAO

  • 34 Posts
  • 22 Reply Likes
I agree with you. I just realized I sent a post to the support desk rather than on here, but along those lines I wish the current and Alpha Maestro software matched the current/Alpha Windows Software. I downloaded the newest update from the Flex Radio Website this morning for my Desktop and it is 1.8.3 but the newest version on the Maestro is 1.9.0.81 (Alpha). Guess what I installed the Maestro but now to use the Windows computer I have to downgrade the radio. It is frustrating that the versions never match.
Photo of David Decoons wo2x

David Decoons wo2x, Elmer

  • 1403 Posts
  • 306 Reply Likes
If you are not an alpha tester then your Maestro should not have received the alpha build. I suggest you contact Flex about that.

Dave wo2x
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
I never even got the chance to try Maestro with 1.8.3, before it was pulled. My guess is I'll never get to try it. 

SmartSDR is going to stay at 1.7.30, until at least 1.8.4, too.

I dislike UPS, just that much.

73, Jay - NO5J 
Photo of Flex_Fan

Flex_Fan

  • 9 Posts
  • 6 Reply Likes
Nothing more true could be said John.

When it comes to software there can not be too many people testing testing and re testing.  
Put the software in the hands of more beta testers with all types of hardware configurations.  

To have software brick equipment so it has to be sent to the factory to be fixed is no way to do software releases and not cost effective.  

Sorry Flex, I love your products but you dropped the ball on this one. And using excuses that this happens to Microsoft and Apple etc are just excuses. Bad programming is just that no matter who does it!
Photo of Varistor

Varistor

  • 334 Posts
  • 74 Reply Likes
The question of how many testers are testing the product is not that interesting. The real question is who and how designs the tests plans, how are the communicated to the testers, and how do the testers follow them. If testers are simply told "Go ahead, test" the overall success and quality of the testing process will be very low.

Test and security planning starts the minute you design (not write!) your system. IBM did a study that fixing a bug during the development process is 80 times cheaper than once the software has been released.
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
Agree
Photo of KY6LA - Howard

KY6LA - Howard, Elmer

  • 3755 Posts
  • 1617 Reply Likes

The Beta Testing group involved more than 50 testers.  None had the Bricking issue.  None had the Power foldback issue that disabled the tube amps.

BUT even when Microsoft and Apple Beta test in the millions of beta testers some things fall thru the cracks.

The bricking issue seems to be a timing issue with a very small number of Maestro's during the update process  The power foldback issue only seems to happen when people use older tube amps...and it appears to be happening when there is a lot of common mode current due to lack of proper common mode chokes

Perhaps the issue with the Beta group is that we are too competent and our stations mostly are very modern with few glitchy stuff ....

In my professional life by far my best beta tester was a secretary in our office who could break anything she touched...if a device could survive her and still operate then I gave it to my XYL who would then try to break it...

I would suggest that Flex add less competent testers  such as several of the people who have responded above... I am sure that they would be happy to help...



Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
I'm certainly less competent! 
I'll volunteer.
For all the good it will do me.
Not going to hold my breath waiting though.
It's hard to know that there is a problem, before there is a problem. 
The arrow of time is not adjustable.
73, Jay - NO5J
Photo of Flex_Fan

Flex_Fan

  • 9 Posts
  • 6 Reply Likes
When I worked in hardware/Software development  we did just that. Handed the software and an installation sheet to the most un savy people we could find. They always found the bugs because they did not assume anything.  There is no excuse for buggy software to kit the consumer.   Sorry make all the excuses in the world but the consumer deserves better.
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
I am less competent I teach 9 year olds.
Photo of Mike W9OJ

Mike W9OJ

  • 104 Posts
  • 38 Reply Likes
The OM-4000 is not an older tube amp.
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3851 Posts
  • 930 Reply Likes
Howard made some good points
Photo of Dave - W6OVP

Dave - W6OVP

  • 316 Posts
  • 99 Reply Likes
In my professional life by far my best beta tester was a secretary in our office who could break anything she touched...

ABSOLUTELY CORRECT! We used to rotate draft Instruction and Installation Manuals among normally bright secretaries who knew nothing about computers or programming. If they could not follow the directions to perfect conclusions without any help we knew WE were not good enough and had more work to do before retesting. And NO excuses were allowed for us!

I have told this story for years to haughty technical cognoscenti and received back only cynical laughs of disbelief. (BTW, from what I've seen, as with virtually all of the computer industry I doubt any of the FLEX product instructions would come even close to passing this tough test.)
Photo of EA4GLI - 8P9EH - Salvador

EA4GLI - 8P9EH - Salvador

  • 1782 Posts
  • 547 Reply Likes
But is Radio Equipment in our hobby really an appliance for the masses? I think ease of use is good, simpler interfaces mostly welcomed, but I think that an SDR amateur radio is just not a device that anyone that reads a manual can use. We are talking about radio frequency, voltages, antennas, patch cables, etc... With a networked radio a familiarity with networking concepts goes a long way. It might be in the best interest of many to read about and understand concepts like TCP, UDP, Latency, Jitter, etc...

I don't disagree that a fresh set of eyes in the Alpha team might be necessary, I am sure Flex has to look at what has happened and make some decisions and most of the advice from users in here is pretty sound.

But I do think that sometimes we forget that this is just no like a usb webcam that you plug in, drivers install and off you go to skype to your hearts content.
Photo of EA4GLI - 8P9EH - Salvador

EA4GLI - 8P9EH - Salvador

  • 1782 Posts
  • 547 Reply Likes
But is Radio Equipment in our hobby really an appliance for the masses? I think ease of use is good, simpler interfaces mostly welcomed, but I think that an SDR amateur radio is just not a device that anyone that reads a manual can use. We are talking about radio frequency, voltages, antennas, patch cables, etc... With a networked radio a familiarity with networking concepts goes a long way. It might be in the best interest of many to read about and understand concepts like TCP, UDP, Latency, Jitter, etc...

I don't disagree that a fresh set of eyes in the Alpha team might be necessary, I am sure Flex has to look at what has happened and make some decisions and most of the advice from users in here is pretty sound.

But I do think that sometimes we forget that this is just no like a usb webcam that you plug in, drivers install and off you go to skype to your hearts content.
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
Although...I would argue that it IS. Thirty plus years ago, not so much. many people built their rigs or fixed their rigs, in the tests to get licensed you had to know sh&t. And op had to know how to even tune their rig, antenna tuners...forget about it. Now, the testing is idiot simple, rigs do all the work for you, turn it on, press the mike off you go. Radios now are appliances. If you can press a button, you can start the radio. If you can start the browser, you can start SSDR. There is an expectation that button pressing is the only requirement.

It is this expectation that a t puts the onus of stability on the people that design and build the radio's. If a legacy radio had a MTBF of a couple of months, word would eventually get out. If ...where windows had to be rebooted once or twice per day, word, eventually, got out. The brilliance of Microsoft lies in their proven ability to convince the consumer market it was perfectly acceptable to have unstable software, moreover, that it should be expected. In the commercial environment it was/is neither expected nor acceptable. In the consumer environment, bsod, is a punch line. In the commercial environment, it signals time to choose a new vendor.

So would it be helpful for everyone to have a Master's degree level of computer science understanding? Sure, if they were planning a career in it. But it's probably not required to understand if one time out of one hundred when they started their browser it crashed that something wasn't right.
(Edited)
Photo of EA4GLI - 8P9EH - Salvador

EA4GLI - 8P9EH - Salvador

  • 1782 Posts
  • 547 Reply Likes
OK Walt, but I am not asking people to go back to school and get a degree in CIS, I am talking about reading about networking which is an important part of how 6xxx radios work.
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
I didn't imply a degree, just that level of understanding. When I mentioned Nagle's Algorithm last night, did you immediately know what I was referring too? In that case you may well have but I am kind of curious, right now how many are googling Nagle. Would it be helpful? Yeah, maybe. My point was merely should it be a requirement to understand the Flex?
BTW, my friend, great conversation...marathon conversation. Thanks, that was completely fun and enjoyable.
(Edited)
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
Clearly the intellect here is beyond me that's why you need some testers that are more average. I once raced a Corvette, I was in a VW bug. The race was 1/4 mile, I won because the Corvette did not take into account the gravel surface and spun his tires. Flex is the Corvette but they need to take into account the average user.
Photo of EA4GLI - 8P9EH - Salvador

EA4GLI - 8P9EH - Salvador

  • 1782 Posts
  • 547 Reply Likes
Why is a Yagi antenna called Yagi?
Why should you have impedance match on the antenna port of your radio?
Why should you ground your equipment?

I would imagine that for the majority here those are simple questions to answer.
But what about these......

What is Network Jitter? 
What is Class A private IP address?
What is the difference between CAT5, CAT6, CAT7?
What is the meaning of APIPA? And DHCP?
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
I wish I knew more about M theory, which is why I have a couple of Lisa Randall books. I wish I had Neal DeGrasse Tyson's level of understanding of Cosmology.
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
I wish everyone had Neal DeGrasse Tyson's level of understanding, of anything and everything. It might make things like learning, and curiosity popular again.

73, Jay - NO5J
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
I'd phase that differently Jay. If more people had curiosity, more people would have something close to his level of understanding. Plus or minus, we all have the inherent ability. It is indifference that is the friction. How many people watch Nova, Frontline?. I don't have the advanced physics for Cosmology yet.
(Edited)
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
Walt

I've always been curious, since birth even. I expect I'l be curious the day I die. I try to learn everything, there is a big difference between "I've heard of that, or "I saw that once" and learning. Learning is the step that follows, Huh? Whazzat. Just because some new discovery didn't kill you, doesn't mean it's safe to ignore it. There is more to survival than avoiding being eaten. "HOT! mustn't touch, No!", should be learned early, but knowing why, might not do you any harm either.  There's more to getting old, than just dumb luck. Naw! I don't wanna mess with all that, just do it for me, may not get it done.

Imagine what life could be like, If we all paid a little more attention.

There might have been less Word Salad, if I'd done a better job of learning to explain myself!

73, Jay - NO5J
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
Jay, honest, I was not referring to you at all.
Photo of KC9NRN

KC9NRN

  • 324 Posts
  • 32 Reply Likes
Less competent, always a great way to insult people, who judges their competency?
Photo of Philip KA4KOE

Philip KA4KOE

  • 175 Posts
  • 22 Reply Likes
When Neil got wrapped up in politics, etc., I lost all respect for him.
Photo of KC9NRN

KC9NRN

  • 324 Posts
  • 32 Reply Likes
100% agreed Philip.
Photo of KC9NRN

KC9NRN

  • 324 Posts
  • 32 Reply Likes
A fresh set of eyes is never a bad thing, some of the reasons for not taking on new beta testers make zero sense to me but heck, I'm just a network engineer, what do I know.
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
Until I'm given the chance to prove my competency, I accept the fact that I'm judged less competent. No one will know the real level of my competency until I'm given the opportunity to demonstrate just how competent, I really might be. The same classification of "less competent" applies to everyone else being underutilized.  Call me names, or watch me work. I don't understand why so many fingers need to pointed at the alpha/beta testers, before any real cause of the problem is known. I'd rather spend time with "Team Less Competent", "Team Finger Pointing" isn't likely to be any more competent, and might be a lot less likely to volunteer to do any of the work. Who wants to be the one being pointed at the next time something fails to fail during testing. The current crop of alpha/beta testers got us all this far. They might have a clue about what really went wrong, and how to avoid a repeat down the road.

So did everyone enjoy seeing this Firework display here in the Community?

73, Jay - NO5J 
Photo of K7NXT

K7NXT

  • 37 Posts
  • 27 Reply Likes
I think the reason that this thread refuses to die is because there are two diametrically opposed viewpoints about the embedded and network software.  

But first I would ask the group - would it be different if we were discussing software that runs your car?  Or is Ham radio a special case?  So here are the two viewpoints that I see:

Viewpoint 1. Ham as a hobby is all about experimenting, tweaking, and getting involved in technology. Therefore, bring on the incremental releases, play with them, learn about networking, WiFI, VPNs, etc. and revel in the technology including the problems and solutions.

Viewpoint 2. People are willing to pay premium prices for premium products - products that just work. Once manufacturers started putting embedded computers into their products and rely on networks, they also take responsibility for the complexity that results.  

I see that the discussion about the number and types of testers needed, the labels given to people who disagree, etc. as evidence of these two viewpoints.


Many of us waited for months to get a Flex on the premise  that the product would be solid when we received it.  I think most of us still have the patience for Flex to work out problems that exist, and because of that, I concur with the idea of fewer and more robust releases.

73s.
Photo of DrTeeth

DrTeeth

  • 1687 Posts
  • 389 Reply Likes
Howard, you have hit the nail on the head perfectly with that post.
Photo of Rick Hadley - W0FG

Rick Hadley - W0FG

  • 600 Posts
  • 130 Reply Likes
This thread has me thinking...a dangerous idea, I realize.  I'm not on the Microsoft Windows early release program because I don't like dealing with OS issues that can screw EVERYTHING up, but I wouldn't at all mind being on the Flex Alpha team, because I know that in most cases I can revert to an earlier release and keep operating if a bug crops up in a new version.  Perhaps Flex should consider implementing a Microsoft-style model and make the pre-release versions available to a greater audience, with the understanding that there most likely will be bugs, and not to sign up for it without a waiver of complaint when they do show up.  Something like that might help catch the less obvious problems earlier.
Photo of Martin Ewing AA6E

Martin Ewing AA6E

  • 323 Posts
  • 76 Reply Likes
I think we have to let Flex management do its thing here.  They have to decide how to structure the test process to get the best product at a tolerable cost.  An open beta program would probably be awful to manage. 

Users are always the final testers.  The risk averse folks will hold back while the bleeding edge guys find some deep bugs.  That always happens.  And we users are always clamoring for new features and releases.
Photo of km9r.mike

km9r.mike

  • 425 Posts
  • 62 Reply Likes
My risk management program has me on 1.6, however , the greater the reward the more risk I will take. I did not get maestro so 1.7 had little reward and I was about to do 1.8 and could have done so w/o any risk, but  some stubborn maestros have temporarily gotten in the way.  
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
And don't forget the hyperactive power foldback when feeding complex impedance's, found on many amateur radio amplifier inputs. We wouldn't want our Flex's getting a complex would we?
I'm really looking forward to 1.8.4+.
Ya see what knobs are good for now. Right?
Still, I kinda prefer my Knobs unbricked. 
So far mine are.

73, Jay - NO5J  
Photo of km9r.mike

km9r.mike

  • 425 Posts
  • 62 Reply Likes
I currently do not have an amp, but a conservative feature is a good thing. Better than the alternative. I am actually a fan of no knobs. Good thing they did not incorporate them when designing the first computers.
Photo of Mike va3mw

Mike va3mw

  • 824 Posts
  • 198 Reply Likes
All

Take a deep breath ... please.  All software has bugs and rest assured, Flex would not ship out anything at anytime if they thought it would impact the community.  That would not help anyone.  

In the software world, you have software unit testing and software functional testing.  Both are part of the product life cycle.

Howard's comments are spot on.  Flex does not have control over the user environment and we all know (or should know) RF can be a strange beast to manage.  

Flex was very smart to allow the community to be able to change versions on the fly, both with SSDR and the Maestro.  That is why you get to see the last 5 versions.   It makes it easy to roll back should a bug get past the palace guards.

World order will be restored in due time.  :)  Just roll back and carry on.

73 all

Mike va3mw
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
Yup blame it on the user environment.
Photo of Jay -- N0FB

Jay -- N0FB, Elmer

  • 539 Posts
  • 213 Reply Likes
Yes Steven, the environment is a major consideration...a multi-tier application such as SmartSDR is reliant on all portions of the system to work flawlessly or at least reasonably well.  That system includes the radio, the client application (SmartSDR), the Client computer, the client computer operating system, the network, the network latency and current utilization at the time of the update, the mode of the network (WiFi vs Direct Connect Ethernet).  There are many points of failure and most are out of the control of FlexRadio Systems.  Unless you fully understand software development in such an environment and what the challenges Flex Radio is up against, you might want to give Flex a little slack.  

I find many of your recent comments not useful.  Most appear to be borne out of ignorance. 
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
Well then don't read them . . . That is your right. Maybe you ought to stop bashing me every time I make a statement. Keep your ignorance to yourself.
Photo of EA4GLI - 8P9EH - Salvador

EA4GLI - 8P9EH - Salvador

  • 1782 Posts
  • 547 Reply Likes
In all fairness Steven, your comment could easily be turned around and used to reply to you..... You made a comment about needed new Alpha testers... that could be understood as bashing the current Alpha testers.

I think we all need to relax a bit, enjoy the family day tomorrow, try to work those "Original 13 Colonies special event" and stop attacking each other. 
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
The recent demise of the Maestro was not caught by the Alpha testers however that is not a reflection of their competency. In fact they may be far more competent than the average amateur and therein may be the problem. If you want to test software you don't give it to Bill Gates, you give it to Joe the plumber, and not just 50 Joes. Flex made a mistake, everybody makes them, let's see if they learned from it.
Photo of Ken ve7kwa

Ken ve7kwa

  • 103 Posts
  • 32 Reply Likes
<<< If you want to test software you don't give it to Bill Gates, you give it to Joe the plumber, and not just 50 Joes. Flex made a mistake, everybody makes them, let's see if they learned from it.>>>  
  
 Burt...Given that you "know" that Flex made a mistake with their Alpha testers , why don't you tell the rest of us how many Alpha testers there are and comment on their demographics so the rest of us can determine if they "made a mistake" as well ??  We all know what happens when you ASS U ME 
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
" ASS U ME " Why I never heard that before, any more like that? How about, "for cryin out loud?" Or, "the truth shall set you free?"

I had read here that there were 50 alpha testers but sadly I didn't record it. If I am wrong I would gladly apologize. I don't see an official response. They are usually great about (another tired cliché coming), "setting the record straight."
Photo of Ken ve7kwa

Ken ve7kwa

  • 103 Posts
  • 32 Reply Likes
Have you considered Decaf Burt ??  B) 
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
I'm guessing, but I bet the Change Log for the next release will include an item stating ...

Corrected: a snippit of code, that bricked a few of the first production Maestro's.

and another.

Optimized: The high SWR power foldback code, and also made it a user selectable option.

Not to bad for a Friday release, before a holiday weekend.

Might also want to refactor the decision to release updates the Friday before a holiday weekend.

Especially holidays, that already feature plenty of other stuff that goes boom. 

73, Jay - NO5J
Photo of EA4GLI - 8P9EH - Salvador

EA4GLI - 8P9EH - Salvador

  • 1782 Posts
  • 547 Reply Likes
Anyone feels like proposing an alternative that will provide bulletproof fault-free releases?
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
Give it to the complainers not the old boys
Photo of SDR Fan

SDR Fan

  • 10 Posts
  • 2 Reply Likes
You test it since your heads in the sand, Even if you find problems you will say its not possible that Flex had any problems and then you will give us more of your excuses and how we are the problem. You have no objectivity.
Photo of Flex_Fan

Flex_Fan

  • 9 Posts
  • 6 Reply Likes
More testing and better testers. If we found the problem after the release so can testers find it before release if they know how to test software.
Photo of Jay -- N0FB

Jay -- N0FB, Elmer

  • 539 Posts
  • 213 Reply Likes
Burt, there maybe some method to your madness.  I don't know the full makeup of the Alpha testers as I'm not one, but it may make some sense to not have all of the Alpha group made up of technically savvy individuals who have shown a propensity to be overly positive.  A non-technical person might provide a helpful alternate view of the health of a given release.  

Again, I'm arm-chair-quarter-backing here.  FRS might already be doing this.
(Edited)
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
If they would weigh in on the make-up of the testers it might turn out this crash might have not been possible to foresee.
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
I made some suggestions to Steve, on this subject, offline last evening. Some things don't readily lend themselves to public discourse.
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes

So why are you making a public comment now?

To show the king has your ear?

Photo of Chris DL5NAM

Chris DL5NAM

  • 670 Posts
  • 152 Reply Likes
... what a helpful information. Your the greatest, so we now can stop here and close thread?
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
Can we at least close it early Tuesday?

73, Jay - NO5J
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
Burt, did you and your friends, when you were all 9 or 10 (were you ever a kid and did you ever have friends?), take scrap plywood and 2x4 and make a rocket for you all to go to the moon? Software development is complex and multidimensional. For that matter so is teaching, what would your reaction be if this topic were all about, ' hey, solving public education issues aren't hard, just double up on class size and fire 1/2 the teachers.

I was just ribbing you on those two stream of consciousness questions.
(Edited)
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
I didn't have many friends when I was 9 but I am making up for it now but with 9 year olds. I am by far the most popular (and strict) teacher in the 4th and 5th grades. I make mistakes in the classroom but don't hide them and secretly tell the principal but not apologize to the students. My analogy is wanting but my point is if you have something to add, add it publically but if you feel you must keep it private don't publically brag about it. I had a teen student years ago who wanted me to evaluate her singing, I said ask your other teachers, they are more qualified . She said, "no, they will all tell me I'm great, you will tell me the truth." The point being for Alpha testers it appears as Flex got the "other teachers."
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
Burt, you, likely as much or more than others, have taken comments I've made, many serious,  as challenges and confrontational. This has sparked, what I know now  to  anticipate, a reaction akin to people feeling they need to defend their girlfriends honour. That was not bragging, it was merely a stmt that as one professional to another I attempted to add perspective and advice. This is a complicated issue and there are a lot of variables in the 'equation'. Some people here know my background, more don't, and yet even more could give a rat's butt less.  I wasn't really addressing the later two categories.. Oh, to be sure, there is some rampant NPD on here. I am not one of them. I don't try to 'strut my stuff' with the PhD's in EE or ME as, I can't compete in that ring. In the software engineering arena, I can, and do, add value. So, to put it in your arena, how do you feel about warehousing and the people who volunteer that as a viable solution to a far more nuanced problem? In other words, there are arenas where you can compete intellectually, very well. This topic, perhaps, isn't one of them. However that  does, in no way, imply they don't exist. If we were in a local property tax funded public education conversation, yes, I'd assume your thoughts, experience, and suggestions would be well backed and worthy of serious consideration. Rather than precipitate a mud sling of credential questioning, you might opt to make a simple stmt and then drop it. Clearly, that didn't work so well. But rather than not respond at all, I thought framing it in terms you would get, might add perspective.
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
NPD? Narcissistic personality disorder?
If that is what it means I am so far out of your league that no further discussion is possible.
I have no idea what the following means, " So, to put it in your arena, how do you feel about warehousing and the people who volunteer that as a viable solution to a far more nuanced problem"
I am getting you think I am too stupid to have viable comments on how Flex could have avoided this mess. Therein lies the problem, with your exceptional talents this crash still got through. The PhD's, EE or ME did not find the problem. Get a clue most amateurs are not PhD's, EE or ME's. The testers ought to be a cross section unless Flex is selling to just PhD's, EE or ME's.
You have demonstrated you can walk with kings, but sadly, only kings.
Photo of Joe WD5Y

Joe WD5Y

  • 176 Posts
  • 24 Reply Likes
Could it be a possibility to give a general flex 6000 user the choice of loading a beta version or even alpha version with the total written understanding of the possible undesirable consequences with no guarantees? Possibly with this in the works the community feedback may be of some help resolving issues, might also stop some of the armchair quarterbacking or pointing fingers and I do not mean this in a derogatory manner, it is human nature for us all to stand back and point fingers when we are not part of the solution.

73's
Joe
WD5Y
Photo of KC9NRN

KC9NRN

  • 324 Posts
  • 32 Reply Likes
Excellent points, I offered to be a beta tester when discussing my purchase. I was told I would need to have been around much longer before this could happen. Seems to me your approach would be a good one, want beta software, sign on the line after reading the risk statement etc.
Photo of Jim Gilliam

Jim Gilliam

  • 932 Posts
  • 218 Reply Likes
I wonder what percentage of the complaints are more to cockpit errors and their lack of experience in dealing with the technology.
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
In general they are not selling to Registered Professional Engineers
Photo of Philip KA4KOE

Philip KA4KOE

  • 175 Posts
  • 22 Reply Likes
In my case, they are indeed doing this very thing!
Photo of KF4HR

KF4HR

  • 778 Posts
  • 264 Reply Likes
I'm sure there are Flex users out there that are constantly chompin'-at-the-bit for the next software upgrade.  I'm not one of them.  I normally sit back and read the feedback after a new software version is released, then only upgrade only if I feel there's a real benefit to the newer version.  In fact, if not for my Maestro purchase I'd probably still be running V1.5.1. 

Putting out new software versions every few months, only to have customers become their own beta testers, reduces my confidence in the product.  Personally I'd prefer Flex reduce their software upgrade frequency, perhaps just moving to an annual software release, if this would result in a more reliable product. 

Granted Flex is on the cutting edge and has its loyal customers.  On the flip side I'm constantly comparing my Flex-6700 reliability to my ICOM IC-7800.  If the bug or configuration frustration levels get too high with my Flex, I find myself reaching for my 7800's On button.   
Photo of KC2QMA_John

KC2QMA_John

  • 755 Posts
  • 309 Reply Likes
Flex may be doing this now but with all of the success of the amateur radio division it’s time to higher a bunch more software engineers to speed development SSDR and Maestro. I am happy with the performance of my 6500 I just wish the software would move along faster to at least have many of the features of PowerSDR.
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
I agree with that 100%. Updates are needed more than 1 every 6 months or so. Bugs need to be corrected as a high priority.
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
You mean "hire" KC2QMA? Are you going to pay them? Flex is not a charity, they need to make money, how they do it on hams amazes me.
Photo of Jon - KF2E

Jon - KF2E

  • 679 Posts
  • 222 Reply Likes
No Burt, they need to hire Darrin KE9NS...and YES, I will contribute to get some of the same work he is doing for PSDR.

Jon...kf2e
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
Sign me up . . . I second that.
Photo of KC2QMA_John

KC2QMA_John

  • 755 Posts
  • 309 Reply Likes
Good catch Burt :)
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3851 Posts
  • 930 Reply Likes
I knew as soon as something ever went wrong some would jump on the opportunity to say Flex does not know what they are doing. And they think they do. Not only do the testers test, but Gerald and Steve as well as many of the Flex staff have used their Maestros for hundreds of hours with out problems.
Photo of Burt Fisher

Burt Fisher

  • 1241 Posts
  • 479 Reply Likes
I wonder if Flex is paying for all shipping if their Maestro is DOU? (dead on upgrade)
Photo of Cal Spreitzer

Cal Spreitzer

  • 447 Posts
  • 113 Reply Likes
Burt,

My DOU Maestro is already on it's way back to Austin for repair.  Flex has paid for shipping both ways.  Top notch customer service once again.  

Cal/N3CAL
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
And these 'bunch' more engineers are going to be paid with the proceeds of the $200 upgrade fee?
Photo of KC9NRN

KC9NRN

  • 324 Posts
  • 32 Reply Likes
The fix isn't more programmers, looking at the software as it is I think they have damned good ones. Windows is everywhere and while people crap on it all the time I find it a good platform and a crap platform when people clueless about computers get their hands on it. I have had to clean up after people for 30 years of PEBCAK it could make your head spin.

It's unfortunate that the severe bugs didn't turn up in testing, the only way to ensure it does get caught is to increase the amount of people testing it. I don't know how many Maestro's were sold so maybe it's not possible to have enough to vet the code as they would like.

I'm going to offer to beta test once my Maestro shows up, I'll see if their attitude is the same at that point.
Photo of K7NXT

K7NXT

  • 37 Posts
  • 27 Reply Likes
Clearly, there is a lot of interest in this topic.  -  I believe that FLEX has a window of opportunity that is closing pretty fast. Let me explain:

An SDR is all about software.  While the Maestro is a beautiful piece of hardware, FLEX will live or die because of its software.

My point is that perceived quality (good or bad) operates as a contagion process, especially when something as innovative as FLEX - Maestro first hits the market.  

So,  when I say that the window is closing for Flex 6X00 products, I mean that once the yet-to-upgrade market begins to reach consensus on the 6X00 reliability, stability, etc.  - it will be very hard to change that market perception. 

Consequently, (and speaking as a former Software Manager) I strongly believe that FLEX should put a hold on new features until the existing software features are considered "rock solid" by the user base.  

My vote is that Rock Solid beats More Features any day of the week.  

I'm wondering what everyone else thinks?

 
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
Ive said that before and couldn't agree with you more
Photo of Jim Jerzycke

Jim Jerzycke

  • 93 Posts
  • 19 Reply Likes
Agree 100% with 'NXT.

I have "spousal approval" for a 6500 now, but I'm hanging back to see how the software issues pan out.

I would MUCH rather have fewer features as long as ALL the included features work properly, and the software is 100% rock solid.

Too many requests for new bells and whistles tends to dilute the effort of the teams working on *ANY* project.

BTDT, and eventually left that company....

73, Jim
Photo of KF4HR

KF4HR

  • 778 Posts
  • 264 Reply Likes
I couldn't agree more.  Get rid of the current bugs before complicating things with more features (and more potential bugs).
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3851 Posts
  • 930 Reply Likes
Jim, the radio it's self is very solid, what is being discussed here mostly is a problem with the latest Maestro update.
Photo of Jim Jerzycke

Jim Jerzycke

  • 93 Posts
  • 19 Reply Likes
OOOPS!

Must have dropped the ball on that. I was under the impression that it was SmartSDR that was being discussed.

I'm not in the market for a Maestro. I saw one at the HAMCON in Torrnace, CA, and while I thought it was a beautiful piece of hardware, I just can't justify the price-to-own for the type of operating I do.

Thanks for the clarification!
Photo of Jim Jerzycke

Jim Jerzycke

  • 93 Posts
  • 19 Reply Likes
I'm pretty happy with my 5000A, but it's an early one and I'm kind of itching to see how much better the new ones are.

I see the type of person you mentioned in many different hobbies I'm in, from cars to photography to shooting sports.

I just shrug them off unless they try and get me involved in their "cause". Just not worth getting all spun-up over....
Photo of Ken ve7kwa

Ken ve7kwa

  • 103 Posts
  • 32 Reply Likes
<<< "rock solid" >>   I see we;ve  gone from buying amateur radio gear to Rocks again...  B(   Everyone simply needs to decide what level of risk exposure they are comfortable with and upgrade (or not) accordingly. There will Always be more risk with new or bleeding edge technology. It's exciting stuff, but... if you aren't comfortable with the inherent risk, then stay an appropriate  margin behind on the curve. I'm currently an SSDR version behind and loving it.
Photo of Jim Jerzycke

Jim Jerzycke

  • 93 Posts
  • 19 Reply Likes
I'm pretty comfortable close to edge. I just don't want software that locks up, crashes, or blue-screens my computer on a regular basis.

I'll tolerate the occasional crash as long as it doesn't happen too often.

Jim
Photo of Jay / NO5J

Jay / NO5J

  • 1674 Posts
  • 262 Reply Likes
Folks
If your not comfortable, having a little fun, then skip the fun parts. Just sit there on the sidelines and watch the rest of us play. Better yet, watch  it in the comfort of your own home on YouTube. 
Just do whatever makes you unhappy. 
We won't really mind.
73, Jay - NO5J
Photo of KD7CAO

KD7CAO

  • 34 Posts
  • 22 Reply Likes
I can agree with what you are saying. At the same time however, V2 of the software was promised shortly after sale yet I have not seen the promised functionality. To me the Maestro is useless without the remote control over the Internet. Heck, it is the whole reason we purchased 4 of the 6700 and Maestro controllers.
Photo of Mike - W8MM

Mike - W8MM

  • 218 Posts
  • 49 Reply Likes
I already have remote over Internet working FB from my office to my home station. It's here, using SoftEther VPN (free). OK, I'm an engineer, but not an IT guy by any stretch. I haven't written a line of code for over 40 years (FORTRAN in college). But, I managed to get my station remoted, even if I made numerous dumb mistakes along the way.

Just cruise around the forum and you can find tons of guidance and plenty of elmers to help out. That's how I managed to get to V2 ahead of schedule
Photo of KC9NRN

KC9NRN

  • 324 Posts
  • 32 Reply Likes
Us creating a working solution is one thing, promising it early on and not delivering is quite another. Coming up with a solution that out right works for all considering the various kinds of networks, firewalls etc out there is tough and I can see why they now say wait for 2.0.
Photo of K7NXT

K7NXT

  • 37 Posts
  • 27 Reply Likes
My comments are not a defense or criticism of FLEX, but simply an observation that if the product is perceived to be buggy, that will limit adoption of new users and the satisfaction of existing users.   

However, I don't think it is an adequate defense to blame users for the problems they experience or to say that the environment is complicated.  Like the others here, I spent a lot of money on my 6500 and Maestro and nowhere in the sales brochure did it say that some features may not work reliably due to the complicated environment.  
Photo of Walt - KZ1F

Walt - KZ1F

  • 3040 Posts
  • 645 Reply Likes
Yup, totally agree Bill. As a former development manager and PSE, myself, yeah, I think you're spot on. But understand, as you've seen, there are some on here that take a suggestion like that as a personal attack as they feel it is their sworn duty to defend FRS's honour.
Photo of WW1SS - Steve

WW1SS - Steve

  • 785 Posts
  • 263 Reply Likes
Yup right to the point that I know nothing about software development and I am "ignorant"
Photo of KC9NRN

KC9NRN

  • 324 Posts
  • 32 Reply Likes
Did I miss the post where they were talking about you?
Photo of KC2QMA_John

KC2QMA_John

  • 755 Posts
  • 309 Reply Likes
It’s not all doom and gloom. Flex has an awesome platform with the 6000 series and Maestro and in my opinion could easily dominate the market. Yes there is a “window of opportunity”. Like I said they may have already “hired” more staff for development? But look they laid the groundwork with the 6000 series hardware, in my opinion now is the time to kick software development into high gear because to me that’s what is holding the platform back from taking off in the market.

As the software stands today (other than the occasional bugs) it basically does what it needs to do to be a great functional radio and DSP works fairly good at this stage. Let me say again I love my flex there is no better option out there for me. I know in time the software will become more of what we all want, the question is how long will it take to get there...?

(Edited)
Photo of Dave - W6OVP

Dave - W6OVP

  • 316 Posts
  • 99 Reply Likes
Flex has an awesome platform with the 6000 series and Maestro and in my opinion could easily dominate the market. Yes there is a “window of opportunity”.

Spot on! The basic product concept is so "awesome" that frustration is caused by what it could be (and was advertised to be) while the progress continues halting and snail-like. Time is an integral part of this scenario, both for the competition which is working day and night, and we hopeful buyers who are not getting any younger.

Is moss beginning to grow at FLEX HQ? Windows of opportunity do not remain open very long and slam shut quickly and often without notice. This window is no longer new. The business plan has to reflect this urgency.

(But there is no moss at FLEX Customer Service and Tech Support which are First Rate!)

This conversation is no longer open for comments or replies.