Is a more structured Alpha/Beta program needed?

Mark_WS7M

I emailed to become a beta tester and received no response.  Personally, I've been a member of 5 different beta teams stemming from Microsoft, Borland, Klipsch, Toshiba, and Dell.   

I think a beta team needs to be a mix of people from technical to non-technical.   The other thing is the beta team needs to be 100% willing to brick there device.  In fact it is an obligation to install every update, try odd combinations, you name it.   In fact a beta member needs to assume their device will be useless most of the time.

I received no response from FR on my beta request.  Not angry or mad about this but frankly they ignored a valuable resource.   I'm sure I could have broken the thing many times over.  I tend to be pretty good at that when it comes to beta testing.

Anyway I agree with Howard.  There is no amount of testing that will catch everything.  As an early adopter of a new device, you should be fully prepared for issues.

Walt - KZ1F

Jay, honest, I was not referring to you at all.

km9r.mike

All this kvetching, finger pointing and out of shape arm chair quarterbacking at the beta team seems foolish at best.  I am specifically referring to the maestro update kerfuffle.

Problem : maestro failed to update and in addition maestro goes TU while attempting to update.

First off lets get the obvious out of the way for the sake of the hard headed amongst us. If this occurred once w/in the beta team, I am 100% confident the team would have reported such. To think otherwise is beyond ignorant.

 Now explain to me exactly how would a beta team catch this if all the updates that they did worked flawlessly or if they never had to perform an update during their beta testing ( just trying to be fair and cover all bases ) ? Face the facts, the process of updating software has been dumbed down for the dumbest of users. The action is neither complicated nor rocket science.

  If the beta team was indeed asked to perform an update to maestro, while performing their beta duties and all updates went flawlessly just what the heck do the perpetual kvetchers expect the beta team to do differently at this point to prevent what occurred ? Please be specific.

The users did nothing different than the beta testers ( if indeed the beta testers were asked to perform an update) yet some users are now claiming they are superior to the beta testers (and FRS) because they simply initiated a thoughtless event according to the instructions before them.

I will not even go into speculating what error occurred and how this error occurred, but for me wildly casting stones at the beta team seems foolish at best.

Burt Fisher

The least you deserve is a response. What excuse could they possibly have for ignoring a customer?

Burt Fisher

You  don't get it, no one faults the selected beta team. The problem of why it wasn't found is the high competency of that team. A beta team needs to be composed of a cross section of users, not just the elite. 50 beta members seems small also. On events like these we usually see Gerald weighing in, where is he?

Burt Fisher

" **** U ME " Why I never heard that before, any more like that? How about, "for cryin out loud?" Or, "the truth shall set you free?"

I had read here that there were 50 alpha testers but sadly I didn't record it. If I am wrong I would gladly apologize. I don't see an official response. They are usually great about (another tired cliché coming), "setting the record straight."

Burt Fisher

If they would weigh in on the make-up of the testers it might turn out this crash might have not been possible to foresee.

Burt Fisher

NPD? Narcissistic personality disorder?
If that is what it means I am so far out of your league that no further discussion is possible.
I have no idea what the following means, " So, to put it in your arena, how do you feel about warehousing and the people who volunteer that as a viable solution to a far more nuanced problem"
I am getting you think I am too **** to have viable comments on how Flex could have avoided this mess. Therein lies the problem, with your exceptional talents this crash still got through. The PhD's, EE or ME did not find the problem. Get a clue most amateurs are not PhD's, EE or ME's. The testers ought to be a cross section unless Flex is selling to just PhD's, EE or ME's.
You have demonstrated you can walk with kings, but sadly, only kings.

Kevin Va3KGS

Everybody needs to get back to DXing !!!
Cheers
Kevin, Va3KGS

Burt Fisher

Kevin, what is discussed here is far more in the spirit of ham radio than talking to a ham for one minute in another country.

Bill -VA3WTB

He must be reading all this shaking his head.
Why keep repeating yourself? It is clear that you and some of the others think Flex has know idea what they are doing. So, your going to teach Gerald how to creat a testing group,,lol umm ok.

As it looks, out of the hundreds they sold, very few are effected, this is a good reason Gerald and the rest of the Flex staff never saw it. It didn't effect the ones they are using. But you know how it goes, if just one person writes in and says it happened to them all **** breaks loos. And Flex would take care of that one all the same.

So we heard the bitching and all you experts,,lets move on...

EA4GLI

Bill, I think there has to be a middle ground between whining and accepting everything. The best promoters of Flex are its customers, but I also think we need to be the most critical. I always find little things that I would do different in almost any equipment I use. The huge advantage here is that I have a forum to voice my opinion and it gets read by the guys that make the equipment and might even be a feature in the future... how cool is that? 

I am sure Gerald et al. can navigate the waters of this forum and keep focus on doing the things they consider the best approach to better their products and ultimately have a profitable and successful business.

I am using my Flex and Maestro with Ver 1.8  without issue but I still take part on this discussion because there are things that can be improved and that will yield a better product. I guess I am bias because I want my radio to be better.

Instead of moving on I welcome different points of view... more so if the add value. Even the "haters" sometimes make good points. 

Bill -VA3WTB

Yes I agree, my point is, that some people comment with their opinion and keep pounding it home over and over. It seems their are some not wishing to be constructive, witch we all want, 

Walt - KZ1F

At least in high tech, when you go for a job interview and the company chooses not to pursue your candidacy, the conversation ends. They do not get involved with extracting themselves from ones candidacy. To do otherwise opens the door to a whole bunch of "yeah but's".

Mike W8MM

I already have remote over Internet working FB from my office to my home station. It's here, using SoftEther VPN (free). OK, I'm an engineer, but not an IT guy by any stretch. I haven't written a line of code for over 40 years (FORTRAN in college). But, I managed to get my station remoted, even if I made numerous dumb mistakes along the way. Just cruise around the forum and you can find tons of guidance and plenty of elmers to help out. That's how I managed to get to V2 ahead of schedule

Walt - KZ1F

I really wish people would disabuse themselves of this lover hater thing. It's childish and a very simplistic view of the world. And Bill, some people comment with their experience. For instance in any software division I've been in there's never been more than 5 or 6 QA Engineers. On any scrum team there has never been more than one QA engineer assigned.

KC9NRN

Less competent, always a great way to insult people, who judges their competency?

KC9NRN

Excellent points, I offered to be a beta tester when discussing my purchase. I was told I would need to have been around much longer before this could happen. Seems to me your approach would be a good one, want beta software, sign on the line after reading the risk statement etc.

KC9NRN

The fix isn't more programmers, looking at the software as it is I think they have **** good ones. Windows is everywhere and while people **** on it all the time I find it a good platform and a **** platform when people clueless about computers get their hands on it. I have had to clean up after people for 30 years of PEBCAK it could make your head spin. It's unfortunate that the severe bugs didn't turn up in testing, the only way to ensure it does get caught is to increase the amount of people testing it. I don't know how many Maestro's were sold so maybe it's not possible to have enough to vet the code as they would like. I'm going to offer to beta test once my Maestro shows up, I'll see if their attitude is the same at that point.

KC9NRN

Us creating a working solution is one thing, promising it early on and not delivering is quite another. Coming up with a solution that out right works for all considering the various kinds of networks, firewalls etc out there is tough and I can see why they now say wait for 2.0.

KC9NRN

Did I miss the post where they were talking about you?

KC9NRN

Mark, Well said, they did tell me I couldn't be one as a new customer, somebody who very easily could catch a problem since I'm new to the platform and might step on something an experienced Flex user might not.

K7NXT

NordicPC, I think the concept that you describe is very important. Experienced users in any technology domain have developed "schemas" (mental models) which tend to create blind spots which are not present for novice users.  

KC9NRN

I'm going to ask them again when my Maestro ships. Although one has to wonder if this new problem will cause a delay. I'll contact them tomorrow and ask, an email I sent last week asking for an update went unanswered.

Walt - KZ1F

The predominate model implicitly being referred to in this thread is, what I will call, "infinite monkey paradox", essentially if one throws enough bodies at a problem the results will get better. I, for one, have been approaching this from a more engineering model, of code coverage analysis, continuous integration with automated smoke testing, Every bug in the tracking system adds a suite of unit/integration tests to the automated process. Every new feature adds a suite of tests to the process. Enforced code coverage floor of 80%.  Only after these hurtles have been passed does a QA 'engineer' get to run highly scripted system and destructive testing, ie, a power failure in the middle of a save, a backhoe induced comm failure during a download. Unscripted testing is problematic at best. Only using this approach is there consistency, reproducibility, and high confidence nothing got dropped.
However, to your point Bill, and NordicPC, at some point you should give it to an un-indoctrinated user with the instructions without benefit of manual, try to break it, or break this feature. I stress, without benefit of manual, as to 'line for line' follow the manual, which few in the real world ever do, will, in fact, script a working solution....yep, that works without breaking anything,

KC9NRN

I am the king of not reading directions or manuals, my wife knows this soooo well.

Peter

Keep it up boys, I am enjoying this. LOL

John-K3MA

VA3WTB I certainly hope that Gerald is reading this thread and shaking his head.  However, not in the way you imply.  

I hope he is shaking his head saying to himself we must improve.  We need to make more frequent releases of the software but we cannot sacrifice quality of these releases.  The software has gotten more complicated and will continue to evolve to meet the needs of our customers and keep our company on the leading edge.  While differentiating us from our competitors and other software companies.  We can't just use a strategy of hoping that what we have done in the past when we released a couple of versions a year will work equally well as we move to releasing a version every few months.  These most recent release issues is a indication that we need to take some steps to evaluate the process that has worked for us to date and what might we change to ensure it continues to work in the future.

The owner of a company cannot be the expert on everything and if indeed Gerald is reading this thread and thinks somewhat in line with what I wrote above instead of what you imply then I make this specific actionable recommendation for his consideration.

Select a leader in the Flex organization and a leader in the current Alpha and Beta testing groups to co-lead a small multifunction group made up of a few users from the customer base that bring varied skills and background in software and quality assurance together.  Task this group to within the next 30 days review and recommend specific changes to the software testing process which when implemented will achieve measureable improvements in the metrics used to measure the effectiveness of software releases while supporting the goals of the company. 

KC9NRN

Well said.

Bill -VA3WTB

And still things will be missed, I really think people are making to big a deal about this. Flex is on top of it and working on it as we speak. I for one will not be explaining to Gerald how to do things. And I'm sure he is working with experts to shore things up better.