Stop bashing the Beta testers

  • 4
  • Praise
  • Updated 1 year ago
Please stop bashing the beta testers for the problems you're experiencing. Unless you've been a beta tester you don't know what's involved in the process.

I certainly sympathize with the Flex beta testers because I am one of hundreds of beta testers for one of the larger streaming services with several million subscribers. We are currently testing a major new application that users have been asking for. The company gets bombarded with emails and posts from impatient users every day demanding the release immediately even though the testing isn't complete.

As a beta tester it is my responsibility to run the app thru every possible scenario that I can think of and report any bugs I find. It is NOT the responsibility of the beta testers to FIX the bugs but only to REPORT the bugs. The fixing is up to the development team. They must balance the need to continue fixes against the intense pressure that users place on them when demanding immediate releases. I'm sure the same applies to the Flex team.

The equation is simple: The more pressure you add for a quick release the more bugs it will have.  You either wait for the product to be completely tested or you wait for all the bugs resulting from a rushed testing phase to be fixed.

The company I'm testing for has taken a firm stand and will not bow to the pressure. They will instead take as much time as required to ensure a smooth roll out even if that means repeated delays. 

Bugs can get by the most rigorous testing. I know. I've been a system developer for over 30 years. Be patient and I'm sure it will work out.
Photo of Pat N6PAT

Pat N6PAT

  • 479 Posts
  • 190 Reply Likes

Posted 1 year ago

  • 4
Photo of Lee Maisel

Lee Maisel

  • 100 Posts
  • 13 Reply Likes
I Totally understand this, and have not had any comments, negative or positive about it.

However, this reminds me of an unresolved issue that has still gone completely unaddressed, I guess all the focus is on the new stuff, and to heck with the user base.

I am still unable to use D-star, and the issue (and simple fix) was expressed in detail, smoke was blown up rear ends (by Flex reps) and still nothing.

So, I can understand people's frustrations.
Photo of Michael Coslo

Michael Coslo

  • 947 Posts
  • 257 Reply Likes
Hi Pat,
I have a number of Radios, a couple Icoms, a Kenwood and a Flex on the HF end and Kenwoods and Yaesu for mobile VHF/UHF. And a PW-1 amp from Icom.

Each radio has some issues. Like the PC tuning caps on the Icom circuit boards, the Kenwood D-700 makes an occasional motorboat on very cold mornings, and so on. And while trying to buy a separation cable for the PW1, they don't even make it any more. But I can buy a whole brand new PW1 Amp from them.  But cures are few and far between. It is just something that goes with the territory.  And you don't open a trouble ticket if the company denies there is a problem.

And you are exactly correct about the pressure applied and the results obtained. The Icom 7610 is suffering from some birthing issues from just that.

Every radio or corporation involved in a radio will end up with some people hating them. Just how things work.

But I defy anyone expecting perfection or better service from some other manufacturer.
Photo of Scott N8UMW

Scott N8UMW

  • 155 Posts
  • 47 Reply Likes
I certainly would not bash them. In the end, their experience helps us all. I would love to be a beta tester.
Photo of Lee

Lee, Elmer

  • 680 Posts
  • 286 Reply Likes
Testing this radio proves to be even more challenging than many other testing situations.  Not only is there the "radio" which now is actually 7 radios plus a maestro, there is the particular computer on which the radio resides and the particular network, as well as particular use cases like Maestro v Mac client v SSDR and the 3rd party programs in use.  The permutations are not trivial, but bugs get squished as we understand the problem.  I did a quick review of my alpha channel inbox and I have collected over 6300 emails related to the development of the "radio" in the past 3.5 years.  

73  W9OY
Photo of Lawrence Gray

Lawrence Gray

  • 154 Posts
  • 74 Reply Likes
Sorry, but not all companies release products with known bugs. I ran an electronics company making sophisticated products containing complex firmware. We did not release firmware versions with known bugs, nor did we allow customer pressure to dictate quality testing procedures or release schedules.

If Flex is releasing software with known bugs because of customer pressure (I have no idea if this is being done), the company is making a serious mistake.
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3636 Posts
  • 875 Reply Likes
If you know much about Gerald, you would know that customer pressure does not drive his release times. And, it is more than likely bugs in software slip by a few testers and found in general public usage being in the many hundreds.
Photo of Lawrence Gray

Lawrence Gray

  • 154 Posts
  • 74 Reply Likes
It is virtually impossible to have much faith in a quality program that allowed a “lockup” bug affecting many users to slip by the testing process?
Photo of WW1SS - Steve

WW1SS - Steve

  • 730 Posts
  • 247 Reply Likes
I agree 100%
Photo of KY6LA - Howard

KY6LA - Howard, Elmer

  • 3700 Posts
  • 1550 Reply Likes

Gee... I guess you do not have much faith in Apple, MS, Google, Oracle, IBM, SAS, etc, etc.   In fact I cannot think of a major software vendor who has ever NOT had a major bug bite them in the ass in spite of thorough beta testing

Yet still it happens...

@Lawrence

Further I really find it hard to believe that your electronic company Never had a bug show up after product release.

Heck.. I too ran a company with real time control products.. Our bugs killed people so we really tested and tested and tested them before releasing them to GA..

Still unforeseen use cases/environments showed up as new bugs...There are just too many real world variations in use cases and environments to be able to successfully test every possibility.  I can tell you a lot of real world unforeseen horror stories

In the case of lockups.. they did not show up in beta testing or obviously they would have been addressed.  They showed up after GA in a much wider breath of environments and use cases.  Many of those new use cases were added to the beta program.

Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3636 Posts
  • 875 Reply Likes
Lawrence and Steve,  it is to bad you guys are having an unhappy time as a Flex user, I say that because if you find anything at all to say something negative you never miss a chance.
It would be great if you both would help the community as owners to give good feedback as to things you find that would help Flex make good changes.
If you want to be part of the extended Flex team, it is about attitude.
Photo of WW1SS - Steve

WW1SS - Steve

  • 730 Posts
  • 247 Reply Likes
Flex is definitly not an Apple or IBM ,like comparing apples to oranges. When Apple has an issue it is fixed in a couple days not weeks or months so dont go there.
Photo of WW1SS - Steve

WW1SS - Steve

  • 730 Posts
  • 247 Reply Likes
Oh and Bill ... my last issue has been Maestro just disconnecting with no networks found in the middle of a QSO and did not hear back from Flex yet as to what the issue is.
(Edited)
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3636 Posts
  • 875 Reply Likes
The comment that was made about Apple and IBM was correct, some of their updates break things. It even happens to them.
I bet if Flex was the same size companies as them, updates and fixes would come faster?
Photo of Rick Hadley - W0FG

Rick Hadley - W0FG

  • 595 Posts
  • 129 Reply Likes
At least the Flex upgrades don’t brick the radio like the Win10; upgrades have with two of our Toshiba laptops. It cost me $150 to get my wife’s fixed and I have to keep restoring mine to a stored restore point every time I need to restart, because I’m too cheap to have mine fixed.
Photo of Lawrence Gray

Lawrence Gray

  • 154 Posts
  • 74 Reply Likes
I rarely post and have posted positive feedback and responses to requests for assistance. I have rarely posted a negative comment. Once the rush is over, I plan to purchase a 6600. I have sold several people on Flex radios. My current 6500 is my 3rd Flex, beginning with a 1500. I am in no way “bashing” the volunteer testers. I am pointing out the relatively large number of people experiencing the radio lockup issue, which appears to have started prior to version 2 being released.

Howard, our products were deployed in the ocean for 1-3 years collecting sensor data. If they “locked up” or otherwise failed, the researchers would lose their data. One occurrence would have curtailed our business. No bugs allowed.

Comparing Flex issues to Windows issues is ridiculous. Windows 10 has 50 million lines of code. There are something over 85 million lines of code in OSX. These software packages are in no way comparable to FlexRadio.

I love my Flex. I use it everyday. I use virtually every mode. I use it remotely with an iPad. It is a generally great product. However, I do not think it serves anyone to avoid pointing out what I believe to be a reasonably serious issue that has existed for awhile based on community posts. Remember, for every person that posts an issue, there are many who don’t bother.

No one is bashing anyone—just trying to make the product better.
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3636 Posts
  • 875 Reply Likes
As long as we do not blow things we read here on the community out of proportion. The main reason this community is here is so people can report problems and seek help before submitting a Help Desk ticket.
When we read here we see lots of different problems, but out there are hundreds of Flex radios without show stopping problems.
Pointing out problems with the radio is what should happen, that helps us all. but to make comments like, losing  or lost faith in their system of testing or to that effect? How would that be helpful?

Take me for instance, I have never had any of the freezes or lockups reported. But I know some have them, and for a while.
They say their working on these issues, so why can't we just let it go and see what they come up with.

Steve went a big step the other day explaining were he feels Flex has failed in testing the software, saying that the testers know the software and just know what to do and what to expect. And he added that for a new Flex user things could and should be better explained to them, and that Flex was sorry.
That was very transparent.
Photo of Rich McCabe

Rich McCabe

  • 1105 Posts
  • 238 Reply Likes
And in all fairness, people that don't "hang out" online rarely go out of their way to make a "this is working great" post. Can't say the same when someone is angry and/or frustrated.

As long as you are on the bleeding edge, you are going to have some issues.

I guess if a guy wants no software bugs he should get one of those radios with "round ADC" devices in the back. I think they call them 6146s.

Rich
Photo of John - K3MA

John - K3MA

  • 322 Posts
  • 94 Reply Likes
Your right that the Beta testers should not be blamed for a ineffective and weak testing program.  It clearly rests on the sholders of Flex management.  In absence of any customers vocalizing that they are not happy there is no pressure for Flex management to improve the testing process.  I would submit the multiple missed rollout dates as a good enough measure that things can be improved.  It does not seem to be working for them (at least from the perspective of their customers) and as such maybe they should benchmark some world class testing processes. If they did so I am sure they would find much low hanging fruit that could easily be implemented to improve there testing process and release success.

Instead of "As a beta tester it is my responsibility to run the app thru every possible scenario that I can think of".

It could be replaced with:
1) Measures of testing success tracked and reported daily or weekly.
2) Detailed testing scripts with tracking timelines and a feedback loop to improve the scripts.
3) Timely follow-up to determine if testing is on track and reallocation of testing resources.
4) Feedback of testing timelines into the overall project plan.
5) A follow-up program to identify "How to improve success of the testing measures."

John
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3634 Posts
  • 875 Reply Likes
I understand that you wish to speak for all Flex customers, how ever it may surprise you that there are many very happy as well.
It is nice to see your opinion here and welcome. Gerald has in the past outlined the testing organisation to us, and it looks like they have things well covered.
The very fact that they have had delays is part of trying to make sure things are right before releasing. And almost all delays were due to late parts arrival.
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 339 Posts
  • 69 Reply Likes
Let's just take on one of your points above and see it it can be improved (With your help) 

"2) Detailed testing scripts with tracking timelines and a feedback loop to improve the scripts."
Let's start with the first part of this statement. 
Don't know which of the the  8 radio models you have, but if you would produce a script for your model and publish it there may be one of the ALPHA testers who would be willing to follow it. Remember to cover all the aspects of SSDR then those that apply only to you radio. Also make sure that you cover all aspects of all possible variances of operating modes and then all aspects of all third party software that connects to the radio. 

Remember FLEX nor the ALPHA team knows exactly what or how you use or apply the tools available. 
I have found several times that I though I had a problem and I did it turned out to be mine for not or miss-understanding instructions or assuming I knew what I was doing. 

Remember also ALPHA testers are not paid even after they tested version 2.0.x they had to then pay the $200.00 for it's used. 

Thanks for your assistance looking forward to reading through your script. 

73's
 
Photo of WW1SS - Steve

WW1SS - Steve

  • 730 Posts
  • 247 Reply Likes
Flex makes a good product. They are getting a bad reputation for the delays and software bugs. Been in several QSO's lately where I just disappeared because of lock-up and got the "what you have to do reboot that flex" . . . That is NOT good.
Photo of Bill -VA3WTB

Bill -VA3WTB

  • 3636 Posts
  • 875 Reply Likes
Steve, that is true. Keep in mind who the people are that are trying to give Flex a bad reputation. Being that Flex is the leader in SDR many people look for anything at all to bash, and they blow things out of proportion and say misleading and poor information.
I hear this all the time on air.
Remember, Flex Radio claim to fame is SDR
Photo of Ria - N2RJ

Ria - N2RJ, Elmer

  • 2308 Posts
  • 945 Reply Likes
Bear in mind that some things are really edge cases where they depend on things like windows versions, equipment configuration and different ways of operating. I have experienced the lockups in the past (even way prior to 2.1) but not with the frequency that you have. My radio is pretty much running constantly. I solve problems on a daily basis and finding a root cause is not always easy. Especially true in a complex device such as a FPGA based SDR.
Photo of Rich McCabe

Rich McCabe

  • 1105 Posts
  • 238 Reply Likes
When were beta (alpha) testers getting bashed?  I think they all did a fine job and all had different operating styles which should help.
Photo of Ria - N2RJ

Ria - N2RJ, Elmer

  • 2308 Posts
  • 946 Reply Likes
Tim or Steve had mentioned this but I'll mention it again.

There are two sets of testing that go on. One is internal testing by the Flex engineering team. This has the scripts, timelines etc. The other is end-user beta testing, which subjects the software to real world conditions.

The Alpha team does the latter, not the former. I don't think that I would want to follow a rigorous test plan every time a new alpha or beta release is made. Instead I just use the software as I normally would, and try to break it. We do get the release notes and a list of issues that were addressed in a particular build. So we pay extra attention to that. 

Like you, I am an end user. I just get alpha releases and provide feedback. I've been in other similar alpha/beta programs, including for ARRL LoTW. I also test devel releases of WSJT-X. They all run pretty much the same. 
Photo of WW1SS - Steve

WW1SS - Steve

  • 730 Posts
  • 247 Reply Likes
Did your feedback include the lock ups and freezing problems people have been experiencing . Or was everything ok.
Photo of Ria - N2RJ

Ria - N2RJ, Elmer

  • 2308 Posts
  • 945 Reply Likes
I cannot comment on actual testing as that is NDA. Ask them if you want details. 
Photo of Steve K9ZW

Steve K9ZW, Elmer

  • 1399 Posts
  • 708 Reply Likes

Backing up Ria on this, one of the BIGGEST reasons I'm hardly answering questions put to me on my blog or by email, and have sharply limited participation here at the community is these endless NDA-violating questions.

Folks, as part of the sign on to be tester we each do an NDA, and more important than the actual document - we pledged to follow the NDA.

Maybe that responsibility to live up to one's word is a variable for some folk, but then figure out who doesn't receive nominations or support on the rare occasions FRS looks to add a few more testers.

Now as a reminder here is what I can do under the NDA - I can talk about anything already released, anything the FRS says can be considered "as released" for discussion, show off my test gear if you come to my shack, include discussions of released hardware/software/features in what I write, and ditto I can do photos/videos of released hardware/software/features.

Like any non-FRS person, I am never going to promise dates, features or anything else.  You got to go to the source (FRS) for any answers on those items.

I can tell you about how, IN MY SHACK, I'm doing with the released hardware/software/features.  I'm good on that, and it matches what the NDA is about.

As I've had an excellent experience as an added "alpha tester" with the gear I'm testing, I couldn't begin to throw fuel on the made up stuff some of the community seems to be imagining.

If I won't talk about it, it doesn't mean I am denying or confirming anything - it just means the question posed directly violates my NDA agreement. 

Oh BTW I gave my word by signing the NDA to FRS, and I don't have any sort of agreement with anyone else that out ranks where I've given my word.  I am deeply offended when fellow hams think I should set aside an agreement I gave my word to, just to please them.  At least in my case I'm never going to knowingly violate agreement I'd made folks. 

Hope this gives some light on the NDA world of the testers.

73

Steve

K9ZW

Photo of N5LB - Lionel B

N5LB - Lionel B

  • 195 Posts
  • 56 Reply Likes
I can't recall any software driven hardware without bugs.  My iPad and iPhone receive software updates fairly often, but then for practical purposes, Apple has unlimited resources. Even with their updates, software bugs remain.  What is the Apple testing procedure? 

I expect bugs in all things software; the fate of an early adopter. 

The Flex team does okay in my experience.
Photo of Craig Williams

Craig Williams

  • 89 Posts
  • 35 Reply Likes
Well, my Swan 500CX has no software issues. Oh the simpler days.
Photo of Dan -- KC4GO

Dan -- KC4GO

  • 339 Posts
  • 69 Reply Likes
Have you checked how clean the TX is lately :) 

I had a 350 lots of power but dirty
Photo of Gerald - K5SDR

Gerald - K5SDR, Employee

  • 825 Posts
  • 1504 Reply Likes
Official Response
First, I want to sincerely thank our alpha/beta test team.  We are truly blessed to have some of the best who range in operating styles across the amateur spectrum.  They work extremely well with our software team and hold nothing back when they find problems.  They treat our team with respect and our team treats them with the respect they deserve.

Second, we do NOT release products with known bugs that are deemed critical or major in nature even though it may delay shipment of hardware products.  That is why we took longer to release the new radios than any of us would want.  That does not mean that we won't find edge cases when the radios get into 10 times as many customers as we have on the alpha/beta team.  It is almost a given that we will.  We have a test rack loaded with every radio that runs SmartSDR.  We run an automated script on every radio for every single software build but that won't catch everything.  It is a fact of life that something will ultimately slip through that we need to address.

Every single week (typically on Monday) our software team logs and reviews all issues reported from the alpha/beta team and from the customer base at large.  Any time there is a new major release there is the opportunity to introduce new bugs that may not be caught through almost any test regimen.  For example, I depend on my iPad for both work and personal use.  Apple has completely broken my iPad twice to where I could not use it for days until they came out with a fix. 
Apple's testing resources are almost infinite when compared to those of the entire amateur radio manufacturer's combined and they have critical bugs that they were not aware of when they released.  I seriously doubt they released iOS knowing it would temporarily brick my iPad.  I personally did not hold it against them because I know how hard it is to be perfect.  By the way, the iPad is a much simpler use case than amateur radio in my opinion.  It only has one connector, has a controlled interface, and it does not connect to an antenna.

Steve has described our software process here on multiple occasions as have I.  We take constant improvement very seriously.  Yes, there are things that may cause problems for some that we have not yet addressed.  There may be some things we will never do and others we have put off for awhile that we may soon get around to.  Everything we do requires triage and trade offs.  We strive for continuous improvement.

I can say that our entire software team is focused only on addressing the most important customer experience issues and anything that affects delivery of quality radios to those of you who are still waiting for radios.  Nothing said here will change that fact.  

So we at FlexRadio extend our warmest thanks to our test team!

73,
Gerald
Photo of Brad A. Steffler

Brad A. Steffler

  • 92 Posts
  • 32 Reply Likes
Thank you, Gerald. I love my 6700. I am eager to get my hands on my PG XL and tuner. But I don't want them with bugs in them. Take your time. I am in a service industrywhere I get time complaints every day.If I go too fast, serious mistakes can be made. Better to do things right the first time. I work as a physician/professor in a medical school.

Bless you, Gerald - and all of Flex.

Brad
KE4XJ