Stop bashing the Beta testers

Pat N6PAT

Please stop bashing the beta testers for the problems you're experiencing. Unless you've been a beta tester you don't know what's involved in the process.

I certainly sympathize with the Flex beta testers because I am one of hundreds of beta testers for one of the larger streaming services with several million subscribers. We are currently testing a major new application that users have been asking for. The company gets bombarded with emails and posts from impatient users every day demanding the release immediately even though the testing isn't complete.

As a beta tester it is my responsibility to run the app thru every possible scenario that I can think of and report any bugs I find. It is NOT the responsibility of the beta testers to FIX the bugs but only to REPORT the bugs. The fixing is up to the development team. They must balance the need to continue fixes against the intense pressure that users place on them when demanding immediate releases. I'm sure the same applies to the Flex team.

The equation is simple: The more pressure you add for a quick release the more bugs it will have.  You either wait for the product to be completely tested or you wait for all the bugs resulting from a rushed testing phase to be fixed.

The company I'm testing for has taken a firm stand and will not bow to the pressure. They will instead take as much time as required to ensure a smooth roll out even if that means repeated delays. 

Bugs can get by the most rigorous testing. I know. I've been a system developer for over 30 years. Be patient and I'm sure it will work out.

Lee Maisel

I Totally understand this, and have not had any comments, negative or positive about it.

However, this reminds me of an unresolved issue that has still gone completely unaddressed, I guess all the focus is on the new stuff, and to heck with the user base.

I am still unable to use D-star, and the issue (and simple fix) was expressed in detail, smoke was blown up rear ends (by Flex reps) and still nothing.

So, I can understand people's frustrations.

Michael Coslo

Hi Pat,
I have a number of Radios, a couple Icoms, a Kenwood and a Flex on the HF end and Kenwoods and Yaesu for mobile VHF/UHF. And a PW-1 amp from Icom.

Each radio has some issues. Like the PC tuning caps on the Icom circuit boards, the Kenwood D-700 makes an occasional motorboat on very cold mornings, and so on. And while trying to buy a separation cable for the PW1, they don't even make it any more. But I can buy a whole brand new PW1 Amp from them.  But cures are few and far between. It is just something that goes with the territory.  And you don't open a trouble ticket if the company denies there is a problem.

And you are exactly correct about the pressure applied and the results obtained. The Icom 7610 is suffering from some birthing issues from just that.

Every radio or corporation involved in a radio will end up with some people hating them. Just how things work.

But I defy anyone expecting perfection or better service from some other manufacturer.

Scott N8UMW

I certainly would not bash them. In the end, their experience helps us all. I would love to be a beta tester.

W9OY

Testing this radio proves to be even more challenging than many other testing situations.  Not only is there the "radio" which now is actually 7 radios plus a maestro, there is the particular computer on which the radio resides and the particular network, as well as particular use cases like Maestro v Mac client v SSDR and the 3rd party programs in use.  The permutations are not trivial, but bugs get squished as we understand the problem.  I did a quick review of my alpha channel inbox and I have collected over 6300 emails related to the development of the "radio" in the past 3.5 years.  

73  W9OY

Lawrence Gray

Sorry, but not all companies release products with known bugs. I ran an electronics company making sophisticated products containing complex firmware. We did not release firmware versions with known bugs, nor did we allow customer pressure to dictate quality testing procedures or release schedules. If Flex is releasing software with known bugs because of customer pressure (I have no idea if this is being done), the company is making a serious mistake.

John - K3MA

Your right that the Beta testers should not be blamed for a ineffective and weak testing program.  It clearly rests on the sholders of Flex management.  In absence of any customers vocalizing that they are not happy there is no pressure for Flex management to improve the testing process.  I would submit the multiple missed rollout dates as a good enough measure that things can be improved.  It does not seem to be working for them (at least from the perspective of their customers) and as such maybe they should benchmark some world class testing processes. If they did so I am sure they would find much low hanging fruit that could easily be implemented to improve there testing process and release success.

Instead of "As a beta tester it is my responsibility to run the app thru every possible scenario that I can think of".

It could be replaced with:
1) Measures of testing success tracked and reported daily or weekly.
2) Detailed testing scripts with tracking timelines and a feedback loop to improve the scripts.
3) Timely follow-up to determine if testing is on track and reallocation of testing resources.
4) Feedback of testing timelines into the overall project plan.
5) A follow-up program to identify "How to improve success of the testing measures."

John

[Deleted User]

Flex makes a good product. They are getting a bad reputation for the delays and software bugs. Been in several QSO's lately where I just disappeared because of lock-up and got the "what you have to do reboot that flex" . . . That is NOT good.

Rich McCabe

When were beta (alpha) testers getting bashed?  I think they all did a fine job and all had different operating styles which should help.

Ria

Tim or Steve had mentioned this but I'll mention it again.

There are two sets of testing that go on. One is internal testing by the Flex engineering team. This has the scripts, timelines etc. The other is end-user beta testing, which subjects the software to real world conditions.

The Alpha team does the latter, not the former. I don't think that I would want to follow a rigorous test plan every time a new alpha or beta release is made. Instead I just use the software as I normally would, and try to break it. We do get the release notes and a list of issues that were addressed in a particular build. So we pay extra attention to that. 

Like you, I am an end user. I just get alpha releases and provide feedback. I've been in other similar alpha/beta programs, including for ARRL LoTW. I also test devel releases of WSJT-X. They all run pretty much the same. 

[Deleted User]

Did your feedback include the lock ups and freezing problems people have been experiencing . Or was everything ok.

Bill -VA3WTB

If you know much about Gerald, you would know that customer pressure does not drive his release times. And, it is more than likely bugs in software slip by a few testers and found in general public usage being in the many hundreds.

Ria

I cannot comment on actual testing as that is NDA. Ask them if you want details. 

Bill -VA3WTB

I understand that you wish to speak for all Flex customers, how ever it may surprise you that there are many very happy as well.
It is nice to see your opinion here and welcome. Gerald has in the past outlined the testing organisation to us, and it looks like they have things well covered.
The very fact that they have had delays is part of trying to make sure things are right before releasing. And almost all delays were due to late parts arrival.

Bill -VA3WTB

Steve, that is true. Keep in mind who the people are that are trying to give Flex a bad reputation. Being that Flex is the leader in SDR many people look for anything at all to bash, and they **** things out of proportion and say misleading and poor information.
I hear this all the time on air.
Remember, Flex Radio claim to fame is SDR

Lionel

I can't recall any software driven hardware without bugs.  My iPad and iPhone receive software updates fairly often, but then for practical purposes, Apple has unlimited resources. Even with their updates, software bugs remain.  What is the Apple testing procedure? 

I expect bugs in all things software; the fate of an early adopter. 

The Flex team does okay in my experience.

Ria

Bear in mind that some things are really edge cases where they depend on things like windows versions, equipment configuration and different ways of operating. I have experienced the lockups in the past (even way prior to 2.1) but not with the frequency that you have. My radio is pretty much running constantly. I solve problems on a daily basis and finding a root cause is not always easy. Especially true in a complex device such as a FPGA based SDR.

Craig Williams

Well, my Swan 500CX has no software issues. Oh the simpler days.

Gerald-K5SDR

First, I want to sincerely thank our alpha/beta test team.  We are truly blessed to have some of the best who range in operating styles across the amateur spectrum.  They work extremely well with our software team and hold nothing back when they find problems.  They treat our team with respect and our team treats them with the respect they deserve.

Second, we do NOT release products with known bugs that are deemed critical or major in nature even though it may delay shipment of hardware products.  That is why we took longer to release the new radios than any of us would want.  That does not mean that we won't find edge cases when the radios get into 10 times as many customers as we have on the alpha/beta team.  It is almost a given that we will.  We have a test rack loaded with every radio that runs SmartSDR.  We run an automated script on every radio for every single software build but that won't catch everything.  It is a fact of life that something will ultimately slip through that we need to address.

Every single week (typically on Monday) our software team logs and reviews all issues reported from the alpha/beta team and from the customer base at large.  Any time there is a new major release there is the opportunity to introduce new bugs that may not be caught through almost any test regimen.  For example, I depend on my iPad for both work and personal use.  Apple has completely broken my iPad twice to where I could not use it for days until they came out with a fix. 
Apple's testing resources are almost infinite when compared to those of the entire amateur radio manufacturer's combined and they have critical bugs that they were not aware of when they released.  I seriously doubt they released iOS knowing it would temporarily brick my iPad.  I personally did not hold it against them because I know how hard it is to be perfect.  By the way, the iPad is a much simpler use case than amateur radio in my opinion.  It only has one connector, has a controlled interface, and it does not connect to an antenna.

Steve has described our software process here on multiple occasions as have I.  We take constant improvement very seriously.  Yes, there are things that may cause problems for some that we have not yet addressed.  There may be some things we will never do and others we have put off for awhile that we may soon get around to.  Everything we do requires triage and trade offs.  We strive for continuous improvement.

I can say that our entire software team is focused only on addressing the most important customer experience issues and anything that affects delivery of quality radios to those of you who are still waiting for radios.  Nothing said here will change that fact.  

So we at FlexRadio extend our warmest thanks to our test team!

73,
Gerald

Steve K9ZW

Backing up Ria on this, one of the BIGGEST reasons I'm hardly answering questions put to me on my blog or by email, and have sharply limited participation here at the community is these endless NDA-violating questions.

Folks, as part of the sign on to be tester we each do an NDA, and more important than the actual document - we pledged to follow the NDA.

Maybe that responsibility to live up to one's word is a variable for some folk, but then figure out who doesn't receive nominations or support on the rare occasions FRS looks to add a few more testers.

Now as a reminder here is what I can do under the NDA - I can talk about anything already released, anything the FRS says can be considered "as released" for discussion, show off my test gear if you come to my shack, include discussions of released hardware/software/features in what I write, and ditto I can do photos/videos of released hardware/software/features.

Like any non-FRS person, I am never going to promise dates, features or anything else.  You got to go to the source (FRS) for any answers on those items.

I can tell you about how, IN MY SHACK, I'm doing with the released hardware/software/features.  I'm good on that, and it matches what the NDA is about.

As I've had an excellent experience as an added "alpha tester" with the gear I'm testing, I couldn't begin to throw fuel on the made up stuff some of the community seems to be imagining.

If I won't talk about it, it doesn't mean I am denying or confirming anything - it just means the question posed directly violates my NDA agreement. 

Oh BTW I gave my word by signing the NDA to FRS, and I don't have any sort of agreement with anyone else that out ranks where I've given my word.  I am deeply offended when fellow hams think I should set aside an agreement I gave my word to, just to please them.  At least in my case I'm never going to knowingly violate agreement I'd made folks. 

Hope this gives some light on the NDA world of the testers.

73

Steve

K9ZW

Dan -- KC4GO

Let's just take on one of your points above and see it it can be improved (With your help) 

"2) Detailed testing scripts with tracking timelines and a feedback loop to improve the scripts."
Let's start with the first part of this statement. 
Don't know which of the the  8 radio models you have, but if you would produce a script for your model and publish it there may be one of the ALPHA testers who would be willing to follow it. Remember to cover all the aspects of SSDR then those that apply only to you radio. Also make sure that you cover all aspects of all possible variances of operating modes and then all aspects of all third party software that connects to the radio. 

Remember FLEX nor the ALPHA team knows exactly what or how you use or apply the tools available. 
I have found several times that I though I had a problem and I did it turned out to be mine for not or miss-understanding instructions or assuming I knew what I was doing. 

Remember also ALPHA testers are not paid even after they tested version 2.0.x they had to then pay the $200.00 for it's used. 

Thanks for your assistance looking forward to reading through your script. 

73's
 

Dan -- KC4GO

Have you checked how clean the TX is lately  

I had a 350 lots of power but dirty

Lawrence Gray

It is virtually impossible to have much faith in a quality program that allowed a “lockup” bug affecting many users to slip by the testing process?

[Deleted User]

I agree 100%

Brad A. Steffler

Thank you, Gerald. I love my 6700. I am eager to get my hands on my PG XL and tuner. But I don't want them with bugs in them. Take your time. I am in a service industrywhere I get time complaints every day.If I go too fast, serious mistakes can be made. Better to do things right the first time. I work as a physician/professor in a medical school.

Bless you, Gerald - and all of Flex.

Brad
KE4XJ

KY6LA_Howard

Gee... I guess you do not have much faith in Apple, MS, Google, Oracle, IBM, SAS, etc, etc.   In fact I cannot think of a major software vendor who has ever NOT had a major bug bite them in the **** in spite of thorough beta testing

Yet still it happens...

@Lawrence

Further I really find it hard to believe that your electronic company Never had a bug show up after product release.

Heck.. I too ran a company with real time control products.. Our bugs killed people so we really tested and tested and tested them before releasing them to GA..

Still unforeseen use cases/environments showed up as new bugs...There are just too many real world variations in use cases and environments to be able to successfully test every possibility.  I can tell you a lot of real world unforeseen horror stories

In the case of lockups.. they did not show up in beta testing or obviously they would have been addressed.  They showed up after GA in a much wider breath of environments and use cases.  Many of those new use cases were added to the beta program.

Bill -VA3WTB

Lawrence and Steve,  it is to bad you guys are having an unhappy time as a Flex user, I say that because if you find anything at all to say something negative you never miss a chance.
It would be great if you both would help the community as owners to give good feedback as to things you find that would help Flex make good changes.
If you want to be part of the extended Flex team, it is about attitude.

[Deleted User]

Flex is definitly not an Apple or IBM ,like comparing apples to oranges. When Apple has an issue it is fixed in a couple days not weeks or months so dont go there.

[Deleted User]

Oh and Bill ... my last issue has been Maestro just disconnecting with no networks found in the middle of a QSO and did not hear back from Flex yet as to what the issue is.

Bill -VA3WTB

The comment that was made about Apple and IBM was correct, some of their updates break things. It even happens to them.
I bet if Flex was the same size companies as them, updates and fixes would come faster?

Rick Hadley - W0FG

At least the Flex upgrades don’t brick the radio like the Win10; upgrades have with two of our Toshiba laptops. It cost me $150 to get my wife’s fixed and I have to keep restoring mine to a stored restore point every time I need to restart, because I’m too cheap to have mine fixed.