Stop bashing the Beta testers

  • 4
  • Praise
  • Updated 7 months ago
Please stop bashing the beta testers for the problems you're experiencing. Unless you've been a beta tester you don't know what's involved in the process.

I certainly sympathize with the Flex beta testers because I am one of hundreds of beta testers for one of the larger streaming services with several million subscribers. We are currently testing a major new application that users have been asking for. The company gets bombarded with emails and posts from impatient users every day demanding the release immediately even though the testing isn't complete.

As a beta tester it is my responsibility to run the app thru every possible scenario that I can think of and report any bugs I find. It is NOT the responsibility of the beta testers to FIX the bugs but only to REPORT the bugs. The fixing is up to the development team. They must balance the need to continue fixes against the intense pressure that users place on them when demanding immediate releases. I'm sure the same applies to the Flex team.

The equation is simple: The more pressure you add for a quick release the more bugs it will have.  You either wait for the product to be completely tested or you wait for all the bugs resulting from a rushed testing phase to be fixed.

The company I'm testing for has taken a firm stand and will not bow to the pressure. They will instead take as much time as required to ensure a smooth roll out even if that means repeated delays. 

Bugs can get by the most rigorous testing. I know. I've been a system developer for over 30 years. Be patient and I'm sure it will work out.
Photo of Pat N6PAT

Pat N6PAT

  • 219 Posts
  • 53 Reply Likes

Posted 7 months ago

  • 4
Photo of Gerald - K5SDR

Gerald - K5SDR, Employee

  • 798 Posts
  • 1452 Reply Likes
Official Response
First, I want to sincerely thank our alpha/beta test team.  We are truly blessed to have some of the best who range in operating styles across the amateur spectrum.  They work extremely well with our software team and hold nothing back when they find problems.  They treat our team with respect and our team treats them with the respect they deserve.

Second, we do NOT release products with known bugs that are deemed critical or major in nature even though it may delay shipment of hardware products.  That is why we took longer to release the new radios than any of us would want.  That does not mean that we won't find edge cases when the radios get into 10 times as many customers as we have on the alpha/beta team.  It is almost a given that we will.  We have a test rack loaded with every radio that runs SmartSDR.  We run an automated script on every radio for every single software build but that won't catch everything.  It is a fact of life that something will ultimately slip through that we need to address.

Every single week (typically on Monday) our software team logs and reviews all issues reported from the alpha/beta team and from the customer base at large.  Any time there is a new major release there is the opportunity to introduce new bugs that may not be caught through almost any test regimen.  For example, I depend on my iPad for both work and personal use.  Apple has completely broken my iPad twice to where I could not use it for days until they came out with a fix. 
Apple's testing resources are almost infinite when compared to those of the entire amateur radio manufacturer's combined and they have critical bugs that they were not aware of when they released.  I seriously doubt they released iOS knowing it would temporarily brick my iPad.  I personally did not hold it against them because I know how hard it is to be perfect.  By the way, the iPad is a much simpler use case than amateur radio in my opinion.  It only has one connector, has a controlled interface, and it does not connect to an antenna.

Steve has described our software process here on multiple occasions as have I.  We take constant improvement very seriously.  Yes, there are things that may cause problems for some that we have not yet addressed.  There may be some things we will never do and others we have put off for awhile that we may soon get around to.  Everything we do requires triage and trade offs.  We strive for continuous improvement.

I can say that our entire software team is focused only on addressing the most important customer experience issues and anything that affects delivery of quality radios to those of you who are still waiting for radios.  Nothing said here will change that fact.  

So we at FlexRadio extend our warmest thanks to our test team!

73,
Gerald