Is a more structured Alpha/Beta program needed?

  • 8
  • Question
  • Updated 2 years ago
  • Answered
I must admit I have no idea how the current Flex Alpha and Beta test programs operate.  I do not know it the selection of the testers is done using a criteria, if they go thru performance evaluations, if they are managed actively or even it they are paid for their time.

At the Flex Dinner in Dayton it was announced that a more rapid release of updates was going to be a focus area for Flex.  I believe it was mentioned that the target was to release a software upgrade once every few months.

WIth the recent 1.8 release it seems a large number of customer are having issues with the changes and additionally it appears from the community posts that the issues are in several functional areas. 

I certainly like the idea of Flex releasing more rapid updates but I am thinking that any improvement in more rapid releases will be lost if the releases are issue prone such as it seem is the case with the recent 1.8 release.  Not only does this leave a bad taste in the customers mouth but resources at Flex need to be deployed to fire fight and fix these issue and might end up effecting the timing of other functionality development efforts.  Also, feedback from the user base might be delayed as people decide it is in their best interest to wait for others to load the updates and see if they work properly before doing so themselves.

Hence, my question.  Does the Alpha and Beta testing process need to be more structured with specific criteria, test scripts, time and deliverable requirement in order to support the quicker release focus?

I am not really expecting an answer either agreeing or defending the process only pointing out the need for Flex management to critically look at the root causes of this recent failure and what can be done to improve the process to support the achievement of the goal of quicker releases.

Regards, John K3MA
Photo of John-K3MA

John-K3MA

  • 102 Posts
  • 29 Reply Likes

Posted 2 years ago

  • 8
Photo of Gerald - K5SDR

Gerald - K5SDR, Employee

  • 794 Posts
  • 1436 Reply Likes
Official Response
Yes, I am shaking my head at all the free management advice.  ;>)  Unfortunately, there is not much that would apply to this specific problem.

I'm just recovering this weekend from jet lag after 10 days in Germany and the UK for Hamradio 2016 and other meetings.  I have not been in the office to meet with the team since June 20th but here is what I know:
  1. There are approximately 50 Maestros in use on the alpha/beta team.
  2. The alpha/beta team is made up of a cross section of hams with varied levels of technical experience and operating style.  Off hand, I don't know the current size of the team since I don't manage it but it is about as large as our engineering team wants to manage.
  3. There were 0.00% upgrade failures in alpha/beta.
  4. If the alpha/beta team were monkeys, there would still be 0.00% failures since this appears to relate to some type of hardware timing issue related to upgrading PIC software on only to a small number of units.  Technical competence of the alpha/beta testers has no bearing on this specific issue.
  5. You could probably upgrade/downgrade over and over forever on those 50 units and never see the problem.  
  6. When we realized there was a problem in the general release, we immediately removed the download and announced it here.
  7. We are working directly with the small number of customers who reported the problem to expedite the fix at our expense.
  8. If your Maestro updated to v1.8, you don't have the problem.  Keep using it.
  9. We have many quality procedures in place such as full team design sessions and daily code pull requests/peer reviews.  We also do automated scripted testing on all radio models on every build.  However, we do not yet have any type of automated testing on Maestro builds.  That would be more difficult.
In another thread I related that I had a very similar thing happen with on my iPad 2 when I upgraded iOS a couple years ago.  The update temporarily bricked the iPad so that it would not complete the upgrade nor could I downgrade.  Apple has comparatively infinite financial and human resources to do testing but they did not catch this issue until it was in general release.  The problem only occurred on some units, which I learned by Googling the issue.  Fortunately, they were able to clear the problem within a few weeks without sending it in.  

I also recently had a Gateway PC that completely bricked on a Windows 10 update.  This was after it had been running on Windows 10 for weeks.  It gets completely stuck in the upgrade cycle.  I haven't been able to recover it yet.

If it can happen to Apple and Microsoft, it can certainly happen to us.  This is no fun for us either so you can be assured we will do an after action review to see what can be done to improve in the future.  

Happy Independence Day!

Gerald