Welcome to the new FlexRadio Community! Please review the new Community Rules and other important new Community information on the Message Board.
If you are having a problem, please refer to the product documentation or check the Help Center for known solutions.
Need technical support from FlexRadio? It's as simple as Creating a HelpDesk ticket.

Stop bashing the Beta testers

Pat N6PAT
Pat N6PAT Member ✭✭
Please stop bashing the beta testers for the problems you're experiencing. Unless you've been a beta tester you don't know what's involved in the process.

I certainly sympathize with the Flex beta testers because I am one of hundreds of beta testers for one of the larger streaming services with several million subscribers. We are currently testing a major new application that users have been asking for. The company gets bombarded with emails and posts from impatient users every day demanding the release immediately even though the testing isn't complete.

As a beta tester it is my responsibility to run the app thru every possible scenario that I can think of and report any bugs I find. It is NOT the responsibility of the beta testers to FIX the bugs but only to REPORT the bugs. The fixing is up to the development team. They must balance the need to continue fixes against the intense pressure that users place on them when demanding immediate releases. I'm sure the same applies to the Flex team.

The equation is simple: The more pressure you add for a quick release the more bugs it will have.  You either wait for the product to be completely tested or you wait for all the bugs resulting from a rushed testing phase to be fixed.

The company I'm testing for has taken a firm stand and will not bow to the pressure. They will instead take as much time as required to ensure a smooth roll out even if that means repeated delays. 

Bugs can get by the most rigorous testing. I know. I've been a system developer for over 30 years. Be patient and I'm sure it will work out.






«1

Comments

  • Lee Maisel
    Lee Maisel Member
    edited February 2018
    I Totally understand this, and have not had any comments, negative or positive about it.

    However, this reminds me of an unresolved issue that has still gone completely unaddressed, I guess all the focus is on the new stuff, and to heck with the user base.

    I am still unable to use D-star, and the issue (and simple fix) was expressed in detail, smoke was blown up rear ends (by Flex reps) and still nothing.

    So, I can understand people's frustrations.

  • Michael Coslo
    Michael Coslo Member ✭✭
    edited March 2018
    Hi Pat,
    I have a number of Radios, a couple Icoms, a Kenwood and a Flex on the HF end and Kenwoods and Yaesu for mobile VHF/UHF. And a PW-1 amp from Icom.

    Each radio has some issues. Like the PC tuning caps on the Icom circuit boards, the Kenwood D-700 makes an occasional motorboat on very cold mornings, and so on. And while trying to buy a separation cable for the PW1, they don't even make it any more. But I can buy a whole brand new PW1 Amp from them.  But cures are few and far between. It is just something that goes with the territory.  And you don't open a trouble ticket if the company denies there is a problem.

    And you are exactly correct about the pressure applied and the results obtained. The Icom 7610 is suffering from some birthing issues from just that.

    Every radio or corporation involved in a radio will end up with some people hating them. Just how things work.

    But I defy anyone expecting perfection or better service from some other manufacturer.
  • Scott N8UMW
    Scott N8UMW Member ✭✭
    edited February 2018
    I certainly would not bash them. In the end, their experience helps us all. I would love to be a beta tester.
  • W9OY
    W9OY Member ✭✭
    edited February 2018
    Testing this radio proves to be even more challenging than many other testing situations.  Not only is there the "radio" which now is actually 7 radios plus a maestro, there is the particular computer on which the radio resides and the particular network, as well as particular use cases like Maestro v Mac client v SSDR and the 3rd party programs in use.  The permutations are not trivial, but bugs get squished as we understand the problem.  I did a quick review of my alpha channel inbox and I have collected over 6300 emails related to the development of the "radio" in the past 3.5 years.  

    73  W9OY
  • Lawrence Gray
    Lawrence Gray Member ✭✭
    edited April 2020
    Sorry, but not all companies release products with known bugs. I ran an electronics company making sophisticated products containing complex firmware. We did not release firmware versions with known bugs, nor did we allow customer pressure to dictate quality testing procedures or release schedules. If Flex is releasing software with known bugs because of customer pressure (I have no idea if this is being done), the company is making a serious mistake.
  • John - K3MA
    John - K3MA Member ✭✭
    edited April 2018
    Your right that the Beta testers should not be blamed for a ineffective and weak testing program.  It clearly rests on the sholders of Flex management.  In absence of any customers vocalizing that they are not happy there is no pressure for Flex management to improve the testing process.  I would submit the multiple missed rollout dates as a good enough measure that things can be improved.  It does not seem to be working for them (at least from the perspective of their customers) and as such maybe they should benchmark some world class testing processes. If they did so I am sure they would find much low hanging fruit that could easily be implemented to improve there testing process and release success.

    Instead of "As a beta tester it is my responsibility to run the app thru every possible scenario that I can think of".

    It could be replaced with:
    1) Measures of testing success tracked and reported daily or weekly.
    2) Detailed testing scripts with tracking timelines and a feedback loop to improve the scripts.
    3) Timely follow-up to determine if testing is on track and reallocation of testing resources.
    4) Feedback of testing timelines into the overall project plan.
    5) A follow-up program to identify "How to improve success of the testing measures."

    John
  • [Deleted User]
    edited April 2020
    Flex makes a good product. They are getting a bad reputation for the delays and software bugs. Been in several QSO's lately where I just disappeared because of lock-up and got the "what you have to do reboot that flex" . . . That is NOT good.
  • Rich McCabe
    Rich McCabe Member ✭✭✭
    edited March 2018
    When were beta (alpha) testers getting bashed?  I think they all did a fine job and all had different operating styles which should help.


  • Ria
    Ria Member ✭✭✭
    edited November 2019
    Tim or Steve had mentioned this but I'll mention it again.

    There are two sets of testing that go on. One is internal testing by the Flex engineering team. This has the scripts, timelines etc. The other is end-user beta testing, which subjects the software to real world conditions.

    The Alpha team does the latter, not the former. I don't think that I would want to follow a rigorous test plan every time a new alpha or beta release is made. Instead I just use the software as I normally would, and try to break it. We do get the release notes and a list of issues that were addressed in a particular build. So we pay extra attention to that. 

    Like you, I am an end user. I just get alpha releases and provide feedback. I've been in other similar alpha/beta programs, including for ARRL LoTW. I also test devel releases of WSJT-X. They all run pretty much the same. 
  • [Deleted User]
    edited April 2020
    Did your feedback include the lock ups and freezing problems people have been experiencing . Or was everything ok.
  • Bill -VA3WTB
    Bill -VA3WTB Member ✭✭✭
    edited February 2018
    If you know much about Gerald, you would know that customer pressure does not drive his release times. And, it is more than likely bugs in software slip by a few testers and found in general public usage being in the many hundreds.
  • Ria
    Ria Member ✭✭✭
    edited March 2018
    I cannot comment on actual testing as that is NDA. Ask them if you want details. 
  • Bill -VA3WTB
    Bill -VA3WTB Member ✭✭✭
    edited February 2018
    I understand that you wish to speak for all Flex customers, how ever it may surprise you that there are many very happy as well.
    It is nice to see your opinion here and welcome. Gerald has in the past outlined the testing organisation to us, and it looks like they have things well covered.
    The very fact that they have had delays is part of trying to make sure things are right before releasing. And almost all delays were due to late parts arrival.
  • Bill -VA3WTB
    Bill -VA3WTB Member ✭✭✭
    edited February 2018
    Steve, that is true. Keep in mind who the people are that are trying to give Flex a bad reputation. Being that Flex is the leader in SDR many people look for anything at all to bash, and they **** things out of proportion and say misleading and poor information.
    I hear this all the time on air.
    Remember, Flex Radio claim to fame is SDR
  • Lionel
    Lionel Member ✭✭✭
    edited March 2020
    I can't recall any software driven hardware without bugs.  My iPad and iPhone receive software updates fairly often, but then for practical purposes, Apple has unlimited resources. Even with their updates, software bugs remain.  What is the Apple testing procedure? 

    I expect bugs in all things software; the fate of an early adopter. 

    The Flex team does okay in my experience.
  • Ria
    Ria Member ✭✭✭
    edited March 2018
    Bear in mind that some things are really edge cases where they depend on things like windows versions, equipment configuration and different ways of operating. I have experienced the lockups in the past (even way prior to 2.1) but not with the frequency that you have. My radio is pretty much running constantly. I solve problems on a daily basis and finding a root cause is not always easy. Especially true in a complex device such as a FPGA based SDR.
  • Craig Williams
    Craig Williams Member ✭✭
    edited April 2020
    Well, my Swan 500CX has no software issues. Oh the simpler days.

  • Steve K9ZW
    Steve K9ZW Member ✭✭✭
    edited March 2018

    Backing up Ria on this, one of the BIGGEST reasons I'm hardly answering questions put to me on my blog or by email, and have sharply limited participation here at the community is these endless NDA-violating questions.

    Folks, as part of the sign on to be tester we each do an NDA, and more important than the actual document - we pledged to follow the NDA.

    Maybe that responsibility to live up to one's word is a variable for some folk, but then figure out who doesn't receive nominations or support on the rare occasions FRS looks to add a few more testers.

    Now as a reminder here is what I can do under the NDA - I can talk about anything already released, anything the FRS says can be considered "as released" for discussion, show off my test gear if you come to my shack, include discussions of released hardware/software/features in what I write, and ditto I can do photos/videos of released hardware/software/features.

    Like any non-FRS person, I am never going to promise dates, features or anything else.  You got to go to the source (FRS) for any answers on those items.

    I can tell you about how, IN MY SHACK, I'm doing with the released hardware/software/features.  I'm good on that, and it matches what the NDA is about.

    As I've had an excellent experience as an added "alpha tester" with the gear I'm testing, I couldn't begin to throw fuel on the made up stuff some of the community seems to be imagining.

    If I won't talk about it, it doesn't mean I am denying or confirming anything - it just means the question posed directly violates my NDA agreement. 

    Oh BTW I gave my word by signing the NDA to FRS, and I don't have any sort of agreement with anyone else that out ranks where I've given my word.  I am deeply offended when fellow hams think I should set aside an agreement I gave my word to, just to please them.  At least in my case I'm never going to knowingly violate agreement I'd made folks. 

    Hope this gives some light on the NDA world of the testers.

    73

    Steve

    K9ZW

  • Dan -- KC4GO
    Dan -- KC4GO Member
    edited March 2018

    Let's just take on one of your points above and see it it can be improved (With your help) 

    "2) Detailed testing scripts with tracking timelines and a feedback loop to improve the scripts."
    Let's start with the first part of this statement. 
    Don't know which of the the  8 radio models you have, but if you would produce a script for your model and publish it there may be one of the ALPHA testers who would be willing to follow it. Remember to cover all the aspects of SSDR then those that apply only to you radio. Also make sure that you cover all aspects of all possible variances of operating modes and then all aspects of all third party software that connects to the radio. 

    Remember FLEX nor the ALPHA team knows exactly what or how you use or apply the tools available. 
    I have found several times that I though I had a problem and I did it turned out to be mine for not or miss-understanding instructions or assuming I knew what I was doing. 

    Remember also ALPHA testers are not paid even after they tested version 2.0.x they had to then pay the $200.00 for it's used. 

    Thanks for your assistance looking forward to reading through your script. 

    73's
     

  • Dan -- KC4GO
    Dan -- KC4GO Member
    edited March 2018
    Have you checked how clean the TX is lately :) 

    I had a 350 lots of power but dirty
  • Lawrence Gray
    Lawrence Gray Member ✭✭
    edited March 2018
    It is virtually impossible to have much faith in a quality program that allowed a “lockup” bug affecting many users to slip by the testing process?
  • [Deleted User]
    edited March 2018
    I agree 100%
  • Brad A. Steffler
    Brad A. Steffler Member ✭✭
    edited March 2018
    Thank you, Gerald. I love my 6700. I am eager to get my hands on my PG XL and tuner. But I don't want them with bugs in them. Take your time. I am in a service industrywhere I get time complaints every day.If I go too fast, serious mistakes can be made. Better to do things right the first time. I work as a physician/professor in a medical school.

    Bless you, Gerald - and all of Flex.

    Brad
    KE4XJ
  • KY6LA_Howard
    KY6LA_Howard Member ✭✭✭
    edited March 2018

    Gee... I guess you do not have much faith in Apple, MS, Google, Oracle, IBM, SAS, etc, etc.   In fact I cannot think of a major software vendor who has ever NOT had a major bug bite them in the **** in spite of thorough beta testing

    Yet still it happens...

    @Lawrence

    Further I really find it hard to believe that your electronic company Never had a bug show up after product release.

    Heck.. I too ran a company with real time control products.. Our bugs killed people so we really tested and tested and tested them before releasing them to GA..

    Still unforeseen use cases/environments showed up as new bugs...There are just too many real world variations in use cases and environments to be able to successfully test every possibility.  I can tell you a lot of real world unforeseen horror stories

    In the case of lockups.. they did not show up in beta testing or obviously they would have been addressed.  They showed up after GA in a much wider breath of environments and use cases.  Many of those new use cases were added to the beta program.

  • Bill -VA3WTB
    Bill -VA3WTB Member ✭✭✭
    edited March 2018
    Lawrence and Steve,  it is to bad you guys are having an unhappy time as a Flex user, I say that because if you find anything at all to say something negative you never miss a chance.
    It would be great if you both would help the community as owners to give good feedback as to things you find that would help Flex make good changes.
    If you want to be part of the extended Flex team, it is about attitude.
  • [Deleted User]
    edited March 2018
    Flex is definitly not an Apple or IBM ,like comparing apples to oranges. When Apple has an issue it is fixed in a couple days not weeks or months so dont go there.
  • [Deleted User]
    edited March 2018
    Oh and Bill ... my last issue has been Maestro just disconnecting with no networks found in the middle of a QSO and did not hear back from Flex yet as to what the issue is.
  • Bill -VA3WTB
    Bill -VA3WTB Member ✭✭✭
    edited March 2018
    The comment that was made about Apple and IBM was correct, some of their updates break things. It even happens to them.
    I bet if Flex was the same size companies as them, updates and fixes would come faster?
  • Rick Hadley - W0FG
    Rick Hadley - W0FG Member ✭✭
    edited March 2018
    At least the Flex upgrades don’t brick the radio like the Win10; upgrades have with two of our Toshiba laptops. It cost me $150 to get my wife’s fixed and I have to keep restoring mine to a stored restore point every time I need to restart, because I’m too cheap to have mine fixed.

Leave a Comment

Rich Text Editor. To edit a paragraph's style, hit tab to get to the paragraph menu. From there you will be able to pick one style. Nothing defaults to paragraph. An inline formatting menu will show up when you select text. Hit tab to get into that menu. Some elements, such as rich link embeds, images, loading indicators, and error messages may get inserted into the editor. You may navigate to these using the arrow keys inside of the editor and delete them with the delete or backspace key.