ARRL and Sherwood testing?

  • 2
  • Question
  • Updated 4 years ago
  • Answered
Just curious mostly. I didn't see a post about this in the last 6 months, but I'm wondering why we still haven't seen any 6000 testing by the ARRL or Sherwood? The posts I saw in the past said that they weren't sure exactly how to measure/test it (?) but seems like they would have figured out a way by now. I know how well my 6700 works, it would just be nice to see it confirmed by people who usually test these things so that the rest of the world knows.
Photo of Robbie - KI4TTZ

Robbie - KI4TTZ

  • 480 Posts
  • 77 Reply Likes
  • happy

Posted 4 years ago

  • 2
Photo of Al / NN4ZZ

Al / NN4ZZ

  • 1710 Posts
  • 581 Reply Likes
Hi Robbie,
I sent you a note offline with some info.  Let me know if it doesn't come through.

Regards, Al / NN4ZZ  
al (at) nn4zz (dot) com
Photo of Gerald - K5SDR

Gerald - K5SDR, Employee

  • 810 Posts
  • 1461 Reply Likes
Official Response
Robbie, I do know that the ARRL very recently purchased a FLEX-6700. I don't know when they will publish the review though. I can confirm that there are ongoing discussions between a number of parties in the industry about how to properly perform receiver testing on direct sampling receivers. Legacy receivers get worse with increasing signal strength while direct sampling receivers actually get better. The ARRL described this phenomenon a couple years back in a sidebar article about their testing of the SDR-IQ. I know enough to say I don't think you will be disappointed. I think that all parties who do these these things genuinely want the data to be meaningful and accurate.
Photo of Robbie - KI4TTZ

Robbie - KI4TTZ

  • 480 Posts
  • 77 Reply Likes
Thanks for the information Gerald!  Looking forward to the results.  I'll have to track down that article you mentioned.
Photo of np2g

np2g

  • 271 Posts
  • 30 Reply Likes
Hello
Have been "Testing" and It will be nice to see the actual Lab grade test  results. In my  real world tests the 6000 series  is Great . 
Photo of James Kirk

James Kirk

  • 79 Posts
  • 12 Reply Likes
One word...dithering, thanks to Phil.
Photo of Lee

Lee, Elmer

  • 677 Posts
  • 285 Reply Likes
I also would like a testing protocol that looks at receiver performance based on architecture.  Cisco's CEO once complained in the late 90's regarding parallel processing that "wintel" was killing innovation.  Back in those days intel would double single core processor speed every year and microsoft would follow suit with small "improvements" in software therefore consuming all the new horsepower.  Your old machine would start to run slowly and an upgrade cycle ensued.  A great strategy for wintel stock holders but a terrible value for customers.  

The ARRL in their best attempt to normalize testing created a procedure back in the 70's so radios could be compared.  Manufacturers realized they could improve radio sales by building their radios to  beat the test.  It gave them bragging rights BUT it also held back innovation in the same way wintel held back the power of parallel processing.  Now in some respects this ancient 40 year old testing methodology acts as an anchor to innovation rather than a motor.  In the 70's the numbers provided a great way to measure and improve today not so much.  In those days people were running HW-101's and receiver performance needed improvement along the criteria the ARRL chose.  "Good enough" numbers were long ago reached and continuing to guild the "numbers" has simply become a marketing ploy and doesn't really add value because if you are already good enough being 10dB + good enough is a non efficient use of resources.  It's a good deal for manufacturer's stock holders but not so much for the ham consumer.

What is needed I think is a new way to motivate.  You can use the same model, and the ARRL would be just the organization to do it.  Build a set of criteria that best stresses radio and station performance along the lines of what radios are all about today.  Things like 3 minutes to digital contacts by setting a few check boxes and drop downs are immensely revolutionary but there is no measure of that in ARRL tests so it basically doesn't exist.  If it was included as a criteria you would see an explosive improvement in software and hardware to accommodate the new criteria.  

There was a discussion on a list I frequent regarding this.  Radios were characterized as "wide" and "sliver" by N9DG.  I think his choice is a good one  Wide radios are radios that basically receive their entire spectrum as a big chunk like the 6xxx series.  That entire spectrum is available for processing in one way or another.  Sliver radios winnow down the bandwidth until 99.999% is discarded and only that 0.001% remains available for processing.  The remaining 99.999% is considered to be "noise" or undesirable.  Why should the wide radio be judged on the same criteria as the sliver radio.  Why should the sliver radio not be held to account for discarding 99.999% of the available data?  Another criteria could be ease of modification.  A sliver radio is basically set in stone.  You may make a few mods but not much can change because of the design.  A wide radio being done primarily in software can be completely rebuilt.  The example here is the change in the "processor" in the Flex 6xxx series this past spring with the advent of W9GR's method of Hilbert function overshoot correction.  It's a radical improvement in methodology but since there is no ARRL criteria for that it barely gets mentioned.  Another example is what the HPSDR people and soon enough Flex are doing regarding predistortion.  Fabulous stuff as far as advancing the state of the art, but no criteria so who cares?  

So in answer to ARRL and Sherwood testing, I think they should be tested but I'm not so sure the testing is any longer nearly as relevant as it once was.  Rob Sherwood has even said once you hit about 85dBm dynamic range you are basically good enough for the vast majority of hams.  I think W8JI also basically made that same statement on one forum or another.  I may be wrong on that one but I'm pretty sure that is a true statement.  Narrow and wide dynamic range of the Flex 1500 based on Sherwood's testing is 88dBm so why are we obsessing?  

73  W9OY   
Photo of John n0snx

John n0snx

  • 229 Posts
  • 54 Reply Likes
Here is the best explanation I have seen of the rating system from Rob Shrewood himself that he posted on another site... I think it is worth a read for anyone concerned with where the FRS 6xxx series rates.  The first and last paragraph pretty well puts all this testing by numbers in a nutshell....

 

  I have my long form reports which cover operational issues, including how the radio performed in a contest. But of course the table on my web site is just numbers.  As I have said at several ham presentations, we have become obsessed with wanting or owning a 100 dB radio.  Of course bigger numbers are generally desirable, at least up to a point. Back when we had 70 dB radios, which was virtually every up-conversion radio made, the difference between a 70 dB radio and an 85 dB radio was huge.  Now the question is, once we have a whole slew of 85 dB or better radios (close-in dynamic range), what else do we look at.  Hopefully all sorts of things: clean receive audio with low fatigue, clean transmitter IMD, good ergonomics, stable software/firmware, reliability, warranty service, etc. 

  Also 85 dB is fine most of the time.  The TS-990S tests out between 85 and 98 dB, depending on the band when measured at 2 kHz.  This is because it happens to be reciprocal mixing dynamic range (RMDR) limited at 2 kHz on all bands.  At 5 or 10 kHz the phase noise is much less of an issue.  If the RMDR is 85 dB, and there is a really strong CW station 2 kHz away, the limit may actually be the key clicks of the very strong station 2 kHz above or below in frequency.  On SSB the transmitted IMD is virtually always the limit when trying to copy an S3 signal with an S9+30 dB signal 3 KHz away. 

Q:  I wouldn't expect your test results to change based on the other software enhancements or fixes that I've seen on the road map. 

Reply:  There actually have been some software issues that affected basic measurements. 

Q:  Even Adaptive Predistortion (when they eventually add that to the 6000) wouldn't affect the receiver numbers.  Does that seem right or am I missing something?

Reply:  Certainly predistortion has nothing to do with basic receiver measurements.  

Q:  Do you think  the way you rank the receivers in your Receiver Test Data listing will change to accommodate the SDRs?  There are enough differences  that you could make a case for that.  

Reply:  The problem is what is the dynamic range (DR3) of a direct sampling radio, both in the lab and on the air with real signals.  If the 6000 series or the Apache ANAN series are tested in the lab, the DR3 value is very dependent on the test level.  Unlike a legacy radio where it is super clean until it starts to overload, there is low level distortion in a direct sampling radio all the time.  It may be odd order, or at times just some other spurious.  Spurious free dynamic range should look at any near-by spur, not just third order.  I never published any data on the SDR-1000 since the general spurious was way above the third-order spurious. 

Q: At any rate I think many of us are anxious to see where the 6700 lands in your list.  (Will it be #2  #5, etc)

Reply:  That is the problem.  If tested at lower levels, like we actually usually have to contend with on the air (S9 + 40 dB), the DR3 might be in the 80s.   If tested at levels like S9 +60 dB or S9 + 70 dB, the DR3 may well be around 100 dB.  As I said earlier, real QRM signals on the band provide incidental dither (a feature not in the 6000 series chip), and may well smear distortion products into broadband noise.  How does one account for this in a table?

Here is my feeling on the subject, and this is a CW contest issue.  Once the DR3 is 85 dB or better, we are going to be fine in a contest / DX pile-up MOST of the time.  SSB contests / DX pile-ups are limited by the other guy’s transmitted IMD products, at least until we have a lot of class A rigs on the air, or a lot of rigs with really well implemented predistortion. 

Lets take the TS-990S vs. the 6700.  Both are in the 85 to 100 dB range, depending on how we measure the radio.  After the radio is overload proof “good enough” from a real-world performance stand point, I am going to pick a radio to purchase on all those other very important aspects of what is important to me.  I don’t happen to own a K3, yet 63% of the radios in the recent WRTC were K3s.   Why was that?  It works well, it is small and doesn’t weigh much, and again once the radio fulfilled the basic needs very well, it came down to the operator skills as to who won.  (They all had the same antenna.)

Q: If the ARRL decides to rate a direct sampling radio vs. band noise, I don’t see any way to directly compare it to my table or the 40 years of history of published data by the ARRL. 

Reply: Here is an example of the problem of any table sorted by close-in (2 kHz) DR3.   That isn’t the whole picture, and I have never said it was.  Take the Hilberling at the top of my table.  It has the highest 2-kHz DR3, and it has outstandingly low phase noise (RMDR).  But it doesn’t have QSK and its selectivity (300 Hz @ -6 dB narrowest selectivity) isn’t adequate in a DX pile-up.  The next one on my table is the KX3.  It has really high DR3 and its RMDR is outstanding.  However, as the foot note clarifies, its opposite sideband rejection is only 65 dB. 

Some hams go nuts over one number, such as  a K3 owner asking me if he should sell his K3 and buy a KX3.  That is a case of not seeing the forest for the trees.  

Flex spent a fortune on making the 6700/6500 have a very high RMDR value, likely higher than practically necessary.  Any OEM has to look at the BOM (build of materials) cost and decide where to allocate money to the radios subsystems. 

I really liked the 6700 in the CQWW 160 CW contest in January.  I also liked the TS-990S in CQWW SSB in October of 2013.  Both radios are very different and have their own quirks.  QSK was broken with FW 1.1 in January.  The preamp gain of the Kenwood was way too high on 10 meters back in October, but has since then been improved.  

Today if I am looking at a purchase, there are at least 10 radios that should be in my consideration list.  The 6000 series would certainly be one of them.  I once bought a $10,000 radio 10 years ago, and it went away after 5 months.  It didn’t do enough better to warrant my investment in the radio, so I sold it and put up two more towers and yagis! 

 

Final comment:   Some of the numbers the League publishes I think are meaningless.  What does a DR3 or blocking dynamic range mean if it is measured with a 1-Hz filter?  Not much as far as I am concerned.  Now we are going to have to come up with a meaningful way to measure direct sampling radios.  Hopefully whatever the ARRL chooses has more relevance than what numbers one can get in the lab with a 1 Hz filter which has no resemblance to how we use a radio on the air. 

 

73, Rob, NC0B    

Photo of Carl/K5HK

Carl/K5HK

  • 47 Posts
  • 9 Reply Likes
Thanks for your explanation Rob.  Like all science the methods and parameters chosen are important and difficult to determine at times as to their relevance and equivalences.  Hopefully ARRL, yourself and others on the cutting edge of that science can come up with "better ways" to determine performance parameters that are important with respect to direct sampling and more traditional radios so we really can compare apples to apples a little better.

73, Carl K5HK
Photo of K1FR

K1FR

  • 109 Posts
  • 9 Reply Likes
Excellent points from all re testing.  No issues.  But.....Well, guess my age (70) is showing in more ways than one.   My 6700 far exceeds my expectations and needs and is simply the best rig I have ever had the joy to use.  I have ear to ear grin every time I turn it on. I recall when my Valiant and HQ-170 made me just as happy!  :)  But.......  Really not very concerned with how the 6700 stacks up against other rigs or if critics throw rocks at it.  I am one happy ham with it and no longer care if others are not.  73, Tom  K1FR
Photo of James Kirk

James Kirk

  • 79 Posts
  • 12 Reply Likes
I have a 5000, 6300, and a HQ-170 that still makes me happy.
(Edited)