The ARRL in their best attempt to normalize testing created a procedure back in the 70's so radios could be compared. Manufacturers realized they could improve radio sales by building their radios to beat the test. It gave them bragging rights BUT it also held back innovation in the same way wintel held back the power of parallel processing. Now in some respects this ancient 40 year old testing methodology acts as an anchor to innovation rather than a motor. In the 70's the numbers provided a great way to measure and improve today not so much. In those days people were running HW-101's and receiver performance needed improvement along the criteria the ARRL chose. "Good enough" numbers were long ago reached and continuing to guild the "numbers" has simply become a marketing ploy and doesn't really add value because if you are already good enough being 10dB + good enough is a non efficient use of resources. It's a good deal for manufacturer's stock holders but not so much for the ham consumer.
What is needed I think is a new way to motivate. You can use the same model, and the ARRL would be just the organization to do it. Build a set of criteria that best stresses radio and station performance along the lines of what radios are all about today. Things like 3 minutes to digital contacts by setting a few check boxes and drop downs are immensely revolutionary but there is no measure of that in ARRL tests so it basically doesn't exist. If it was included as a criteria you would see an explosive improvement in software and hardware to accommodate the new criteria.
There was a discussion on a list I frequent regarding this. Radios were characterized as "wide" and "sliver" by N9DG. I think his choice is a good one Wide radios are radios that basically receive their entire spectrum as a big chunk like the 6xxx series. That entire spectrum is available for processing in one way or another. Sliver radios winnow down the bandwidth until 99.999% is discarded and only that 0.001% remains available for processing. The remaining 99.999% is considered to be "noise" or undesirable. Why should the wide radio be judged on the same criteria as the sliver radio. Why should the sliver radio not be held to account for discarding 99.999% of the available data? Another criteria could be ease of modification. A sliver radio is basically set in stone. You may make a few mods but not much can change because of the design. A wide radio being done primarily in software can be completely rebuilt. The example here is the change in the "processor" in the Flex 6xxx series this past spring with the advent of W9GR's method of Hilbert function overshoot correction. It's a radical improvement in methodology but since there is no ARRL criteria for that it barely gets mentioned. Another example is what the HPSDR people and soon enough Flex are doing regarding predistortion. Fabulous stuff as far as advancing the state of the art, but no criteria so who cares?
So in answer to ARRL and Sherwood testing, I think they should be tested but I'm not so sure the testing is any longer nearly as relevant as it once was. Rob Sherwood has even said once you hit about 85dBm dynamic range you are basically good enough for the vast majority of hams. I think W8JI also basically made that same statement on one forum or another. I may be wrong on that one but I'm pretty sure that is a true statement. Narrow and wide dynamic range of the Flex 1500 based on Sherwood's testing is 88dBm so why are we obsessing?
I have my long form reports which cover operational issues, including how the radio performed in a contest. But of course the table on my web site is just numbers. As I have said at several ham presentations, we have become obsessed with wanting or owning a 100 dB radio. Of course bigger numbers are generally desirable, at least up to a point. Back when we had 70 dB radios, which was virtually every up-conversion radio made, the difference between a 70 dB radio and an 85 dB radio was huge. Now the question is, once we have a whole slew of 85 dB or better radios (close-in dynamic range), what else do we look at. Hopefully all sorts of things: clean receive audio with low fatigue, clean transmitter IMD, good ergonomics, stable software/firmware, reliability, warranty service, etc.
Also 85 dB is fine most of the time. The TS-990S tests out between 85 and 98 dB, depending on the band when measured at 2 kHz. This is because it happens to be reciprocal mixing dynamic range (RMDR) limited at 2 kHz on all bands. At 5 or 10 kHz the phase noise is much less of an issue. If the RMDR is 85 dB, and there is a really strong CW station 2 kHz away, the limit may actually be the key clicks of the very strong station 2 kHz above or below in frequency. On SSB the transmitted IMD is virtually always the limit when trying to copy an S3 signal with an S9+30 dB signal 3 KHz away.
wouldn't expect your test results to change based on the other software
enhancements or fixes that I've seen on the road map.
Reply: There actually have been some software issues that affected basic measurements.
Q: Even Adaptive Predistortion (when they eventually add that to the 6000) wouldn't affect the receiver numbers. Does that seem right or am I missing something?
Reply: Certainly predistortion has nothing to do with basic receiver measurements.
Q: Do you think the way you rank the receivers in your Receiver Test Data listing will change to accommodate the SDRs? There are enough differences that you could make a case for that.
Reply: The problem is what is the dynamic range (DR3) of a direct sampling radio, both in the lab and on the air with real signals. If the 6000 series or the Apache ANAN series are tested in the lab, the DR3 value is very dependent on the test level. Unlike a legacy radio where it is super clean until it starts to overload, there is low level distortion in a direct sampling radio all the time. It may be odd order, or at times just some other spurious. Spurious free dynamic range should look at any near-by spur, not just third order. I never published any data on the SDR-1000 since the general spurious was way above the third-order spurious.
Q: At any rate I think many of us are anxious to see where the 6700 lands in your list. (Will it be #2 #5, etc)
That is the problem. If tested at lower levels, like we actually
usually have to contend with on the air (S9 + 40 dB), the DR3 might be
in the 80s.
If tested at levels like S9 +60 dB or S9 + 70 dB, the DR3 may well be
around 100 dB. As I said earlier, real QRM signals on the band provide
incidental dither (a feature not in the 6000 series chip), and may well
smear distortion products into broadband
noise. How does one account for this in a table?
Here is my feeling on the subject, and this is a CW contest issue. Once the DR3 is 85 dB or better, we are going to be fine in a contest / DX pile-up MOST of the time. SSB contests / DX pile-ups are limited by the other guy’s transmitted IMD products, at least until we have a lot of class A rigs on the air, or a lot of rigs with really well implemented predistortion.
Lets take the TS-990S vs. the 6700. Both are in the 85 to 100 dB range, depending on how we measure the radio. After the radio is overload proof “good enough” from a real-world performance stand point, I am going to pick a radio to purchase on all those other very important aspects of what is important to me. I don’t happen to own a K3, yet 63% of the radios in the recent WRTC were K3s. Why was that? It works well, it is small and doesn’t weigh much, and again once the radio fulfilled the basic needs very well, it came down to the operator skills as to who won. (They all had the same antenna.)
the ARRL decides to rate a direct sampling radio vs. band noise, I
don’t see any way to directly compare it to my table or the 40 years of
history of published
data by the ARRL.
Reply: Here is an example of the problem of any table sorted by close-in (2 kHz) DR3. That isn’t the whole picture, and I have never said it was. Take the Hilberling at the top of my table. It has the highest 2-kHz DR3, and it has outstandingly low phase noise (RMDR). But it doesn’t have QSK and its selectivity (300 Hz @ -6 dB narrowest selectivity) isn’t adequate in a DX pile-up. The next one on my table is the KX3. It has really high DR3 and its RMDR is outstanding. However, as the foot note clarifies, its opposite sideband rejection is only 65 dB.
Some hams go nuts over one number, such as a K3 owner asking me if he should sell his K3 and buy a KX3. That is a case of not seeing the forest for the trees.
Flex spent a fortune on making the 6700/6500 have a very high RMDR value, likely higher than practically necessary. Any OEM has to look at the BOM (build of materials) cost and decide where to allocate money to the radios subsystems.
I really liked the 6700 in the CQWW 160 CW contest in January. I also liked the TS-990S in CQWW SSB in October of 2013. Both radios are very different and have their own quirks. QSK was broken with FW 1.1 in January. The preamp gain of the Kenwood was way too high on 10 meters back in October, but has since then been improved.
Today if I am looking at a purchase, there are at least 10 radios that should be in my consideration list. The 6000 series would certainly be one of them. I once bought a $10,000 radio 10 years ago, and it went away after 5 months. It didn’t do enough better to warrant my investment in the radio, so I sold it and put up two more towers and yagis!
Final comment: Some of the numbers the League publishes I think are meaningless. What does a DR3 or blocking dynamic range mean if it is measured with a 1-Hz filter? Not much as far as I am concerned. Now we are going to have to come up with a meaningful way to measure direct sampling radios. Hopefully whatever the ARRL chooses has more relevance than what numbers one can get in the lab with a 1 Hz filter which has no resemblance to how we use a radio on the air.
73, Rob, NC0B
73, Carl K5HK