Reflections on an API-based controller

John G3WGV

I've been busy building a controller for my Flex 6500. It uses the API and is implemented on an Arduino SBC. It's been a fascinating exercise with interesting challenges, both hardware and software - I get a lot of enjoyment out of that sort of thing. If anyone's interested, there's a Blog covering the development at http://g3wgvflex.blogspot.co.uk/. This is what it looks like:



One of the things I was keen to explore was the overall feasibility of finely controlling an SDR via the API. In general I've found that it works very well but there are a couple of things that seem to be not quite right. These may be limitations imposed by the networked nature of the interface, or could be processing constraints, either in the radio or in the controller. I'd be interested in what others think and, especially, in any response from the company.

So, the good bits first...

The API is great! It's quite easy to implement and I've yet to find anything I want to do that is not supported. I love the extremely logical structure of the API: once you understand the basic principles, the entire API is an open book.

As a result I have been able to create a controller that is turning out to be great fun to use in contests, DXing and general "messing about with radio".

And the not quite so good?

Tuning. My controller has VFO encoders that produce 400 pulses per revolution. I hoped that this would give really smooth tuning, permitting me to use very small steps (5Hz, possibly less), so the radio would tune just like an analogue VFO.

It doesn't. At anything other than fairly slow tuning rates the radio seems to miss some of the frequency updates, resulting in much bigger actual tuning steps. It makes the tuning sound like an early synthesiser radio with 100Hz steps.

Clicks. Associated with tuning is apparently random clicks in the audio while tuning. This seems to be especially noticeable when tuning through strong signals. I'm guessing that this is happening when the radio doesn't keep up with the tune commands. I also hear these clicks from time to time on other commands sent to the radio but they are less obtrusive in that case.

I also have a Maestro. I note two interesting things: Firstly, the Maestro only has 64 pulse/revolution VFO encoders and secondly, the same two effects are noticeable (missed frequency updates and the clicks). I was surprised by the relatively low resolution of the VFO encoders but I wonder now if that was a deliberate design decision, to reduce the rate of frequency updates going to the radio.

Let me say at once that these are observations, not criticisms. The API works remarkably well but it is bound to have limitations, especially when there's lots of control transactions going on that need to transit the network and be processed. What I'd like to understand is where the bottleneck is and whether it is possible to do anything about it.

In the meantime, I have taken the same approach as the Maestro, reducing the VFO update rate per revolution. It helps but it doesn't entirely solve the problem.

I'd be grateful for any feedback.

Ken - NM9P

John,  That is very nice work.
I have a few suggestions for your encouragement.  if you have already done this, then please disregard...

1) This may be what you meant by "reducing the VFO update rate per revolution..."  But you might take a look a different "slow-down" routine for your VFO encoders.  When William was working on his interface for the CMD-1 DJ Controller, he had the same problem.  He put in a simple line that required a certain number of "tics" of the tuning knob before it sent a step tune command to the rig.  It smoothed things our very nicely.  He even added a routine that reduced the number of "tics" required as you tuned increasingly faster.  This allowed a customized variable speed tuning.  For really fast swipes of the knob, he even had it change the size of the step.  You can easily vary the number of "tics-per-click" that you want in your routine.  The beauty of working with the API is that you can customize your controller exactly the way YOU want it to be.  as the old adage goes..."It's just a simple matter of programming!"  (alas. nothing is REALLY simple...but it IS possible!)

You may benefit from looking at his source code for the Behringer CMD-1 Micro DJ Controller, posted on another thread.  There are lots of good Ideas to be had there....

2) Don't sweat the "clicks" during tuning.  I get these when using my FlelxControl Knob, too.  This has been reported multiple times and I am confident that FRS will get it sorted out in their code in due time.

3) Keep up the good work.  This is part of the fun of SDR!

Ken - NM9P

John G3WGV

Hi Ken,

Thanks for the comments and encouragement.

Yes, that's exactly what I meant when I said I reduced the update rate. I've found that 100 pulses per revolution (PPR) works pretty well, together with a 10Hz step, giving 1kHz/revolution tuning rate. 200 PPR works almost as well, giving 2kHz per revolution. I can change the division ratios via a front panel control. Automatically variable tuning rates to permit rapid excursions up and down the band, a la Maestro, is next on my list.

On the clicks, it's good to hear that it's a known problem. It's only slightly irritating and the slower tuning rate keeps it more or less under control. I would certainly add my weight to the request that it be fixed.

I like the Maestro but to my mind it has muddied the waters somewhat, taking a totally soft radio and cast it into a physical implementation (i.e. a bunch of physical controls that each have a fixed function). One might argue that these controls will always be needed, hence fixing them physically is no big deal.

My controller takes a different approach. The buttons and encoders are completely "soft" and can be allocated to any function via a "profile" file. This means that you can have multiple layout profiles for any given physical implementation. For example, one might have a profile for contesting, another for data modes, and so on. Or swap VFO-A and VFO-B over, for left handed operators perhaps. It's been interesting developing these soft panel concepts and I think there's a lot more progress yet to come.

73, John, G3WGV

Peter K1PGV

What a terrific project. Very nice work! Super ambitious, and nicely realized. You should be very proud of what you've achieved. Having worked on a number of similar sorts of algorithms, I agree with Ken's suggestion to "batch up" and send tuning commands to the radio in larger groups. IOW, you can make the updates you send to the radio time-based. To eliminate initial tuning lag, you can "prime the pump" by sending a few initial tuning changes immediately when the user touches the knob, and then send follow-on changes every x milliseconds. The trick is finding x, obviously... Balancing between getting the necessary perceived responsiveness and not swamping the radio beyond its capability to respond. I absolutely love your soft button approach. It make tons of sense to me. Have you considered using buttons with programmable displays to show the function that's currently assigned? There are some really super ones available, depending on the interface bus you want to use (i2c, spi, whatever). That would be very cool, I think. Anyhow.... Really great project. Bravo! Please keep us apprised here of how it develops. Peter K1PGV

John G3WGV

Hi Peter,

Thanks for your kind words.

It's an interesting idea to time command transmissions to the rig. As you say, finding the optimum timing would be tricky... and probably different for each implementation. Now I have the code more or less completed, it's the sort of thing I will be looking into.

My immediate concern is that any sort of throttling would in effect create a queue, with the attendant risk of VFO overshoot. Something to play with. Anyway, you seem to be confirming my suspicion that there is indeed a finite rate at which 'slice tune' commands can be processed. And it's not particularly high either.

This got me thinking that it is odd that the only way to tune the radio is by sending the complete new frequency as a float in MHz. I'd imagine that the radio has to do all sorts of processing that wouldn't be required for simple incremental tuning.

What's perhaps needed is a 'slice step' command that would simply step the frequency up or down by the current step value. I would have thought that would require quite a bit less processing and could therefore be accepted more often.

I like the idea of buttons with programmable displays. Something for the Mk2 controller!

73, John, G3WGV

Ken - NM9P

I don't know if a "Slice A Step +" or "Slice A Step -" command exists, but I agree that it would be very nice!

Ken - NM9P

John G3WGV

If it exists, it's not documented anywhere. I can't find it in the FlexLib code, which is said to be the API documentation of record. It would be nice to have if it improved the tuning rate, otherwise probably not particularly useful.

Peter K1PGV

With respect, I don't think you want Freq Up and Freq Down commands. Those WILL lead to either overshoot or undershoot., as the radio processes late or drops increments. Set Freq is what you really need for this job, I think, and that's what we've been given. I would recommend you try driving the displayed frequency from the user's input and not from the notifications of frequency change from the radio (I'm assuming your writing the display code for each VFO display) ... There's just too much latency and variance in waiting for the update, and it's not necessary. Then, accumulate your frequency changes and sendyour timer-based frequency updates... I'd ever start with trying every 50ms. That almost certainly will be fast enough to feel like "real time" and 20 updates a second might not swamp the radio. This will entirely decouple the encoder rate from the frequency of updates to the radio, and you can adjust from there. The real trick, I bet, will be how much CPU capacity you have for this sort of thing. An I-5? No problem. That RPI3? Hmmmmmm.... I wonder. Anyhow, good luck! What a worthwhile, challenging, and interesting project. I'm both impressed and... Well... A bit jealous. I admit it. I'd love to be able to take on a project like the one you're doing. Best I can do at this point is to participate vicariously :-) Best of luck as you move your design forward to perfection... Peter K1PGV

John G3WGV

HI Peter,

Your point regarding the step function is well made.

I do display the frequency directly from user input but of course I also have to process radio updates to keep up with changes that any other client makes. I quickly learned that waiting for the radio to respond gave a soggy feel to the tuning.

I checked my code (this bit was written a long time ago!) and found that I do in fact send commands to the radio on a timer basis. Currently this is set to 10ms and that does seem to be a bit of a problem. That's something of a surprise really - 10ms is an ocean of time in modern-day processing terms. I guess the radio is busy with other stuff. It's probably time for me to revisit this code, as my understanding is considerably greater now than it was back in March when I wrote it.

I'm actually using an Arduino Due and as far as I can tell it is not even close to being a processing bottleneck. Of course I am not trying to deal with the pan adapter/waterfall data - that, I am quite sure would overwhelm the Arduino (or at least the display system). And, of course, the Arduino has no O/S overhead. I'm already working on a Mk2 controller which will use a quad-core X86 processor under W7 but that is more to do with displaying pan/waterfall than basic radio control.

Anyway, as you say, it's an interesting project and I've had a lot of fun learning all about the Flex radio and its API. Thanks for your kind comments.

73, John, G3WGV

Asher - K0AU

John - your project looks very cool.  I'm also slightly inclined to point to the Arduino as the problem.  I've been working on a web based controller using the Chrome browser as a front end.  Chrome supports Midi devices native.

My feedback loop is Midi device -> browser -> commands to server -> commands to radio -> response from radio to server -> response to browser -> update display and it's responsive.  I'm currently running the web server (IIS Express) and the Chrome browser on the same Windows10 i5.

John G3WGV

Hi Asher,

Are you saying that the Arduino is creating the audio clicks and missed updates? I'm afraid I don't get that. I can see with traces that the Arduino is sending the updates to the Flex but I can hear that the Flex is missing some of them. Very early on in my development I discovered that I had to limit the update rate to the radio otherwise the clicks were far worse.

I've considerably improved things by limiting the update period to 10ms but that, of course, either means very slow tuning rates or the tuning step becoming so big you can hear it as a discrete step. I can't really see how the Arduino can be the culprit, especially as it seems to be spending almost all its time in the main processing loop, which it only does when it's not processing interrupts (the VFO runs on interrupts).

I think there is definitely some mileage in experimenting with the setFreq update rate (10ms at the moment) and with the relationship between radio step value and the number of steps per revolution. I'm just a bit surprised it's necessary.

73, John, G3WGV

Asher - K0AU

Not sure if you're processing packet audio through the Arduino...  Historically I've seen lost or delayed audio packets cause clicking, but I can't say I know how this codec behaves.

Do you have any way to benchmark your network and packet decode stack?
WIth 5 Hz steps, 1 KHz/s tuning rate is only 200pps and I'd imagine the Duo SPI can keep up, but it's still worth measuring. 

IIRC the radio may send multiple response packets for each tune command (I think there's an ack followed by a slice change message) so the receive packet rates will be higher.  In my case with a 100Hz tune step I see around 200 kbps bit rate increase from the radio when tuning at 1 KHz/s. I don't have a quick way to look at packet rates; sorry!

John G3WGV

You raise some interesting points Asher and I think I need to go do some more research. As it happens, I have just the tool for the job: I can run my Delphi API on my super fast i7-based development system and try throwing frequency commands at the radio, perhaps just scanning up the band, with various different update/step rates. It's then easy to put Wireshark on that and see what happens. It'll take me a couple of days or so to get this sorted... watch this space, I will report back.

No, I'm not processing audio, in fact I'm not even subscribed to that stream. Nor am I subscribed to pan/waterfall streaming data. The only streaming data I subscribe to is meter data.

73, John, G3WGV

IW7DMH, Enzo

Tuning. My controller has VFO encoders that produce 400 pulses per revolution. I hoped that this would give really smooth tuning, permitting me to use very small steps (5Hz, possibly less), so the radio would tune just like an analogue VFO.

It doesn't. At anything other than fairly slow tuning rates the radio seems to miss some of the frequency updates, resulting in much bigger actual tuning steps. It makes the tuning sound like an early synthesiser radio with 100Hz steps.

Hello John,

you can avoid sending a frequency update each time you read a encoder signal change.
In my controller I use a thresold counter (vfoHits) to do the job and this is very helpful for low rotation speeds.
Anyway this is not enough for high speed rotations. When they happens you can limit the frequency update amount using a logarithm function.
It is the way I implemented a speed sensitive VFO knob and it works very well.

Finally, if you adopted the interrupt driven approach for your encoder handlers (as I suppose), I do not advise you to update the rig frequency using a timer approach code.

73' Enzo
iw7dmh

....
void ExePanVFO() {
  int increment = VFO.getIncrement();
  if (increment) {
    if (vfoHits>7) {
      int v=log(vfoHits*increment);
      VFO.updateCurrentControl(-v); 
    } else {
      VFO.updateCurrentControl(-increment);
    }

    int r=VFO.getCurrentControl().value%VFO.getCurrentControl().step;
    VFO.updateCurrentControl(r); 
    fRig.vfo=true;
  }
}
....

Ken - NM9P

Good point, Peter. I hadn't thought of that aspect of the tuning delay/skipping issue.

KC2QMA_John

AWESOME!  "FlexControl Ultra" ?

Great work!

Steve-N5AC

You should not send a barrage of 1Hz steps to the radio.  Rather you should pick a sampling interval, no faster than say 50Hz, and send coalesced commands to the radio.  So if you see 12 clicks of your encoder in 20ms, you should send a tune command for 12Hz (assuming a 1Hz step size)

John G3WGV

HI Steve,

Thanks for the useful info. I'm actually encoding in 5Hz or 10Hz steps but I can certainly send them faster than once every 20ms. At the moment I send whatever the "coalesced" update is every 10ms but from what you say that is too fast, so I'll make it every 20ms and see how I get on. Thanks for your help.

Update: It was such a simple change that I coded it there and then. Yes, 20ms does seem to work better than 10ms, so I'll stick with that for the time being. Noticeably fewer audio clicks too. May try slowing it down even more to find the point at which the tuning feels jerky - I suspect perhaps 50ms or so.

Update 2: 50ms is too slow, but not by much. The tuning is fine when tuning slowly but just a little jerky at moderate rates of change. So it seems like 20ms, maybe 30ms is a good compromise. Interesting exercise! This could do to be documented somewhere.

Peter K1PGV

Thanks for keeping us updated. Sounds like 20ms or 30ms will be a winner for you. User Experience (UX) coding can be very difficult, but very rewarding. It has to do with feel and the user perception of the interface. As much art as science, in my experience, Peter K1PGV

John G3WGV

As much art as science, in my experience

Ain't that the truth! I suppose it just never occurred to me that the radio would need tens of ms to execute commands. It's not a problem that it does but it was just rather unexpected. So that's another piece on knowledge to add to the pile.

I'm starting to think that I could probably write a useful article on the practical aspects of writing controller code for the API.

Ken - NM9P

John,  Before you go pulling your hair out about the clicking....Check out the new release V.1.9.7 ... Just released today (September 13, 2016) It should have solved the problem of clicking during tuning the band or moving the pan display.

John G3WGV

Yes, Ken, I just saw the announcement and was thinking that this might make a useful difference. That said, I'm a cautious soul and will QRX until 1.9.7 has been out in the wild for a week or two before upgrading. I don't want to have to pack my 6500 off to Germany again this year!

Eric-KE5DTO

You may also want to test this against SmartSDR v1.9.7 that was just released as we improved some of the performance around tuning.

John G3WGV

Hi Eric, yes, I plan to do just that in a few days' time. Will report back!

John G3WGV

Eric - MUCH improved, see my praise posting. Thanks to you and the team.