Android App

Stan VA7NF

Thanks Walt for jumping in.  Pardon if the terms are wrong but so many times I see an "event" is posted but not cleared resulting in continuous posting, or as William inferred he didn't have a blocking condition. 

W4WHL

Not sure what I did (I really need to implement GIT), but somehow its working very good at the moment.  Wish I knew what change made the difference

I think the issue was a publishprogress command was outside a "IF" statement.  It was causing the UI to be updated constantly, vs only when the statement was true.

But not 100% sure

William

W4WHL

latest video of the panadapter display


https://www.youtube.com/watch?v=A7maMbRqjVE

Walt - KZ1F

And thank you too Stan..Again, I haven't seen William's code so I was completely shooting from the hip (a bit). However, having said that. on the chance you didn't realize this (I try not to routinely toot my own horn) I've been doing software development and architecture for the past 43+ years. I just retired 2 months ago. I have been almost exclusively in the Java space since 2000. Much of what makes queries on Monster so blindingly fast is my code. The UI to Monster is .NET and on a good day I can spell .NET. The stuff that does the real work in ... I don't know how much I should go into this...Java. I am very experienced with nonblocking IO in Java, which uses Executor services. I was telling William earlier that what I am doing is using Netty for it's IO subsystem, not that it needs that level of scalability but it can scale to manage seamlessly thousands of concurrent connections. My reason to be in this space at all is I hate Microsoft products and strategies. I used to be a real Windows **** in the 80's until I moved to OS/2.  I am running an OpenStack cloud in my basement 'farm' and my main computer here (and my wife's as well) is Linux. Right now I am running SSDR on a 15" laptop I created a WIndows partition for (for TurboTax and, now, SSDR). It is safe to say running SSDR plus logging plus dxcluster plus cw skimmer +++ can not be reasonably done on a 15" single monitory. On my qrs page you can see my 'office'. What I am writing is a portable S(er)SDR that can run on anything with detachable panadaptors, better integration with other tools and some things I don't want to go into. This requires a complete rewrite of Flexlib. There is a reason I chose to do that which, again, I won't get into here, but it was the most painful path to having a non Windows SSDR. Every time FRS makes a change I have to  add the new source to my repo, pull the diffs and one by one do a facsimile of those changes for my code. The process is non-productive and, largely, thankless. The outcome though will be an almost identical interface to identical behavior. A few posts back I pointed to a very interesting app that quite well demonstrates on your own hardware what graphics performance you can expect. Prior to ordering my 6500 I had a conversation with FRS about ... well, I was questioning their decision. The response was I was wrong and would be quite disappointed with the results. Not at all. The igthub repo I referenced above will demonstrate this path is more than sufficient to handle multiple active pans w/waterfalls. After the Linux version I will release the visually almost identical same thing to Android then do an HTTP backend such that one could simply use a thin client to control the radio, similar to RHR but more sophisticated. In all cases, all controlling the 6000 series. But, as this is William's thread, I'll let William's work drive it.

William, the only thing running in the UI thread should be UI work. Nothing else. If you use a synchronize, you need to be 100% sure you understand precisely why you are using a synchronize. You should also remove the sleep()'s.
http://developer.android.com/reference/android/os/AsyncTask.html

Ken - NM9P

Really nice, WIlliam.  That would be nice to use with my rig in the camper running mobile.  Stand alone without a windows computer!  Keep up the good work!

W4WHL

I have a ton to learn.  My biggest issue is understanding how to get variables back from other threads.  Currently all my data parsing is in the UI thread, because I don't have enough knowledge on threads.

Maybe you can point me on the best method to pass the parsing functions to a separate thread and return the results.

From my reading in android, only the UI thread can write to the screen.  So my challenge is getting the data to the screen.  Hence why I'm currently parsing all my data in onprogressupdate.   I think this data parsing is what is causing all my performance issues.

Please excuse my ignorance here.  I'm very new at this!

William




W4WHL

That is still far away   But I'm making progress.  I have to totally redesign the UI.

William

Walt - KZ1F

First of all, you've come a very long way in a very short time. You are to be commended. Here is what I would suggest, and this is largely what I suggested earlier but I will expand on some rationale.

Threads are inexpensive, relative to processes but they aren't free. They aren't free to create and they aren't free as far as context switching. There are algorithms to aid in determining ideal thread count but that is outside the scope of where you are and most developers are. There is a pattern in communications programming called leader-follower. There is a relationship between the number of processors you have, on Linux it is found at /proc/cpuinfo. Processors are listed 0->n. On WIndows I think you can just look at Computer/System Info (I believe). If you have an I7 divde the number it gives you in half. Any program or program segment has an "IO-ness". There are few things that are CPU bound and once a task that is running on a particular processor is blocked, for paging, network, user input, etc, that processor handles another task/thread that is a runnable state, in other words it is not waiting for anything beyond a processor to be dispatched to.

The more IO bound a program or segment or activity fragment is, the better...in a sense. The more a task gets blocked the more other things can be done on the same processor. So what are the tasks to be done in SSDR

1) network IO - very IO bound.
2) preprocessing network traffic - this could be cpu bound depending on what has to be done.
3) building a layer or pane or fragment
4) drawing and processing user input.

The more threads you have the more time the system spends in what's referred to as context switching. The more threads you have, relative to processors, the more context switching will occur. While necessary, context switching is 'non productive' time. Actually, I am leading to something so bear with me.

I mentioned leader-follower. The idea of a thread pool or Executor Service is +/- leader follower. You have a finite number of threads (dispatchable units of work), all started waiting to be dispatched. The first available thread gets tagged because there was an IO event on a selector (on one of the connections something occurred, a (partial) packet is ready to be read, the write queue has available space. One of the prestarted threads handles that and, when done returns to the queue. If, while that first thread is handling that the prior packet is ready to be parsed, a second prestarted thread is dispatched to do that.

Stepping back out for a second, consider what needs to happen happen next. What you don't want to happen is data to be ready to read but no available thread to read it. You need to deal with that IO very quickly. Remember I mentioned the LinkedBlockingQueue, dump the record into it, this might be a UDP packet or might be a line from the TCP connection. Everything on the TCP side is English lines terminated by cr/lf. The TCP stuff doesn't need a lot of processing and it isn't all that time critical so each.

There are a lot of 'laws' in software, i.e. Murphy's, Butter's, Moores', etc. The one I especially like, attributed to anonymous is, "there is no problem in computer science that can not be solved by adding another layer of indirection". Some of the readers may actually smile at that, but no matter. The one I am referring to may have been coined by  Knuth, "Get it working, get it working correctly and, if and only if there is a performance problem, optimize it.

So with that, the goal of the IO thread is move an inbound packet from the IO stack to the internal processing queue. A second thread, that may actually be kept busy, it the one that processes the inbound packets and that puppy is the consumer of data pumped into the queue. It starts something like this:

running, is a switch, initially off, but turned on (set to true) once the channel/connection is opened to the receiving port. I never like "while (true)" stmts.
boolean running;
while (running) {
  DatagramPacket  packet = inputQueue.take();
  parse it
   do whatever geometry on it to prepare for the actual draw operation.
   pass it to the UI thread
}
I am thinking you could accomplish all your goals with 3-4 threads in an ExecutorService.
There is a good write-up here https://developer.android.com/training/multiple-threads/communicate-ui.html.
The way to create a small thread pool is:
Executors.newFixedThreadPool(4);
What you don't want to do is a whole slew of non- 'while' runnables.

The other thing that may seem like an alien concept is generally speaking what's running on the Android, and presumably iPhone, is always running. There is no exit. There are state changes so, the while (running) is useful because if / when you shut off the radio the connections to the radio will be broken. Establishing them should be in a thread that waits so many seconds and sees if it can contact the radio. Once it does, set contact established (running = true) restart your worker threads and get to ball rolling again.
It's coming along very nicely...Good Job!

Walt - KZ1F

There is a state machine for Android activities.
If you haven't seen this, this will walk you through it
http://developer.android.com/training/index.html - look at lifecycle

Walt - KZ1F

A really good stretch goal... Use the camera and digi mode to do SSTV or even Heil?

IW7DMH, Enzo

Walt,
I'm already doing this.  But its not working as I expected.  For some reason the Asynctask are still affecting the performance of the other.  Adding thread.sleep for a few ms in each, held some, but I still get hangs.
William

I would consider abandoning the event driven approach for this task. It is elegant but time consuming primarily in small Android devices.
I would consider building my asynchronous thread in this way:

I am assuming that in your code you have already implemented all the rig objects (like Radio, Transmit, Slices and so on).

OUTBOND COMMANDS
- each GUI control simply adds its command to a circular list and updates immediately the related rig object.
- a specific thread access the circular list and empty it sending all queued commands all at once.

INCOMING PACKETS
- a specific thread get status packets and updates immediately the related rig object.
- any modification to a rig object sets a "dirty" flag so the GUI can be aware in any moment if a specific control set is outdated.
- when a dirty flag is found the whole control set is updated (you should use your specific GUI thread for this task).

I would avoid, also, using dynamic linked lists for circular lists. Dynamic data structures are nice to use but time consuming, so it would be better implementing them using simple fixed-length arrays.

In this way you should have a more responsive GUI.

Walt - KZ1F

Enzo, I wouldn't do that, nor would I advise anyone else to.

W4WHL

Walt,

Tons of good information.  Some of that I understand and some I have to research.  Thanks a lot!  I just went through my code, and I told you wrong.  All of my data parsing is done in Async task.

For the TCP task a  while loop constantly waits for new data then if new data is available if parses the data and updates all the necessary variables.  After parsing onprogressupdate is called to update any textviews etc.

The UDP Asynctask is nearly the same.  It constantly checks for data.  If new data, the packet is dumped into a buffer.  The data is then pulled from the buffer, checked for class, if correct class the data is parse and placed into an array.  Onprogressupdate is then called to UI.

  I need to study up on how to better Queue the data.

William

Walt - KZ1F

Interestingly the Android development tutorial also suggests LinkedBlockingQueue, very straight forward. Above Enzo suggested an array based BlockingQueie. This is bad because you never want the UDP recevier to block waiting for a table on the blocking queue. You want to get the datagram and put it on the queue and wait for the next datagram. That needs to be the most blindingly fast piece of code you have as everything else depends on it. If that is slow, everything will be slow. (weakest link antipattern) I would bag the progress indicator all together. Where you'll be getting an infinite number of packets there is no concept of 'almost done'. It's probably a good time to ask this next question. What's your programming background. When you say you are really new to this, do you mean Android programming or programming in general? One of the things Stan picked up on is your use of the phrase "constantly checks for data". This is, plus or minus, what you should be doing in a non-UI thread public void client() throws IOException{ DatagramSocket client_socket = new DatagramSocket(4991); InetAddress IPAddress = InetAddress.getByName("10.80.1.95"); while (running) { try { DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length); client_socket.receive(receivePacket); queue.put(receivePacket); } catch (SocketException e) { running = false; } } }

Walt - KZ1F

Interestingly the Android development tutorial also suggests LinkedBlockingQueue, very straight forward. Above Enzo suggested an array based BlockingQueie. This is bad because you never want the UDP recevier to block waiting for a table on the blocking queue. You want to get the datagram and put it on the queue and wait for the next datagram. That needs to be the most blindingly fast piece of code you have as everything else depends on it. If that is slow, everything will be slow. (weakest link antipattern) I would bag the progress indicator all together. Where you'll be getting an infinite number of packets there is no concept of 'almost done'. It's probably a good time to ask this next question. What's your programming background. When you say you are really new to this, do you mean Android programming or programming in general? One of the things Stan picked up on is your use of the phrase "constantly checks for data".

This is, plus or minus, what you should be doing in a non-UI thread
public void client() throws IOException{

DatagramSocket client_socket = new DatagramSocket(4991);
InetAddress IPAddress = InetAddress.getByName("10.80.1.95");

   while (running) {       try {      DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);       
            client_socket.receive(receivePacket);             queue.put(receivePacket);       } catch (SocketException e) {             running = false;          }       }    }

something to this effect. 
The receive blocks so this is not a spin loop.
The thread that digests the queue is also non-UI and posts
final data to the UI thread.

W4WHL

Walt,

I'm new to both android and program in general.  Its been 20years since I wrote any code.  Anyway I appropriate the advise.  Just for reference, this is my current UDP thread.

@Override
        protected int[] doInBackground(int[]... arg0) {
                try {
                        int port = 4991;
                        DatagramSocket dsocket2 = new DatagramSocket(port);
                        byte[] buffer = new byte[1444];
                        DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
                        boolean wait=true;
                        while(true) { //Loop
                            dsocket2.receive(packet);
                            byte vitacontents[] = packet.getData();
                            int ffttrue = pattern.indexOf(vitacontents, fftclass); //Check if FFT data
                            if (ffttrue > 0) { //If FFT process
                                ffttemp = bytearray2intarray(vitacontents); //Call Method to convert byte to int
                                int[] arrayb = new int[ffttemp.length - 45];
                                if (ffttemp.length > 1443) {
                                    System.arraycopy(ffttemp, 45, arrayb, 0, (ffttemp.length) - 45); //Delete header
                                    int[] temp = new int[arrayb.length];
                                    int numberOfZeros = 0;
                                    for (int i=0; i<arrayb.length; i++){
                                        if (arrayb[i] != 0){
                                            temp[i-numberOfZeros] = arrayb[i];
                                        } else {
                                            numberOfZeros++;
                                        }
                                    }
                                    fttdata = new int[temp.length-numberOfZeros];
                                    System.arraycopy(temp, 0, fttdata, 0, fttdata.length); //create FTT data array
                                    publishProgress(null); // Update UI
                                }
                            }

                            packet.setLength(buffer.length);
                        }


                    }catch(Exception e){
                        Log.e("test", "" + e);
                    }




            return null;
        }

Walt - KZ1F

well, good luck

W9OY

I think being able to set tune power and being able to watch Power out and SWR would be extremely useful when tuning an antenna out at the tower

73  W9OY

GI4FZD

For anyone in the UK, this runs fine on the Hudl2 which can be had for doubled up vouchers at your local T****s supermarket..nice work William

W4WHL

I found the main source for my performance issues with the panadapter.  Its cause by constant calls  to  GC(Garbage collector)  to free up memory.  The panadapter is trying to update faster than the device can free up memory.  This causes WAIT's that hang the App.  This all stems from the constant screen writes displaying the panadapter.

I'm using a 3rd party charting library to handle drawing the graph.  This is probably the root of my problem.  My next action is to write my own code to build the graph.  I wrote the original test in JAVA which uses direct draws to canvas, so I should be able to use that as a starting point.

Slowing down the refresh rate of the panadapter mask the issue, but makes the panadapter useless.

On a side note, OPUS audio is only supported in Android 5+, and I do not have a device to play with at the moment.  I have successfully capture and buffered DAX audio, but Android also does not support 32 bit floating point audio except in Android 5 .  So I could write an algorithm to convert  the floating points to 16bit PCM, but not sure its worth the effort.  For fun I graphed the floating point audio, so I could view the spectrum, but I'm not even sure the Android could even keep up.  Floating point is a hog on resources I here.

So for now I'm not sure where to go with this project.  As my real goal is Opus (aka remote audio) and that will just have to wait till I get a test device.

Without Audio, the spectrum and waterfall are useless.  So not sure if I should even continue messing with it till I have an audio solution.  But I will keep playing and learning!

William

Stan VA7NF

Take a deep breath, relax, breathe out!  Every project hits a block, if not it didn't delve far enough into R&D.

Re the buffers:  For a long running (measure on a clock second hand) tasks I would design a thread safe push down stack of buffers, create one when empty, pull one and use it then push it back when done.  On a PC I also decommit the contents so it doesn't waste real memory.  Managed code is (almost always) a waste of resources, but it is fast to write.

Your audio question seems to be the one that experts in the area (Walt?) may be able to assist.  It seems a tight subroutine that is array aware may convert float to fixed fairly quickly.  Must you have 16 bit PCM?  That doesn't seem to have much dynamic range.

Oh yes, take another deep breath.

Best regards,  Stan

Tim - W4TME

I am sure a KickStarter campaign could raise the funds quickly for an Android 5+ compatible tablet so you can continue.  You are sooooo close.

Walt - KZ1F

William, a couple of observations:
Most any Nexus tablet will be upgraded to 5.x in the coming days.
Opus is the very least of your problems and deals only with remote audio so is independent of the waterfall.
You need to look again at the advise I've been giving you. You are creating lots of independent copies of thing, big things.
What version of Java are you using?  You want to be using at least Java 7, if not 8.
My guess is you are leaking memory. Again, go back over my suggestions.
The take a breath is great advise.
Why do you think you have a GC issue?
Do you understand garbage collection?
That would imply you are really good at profiling and, for someone with no programming experience, being really good at profiling is way beyond where you are.
Java has multiple levels of memory. The first, and least expensive is called Eden space. If you quickly use something and then drop it then it is never 'collected' as all of eden space is routinely destroyed. I'd go back over that snippet of code you showed and, for every line, ask yourself why are you doing that, does it serve any useful purpose.
I am going to say something that will sound snarky and likely offend. If software development were easy, it would be a minimum wage job. For someone with no knowledge, you are doing pretty good. But that means, for you, it will very much be an iterative process.

It's not the runtime's fault.

Tim, could we all do a kick starter program to upgrade our radios for free? I'd especially like a Velocity XL, could I do one to fund that?

W4WHL

Walt,

I agree with all you say.  The reason I think the pauses are GC related is because I'm seeing tons of them being caused by my app.  Specifically the charting library I use.  Killing the chart repaint stops the messages in logcat.  I only killed the repaint to screen, not the processing of data.  That being said, I'm sure there are tons redundancies and other rookie issues with my code. 

I'm not giving up, I was just saying I'm giving up on the audio till I have a Android 5.  This is because I want to wait till OPUS is natively supported.  I'm not even close to experienced enough to develop my own code from the OPUS source.  That's just way to far above my head right now.

As for Android 5, my phone just upgraded to lollipop yesterday.  And my Samsung tablet should upgrade in the coming months.  So at least now I have a device to play with.

I'm currently taking online nano degree in programming.  This is a work payed for program.  And on top of that in my spare time taking Udacity courses in java and Android.  So I'm far from stopping, just stepping back and slowing down a bit.

But don't worry, I'm not giving up by a long shot.  You hit the nail on the head with "Iterative".  This is where I need to step back.  My code is getting long and hard to follow.  I need to break it up into pieces into separate classes.  Its getting way to complicate currently.

William

EA4GLI

Thanks again for doing this WIlliam! Good luck!

Walt - KZ1F

Here's the thing, I am trying not to sound too 'professorial'. I've taught programming before, it's fun but it's slow and the student has to want to be the student. The very worst thing is to say 'do this' only to have someone else say "I think you should do something else". Then it turns into a food fight. Besides, at no time did you ask to be taught.  It would be presumptuous for me or anyone to tell you what you ought to do. unless you, you know, ask for it. Consequently, I've been trying to soft-pedal any advice. When you added that code fragment my response was 'well, good luck' as that was kind of 'code' for 'you don't want to do that' without the negative connotation. 

Something else I haven't commented on is you have referenced you are using multiple third-party packages. That's hugely problematic as the 3rd party package could be buggy, could not be suitable for what you are trying to do with it. You also don't want to be adding sleep() or progress indicators or any other superfluous stuff. Start with the foundation, get it working rock solid before you build on it.

It's not clear to me the version of java you are using is the issue. It's not clear it is not an issue. However, don't think Lollipop is going to be a cure-all. While it might help mask the problem,  it won't be the cure. Again, Opus is not on your list of problems right now.

The problem is you shouldn't be holding on to any data long enough for it to trigger a garbage collection problem, a full gc. 

W4WHL

Walt,

I will take advice any way I can get it.  And since I'm just learning this is really helpful.  I agree with the 3rd party libraries.  The charting library I am using is prob not designed for real-time data.  I realize this, and plan to write my own eventually.

This all started as a test, just to see if I could do it.  You know a personal challenge.  But its grown into more than that.  Its grown to the point where my goals exceed my knowledge.  But that is not necessarily a bad thing.   Its forcing me to step back and learn the fundamentals.  While this can be frustrating, it will help me in the long run.

I have made great personal goals real fast, and with each step forward, I have to take 3 steps back.  This is very frustrating, but this my personal learning style. 

I am definitely listening to your advise, and appreciate it.   Some of it, I just don't understand yet, not that I don't agree. 

As for Java, I am using JDK 8.  My only reason for waiting on Android 5 is native audio support.  This does not mean I stopping till then.  I just meant I am working on anything audio related till then.  I also agree that I have bigger fish to fry.

Thanks for the advise

William

James Whiteway

William, just in case you missed it, Android 5.1 is being pushed out now. Both my S5 phone ( AT&T) and my Nexus 7 took the update. The phone last night and the Nexus 7 today.  So, maybe your device will get the update soon.
james
WD5GWY

Curious about one thing. Do you plan to distribute the source code on the Android app like you have on the CMD Micro and other controllers? If so, I'd be interested in seeing how you are doing some things. Especially the knobs you are using. I cannot tell if those are user drawn controls or custom controls from a 3rd party. I downloaded Android Studio (again) and just got around installing it. I had the older version on my system, but, figured a clean install of everything with Android 5 would be a good idea.

W5XZ - dan

( apologies for getting out in the weeds )  James, did you 'root' your at&t S-5? before 5.1 release?  I did, trying to de-bloat it, but it didn't work very well....73

James Whiteway

No, stock phone, no modes or rooting. james WD5GWY