Everything works fine if I use the VPN.
According to the Facebook experts the radio is having fragmentation and that is why things are broke only on the smartlink mode. Ok I was remote last week and decided to wait until I was physically at the radio to test.
I did a wireshark/tcpdump and see the fragmentation.
Here is the problem. I have jumbo frames and the MTU set to 9000 the entire path to the cable modem ( I have a 1gig symetric connection.)
According to my switch, the radio is connection at 1000 full duplex. It is impossible to hard set the network settings on the radio, thus it is impossible to troubleshoot.
The support I have seen on this is that people have changed a setting and things start working?
I do not run consumer gear. Thus ideas where to go?
1. I have jumbo frames enabled the entire path.
2. MTU is set to 9000 the entire path except the radio which I can not verifiy.
3. The radio ethernet is connecting at 1000 full duplex.
4. I ran a sniffer via a spanned port on the local switch and and see fragmentation on the local port. This means the radio is probally using a low mtu for some odd reason.
5. I have 1gig symetric with static ip space.
6. I am running a complete ubiquiti stack, all of the firmwares are current version.
7. Other udp fragmentation sensative protocols like voip, and L2TP work fine.
8. Everything works fine when accessing the radio via L2tp, openvpn, and Softether vpns.
9. The radio is installed on a secure iot lan segment due to the fact that the OS is not accesible thus its security level can not be verified.
This particular 2.x problem is a showstopper since I can have the same working functionality with a 1.x firmware. I am also having the audio drop problem when running digital modes, but that has a work around.
I would be I only know enough about jumbo frames to be dangerous other than everyone has to play nice in both directions through the entire chain.
If you have jumbo frames turned off totally, do you have the same problem?
I also had your problem, and I was using pfSense as my router at the radio end. At some point, I started to see the same issue.
As a test, I swapped out the pfSense router and replaced it with an IQrouter. That was 3 weeks ago and it has been flawless since.
I suspect it was due to traffic shaping on the pfSense router that caused it. Others (like Ria) are running pfSense without issue. For me, my upstream bandwidth was limited at 1mb/sec as that is all I could get from the ISP.
My 2 cents (and certainly not the solution).
(speaking as a user, not an employee)
David has done a great job of providing some data that may help to isolate this.
However (speaking on my past IT career--not on behalf of FRS or our wonderful engineering team), this may or may not be a FlexRadio issue.
You may find it a surprise that there is no standard when communicating over the internet. Just a pile of comprises. :)
The black box of death is just a symptom of something else and engineering may be able to figure it out once they know what is causing it. Right now, it is not that simple.
For most (not all), SL works. All of us have little control on the pipeline that connects one end to the other and there are a LOT of players in between. Wifi radios, routers, switches, etc. It only takes one not to play nice before we notice it.
Other applications like streaming TV, etc, it may already happen, but with those modes, you can have a 5% packet loss and not even notice.
my 2 cents as someone who has had to deal with moving data in a hurry from my previous career.
The black box of death is just a symptom of something else and engineering may be able to figure it out once they know what is causing it.
The number of bytes that comprises a frame of display data can, and usually is, greater than 1500 bytes. We send display data to the client in single frames for rendering. The display data transport is UDP (VITA-49) and the maximum Ethernet frame size on the Internet is 1518 bytes which include a 1500 byte data payload. Since the display frame data can exceed the maximum size of a single Ethernet frame, the radio's TCP/IP stack fragments the data, meaning that the display data frame is divided up into multiple VITA-49 packets and the client host's TCP/IP stack reassembles them into a single display frame. IP fragmentation/reassembly is an Internet Protocol (IP) process that breaks datagrams into smaller pieces (fragments) which are reassembled by the receiving host running SmartSDR. This capability is standard and required in all TCP/IP protocol stacks as per RFCs 791, 815 and 1191.
If somewhere in the connection between the radio (this includes the LAN and the LAN router), the multitude of transport providers on the Internet, and the LAN and LAN router at the client end of the connection does not allow fragmented packets to pass through so that the client host can reassemble them into a single display frame, then you will experience the frozen display and no waterfall data.
Now, it is very unlikely that the transport providers are not in compliance with RFCs 791 and 815 so that leaves the radio and client ends (endpoints) of the connection as probable areas where the issue is occurring. Edge routers that are used for these endpoints usually have firewall features and by their very nature block different traffic types. With some routers, you have control over the features and in others, you have very limited control. Some client locations prevent fragmented packets as a security feature because, in the past, hackers have used this part of the TCP/IP protocol as a DoS attack vector. Rather than using a stateful process to inspect the packets they use brute force methods of denying all of them. Other client locations deny UDP fragments because it represents a possible high bandwidth usage scenario that they want to prevent rather than using a stateful firewall that operates at layer 6 and 7, again, another brute force method to control data flow. If one of these features is preventing the multiple VITA-49 packet fragments from traversing the firewall/router, then the SmartSDR client is not getting the data it needs to reassemble the packet fragments and build a complete frame of display data.
This behavior is easy to validate. If you have this issue, start reducing the size of the SmartSDR application so that the panafall's geometry begins to get smaller. At a certain point, it will reach a size where all of the data for a single display frame is 1500 bytes or less. In this scenario we do not need multiple VITA-49 packets to construct a display frame; fragmentation is not required and the display will begin rendering.
When we engineer a network feature, we do so in order to achieve the most efficient transfer of data while being compatible with the greatest number of client configurations. In the case of how we transfer display frame data, we chose the method described above because it uses the least amount of CPU resources on the radio where they are finite (the fragmentation/reassembly is performed using the TCP/IP kernel mode daemon) and fragmentation is a standard component of the TCP/IP protocol stack, so it is universal. We have been using this method of rendering display data since before SmartSDR v1.0.0 was released and it has worked flawlessly.
Since a majority of customers who are using SmartLink are not globally experiencing this particular problem, meaning that they connect to their radio from a different location or Internet provider and it functions as intended, our primary design goal has been met. For those instances where this was not the case, turning off certain firewall/router or client based Internet security features has resolved the problems. This does not mean that every use case will experience success. Cases, where the radio or the client end of the connection (LANs), are using a variety of firewall policies (either configured or unbeknownst to them which is usually the case) or configuration that would be considered advanced in nature may not operate properly. In these cases, we will work to identify the cause of the problem, but FlexRadio's support mandate does not extend to advanced network troubleshooting, analysis or issue resolution.
Since we are aware of the situation, we have defect listed in our bug tracker to investigate possible remedies for datapaths that are blocking VITA-49 fragments. We have some ideas, but until we investigate and analyze the impact that has on the radio operational performance, we are not ready to commit to an alternate solution or a time frame. I hope this clears up the nature of this situation at hand.
And I hope David can find a resolution to his issue as he has been provided the key information as to why his issue is occurring.
David, If you believe the radio is at fault, then open a HelpDesk ticket and send it in for validation. We'll put in on our LAN and I'll do the client testing myself since both networks are known good running SmartLink daily. I use my location for worldwide demo purposes, most recently from Germany and Japan. If there is a problem, we'll correct it.
Next I switched the switch to 1522 to make sure I did not have some forgotten vlan in there. SAME PROBLEM.
I then changed layer 3 to as CISCO ASA I had sitting around... Guess what - SAME PROBLEM.
Then I plugged the radio directly into the ASA - SAME PROBLEM.
I then took the ASA and changed it from my primary COX Business connection to my google fiber home connection. - SAME PROBLEM.
I then took the ASA and hooked it to my netgear LTE modem that I use as a backup path, and guess what SAME PROBLEM.
Next I took the ethernet cable hooked to the radio and the asa and then repeated the above scenarios.
I am now convinced the problem is either in the software somewhere or the radio is broke.
I will open a ticket tommorow. My problem is that I live on the road and honestly dont have the personal time to troubleshoot IT crap.
This problem gave me the justification to rework my stack at home and make damn sure everything is current. I am at wits end.
I really do like this radio. I want to step up someday, but things like this make me want to pull my hair out.
I have a similar problem. It does not work on Brighthouse/Spectrum networks and I have read on here complaints of similar problems from other amateurs using different networks. I get around it by using a Verizon mifi, which is far from ideal due to poor signal and additional cost. Flex Radio Systems (FRS) blame the ISP the ISP say it is not their fault they are blocking nothing so its FRS fault.
In my mind Its like designing an app which works on AT&T but not T-Mobile. That would appear to be a design constraint or a Project/Quality Management issue in that important aspects which should of been captured when assessing the technological environment were missed and not incorporated into the design.
However there are other aspects of smartlink which are fantastic, I love the ease of connectivity, that is a great job!
I think FRS should consider the limitation and implement a design change asap. In the interim it is only right that consumers are made aware and FRS should put this limitation on their sales documentation i.e. that smartlink software does not work on all networks (perhaps with a list of networks it works on and those it doesn't work on) so that consumers can make an informed decision.
The unfortunate thing is that the fix we've seen solve the problem has typically been to switch a setting or for the customer to replace their router. I don't want anyone to have to do either of these things if it's something we can fix. Our alternative to using the fragmented protocol is write our own fragmenter/assembler on either end and not rely on this part of the UDP/IP protocol. We have an issue for our engineering team to go investigate this and see if we can fix it (I suspect we can). My preference would be a fix in our software that doesn't rely on this part of the protocol or checks to see if we can and, if not, ratchets down to our own fragmenter/assembler. We'll get to this as soon as we can. If you are having the problem, please enter a help desk ticket and let Tim know and when we believe we have a solution I'd like to send it to you to verify (because we have not see this problem except in one case where we couldn't get a Maestro to play remotely).
This conversation is no longer open for comments or replies.
This conversation is no longer open for comments or replies.