Single Point of Failure Remote

KY6LA_Howard

I have been REMOTING from my son's winery in Northern California this Week. When I went to try SmartSDR for iOS v1.0.2 yesterday I could not connect. Trying my various backup connections it is pretty obvious that my internet connection in La Jolla is down as none of my other servers are reachable as well. Fortunately I am still in the USA and will fly back to La Jolla on Friday. So I won't be off the air too long this time. The DREADED SINGLE POINT OF FAILURE HAS STRUCK. I have backups and UPS on everything as well as multiple servers etc. BUT my single point of failure is a single Internet connection with a single modem and single router on the critical path. In this case, I am pretty sure it is the one UPS that powers the modem and router that failed. Likely the battery had gone which is an easy fix. My most obvious solution is to add a second Internet line, modem and router and run a second subnet. Possible but I am time limited before we head back to France next week so I don't think I can get that working before I leave Anyone have any bright ideas as to how I might eliminate the single point of failure?

Jim Gilliam

Hi, Howard,

Last year I had a similar problem where my Internet went down while waiting a the airport to go on a two week cruise. Blah!! When I finally got home there was no apparent failure except I could not connect to the Internet. I had to power-off/power-on my modem before the ISP would recognize my router. So, apparently someone was screwing around at the Central office. Since then, I have put my modem on my remotable powerline **** should that happen again. Also, when I went through the power cycle of my modem, my I/P address changed and had to had to adjust my Dyn.com DNS service to accommodate the change.

Jim, K6QE

Arnie

I think they call it "Murphy's Law." Whatever may fail, will fail, but only at the most inopportune moment!

73 de Arnie W8DU

KY6LA_Howard

But I am looking for a clever way to circumvent Murphy

Ian1

Howard Quality time with your son is assured now. Like you I travel the globe for work here is my backup solution for those Expeditions and rare DX http://www.remotehamradio.com Then you would have a seperate backup network to utlize. Ian

Ken - NM9P

Howard, just yesterday, my "Remote ON: ENGAGED" button somehow reset itself to "Disengaged" leaving me no possible way to turn the rig on remotely, even by remotely power cycling it.  If I had been able to figure that out, I could have called home and had Dee punch the power button to boot up the 6500.

This is also a possible Single point of failure that would render a rig totally unusable without human or electro-mechanical intervention.  

As far as your question - If you want true internet fail-safe diversity, you might wish to obtain a second internet connection from a different provider/means.  i.e. if your normal connection is by Cable Modem, then have a DSL line installed for backup, so that it would require two different companies to lose service at the same time.  

If I were in a communications-critical situation, I would pursue that further.  But it is much too expensive for my small-time operation.  I don't even want to pay for 3Mb upload speed yet!

Ken - NM9P

EA4GLI

I operate remotely and I have gone through the pains of finding many weak points on the setup. I laugh at those that look down on remote operation... it is not a trivial setup.

I would use UPS on the router only if I am at the location. It could be convenient to have internet access in case of a power outage. Some routers will allow you to have a 4g dongle connected and revert to a 4g connection if the main WAN fails.

I would however, only use a surge protector and not UPS if I am not there. That doesn't cover the eventuality of needing an internet/router reset without having internet access. That is a very tricky one.

I have access to neighbors WiFi as back up. I use Fiber and they use ADSL so we are on different connections and providers. I use these tiny https://www.itead.cc/sonoff-wifi-wireless-switch.html wifi switches that will allow me to restart devices. I have the router connected to 2 WiFi switches in series one on my neighbors WiFi and one with my own WiFi.

My first thought though is that these tiny wifi switches have the option of a Default to ON and also a timer ON-OFF capability. This means that you could setup your internet router switch to shutdown and restart once a week. That means it will reset itself every so often regardless of status. This is just a thought, not thrilled about rebooting the router every week... but if you are going to be gone without anyone with physical access that might be the way to go.

Advanced remote switches/routers have a feature where if they don't respond to a ping request several times it will reboot. Not cheap though. 

Some other things I have setup on my remote operation:

• All the PCs will power back on in case of a power failure. This is a BIOS setting and not the default one. I also do not Halt on errors.
• Make sure your Ethernet and wifi adapters do not have energy saving feature on. It will turn them off.
• I have 3 PCs in the same LAN which I can access remotely. I keep them on but with no monitors and on lowest power consumption setting. Each one is on a different segments of the network. One directly to one of the internet router ports, one on wifi and one on the main gigabit switch that distributes internet in the home.
• I have 2 systems to log into the devices, Teamviewer (with 2 step verification) and Chrome Remote.
• I can also SSH into a linux box (a Raspberry Pi).
• I can access the internet router from the outside.
• I have cameras looking into radioshack room and server room.
• I have smoke sensors and water sensors on the floor.
• I rely heavily on direct wire connections, do not trust WiFi, so I have ON OFF switches that are ethernet connected to the LAN.
• In some cases, if I am not going to be in the location for a while I will run and ethernet cable on the ground from one location to another instead of relying on WiFi.

I am working on creating my own event handling by reprograming these Sonoff WiFi switches which are basically ESP8266 devices.

This could be a great thread, I am certainly interested in hearing other ideas.

EA4GLI

Jim,
DynDNS has a free app you can install in any machine in the LAN that will update the IP address in case your router doesn't have that feature.
http://dyn.com/apps/

Arnie

"Hah!" says Murphy. He (or she) is much cleverer than we are. Stil, this is a nice thread to be sharing ways we give ourselves the 'illusion' that we are circumventing Mr. or Ms. Murphy. I like the idea of redundancy but eventually it comes down to how much $$$ are we willing to spend to diminish the rate of Murphy strikes. Building a smarter mousetrap just encourages the development smarter mice. hi hi.
73 de Arnie W8DU

Mike va3mw

Howard

As I have been running a remote base for over 10 years (2006), my modem was my first SPoF.  I called the ISP and they said:  "Your modem is in a state - just reboot it".  Yeah, right.  A 3 hr winter drive away.   

The perfect solution is another ISP (cable or DSL).  However, in my case, I immediately added a Christmas tree timer to power cycle the router/modem every morning at 3am.  It is down for about 2 minutes.  Since then, I've been pretty good.

Lately, my new modem has a network port that disappears.  I proved this and called the cable company.  They said "No one has reported this, so that isn't it".  I said "I am  not surprised as the first thing you tell everyone is to reboot everything and therefor the problem goes away."  The rep said:  "oh"  

I used the KISS principle.  

Mike va3mw

[Deleted User]

Just think if it goes down and you can't get on the air that gives you more time to go to the shoe store with your wife

Walt

Have Flex ship a 6700 UPS overnight to your son's farm, same with a computer and router and throw a wire out the window for the time being.

When you come home next, have the tower and beams installed and you will have a 2nd station for emergency use.  Will work good for the next S. Cal earthquake.   And might be a real low-noise location !

Always fun installing a new station from scratch.

Cheers

Ken - NM9P

I would recommend that if you want to have both UPS backup and the ability to power-cycle your router or other devices that are supported by the UPS, that you place a WiFi power switch BETWEEN the UPS and the device(s) you want to control.  Howard uses WiMo brand, I use NEO, others use more elaborate (and more expensive) laboratory remote power switches.

I looked at post for the ones you are using.  They look very interesting, and are even more affordable.  Do they have a model, or modification, that allows the output to be a simple Normally Open / Normally Closed relay contact with no voltage on it?

Ken - NM9P

KY6LA_Howard

I won't have time to add a second ISP until I am back in the USA in October.

KY6LA_Howard

Sal and Ken All good suggestions. I will get a second ISP when I am back in the USA in October. But too time limited now as I am already it raveling I will be back in LJ Friday nite to diagnose the issue but I am 99% sure it's an UPS failure.

EA4GLI

They do Ken

https://www.itead.cc/smart-home/sonoff-sv.html

KY6LA_Howard

I also laugh at those who think Remoting is trivial. Try Remoting being chased down a beach on Mt Athos while trying to maintain a connection and then tell me it's trivial.

EA4GLI

Also @Ken, at $5 a piece I have a ton of them.
I can buy 10 for the price of a Wimo or Neo. So you start getting creative with them.
With IFTTT and a bit of programming you can actuate them with the Amazon Echo
via voice.

@Howard, tell me about it. It is quite challenging to go completely remote. It is frustrating at times, but ultimately very rewarding.

Another bit, I use a software program called Zentimo xStorage Manager that will allow me to deactivate and reactivate USB devices. Like physically removing them and plugging them back in. Tons of USB devices will hang for no apparent reason (serial to USB) and require an unplug.

Rob Fissel

Howard, 

One thing you might wish to try is a second ISP, and using DD-WRT on your primary router. DD-WRT will allow you to connect multiple WAN sources, designate one as a primary, and use the secondary to fail over to. This feature is typically only found on much more expensive enterprise-grade routers for business. Just a thought...

Rob

Barry-W4TGA

Give a key to a technically savvy neighbor. Neighbors don't let neighbors be off line.

EA4GLI

Another tip, if you enable Remote ON and place an RCA connector with Tip and Ring soldered together on the Flex Radio, it will turn on when DC power is applied to the radio. You can still turn the radio off with the front button without removing the RCA connector in the back.

I would love to have a software button to turn off the radio remotely. As my only option now is to turn the power supply off which is probably not a good practice to turn off the radio.

KY6LA_Howard

I use a WEMO to control a relay across the Remote On connector. So in effect I have a software switch to turn the rig on and off. Definitely not a good idea to turn the rig on and off by shutting Power Supply. Also with battery backups on PS, thePS is usually on even when AC is not there so I have a separate WEMO to control the PS.

EA4GLI

So the remote on works as a power off too?

EDIT: I just tested it and it does.

KY6LA_Howard

That is the final default process. But my configuration is far too complex for any neighbors to reboot. So I have produced a number of YouTube videos for some of my ham friends to reboot things

However this single point of failure is not one of my videos yet It will be by the weekend when I get back.

KY6LA_Howard

On the bright side it failed while I am still in the USA so it can be fixed before I head back overseas. It would be disastrous if it failed while. I am in Andorra 9/17-9/19.

KY6LA_Howard

Yes.

Paul Christensen, W9AC

Howard:

What? You don't have an OC-192 SONET ring installed at your home QTH with parallel backed inverters fed from a Telco +48V distribution system?  Shame on you. :-) 

Boy you're right on the remote complexities.  N4CC and I operate a remote station on the edge of the Okefenokee Swap.  Great QTH into Asia from FL since the site looks down into a north-south river valley. 

I purchased a used telecom shelter from Sprint Cellular.  It has redundant HVAC, genset input, TVSS surge suppression, and security/environmental monitoring all left over from the Sprint folks.   

A rack-mounted Digital Loggers webswitch automatically pings various devices on the network in search of life. If none, the webswitch initiates a hard boot of the crashed device.  BIOS is also set on shack PC to re-start after a power failure and at 5am as a Hail Mary if necessary.  Very few 1 hour long trips to the site.  Between VNC and TeamViwer, just about every problem can be reset remotely.  A telco-operated AC power switch is at the site but we haven't had the telco line installed.  That's a great way to restart a DSL/Cable modem when it can't be pinged from WAN.  Next, we're looking to add high power Wi-Fi transceivers for connection to another site about 7 miles away for a redundant ISP.  The main tower is 140 ft AGL and at that height, there's good line of sight into Jacksonville.

Paul, W9AC

W5UN_Dave

Back in the "old" days before internet I used a device called shack master, which worked over the telephone land line, to work my home radio from the office.. Perhaps such  a device could be used for things like rebooting, etc. when internet has failed.

EA4GLI

Wow, what a great setup! Thanks for sharing.

VK7WH Winston

Howard, I've just swapped over the internet connection at my remote site from a 4G LTE feed to 100 MB/s fiber connection via a 3 hop 5G WIFI link. For the time being I left my 3 Wemo switches connected to the old 4G LTE router.

This is not a totally redundant arrangement at the moment, but as the 4G LTE route has a fall back capability built in, I intend to rout the fiber feed via the 4G LTE router which will then give me less dependency on a single connection in the event of the loss of the primary connection.

Good luck. I hope you get it sorted before you leave

Winston

Dan -- KC4GO

Mike,
 It makes sense to simple.. Here to the first thing the cable company tells you to do is reset all your network stuff. Simple plug the timer into the UPS and let it do the reset every 24 hours at dark 02:30 or what ever time works from where you are going to be a remote operator.
      Now to make it fancy have it be a remote programmable timer so as you move around the reset time can be changed..  Embrace the challenge once you fix this it will be something else.  
       Thanks for the suggestion it's with in my limited budget.
Dan -- KC4GO