Is SmartLink down? (2026.06.12 @ 21h22 UTC)

W4AFK Bruno

Trying to connect to two remote 6400 radios. Tried using SmartLink and AetherSDR and they never see the radios. However, when I use RemotePC and connect to a computer that's co-located to my personal 6400 and to our club's 6400, I can run SmartSDR on both LANs and it works fine. So it's definitely SmartLink as the radios fail to appear.

A week ago, many reported a weird SmartLink failure. Like me, they would get "exceed retry times" and "wrong UID/PWD" messages. Flex confirmed this in the forum started monitoring. But the problem went away after a day.

Today, I (we?) can't use "SmartLink" …

Eric-KE5DTO

Abstract

A June 10th AWS maintenance restart caused a mass reconnect that, combined with newly enforced limits at our login provider, briefly blocked some SmartLink logins. SmartLink access was fully restored the same afternoon. Changes to prevent this from reoccurring will be included in the upcoming SmartSDR v4.3.

Summary

On Wednesday, 2026-06-10, some users experienced failures logging in and connecting to their radios via SmartLink. Early that morning, at approximately 0750Z (2:50 AM CDT), a scheduled AWS infrastructure update required all connected radios and applications to reconnect and reauthenticate, producing a large, simultaneous wave of authentication requests.  This is normal behavior that our clients perform after any such restart.

On this occasion, that wave exceeded the rate limit on our authentication provider's (Auth0) Authentication API, and legitimate login and token requests were temporarily rejected.  Two conditions combined to turn a routine event into a disruption: an unnecessary request from our SmartSDR software, and a recent change in how our authentication provider enforces its rate limits.

The SmartSDR client made unnecessary software requests for the JSON Web Key Set (JWKS), the public keys used to verify login tokens.  These are not necessary in the client as they are checked and enforced in the SmartLink server.  These keys change very rarely and are intended to be cached.  This behavior has been present in SmartSDR since April 2024 and, until now, never caused a problem: previous server restarts produced the same redundant requests, but the stated rate limit was not being actively enforced, so the excess was absorbed without visible impact or warning.

Recently, Auth0 began enforcing its published Authentication API rate limits. With enforcement in effect, the mass reauthentication triggered by the infrastructure update drove the volume of JWKS requests past the global limit for our account, and the provider began rejecting requests. Because that limit is global across all authentication activity, the impact was not confined to the clients generating the excess requests; unrelated users on other devices and software also experienced login failures.

Based on the available logs, the impact appears to have ended around 2026-06-10 19:45Z. We are implementing the corrective actions below.

Leadup

We rely on a hosted authentication provider (Auth0) for SmartLink login.  Its Authentication API has a global limit, shared across all login and token activity for our account.

Starting with v3.8 (and through the current v4.2), SmartSDR applications (including CAT, DAX, Maestro, and M model) have requested the JWKS signing keys on each SmartLink reconnection instead of caching them.  On its own, this is inefficient but not harmful, and for more than two years, it produced no visible problem, including during previous infrastructure events that forced radios and applications to reconnect.  The provider was not actively enforcing the stated rate limit, and the reconnection load, while occasionally high, was absorbed without rejection, making the issue less visible.

That changed when Auth0 recently began enforcing its published rate limits. From that point, reconnection behavior that had previously been tolerated became subject to hard rejection once it crossed the stated threshold.

Fault

Affected clients request the JWKS signing keys on every reconnection rather than caching them. JWKS keys rotate infrequently and are meant to be fetched once and refreshed only when a key changes. Aggregated across many radios reconnecting at once, these redundant requests consume a large share of the global Authentication API budget. With rate-limit enforcement now active, exceeding that budget causes the provider to reject requests, including legitimate logins from users whose own devices are behaving correctly.

Impact

During the affected window on 2026-06-10, some users were intermittently unable to log in or connect to their radios via SmartLink. Because the limit is global to all authentication requests, the impact was not confined to the radios generating the excess traffic: users on other software and devices could also experience login failures or delays whenever the limit was being exceeded.

Response

Rate-limit warnings and errors were observed in our authentication provider's monitoring. Investigation of the authentication logs showed that the large majority of rate-limit events were requests to the JWKS signing-key endpoint, hitting the global Authentication API limit rather than any per-user limit.

We confirmed that our provider had recently begun enforcing its published rate limits. This, combined with the morning's mass reauthentication, accounts for why a long-standing behavior produced a failure for the first time on this date.

As more radios and clients were able to connect to the server successfully, fewer clients were making attempts and the issue subsided as reconnection activity subsided.

Timeline (UTC)

2026-06-10 1100Z: Authentication rate-limit warnings begin

2026-06-10 1630Z: Investigation begins

2026-06-10 1945Z: Impact ends

2026-06-10 2100Z: Root cause identified (redundant JWKS requests exhausting the global Authentication API limit)

Root Cause

The incident resulted from three factors occurring together, none of which would have caused an outage on its own:

• Latent fault.  Since SmartSDR v3.8, clients fetched the JWKS signing keys on each SmartLink server reconnection instead of caching them, generating far more requests to the authentication provider than necessary.
• Environmental change.  Auth0 recently began enforcing its published Authentication API rate limits. The stated global limit had not previously been enforced in a way that affected us, so the inefficient behavior had never produced rejections.
• Trigger.  The scheduled AWS infrastructure update forced all radios and applications to reconnect and reauthenticate simultaneously (ordinary post-restart behavior), generating a concentrated burst of JWKS requests.

With enforcement newly in effect, that burst exceeded the global limit, and the provider rejected authentication requests across the account, affecting users regardless of which software they were running.

Lessons Learned

Public signing keys (JWKS) are designed to be cached and should only be used if token decryption is necessary. Clients that need them should fetch them once and refresh only on key rotation or a cache miss, not on every request.

A single global rate limit means one misbehaving code path can degrade authentication for all users. Adequate headroom and isolation between clients are both important.

Corrective Actions

Fix client-side JWKS handling so the signing keys are not referenced if not needed. If they are necessary for token decryption (likely server-side only), ensure they are cached and reused, refreshing only on key rotation or cache miss, and add backoff with jitter on connection or validation failures so a systemic error does not produce a tight retry loop. [Tracked as SMART-12880, coming in SmartSDR v4.3]

Review other SmartLink server and firmware authentication code paths for similar non-cached or non-backoff patterns. [Tracked as SMART-12886]

Evaluate increasing our authentication service capacity (a higher provider tier or rate-limit allowance) so that an anomaly does not immediately exhaust the shared budget. This is a buffer, not a substitute for the client-side fix above.

KD0RC

Hi Bruno, Yes, it looks like SmartLink has an issue. I can get logged in, but it doesn't find radios that I know to be online. Hopefully, it will be fixed soon.

W4AFK Bruno

Thanks Len … This exposes the "single point of failure" around the use of SmartLink. If the server is down, SmartLink doesn't work. So yes, it's simple and reliable BUT also kills everyone if it glitches.

I can use RemotePC to connect to a co-located computer. It's normally used for updates but it can get me on-line using SmartSDR or AetherSDR which is on that computer. Unfortunately, it doesn't help me when I use my Maestro C from home. That requires SmartLink or running a VPN at the radio site.

I am "exploring" the VPN option and getting my ducks in a row before jumping on the bandwagon. Yes … RemotePC is also a "single point of failure", so the next best way is via the VPN.

Here is my question: If the SmartLink server is only used to validate the radio and the user, then lets the user talk directly to the radio, why can't we do the same? After all, the SmartLink approach does NOT require a VPN on the radio side. The radio is broadcasting its availability to the SmartLink server, why can't it be configured so that a user can do the same?

johnn1jm

Down earlier but up here now.

W4AFK Bruno

Hmm … nice, but it's still down for me (San Diego area).

I can log-in but neither SmartSDR/Link nor AetherSDR show any radio. Connecting to the co-located computer via RemotePC and the radio works fine on using SmartSDR or AetherSDR (LAN side).

W4AFK Bruno

2026.06.14, 19h42 PDT - SmartLink is now working to reach my 6400 Florida remote. Using my Maestro C now. Still can't reach the Palomar Mountain 6400 radio via SmartLink. We are looking on our side too.

stevenab

June 15th do radios showing in smartlink? I also have online remote radio setup, using that this morning.

It’s working fine. Anyone have an update?

KD0RC

It looks like SmartLink just came back up for me.

stevenab

mine started working when I rebooted the radio? No signs of power issues, I’m not sure why this happened. First time….

KD0RC

Hi Steven, it was most likely a SmartLink issue, not anything at your end.

Eric-KE5DTO

Yes, we did have an event Sunday morning that lasted around 17 min where the SmartLink server was unavailable. Unscheduled outages on SmartLink have been unusual lately, so we're digging into the root cause on this. You can typically find real time status info at status.flexradio.com. The data that feeds that site is the same one that alarms us when there is an issue.

on7wp

While the status pages shows all green my two remote stations are offline.

Both are running v4.1.5 and as all works (worked…) great I refrain from upgrading…

Is this the problem. One should understand stations 1000 miles away are not easy to upgrade…

ON7WP

rhlanham

Not working for me still. Guess I'll try rebooting the 6600 and my PC?