A Surprise Ending to a Networking Problem

I’ve got a networking adventure to tell you about that has a very surprising ending. While the specifics of the story will be recounted purely for your entertainment, there are some steps along the way that might help you in diagnosing technical problems. Remember, we have two engineers with masters degrees in the house so you should expect rigorous, controlled experiments to eliminate variables.

Some elements of this story were reported earlier in an article entitled Eero, MoCA, ONT – Not a Happy Networking Story. I won’t repeat all of that previous story but parts of it must be interwoven for this story to make sense.

Embracing the Smart Home

Steve and I have embraced the “smart home” promise. We’re on the HomeKit wagon, and our computers plus IoT products add up to just under 70 devices on our network. If you look at our device listing, you’d think that I’d employed a strategy of choosing as many different IoT vendors as humanly possible, but I promise I didn’t do it on purpose.

We’ve got switches from Wemo and iDevices and Meross, lights from Hue and Elgato and Nanoleaf, door and window sensors and external cameras from Ring, internal cameras from Eufy, external cameras from Wyze, water sensors from Ring and Ambient, a lock from August, a smart pet feeder from Petlibro, Daikin thermostats, smoke alarms by Google Nest, a smart sprinkler system from Rachio, a Tailwind garage door opener, and a weather station from Ambient.

This entire mess is controlled by an Eero mesh network. Over the years I’ve upgraded or added Eeros here and there, which left us with a bit of a mixture of versions of Eeros over time.

Canary in the Coal Mine

Starting maybe 6 months ago, things started to go wonky on our network. The canary in the coal mine throughout this story has been the Eufy cameras. We’d know things were going wrong when we’d get a notification that said, “Dining room Eufy has gone offline” followed shortly after that by “Dining room Eufy is back online.”

Cycle the Eufys

In the earliest days of our naïveté about how complicated this problem would become, our solution would be to unplug the errant Eufy cam and plug it back in. As soon as we did this to the dining room Eufy, the living room Eufy would go offline. We’d chase the problem around the house, and eventually, it would stop happening…only to start up again a day or two later. At one point we realized the way to solve the problem was just to turn off notifications. Band-Aid successfully deployed!

But then other things started to get weird. We use YouTube TV as our cord-cutting solution, but we hate the interface. I set up our Synology as a Channels server, which lets us use it as a DVR with a much nicer interface. We started noticing stuttering and jankiness of the video. The Tailwind garage door opener stopped responding to Siri. The August lock stopped opening when we got within range. No Band-Aid would help this situation.

Phone a Friend

Our good friend Pat Dengler has a lot of experience working with Eero, and I called her to get advice. Her best piece of advice was to call Eero and ask them to help out. I thought this was a crazy idea, but gave it a shot, and actually got some great help.

When I got tech support on the phone, they scanned my network and the tech helping me noted that I had our oldest and worst Eero set up as the Gateway connected to our FiOS, and the best one was down in the dining room. The dining room was an Eero Pro 6E, while the other three were Eero 6 Pros.

I remember I’d been uncertain whether it would work to put a new one in as the Gateway because with 70-ish devices I did not want Steve to have to go back in and rename everything. The tech assured me that we wouldn’t have to go through all of that. They had me shut everything down, swap the 6E in as the Gateway, wait for it to come up, and then bring the remaining Eeros back online.

Things seemed stable and we were hopeful we’d solved the problem.

MoCA Adapter Problem

Our joy was short-lived. This is the part of the story you may have heard before. I had a call with Bart and my video was hot garbage for him. While the Eeros were reporting that we were getting the full 500/500Mbps symmetrical we were paying for, they were lying. At one point, I measured speeds of less than 3Mbps.

As I explained in my previous article, we don’t have wired Ethernet in our house, we have coax cable. Down at the Frontier Optical Network Terminal (ONT) where our fiber comes in, there’s a MoCA adapter that turns the Ethernet out of the ONT into coax. Then upstairs near the gateway Eero, there’s another MoCA adapter that converts it back to Ethernet.

After many controlled experiments, Steve got the idea to unplug the MoCA adapter on the ONT and plug it back in. Boom! Our network was back up to speed.

My previous article ended with joy that we’d finally solved our network woes. Bless our hearts.

And Then the MoCA Adapter Failed Again

A few days after I posted that article, the network went in the pooper yet again. While unplugging the MoCA adapter fixed the problem the second time, this was clearly not a sustainable solution. I called Frontier and asked them to send a new MoCA adapter but they said they had to send out a tech. I thought it was overkill but they were hard over that this was the best path forward.

I’m glad they won the argument because the lovely Frontier tech not only replaced the recalcitrant MoCA adapter connected to the ONT, but he also replaced the one inside our house, and just for good measure replaced the ONT itself with a shiny new one.

Whew! All was great in the Sheridan network!

Throw New Eeros at the Problem

Until a few days later when the living room Eufy camera went offline.

Back before we figured out that the MoCA adapter was misbehaving, I decided to throw money at the problem and drove over to Best Buy and bought a 3-pack of Eero Pro 6Es. I didn’t install them right away because we thought the MoCA adapter was going to solve all of our network ills.

While we’re still certain the MoCA adapter was a problem, we clearly hadn’t solved the original problem. Steve was also having to fight with the Hue carriage lights on the front of our garage, and the switch in the Living room wasn’t working right either.

It was time to deploy the new Eero Pro 6Es. Adding to our one existing 6E, we could have maintained four total Eeros. But we thought perhaps adding one more Eero to the Garage would fix our problems, so we kept the newest of the old Eeros.

Our final layout was then four Eero Pro 6Es and we put the one older Eero Pro 6 in the main bedroom. We have a 2800 square-foot house, so three Eeros should have done the trick but why not have five?

Well, things didn’t get better. While the carriage lights and the living room switch miraculously started working when we added in the Garage Eero, anarchy started moving from device to device. One of my favorite symptoms was that while my studio is right outside of the room where the Gateway Eero is, my devices insisted on connecting to the garage Eero. It’s not the dumbest choice as my studio is also above the garage, but I was getting terrible speeds with the signal going through the garage ceiling.

I talked to Pat again, and she told me that she’s convinced that having too many wireless routers actually starts to cause problems. I thought they should be designed to do all the handoffs gracefully but at this point, we were ready to try anything.

We unplugged our beloved garage Eero and things stabilized. But of course, that joy was also short-lived with Eufy cams going off and online. We plugged the garage Eero back in because we might as well have the carriage lights and living room switch working even if something else was still wrong with the network.

Let’s Go to Hawaii

We decided to abandon our house and take our kids on vacation to Hawaii. While we were gone, I went to check something on our network, I think it was the automated PetLibro pet feeder, and to my surprise, I couldn’t connect to it. I tried my Tailscale network and I couldn’t reach the Synology or the Mac mini that’s always online. Our entire house had gone offline.

I was very concerned about the cats not getting fed but hoped that the PetLibro would still portion out their food at regular intervals even without a network. I asked our cat sitter and their dad to check our house and see if they could find anything obvious about the network. They confirmed that the Synology had blinky lights on it and that the cats were being fed, but they weren’t able to connect to our WiFi.

I knew we’d gone to the limit of their nerd skills, so I contacted our good buddy Ron and asked him if he could go over and do some diagnostics. You’ve heard Ron on the show before – he’s easily as nerdy as us if not more so.

No Lights on the Gateway Eero

When he took a look at the Gateway Eero, he was surprised to see that it had no light on it at all. Well, that might just be our problem.

The Gateway Eero is plugged into the battery backup side of our Cyberpower 1500VA uninterruptible power supply (UPS). Ron reported the Synology was up with its blinky lights and it was plugged into the same UPS also on the battery backup side.

My only thought was to suggest Ron unplug the Eero from the UPS and plug it directly into the wall outlet. To everyone’s surprise, the Eero turned on and the network came back up. Ron was our hero. He went home and we went back to sipping our Mai Tais and going down the lazy river with our grandkids.

Home Again

After almost enough Mai Tais and lazy river rides and quite a few water slide trips, it was time to face the network music at home. When we arrived home, the network was technically up but things were jankier than ever.

Our Ring alarm was on cellular backup rather than WiFi. Our motion-detection Hue lights wouldn’t turn off automatically. Those pesky front carriage lights from Hue also weren’t responding. Channels wasn’t working on the Synology.

I turned a critical eye towards our gigabit network switch. The Hue lights go through a hub which is connected via the switch, as does the Ring alarm system. The Synology is also on that same switch. The lights looked ok on the switch so I was pretty sure it wasn’t the root cause.

All of these devices get their power through the UPS. The Ring and Hue hubs had power, as did the Synology. On closer inspection though, we noticed the display was blank on the UPS. Steve suggested pressing the reset button on the UPS, and that brought it back to life. The Ring alarm went back to using WiFi instead of cellular, and all of the Hue lights started responding again.

We plugged the Eero back into the battery backup side of the UPS. We also plugged our new laser printer into the UPS for the first time, but the UPS started screaming at us so we plugged it directly into the wall again.

Even though we had things working again, we weren’t sure what had caused the Eero to shut down in the first place. Pat was highly suspicious of the UPS at this point, but we weren’t sure. It’s only six years old and we replaced the UPS battery only two years ago. It’s also frustrating to deal with this because we have a whole-home battery system so we essentially have our entire house on a surge protector battery.

Things Were Still Unstable

We thought we finally had everything working properly but then the Eufy cameras started going off and online again. And yes, I did consider throwing them all in the bin.

And that’s when we come to the final solution, or at least we think we have the final solution.

Steve noticed that the older Eero Pro 6 in our main bedroom had a blinking amber light. We thought we’d seen every light combo before on the Eeros but this was a new one. I looked it up, and it means “Unapproved USB-C power source used.” I looked closely at the power brick and it clearly said “Eero” on it.

However, the connection point of the cable into the power brick had a crack in it.

I went to Amazon and bought a replacement 27W official Eero power brick for $30 and replaced the plug. The blinking amber lights went away.

And guess what? It’s been over 3 weeks since we replaced the power supply on the Eero Pro 6, and not once has a Eufy cam gone offline, or a Hue light refused to react correctly. The Tailwind garage door is even opening with Siri 100% of the time. This is by far the longest time in six months that we’ve gone without that canary singing in the coal mine.

What About that UPS?

Now that the network is stable, or at least we think it is, it’s time to talk about that pesky UPS. Quite awhile ago we had a power outage, and I remember that same UPS didn’t fail over to the battery. I forget why I didn’t address it back then, but today we ran a controlled experiment on it. The display said it was providing power, and it had a full five bars of battery. I gracefully shut down the Synology and then unplugged the UPS from the wall. It screeched at us and showed Error 21 and “fault” on the display, and all devices connected to it powered down.

Steve did a bit of research on the UPS and found that Error 21 could be a bad battery or it could be a bad motherboard. But he also found a really interesting note that said, “Never plug a laser printer into a UPS.” Well, that would have been good information a long time ago!

I don’t think we blew out the motherboard of the UPS when we briefly plugged in the new laser printer, because it failed us on vacation before we did that, and I’m pretty sure it wasn’t working correctly for a long time before that. However, we did have a previous laser printer that could have caused the original problem.

Steve looked up the specs on our new laser printer and the UPS. The laser printer draws 960W peak, and the UPS is rated for 900W.

A new CyberPower CP1500AVRLCD3 UPS is on its way even as we speak.

Bottom Line

The bottom line is that there were a great many failures to track down. We definitely had dodgy MoCA adapters, and the UPS was no longer capable of doing the one job we hired it to do. Today I learned you can’t run a laser printer off of a UPS either. But to me, the most interesting root cause was the dodgy power supply on the Eero.

I don’t know how the mesh software works exactly, but I suspect it must have been constantly readjusting which devices went on which routers. The Eufy cams never should have connected to this particular Eero as it is the farthest one away from them, but all of the other devices must have been getting shuffled around and maybe that knocked them offline.