Those who have followed here for a while might recall that back in 2017 we switched our internet service from Comcast to a local wireless service through a company called Common Networks. They set up their network as a mesh network, with a couple of different locations at that time with Internet access (the Alameda movie theatre on the east end of the island, and the Digital Realty data center just over the estuary from us in Oakland providing a primary connection for the west end). If a node went down, no problem – traffic was routed around it automatically.
Monkey Brains
Fast forward a few years, and they were struggling and ended up being bought by a San Francisco based outfit called Monkey Brains who operate wireless networks in SF, and a few other locations around the bay area.
They came along and ripped out the little PC that we had on site managing the routing, and replaced the dishes on the roof. There were lots of outages and a lot of our neighbors left and switched to AT&T fiber, which was just starting to offer service around Alameda at the time. We had not had any serious outages, so we stayed.
No More Mesh
That didn’t last. The first few times we had issues, they did not last long, but the feedback was always the same from the Monkey Brains support team: there is a problem at an upstream tower that is affecting you.
Back in 2021, I had posted a reply on Twitter to one of their posts about their improvements to the network in Alameda, commenting that some redundancy would be nice.
A bit more redundancy would be nice too. We seem to have lost the mesh part of the network and be just point to point now.
— John Gordon (@bluedonkey) October 28, 2021
The response was:
We have a redundant backbone mesh. The mesh Common had built didn’t, uh, mesh with our network. Writing this tweet makes me realize we could turn parts of the meshiness back on now. Thanks, gentle reader, for sparking some synapses.
— Monkeybrains ISP (@monkeybrainsnet) October 28, 2021
I posted a long thread about his on Mastodon earlier this year too.
Today’s Outage
This afternoon the network goes down again (it was out briefly in the morning too), and we are left with little communication from Monkey Brains, but at least Alameda is finally on the map. The outage map that is.
That number there is the number of buildings that are offline in the red region. When they eventually called back with status, it was the same problem. An upstream tower is down. Tonight there was the added comment that they won’t have access to the building until 9am tomorrow morning, and they have an ETA for the internet being stable again by 11am tomorrow.
Lack Of Control
As I have commented to them several times now, and I have written in the thread I linked to above, you cannot offer a reliable wireless internet service without either redundancy (for example, a mesh, so traffic can be routed around problem nodes) or control over the equipment and, critically, the power supply for that equipment.
Today, they have connected us up in a way that some nodes at least, should they go down, will take down all the buildings downstream of them. Those nodes have dishes on the root that are connected to power inside the building, without any backup power. Now imagine I am having some work done on the house, and for a while they need to shut off the power. I am without internet as expected, but so are all the people downstream of me. And they didn’t expect it!
Or, I’m going away for a couple of weeks, and decide to just power down all the equipment in my house because I won’t be needing it. Now, not only are all the downstream buildings offline, but Monkey Brains has no access to the equipment to re-energize it, and I am away for two weeks.
Time to Switch?
I wanted to support our local wireless ISP, and I really wanted it to be a success. I was sad to see Common Networks fail so early, but happy to see another local company step in and take over the service. We stuck with them though the transition period, believing that things would improve, and understanding that they needed to make changes, which could lead to temporary instability in the network.
Additionally, wireless has been a big chunk of my career, and something about having a wireless ISP appealed from a technology perspective too. That said, I need to have a internet connection I can rely on and, if Monkey Brains are not willing, or not able, to invest in the redundancy that a wireless network needs to be reliable, then we may well be forced to look at alternatives. Most likely the AT&T fiber (perhaps through Sonic, who I used for DSL at my apartment and was very happy with – sadly, when we moved here they couldn’t offer us service, otherwise we may have still been with them).