Automation is not Laziness

It occurred to me recently that when I have been doing things around the house that automate something, those activities have often been branded as me being lazy.

The Roomba

The Roomba was the first example of this, though at the time I just ignored the labeling. Having a device that can trundle around cleaning the floors for me without my needing to be pushing it is great. The argument was made that the Roomba was not as effective, nor as fast, as the Dyson in the closet. But that argument is false. On paper, the Roomba clearly does not have the suction power of the Dyson, but it can do the job unattended, meaning the floors are vacuumed more often than they would be with the Dyson. As for the speed, while it does take much longer for the Roomba to complete the job, it doesn’t expend any of my time at all. That makes it infinitely faster from my perspective.

Is it lazy though to push vacuuming onto a robot? I don’t think so. Especially not if I am spending the time I would have spent pushing the Dyson around doing something more valuable. It also impacts the overall quality of life for the family; one less chore that needs to be delegated.

The Dishwasher

Another one the household appliances that was seemingly tagged as making people lazy was the humble dishwasher (though, oddly, I have not heard the same complaint about the washing machine in our laundry room). For me, the dishwasher not only saves my time, it also typically does a better job at getting things clean, and saves on water. Once again, it can run when I am doing other things around the house, or it gives me some time to spend with the kids rather than standing at the kitchen sink washing dishes.

Alexa

Just before Thanksgiving, I finally bit the bullet and upgraded the Insteon hub here in the house to the newer version that can integrate with Amazon’s fantastic Alexa gadget (something we’ve had in the house for a long time now and been using for music, weather, news, shopping lists and more). In addition to the Insteon lights, I hooked up the Nest and August integrations, so Alexa could control the house lights, the thermostats and our front door lock (she will only check the status or lock the door though, not unlock it).

My kids were hooked on the light control immediately, and our five year old has been showing everybody who visits Alexa’s newest trick. But is it laziness to ask Alexa to turn a light on or off rather than getting up and walking to the switch? Or to have her adjust the temperature on the thermostat instead on getting up and adjusting it manually? Of course, in both cases I could also just pull out the phone and use that, and in the case of the Nest, I can even make the change from my watch. I imagine those are also considered “lazy” options.

Is voice control ever useful? Sure! in the month or so we’ve had it, I’ve already used it a few times when I’ve had my hands full and needed a light turned on or off. I don’t think that is laziness; I see it more as improving efficiency.

Oddly, even though controlling the lights or thermostats remotely is laziness, I noticed that controlling the TV without getting up was not considered to be lazy. I wonder if the TV remote control was branded as laziness when it was first introduced?

Smarthome

Right now, I have the basic elements of a smart home, but I don’t think the home is really that smart. Sure, the August can tell the Nest when I go out. The Nest smoke detectors can tell the Nest thermostats when there is a carbon monoxide issue or a fire (and have the heating shut off). The thermostats also tell the smoke detectors when we are out so they can run self-tests without disturbing us.

But I am looking for more. I’d like the August to turn on the lights inside when I unlock the door at night. Or if the Roomba could disable the motion sensors on our home alarm system while it was cleaning, and re-enable them once it was done (so I could schedule the vacuuming to run while we are out without it resulting in the police breaking the door down to check for intruders).

Luckily, for this there are more and more options coming online, connecting the various APIs for each service. All I need now is some time to set them all up. And to keep replacing components in the home with ones that have APIs. I’d love to have our home Wi-Fi router be able to report when certain devices come and go, or trigger other activities based on who is at home, or who is arriving home. Our Apple Extreme cannot do this, but perhaps the next router I buy will have that capability (or something else I can connect to the house will).

Presence (for the future)

The next step I suspect will be to add beacon sensors in certain rooms so the house “knows” which room I am in based on either my phone, or my watch or some other wearable device that I choose to register as synonymous with me being present. Would it be lazy to have the kitchen lights automatically come on, and perhaps have a coffee start brewing when I come down at 6:30am to make lunch for our pre-schooler? Maybe Alexa would even know I walked in and greet me, or start my flash news briefing or a review of my calendar automatically too.

None of this is actually that new. I attended a lecture talking about “smart” badges probably 20+ years ago in the UK. But the age of Bluetooth wearables and smartphones makes a special badge unnecessary. And services like IFTTT make all of the interconnects simple too. We live in exciting times, as long as you think beyond automation being lazy of course!

Comcast: Now It’s The Wi-Fi

Last Thursday I spent almost 2 hours on the phone with a couple of engineers from Comcast. The one who lead the call was sitting in the Hayward head end location, connected to the same network I was using (I’m not sure where the other one was physically located – he was dialed into the call). Around 10pm, we both started to see “anomolies” in the Speedtest results. As well seeing packet loss in the traceroute results from the Sunnyvale router. We also saw longer than typical ping times to Google (normally low tens of milliseconds from my home).

He tried a number of things, but nothing made a difference. Until, at 10:20pm, it rapidly went back to being normal. The pings improved. The Speedtest results improved. The traceroutes started showing responses from Sunnyvale again (and by 10:30pm they were also consistently good times). Unfortunately, it didn’t go bad again that night while we were on the phone, but I was hopeful that having had two of their engineers see what I was seeing there might be a light at the end of the tunnel. I should have known better.

Wi-Fi

During that call there was a question about how I was connected to the network, and we even switched my computer between the 2.4GHz and 5GHz bands on my Wi-Fi to verify that made no difference. It did not. Where I live, both bands are pretty clear of interference. Normally I have the two bands using the same SSID, but during the call I switched the configuration so they were separated (allowing me to easily switch between them).

I did also point out at the time that it was somewhat irrelevant since I had seen and documented the problem in many configurations, from many devices, including some of my neighbors homes and even some friends in other bay area locations.

To be clear here, I have seen the problem:

  • On Wi-Fi connected computers (both bands)
  • On Wi-Fi connected mobile devices (both bands)
  • On ethernet connected computers through my router
  • On ethernet connected computers direct to a Comcast-provided modem
  • On both our wired and wireless Roku boxes

Additionally, at other times in the day there is no problem at all from the same computer I was using during the call. Here is a speed test from that same computer taken around 6:45pm this evening, while we were streaming HD video from Netflix on the Roku and a visitor was watching a basketball game on his Mac. As I’ve made clear throughout this, outside of the 10pm to midnight (some days a little later) window, the performance is very acceptable.

That was run from the browser on my Mac Mini connecting to the internet over the same Wi-Fi network. The only difference was I ran it at 6:45pm rather than 10pm.

Home Visit

So, today’s call from the engineering team did not go well. Firstly, even though I saw the slow down over the weekend and yesterday (I was not online at the right times on Sunday or Monday), apparently the equipment in the Hayward location saw none of it.

Furthermore, he now believes that the “anomalies” seen in the Speedtest results during our call last week were related to Adobe Flash and not the network. While I am not going to defend Flash at all, I will say that Speedtest is rock solid, and the versions I am running are not Flash based, so that doesn’t apply to my results.

Finally, it was suggested that perhaps the issue is in my Wi-Fi network and they should come to my house to check the setup in the house. Really. I am not kidding. Once again they are concluding that their network is fine and any issue must be in my house.

When I pointed out that several of their engineers have already been here, and perhaps he should just pick the phone up and call one of them, I was told that they would need to come out to see it themselves. I guess they don’t trust any of their colleagues to check the modem and signal quality.

Once again, Comcast folks, if you believe that something in my house is the cause of this, please explain to me how so many of my neighbours see the same issue. Is my Apple Airport Extreme somehow managing to interfere with them all?

Also, I’m willing to accept that the packet losses in Sunnyvale are coincidence, if you can show me what else in your network could cause so many people in this area to see this issue. And why they correspond so well in terms of time to when the network quality is poor. Right now, when the network is working well, I see this from my traces:

5 be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 11.224 ms 16.193 ms 13.003 ms

The only time I see slower ping times or no responses from that node is when all the other things I try are bad too. What are those other things? Well, here’s a list of the things we often see when it is behaving poorly:

  • Web pages, Facebook content, Twitter content and emails not loading;
  • Netflix and Amazon Prime video not being able to load their index screens;
  • If we have the index loaded, Netflix and Amazon not being able to start playing videos;
  • Videos that are playing keep pausing to load, which can take many minutes to recover from, and in many cases times out;
  • Speedtest shows results under 1Mbps for download (compare to the speed above when it is working)

Experience

What I haven’t mentioned to Comcast until today is that I have a little bit more knowledge about Wi-Fi than perhaps their regular customer. Having been working in a Wi-Fi software company since 2003, and even been part of the creation of some of the Wi-Fi standards, I think I am pretty well placed to know whether my Wi-Fi setup here is likely to be the cause.

I have spent perhaps too much time inside shielded rooms helping access point manufacturers fine tune their software to squeeze the maxiumum performance from their Wi-Fi stacks. I am well aware that the 802.11ac system I have here far exceeds the capabilities of the Comcast network it connects to for Internet access.

I have spent too much time connecting to Wi-Fi networks all over the world and running performance tests on them (my wife is regularly upset by me searching for public Wi-Fi to use and test when we are out).

Tonight

Of couse, tonight I didn’t see any issue while I was reading on my iPad (normally I will encounter a period of time when media in particular doesn’t load), and my timed download script didn’t see anything either tonight.

Android 7.1.1 and no LTE on Nexus 5X Phones

I have been carrying a Nexus 5X for a while now, and for the most part I love it. Having a pure Android experience, rather than one with a UI skin forced over the top of it, makes the Android experience very close to the iOS one for the most part in my opinion.

Partly because it is not the only phone I carry, I also decided a few weeks ago to sign it up for the Android beta program. I was already running Nougat (Android 7.0), but I wanted to see what else they were planning in the Wi-Fi space particularly. Everything seemed to be going well until the beta 7.1.1 drop landed on the phone. At that point, I started having problems with Wi-Fi stability and, worse, lost LTE completely.

After a few days of that, and at least one attempt to find where I could send feedback that might be noticed (Google really needs to learn from Apple here and include a dedicated beta program feedback app in the builds), I gave up and worked out how to get the phone out of the beta program. This is one place where Google beats Apple; as soon as I did that on their site, the phone indicated that it had an “update” to revert me to Android 7.0. It was a full install (so all data wiped), but that’s OK.

Missing Backups

At the end of that I opted to setup using a backup. There was the first problem. The latest backup it showed me was from my old Nexus 5. Nothing from the 5X despite it being setup for backups all the time I’ve been running it. No big deal, the Nexus 5 backup will be 90% or more, and the photos (which are all I really need) are all in Google Photos anyway.

The process was as smooth as usual, and in no time I was back to running Android 7.0 and my LTE signal was connected again. Then, it offered me an update to the public Android 7.1.1 release. This had a different build ID to the one I had been running, so I took a chance (didn’t really have much to lose).

Public Android 7.1.1

This is an OTA upgrade, so not a full wipe & it doesn’t take long. When it was done, the phone restarted and my LTE was gone again. Seriously. In a public release of their new flagship OS, LTE is not working on the Nexus 5X device (connected to AT&T). It works fine with Android 7.0 (and previous releases too). It amazes me that nobody at Google had noticed this before they shipped it!

Comcast: Still Fixated by the Modem

Last week’s email to the CEO’s office generated some communication again. On Friday afternoon, I received a call from a lady in their Executive Customer Relations department letting me know that she had reached out to the engineering team the day before, but still had not heard anything back from them.

Earlier today I got another call from her, relaying their answer: that they had installed a parallel modem in the house and then paid for a replacement modem, and that had exhausted their options. I kid you not. Finally, moments ago I received this in a DM from the Twitter support folks, relaying questions from, I assume, the same engineering team:

Good morning. The engineering team is asking if you see the same results with a wired test during the times shown in the screenshots. They also noted that there are no events shown in the node history corresponding with the dates and times pointed out. They also wanted to know if you changed the location of your modem at all recently. Thank you. -FL

Despite everything they’ve been sent, and the fact that the entire recent discussion was initiated by my frustration that they cannot see anything past the cable modem, they are still fixated on it being a modem problem.

So, given this fixation with the modem, here are my questions back to them:

Dear Comcast Engineering,

There is more to your network than the cable modems & headends. In fact, in the grand scheme of things, that link is pretty short. But given your fixation on the modem, and the wiring in my house, despite several of your techs agreeing that it all seems to be just fine, here are my questions to you:

1. Please explain how my modem here in one house in Alameda can be causing packet loss at the 5th hop of the trace, and only that hop (which has been a reliable symptom of this since I first reported it, despite all the modem swaps). Note: that router is located ~45 miles away in Sunnyvale assuming your hostnames are accurate (the node is 68.86.90.93, which, according to DNS corresponds to be-33651-cr01.sunnyvale.ca.ibone.comcast.net); it is neither in my house, nor at the end of the wire my modem is connected to.

2. Additionally, please explain how my modem can also be impacting all the other people in Alameda and Oakland who have so far commented on my Nextdoor and Facebook threads sharing the blog post last week confirming they too see the issue with the internet becoming unusable for periods of time during the late evening.

3. Finally, please include in your explanation how this happens only in the late evening, and at other times the same modem, same wiring and same devices are able to get 50 Mbps download speeds reliably.

Thanks in advance,
John

Sad Day For Pebble

Black Pebble Time SteelYesterday the rumour from the weekend about Fitbit acquiring Pebble were confirmed. Sadly though, the details made it clear that Fitbit didn’t really acquire Pebble at all. They acquired Pebble’s software platform and are offering some of the engineers jobs at Fitbit. The hardware will no longer exist.

That is very sad as the closest thing Fitbit has to a smartwatch is the Blaze, and to be frank it is perhaps the ugliest smartwatch on the market. As the product page makes clear, it is really a wrist-worn fitness monitor first, second and third (though, somewhat ironically for a fitness device, it isn’t water proof beyond “sweat, rain and splash proof”). Then somewhere, much lower in the priority list they added a watch and some basic smartwatch features (answer/reject phone calls and see notifications). No mention of custom watch faces. No mention of third party apps – my Pebble is configured with one button access to my Nest so I can monitor and adjust the temperature in the house from my wrist. And there any many, many more apps for the Pebble platform that allow people to do what they want to with their watch.

I think Pebble have said it before, but they set out to design a watch first. Something that would look good, and work well as a watch. My current Pebble Time Steel looks like a watch. I’m hoping that as well as the platform, some of the design from the Pebble platform will start to appear in future Fitbit, but it is sad to realize that the features supported by my watch will start to degrade over time (either because of changes to iOS, or, as BoingBoing suggests, because cloud support features get turned off by Fitbit).

Fitbit have missed an opportunity IMHO in not taking the hardware designs forward too. Most of the things that I think put Pebble ahead of everybody, including Apple, were hardware features (water resistance, long battery life, always on screen, simple button based UI that actually works). Sure, there were some software features too (the web based developer environment was fantastic, the new iOS app looks great too, and, of course, the choices of third party watchfaces were amazing).

Over the years they had their bad moments too. Support was always a bit of a crapshoot. Sometimes it was great, other times not so much. At the end of the day though, things normally improved (at least until the next iOS update broke them again – which is not Pebble’s fault at all).

Despite its shortcomings, it might actually be time to switch to an Apple watch. After several years of Pebble use (from the original Kickstarter edition, to a Pebble Steel, to a Pebble Time Steel, and I backed the PT2 as well), going without a smartwatch isn’t an option and not having it be waterproof and support basic apps is a non-starter too. So, sorry Fitbit, I’ll be sticking with my One for step tracking, but I don’t need a wrist-worn fitness tracker; I need a smartwatch.

Comcast: Failure to Manage IP Network

Over the last few days I have been posting about my recent interactions with Comcast concerning an issue we see every night around 10pm here with poor network performance, which, based on trace route data seems to be caused by one node in their IP network, in Sunnyvale based on the hostname, becoming overloaded. Our neighbors here also see the issue (one I spoke to has decided he cannot do video calls with Asia at 10pm because it repeatedly drops the call). This has been going on for over a year now though, and still they are continuing to focus on the modem. So, I thought I would create a full timeline of this issue:

October 30, 2015

Initial report of the problem on Twitter, complete with trace route data showing the problem. At this point, the router wasn’t dropping packets, it was just extremely slow:

1-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 12.915 ms 11.091 ms 2893.567 ms

November 7-9, 2015

More traces sent via Twitter DM, all showing packets being dropped completely at the 5th hop (which is the Sunnyvale one):

0 192.168.1.1 27.53ms 1.74ms 1.51ms
1 73.162.38.1 162.18ms 169.54ms 162.59ms
2 …0-6-sur03.hayward.ca.sfba.comcast.net (68.86.248.125) 208.8ms 186.68ms 182.83ms
3 …0-ar01.santaclara.ca.sfba.comcast.net (68.87.192.209) 192.47ms
3 …1-ar01.santaclara.ca.sfba.comcast.net (68.87.192.213) 191.39ms 201.32ms
4 – * * *
5 …1-pe02.529bryant.ca.ibone.comcast.net (68.86.86.146) 292.25ms 306.23ms 305.94ms

Following these, the response from the Twitter support team was to send somebody out to check the local connection. Despite the data being very clear that the issue was not local to me.

December 27, 2015

After reporting the problem was still present, I get this response first:

I apologize for the internet speed issues you are experiencing, are you directly connected to our modem or over WIFI connection? -Will

Pointing out that the issue is in Sunnyvale, and so the method I am using to connect to the modem is somewhat irrelevant (especially since it works just fine for most of the day), I get this back:

I have created a internal escalation (ESL02290781) per your concerns, generally these issues fix themselves. -Will

The conversation then moves to email, and for much of January I was collecting and sending data; including a period of time where I ran pings every minute and recorded the times. That resulted in this graph for them:

Comcast Ping Times

I also demonstrated just how unpredictable it was with this series of speed tests, run just one minute apart:

Speedtest 9:44pm

Speedtest 9:45pm

Speedtest 9:46pm

That email thread got nowhere, and my final message on it (February 5, 2016) was not replied to. Comcast just seemed to decide that there was nothing they could (or would) do about the problem and dropped it.

May 29, 2016

After months of seeing the same thing night after night, I finally reported it on Twitter again hoping to find somebody at Comcast who understood IP networking (rather than just assuming all problems are in the cable modem or the local wiring). That was a waste of time, with this being the initial response:

I would like to recommend setting up an appointment for a technician to come out to your home. Please let me know a good time and date that is best for you and I will see what we have available for you. – Chad

When I point out that we’ve been down that path, and that really my modem & wiring cannot affect the performance of the router 45 miles away in Sunnyvale, I get this back instead:

After reviewing your account, I saw your modem has reached the End-Of-Life (EOL) list. Please visit, mydeviceinfo.xfinity.com/device.php?dev… to preview your device. If you would like to purchase another modem, please visit, mydeviceinfo.xfinity.com and select the speed tier you are subscribed to and a list of Comcast approved modems will be provided below to choose from. After you purchase your device you will need to provide us with the Make, Model, and the MAC Address to add your device to the account. – Chad

What is even more amusing about that is that the device I had at the time, an SB6121, is still on the list of approved modems on their site today. It is not end-of-life at all. It is listed as a fully supported modem:

SB6121 Supported

At the time, I was paying for a 25 Mbps tier; the support page for the modem clearly states it will support up to 75 Mbps. Additionally, outside of the times when the Sunnyvale router was overloaded, I was getting 25 Mbps down. Clearly it was not the modem, nor the wiring. But Comcast tech support was unable to accept that.

May 30, 2016 – June 7, 2016

More data sent showing the same problem still existing in the Sunnyvale node, around the same time at night. That resulted in this response on June 3, 2016:

Our team has reviewed this matter, and wanted to provide update with findings and resolved. The short version is that ICMP packets (the type of packets used in ping and traceroute) are, by design, low priority packets. If a router is busy, it will respond slowly or discard those packets instead of responding. A router is optimized for routing packets, passing them to a new destination, and low priority packets designed to stop at it can generally be responded to last or discarded if the CPU is needed for doing it’s actual job, forwarding packets. In many cases, routers are deliberately configured to drop these packets as a defense against certain types of DDoS attacks. The important part of the information from traceroute is how long does a packet take to make it through Comcast’s network, is there a pattern of packet loss or latency to all stops after a certain point. -AD

Finally, an admission that the node must be busy: “If a router is busy, it will respond slowly or discard those packets instead of responding.” But they are using that as an indication that it is behaving correctly. Of course, at one level it is, but the problem is when it gets to the point where it is dropping my ICMP packets, it is also dropping other packets making web content and streaming video unbearably slow, or even failing completely.

On June 4, they just wrote it off as not being a problem:

Our engineering team has review this matter, to ensure it’s no possible routing issues.with you explaining what you experience and neighbors I wanted to make sure we thoroughly reviewed and research this matter. They have reviewed your device and confirm it’s no issues. They have review plant integrity which is 100% also confirm device levels have been good since April. We have exhausted all options with engineering leadership team and the findings have confirm no possible routing issues with your device or area. -AD

And yet it continued to be a problem every night.

June 14-16, 2016

More data sent to them, and once again we are back to the modem as the problem:

I checked the area and modem, signal wise. I’m now seeing some slight noise on your connection, unique of the neighborhood. Would you open to having someone come check it out? – CR

I allowed them to waste their time sending somebody to the house, and that tech arrived on June 16. He found no issues at the house at all, with all the signals looking good. Additionally, when he looked at the data I had been sending he was confused as to why he had even been sent out for a problem that was clearly in the IP network in Sunnyvale and nothing to do with the cable modem or even the network in Alameda/Hayward.

Of course, there was still no change in the behavior with the network slowing down, or failing completely around 10pm almost every night as before.

June 20, 2016

This time I get a response saying that the modem I have, the SB6121, is indeed not end of life at all (that is only for rentals), and is a supported modem:

Hi John, the sb6121 is only marked end of life for rental units. We are continuing to activate retail modems on our network, and it is supported for the performance tier of service. Going back through your traces, they are much improved over previous, but it doesn’t appear to be an issue with Sunnyvale. There had been an area issue around that time that may have been related. How is the connection tonight? – CR

While the claim here is that a different problem was impacting performance, my data continued to show the same thing: the Sunnyvale node was overloading and dropping packets.

June 29, 2016

Again, after sending data from the evening of June 28, I receive the standard modem focused response:

Hi John, I’d like to send a signal to your modem that may interrupt your service for a few minutes. Can I do that now? -VG

Of course, as with all the previous threads, this got nowhere at all.

July 22 – August 2, 2016

Still been an issue, and still nothing is being done to even investigate the real problem. The response I get this time from the support folks is this:

This is something that we can’t repair here or view remotely. Thank you for your address, Now I can schedule a maintenance tech to go out to the affected area to see what’s going on and work towards getting this addressed and resolved. -KJ

So, once again, we’re back to the myopic focus on the modem and the cable connection in spite of all the data pointing at a problem elsewhere in the network. After much back and forth, and more and more data, all pointing at the same place, I get this:

Hi. Our engineer team sees no erroneous activity on the node and wanted me to verify if you were using wifi with these tests. Also, the modem has not been refreshed in over 56 days. Could you please give it a hard reset? If you are using wifi please try testing with ethernet if you are able. Thanks. -FL

Incredibly, they are back to the modem and the connection in my house which somehow can make packets drop in Sunnyvale. But not anywhere else. Where do they recruit these people? Not one of them has been able to explain how my modem, the wiring in my house or whether I am using ethernet or Wi-Fi for the local connection could cause packet loss in a router 5 hops away, while not impacting the ones in between. Oh, and of course, we get the IT standard response: reboot it.

August 10-14, 2016

I point out that other neighbors are seeing the same problem, and even provide some data from one of them showing the same thing happening at their house, again around 10pm. This is not isolated to my house, so I am hoping this will get them to move on from the modem/wiring issue.

On the 14th I get this back:

The node is showing no problems. I can also confirm that it is very far from being overloaded. Once you get a chance check it out and let us know. If the problem persists then we will have someone out to take a look free of charge. Thanks. -FL

OK, probably true at the time you looked, but how about at 10pm? When earlier you admitted it would drop ICMP packets only when it became “busy.” And it shouldn’t be dropping other packets unless it is actually overloaded. And “if the problem persists” is a ridiculous response to a problem that I have been reporting for 9 months already. Of course it is persisting.

August 20, 2016

A response about the meaning of the trace route data leads to a discussion about that. Really, they have actually been trying to suggest that dropping ICMP is something that happens, and is not an indication of other issues. Of course, since, as they noted earlier, it only happens when the node is getting loaded, that is nonsense. Furthermore, since the speed tests, web page loads and streaming video connections were also failing, I am pretty certain the problem is overloading somewhere, and the only node dropping ICMP is the Sunnyvale one. Doesn’t take a genius, but apparently it takes somebody smarter than a Comcast employee.

At the end of that, I get this:

Hello. A suggestion was made by one of the network engineers to possibly upgrade your modem to one with more channels. The SB6121 is 4×4, it is recommended that perhaps an SB6183 would work better for you. They are also interested in the make/model of the router you are using and are curious if the results you’ve provided are with ethernet directly to the modem without the router in the equation. Thanks. -FL

Yes, amazingly, they are back to the modem causing the packet loss in Sunnyvale. I don’t know what it takes to get these people to move on from the modem. Apparently almost a year of data pointing at the IP network in Sunnyvale isn’t sufficient. Either that or they have the most unreliable cable network in the world, and suspect that it will fail there all the time.

End of August through September, 2016

After sending a troubleshooter to the house to discover that the connection and signals there are still perfect, he installs a parallel modem taken from one of their offices in the house. During the next month, I collect data over both showing that they both experience the same thing. I was hoping this would rule out any further suggestion that it was the modem, but no. Instead, they send me a list of approved modems and tell me to buy one and they would reimburse me. I point out that my current one is still on that list, but they insist on a newer model. So, on September 19, 2016 I replace the SB6121 with an SB6183. The problem is unchanged of course since it is not a modem issue.

I send a lot more data, and get no response.

December 5-7, 2016

Monday night it was bad as usual, but my wife wanted to watch something on Netflix and we couldn’t even get the Netflix app on the Roku to launch as it was not able to contact Netflix’s servers. Once again, I posted the trace data from my iPad:

Trace Data

On Tuesday morning I get the typical response, and we’re back to the modem being the issue again:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

That thread ends with this statement:

I would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

Of course, because as I have been saying for over a year now, the problem is nothing to do with my modem, the wiring or the connection to the headend. It is in Sunnyvale. In the IP network. And it only happens at 10pm. Seriously, I don’t believe these people “review” anything when they say they review the account. And they certainly didn’t bother reading the detail in the blog post (it was the tweet about the post on Monday that they responded to first on Twitter).

Then this morning I get this:

Hi, John. I completely understand your frustration 100%. I reviewed the signals in your modem and I’m reading T4 time-outs that would be best resolved by a tech visit. I would highly recommend this as they can file any requests for maintenance in the area. Is there a date and time that would work best with your availability? I really want to get this resolved for you. -CE

That despite the fact that the speed test results just after he sent that showed a very acceptable throughput. And since yesterday when it was all OK.

Speedtest on 12/7

And then they claim they cannot resolve the issue because they need to send a tech to look at my modem. This guy needs a new job:

I understand completely. I’m a corporate staff member for Comcast and again, I really want to help. As my colleagues have previously mentioned, to best assist you, we need to have a tech come out and take a look at your equipment in-person and diagnose what is causing your service issues. Otherwise, we are unable to proceed with reaching a resolution. Feel free to reach out if you change your mind. Our corporate Digital Care team is available 24/7 on social media. -CE

Hopefully after this lands on his boss’ desk he will rethink his position and perhaps try to get somebody to review the load history on the Sunnyvale node. Perhaps looking at what happens between 10pm and midnight rather than checking in the middle of the day and writing it off as ‘all OK’ quickly.

Comcast Responds: Reboot the Modem

I kid you not. First they respond on Twitter with the standard “please DM us” – ostensibly to get my account number but this is really about getting the conversation out of the public channel. 

Then, when I respond via DM that they have all the information and I just want to speak to somebody at this point who can see beyond the modem and the wiring at my house, I get this back:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

Never mind the details I provided last night, and all the information over the past year. 

Never mind the information about having the modem replaced and even having Comcast install their own alongside mine to check in parallel. 

Never mind the fact that I said this happens almost every night. So the 47 days is irrelevant. It happens almost every night. 

All their support people are capable of is sending canned responses now. Clearly, his review of the account must have been very cursory or he might have seen the information they’ve been sent and that several people have checked the wiring at the house. And the modems. And the router. 

Totally useless. 

Update 1

I pointed out that it wasn’t the modem. And that we have totally exhausted that route. Here’s what I get back:

I would be happy to help get this corrected. Since the modem has been online for 47 days I would like to start by pushing a provisioning signal to your modem. This will make sure the speeds and firmware are correct on the device. This will take the modem offline for 3-5 minutes. -CN

I am amazed they can get anything done with responses like this.

Update 2

Still insisting that the dropped packets in Sunnyvale, ~45 miles from Alameda, are caused by the modem it seems, as I got this back:

In order to properly troubleshoot we have to start at the modem and then work down the line. In order to start to help you I will need to reset the modem since it has not been reset for 47 days. -CN

Update 3

Finally, moved away from the modem, but apparently still not understanding that the signal quality in Alameda at 1:30pm is totally irrelevant to an IP problem in Sunnyvale that happens around 10pm:

would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

And throughout this, my requests to either escalate to somebody who understands the trace route or to get a VP to call me have been ignored. Totally ignored.

Comcast Can’t Handle The Traffic

For more than a year I’ve been reporting that around 10pm every night our Comcast internet becomes unusable. Streaming video fails, web pages start timing out. Media content in Facebook and Twitter stop loading. Every time I have looked, the problem is the same: the Sunnyvale node is dropping packets. Like this:


That 5th node is Sunnyvale. In this run it wasn’t even able to get the reverse DNS for that node.

Over the course of the year, I have sent them pings, traces & speed tests over my network. From neighbors who all report the same thing, and even over a modem they installed in parallel with mine here (when they were claiming it was my modem in Alameda causing the packet loss in Sunnyvale!). It is always the same. Around 10pm the network becomes intermittently unusable, and it is always the Sunnyvale node that is dropping packets. It will stay bad until after midnight most nights. 

I’ve been told that the router dropping ICMP is normal (not true: it is true that it will drop ICMP packets first when overloaded though, so the high loss rate indicates an overloaded router; the failing video & web downloads tell me it is dropping other packets too).

I’ve been told it must be my router, my modem or the cable into the house. Nobody has been able to explain how any of those things could result in packet loss at Sunnyvale though. 

When the network is working, Speedtest results look like this:

Just after 10pm that periodically changes to this:

Every time I report it we go through the same sequence. First there is the request to send a tech to my house to check the wiring (total waste of everybody’s time since it is obvious the problem is in Sunnyvale, about 45 miles from here), then the request to send the logs (I’ve been sending them for months). Then radio silence. Until I ask again, when they repeat the process. 

At no point do they ever get the networking team to admit their network is becoming overloaded. A year after I first reported it, their network is apparently still unable to cope with the traffic passing over it between 10pm & midnight. Not continuously though. It is intermittent. Randomly, there will be a few minutes where nothing works, then it will be back. Streaming video fails, web loads timeout. My neighbors tell me they get dropped from conference calls too, making it hard to do late night calls with overseas teams. 

I am starting to wonder whether Comcast actually knows how to run an IP network. It certainly seems that every time I contact them they try to find a fault in the cable TV related parts of the system. Never in the IP network. I can’t believe they don’t have load charts for that node. Or statistics on packet drop rates. I just suspect they aren’t even looking. 

Time Dock Pebble 2

Around the time I signed up on Kickstarter for the Pebble Time 2, I also backed a project to create the second generation of a charging dock for Pebble Time watches. The gadget to the right arrived recently in the mail.

Up until now, when I charge my watch (roughly once a week), I leave it lying on the desk with the factory charging cable attached underneath. This turns my watch temporarily into a desk clock, so I can still see notifications etc while it charges. 

The Product

Firstly, what is it exactly? Well, it is an anodized aluminium (that’s aluminum for US readers) stand with a magnetic charging connector that allows the watch to be mounted on it, held in place by the two magnets, with the strap still attached whether it is a loop, or a two piece band. Not sure how well it would work with a single band NATO strap which goes under the watch, covering the charging port, but with my two piece Clockwork Synergy band it works well.

Behind the stand, a standard micro-USB connector provides the connection to power, and there is a notch in the base that grips the provided USB cable tightly, keeping it neat & tidy.

In Use

In use, the watch snaps easily onto the magnets & remains visible as a small desk clock thanks to a slight angle back:


Charging functions the same way as with the factory cable, but now it is held safely in place while it charges.

It is designed to be compatible with the Pebble Time 2 as well, so when that arrives I should be able to keep using it to charge the new watch as well.

When Cloud Based IoT Dies

Update November 7, 2016:

It looks like the Lono cloud service is back online. This was not a normal outage however as their domain completely disappeared from DNS. The bigger question of what they will be doing to ensure that the device works locally, even when the cloud is unavailable still deserves an answer (and I filed a support ticket this evening asking both about the outage & about plans for graceful degradation of service should the cloud component fail again).


A while ago now I backed a project on Kickstarter that was creating a more modern sprinkler controller. That actually wasn’t hard to imagine since the user interface of the one our home’s builder attached to wall consisted of a rotating switch, some buttons and an LCD display which could handle numbers & a few other preset things. Like something from the 1980s.

That project was Lono, and, like most Kickstarters, it delivered late & somewhat incomplete. But the hardware looked good, was dead simple to install & seemed to work. The software less so. Over time, things improved a bit though.  I could access the controller, via the iPhone app, from anywhere. Scheduling was added, as was weather and a few other features. I don’t think I saw the truly smart scheduling that was promised, but it was delivering what I needed. Until today.

Cloud Dependency

Today, the Lono died. Well. More specifically, the cloud service behind the Lono died. Now the attractive black & green box on the wall of my garage is essentially useless. Obviously, that is frustrating because I can no longer control my sprinklers, even from home when my phone & the Lono are on the same network. But it frustrates me on another level too. These IoT devices are clearly more powerful when connected to the cloud, but they should not be designed to be dependent on that cloud to do anything. 

There is absolutely no reason why the Lono, discovering it could no longer reach its cloud based control center, couldn’t have dropped back to a LAN only mode. Whether the outage is caused by the company failing (which seems to be the case here), or other things (maybe an ISP failing, or being temporarily offline), there really is no excuse for these things to stop working based on their last known settings & reverting to more local control.

Kickstarter 

I’ve backed a number of different things on Kickstarter & Indiegogo. Typically, while they may have received firmware updates etc, only a couple were really dependent on a cloud based service & only Lono has failed. It does make me think I will be more wary of cloud backed IoT projects in future. Perhaps such projects will need to explain their plans for this scenario. At the very least, it would be good to see they’ve considered this & have some level of “disconnected” functionality baked in.

If they want to truly impress me, they should have hardware design, firmware & app software in an escrow service, with public (or at least customer) release triggered on company failure. Then, maybe, the community could rally around and perhaps continue support for these devices.