Smartwatch Round Up

The recent acquisition of Pebble by Fitbit, or rather the recent acquisition of some of Pebble’s technology & employees by Fitbit, has left me with a bit of a dilemma. There is a blog post stating that “Fitbit is going out of its way to keep Pebble software and services running through 2017.” Of course, that is only 2017 (and who knows what that really means). But they didn’t buy the hardware, so there will not be new Pebble watches. Of course, it seems likely that there will be a future Fitbit watch with some or all of the Pebble software in it (why else would they buy it), but their existing watches are not very attractive to me.

Then, today, as I was walking to the ferry terminal, this popped up on my phone:

I hadn’t changed watches as far as I knew; in fact, I hadn’t even touched the watch. I happened to look at the phone because I was trying to send a text message. Of course, since the watch had mysteriously reset, I did not get the notification on my wrist. Indeed, my watch had reset completely. Also, when I launched the app on my phone, it wanted me to login again. Logging in, got me to what looked like an empty account. It also indicated that my health database was corrupted (all that data is lost apparently).

Slowly, bits of the old Pebble environment came back. Some of the watchfaces appeared in the locker, then apps. But I couldn’t make any of the watchfaces active. Then, a little later I could again.

If I had to guess (and so far I have not been able to get an answer from the @FitbitSupport folks on Twitter confirming or denying this), I would say that somebody migrated a backend and in doing so broke something. Or maybe the app just crashed and corrupted something (not like it doesn’t crash a lot recently). For now it is back (minus the health data), but I am wondering how long for, and also whether anybody will fix the iOS app, or even update it to remain compatible with future iOS updates.

Alternatives

That led me to looking for alternatives. There was Vector, until Fitbit acquired them too (and said there won’t be more Vector watches). So, what else have I found?

Withings SteelHR

The Withings SteelHR is an interesting device in that it really is a watch, but I think I would be giving up too much to go back to something that basically had an indicator for notifications on it rather than showing me some of the text.

I’m also kind of used to having options on the watch face (something that a real watch can’t do), and the option to combine several pieces of information on the one display (currently, I have steps, sleep patterns, analog time & date as well as status indicators for battery life and bluetooth connectivity).

CoWatch

The CoWatch reminds me more of an Android or Samsung smartwatch. It has one feature that really stood out though: Amazon’s Alexa integrated inside.

This one checks all the boxes in terms of flexibility, and I do quite like the idea of a round screen. Where is fails is in the reviews that suggest it is not quite there yet. When I bought the original Pebble on Kickstarter, I was also buying into an experiment to see how I liked the idea of a smartwatch. My current Pebble is the third one I’ve owned, and I am totally hooked on the concept. Going back to a beta level watch, with all the connectivity problems and missing features doesn’t appeal at this point.

Also, much like the Android and Apple offerings, this one basically requires charging once a day. That is something that puts me off having grown accustomed to a once a week cycle with the Pebble.

Martian mVoice

The Martian Watches mVoice is another smartwatch with Amazon’s Alexa integrated (and by all accounts the integration is better than the current CoWatch one). But, like the SteelHR, this is an analog watch with a small screen. While it does look like the screen could handle more text than the SteelHR one, it is still much more limited than getting the text message full screen on my Pebble where I can read it.

It is also only splash resistant which makes it a non-starter for me (in the same way that the original Apple watch is not interesting because I can’t swim in it).

Apple Watch Series 2

That takes us to the Apple Watch Series 2. These are water proof, and they certainly deliver on the smartwatch features (although the lack of 3rd party watch faces is still a little odd given they’re up to version 3 of the OS, and there are already native watch apps).

This one also checks all the boxes for fitness tracking, and I don’t think there’s much risk of them being acquired by Fitbit. But, the price! In black stainless steel (which is what my current Pebble is made of) one of these would set me back $600. With a silicone band. Add another $25 for a third party nylon band. (In reality, I’d probably opt for the black aluminium version at $400 with the nylon band, but still, that’s over double what I paid for the Pebble – in fact, I didn’t pay much more than that for all three Pebbles combined!).

The other big negative on this one for me is the short battery life. Charging daily would require thinking about when to charge it; overnight makes sense, unless you want to use the watch for sleep tracking too, which I have grown to like on the Pebble. In the office might work, but what about weekends when I am not sitting at my desk?

Matrix Powerwatch

The Matrix Powerwatch is an outlier in that it doesn’t exist yet. It is still in the crowdfunding stage over on Indiegogo. Its bold claim is that it doesn’t need charging at all – instead it is powered using thermodynamics from your body heat. Take it off and it sleeps (retaining time etc of course); put it back on and it wakes up.

Given its early stage of development, I am not even considering it. It also isn’t clear what the feature set will be in terms of smartwatch capabilities (the photos all seem to be fitness oriented). But never having to charge it sounds great to me! I had a solar powered watch many years ago and loved that aspect of it. I wonder how good this thermodynamics concept is, but if it works, I think it is a great idea.

Conclusions?

I love the increasing number of options in the space, and I still believe that once you have tried a smartwatch you will not go back (even if it is just for the ability to keep your phone in silent mode and filter the interruptions to just the things you really care about). The Apple option is clearly a strong player, but for a little longer at least I am going to stick with my Pebble I think. If it does get to the point that it really isn’t working, right now I think the only viable option for what I’m after is the Apple watch.

Comcast: Not Resolved

It is holiday season so I wasn’t expecting much to happen with the Comcast issue (and to be honest, I haven’t been online at 10pm as much either to notice it), but on December 30th I did get a call from Mark N. in Executive Customer Relations updating me on the fact that the engineering team had not been able to find anything, and telling me they were closing the ticket. He also stated if I saw the issue again that I should let him know and he would re-open it.

At the time, I commented that while I had not been online at the affected hour (10pm Pacific for those new to this thread) to be able to update him on my experience, I felt it was unlikely to have changed given that it has been ongoing for over a year now.

Then, on January 1st, I happened to be online and notice the typical stalls in video streaming and poor performance in Twitter and Facebook with images & videos not loading. I ran some speed tests and traces too, and got these:

Speedtest results

Notice the chart for the download starts high, and then drops fast to a very low level. That is a bit different to previous results (although this was 10:35pm too, and things were starting to improve, especially the video stream). The average is also much higher than it is at the peak of the problem (where I typically see under 1 Mbps download). But, at other times of the day I see 50Mbps or more for the download speeds, and a nice flat intra-test chart.

Traceroute

The trace shows the packet losses at Sunnyvale though, same as always. I suspect that this is the cause of the fast drop in the speed test too as the TCP window size adapts down to cope with the higher than ideal packet loss.

The burst also explains why my command line speed test script didn’t see anything even when the GUI version was showing degradation. I suspect the command line tool is using only a small file, whereas the GUI version I know adapts how much it downloads to make sure it gets a good average.

Another Report

Tonight, I happened to be on Twitter and noticed this tweet about another Comcast customer in the SF Bay Area having issues, and pointing at the Sunnyvale router:

When I reached out to @Pixel, he got back to me with the comment that he sees issues almost every night too around 10pm ± 1 hour. And he is in Santa Clara. So, now I have reports of the same performance problems from Alameda, Oakland, San Francisco and Santa Clara. But still I get this from Comcast support:

I seriously think that Comcast needs to invest in some training for their support people. When Frank Eliason was running the Twitter support team it seemed to be staffed with people who were on the top of their game. Sadly, that doesn’t seem to be the case any longer.

The same support person however did let me know that the ticket (ESL02794458) had been closed with the comment that it was resolved. Let me be crystal clear now:

I have never stated that the issue was resolved.

The request to close the ticket came from Comcast because they were unable to find anything. As I mentioned above, my response to that was that I felt it was unlikely to have changed since it had been the same for over a year, but I had not been home at the right time to be able to confirm or deny it was resolved. That is most definitely not the same as me reporting the issue as resolved. Furthermore, for the record:

The issue is not resolved.

And, as should be apparent to the people at Comcast by now, I am not the only customer affected by this. I strongly suggest they get their act together and find out what is causing these performance issues. And I would also suggest that perhaps rather than sitting in the Hayward head end trying to find ways to pin the blame on my modem/router/Wi-Fi, they take a trip to the Sunnyvale router’s location and have a look at what is happening there. It seems too much of a coincidence that that one node always shows high packet loss rates when there are performance issues, and not when everything is working OK.

Not Just Traceroute & Speedtest

One more thing I’ve heard is that these tools might not show problems when the rest of the network performance is OK. Let me be very clear on this for the folks at Comcast too:

I only run Speedtest and traceroute when I see other things failing.

More specifically, the applications I have seen failing are:

  • Video streaming from Amazon and Netflix (stalling, dropping down to the lowest quality stream, and even failing to load the catalog on our Roku box sometimes)
  • Media loading in Facebook & Twitter (text loads, but no images, including avatars/profile pics in many cases)
  • Web pages timing out on loading (especially more complex sites like the NY Times, the Washington Post or even the Amazon home page)
  • Video calling (Google Hangouts, FaceTime etc) disconnecting repeatedly

Only when I see something like that failing do I think to run speed tests or traceroute on my home network.

Automation is not Laziness

It occurred to me recently that when I have been doing things around the house that automate something, those activities have often been branded as me being lazy.

The Roomba

The Roomba was the first example of this, though at the time I just ignored the labeling. Having a device that can trundle around cleaning the floors for me without my needing to be pushing it is great. The argument was made that the Roomba was not as effective, nor as fast, as the Dyson in the closet. But that argument is false. On paper, the Roomba clearly does not have the suction power of the Dyson, but it can do the job unattended, meaning the floors are vacuumed more often than they would be with the Dyson. As for the speed, while it does take much longer for the Roomba to complete the job, it doesn’t expend any of my time at all. That makes it infinitely faster from my perspective.

Is it lazy though to push vacuuming onto a robot? I don’t think so. Especially not if I am spending the time I would have spent pushing the Dyson around doing something more valuable. It also impacts the overall quality of life for the family; one less chore that needs to be delegated.

The Dishwasher

Another one the household appliances that was seemingly tagged as making people lazy was the humble dishwasher (though, oddly, I have not heard the same complaint about the washing machine in our laundry room). For me, the dishwasher not only saves my time, it also typically does a better job at getting things clean, and saves on water. Once again, it can run when I am doing other things around the house, or it gives me some time to spend with the kids rather than standing at the kitchen sink washing dishes.

Alexa

Just before Thanksgiving, I finally bit the bullet and upgraded the Insteon hub here in the house to the newer version that can integrate with Amazon’s fantastic Alexa gadget (something we’ve had in the house for a long time now and been using for music, weather, news, shopping lists and more). In addition to the Insteon lights, I hooked up the Nest and August integrations, so Alexa could control the house lights, the thermostats and our front door lock (she will only check the status or lock the door though, not unlock it).

My kids were hooked on the light control immediately, and our five year old has been showing everybody who visits Alexa’s newest trick. But is it laziness to ask Alexa to turn a light on or off rather than getting up and walking to the switch? Or to have her adjust the temperature on the thermostat instead on getting up and adjusting it manually? Of course, in both cases I could also just pull out the phone and use that, and in the case of the Nest, I can even make the change from my watch. I imagine those are also considered “lazy” options.

Is voice control ever useful? Sure! in the month or so we’ve had it, I’ve already used it a few times when I’ve had my hands full and needed a light turned on or off. I don’t think that is laziness; I see it more as improving efficiency.

Oddly, even though controlling the lights or thermostats remotely is laziness, I noticed that controlling the TV without getting up was not considered to be lazy. I wonder if the TV remote control was branded as laziness when it was first introduced?

Smarthome

Right now, I have the basic elements of a smart home, but I don’t think the home is really that smart. Sure, the August can tell the Nest when I go out. The Nest smoke detectors can tell the Nest thermostats when there is a carbon monoxide issue or a fire (and have the heating shut off). The thermostats also tell the smoke detectors when we are out so they can run self-tests without disturbing us.

But I am looking for more. I’d like the August to turn on the lights inside when I unlock the door at night. Or if the Roomba could disable the motion sensors on our home alarm system while it was cleaning, and re-enable them once it was done (so I could schedule the vacuuming to run while we are out without it resulting in the police breaking the door down to check for intruders).

Luckily, for this there are more and more options coming online, connecting the various APIs for each service. All I need now is some time to set them all up. And to keep replacing components in the home with ones that have APIs. I’d love to have our home Wi-Fi router be able to report when certain devices come and go, or trigger other activities based on who is at home, or who is arriving home. Our Apple Extreme cannot do this, but perhaps the next router I buy will have that capability (or something else I can connect to the house will).

Presence (for the future)

The next step I suspect will be to add beacon sensors in certain rooms so the house “knows” which room I am in based on either my phone, or my watch or some other wearable device that I choose to register as synonymous with me being present. Would it be lazy to have the kitchen lights automatically come on, and perhaps have a coffee start brewing when I come down at 6:30am to make lunch for our pre-schooler? Maybe Alexa would even know I walked in and greet me, or start my flash news briefing or a review of my calendar automatically too.

None of this is actually that new. I attended a lecture talking about “smart” badges probably 20+ years ago in the UK. But the age of Bluetooth wearables and smartphones makes a special badge unnecessary. And services like IFTTT make all of the interconnects simple too. We live in exciting times, as long as you think beyond automation being lazy of course!

Comcast: Now It’s The Wi-Fi

Last Thursday I spent almost 2 hours on the phone with a couple of engineers from Comcast. The one who lead the call was sitting in the Hayward head end location, connected to the same network I was using (I’m not sure where the other one was physically located – he was dialed into the call). Around 10pm, we both started to see “anomolies” in the Speedtest results. As well seeing packet loss in the traceroute results from the Sunnyvale router. We also saw longer than typical ping times to Google (normally low tens of milliseconds from my home).

He tried a number of things, but nothing made a difference. Until, at 10:20pm, it rapidly went back to being normal. The pings improved. The Speedtest results improved. The traceroutes started showing responses from Sunnyvale again (and by 10:30pm they were also consistently good times). Unfortunately, it didn’t go bad again that night while we were on the phone, but I was hopeful that having had two of their engineers see what I was seeing there might be a light at the end of the tunnel. I should have known better.

Wi-Fi

During that call there was a question about how I was connected to the network, and we even switched my computer between the 2.4GHz and 5GHz bands on my Wi-Fi to verify that made no difference. It did not. Where I live, both bands are pretty clear of interference. Normally I have the two bands using the same SSID, but during the call I switched the configuration so they were separated (allowing me to easily switch between them).

I did also point out at the time that it was somewhat irrelevant since I had seen and documented the problem in many configurations, from many devices, including some of my neighbors homes and even some friends in other bay area locations.

To be clear here, I have seen the problem:

  • On Wi-Fi connected computers (both bands)
  • On Wi-Fi connected mobile devices (both bands)
  • On ethernet connected computers through my router
  • On ethernet connected computers direct to a Comcast-provided modem
  • On both our wired and wireless Roku boxes

Additionally, at other times in the day there is no problem at all from the same computer I was using during the call. Here is a speed test from that same computer taken around 6:45pm this evening, while we were streaming HD video from Netflix on the Roku and a visitor was watching a basketball game on his Mac. As I’ve made clear throughout this, outside of the 10pm to midnight (some days a little later) window, the performance is very acceptable.

That was run from the browser on my Mac Mini connecting to the internet over the same Wi-Fi network. The only difference was I ran it at 6:45pm rather than 10pm.

Home Visit

So, today’s call from the engineering team did not go well. Firstly, even though I saw the slow down over the weekend and yesterday (I was not online at the right times on Sunday or Monday), apparently the equipment in the Hayward location saw none of it.

Furthermore, he now believes that the “anomalies” seen in the Speedtest results during our call last week were related to Adobe Flash and not the network. While I am not going to defend Flash at all, I will say that Speedtest is rock solid, and the versions I am running are not Flash based, so that doesn’t apply to my results.

Finally, it was suggested that perhaps the issue is in my Wi-Fi network and they should come to my house to check the setup in the house. Really. I am not kidding. Once again they are concluding that their network is fine and any issue must be in my house.

When I pointed out that several of their engineers have already been here, and perhaps he should just pick the phone up and call one of them, I was told that they would need to come out to see it themselves. I guess they don’t trust any of their colleagues to check the modem and signal quality.

Once again, Comcast folks, if you believe that something in my house is the cause of this, please explain to me how so many of my neighbours see the same issue. Is my Apple Airport Extreme somehow managing to interfere with them all?

Also, I’m willing to accept that the packet losses in Sunnyvale are coincidence, if you can show me what else in your network could cause so many people in this area to see this issue. And why they correspond so well in terms of time to when the network quality is poor. Right now, when the network is working well, I see this from my traces:

5 be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 11.224 ms 16.193 ms 13.003 ms

The only time I see slower ping times or no responses from that node is when all the other things I try are bad too. What are those other things? Well, here’s a list of the things we often see when it is behaving poorly:

  • Web pages, Facebook content, Twitter content and emails not loading;
  • Netflix and Amazon Prime video not being able to load their index screens;
  • If we have the index loaded, Netflix and Amazon not being able to start playing videos;
  • Videos that are playing keep pausing to load, which can take many minutes to recover from, and in many cases times out;
  • Speedtest shows results under 1Mbps for download (compare to the speed above when it is working)

Experience

What I haven’t mentioned to Comcast until today is that I have a little bit more knowledge about Wi-Fi than perhaps their regular customer. Having been working in a Wi-Fi software company since 2003, and even been part of the creation of some of the Wi-Fi standards, I think I am pretty well placed to know whether my Wi-Fi setup here is likely to be the cause.

I have spent perhaps too much time inside shielded rooms helping access point manufacturers fine tune their software to squeeze the maxiumum performance from their Wi-Fi stacks. I am well aware that the 802.11ac system I have here far exceeds the capabilities of the Comcast network it connects to for Internet access.

I have spent too much time connecting to Wi-Fi networks all over the world and running performance tests on them (my wife is regularly upset by me searching for public Wi-Fi to use and test when we are out).

Tonight

Of couse, tonight I didn’t see any issue while I was reading on my iPad (normally I will encounter a period of time when media in particular doesn’t load), and my timed download script didn’t see anything either tonight.

Android 7.1.1 and no LTE on Nexus 5X Phones

I have been carrying a Nexus 5X for a while now, and for the most part I love it. Having a pure Android experience, rather than one with a UI skin forced over the top of it, makes the Android experience very close to the iOS one for the most part in my opinion.

Partly because it is not the only phone I carry, I also decided a few weeks ago to sign it up for the Android beta program. I was already running Nougat (Android 7.0), but I wanted to see what else they were planning in the Wi-Fi space particularly. Everything seemed to be going well until the beta 7.1.1 drop landed on the phone. At that point, I started having problems with Wi-Fi stability and, worse, lost LTE completely.

After a few days of that, and at least one attempt to find where I could send feedback that might be noticed (Google really needs to learn from Apple here and include a dedicated beta program feedback app in the builds), I gave up and worked out how to get the phone out of the beta program. This is one place where Google beats Apple; as soon as I did that on their site, the phone indicated that it had an “update” to revert me to Android 7.0. It was a full install (so all data wiped), but that’s OK.

Missing Backups

At the end of that I opted to setup using a backup. There was the first problem. The latest backup it showed me was from my old Nexus 5. Nothing from the 5X despite it being setup for backups all the time I’ve been running it. No big deal, the Nexus 5 backup will be 90% or more, and the photos (which are all I really need) are all in Google Photos anyway.

The process was as smooth as usual, and in no time I was back to running Android 7.0 and my LTE signal was connected again. Then, it offered me an update to the public Android 7.1.1 release. This had a different build ID to the one I had been running, so I took a chance (didn’t really have much to lose).

Public Android 7.1.1

This is an OTA upgrade, so not a full wipe & it doesn’t take long. When it was done, the phone restarted and my LTE was gone again. Seriously. In a public release of their new flagship OS, LTE is not working on the Nexus 5X device (connected to AT&T). It works fine with Android 7.0 (and previous releases too). It amazes me that nobody at Google had noticed this before they shipped it!

Comcast: Still Fixated by the Modem

Last week’s email to the CEO’s office generated some communication again. On Friday afternoon, I received a call from a lady in their Executive Customer Relations department letting me know that she had reached out to the engineering team the day before, but still had not heard anything back from them.

Earlier today I got another call from her, relaying their answer: that they had installed a parallel modem in the house and then paid for a replacement modem, and that had exhausted their options. I kid you not. Finally, moments ago I received this in a DM from the Twitter support folks, relaying questions from, I assume, the same engineering team:

Good morning. The engineering team is asking if you see the same results with a wired test during the times shown in the screenshots. They also noted that there are no events shown in the node history corresponding with the dates and times pointed out. They also wanted to know if you changed the location of your modem at all recently. Thank you. -FL

Despite everything they’ve been sent, and the fact that the entire recent discussion was initiated by my frustration that they cannot see anything past the cable modem, they are still fixated on it being a modem problem.

So, given this fixation with the modem, here are my questions back to them:

Dear Comcast Engineering,

There is more to your network than the cable modems & headends. In fact, in the grand scheme of things, that link is pretty short. But given your fixation on the modem, and the wiring in my house, despite several of your techs agreeing that it all seems to be just fine, here are my questions to you:

1. Please explain how my modem here in one house in Alameda can be causing packet loss at the 5th hop of the trace, and only that hop (which has been a reliable symptom of this since I first reported it, despite all the modem swaps). Note: that router is located ~45 miles away in Sunnyvale assuming your hostnames are accurate (the node is 68.86.90.93, which, according to DNS corresponds to be-33651-cr01.sunnyvale.ca.ibone.comcast.net); it is neither in my house, nor at the end of the wire my modem is connected to.

2. Additionally, please explain how my modem can also be impacting all the other people in Alameda and Oakland who have so far commented on my Nextdoor and Facebook threads sharing the blog post last week confirming they too see the issue with the internet becoming unusable for periods of time during the late evening.

3. Finally, please include in your explanation how this happens only in the late evening, and at other times the same modem, same wiring and same devices are able to get 50 Mbps download speeds reliably.

Thanks in advance,
John

Sad Day For Pebble

Black Pebble Time SteelYesterday the rumour from the weekend about Fitbit acquiring Pebble were confirmed. Sadly though, the details made it clear that Fitbit didn’t really acquire Pebble at all. They acquired Pebble’s software platform and are offering some of the engineers jobs at Fitbit. The hardware will no longer exist.

That is very sad as the closest thing Fitbit has to a smartwatch is the Blaze, and to be frank it is perhaps the ugliest smartwatch on the market. As the product page makes clear, it is really a wrist-worn fitness monitor first, second and third (though, somewhat ironically for a fitness device, it isn’t water proof beyond “sweat, rain and splash proof”). Then somewhere, much lower in the priority list they added a watch and some basic smartwatch features (answer/reject phone calls and see notifications). No mention of custom watch faces. No mention of third party apps – my Pebble is configured with one button access to my Nest so I can monitor and adjust the temperature in the house from my wrist. And there any many, many more apps for the Pebble platform that allow people to do what they want to with their watch.

I think Pebble have said it before, but they set out to design a watch first. Something that would look good, and work well as a watch. My current Pebble Time Steel looks like a watch. I’m hoping that as well as the platform, some of the design from the Pebble platform will start to appear in future Fitbit, but it is sad to realize that the features supported by my watch will start to degrade over time (either because of changes to iOS, or, as BoingBoing suggests, because cloud support features get turned off by Fitbit).

Fitbit have missed an opportunity IMHO in not taking the hardware designs forward too. Most of the things that I think put Pebble ahead of everybody, including Apple, were hardware features (water resistance, long battery life, always on screen, simple button based UI that actually works). Sure, there were some software features too (the web based developer environment was fantastic, the new iOS app looks great too, and, of course, the choices of third party watchfaces were amazing).

Over the years they had their bad moments too. Support was always a bit of a crapshoot. Sometimes it was great, other times not so much. At the end of the day though, things normally improved (at least until the next iOS update broke them again – which is not Pebble’s fault at all).

Despite its shortcomings, it might actually be time to switch to an Apple watch. After several years of Pebble use (from the original Kickstarter edition, to a Pebble Steel, to a Pebble Time Steel, and I backed the PT2 as well), going without a smartwatch isn’t an option and not having it be waterproof and support basic apps is a non-starter too. So, sorry Fitbit, I’ll be sticking with my One for step tracking, but I don’t need a wrist-worn fitness tracker; I need a smartwatch.

Comcast: Failure to Manage IP Network

Over the last few days I have been posting about my recent interactions with Comcast concerning an issue we see every night around 10pm here with poor network performance, which, based on trace route data seems to be caused by one node in their IP network, in Sunnyvale based on the hostname, becoming overloaded. Our neighbors here also see the issue (one I spoke to has decided he cannot do video calls with Asia at 10pm because it repeatedly drops the call). This has been going on for over a year now though, and still they are continuing to focus on the modem. So, I thought I would create a full timeline of this issue:

October 30, 2015

Initial report of the problem on Twitter, complete with trace route data showing the problem. At this point, the router wasn’t dropping packets, it was just extremely slow:

1-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 12.915 ms 11.091 ms 2893.567 ms

November 7-9, 2015

More traces sent via Twitter DM, all showing packets being dropped completely at the 5th hop (which is the Sunnyvale one):

0 192.168.1.1 27.53ms 1.74ms 1.51ms
1 73.162.38.1 162.18ms 169.54ms 162.59ms
2 …0-6-sur03.hayward.ca.sfba.comcast.net (68.86.248.125) 208.8ms 186.68ms 182.83ms
3 …0-ar01.santaclara.ca.sfba.comcast.net (68.87.192.209) 192.47ms
3 …1-ar01.santaclara.ca.sfba.comcast.net (68.87.192.213) 191.39ms 201.32ms
4 – * * *
5 …1-pe02.529bryant.ca.ibone.comcast.net (68.86.86.146) 292.25ms 306.23ms 305.94ms

Following these, the response from the Twitter support team was to send somebody out to check the local connection. Despite the data being very clear that the issue was not local to me.

December 27, 2015

After reporting the problem was still present, I get this response first:

I apologize for the internet speed issues you are experiencing, are you directly connected to our modem or over WIFI connection? -Will

Pointing out that the issue is in Sunnyvale, and so the method I am using to connect to the modem is somewhat irrelevant (especially since it works just fine for most of the day), I get this back:

I have created a internal escalation (ESL02290781) per your concerns, generally these issues fix themselves. -Will

The conversation then moves to email, and for much of January I was collecting and sending data; including a period of time where I ran pings every minute and recorded the times. That resulted in this graph for them:

Comcast Ping Times

I also demonstrated just how unpredictable it was with this series of speed tests, run just one minute apart:

Speedtest 9:44pm

Speedtest 9:45pm

Speedtest 9:46pm

That email thread got nowhere, and my final message on it (February 5, 2016) was not replied to. Comcast just seemed to decide that there was nothing they could (or would) do about the problem and dropped it.

May 29, 2016

After months of seeing the same thing night after night, I finally reported it on Twitter again hoping to find somebody at Comcast who understood IP networking (rather than just assuming all problems are in the cable modem or the local wiring). That was a waste of time, with this being the initial response:

I would like to recommend setting up an appointment for a technician to come out to your home. Please let me know a good time and date that is best for you and I will see what we have available for you. – Chad

When I point out that we’ve been down that path, and that really my modem & wiring cannot affect the performance of the router 45 miles away in Sunnyvale, I get this back instead:

After reviewing your account, I saw your modem has reached the End-Of-Life (EOL) list. Please visit, mydeviceinfo.xfinity.com/device.php?dev… to preview your device. If you would like to purchase another modem, please visit, mydeviceinfo.xfinity.com and select the speed tier you are subscribed to and a list of Comcast approved modems will be provided below to choose from. After you purchase your device you will need to provide us with the Make, Model, and the MAC Address to add your device to the account. – Chad

What is even more amusing about that is that the device I had at the time, an SB6121, is still on the list of approved modems on their site today. It is not end-of-life at all. It is listed as a fully supported modem:

SB6121 Supported

At the time, I was paying for a 25 Mbps tier; the support page for the modem clearly states it will support up to 75 Mbps. Additionally, outside of the times when the Sunnyvale router was overloaded, I was getting 25 Mbps down. Clearly it was not the modem, nor the wiring. But Comcast tech support was unable to accept that.

May 30, 2016 – June 7, 2016

More data sent showing the same problem still existing in the Sunnyvale node, around the same time at night. That resulted in this response on June 3, 2016:

Our team has reviewed this matter, and wanted to provide update with findings and resolved. The short version is that ICMP packets (the type of packets used in ping and traceroute) are, by design, low priority packets. If a router is busy, it will respond slowly or discard those packets instead of responding. A router is optimized for routing packets, passing them to a new destination, and low priority packets designed to stop at it can generally be responded to last or discarded if the CPU is needed for doing it’s actual job, forwarding packets. In many cases, routers are deliberately configured to drop these packets as a defense against certain types of DDoS attacks. The important part of the information from traceroute is how long does a packet take to make it through Comcast’s network, is there a pattern of packet loss or latency to all stops after a certain point. -AD

Finally, an admission that the node must be busy: “If a router is busy, it will respond slowly or discard those packets instead of responding.” But they are using that as an indication that it is behaving correctly. Of course, at one level it is, but the problem is when it gets to the point where it is dropping my ICMP packets, it is also dropping other packets making web content and streaming video unbearably slow, or even failing completely.

On June 4, they just wrote it off as not being a problem:

Our engineering team has review this matter, to ensure it’s no possible routing issues.with you explaining what you experience and neighbors I wanted to make sure we thoroughly reviewed and research this matter. They have reviewed your device and confirm it’s no issues. They have review plant integrity which is 100% also confirm device levels have been good since April. We have exhausted all options with engineering leadership team and the findings have confirm no possible routing issues with your device or area. -AD

And yet it continued to be a problem every night.

June 14-16, 2016

More data sent to them, and once again we are back to the modem as the problem:

I checked the area and modem, signal wise. I’m now seeing some slight noise on your connection, unique of the neighborhood. Would you open to having someone come check it out? – CR

I allowed them to waste their time sending somebody to the house, and that tech arrived on June 16. He found no issues at the house at all, with all the signals looking good. Additionally, when he looked at the data I had been sending he was confused as to why he had even been sent out for a problem that was clearly in the IP network in Sunnyvale and nothing to do with the cable modem or even the network in Alameda/Hayward.

Of course, there was still no change in the behavior with the network slowing down, or failing completely around 10pm almost every night as before.

June 20, 2016

This time I get a response saying that the modem I have, the SB6121, is indeed not end of life at all (that is only for rentals), and is a supported modem:

Hi John, the sb6121 is only marked end of life for rental units. We are continuing to activate retail modems on our network, and it is supported for the performance tier of service. Going back through your traces, they are much improved over previous, but it doesn’t appear to be an issue with Sunnyvale. There had been an area issue around that time that may have been related. How is the connection tonight? – CR

While the claim here is that a different problem was impacting performance, my data continued to show the same thing: the Sunnyvale node was overloading and dropping packets.

June 29, 2016

Again, after sending data from the evening of June 28, I receive the standard modem focused response:

Hi John, I’d like to send a signal to your modem that may interrupt your service for a few minutes. Can I do that now? -VG

Of course, as with all the previous threads, this got nowhere at all.

July 22 – August 2, 2016

Still been an issue, and still nothing is being done to even investigate the real problem. The response I get this time from the support folks is this:

This is something that we can’t repair here or view remotely. Thank you for your address, Now I can schedule a maintenance tech to go out to the affected area to see what’s going on and work towards getting this addressed and resolved. -KJ

So, once again, we’re back to the myopic focus on the modem and the cable connection in spite of all the data pointing at a problem elsewhere in the network. After much back and forth, and more and more data, all pointing at the same place, I get this:

Hi. Our engineer team sees no erroneous activity on the node and wanted me to verify if you were using wifi with these tests. Also, the modem has not been refreshed in over 56 days. Could you please give it a hard reset? If you are using wifi please try testing with ethernet if you are able. Thanks. -FL

Incredibly, they are back to the modem and the connection in my house which somehow can make packets drop in Sunnyvale. But not anywhere else. Where do they recruit these people? Not one of them has been able to explain how my modem, the wiring in my house or whether I am using ethernet or Wi-Fi for the local connection could cause packet loss in a router 5 hops away, while not impacting the ones in between. Oh, and of course, we get the IT standard response: reboot it.

August 10-14, 2016

I point out that other neighbors are seeing the same problem, and even provide some data from one of them showing the same thing happening at their house, again around 10pm. This is not isolated to my house, so I am hoping this will get them to move on from the modem/wiring issue.

On the 14th I get this back:

The node is showing no problems. I can also confirm that it is very far from being overloaded. Once you get a chance check it out and let us know. If the problem persists then we will have someone out to take a look free of charge. Thanks. -FL

OK, probably true at the time you looked, but how about at 10pm? When earlier you admitted it would drop ICMP packets only when it became “busy.” And it shouldn’t be dropping other packets unless it is actually overloaded. And “if the problem persists” is a ridiculous response to a problem that I have been reporting for 9 months already. Of course it is persisting.

August 20, 2016

A response about the meaning of the trace route data leads to a discussion about that. Really, they have actually been trying to suggest that dropping ICMP is something that happens, and is not an indication of other issues. Of course, since, as they noted earlier, it only happens when the node is getting loaded, that is nonsense. Furthermore, since the speed tests, web page loads and streaming video connections were also failing, I am pretty certain the problem is overloading somewhere, and the only node dropping ICMP is the Sunnyvale one. Doesn’t take a genius, but apparently it takes somebody smarter than a Comcast employee.

At the end of that, I get this:

Hello. A suggestion was made by one of the network engineers to possibly upgrade your modem to one with more channels. The SB6121 is 4×4, it is recommended that perhaps an SB6183 would work better for you. They are also interested in the make/model of the router you are using and are curious if the results you’ve provided are with ethernet directly to the modem without the router in the equation. Thanks. -FL

Yes, amazingly, they are back to the modem causing the packet loss in Sunnyvale. I don’t know what it takes to get these people to move on from the modem. Apparently almost a year of data pointing at the IP network in Sunnyvale isn’t sufficient. Either that or they have the most unreliable cable network in the world, and suspect that it will fail there all the time.

End of August through September, 2016

After sending a troubleshooter to the house to discover that the connection and signals there are still perfect, he installs a parallel modem taken from one of their offices in the house. During the next month, I collect data over both showing that they both experience the same thing. I was hoping this would rule out any further suggestion that it was the modem, but no. Instead, they send me a list of approved modems and tell me to buy one and they would reimburse me. I point out that my current one is still on that list, but they insist on a newer model. So, on September 19, 2016 I replace the SB6121 with an SB6183. The problem is unchanged of course since it is not a modem issue.

I send a lot more data, and get no response.

December 5-7, 2016

Monday night it was bad as usual, but my wife wanted to watch something on Netflix and we couldn’t even get the Netflix app on the Roku to launch as it was not able to contact Netflix’s servers. Once again, I posted the trace data from my iPad:

Trace Data

On Tuesday morning I get the typical response, and we’re back to the modem being the issue again:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

That thread ends with this statement:

I would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

Of course, because as I have been saying for over a year now, the problem is nothing to do with my modem, the wiring or the connection to the headend. It is in Sunnyvale. In the IP network. And it only happens at 10pm. Seriously, I don’t believe these people “review” anything when they say they review the account. And they certainly didn’t bother reading the detail in the blog post (it was the tweet about the post on Monday that they responded to first on Twitter).

Then this morning I get this:

Hi, John. I completely understand your frustration 100%. I reviewed the signals in your modem and I’m reading T4 time-outs that would be best resolved by a tech visit. I would highly recommend this as they can file any requests for maintenance in the area. Is there a date and time that would work best with your availability? I really want to get this resolved for you. -CE

That despite the fact that the speed test results just after he sent that showed a very acceptable throughput. And since yesterday when it was all OK.

Speedtest on 12/7

And then they claim they cannot resolve the issue because they need to send a tech to look at my modem. This guy needs a new job:

I understand completely. I’m a corporate staff member for Comcast and again, I really want to help. As my colleagues have previously mentioned, to best assist you, we need to have a tech come out and take a look at your equipment in-person and diagnose what is causing your service issues. Otherwise, we are unable to proceed with reaching a resolution. Feel free to reach out if you change your mind. Our corporate Digital Care team is available 24/7 on social media. -CE

Hopefully after this lands on his boss’ desk he will rethink his position and perhaps try to get somebody to review the load history on the Sunnyvale node. Perhaps looking at what happens between 10pm and midnight rather than checking in the middle of the day and writing it off as ‘all OK’ quickly.

Comcast Responds: Reboot the Modem

I kid you not. First they respond on Twitter with the standard “please DM us” – ostensibly to get my account number but this is really about getting the conversation out of the public channel. 

Then, when I respond via DM that they have all the information and I just want to speak to somebody at this point who can see beyond the modem and the wiring at my house, I get this back:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

Never mind the details I provided last night, and all the information over the past year. 

Never mind the information about having the modem replaced and even having Comcast install their own alongside mine to check in parallel. 

Never mind the fact that I said this happens almost every night. So the 47 days is irrelevant. It happens almost every night. 

All their support people are capable of is sending canned responses now. Clearly, his review of the account must have been very cursory or he might have seen the information they’ve been sent and that several people have checked the wiring at the house. And the modems. And the router. 

Totally useless. 

Update 1

I pointed out that it wasn’t the modem. And that we have totally exhausted that route. Here’s what I get back:

I would be happy to help get this corrected. Since the modem has been online for 47 days I would like to start by pushing a provisioning signal to your modem. This will make sure the speeds and firmware are correct on the device. This will take the modem offline for 3-5 minutes. -CN

I am amazed they can get anything done with responses like this.

Update 2

Still insisting that the dropped packets in Sunnyvale, ~45 miles from Alameda, are caused by the modem it seems, as I got this back:

In order to properly troubleshoot we have to start at the modem and then work down the line. In order to start to help you I will need to reset the modem since it has not been reset for 47 days. -CN

Update 3

Finally, moved away from the modem, but apparently still not understanding that the signal quality in Alameda at 1:30pm is totally irrelevant to an IP problem in Sunnyvale that happens around 10pm:

would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

And throughout this, my requests to either escalate to somebody who understands the trace route or to get a VP to call me have been ignored. Totally ignored.

Comcast Can’t Handle The Traffic

For more than a year I’ve been reporting that around 10pm every night our Comcast internet becomes unusable. Streaming video fails, web pages start timing out. Media content in Facebook and Twitter stop loading. Every time I have looked, the problem is the same: the Sunnyvale node is dropping packets. Like this:


That 5th node is Sunnyvale. In this run it wasn’t even able to get the reverse DNS for that node.

Over the course of the year, I have sent them pings, traces & speed tests over my network. From neighbors who all report the same thing, and even over a modem they installed in parallel with mine here (when they were claiming it was my modem in Alameda causing the packet loss in Sunnyvale!). It is always the same. Around 10pm the network becomes intermittently unusable, and it is always the Sunnyvale node that is dropping packets. It will stay bad until after midnight most nights. 

I’ve been told that the router dropping ICMP is normal (not true: it is true that it will drop ICMP packets first when overloaded though, so the high loss rate indicates an overloaded router; the failing video & web downloads tell me it is dropping other packets too).

I’ve been told it must be my router, my modem or the cable into the house. Nobody has been able to explain how any of those things could result in packet loss at Sunnyvale though. 

When the network is working, Speedtest results look like this:

Just after 10pm that periodically changes to this:

Every time I report it we go through the same sequence. First there is the request to send a tech to my house to check the wiring (total waste of everybody’s time since it is obvious the problem is in Sunnyvale, about 45 miles from here), then the request to send the logs (I’ve been sending them for months). Then radio silence. Until I ask again, when they repeat the process. 

At no point do they ever get the networking team to admit their network is becoming overloaded. A year after I first reported it, their network is apparently still unable to cope with the traffic passing over it between 10pm & midnight. Not continuously though. It is intermittent. Randomly, there will be a few minutes where nothing works, then it will be back. Streaming video fails, web loads timeout. My neighbors tell me they get dropped from conference calls too, making it hard to do late night calls with overseas teams. 

I am starting to wonder whether Comcast actually knows how to run an IP network. It certainly seems that every time I contact them they try to find a fault in the cable TV related parts of the system. Never in the IP network. I can’t believe they don’t have load charts for that node. Or statistics on packet drop rates. I just suspect they aren’t even looking.