Over the last few days I have been posting about my recent interactions with Comcast concerning an issue we see every night around 10pm here with poor network performance, which, based on trace route data seems to be caused by one node in their IP network, in Sunnyvale based on the hostname, becoming overloaded. Our neighbors here also see the issue (one I spoke to has decided he cannot do video calls with Asia at 10pm because it repeatedly drops the call). This has been going on for over a year now though, and still they are continuing to focus on the modem. So, I thought I would create a full timeline of this issue:

October 30, 2015

Initial report of the problem on Twitter, complete with trace route data showing the problem. At this point, the router wasn’t dropping packets, it was just extremely slow:

1-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 12.915 ms 11.091 ms 2893.567 ms

November 7-9, 2015

More traces sent via Twitter DM, all showing packets being dropped completely at the 5th hop (which is the Sunnyvale one):

0 192.168.1.1 27.53ms 1.74ms 1.51ms
1 73.162.38.1 162.18ms 169.54ms 162.59ms
2 …0-6-sur03.hayward.ca.sfba.comcast.net (68.86.248.125) 208.8ms 186.68ms 182.83ms
3 …0-ar01.santaclara.ca.sfba.comcast.net (68.87.192.209) 192.47ms
3 …1-ar01.santaclara.ca.sfba.comcast.net (68.87.192.213) 191.39ms 201.32ms
4 – * * *
5 …1-pe02.529bryant.ca.ibone.comcast.net (68.86.86.146) 292.25ms 306.23ms 305.94ms

Following these, the response from the Twitter support team was to send somebody out to check the local connection. Despite the data being very clear that the issue was not local to me.

December 27, 2015

After reporting the problem was still present, I get this response first:

I apologize for the internet speed issues you are experiencing, are you directly connected to our modem or over WIFI connection? -Will

Pointing out that the issue is in Sunnyvale, and so the method I am using to connect to the modem is somewhat irrelevant (especially since it works just fine for most of the day), I get this back:

I have created a internal escalation (ESL02290781) per your concerns, generally these issues fix themselves. -Will

The conversation then moves to email, and for much of January I was collecting and sending data; including a period of time where I ran pings every minute and recorded the times. That resulted in this graph for them:

I also demonstrated just how unpredictable it was with this series of speed tests, run just one minute apart:

That email thread got nowhere, and my final message on it (February 5, 2016) was not replied to. Comcast just seemed to decide that there was nothing they could (or would) do about the problem and dropped it.

May 29, 2016

After months of seeing the same thing night after night, I finally reported it on Twitter again hoping to find somebody at Comcast who understood IP networking (rather than just assuming all problems are in the cable modem or the local wiring). That was a waste of time, with this being the initial response:

I would like to recommend setting up an appointment for a technician to come out to your home. Please let me know a good time and date that is best for you and I will see what we have available for you. – Chad

When I point out that we’ve been down that path, and that really my modem & wiring cannot affect the performance of the router 45 miles away in Sunnyvale, I get this back instead:

After reviewing your account, I saw your modem has reached the End-Of-Life (EOL) list. Please visit, mydeviceinfo.xfinity.com/device.php?dev… to preview your device. If you would like to purchase another modem, please visit, mydeviceinfo.xfinity.com and select the speed tier you are subscribed to and a list of Comcast approved modems will be provided below to choose from. After you purchase your device you will need to provide us with the Make, Model, and the MAC Address to add your device to the account. – Chad

What is even more amusing about that is that the device I had at the time, an SB6121, is still on the list of approved modems on their site today. It is not end-of-life at all. It is listed as a fully supported modem:

At the time, I was paying for a 25 Mbps tier; the support page for the modem clearly states it will support up to 75 Mbps. Additionally, outside of the times when the Sunnyvale router was overloaded, I was getting 25 Mbps down. Clearly it was not the modem, nor the wiring. But Comcast tech support was unable to accept that.

May 30, 2016 – June 7, 2016

More data sent showing the same problem still existing in the Sunnyvale node, around the same time at night. That resulted in this response on June 3, 2016:

Our team has reviewed this matter, and wanted to provide update with findings and resolved. The short version is that ICMP packets (the type of packets used in ping and traceroute) are, by design, low priority packets. If a router is busy, it will respond slowly or discard those packets instead of responding. A router is optimized for routing packets, passing them to a new destination, and low priority packets designed to stop at it can generally be responded to last or discarded if the CPU is needed for doing it’s actual job, forwarding packets. In many cases, routers are deliberately configured to drop these packets as a defense against certain types of DDoS attacks. The important part of the information from traceroute is how long does a packet take to make it through Comcast’s network, is there a pattern of packet loss or latency to all stops after a certain point. -AD

Finally, an admission that the node must be busy: “If a router is busy, it will respond slowly or discard those packets instead of responding.” But they are using that as an indication that it is behaving correctly. Of course, at one level it is, but the problem is when it gets to the point where it is dropping my ICMP packets, it is also dropping other packets making web content and streaming video unbearably slow, or even failing completely.

On June 4, they just wrote it off as not being a problem:

Our engineering team has review this matter, to ensure it’s no possible routing issues.with you explaining what you experience and neighbors I wanted to make sure we thoroughly reviewed and research this matter. They have reviewed your device and confirm it’s no issues. They have review plant integrity which is 100% also confirm device levels have been good since April. We have exhausted all options with engineering leadership team and the findings have confirm no possible routing issues with your device or area. -AD

And yet it continued to be a problem every night.

June 14-16, 2016

More data sent to them, and once again we are back to the modem as the problem:

I checked the area and modem, signal wise. I’m now seeing some slight noise on your connection, unique of the neighborhood. Would you open to having someone come check it out? – CR

I allowed them to waste their time sending somebody to the house, and that tech arrived on June 16. He found no issues at the house at all, with all the signals looking good. Additionally, when he looked at the data I had been sending he was confused as to why he had even been sent out for a problem that was clearly in the IP network in Sunnyvale and nothing to do with the cable modem or even the network in Alameda/Hayward.

Of course, there was still no change in the behavior with the network slowing down, or failing completely around 10pm almost every night as before.

June 20, 2016

This time I get a response saying that the modem I have, the SB6121, is indeed not end of life at all (that is only for rentals), and is a supported modem:

Hi John, the sb6121 is only marked end of life for rental units. We are continuing to activate retail modems on our network, and it is supported for the performance tier of service. Going back through your traces, they are much improved over previous, but it doesn’t appear to be an issue with Sunnyvale. There had been an area issue around that time that may have been related. How is the connection tonight? – CR

While the claim here is that a different problem was impacting performance, my data continued to show the same thing: the Sunnyvale node was overloading and dropping packets.

June 29, 2016

Again, after sending data from the evening of June 28, I receive the standard modem focused response:

Hi John, I’d like to send a signal to your modem that may interrupt your service for a few minutes. Can I do that now? -VG

Of course, as with all the previous threads, this got nowhere at all.

July 22 – August 2, 2016

Still been an issue, and still nothing is being done to even investigate the real problem. The response I get this time from the support folks is this:

This is something that we can’t repair here or view remotely. Thank you for your address, Now I can schedule a maintenance tech to go out to the affected area to see what’s going on and work towards getting this addressed and resolved. -KJ

So, once again, we’re back to the myopic focus on the modem and the cable connection in spite of all the data pointing at a problem elsewhere in the network. After much back and forth, and more and more data, all pointing at the same place, I get this:

Hi. Our engineer team sees no erroneous activity on the node and wanted me to verify if you were using wifi with these tests. Also, the modem has not been refreshed in over 56 days. Could you please give it a hard reset? If you are using wifi please try testing with ethernet if you are able. Thanks. -FL

Incredibly, they are back to the modem and the connection in my house which somehow can make packets drop in Sunnyvale. But not anywhere else. Where do they recruit these people? Not one of them has been able to explain how my modem, the wiring in my house or whether I am using ethernet or Wi-Fi for the local connection could cause packet loss in a router 5 hops away, while not impacting the ones in between. Oh, and of course, we get the IT standard response: reboot it.

August 10-14, 2016

I point out that other neighbors are seeing the same problem, and even provide some data from one of them showing the same thing happening at their house, again around 10pm. This is not isolated to my house, so I am hoping this will get them to move on from the modem/wiring issue.

On the 14th I get this back:

The node is showing no problems. I can also confirm that it is very far from being overloaded. Once you get a chance check it out and let us know. If the problem persists then we will have someone out to take a look free of charge. Thanks. -FL

OK, probably true at the time you looked, but how about at 10pm? When earlier you admitted it would drop ICMP packets only when it became “busy.” And it shouldn’t be dropping other packets unless it is actually overloaded. And “if the problem persists” is a ridiculous response to a problem that I have been reporting for 9 months already. Of course it is persisting.

August 20, 2016

A response about the meaning of the trace route data leads to a discussion about that. Really, they have actually been trying to suggest that dropping ICMP is something that happens, and is not an indication of other issues. Of course, since, as they noted earlier, it only happens when the node is getting loaded, that is nonsense. Furthermore, since the speed tests, web page loads and streaming video connections were also failing, I am pretty certain the problem is overloading somewhere, and the only node dropping ICMP is the Sunnyvale one. Doesn’t take a genius, but apparently it takes somebody smarter than a Comcast employee.

At the end of that, I get this:

Hello. A suggestion was made by one of the network engineers to possibly upgrade your modem to one with more channels. The SB6121 is 4×4, it is recommended that perhaps an SB6183 would work better for you. They are also interested in the make/model of the router you are using and are curious if the results you’ve provided are with ethernet directly to the modem without the router in the equation. Thanks. -FL

Yes, amazingly, they are back to the modem causing the packet loss in Sunnyvale. I don’t know what it takes to get these people to move on from the modem. Apparently almost a year of data pointing at the IP network in Sunnyvale isn’t sufficient. Either that or they have the most unreliable cable network in the world, and suspect that it will fail there all the time.

End of August through September, 2016

After sending a troubleshooter to the house to discover that the connection and signals there are still perfect, he installs a parallel modem taken from one of their offices in the house. During the next month, I collect data over both showing that they both experience the same thing. I was hoping this would rule out any further suggestion that it was the modem, but no. Instead, they send me a list of approved modems and tell me to buy one and they would reimburse me. I point out that my current one is still on that list, but they insist on a newer model. So, on September 19, 2016 I replace the SB6121 with an SB6183. The problem is unchanged of course since it is not a modem issue.

I send a lot more data, and get no response.

December 5-7, 2016

Monday night it was bad as usual, but my wife wanted to watch something on Netflix and we couldn’t even get the Netflix app on the Roku to launch as it was not able to contact Netflix’s servers. Once again, I posted the trace data from my iPad:

On Tuesday morning I get the typical response, and we’re back to the modem being the issue again:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

That thread ends with this statement:

I would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

Of course, because as I have been saying for over a year now, the problem is nothing to do with my modem, the wiring or the connection to the headend. It is in Sunnyvale. In the IP network. And it only happens at 10pm. Seriously, I don’t believe these people “review” anything when they say they review the account. And they certainly didn’t bother reading the detail in the blog post (it was the tweet about the post on Monday that they responded to first on Twitter).

Then this morning I get this:

Hi, John. I completely understand your frustration 100%. I reviewed the signals in your modem and I’m reading T4 time-outs that would be best resolved by a tech visit. I would highly recommend this as they can file any requests for maintenance in the area. Is there a date and time that would work best with your availability? I really want to get this resolved for you. -CE

That despite the fact that the speed test results just after he sent that showed a very acceptable throughput. And since yesterday when it was all OK.

And then they claim they cannot resolve the issue because they need to send a tech to look at my modem. This guy needs a new job:

I understand completely. I’m a corporate staff member for Comcast and again, I really want to help. As my colleagues have previously mentioned, to best assist you, we need to have a tech come out and take a look at your equipment in-person and diagnose what is causing your service issues. Otherwise, we are unable to proceed with reaching a resolution. Feel free to reach out if you change your mind. Our corporate Digital Care team is available 24/7 on social media. -CE

Hopefully after this lands on his boss’ desk he will rethink his position and perhaps try to get somebody to review the load history on the Sunnyvale node. Perhaps looking at what happens between 10pm and midnight rather than checking in the middle of the day and writing it off as ‘all OK’ quickly.

I kid you not. First they respond on Twitter with the standard “please DM us” – ostensibly to get my account number but this is really about getting the conversation out of the public channel. 

Then, when I respond via DM that they have all the information and I just want to speak to somebody at this point who can see beyond the modem and the wiring at my house, I get this back:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

Never mind the details I provided last night, and all the information over the past year. 

Never mind the information about having the modem replaced and even having Comcast install their own alongside mine to check in parallel. 

Never mind the fact that I said this happens almost every night. So the 47 days is irrelevant. It happens almost every night. 

All their support people are capable of is sending canned responses now. Clearly, his review of the account must have been very cursory or he might have seen the information they’ve been sent and that several people have checked the wiring at the house. And the modems. And the router. 

Totally useless. 

Update 1

I pointed out that it wasn’t the modem. And that we have totally exhausted that route. Here’s what I get back:

I would be happy to help get this corrected. Since the modem has been online for 47 days I would like to start by pushing a provisioning signal to your modem. This will make sure the speeds and firmware are correct on the device. This will take the modem offline for 3-5 minutes. -CN

I am amazed they can get anything done with responses like this.

Update 2

Still insisting that the dropped packets in Sunnyvale, ~45 miles from Alameda, are caused by the modem it seems, as I got this back:

In order to properly troubleshoot we have to start at the modem and then work down the line. In order to start to help you I will need to reset the modem since it has not been reset for 47 days. -CN

Update 3

Finally, moved away from the modem, but apparently still not understanding that the signal quality in Alameda at 1:30pm is totally irrelevant to an IP problem in Sunnyvale that happens around 10pm:

would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

And throughout this, my requests to either escalate to somebody who understands the trace route or to get a VP to call me have been ignored. Totally ignored.

For more than a year I’ve been reporting that around 10pm every night our Comcast internet becomes unusable. Streaming video fails, web pages start timing out. Media content in Facebook and Twitter stop loading. Every time I have looked, the problem is the same: the Sunnyvale node is dropping packets. Like this:

That 5th node is Sunnyvale. In this run it wasn’t even able to get the reverse DNS for that node.

Over the course of the year, I have sent them pings, traces & speed tests over my network. From neighbors who all report the same thing, and even over a modem they installed in parallel with mine here (when they were claiming it was my modem in Alameda causing the packet loss in Sunnyvale!). It is always the same. Around 10pm the network becomes intermittently unusable, and it is always the Sunnyvale node that is dropping packets. It will stay bad until after midnight most nights. 

I’ve been told that the router dropping ICMP is normal (not true: it is true that it will drop ICMP packets first when overloaded though, so the high loss rate indicates an overloaded router; the failing video & web downloads tell me it is dropping other packets too).

I’ve been told it must be my router, my modem or the cable into the house. Nobody has been able to explain how any of those things could result in packet loss at Sunnyvale though. 

When the network is working, Speedtest results look like this:

Just after 10pm that periodically changes to this:

Every time I report it we go through the same sequence. First there is the request to send a tech to my house to check the wiring (total waste of everybody’s time since it is obvious the problem is in Sunnyvale, about 45 miles from here), then the request to send the logs (I’ve been sending them for months). Then radio silence. Until I ask again, when they repeat the process. 

At no point do they ever get the networking team to admit their network is becoming overloaded. A year after I first reported it, their network is apparently still unable to cope with the traffic passing over it between 10pm & midnight. Not continuously though. It is intermittent. Randomly, there will be a few minutes where nothing works, then it will be back. Streaming video fails, web loads timeout. My neighbors tell me they get dropped from conference calls too, making it hard to do late night calls with overseas teams. 

I am starting to wonder whether Comcast actually knows how to run an IP network. It certainly seems that every time I contact them they try to find a fault in the cable TV related parts of the system. Never in the IP network. I can’t believe they don’t have load charts for that node. Or statistics on packet drop rates. I just suspect they aren’t even looking. 



Around the time I signed up on Kickstarter for the Pebble Time 2, I also backed a project to create the second generation of a charging dock for Pebble Time watches. The gadget to the right arrived recently in the mail.

Up until now, when I charge my watch (roughly once a week), I leave it lying on the desk with the factory charging cable attached underneath. This turns my watch temporarily into a desk clock, so I can still see notifications etc while it charges. 

The Product

Firstly, what is it exactly? Well, it is an anodized aluminium (that’s aluminum for US readers) stand with a magnetic charging connector that allows the watch to be mounted on it, held in place by the two magnets, with the strap still attached whether it is a loop, or a two piece band. Not sure how well it would work with a single band NATO strap which goes under the watch, covering the charging port, but with my two piece Clockwork Synergy band it works well.

Behind the stand, a standard micro-USB connector provides the connection to power, and there is a notch in the base that grips the provided USB cable tightly, keeping it neat & tidy.

In Use

In use, the watch snaps easily onto the magnets & remains visible as a small desk clock thanks to a slight angle back:

Charging functions the same way as with the factory cable, but now it is held safely in place while it charges.

It is designed to be compatible with the Pebble Time 2 as well, so when that arrives I should be able to keep using it to charge the new watch as well.

Updated August 11, 2016: See new comments at the end.

One thing that being an Amazon Prime member teaches you is how good the various shipping companies are at getting packages delivered to the right place, at the right time. Amazon uses pretty much all the options, including, recently, their own Amazon Logistics delivery vans. From all those deliveries, the ones that regularly arrive late, or not at all, are the ones carried by UPS. Amazon Logistics and OnTrac are always on time or even early (often next day instead of two days).

In the last month, we’ve placed 8 Amazon orders. Two shipped by USPS (arrived one day early), two UPS (both late), and the other four came with Amazon Logistics (two early, two on time).

Second Day Air

The first late delivery was ordered using the Prime 2 day delivery. On a Wednesday morning. Normally, that would mean delivery by Friday. But in the world of UPS second day air, it meant Monday. And late Monday too (almost up to the 8pm deadline). That, but my calculation, is 5 days after the order. They dispute this by claiming they don’t count weekend days. Well I do, and so do their competitors who happily deliver Saturday & Sunday. 

Here’s the rub though, early on Sunday morning I ordered another item from Amazon, using prime 2 day shipping. It was delivered early on Monday morning. One day early and several hours before the order from the previous Wednesday. But it was delivered by Amazon Logistics, who apparently can not only move packages over the weekend, but deliver early when they can. Even the regular postal service delivers over the weekend, Sunday included. In fact, many of my Amazon two day orders arrive on Sunday via USPS. But not UPS. 

Next Day Guaranteed

Last night I needed a micro USB to USB C adapter quickly. I ordered them & paid extra for the next day delivery upgrade (still cheaper than buying one from a Target or Best Buy, but the shipping was almost as much as the adapters). Today at lunch time I received an alert from Amazon that my delivery had been delayed: 

So, somehow UPS managed to send the package to the wrong place, but South San Francisco isn’t far away. And that notice still suggests it might arrive today. Their own website seems less confident, but still not definitive that the package won’t make it on time:

At least I ordered mid-week too. Otherwise that one day delay might be a three day delay. 

Given that I’d not received any updates by 5pm, I sent an email asking whether there was any chance of it being delivered today (the website was still vague at best). Here’s the reply I received:

Not only is the package going to miss the guaranteed delivery time, they don’t even seem to know when it will be delivered. How can that be? Surely, the correct answer should be first thing the next morning? Even without the special Express handling option, UPS has an option for guaranteed before 9:30am delivery (Next Day Early). And that works from a lot further away than South SF. It should have been simple to guarantee delivery by 9:30am if they cared. 

A smart organization, when they make a mistake like this, would upgrade the package to the fastest possible option. But not UPS. I called the number Amazon support sent me to get better tracking information & the only thing the person who answered could say was it would arrive by 8pm tomorrow. A whole 24 hours late. She showed absolutely no concern for the fact that I had paid extra for next day shipping for a reason. Like I needed it today; not tomorrow. 

Mistakes Happen

I understand mistakes happen (although I kind of assumed the package sorting would be an automated process, at least near an Amazon distribution facility). What really counts is how the organization handles it. UPS had two options:

1. Promise delivery before 9:30am the next day (and keep that promise);
2. Show zero concern for missing the delivery deadline, and not even provide an updated delivery guarantee. 

The first is good customer care, and should be the standard policy in cases where the mistake is clearly internal as it was in this case. The second treats customers as if they don’t matter. UPS went with option 2. 

Update 1 (August 11, 2016)

I had the chance to talk to a very nice lady from UPS’ Customer Relations department at HQ this morning, and go over some of the concerns I raised here. As I noted above, UPS feels the earlier 2-day package was delivered on time (and it did arrive on the day they said it would). My main concern there is that if Amazon continues to ship packages using UPS at the end of the week, the 2 day prime shipping becomes 4-5 day shipping. Even more so, since not only do they not deliver on the weekend (unless the special Saturday option was selected when the shipment was sent), they also don’t even move the package towards its destination. Essentially, it freezes on Friday night and doesn’t move again until Monday morning.

The second issue, with the next day package turns out to be partly Amazon’s fault. It seems they decided that the ground transit time from their Las Vegas distribution center to here was short enough that they could ship it using UPS ground rather than a guaranteed next day service. When I suggested that having made an error like this, it would be a smart move for UPS to expedite the package and minimize the delay; the response to that was that it wouldn’t make business sense (and that it would also be potentially complex to determine which packages needed to be expedited, though I don’t buy that at all since it was possible to send me the alert when the mistake was detected). Given that Amazon chose ground shipping for a premium rate next day delivery though, at least part of the blame lies with them. (They did refund the shipping costs, but I would have preferred the items on time so I wasn’t rushing to complete things before traveling). I suspect there is an API somewhere which the Amazon brain connects into and queries the expected delivery time using all options and then picks the cheapest.

Just to be clear up front, I haven’t experienced anything that would suggest any personal data had been stolen from AT&T. This post concerns a risky policy they seem to have switched to where they collect more personal information than they need.

Continue reading →

Episode VII of the Star Wars franchise is something few can have missed the hype & merchandising for, at least here in the US. That aside, the movie gets a solid thumbs up from me. My one line summary would be that this is episode IV for a new generation of fans. 

Spoilers after the jump though, so if you’re still trying to be surprised, don’t click through.

Continue reading →

I’ve seen a number of articles suggesting that the problem with gun control in the US is that, while a majority of people support better background checks at time of purchase, congress is so worried about the loss of NRA funds that they refuse to pass any real laws that might restrict gun ownership. One of the best articles I’ve read, entitled “We’re Just Haggling Over Price“, suggests 90% of Americans support the enhanced checks, and that the changes would only delay purchasing by a few minutes.

While some may consider it to be a step in the right direction; I don’t. I’m sorry, but a law like that would be just another lame compromise, a bit like the Affordable Care Act was a massive compromise. Of course, the difference between the two was that the ACA was passed, but gun control laws fail every time they are proposed, no matter how many people are killed.

Continue reading →