I received an email over the weekend from Bird, one of the companies behind electric scooter sharing here in the SF area stating the following:

The City of Alameda will consider banning shared electric scooters at a City Council hearing on December 4th at 7:00 p.m. at City Hall! Let’s remind city leaders that Alameda residents want access to affordable, environmentally-friendly options such as Bird in their city. 

The backlash against these scooters in San Francisco came as a surprise in a city that claims to be progressive, and it was disappointing to see the city ban them, then decide to run small scale trials of them with different companies from the two who pioneered the service, and in limited numbers. To see Alameda follow this crazy, illogical path is even more depressing.

Those who have been following along here (or on Twitter) might recall all the strange issues we were having with Comcast’s internet service, specifically abysmal performance during the late evenings. For most of that time, and it was over a year I was trying to get them to fix the issue, they were sadly the only option we had for internet access. They had previously priced our city owned cable TV & ISP out of business, and then bought the customers. While that was happening, AT&T decided to simply drop out of providing internet service to the almost 500 homes in our neighborhood.

Then one day a friend posted information about a new wireless ISP starting up here in Alameda: Common Networks. I signed up to be on their waiting list immediately. Not long afterwards I was surprised by the email requesting information & selection of a date to install. They were still in beta at the time and suggested I keep Comcast as a backup for a little while, but with just one brief outage in the first two months they were easily more reliable than Comcast. Not to mention being faster. Here’s a speed test result from my phone:

Latterly with Comcast I was paying the same as Common charges, but getting max 25Mbps down and 5Mbps up.

Continue reading →

Last week’s email to the CEO’s office generated some communication again. On Friday afternoon, I received a call from a lady in their Executive Customer Relations department letting me know that she had reached out to the engineering team the day before, but still had not heard anything back from them.

Earlier today I got another call from her, relaying their answer: that they had installed a parallel modem in the house and then paid for a replacement modem, and that had exhausted their options. I kid you not. Finally, moments ago I received this in a DM from the Twitter support folks, relaying questions from, I assume, the same engineering team:

Good morning. The engineering team is asking if you see the same results with a wired test during the times shown in the screenshots. They also noted that there are no events shown in the node history corresponding with the dates and times pointed out. They also wanted to know if you changed the location of your modem at all recently. Thank you. -FL

Despite everything they’ve been sent, and the fact that the entire recent discussion was initiated by my frustration that they cannot see anything past the cable modem, they are still fixated on it being a modem problem.

So, given this fixation with the modem, here are my questions back to them:

Dear Comcast Engineering,

There is more to your network than the cable modems & headends. In fact, in the grand scheme of things, that link is pretty short. But given your fixation on the modem, and the wiring in my house, despite several of your techs agreeing that it all seems to be just fine, here are my questions to you:

1. Please explain how my modem here in one house in Alameda can be causing packet loss at the 5th hop of the trace, and only that hop (which has been a reliable symptom of this since I first reported it, despite all the modem swaps). Note: that router is located ~45 miles away in Sunnyvale assuming your hostnames are accurate (the node is 68.86.90.93, which, according to DNS corresponds to be-33651-cr01.sunnyvale.ca.ibone.comcast.net); it is neither in my house, nor at the end of the wire my modem is connected to.

2. Additionally, please explain how my modem can also be impacting all the other people in Alameda and Oakland who have so far commented on my Nextdoor and Facebook threads sharing the blog post last week confirming they too see the issue with the internet becoming unusable for periods of time during the late evening.

3. Finally, please include in your explanation how this happens only in the late evening, and at other times the same modem, same wiring and same devices are able to get 50 Mbps download speeds reliably.

Thanks in advance,
John

Over the last few days I have been posting about my recent interactions with Comcast concerning an issue we see every night around 10pm here with poor network performance, which, based on trace route data seems to be caused by one node in their IP network, in Sunnyvale based on the hostname, becoming overloaded. Our neighbors here also see the issue (one I spoke to has decided he cannot do video calls with Asia at 10pm because it repeatedly drops the call). This has been going on for over a year now though, and still they are continuing to focus on the modem. So, I thought I would create a full timeline of this issue:

October 30, 2015

Initial report of the problem on Twitter, complete with trace route data showing the problem. At this point, the router wasn’t dropping packets, it was just extremely slow:

1-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 12.915 ms 11.091 ms 2893.567 ms

November 7-9, 2015

More traces sent via Twitter DM, all showing packets being dropped completely at the 5th hop (which is the Sunnyvale one):

0 192.168.1.1 27.53ms 1.74ms 1.51ms
1 73.162.38.1 162.18ms 169.54ms 162.59ms
2 …0-6-sur03.hayward.ca.sfba.comcast.net (68.86.248.125) 208.8ms 186.68ms 182.83ms
3 …0-ar01.santaclara.ca.sfba.comcast.net (68.87.192.209) 192.47ms
3 …1-ar01.santaclara.ca.sfba.comcast.net (68.87.192.213) 191.39ms 201.32ms
4 – * * *
5 …1-pe02.529bryant.ca.ibone.comcast.net (68.86.86.146) 292.25ms 306.23ms 305.94ms

Following these, the response from the Twitter support team was to send somebody out to check the local connection. Despite the data being very clear that the issue was not local to me.

December 27, 2015

After reporting the problem was still present, I get this response first:

I apologize for the internet speed issues you are experiencing, are you directly connected to our modem or over WIFI connection? -Will

Pointing out that the issue is in Sunnyvale, and so the method I am using to connect to the modem is somewhat irrelevant (especially since it works just fine for most of the day), I get this back:

I have created a internal escalation (ESL02290781) per your concerns, generally these issues fix themselves. -Will

The conversation then moves to email, and for much of January I was collecting and sending data; including a period of time where I ran pings every minute and recorded the times. That resulted in this graph for them:

I also demonstrated just how unpredictable it was with this series of speed tests, run just one minute apart:

That email thread got nowhere, and my final message on it (February 5, 2016) was not replied to. Comcast just seemed to decide that there was nothing they could (or would) do about the problem and dropped it.

May 29, 2016

After months of seeing the same thing night after night, I finally reported it on Twitter again hoping to find somebody at Comcast who understood IP networking (rather than just assuming all problems are in the cable modem or the local wiring). That was a waste of time, with this being the initial response:

I would like to recommend setting up an appointment for a technician to come out to your home. Please let me know a good time and date that is best for you and I will see what we have available for you. – Chad

When I point out that we’ve been down that path, and that really my modem & wiring cannot affect the performance of the router 45 miles away in Sunnyvale, I get this back instead:

After reviewing your account, I saw your modem has reached the End-Of-Life (EOL) list. Please visit, mydeviceinfo.xfinity.com/device.php?dev… to preview your device. If you would like to purchase another modem, please visit, mydeviceinfo.xfinity.com and select the speed tier you are subscribed to and a list of Comcast approved modems will be provided below to choose from. After you purchase your device you will need to provide us with the Make, Model, and the MAC Address to add your device to the account. – Chad

What is even more amusing about that is that the device I had at the time, an SB6121, is still on the list of approved modems on their site today. It is not end-of-life at all. It is listed as a fully supported modem:

At the time, I was paying for a 25 Mbps tier; the support page for the modem clearly states it will support up to 75 Mbps. Additionally, outside of the times when the Sunnyvale router was overloaded, I was getting 25 Mbps down. Clearly it was not the modem, nor the wiring. But Comcast tech support was unable to accept that.

May 30, 2016 – June 7, 2016

More data sent showing the same problem still existing in the Sunnyvale node, around the same time at night. That resulted in this response on June 3, 2016:

Our team has reviewed this matter, and wanted to provide update with findings and resolved. The short version is that ICMP packets (the type of packets used in ping and traceroute) are, by design, low priority packets. If a router is busy, it will respond slowly or discard those packets instead of responding. A router is optimized for routing packets, passing them to a new destination, and low priority packets designed to stop at it can generally be responded to last or discarded if the CPU is needed for doing it’s actual job, forwarding packets. In many cases, routers are deliberately configured to drop these packets as a defense against certain types of DDoS attacks. The important part of the information from traceroute is how long does a packet take to make it through Comcast’s network, is there a pattern of packet loss or latency to all stops after a certain point. -AD

Finally, an admission that the node must be busy: “If a router is busy, it will respond slowly or discard those packets instead of responding.” But they are using that as an indication that it is behaving correctly. Of course, at one level it is, but the problem is when it gets to the point where it is dropping my ICMP packets, it is also dropping other packets making web content and streaming video unbearably slow, or even failing completely.

On June 4, they just wrote it off as not being a problem:

Our engineering team has review this matter, to ensure it’s no possible routing issues.with you explaining what you experience and neighbors I wanted to make sure we thoroughly reviewed and research this matter. They have reviewed your device and confirm it’s no issues. They have review plant integrity which is 100% also confirm device levels have been good since April. We have exhausted all options with engineering leadership team and the findings have confirm no possible routing issues with your device or area. -AD

And yet it continued to be a problem every night.

June 14-16, 2016

More data sent to them, and once again we are back to the modem as the problem:

I checked the area and modem, signal wise. I’m now seeing some slight noise on your connection, unique of the neighborhood. Would you open to having someone come check it out? – CR

I allowed them to waste their time sending somebody to the house, and that tech arrived on June 16. He found no issues at the house at all, with all the signals looking good. Additionally, when he looked at the data I had been sending he was confused as to why he had even been sent out for a problem that was clearly in the IP network in Sunnyvale and nothing to do with the cable modem or even the network in Alameda/Hayward.

Of course, there was still no change in the behavior with the network slowing down, or failing completely around 10pm almost every night as before.

June 20, 2016

This time I get a response saying that the modem I have, the SB6121, is indeed not end of life at all (that is only for rentals), and is a supported modem:

Hi John, the sb6121 is only marked end of life for rental units. We are continuing to activate retail modems on our network, and it is supported for the performance tier of service. Going back through your traces, they are much improved over previous, but it doesn’t appear to be an issue with Sunnyvale. There had been an area issue around that time that may have been related. How is the connection tonight? – CR

While the claim here is that a different problem was impacting performance, my data continued to show the same thing: the Sunnyvale node was overloading and dropping packets.

June 29, 2016

Again, after sending data from the evening of June 28, I receive the standard modem focused response:

Hi John, I’d like to send a signal to your modem that may interrupt your service for a few minutes. Can I do that now? -VG

Of course, as with all the previous threads, this got nowhere at all.

July 22 – August 2, 2016

Still been an issue, and still nothing is being done to even investigate the real problem. The response I get this time from the support folks is this:

This is something that we can’t repair here or view remotely. Thank you for your address, Now I can schedule a maintenance tech to go out to the affected area to see what’s going on and work towards getting this addressed and resolved. -KJ

So, once again, we’re back to the myopic focus on the modem and the cable connection in spite of all the data pointing at a problem elsewhere in the network. After much back and forth, and more and more data, all pointing at the same place, I get this:

Hi. Our engineer team sees no erroneous activity on the node and wanted me to verify if you were using wifi with these tests. Also, the modem has not been refreshed in over 56 days. Could you please give it a hard reset? If you are using wifi please try testing with ethernet if you are able. Thanks. -FL

Incredibly, they are back to the modem and the connection in my house which somehow can make packets drop in Sunnyvale. But not anywhere else. Where do they recruit these people? Not one of them has been able to explain how my modem, the wiring in my house or whether I am using ethernet or Wi-Fi for the local connection could cause packet loss in a router 5 hops away, while not impacting the ones in between. Oh, and of course, we get the IT standard response: reboot it.

August 10-14, 2016

I point out that other neighbors are seeing the same problem, and even provide some data from one of them showing the same thing happening at their house, again around 10pm. This is not isolated to my house, so I am hoping this will get them to move on from the modem/wiring issue.

On the 14th I get this back:

The node is showing no problems. I can also confirm that it is very far from being overloaded. Once you get a chance check it out and let us know. If the problem persists then we will have someone out to take a look free of charge. Thanks. -FL

OK, probably true at the time you looked, but how about at 10pm? When earlier you admitted it would drop ICMP packets only when it became “busy.” And it shouldn’t be dropping other packets unless it is actually overloaded. And “if the problem persists” is a ridiculous response to a problem that I have been reporting for 9 months already. Of course it is persisting.

August 20, 2016

A response about the meaning of the trace route data leads to a discussion about that. Really, they have actually been trying to suggest that dropping ICMP is something that happens, and is not an indication of other issues. Of course, since, as they noted earlier, it only happens when the node is getting loaded, that is nonsense. Furthermore, since the speed tests, web page loads and streaming video connections were also failing, I am pretty certain the problem is overloading somewhere, and the only node dropping ICMP is the Sunnyvale one. Doesn’t take a genius, but apparently it takes somebody smarter than a Comcast employee.

At the end of that, I get this:

Hello. A suggestion was made by one of the network engineers to possibly upgrade your modem to one with more channels. The SB6121 is 4×4, it is recommended that perhaps an SB6183 would work better for you. They are also interested in the make/model of the router you are using and are curious if the results you’ve provided are with ethernet directly to the modem without the router in the equation. Thanks. -FL

Yes, amazingly, they are back to the modem causing the packet loss in Sunnyvale. I don’t know what it takes to get these people to move on from the modem. Apparently almost a year of data pointing at the IP network in Sunnyvale isn’t sufficient. Either that or they have the most unreliable cable network in the world, and suspect that it will fail there all the time.

End of August through September, 2016

After sending a troubleshooter to the house to discover that the connection and signals there are still perfect, he installs a parallel modem taken from one of their offices in the house. During the next month, I collect data over both showing that they both experience the same thing. I was hoping this would rule out any further suggestion that it was the modem, but no. Instead, they send me a list of approved modems and tell me to buy one and they would reimburse me. I point out that my current one is still on that list, but they insist on a newer model. So, on September 19, 2016 I replace the SB6121 with an SB6183. The problem is unchanged of course since it is not a modem issue.

I send a lot more data, and get no response.

December 5-7, 2016

Monday night it was bad as usual, but my wife wanted to watch something on Netflix and we couldn’t even get the Netflix app on the Roku to launch as it was not able to contact Netflix’s servers. Once again, I posted the trace data from my iPad:

On Tuesday morning I get the typical response, and we’re back to the modem being the issue again:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

That thread ends with this statement:

I would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

Of course, because as I have been saying for over a year now, the problem is nothing to do with my modem, the wiring or the connection to the headend. It is in Sunnyvale. In the IP network. And it only happens at 10pm. Seriously, I don’t believe these people “review” anything when they say they review the account. And they certainly didn’t bother reading the detail in the blog post (it was the tweet about the post on Monday that they responded to first on Twitter).

Then this morning I get this:

Hi, John. I completely understand your frustration 100%. I reviewed the signals in your modem and I’m reading T4 time-outs that would be best resolved by a tech visit. I would highly recommend this as they can file any requests for maintenance in the area. Is there a date and time that would work best with your availability? I really want to get this resolved for you. -CE

That despite the fact that the speed test results just after he sent that showed a very acceptable throughput. And since yesterday when it was all OK.

And then they claim they cannot resolve the issue because they need to send a tech to look at my modem. This guy needs a new job:

I understand completely. I’m a corporate staff member for Comcast and again, I really want to help. As my colleagues have previously mentioned, to best assist you, we need to have a tech come out and take a look at your equipment in-person and diagnose what is causing your service issues. Otherwise, we are unable to proceed with reaching a resolution. Feel free to reach out if you change your mind. Our corporate Digital Care team is available 24/7 on social media. -CE

Hopefully after this lands on his boss’ desk he will rethink his position and perhaps try to get somebody to review the load history on the Sunnyvale node. Perhaps looking at what happens between 10pm and midnight rather than checking in the middle of the day and writing it off as ‘all OK’ quickly.