Last Thursday I spent almost 2 hours on the phone with a couple of engineers from Comcast. The one who lead the call was sitting in the Hayward head end location, connected to the same network I was using (I’m not sure where the other one was physically located – he was dialed into the call). Around 10pm, we both started to see “anomolies” in the Speedtest results. As well seeing packet loss in the traceroute results from the Sunnyvale router. We also saw longer than typical ping times to Google (normally low tens of milliseconds from my home).
He tried a number of things, but nothing made a difference. Until, at 10:20pm, it rapidly went back to being normal. The pings improved. The Speedtest results improved. The traceroutes started showing responses from Sunnyvale again (and by 10:30pm they were also consistently good times). Unfortunately, it didn’t go bad again that night while we were on the phone, but I was hopeful that having had two of their engineers see what I was seeing there might be a light at the end of the tunnel. I should have known better.
During that call there was a question about how I was connected to the network, and we even switched my computer between the 2.4GHz and 5GHz bands on my Wi-Fi to verify that made no difference. It did not. Where I live, both bands are pretty clear of interference. Normally I have the two bands using the same SSID, but during the call I switched the configuration so they were separated (allowing me to easily switch between them).
I did also point out at the time that it was somewhat irrelevant since I had seen and documented the problem in many configurations, from many devices, including some of my neighbors homes and even some friends in other bay area locations.
To be clear here, I have seen the problem:
- On Wi-Fi connected computers (both bands)
- On Wi-Fi connected mobile devices (both bands)
- On ethernet connected computers through my router
- On ethernet connected computers direct to a Comcast-provided modem
- On both our wired and wireless Roku boxes
Additionally, at other times in the day there is no problem at all from the same computer I was using during the call. Here is a speed test from that same computer taken around 6:45pm this evening, while we were streaming HD video from Netflix on the Roku and a visitor was watching a basketball game on his Mac. As I’ve made clear throughout this, outside of the 10pm to midnight (some days a little later) window, the performance is very acceptable.
That was run from the browser on my Mac Mini connecting to the internet over the same Wi-Fi network. The only difference was I ran it at 6:45pm rather than 10pm.
So, today’s call from the engineering team did not go well. Firstly, even though I saw the slow down over the weekend and yesterday (I was not online at the right times on Sunday or Monday), apparently the equipment in the Hayward location saw none of it.
Furthermore, he now believes that the “anomalies” seen in the Speedtest results during our call last week were related to Adobe Flash and not the network. While I am not going to defend Flash at all, I will say that Speedtest is rock solid, and the versions I am running are not Flash based, so that doesn’t apply to my results.
Finally, it was suggested that perhaps the issue is in my Wi-Fi network and they should come to my house to check the setup in the house. Really. I am not kidding. Once again they are concluding that their network is fine and any issue must be in my house.
When I pointed out that several of their engineers have already been here, and perhaps he should just pick the phone up and call one of them, I was told that they would need to come out to see it themselves. I guess they don’t trust any of their colleagues to check the modem and signal quality.
Once again, Comcast folks, if you believe that something in my house is the cause of this, please explain to me how so many of my neighbours see the same issue. Is my Apple Airport Extreme somehow managing to interfere with them all?
Also, I’m willing to accept that the packet losses in Sunnyvale are coincidence, if you can show me what else in your network could cause so many people in this area to see this issue. And why they correspond so well in terms of time to when the network quality is poor. Right now, when the network is working well, I see this from my traces:
5 be-33651-cr01.sunnyvale.ca.ibone.comcast.net (18.104.22.168) 11.224 ms 16.193 ms 13.003 ms
The only time I see slower ping times or no responses from that node is when all the other things I try are bad too. What are those other things? Well, here’s a list of the things we often see when it is behaving poorly:
- Web pages, Facebook content, Twitter content and emails not loading;
- Netflix and Amazon Prime video not being able to load their index screens;
- If we have the index loaded, Netflix and Amazon not being able to start playing videos;
- Videos that are playing keep pausing to load, which can take many minutes to recover from, and in many cases times out;
- Speedtest shows results under 1Mbps for download (compare to the speed above when it is working)
What I haven’t mentioned to Comcast until today is that I have a little bit more knowledge about Wi-Fi than perhaps their regular customer. Having been working in a Wi-Fi software company since 2003, and even been part of the creation of some of the Wi-Fi standards, I think I am pretty well placed to know whether my Wi-Fi setup here is likely to be the cause.
I have spent perhaps too much time inside shielded rooms helping access point manufacturers fine tune their software to squeeze the maxiumum performance from their Wi-Fi stacks. I am well aware that the 802.11ac system I have here far exceeds the capabilities of the Comcast network it connects to for Internet access.
I have spent too much time connecting to Wi-Fi networks all over the world and running performance tests on them (my wife is regularly upset by me searching for public Wi-Fi to use and test when we are out).
Of couse, tonight I didn’t see any issue while I was reading on my iPad (normally I will encounter a period of time when media in particular doesn’t load), and my timed download script didn’t see anything either tonight.