Comcast: Back to the House

Amazingly, despite overwhelming amounts of data from myself, and data from people in Oakland, San Francisco and Santa Clara all commenting that their network sees the same issues around the same time of day (late evenings), once again I get a call from Comcast saying they want to come out to my house and attach things to the network here. They truly are fixated with the modem & the cable part of their network. And yet they cannot explain how something in my house could:

  1. Affect me only for bursts of time in the late evening, but be fine most of the rest of the time
  2. Affect other people here, and in various other cities in the bay area

Apparently, they just cannot see that this is probably impacting many of their customers. I have spoken to many here who see the issue but don’t have the patience to deal with Comcast’s totally abysmal customer support system. Even when it gets escalated, it appears they cannot get past it being localized to one house.

If they don’t want to look at the data in their routers, perhaps they should try calling some of their customers in the bay area and asking whether they use the network in the late evening regularly, and whether they have seen any issues with it.

Here’s the latest data I sent them:

January 16, 2017 (averaging 37% packet loss in the test):

January 17, 2017 (averaging 18% packet loss in the test):

Also from January 17, the trace showing losses at Sunnyvale again, and from nodes beyond it (likely dropped at Sunnyvale too – something they’ve always said was not happening):

I get it that their network can get loaded; I work in an industry that sees how much data traffic increases month on month as people use more and more high bandwidth services. But Comcast’s only role here is to provide the pipe to the Internet and keep it from being overloaded. I get it that there are huge swings in traffic volume across a 24 hour period; I don’t necessarily expect to see the full bandwidth during peak hours (right now, at 11am on a weekday, for reference, I am seeing about 30Mbps), but dropping to under 1Mbps is unacceptable. And the times when the packet losses are so great that my router decides the Internet is inaccessible are completely unacceptable. Personally, I would say once it drops below 50% of the bandwidth I’m paying for, that is a problem; I suspect they have a lower percentage in mind, but I doubt it is as low as 1%. If they do consider 1% of the contract bandwidth to be acceptable, perhaps the FCC should take a closer look at the service Comcast provides.

Comcast: Now It’s The Wi-Fi

Last Thursday I spent almost 2 hours on the phone with a couple of engineers from Comcast. The one who lead the call was sitting in the Hayward head end location, connected to the same network I was using (I’m not sure where the other one was physically located – he was dialed into the call). Around 10pm, we both started to see “anomolies” in the Speedtest results. As well seeing packet loss in the traceroute results from the Sunnyvale router. We also saw longer than typical ping times to Google (normally low tens of milliseconds from my home).

He tried a number of things, but nothing made a difference. Until, at 10:20pm, it rapidly went back to being normal. The pings improved. The Speedtest results improved. The traceroutes started showing responses from Sunnyvale again (and by 10:30pm they were also consistently good times). Unfortunately, it didn’t go bad again that night while we were on the phone, but I was hopeful that having had two of their engineers see what I was seeing there might be a light at the end of the tunnel. I should have known better.

Wi-Fi

During that call there was a question about how I was connected to the network, and we even switched my computer between the 2.4GHz and 5GHz bands on my Wi-Fi to verify that made no difference. It did not. Where I live, both bands are pretty clear of interference. Normally I have the two bands using the same SSID, but during the call I switched the configuration so they were separated (allowing me to easily switch between them).

I did also point out at the time that it was somewhat irrelevant since I had seen and documented the problem in many configurations, from many devices, including some of my neighbors homes and even some friends in other bay area locations.

To be clear here, I have seen the problem:

  • On Wi-Fi connected computers (both bands)
  • On Wi-Fi connected mobile devices (both bands)
  • On ethernet connected computers through my router
  • On ethernet connected computers direct to a Comcast-provided modem
  • On both our wired and wireless Roku boxes

Additionally, at other times in the day there is no problem at all from the same computer I was using during the call. Here is a speed test from that same computer taken around 6:45pm this evening, while we were streaming HD video from Netflix on the Roku and a visitor was watching a basketball game on his Mac. As I’ve made clear throughout this, outside of the 10pm to midnight (some days a little later) window, the performance is very acceptable.

That was run from the browser on my Mac Mini connecting to the internet over the same Wi-Fi network. The only difference was I ran it at 6:45pm rather than 10pm.

Home Visit

So, today’s call from the engineering team did not go well. Firstly, even though I saw the slow down over the weekend and yesterday (I was not online at the right times on Sunday or Monday), apparently the equipment in the Hayward location saw none of it.

Furthermore, he now believes that the “anomalies” seen in the Speedtest results during our call last week were related to Adobe Flash and not the network. While I am not going to defend Flash at all, I will say that Speedtest is rock solid, and the versions I am running are not Flash based, so that doesn’t apply to my results.

Finally, it was suggested that perhaps the issue is in my Wi-Fi network and they should come to my house to check the setup in the house. Really. I am not kidding. Once again they are concluding that their network is fine and any issue must be in my house.

When I pointed out that several of their engineers have already been here, and perhaps he should just pick the phone up and call one of them, I was told that they would need to come out to see it themselves. I guess they don’t trust any of their colleagues to check the modem and signal quality.

Once again, Comcast folks, if you believe that something in my house is the cause of this, please explain to me how so many of my neighbours see the same issue. Is my Apple Airport Extreme somehow managing to interfere with them all?

Also, I’m willing to accept that the packet losses in Sunnyvale are coincidence, if you can show me what else in your network could cause so many people in this area to see this issue. And why they correspond so well in terms of time to when the network quality is poor. Right now, when the network is working well, I see this from my traces:

5 be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 11.224 ms 16.193 ms 13.003 ms

The only time I see slower ping times or no responses from that node is when all the other things I try are bad too. What are those other things? Well, here’s a list of the things we often see when it is behaving poorly:

  • Web pages, Facebook content, Twitter content and emails not loading;
  • Netflix and Amazon Prime video not being able to load their index screens;
  • If we have the index loaded, Netflix and Amazon not being able to start playing videos;
  • Videos that are playing keep pausing to load, which can take many minutes to recover from, and in many cases times out;
  • Speedtest shows results under 1Mbps for download (compare to the speed above when it is working)

Experience

What I haven’t mentioned to Comcast until today is that I have a little bit more knowledge about Wi-Fi than perhaps their regular customer. Having been working in a Wi-Fi software company since 2003, and even been part of the creation of some of the Wi-Fi standards, I think I am pretty well placed to know whether my Wi-Fi setup here is likely to be the cause.

I have spent perhaps too much time inside shielded rooms helping access point manufacturers fine tune their software to squeeze the maxiumum performance from their Wi-Fi stacks. I am well aware that the 802.11ac system I have here far exceeds the capabilities of the Comcast network it connects to for Internet access.

I have spent too much time connecting to Wi-Fi networks all over the world and running performance tests on them (my wife is regularly upset by me searching for public Wi-Fi to use and test when we are out).

Tonight

Of couse, tonight I didn’t see any issue while I was reading on my iPad (normally I will encounter a period of time when media in particular doesn’t load), and my timed download script didn’t see anything either tonight.

Comcast: Still Fixated by the Modem

Last week’s email to the CEO’s office generated some communication again. On Friday afternoon, I received a call from a lady in their Executive Customer Relations department letting me know that she had reached out to the engineering team the day before, but still had not heard anything back from them.

Earlier today I got another call from her, relaying their answer: that they had installed a parallel modem in the house and then paid for a replacement modem, and that had exhausted their options. I kid you not. Finally, moments ago I received this in a DM from the Twitter support folks, relaying questions from, I assume, the same engineering team:

Good morning. The engineering team is asking if you see the same results with a wired test during the times shown in the screenshots. They also noted that there are no events shown in the node history corresponding with the dates and times pointed out. They also wanted to know if you changed the location of your modem at all recently. Thank you. -FL

Despite everything they’ve been sent, and the fact that the entire recent discussion was initiated by my frustration that they cannot see anything past the cable modem, they are still fixated on it being a modem problem.

So, given this fixation with the modem, here are my questions back to them:

Dear Comcast Engineering,

There is more to your network than the cable modems & headends. In fact, in the grand scheme of things, that link is pretty short. But given your fixation on the modem, and the wiring in my house, despite several of your techs agreeing that it all seems to be just fine, here are my questions to you:

1. Please explain how my modem here in one house in Alameda can be causing packet loss at the 5th hop of the trace, and only that hop (which has been a reliable symptom of this since I first reported it, despite all the modem swaps). Note: that router is located ~45 miles away in Sunnyvale assuming your hostnames are accurate (the node is 68.86.90.93, which, according to DNS corresponds to be-33651-cr01.sunnyvale.ca.ibone.comcast.net); it is neither in my house, nor at the end of the wire my modem is connected to.

2. Additionally, please explain how my modem can also be impacting all the other people in Alameda and Oakland who have so far commented on my Nextdoor and Facebook threads sharing the blog post last week confirming they too see the issue with the internet becoming unusable for periods of time during the late evening.

3. Finally, please include in your explanation how this happens only in the late evening, and at other times the same modem, same wiring and same devices are able to get 50 Mbps download speeds reliably.

Thanks in advance,
John

Comcast: Failure to Manage IP Network

Over the last few days I have been posting about my recent interactions with Comcast concerning an issue we see every night around 10pm here with poor network performance, which, based on trace route data seems to be caused by one node in their IP network, in Sunnyvale based on the hostname, becoming overloaded. Our neighbors here also see the issue (one I spoke to has decided he cannot do video calls with Asia at 10pm because it repeatedly drops the call). This has been going on for over a year now though, and still they are continuing to focus on the modem. So, I thought I would create a full timeline of this issue:

October 30, 2015

Initial report of the problem on Twitter, complete with trace route data showing the problem. At this point, the router wasn’t dropping packets, it was just extremely slow:

1-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93) 12.915 ms 11.091 ms 2893.567 ms

November 7-9, 2015

More traces sent via Twitter DM, all showing packets being dropped completely at the 5th hop (which is the Sunnyvale one):

0 192.168.1.1 27.53ms 1.74ms 1.51ms
1 73.162.38.1 162.18ms 169.54ms 162.59ms
2 …0-6-sur03.hayward.ca.sfba.comcast.net (68.86.248.125) 208.8ms 186.68ms 182.83ms
3 …0-ar01.santaclara.ca.sfba.comcast.net (68.87.192.209) 192.47ms
3 …1-ar01.santaclara.ca.sfba.comcast.net (68.87.192.213) 191.39ms 201.32ms
4 – * * *
5 …1-pe02.529bryant.ca.ibone.comcast.net (68.86.86.146) 292.25ms 306.23ms 305.94ms

Following these, the response from the Twitter support team was to send somebody out to check the local connection. Despite the data being very clear that the issue was not local to me.

December 27, 2015

After reporting the problem was still present, I get this response first:

I apologize for the internet speed issues you are experiencing, are you directly connected to our modem or over WIFI connection? -Will

Pointing out that the issue is in Sunnyvale, and so the method I am using to connect to the modem is somewhat irrelevant (especially since it works just fine for most of the day), I get this back:

I have created a internal escalation (ESL02290781) per your concerns, generally these issues fix themselves. -Will

The conversation then moves to email, and for much of January I was collecting and sending data; including a period of time where I ran pings every minute and recorded the times. That resulted in this graph for them:

Comcast Ping Times

I also demonstrated just how unpredictable it was with this series of speed tests, run just one minute apart:

Speedtest 9:44pm

Speedtest 9:45pm

Speedtest 9:46pm

That email thread got nowhere, and my final message on it (February 5, 2016) was not replied to. Comcast just seemed to decide that there was nothing they could (or would) do about the problem and dropped it.

May 29, 2016

After months of seeing the same thing night after night, I finally reported it on Twitter again hoping to find somebody at Comcast who understood IP networking (rather than just assuming all problems are in the cable modem or the local wiring). That was a waste of time, with this being the initial response:

I would like to recommend setting up an appointment for a technician to come out to your home. Please let me know a good time and date that is best for you and I will see what we have available for you. – Chad

When I point out that we’ve been down that path, and that really my modem & wiring cannot affect the performance of the router 45 miles away in Sunnyvale, I get this back instead:

After reviewing your account, I saw your modem has reached the End-Of-Life (EOL) list. Please visit, mydeviceinfo.xfinity.com/device.php?dev… to preview your device. If you would like to purchase another modem, please visit, mydeviceinfo.xfinity.com and select the speed tier you are subscribed to and a list of Comcast approved modems will be provided below to choose from. After you purchase your device you will need to provide us with the Make, Model, and the MAC Address to add your device to the account. – Chad

What is even more amusing about that is that the device I had at the time, an SB6121, is still on the list of approved modems on their site today. It is not end-of-life at all. It is listed as a fully supported modem:

SB6121 Supported

At the time, I was paying for a 25 Mbps tier; the support page for the modem clearly states it will support up to 75 Mbps. Additionally, outside of the times when the Sunnyvale router was overloaded, I was getting 25 Mbps down. Clearly it was not the modem, nor the wiring. But Comcast tech support was unable to accept that.

May 30, 2016 – June 7, 2016

More data sent showing the same problem still existing in the Sunnyvale node, around the same time at night. That resulted in this response on June 3, 2016:

Our team has reviewed this matter, and wanted to provide update with findings and resolved. The short version is that ICMP packets (the type of packets used in ping and traceroute) are, by design, low priority packets. If a router is busy, it will respond slowly or discard those packets instead of responding. A router is optimized for routing packets, passing them to a new destination, and low priority packets designed to stop at it can generally be responded to last or discarded if the CPU is needed for doing it’s actual job, forwarding packets. In many cases, routers are deliberately configured to drop these packets as a defense against certain types of DDoS attacks. The important part of the information from traceroute is how long does a packet take to make it through Comcast’s network, is there a pattern of packet loss or latency to all stops after a certain point. -AD

Finally, an admission that the node must be busy: “If a router is busy, it will respond slowly or discard those packets instead of responding.” But they are using that as an indication that it is behaving correctly. Of course, at one level it is, but the problem is when it gets to the point where it is dropping my ICMP packets, it is also dropping other packets making web content and streaming video unbearably slow, or even failing completely.

On June 4, they just wrote it off as not being a problem:

Our engineering team has review this matter, to ensure it’s no possible routing issues.with you explaining what you experience and neighbors I wanted to make sure we thoroughly reviewed and research this matter. They have reviewed your device and confirm it’s no issues. They have review plant integrity which is 100% also confirm device levels have been good since April. We have exhausted all options with engineering leadership team and the findings have confirm no possible routing issues with your device or area. -AD

And yet it continued to be a problem every night.

June 14-16, 2016

More data sent to them, and once again we are back to the modem as the problem:

I checked the area and modem, signal wise. I’m now seeing some slight noise on your connection, unique of the neighborhood. Would you open to having someone come check it out? – CR

I allowed them to waste their time sending somebody to the house, and that tech arrived on June 16. He found no issues at the house at all, with all the signals looking good. Additionally, when he looked at the data I had been sending he was confused as to why he had even been sent out for a problem that was clearly in the IP network in Sunnyvale and nothing to do with the cable modem or even the network in Alameda/Hayward.

Of course, there was still no change in the behavior with the network slowing down, or failing completely around 10pm almost every night as before.

June 20, 2016

This time I get a response saying that the modem I have, the SB6121, is indeed not end of life at all (that is only for rentals), and is a supported modem:

Hi John, the sb6121 is only marked end of life for rental units. We are continuing to activate retail modems on our network, and it is supported for the performance tier of service. Going back through your traces, they are much improved over previous, but it doesn’t appear to be an issue with Sunnyvale. There had been an area issue around that time that may have been related. How is the connection tonight? – CR

While the claim here is that a different problem was impacting performance, my data continued to show the same thing: the Sunnyvale node was overloading and dropping packets.

June 29, 2016

Again, after sending data from the evening of June 28, I receive the standard modem focused response:

Hi John, I’d like to send a signal to your modem that may interrupt your service for a few minutes. Can I do that now? -VG

Of course, as with all the previous threads, this got nowhere at all.

July 22 – August 2, 2016

Still been an issue, and still nothing is being done to even investigate the real problem. The response I get this time from the support folks is this:

This is something that we can’t repair here or view remotely. Thank you for your address, Now I can schedule a maintenance tech to go out to the affected area to see what’s going on and work towards getting this addressed and resolved. -KJ

So, once again, we’re back to the myopic focus on the modem and the cable connection in spite of all the data pointing at a problem elsewhere in the network. After much back and forth, and more and more data, all pointing at the same place, I get this:

Hi. Our engineer team sees no erroneous activity on the node and wanted me to verify if you were using wifi with these tests. Also, the modem has not been refreshed in over 56 days. Could you please give it a hard reset? If you are using wifi please try testing with ethernet if you are able. Thanks. -FL

Incredibly, they are back to the modem and the connection in my house which somehow can make packets drop in Sunnyvale. But not anywhere else. Where do they recruit these people? Not one of them has been able to explain how my modem, the wiring in my house or whether I am using ethernet or Wi-Fi for the local connection could cause packet loss in a router 5 hops away, while not impacting the ones in between. Oh, and of course, we get the IT standard response: reboot it.

August 10-14, 2016

I point out that other neighbors are seeing the same problem, and even provide some data from one of them showing the same thing happening at their house, again around 10pm. This is not isolated to my house, so I am hoping this will get them to move on from the modem/wiring issue.

On the 14th I get this back:

The node is showing no problems. I can also confirm that it is very far from being overloaded. Once you get a chance check it out and let us know. If the problem persists then we will have someone out to take a look free of charge. Thanks. -FL

OK, probably true at the time you looked, but how about at 10pm? When earlier you admitted it would drop ICMP packets only when it became “busy.” And it shouldn’t be dropping other packets unless it is actually overloaded. And “if the problem persists” is a ridiculous response to a problem that I have been reporting for 9 months already. Of course it is persisting.

August 20, 2016

A response about the meaning of the trace route data leads to a discussion about that. Really, they have actually been trying to suggest that dropping ICMP is something that happens, and is not an indication of other issues. Of course, since, as they noted earlier, it only happens when the node is getting loaded, that is nonsense. Furthermore, since the speed tests, web page loads and streaming video connections were also failing, I am pretty certain the problem is overloading somewhere, and the only node dropping ICMP is the Sunnyvale one. Doesn’t take a genius, but apparently it takes somebody smarter than a Comcast employee.

At the end of that, I get this:

Hello. A suggestion was made by one of the network engineers to possibly upgrade your modem to one with more channels. The SB6121 is 4×4, it is recommended that perhaps an SB6183 would work better for you. They are also interested in the make/model of the router you are using and are curious if the results you’ve provided are with ethernet directly to the modem without the router in the equation. Thanks. -FL

Yes, amazingly, they are back to the modem causing the packet loss in Sunnyvale. I don’t know what it takes to get these people to move on from the modem. Apparently almost a year of data pointing at the IP network in Sunnyvale isn’t sufficient. Either that or they have the most unreliable cable network in the world, and suspect that it will fail there all the time.

End of August through September, 2016

After sending a troubleshooter to the house to discover that the connection and signals there are still perfect, he installs a parallel modem taken from one of their offices in the house. During the next month, I collect data over both showing that they both experience the same thing. I was hoping this would rule out any further suggestion that it was the modem, but no. Instead, they send me a list of approved modems and tell me to buy one and they would reimburse me. I point out that my current one is still on that list, but they insist on a newer model. So, on September 19, 2016 I replace the SB6121 with an SB6183. The problem is unchanged of course since it is not a modem issue.

I send a lot more data, and get no response.

December 5-7, 2016

Monday night it was bad as usual, but my wife wanted to watch something on Netflix and we couldn’t even get the Netflix app on the Roku to launch as it was not able to contact Netflix’s servers. Once again, I posted the trace data from my iPad:

Trace Data

On Tuesday morning I get the typical response, and we’re back to the modem being the issue again:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

That thread ends with this statement:

I would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

Of course, because as I have been saying for over a year now, the problem is nothing to do with my modem, the wiring or the connection to the headend. It is in Sunnyvale. In the IP network. And it only happens at 10pm. Seriously, I don’t believe these people “review” anything when they say they review the account. And they certainly didn’t bother reading the detail in the blog post (it was the tweet about the post on Monday that they responded to first on Twitter).

Then this morning I get this:

Hi, John. I completely understand your frustration 100%. I reviewed the signals in your modem and I’m reading T4 time-outs that would be best resolved by a tech visit. I would highly recommend this as they can file any requests for maintenance in the area. Is there a date and time that would work best with your availability? I really want to get this resolved for you. -CE

That despite the fact that the speed test results just after he sent that showed a very acceptable throughput. And since yesterday when it was all OK.

Speedtest on 12/7

And then they claim they cannot resolve the issue because they need to send a tech to look at my modem. This guy needs a new job:

I understand completely. I’m a corporate staff member for Comcast and again, I really want to help. As my colleagues have previously mentioned, to best assist you, we need to have a tech come out and take a look at your equipment in-person and diagnose what is causing your service issues. Otherwise, we are unable to proceed with reaching a resolution. Feel free to reach out if you change your mind. Our corporate Digital Care team is available 24/7 on social media. -CE

Hopefully after this lands on his boss’ desk he will rethink his position and perhaps try to get somebody to review the load history on the Sunnyvale node. Perhaps looking at what happens between 10pm and midnight rather than checking in the middle of the day and writing it off as ‘all OK’ quickly.

Comcast Responds: Reboot the Modem

I kid you not. First they respond on Twitter with the standard “please DM us” – ostensibly to get my account number but this is really about getting the conversation out of the public channel. 

Then, when I respond via DM that they have all the information and I just want to speak to somebody at this point who can see beyond the modem and the wiring at my house, I get this back:

I reviewed the account and the signal for the area and the signal history on the modem. At this time all the signal in the area is prefect. I am showing the modem has been online for 47 days. Regularly reboot your modem and router help keep it running smother. Software updates are made from time to time that require a restart – just like any other consumer device. You can easily and automatically reboot your Xfinity equipment by using our My Account app. You can also use the app to view, change or share your WiFi network name and password. -CN

Never mind the details I provided last night, and all the information over the past year. 

Never mind the information about having the modem replaced and even having Comcast install their own alongside mine to check in parallel. 

Never mind the fact that I said this happens almost every night. So the 47 days is irrelevant. It happens almost every night. 

All their support people are capable of is sending canned responses now. Clearly, his review of the account must have been very cursory or he might have seen the information they’ve been sent and that several people have checked the wiring at the house. And the modems. And the router. 

Totally useless. 

Update 1

I pointed out that it wasn’t the modem. And that we have totally exhausted that route. Here’s what I get back:

I would be happy to help get this corrected. Since the modem has been online for 47 days I would like to start by pushing a provisioning signal to your modem. This will make sure the speeds and firmware are correct on the device. This will take the modem offline for 3-5 minutes. -CN

I am amazed they can get anything done with responses like this.

Update 2

Still insisting that the dropped packets in Sunnyvale, ~45 miles from Alameda, are caused by the modem it seems, as I got this back:

In order to properly troubleshoot we have to start at the modem and then work down the line. In order to start to help you I will need to reset the modem since it has not been reset for 47 days. -CN

Update 3

Finally, moved away from the modem, but apparently still not understanding that the signal quality in Alameda at 1:30pm is totally irrelevant to an IP problem in Sunnyvale that happens around 10pm:

would be more then happy to assist but currently after reviewing the signal for the area and on the modem there is no issue. -CN

And throughout this, my requests to either escalate to somebody who understands the trace route or to get a VP to call me have been ignored. Totally ignored.

Comcast Can’t Handle The Traffic

For more than a year I’ve been reporting that around 10pm every night our Comcast internet becomes unusable. Streaming video fails, web pages start timing out. Media content in Facebook and Twitter stop loading. Every time I have looked, the problem is the same: the Sunnyvale node is dropping packets. Like this:


That 5th node is Sunnyvale. In this run it wasn’t even able to get the reverse DNS for that node.

Over the course of the year, I have sent them pings, traces & speed tests over my network. From neighbors who all report the same thing, and even over a modem they installed in parallel with mine here (when they were claiming it was my modem in Alameda causing the packet loss in Sunnyvale!). It is always the same. Around 10pm the network becomes intermittently unusable, and it is always the Sunnyvale node that is dropping packets. It will stay bad until after midnight most nights. 

I’ve been told that the router dropping ICMP is normal (not true: it is true that it will drop ICMP packets first when overloaded though, so the high loss rate indicates an overloaded router; the failing video & web downloads tell me it is dropping other packets too).

I’ve been told it must be my router, my modem or the cable into the house. Nobody has been able to explain how any of those things could result in packet loss at Sunnyvale though. 

When the network is working, Speedtest results look like this:

Just after 10pm that periodically changes to this:

Every time I report it we go through the same sequence. First there is the request to send a tech to my house to check the wiring (total waste of everybody’s time since it is obvious the problem is in Sunnyvale, about 45 miles from here), then the request to send the logs (I’ve been sending them for months). Then radio silence. Until I ask again, when they repeat the process. 

At no point do they ever get the networking team to admit their network is becoming overloaded. A year after I first reported it, their network is apparently still unable to cope with the traffic passing over it between 10pm & midnight. Not continuously though. It is intermittent. Randomly, there will be a few minutes where nothing works, then it will be back. Streaming video fails, web loads timeout. My neighbors tell me they get dropped from conference calls too, making it hard to do late night calls with overseas teams. 

I am starting to wonder whether Comcast actually knows how to run an IP network. It certainly seems that every time I contact them they try to find a fault in the cable TV related parts of the system. Never in the IP network. I can’t believe they don’t have load charts for that node. Or statistics on packet drop rates. I just suspect they aren’t even looking. 

Time Dock Pebble 2

Around the time I signed up on Kickstarter for the Pebble Time 2, I also backed a project to create the second generation of a charging dock for Pebble Time watches. The gadget to the right arrived recently in the mail.

Up until now, when I charge my watch (roughly once a week), I leave it lying on the desk with the factory charging cable attached underneath. This turns my watch temporarily into a desk clock, so I can still see notifications etc while it charges. 

The Product

Firstly, what is it exactly? Well, it is an anodized aluminium (that’s aluminum for US readers) stand with a magnetic charging connector that allows the watch to be mounted on it, held in place by the two magnets, with the strap still attached whether it is a loop, or a two piece band. Not sure how well it would work with a single band NATO strap which goes under the watch, covering the charging port, but with my two piece Clockwork Synergy band it works well.

Behind the stand, a standard micro-USB connector provides the connection to power, and there is a notch in the base that grips the provided USB cable tightly, keeping it neat & tidy.

In Use

In use, the watch snaps easily onto the magnets & remains visible as a small desk clock thanks to a slight angle back:


Charging functions the same way as with the factory cable, but now it is held safely in place while it charges.

It is designed to be compatible with the Pebble Time 2 as well, so when that arrives I should be able to keep using it to charge the new watch as well.

The Useless Parcel Service

Updated August 11, 2016: See new comments at the end.


One thing that being an Amazon Prime member teaches you is how good the various shipping companies are at getting packages delivered to the right place, at the right time. Amazon uses pretty much all the options, including, recently, their own Amazon Logistics delivery vans. From all those deliveries, the ones that regularly arrive late, or not at all, are the ones carried by UPS. Amazon Logistics and OnTrac are always on time or even early (often next day instead of two days).

In the last month, we’ve placed 8 Amazon orders. Two shipped by USPS (arrived one day early), two UPS (both late), and the other four came with Amazon Logistics (two early, two on time).

Second Day Air

The first late delivery was ordered using the Prime 2 day delivery. On a Wednesday morning. Normally, that would mean delivery by Friday. But in the world of UPS second day air, it meant Monday. And late Monday too (almost up to the 8pm deadline). That, but my calculation, is 5 days after the order. They dispute this by claiming they don’t count weekend days. Well I do, and so do their competitors who happily deliver Saturday & Sunday. 

Here’s the rub though, early on Sunday morning I ordered another item from Amazon, using prime 2 day shipping. It was delivered early on Monday morning. One day early and several hours before the order from the previous Wednesday. But it was delivered by Amazon Logistics, who apparently can not only move packages over the weekend, but deliver early when they can. Even the regular postal service delivers over the weekend, Sunday included. In fact, many of my Amazon two day orders arrive on Sunday via USPS. But not UPS. 

Next Day Guaranteed

Last night I needed a micro USB to USB C adapter quickly. I ordered them & paid extra for the next day delivery upgrade (still cheaper than buying one from a Target or Best Buy, but the shipping was almost as much as the adapters). Today at lunch time I received an alert from Amazon that my delivery had been delayed: 

So, somehow UPS managed to send the package to the wrong place, but South San Francisco isn’t far away. And that notice still suggests it might arrive today. Their own website seems less confident, but still not definitive that the package won’t make it on time:


At least I ordered mid-week too. Otherwise that one day delay might be a three day delay. 

Given that I’d not received any updates by 5pm, I sent an email asking whether there was any chance of it being delivered today (the website was still vague at best). Here’s the reply I received:


Not only is the package going to miss the guaranteed delivery time, they don’t even seem to know when it will be delivered. How can that be? Surely, the correct answer should be first thing the next morning? Even without the special Express handling option, UPS has an option for guaranteed before 9:30am delivery (Next Day Early). And that works from a lot further away than South SF. It should have been simple to guarantee delivery by 9:30am if they cared. 

A smart organization, when they make a mistake like this, would upgrade the package to the fastest possible option. But not UPS. I called the number Amazon support sent me to get better tracking information & the only thing the person who answered could say was it would arrive by 8pm tomorrow. A whole 24 hours late. She showed absolutely no concern for the fact that I had paid extra for next day shipping for a reason. Like I needed it today; not tomorrow. 

Mistakes Happen

I understand mistakes happen (although I kind of assumed the package sorting would be an automated process, at least near an Amazon distribution facility). What really counts is how the organization handles it. UPS had two options:

  1. Promise delivery before 9:30am the next day (and keep that promise);
  2. Show zero concern for missing the delivery deadline, and not even provide an updated delivery guarantee. 

The first is good customer care, and should be the standard policy in cases where the mistake is clearly internal as it was in this case. The second treats customers as if they don’t matter. UPS went with option 2. 

Update 1 (August 11, 2016)

I had the chance to talk to a very nice lady from UPS’ Customer Relations department at HQ this morning, and go over some of the concerns I raised here. As I noted above, UPS feels the earlier 2-day package was delivered on time (and it did arrive on the day they said it would). My main concern there is that if Amazon continues to ship packages using UPS at the end of the week, the 2 day prime shipping becomes 4-5 day shipping. Even more so, since not only do they not deliver on the weekend (unless the special Saturday option was selected when the shipment was sent), they also don’t even move the package towards its destination. Essentially, it freezes on Friday night and doesn’t move again until Monday morning.

The second issue, with the next day package turns out to be partly Amazon’s fault. It seems they decided that the ground transit time from their Las Vegas distribution center to here was short enough that they could ship it using UPS ground rather than a guaranteed next day service. When I suggested that having made an error like this, it would be a smart move for UPS to expedite the package and minimize the delay; the response to that was that it wouldn’t make business sense (and that it would also be potentially complex to determine which packages needed to be expedited, though I don’t buy that at all since it was possible to send me the alert when the mistake was detected). Given that Amazon chose ground shipping for a premium rate next day delivery though, at least part of the blame lies with them. (They did refund the shipping costs, but I would have preferred the items on time so I wasn’t rushing to complete things before traveling). I suspect there is an API somewhere which the Amazon brain connects into and queries the expected delivery time using all options and then picks the cheapest.

TriNet Experiences

It has been just over a year now that we have had TriNet as our HR service at work, and my opinion of them gets worse & worse with each interaction. There are definitely a few bright points, but still my overall advice for any small company thinking about using them would be simply, don’t. That is from an employee perspective of course, but hopefully when choosing something like this the employee experience is an important element too.

Continue reading