Managing Traffic Spikes

Standing in the line for the return ferry tonight, along with hundreds of others who had been forced to try the ferry by the BART transbay outage, I had a chance to ask one of the SF Bay Ferry employees walking the line why they had not separated the Oakland & Alameda traffic with the extra boats they had added. That was something they had been doing when the free spare the air day rides increased the traffic. The answer was that they had not had time (or it simply didn’t occur to anybody to do it).

It reminded me though of the work we had done on our web services in the early days of scaling up to deal with rapid load increases. Sure, the first thing you do is add extra capacity. For the web service, more servers; for the ferry, more boats. That helps, but you might still find that some requests are hogging more of the resources than others, and delaying otherwise quick services.

Waiting in Alameda felt a bit like that yesterday morning as all the extra boats raced to Oakland & filled up, either arriving at Alameda with very little space for more people, or bypassing us completely because they were already full.

In our web world we faced a similar situation, with more complex requests holding up light weight requests that needed a fast response. The solution was simple: split the servers into two pools & manage the capacity for each pool separately. We actually ended up with more than two pools, but this simple approach has allowed us to manage resources much more efficiently and also implement automatic scaling for each pool independently. Sharding is often used for scaling databases, but it can also be effective for managing traffic.

The SF bay ferry service experienced a big spike in traffic today, and while they responded by adding additional capacity quickly (five boats instead of one this morning), they could have used that additional capacity more wisely and kept the lines at both Oakland and Alameda flowing more evenly.

 

Over a Year

Can’t believe it has been over a year since I’ve posted here. I actually started a post just over a year ago detailing the birth of my little boy. Then I decided that was too much information for public consumption & didn’t post it.

Now, over a year later, that little baby who seemed so fragile has become a toddler, running around our house getting in to everything he can reach. While my life is definitely still busier than it had been before his epic arrival, I think I can probably get back to posting here more frequently. Especially if I can get this Blogsy app for my iPad to work well – allowing me to write on BART.

You can probably expect a few reviews of baby products, as well as any commute horror stories (it is ironic that I am writing that today, the day of the big fire at West Oakland BART which caused havoc for my morning commute), but I’ll try to mix in some tech and photography posts too, promise.

 

New Art at Imagekind

Leucospermum catherinaeI have uploaded some of my recent favourite photos to my Imagekind store. Included in the new uploads are a sea turtle and a sunset from Hawaii, the leafy sea dragon from the Monterey Bay Aquarium and a Catherine Wheel Pincushion that we recently added to our dining room wall (as a 16×20 canvas).

In addition to the Imagekind website, you can now browse and buy prints, framed or not, and canvases directly from your iPhone with the new Artview app.

UFO

UFO?In this case, that’s an unidentified floating object! These jellies, once part of the jellies as art exhibit, have made a re-appearance and they are still quite amazing to watch.

With this black background and tiny specs reflecting the light, I think they look more like alien space craft than creatures from the deep.

Iconic Sea Nettles

Iconic Sea NettlesYou knew, given all the other photos from the Monterey Bay Aquarium, that at some point I had to post the iconic blue & orange sea nettle photo. Well, here it is.

For anybody who’s been down there, you’ll know just how hard it is to get a good shot of these guys. The room is dark, and people are pushing to get to the front and be right up close to the jellies. Oh, and did I mention that they’re constantly moving? Well, they are, not fast, but fast enough to make it that little bit harder to catch them well.

Leopard Shark

Leopard ProwlingAnother shot from the Monterey Bay Aquarium. We rested for a while in the back row of the little theatre in front of the kelp forest tank. While there, an albatross was brought out to be introduced to the crowd, but in the background one of the leopard sharks started swimming close to the glass.

BART

For those not living in the San Francisco area, BART is a regional train service here that connects many of the East Bay cities with San Francisco and the cities on the SF peninsula down as far as the international airport (although connecting to the airport was something of an afterthought that was only added in the time I’ve been living here).

With the ending of the shuttle I was taking from SF to San Bruno, where Devicescape‘s headquarters is located, my only option from Embarcadero to San Bruno is BART. Now, over three months into that commute, I have some observations about BART that as an infrequent user I hadn’t really noticed all of.

Noisy
OK, it would be a lie to say that I had not noticed the screeching BART trains make through the tunnels as an infrequent user of the service, but now I am on their trains for an hour each day (30 minutes each way), the screeching is unbelievable. I’ve never been on a train service anywhere else in the world that is so loud. Most of the time a conversation with the person sitting next to you is impossible. My noise canceling headphones only managed a small reduction in the noise level.

Dirty
Choosing cloth carpets for a commuter train in a place where rain is not that uncommon has to rank up there as one of the most stupid decisions ever made. And now the cars have been in service for many years, those carpets are stained and disgusting. Granted, there are a few cars with new hard floors in use, but some of them are labeled as “demonstration” cars, and they are clearly getting old themselves, so unless the demonstration period was measured in decades, it looks as though no plan to upgrade the remaining cars was ever agreed upon.

Cloth seats were also perhaps not the best choice, but at least that is more common on trains that serve cities as far apart as BART does. The problem is that most train services clean them, or replace them when they become too soiled. Apparently not BART; frequently I see seats that are so badly stained I would rather stand than go near them. And that’s just what I can see. A recent report found all kinds of unpleasant stuff hiding in the BART seating.

No Food & Drink
Even more bizarre for a regional train service with journey times of 30 minutes to over an hour, food & drink is banned. They even interrupt useful next train information announcements with warnings about large fines for consuming food or drink on board.

Most train services are happy to have concession stands and/or vending machines within their stations selling food & drink because they get much needed revenue from such stands. Some train systems even have carts being pushed through the trains, or specific cars where food & drink is sold. Not BART.

The reason? They want to keep the trains clean. Clearly that’s not working. They’d be better off selling food & drink in the stations & actually cleaning the trains.

Inconsistent Announcements
I mentioned in the last section that the no food & drink announcements frequently cut off information announcements about the next train (sometimes even the one about the train currently at the station). It is not just the food & drink ones though. BART has a plethora of poor audio quality (often complete with office noise in the background) announcements that are played from the central control centre and will always override the station specific train information.

Bad announcements are not just limited to the stations though. On board announcements are made by the operator. Some are really good at this, others not so good. Either way, it appears that there is no script for what should be said. I often catch a train in the mornings where the operator announces the destination station name at each stop – very disconcerting if you weren’t paying attention. Others give a little more detail, like “This is Daly City; Millbrae train,” which makes it much clearer.

BART frequently stops mid-tunnel, especially the evening trains for some reason, and once again the operator information varies from total silence to a clear reason. This sort of thing should be consistently clear.

Escalators & Ticket Barriers
Everybody understands that escalators & elevators break down, but not daily. Embarcadero station, it seems, has its escalators out of service more often than in service these days. Shopping malls manage to keep their escalators running most of the time, why is it so hard for BART?

Even more annoying, when the escalator is out of service, they chain them off, forcing everybody into the narrow staircase next to it. An escalator can still be used as stairs when it is not running, but it seems BART is unaware of this.

Delays
Delays on train services (at least outside of Japan) are not uncommon, but BART seems to have them more frequently than I would expect, and without reason. We will frequently stop between stations for minutes at a time with no more explanation than “there is a traffic jam ahead” (I kid not, that was an actual operator announcement from a few weeks ago).

Indicator Boards
Finally, the overhead indicator boards that are on each platform have 4 lines of text on them than could be used to continually advise about the time for the next train. Instead, BART has chosen to fill them with more of their pointless announcements most of the time, or have them all show just the current time (something I can get from my watch or cellphone).

There is enough space on these boards that they could easily reserve one or two lines for next train information all the time, and scroll their other messages in the other portion of the screen. The current usage of these displays is just another indication that BART management knows very little about running a train service.