ところが同じ5月、「Why Can't Twitter Scale? Blaine Cook Tries To Explain(なんでTwitterってスケールしないの?)」という、blog紹介記事がSilicon Alley Insiderに掲載される。記事の元になったblogエントリは、Twitterの前チーフアーキテクトだったBlaine Cook氏によるもの。Cook氏によれば、TwitterのスケールとRubyは何の関係もないという。

Why Can't Twitter Scale? Blaine Cook Tries To Explain

In Twitter's case, there is zero chance that the problems there are in any way related to their language. It is likely that there are architectural challenges which come from the fact that it is very hard to cache a Twitter data request since no two people ever get the same data. And even for a given user, the data requests change quickly since users are always receiving tweets. This is a hard, though not unsolvable problem that requires a very specialized caching architecture. Eran Hammer-Lahav, has done some interesting work in this area and talks about it in an extensive blog post.



その答えは、上記記事でもポイントされているように、Hueniverse社のfounderであるEran Hammer-Lahav氏による解説記事「Scaling a Microblogging Service - Part I」で触れられている。

Scaling a Microblogging Service - Part I
People don’t seem to get what is so hard about scaling the Twitter service. Some think it has to do with Rails, while others look at the hosting company or other operating factors. Then there are those who incorrectly think a distributed or federated service is the solution (which will only make things worse). The idea that building a large scale web application is trivial or a solved problem is simply ridiculous. In a way it is surprising that there are so few companies out there dealing with commoditizing web developing scaling.


There are two main subsystems from an infrastructure standpoint (regardless of the actual implementation). The first is the distribution (or delivery) system which takes every incoming message and delivers it to the right list of followers via Instant Messaging, SMS, or email. The second is the retrieval (or query) system which allows users to query what the people they are following and the general population (the public timeline) have been up to (few messages at a time).
Building the delivery system isn’t trivial, but much easier to scale. Subscription lists are generally static (who is following who) and tend to be small (even thousands of followers is still a small list). Twitter only allows following all updates of a certain user or tracking all updates of a certain list of words, which means there isn’t any heavy calculation being done at the distribution point as to who should receive which message.


In its current state, pushing updates out isn’t a challenge even at a large scale, and something that has generally scaled well and has showed very solid performance and stability in the past year. If Twitter was a push-only service, they would have no scaling issues at all – but in a way a much more difficult time competing with little-to-none technology barriers to slow competitors.

The retrieval system is where things are not as simple. Unlike webmail services where refreshing a user’s inbox only queries a very simple data set (is there anything new in MY inbox?), refreshing a user’s home page on Twitter queries a much more complex data set (are there any new updates in ALL my friends’ pages?) and the nature of the service means that the ratio of reads to writes is significantly different from most other web services.


It is these constant queries that bring Twitter down during popular events. The fact that a single request for a user’s home page or an API request for a user’s friends timeline must perform a complex iterative query is what causing some requests to take too long, at best timeout and at worst cause the entire system to lag behind. These custom views are expensive and mean that it is much more difficult to cache the final result of a request.

Going through a timeline request, the server first looks up the list of people the user is following. Then for each person, checks if their timeline is public or private, and if private, if the requesting user has the rights to view it. If the user has rights, the last few messages are retrieved, and the same is repeated for each person being followed. When done, all the messages are collected, sorted, and the latest messages are converted from their internal representation to the requested format (JASON, XML, RSS, or ATOM).


As long as all the messages in the system can fit into a single database table, this can be done with a pretty straight-forward query leaving the heavy lifting to the database. But once a single table or even a single server isn’t enough to hold the entire message base (or support the load), an application has to perform multiple database requests to gather the data. Partitioning the database which for many application is enough to offer scalability, solves the issue of a large and inefficient single table scenario, but is also the reason why the system slows down. Sharding takes away the ability to leverage the database indexing services to optimize users’ views. If the people you are following are not all on the same partition, multiple data queries are required.



One simple but painfully restrictive solution is to duplicate the data for each user. Basically what this means is turning the service into an email system. Each user is given a mailbox and whenever someone they are following publishes a status, it is copied into their inbox, as well as into all the other followers’ inboxes. This brings the solution in line with existing systems such as webmail services where data partitioning alone can accomplish great scaling. But this comes at a price. First the data storage grows significantly faster and requires much more disk space, together with increased backup cost. Second, it makes other actions much more complex such as erasing a message once sent.






blogテーブルが肥大化してくると問題が顕在化します。例えば1日平均で10万件程度の日記が書かれるとすると、1年でおよそ3500万〜4000万レコードのテーブルとなります。すると、WHERE句とORDER BY句が並存した場合にインデックスが効果的に使われない*ため、パフォーマンスが劣化していきます。例えばuser_idとpublication_datetimeにマルチカラムインデックスが張られていても、

SELECT title FROM blog WHERE user_id IN (...)




  • 表示されるデータがユーザーごとに異なるため、キャッシュのヒット率が非常に低い
  • データの性質上即時性が求められるため、あまりキャッシュしたくない




  • リクエストがあったタイミングでデータを取得する


  • データ更新のタイミングであらかじめ必要なデータを非同期に構築しておく



このあたりの話は、httpでいかにC10K問題を解決するか、という文脈で語られることが多い。データの格納をWebサーバと非同期でやるべきという考え方は、Life is beautiful(中島聡さん)の記事「マルチスレッド・プログラミングの落とし穴、その2」でも紹介されている。

前述のHammer-Lahav氏は、現在Real-Time Content DeliveryエンジンであるNouncer開発プロジェクトに従事されているということで、これを使えば開発者は簡単にスケールするMicro-blog(Twitterみたいなサービスの総称)サービスを作れちゃうらしい(サイトを見たところプロジェクトが停滞しているようだけれど..)。 こうやってまたひとつ技術がコモディティ化していくんだなー。。