Riak Core: Dynamo Building Blocks
Highly recommended!
Summary:
Andy Gross discusses the design philosophy behind Riak based on Amazon Dynamo – Gossip Protocol, Consistent Hashing, Vector clocks, Read Repair, etc. -, overviewing its main features and architecture.
Amazon’s Dynamo paper:
Dynamo: Amazon’s Highly Available Key-value Store (HTML)
Dynamo: Amazon’s Highly Available Key-value Store (PDF)
One of the more intriguing slide represented http/apps/dbs as a stack to show that while scaling of the http layer is well-known, scaling of apps is more difficult but still doable, the scaling of storage is the most expensive and difficult.
I mention that because scaling of databases I suspect has a lot in common with scaling of topic maps.
On the issue of consistency, the point was made that “expires” can be included in HTTP headers, which indicate a fact is good until some time. I wonder, could a topic have a “last merged” property? So that a user can choose the timeliness they need? So that “last merged” 7 days ago is public information, “last merged” 3 days ago is subscriber information and the most recent “last merged” is premium information.
For example, instead of trying to regulate insider trading, the SEC could create a topic map of stocks and sell insider trading information, suitably priced to keep its “insider” character, except that for enough money, anyone could play. The SEC portion of the subscription + selling price could be used to finance other enforcement activities.
This presentation plus the Amazon paper make nice weekend reading/viewing.
[…] may want to compare the presentation of Andy Gross at Riak Core: Dynamo Building Blocks. Basically the same material but worded […]
Pingback by Vol. 15: Understanding Dynamo — with Andy Gross « Another Word For It — June 20, 2011 @ 3:32 pm