From the webpage:
Sparkey is an extremely simple persistent key-value store. You could think of it as a read-only hashtable on disk and you wouldn’t be far off. It is designed and optimized for some server side usecases at Spotify but it is written to be completely generic and makes no assumptions about what kind of data is stored.
Some key characteristics:
- Supports data sizes up to 2^63 – 1 bytes.
- Supports iteration, get, put, delete
- Optimized for bulk writes.
- Immutable hash table.
- Any amount of concurrent independent readers.
- Only allows one writer at a time per storage unit.
- Cross platform storage file.
- Low overhead per entry.
- Constant read startup cost
- Low number of disk seeks per read
- Support for block level compression.
- Data agnostic, it just maps byte arrays to byte arrays.
What it’s not:
- It’s not a distributed key value store – it’s just a hash table on disk.
- It’s not a compacted data store, but that can be implemented on top of it, if needed.
- It’s not robust against data corruption.
The usecase we have for it at Spotify is serving data that rarely gets updated to users or other services. The fast and efficient bulk writes makes it feasible to periodically rebuild the data, and the fast random access reads makes it suitable for high throughput low latency services. For some services we have been able to saturate network interfaces while keeping cpu usage really low.
If you are looking for a very high-performance key-value store with little to no frills, your search may be over.
Originating with Spotify and being able to saturate network interfaces bodes well for those needing pure performance.
I first saw this in Nat Torkington’s Four short links: 10 September 2013.