Benchmarking Honesty by David Rosenthal.
From the post:
Recently, someone brought to my attention a blog post that benchmarks FoundationDB and another responding to the benchmark itself. I’ll weigh in: I think this benchmark is unfair because it gives people too good an impression of FoundationDB’s performance. In the benchmark, 100,000 items are loaded into each database/storage engine in both sequential and random patterns. In the case of FoundationDB and other sophisticated systems like SQL Server, you can see that the performance of random and sequential writes are virtually the same; this points to the problem. In the case of FoundationDB, an “absorption” mechanism is able to cope with bursts of writes (on the order of a minute or two, usually) without actually updating the real data structures holding the data (i.e. only persisting a log to disk, and making changes available to read from RAM). Hence, the published test results are giving FoundationDB an unfair advantage. I think that you will find that if you sustain this workload for a longer time, like in real-world usages, FoundationDB might be significantly slower.
If you don’t recognize the name, David Rosenthal is the co-founder and CEO of FoundationDB.
What?
A CEO saying a benchmark favorable to his product is “unfair?”
Odd as it may sound, I think there is an honest CEO on the loose.
Statistically speaking, it had to happen eventually. 😉
Seriously, high marks to David Rosenthal. We need more CEOs, engineers and presenters with a sense of honesty.