Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

November 10, 2012

LDBC: Linked Data Benchmark Council [I count 15 existing Graph/RDF benchmarks. You?]

Filed under: Benchmarks,Graphs,Linked Data — Patrick Durusau @ 11:01 am

LDBC: Linked Data Benchmark Council

From the webpage:

In the last years we have seen an explosion of massive amounts of graph shaped data coming from a variery of applications that are related to social networks like facebook, twitter, blogs and other on-line media and telecommunication networks. Furthermore, the W3C linking open data initiative has boosted the publication and interlinkage of a large number of datasets on the semantic web resulting to the Linked Data Cloud. These datasets with billions of RDF triples such as Wikipedia, U.S. Census bureau, CIA World Factbook, DBPedia, and government sites have been created and published online. Moreover, numerous datasets and vocabularies from e-science are published nowadays as RDF graphs most notably in life and earth sciences, astronomy in order to facilitate community annota- tion and interlinkage of both scientific and scholarly data of interest.

Technology and bandwidth now provide the opportunities for compiling, publishing and sharing massive Linked Data datasets. A significant number of commercial semantic repositories (RDF databases with reasoner and query-engine) which are the cornerstone of the Semantic Web exist.

Neverthless at the present time,

  • there is no comprehensive suite of benchmarks that encourage the advancement of technology by providing both academia and industry with clear targets for performance and functionality and
  • no independent authority for developing benchmarks and verifying the results of those engines. The same holds for the emerging field of noSQL graph databases, which share with RDF a graph data model and pattern- and pathoriented query languages.

The Linked Data Benchmark Council (LDBC) project aims to provide a solution to this problem by making insightful the critical properties of graph and RDF data management technology, and stimulating progress through compettion. This is timely and urgent since non-relational data management is emerging as a critical need for the new data economy based on large, distributed, heterogeneous, and complexly structured data sets. This new data management paradigm also provides an opportunity for research results to impact young innovative companies working on RDF and graph data management to start playing a significant role in this new data economy.

This announcement puzzled me because I know I have seen (and written about) graph benchmarks.

A quick search with a popular search engine turned up three of the better known graph benchmarks (in the first ten “hits”):

  1. BHOSLIB: Benchmarks with Hidden Optimum Solutions for Graph Problems (Maximum Clique, Maximum Independent Set, Minimum Vertex Cover and Vertex Coloring) —— Hiding Exact Solutions in Random Graphs by Ke XU, Beijing University of Aeronautics and Astronautics.
  2. HPC Graph Analysis From the homepage:

    We maintain a parallel graph theory benchmark that solves multiple graph analysis kernels on small-world networks. An early version of the benchmark was part of the DARPA High Productivity Computing Systems (HPCS) Compact Application (SSCA) suite. The benchmark performance across current HPC systems can be compared using a single score called TrEPS (Traversed Edges Per Second).

  3. Graph 500. From their specifications page:

    There are five problem classes defined by their input size:

    toy 17GB or around 1010 bytes, which we also call level 10,

    mini 140GB (1011 bytes, level 11),

    small 1TB (1012 bytes, level 12),

    medium 17TB (1013 bytes, level 13),

    large 140TB (1014 bytes, level 14), and

    huge 1.1PB (1015 bytes, level 15).

On RDF graphs in particular, the W3C wiki page: RDF Store Benchmarking, has a host of resources, including twelve (12) benchmarks for RDF Stores:

For a total of fifteen (15) graph/RDF benchmarks, discoverable in just a few minutes.

Given the current financial difficulties at the EU, duplicating research already performed/underway by others is a poor investment.

PS: Pass my name along to anyone you know in the EU research approval committees. I would be happy to do new/not-new evaluations of proposals on a contract basis.

PPS: As always, if you know of other graph/RDF benchmarks, I would be happy to add them. I deliberately did not attempt an exhaustive survey of graph or RDF benchmarks. If you or your company are interested in such a survey, ping me.

2 Comments

  1. Dear Patrick,

    I agree that the wording on the LDBC landing page, might be taken to suggest that there are no RDF and graph benchmarks around. This is, as you show, not the case; and will be corrected.

    This text was copied and pasted without the proper other context (it comes from the first sentences of the project proposal introduction , which elaborates much more).

    So, we are improving the website thanks to your comments.

    Rest assured that the project proposal indeed mentions many more benchmark efforts; including 5 more graph benchmarks and 8 more RDF benchmarks; so your tally could be 23. This is not our point, though.

    The mission of the LDBC can be compared to that of the Transaction Processing Council (TPC) that Jim Gray founded (www.tpc.org). The main difference here is between the rather academic project of defining a benchmark and the industrial application of those benchmarks. LDBC aims for the latter and the EU project aims to make the transfer from academic to industrial.

    LDBC will create a body in which industry (i.e. vendors of RDF and graph database systems) agree on relevant benchmarks and benchmark practices; and will publish official benchmark results.

    “agreeing on benchmark practices” means agreeing on the exact rules and metrics with which products can be compared. Without such rules, it is very easy to skew any benchmark result in one’s favor; e.g. by precomputing (partial) answers; by implementing benchmark-special functionalities, by being not open about hot or cold runs; by comparing results on wholly different hardware (with wholly different price-tags). There are many ways in which one can game the result.

    “agreeing on metrics” is important as without clear metrics, it is easy to pick the benchmark observations or statistics that favor your algorithm/system/product (conveniently forgetting about other metrics relevant for the benchmark on which the performance maybe to so favorable — often systems must make trade-offs, so a win on one metric can become a loss on another; see e.g. the difference between OLTP and OLAP workloads). What distinguishes TPC metrics from all the benchmarks in your list of 15 is that it includes a notion of score/$, taking into account hardware+software+maintenance cost aspects in the results.

    The purpose of LDBC is thus to create credible benchmarks with credible metrics; and a body of industry consensus so that we will see multiple relevant systems report comparable results on the same tests.

    This goes beyond just defining a benchmark, but also about enforcing fairness in reaching benchmark results. This process will involve the use of independent auditors to check each claimed benchmark result because it becomes official. It will also involve the protection of the LDBC rules in the marketing statements of companies quoting LDBC results.

    The usefulness of this approach was learned in relational database world (the TPC). Thus, LDBC is going to take lessons taken from the relational database community and try to enrich benchmark culture in the RDF and graph database areas. We think this will be very useful for e.g. IT practitioners wondering whether graph or RDF database technology is to be chosen over the well-entrenched relational systems, and if so, which system to select. It will also help invigorate technical progress among vendors of competing technologies; in their unavoidable competition for the highest benchmark scores.

    The LDBC EU project also has academic participation, essentially to kick-start it by helping in selecting/defining an initial set of benchmarks. Even though in RDF there are already many benchmarks, generally the RDF community does acknowledges flaws in them (e.g. they often manipulate essentially relational data, do not represent a meaningful workload, fail to test reasoning performance, etc). On the graph database side, there is the challenge that “graph database system” is a rather wide and vague concept (not to mention the lack of a standard data model and query language). So there are plenty technical challenges there for the academics to work on.

    But rest assured: the LDBC project is taking into account previous work and it is likely that LDBC endorsed benchmarks will be formulated by taking components of the above mentioned previous efforts as a basis.

    Thanks again for your comment, which we are as I mentioned using to change the text on the landing page of http://www.ldbc.eu

    best,

    Peter Boncz
    researcher at CWI and Vrije Universiteit Amsterdam
    scientific director LDBC

    PS
    I can confirm that this message is being read by EU research officers, who decide who to invite in “new/not new” evaluations — so they do know your name, by now 😉 My impression of their decision making is that it is rather thorough.

    Comment by peterboncz — November 15, 2012 @ 12:54 pm

  2. Dear Peter,

    Thanks for the detailed response.

    The project you describe is one that I will be following with interest!

    I have added the LDBC homepage to my RSS feed.

    Will there be a mailing list for draft documents?

    Looking forward to hearing more from your project!

    Patrick

    Comment by Patrick Durusau — November 15, 2012 @ 2:04 pm

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress