Running Spark GraphX algorithms on Library of Congress subject heading SKOS by Bob Ducharme.
From the post:
Well, one algorithm, but a very cool one.
Last month, in Spark and SPARQL; RDF Graphs and GraphX, I described how Apache Spark has emerged as a more efficient alternative to MapReduce for distributing computing jobs across clusters. I also described how Spark’s GraphX library lets you do this kind of computing on graph data structures and how I had some ideas for using it with RDF data. My goal was to use RDF technology on GraphX data and vice versa to demonstrate how they could help each other, and I demonstrated the former with a Scala program that output some GraphX data as RDF and then showed some SPARQL queries to run on that RDF.
Today I’m demonstrating the latter by reading in a well-known RDF dataset and executing GraphX’s Connected Components algorithm on it. This algorithm collects nodes into groupings that connect to each other but not to any other nodes. In classic Big Data scenarios, this helps applications perform tasks such as the identification of subnetworks of people within larger networks, giving clues about which products or cat videos to suggest to those people based on what their friends liked.
…
As so typically happens when you are reading one Bob DuCharme post, you see another that one requires reading!
Bob covers storing RDF in RDD (Resilient Distributed Dataset), the basic Spark data structure, creating the report on connected components and ends with heavily commented code for his program.
Sadly the “related” values assigned by the Library of Congress don’t say how or why the values are related, such as:
…
“Hiding places”
“Secrecy”
“Loneliness”
“Solitude”
“Privacy”
…
Related values could be useful in some cases but if I am searching on “privacy,” as in the sense of being free from government intrusion, then “solitude,” “loneliness,” and “hiding places” aren’t likely to be helpful.
That’s not a problem with Spark or SKOS, but a limitation of the data being provided.