Archive for the ‘OAI’ Category

BASE – Bielefeld Academic Search Engine

Wednesday, June 10th, 2015

BASE – Bielefeld Academic Search Engine

From the post:

BASE is one of the world’s most voluminous search engines especially for academic open access web resources. BASE is operated by Bielefeld University Library.

As the open access movement grows and prospers, more and more repository servers come into being which use the “Open Archives Initiative Protocol for Metadata Harvesting” (OAI-PMH) for providing their contents. BASE collects, normalises, and indexes these data. BASE provides more than 70 million documents from more than 3,000 sources. You can access the full texts of about 70% of the indexed documents. The index is continuously enhanced by integrating further OAI sources as well as local sources. Our OAI-PMH Blog communicates information related to harvesting and aggregating activities performed for BASE.

One feature of the search interface that is particularly appealing is the ability to “boost” open access documents, request verbatim search, request additional word forms, and to invoke multilingual synonyms (Eurovoc Thesaurus).

I first saw this in a tweet by Amanda French

BASE indexed 50 million OAI-records

Wednesday, August 28th, 2013

BASE indexed 50 million OAI-records by Sarah Dister.

From the post:

BASE, a search engine for academic open access web resources, has indexed more than 50,000,000 OAI-records. The records are provided by about 2,700 repositories among which many are related to agriculture.

BASE is a multi-disciplinary search engine for academically relevant OAI-Sources worldwide, which was created and developed by Bielefeld University Library.

Take a few minutes (or longer) to explore BASE.

It is a remarkable resource. For example, users can invoke the Eurovoc Thesaurus as part of their search query.

ResourceSync Framework Specification

Monday, February 11th, 2013

NISO and OAI Release Draft for Comments of ResourceSync Framework Specification

From the post:

NISO and the Open Archives Initiative (OAI) announce the release of a beta draft for comments of the ResourceSync Framework Specification for the web consisting of various capabilities that allow third-party systems to remain synchronized with a server’s evolving resources. The ResourceSync joint project, funded with support from the Alfred P. Sloan Foundation and the JISC, was initiated to develop a new open standard on the real-time synchronization of Web resources.

Increasingly, large-scale digital collections are available from multiple hosting locations, are cached at multiple servers, and leveraged by several services. This proliferation of replicated copies of works or data on the Internet has created an increasingly challenging problem of keeping the repositories’ holdings and the services that leverage them up-to-date and accurate. The ResourceSync draft specification introduces a range of easy to implement capabilities that a server may support in order to enable remote systems to remain more tightly in step with its evolving resources.

The draft specification is available on the OAI website at: Comments on the draft can be posted on the public discussion forum at:!forum/resourcesync.

For more on the ResourceSync Framework, see the article in the January/February 2013 issue of D-Lib.

For those interested in synchronization of resources. Say from or to topic maps.

Similarity and Duplicate Detection System for an OAI Compliant Federated Digital Library

Tuesday, September 28th, 2010

Similarity and Duplicate Detection System for an OAI Compliant Federated Digital Library Authors: Haseebulla M. Khan, Kurt Maly and Mohammad Zubair Keywords: OAI – duplicate detection – digital library – federation service


The Open Archives Initiative (OAI) is making feasible to build high level services such as a federated search service that harvests metadata from different data providers using the OAI protocol for metadata harvesting (OAI-PMH) and provides a unified search interface. There are numerous challenges to build and maintain a federation service, and one of them is managing duplicates. Detecting exact duplicates where two records have identical set of metadata fields is straight-forward. The problem arises when two or more records differ slightly due to data entry errors, for example. Many duplicate detection algorithms exist, but are computationally intensive for large federated digital library. In this paper, we propose an efficient duplication detection algorithm for a large federated digital library like Arc.

The authors discovered that title weight was more important than author weight in the discovery of duplicates. Working with a subset of 73 archives with 465,440 records. Would be interesting to apply this insight to a resource like WorldCat, where duplicates are a noticeable problem.