Another Word For It Patrick Durusau on Topic Maps and Semantic Diversity

June 2, 2013

MapReduce with Python and mrjob on Amazon EMR

Filed under: Amazon EMR,MapReduce,Natural Language Processing,Python — Patrick Durusau @ 10:59 am

MapReduce with Python and mrjob on Amazon EMR by Sujit Pal.

From the post:

I’ve been doing the Introduction to Data Science course on Coursera, and one of the assignments involved writing and running some Pig scripts on Amazon Elastic Map Reduce (EMR). I’ve used EMR in the past, but have avoided it ever since I got burned pretty badly for leaving it on. Being required to use it was a good thing, since I got over the inertia and also saw how much nicer the user interface had become since I last saw it.

I was doing another (this time Python based) project for the same class, and figured it would be educational to figure out how to run Python code on EMR. From a quick search on the Internet, mrjob from Yelp appeared to be the one to use on EMR, so I wrote my code using mrjob.

The code reads an input file of sentences, and builds up trigram, bigram and unigram counts of the words in the sentences. It also normalizes the text, lowercasing, replacing numbers and stopwords with placeholder tokens, and Porter stemming the remaining words. Heres the code, as you can see, its fairly straightforward:

Knowing how to exit and confirm exit from a cloud service are the first things to learn about a cloud system.

No Comments

No comments yet.

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress