Apache Pig for batch data analysis over Hadoop

Datetime:2016-08-23 02:38:25          Topic: Apache Pig  Data Analysis           Share

In these days I’m playing with Apache Pig for running data analysis over Apache Hadoop . Below a sample wordcloud generated from the top word count of nouns of the Italian translation of the Bible

la-sacra-bibbia-frequenza-parole

Wordcount (pig script):

a = load '/user/matteo/book.txt';
b = foreach a {
	line = LOWER(REPLACE((chararray)$0, '[!?\\.»«:;,\']', ' '));
    generate flatten(TOKENIZE(line)) as word;
}
c = group b by word;
d = foreach c generate group, COUNT(b) as cnt;
d_ordered = ORDER d BY cnt DESC;
store d_ordered into '/user/matteo/book-wordcount';

Wordcloud ( R script)

library(wordcloud)
p = read.table(file="book.txt")
png("/home/matteo/la-sacra-bibbia-frequenza-parole.png", width=900, height=900)
wordcloud(p$V1, p$V2, scale=c(8,.3),min.freq=2,max.words=200, random.order=T, rot.per=.15)
dev.off()




About List