This article discuss about running mapreduce jobs using the apache tools called pig and hive.Before we can process the data we need to upload the files to be processed to HDFS\/S3.\u00a0 We recommend uploading to hdfs and keeping the important files in s3 for backup is a better practice. s3 is easily accessible from commandline using tools like s3cmd. HDFS is a failover cluster filesystem which provides enough protection to your data over instance failures.<\/p>\n
Mapreduce<\/p>\n
MapReduce is a programming model and an associated implementation for processing and generating large data sets. We can specify a map function that processes a key\/value pair to generate a set of intermediate key\/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key.<\/p>\n
The main steps hadoop takes to run a job are<\/p>\n
Hadoop Map\/Reduce is very powerful, but<\/p>\n
o\u00a0\u00a0 Requires a Java Programmer.<\/p>\n
o\u00a0\u00a0 Harder to write and also time consuming.<\/p>\n
o\u00a0\u00a0 Difficult to update frequently.<\/p>\n
A solution is to Run jobs using pig(Piglatin)\/hive(HiveQL).<\/p>\n
Pig<\/strong><\/p>\n \u2022 An engine for executing programs on top of Hadoop<\/p>\n \u2022 It provides a language, Pig Latin, to specify these programs<\/p>\n Pig has Two main parts:<\/p>\n \u2013 A high level language to express data analysis<\/p>\n \u2013 Compiler to generate mapreduce programs (which can run on top of Hadoop)<\/p>\n Pig Latin is the name of the language with which Pig scripts are written. Pig also provides an interactive shell for executing simple commands, called Grunt. Pig Latin is a high level language. Pig runs on top of Hadoop. It collect the data for processing from Hadoop HDFS filesystem and Submit the jobs to the Hadoop mapreduce system.<\/p>\n A sample mapreduce job (like a Hello World program) using pig is given below<\/p>\n It is assumed that you are on one of the machines which is a part of a hadoop cluster having NameNode\/DataNode as well as JobTracker\/TaskTracker setup.<\/p>\n We will be executing piglatin commands using grunt shell. Switch to hadoop user first .<\/p>\n Consider we have a file \u2018users\u2019 on our local filesystem which contain data to be processed.First we have to upload it to hdfs. Then<\/p>\n # pig -x mapreduce<\/p>\n this command will take you to grunt shell. Pig Latin statements are generally<\/p>\n organized in the following manner:<\/p>\n A LOAD statement reads data from the file system.Then we process the data.And writes output to the file system using STORE statement. A DUMP statement displays output to the screen.<\/p>\n grunt> Users = load ‘users’ as (name, age);<\/p>\n grunt> Fltrd = filter Users by age >= 18 and age <= 25;<\/p>\n grunt> Pages = load ‘pages’ as (user, url);<\/p>\n grunt> Jnd = join Fltrd by name, Pages by user;<\/p>\n grunt> Grpd = group Jnd by url;<\/p>\n grunt> Smmd = foreach Grpd generate group, COUNT(Jnd) as clicks;<\/p>\n grunt> Srtd = order Smmd by clicks desc;<\/p>\n grunt> Top5 = limit Srtd 5;<\/p>\n grunt> store Top5 into ‘top5sites’;<\/p>\n We can also view the progress of the job through the web interface http:\/\/<ipaddress of jobtracker machine>:50030.<\/p>\n Tools like PigPen (an eclipse plugin) are available\u00a0 that helps us create pig-scripts, test them using the example generator and then submit them to a hadoop cluster.<\/p>\n There is another tool called oozie – Oozie is a server based Workflow Engine <\/em>specialized in running workflow jobs with actions that run Hadoop Map\/Reduce and Pig jobs.<\/p>\n Pig tasks can be modeled as a workflow in oozie. These are deployed to the Oozie server using a command line utility. Once deployed, the workflows can be started and manipulated as necessary using the same utility. Once the workflow is started Oozie will run through each flow.. The web console for Oozie server can be used to monitor the progress of various workflow jobs being managed by the server.<\/p>\n Hive<\/strong><\/p>\n <\/p>\n Pig, was causing some slowdowns at Facebook company as it needed training to bring business intelligence users up to speed. So the development team decided to write Hive which has an SQL like syntax.<\/p>\n Apache Hive is a data warehouse infrastructure built on top of Apache Hadoop. It provides tools for querying and analysis of large data sets stored in Hadoop files. Hive defines a simple SQL-like query language, called HiveQL, that enables users familiar with SQL to query the data. Also it allows custom mappers and reducers to perform more sophisticated analysis that may not be supported by the built-in capabilities of the language.<\/p>\n Some of the queries in HiveQL are given below, which is very similar to the SQL.<\/p>\n # show tables;<\/p>\n # describe <tablename>;<\/p>\n # SELECT * FROM <tablename> LIMIT 10;<\/p>\n #\u00a0 CREATE TABLE table_name<\/p>\n #\u00a0 ALTER TABLE table_name RENAME TO new_table_name<\/p>\n #\u00a0 DROP TABLE table_name<\/p>\n NoSQL databases like Cassandra provide support for hadoop. Cassandra supports running Hadoop MapReduce jobs against the Cassandra cluster. With proper cluster configuration, MapReduce jobs can retrieve data from Cassandra and then output results either back into Cassandra, or into a file system.<\/p>\n","protected":false},"excerpt":{"rendered":" This article discuss about running mapreduce jobs using the apache tools called pig and hive.Before we can process the data we need to upload the files to be processed to HDFS\/S3.\u00a0 We recommend uploading to hdfs and keeping the important files in s3 for backup is a better practice. s3 is easily accessible from commandline […]<\/p>\n","protected":false},"author":17,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[34,51,3],"tags":[434,435,442,88,446,87,443,212,447,440,437,441,89,90,94,93,97,444,448,438,436,95,92,96,197,433,136,445,439],"class_list":["post-988","post","type-post","status-publish","format-standard","hentry","category-amazon-ec2","category-amazon-web-services","category-cloud-computing","tag-apache-hadoop","tag-apache-hive","tag-apache-tools","tag-aws-ec2","tag-business-intelligence","tag-cassandra","tag-commandline-using-tools","tag-company-facebook","tag-data-warehouse","tag-data-warehouse-infrastructure","tag-data-intensive-computing","tag-grunt-store-top5","tag-hadoop","tag-hadoop-cluster","tag-hdfs","tag-hive","tag-hiveql","tag-java","tag-java-programmer","tag-jumbune","tag-mapr","tag-mapreduce","tag-pig","tag-piglatin","tag-social-media-networking","tag-sql","tag-technologyinternet","tag-web-console","tag-web-interface"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t