Apache Sqoop 1.4.x Installation

Apache Sqoop is a command-line interface application for transferring data between relational databases and Hadoop. It supports incremental loads of a single table or a free form SQL query as well as saved jobs which can be run multiple times to import updates made to a database since the last import. Imports can also be used to populate tables in Hive or HBase. Exports can be used to put data from Hadoop into a relational database. Sqoop got the name from sql+hadoop. Sqoop became a top-level Apache project in March 2012.

Pre Requirements

1) A machine with Ubuntu 14.04 LTS operating system.

2) Apache Hadoop pre installed (How to install Hadoop on Ubuntu 14.04)

3) MySQL Database pre installed (How to install MySQL Database on Ubuntu 14.04)

4) Apache Sqoop 1.4.6 Software (Download Here)

NOTE

Apache Sqoop comes with Hadoop compatible version. Check with your Hadoop version and download sqoop.

Sqoop 1.4.6 Installation on Ubuntu

Installation Steps

Step 1 - Update. Open a terminal (CTRL + ALT + T) and type the following sudo command. It is advisable to run this before installing any package, and necessary to run it to install the latest updates, even if you have not added or removed any Software Sources.

$ sudo apt-get update

Step 2 - Installing Java 7.

$ sudo apt-get install openjdk-7-jdk

Step 3 - Creating sqoop directory.

$ sudo mkdir /usr/local/sqoop

Step 4 - Change the ownership and permissions of the directory /usr/local/sqoop. Here 'hduser' is an Ubuntu username.

$ sudo chown -R hduser /usr/local/sqoop

$ sudo chmod -R 755 /usr/local/sqoop

Step 5 - Change the directory to /home/hduser/Desktop , In my case the downloaded sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz file is in /home/hduser/Desktop folder. For you it might be in /downloads folder check it.

$ cd /home/hduser/Desktop/

Step 6 - Untar the sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz file.

$ tar xzf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz

Step 7 - Move the contents of sqoop-1.4.6.bin__hadoop-2.0.4-alpha folder to /usr/local/hadoop

$ mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha/* /usr/local/sqoop

Step 8 - Edit $HOME/.bashrc file by adding the sqoop path.

$ sudo gedit $HOME/.bashrc

$HOME/.bashrc file. Add the following lines

export SQOOP_HOME=/usr/local/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

Step 9 - Reload your changed $HOME/.bashrc settings

$ source $HOME/.bashrc

Step 10 - Change the directory to /usr/local/sqoop/conf

$ cd $SQOOP_HOME/conf

Step 11 - Copy the default sqoop-env-template.sh to sqoop-env.sh

$ cp sqoop-env-template.sh sqoop-env.sh

Step 12 - Edit sqoop-env.sh file.

$ gedit sqoop-env.sh

Step 13 - Add the below lines to sqoop-env.sh file. Save and Close.

export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop

Step 14 - Copy the mysql-connector-java-5.1.28.jar to /sqoop/lib/ folder.

$ cp /usr/share/java/mysql-connector-java-5.1.28.jar /usr/local/sqoop/lib

Step 15 - Change the directory to /usr/local/sqoop/bin

$ cd $SQOOP_HOME/bin

Step 16 - Verify Installation

$ sqoop-version

OR

$ sqoop version

Please share this blog post and follow me for latest updates on