NATIVE HADOOP LIBRARY
Due to non-availability of Java implementations Hadoop has native implementations of certain components. The library in which these components are available is called the Native Hadoop Library. This library On the *nix platforms is named libhadoop.so.
Read our post: Hadoop Commands Guide
Usage of Native Hadoop Library
- Step 1
Start with reviewing the components.
- Step 2
Review the supported platforms.
- Step 3
You can download the hadoop version which will include a pre-built version of the native hadoop library, or build your own version of the native hadoop library. Which ever is easy for you to start with.
- Step 4
Install the compression codec development packages (>zlib-1.2, >gzip-1.2).
- Step 5
Check the runtime log files.
Components of Native Hadoop Library
Main component are followings:
- Compression Codecs (bzip2, lz4, snappy, zlib).
- Native IO utilities for HDFS Short-Circuit Local Reads and Centralized Cache Management in HDFS.
- CRC32 checksum implementation.
Read our post: WHAT IS HDFS?
How to Build Hadoop Library
Hadoop Library is built using the GNU autotools-chain. For exmaple: autoconf, autoheader, automake, autoscan, libtool. Hadoop Library is written in ANSI C so it is very easy to build this hadoop library using any common C-compiler and the GNU autotools-chain.
- C compiler.
- GNU Autools Chain.
- zlib-development package.
- openssl-development package.
Use the following standard hadoop pom.xml file after you have installed all the prerequisite packages and build the native library:
$ mvn package -Pdist,native -DskipTests -Dtar
After you have build the native library you can find the new library on below given path:
Use the following commands to ensure that the native hadoop library is on the library path via the system property:
Check the hadoop log files for your MapReduce tasks.
- If everything is all right, then: DEBUG util.NativeCodeLoader will try to load the custom-built native-hadoop library.
- If something goes wrong, then : INFO util.NativeCodeLoader will be unable to load native-hadoop library.
How to Check Native Hadoop Library is loaded correctly or not
We have a tool know as Native Library Checker. This tool checks whether library is loading correctly or not. Use the following code to launch Native Library Checker.
$ hadoop checknative -a
14/12/06 01:30:45 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
14/12/06 01:30:45 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop: true /home/ozawa/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib/x86_64-linux-gnu/libz.so.1
snappy: true /usr/lib/libsnappy.so.1
lz4: true revision:99