Posted on

Hadoop – What is Native Hadoop Library

NATIVE HADOOP LIBRARY

NATIVE HADOOP LIBRARY

Due to non-availability of Java implementations Hadoop has native implementations of certain components. The library in which these components are available is called the Native Hadoop Library. This library On the *nix platforms is named libhadoop.so.

NATIVE HADOOP LIBRARY

Read our post: Hadoop Commands Guide

Usage of Native Hadoop Library

  • Step 1

Start with reviewing the components.

  • Step 2

Review the supported platforms.

  • Step 3

You can download the hadoop version which will include a pre-built version of the native hadoop library, or build your own version of the native hadoop library. Which ever is easy for you to start with.

  • Step 4

Install the compression codec development packages (>zlib-1.2, >gzip-1.2).

  • Step 5

Check the runtime log files.

Components of Native Hadoop Library

Main component are followings:

  1. Compression Codecs (bzip2, lz4, snappy, zlib).
  2. Native IO utilities for HDFS Short-Circuit Local Reads and Centralized Cache Management in HDFS.
  3. CRC32 checksum implementation.

Read our post: WHAT IS HDFS?

How to Build Hadoop Library

Hadoop Library is built using the GNU autotools-chain. For exmaple: autoconf, autoheader, automake, autoscan, libtool. Hadoop Library is written in ANSI C so it is very easy to build this hadoop library using any common C-compiler and the GNU autotools-chain.

Prerequisite packages

  1. C compiler.
  2. GNU Autools Chain.
  3. zlib-development package.
  4. openssl-development package.

Use the following standard hadoop pom.xml file after you have installed all the prerequisite packages and build the native library:

After you have build the native library you can find the new library on below given path:

Use the following commands to ensure that the native hadoop library is on the library path via the system property:

Check the hadoop log files for your MapReduce tasks.

  • If everything is all right, then: DEBUG util.NativeCodeLoader will try to load the custom-built native-hadoop library.
  • If something goes wrong, then : INFO util.NativeCodeLoader will be unable to load native-hadoop library.

How to Check Native Hadoop Library is loaded correctly or not

We have a tool know as Native Library Checker. This tool checks whether library is loading correctly or not. Use the following code to launch Native Library Checker.