nutch

tutorial

Requirements

Java 1.4.x, either from Sun or IBM on Linux is preferred. Set NUTCH_JAVA_HOME to the root of your JVM installation.
Apache's Tomcat 4.x.
On Win32, cygwin, for shell support. (If you plan to use CVS on Win32, be sure to select the cvs and openssh packages when you install, in the "Devel" and "Net" categories, respectively.)
Up to a gigabyte of free disk space, a high-speed connection, and an hour or so.

Getting Started

First, you need to get a copy of the Nutch code. You can download a release from http://www.nutch.org/release/. Unpack the release and connect to its top-level directory. Or, check out the latest source code from CVS and build it with Ant.

Try the following command:

bin/nutch

This will display the documentation for the Nutch command script.

Concepts

Nutch data is of two types:

The web database. This contains information about every page known to Nutch, and about links between those pages.
A set of segments. Each segment is a set of pages that are fetched and indexed as a unit. Segment data consists of the following types:

a fetchlist is a file that names a set of pages to be fetched
the fetcher output is a set of files containing the fetched pages
the index is a Lucene-format index of the fetcher output.

In the following examples we will keep our web database in a directory named db and our segments in a directory named segments:

mkdir db
mkdir segments

Boostrapping the Web Database

The admin tool is used to create a new, empty database:

bin/nutch admin db -create

The injector adds urls into the database. Let's inject URLs from the DMOZ Open Directory. First we must download and uncompress the file listing all of the DMOZ pages. (This is a 200+Mb file, so this will take a few minutes.)

wget http://rdf.dmoz.org/rdf/content.rdf.u8.gz
gunzip content.rdf.u8.gz

Next we inject a random subset of these pages into the web database. (We use a random subset so that everyone who runs this tutorial doesn't hammer the same sites.) DMOZ contains around three million URLs. We inject one out of every 3000, so that we end up with around 1000 URLs:

bin/nutch inject db -dmozfile content.rdf.u8 -subset 3000

This also takes a few minutes, as it must parse the full file.

Now we have a web database with around 1000 as-yet unfetched URLs in it.

Fetching

To fetch, we first generate a fetchlist from the database:

bin/nutch generate db segments

This generates a fetchlist for all of the pages due to be fetched. The fetchlist is placed in a newly created segment directory. The segment directory is named by the time it's created. We save the name of this segment in the shell variable >s1:

s1=`ls -d segments/2* | tail -1`
echo $s1

Now we run the fetcher on this segment with:

bin/nutch fetch $s1

When this is complete, we update the database with the results of the fetch:

bin/nutch updatedb db $s1

Now the database has entries for all of the pages referenced by the initial set.

Next we run five iterations of link analysis on the database in order to prioritize which pages to next fetch:

bin/nutch analyze db 5

Now we fetch a new segment with the top-scoring 1000 pages:

bin/nutch generate db segments -topN 1000
s2=`ls -d segments/2* | tail -1`
echo $s2

bin/nutch fetch $s2
bin/nutch updatedb db $s2
bin/nutch analyze db 2

Let's fetch one more round:

bin/nutch generate db segments -topN 1000
s3=`ls -d segments/2* | tail -1`
echo $s3

bin/nutch fetch $s3
bin/nutch updatedb db $s3
bin/nutch analyze db 2

By this point we've fetched a few thousand pages. Let's index them!

Indexing

To index each segment we use the index command, as follows:

bin/nutch index $s1
bin/nutch index $s2
bin/nutch index $s3

Then, before we can search a set of segments, we need to delete duplicate pages. This is done with:

bin/nutch dedup segments dedup.tmp

Now we're ready to search!

Searching

To search you need to put the nutch war file into your servlet container. (If instead of downloading a Nutch release you checked the sources out of CVS, then you'll first need to build the war file, with the command ant war.)

Assuming you've unpacked Tomcat as ~/local/tomcat, then the Nutch war file may be installed with the commands:

rm -rf ~/local/tomcat/webapps/ROOT*
cp nutch*.war ~/local/tomcat/webapps/ROOT.war

The webapp finds its indexes in ./segments, relative to where you start Tomcat, so, don't change directories, and give the command:

~/local/tomcat/bin/catalina.sh start

Then visit http://localhost:8080/ and have fun!