You have probably come across some of those large text books and noticed the index at the end. With a hard copy, it is nice to have such an index to navigate to the desired page quickly. I have recently published a very short book, and when it came to setting the index, the task seemed daunting even though the book is very short. The book doesn't have an index yet anyway.

If you have been following my articles, you will notice that I mainly write about Python and how it can help us in solving different issues in a simple manner. So let's see how we can set a book index using Python.

Without further ado, let's get started.

What Is a Book Index?

I'm pretty sure that most of you know what a book index is, but I just want to quickly clarify this concept.

A book index is simply a collection of words and/or phrases that are considered important to the book, along with their locations in the book. The index does not contain every word/phrase in the book. The reason for that is shown in the next section.

What Makes a Good Book Index?

What if you had an index through which you can find the location of each word or phrase in the book? Wouldn't that be considered as the index of choice? Wrong!

The index of choice, or what would be considered a good index, is that which points to the important words and phrases in the book. You might be questioning the reason for that. Let's take an example. Say that we have a book that consists only of the following sentence:

My book is short

What would happen if we try to index each word and phrase in that very short sentence, assuming that the location is the word number in the sentence? This is the index that we would have in this case:

my book is short: 1
my book is: 1
my book: 1
my: 1
short: 4
is short: 3
is: 3
book is short: 2
book is: 2
book: 2

From the example above, we can see that such an index would be larger than the book itself! So a good index would be one that contains the words and phrases considered important to the reader.

Setup

Natural Language Toolkit (NLTK)

In this tutorial, we will be using the Natural Language Toolkit (NLTK) library, which is used to work with human language data. As mentioned in the documentation, NLTK has been called “a wonderful tool for teaching, and working in, computational linguistics using Python,” and “an amazing library to play with natural language.”

I'm currently writing this tutorial from my Ubuntu machine, and the steps for installing NLTK in this section will be relevant to the Ubuntu Operating System. But don't worry, you can find the steps for installing NLTK on other Operating Systems on the NLTK website.

In order to install NLTK, I'm going to use pip. If you don't have pip installed already, you can use the following command in your terminal to install pip:

sudo easy_install3 pip

To make sure you have pip installed, type the following command:

pip --version

You should get something similar to the following:

pip 8.1.2 from /usr/local/lib/python3.5/dist-packages/pip-8.1.2-py3.5.egg (python 3.5)

Now, to install NLTK, simply run the following command in your terminal:

sudo pip install -U nltk

You can test the nltk installation by typing python, and then importing nltk in your terminal. If  you get ImportError: No module named nltk, this thread might help you out.

Test File

At this point, we need a test file (book) to use for creating a book index. I'll grab this book: The Rate of Change of the Rate of Change by the EFF. You can download the text file of the book from Dropbox. You can of course use any book of your choice; you just need something to experiment with in this tutorial.

Program

Let's start with the interesting part in this tutorial, the program that will help us form the book index. The first thing we want to do is find the word frequency in the book. I have shown how we can do that in another tutorial, but I want to show you how we can do that using the NLTK library.

This can be done as follows:

import nltk, collections
from nltk.collocations import *

frequencies = collections.Counter()
with open('bigd10.txt') as book:
    read_book = book.read()
words = nltk.word_tokenize(read_book)

for w in words:
    frequencies[w] += 1
    
print (frequencies) 

When you run the program, you will notice that we will have a very long list of words and their frequencies.

Before moving further, let's analyze the above code a bit. In the following line:

frequencies = collections.Counter()

We are trying to use the Counter() function in order to get the word frequencies in the book (how many times the word occurred in the book).

word_tokenize, on the other hand, splits the sentences into their constituent parts. Let's take a simple example to see how word_tokenize actually works:

from nltk.tokenize import word_tokenize
sentence = 'My name is Abder. I like Python. It\'s a pretty nice programming language'
print (word_tokenize(sentence))

The output of the above script is as follows:

['My', 'name', 'is', 'Abder', '.', 'I', 'like', 'Python', '.', 'It', "'s", 'a', 'pretty', 'nice', 'programming', 'language']

We then loop through the words and find the frequency of occurrence of each word.
What about phrases (combination of words)? Those are called collocations (a sequence of words that occur together often). An example of collocations is bigrams, that is a list of word pairs. Similar to that is trigrams (a combination of three words), and so forth (i.e. n-grams).

Let's say we want to extract the bigrams from our book. We can do that as follows:

bigram = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(words)
finder.apply_freq_filter(2)

The number 2 in the apply_freq_filter( ) function is telling us to ignore all bigrams that occur less than two times in the book.

If we want to find the 30 most occurring bigrams in the book, we can use the following code statement:

print (finder.nbest(bigram.pmi, 30))

Finally, if we would like to find the location, which is in our case where the word or phrase occurs in the book (not the page number), we can do the following:

print (read_book.index('computer'))
print (read_book.index('Assisted Reporting'))

The above statements seem to return the word location in a sentence, similar to what we have seen in our short sentence example at the beginning of the tutorial.

Putting It All Together

Let's put what we have learned in a single Python script. The following script will read our book and return the word frequencies, along with the 30 most occurring bigrams in the book, in addition to the location of a word and a phrase in the book:

import nltk, collections
from nltk.collocations import *

frequencies = collections.Counter()
with open('bigd10.txt') as book:
    read_book = book.read()
words = nltk.word_tokenize(read_book)

for w in words:
    frequencies[w] += 1

bigram = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(words)
finder.apply_freq_filter(2)
 
print ('Those are the words and their frequency of occurrence in the book:')
print (frequencies) 
print ('#################################################################')
print ('Those are the 30 most bigrams occurring in the book:')
print (finder.nbest(bigram.pmi, 30))
print (read_book.index('computer'))
print (read_book.index('Assisted Reporting'))

Conclusion

As we have seen in this tutorial, even a short text can be very daunting when it comes to building an index for that text. Also, an automated way for building the optimum index for the book might not be feasible enough.

We were able to solve this issue through using Python and the NLTK library, where we could pick the best words and phrases for the book index based on their frequency of occurrence (i.e. importance) in the book.

There is, of course, more you can do with NLTK, as shown in the library's documentation. You can also refer to the book Natural Language Processing with Python if you would like to go deeper in this library.