Click here to listen to the Texan Translation segment on KUT Public Radio in which the research presented here was discussed (aired on 22 January, 2021).
Most frequent words
For the first analysis we’ll remove stopwords (see a definition of stopwords here). And then we will count the occurrences of each distinct word and see what’s most frequent!
Figure 1: Top 15 terms in each inaugural address by frequency.
Most important words by tf-idf
An advanced measure of the importance of words in documents is tf-idf. It is calculated using a formula that takes into account both each word’s raw frequency as well as the number of documents in the corpus in which it is used. This method is a way of finding out the most distinctive words in each text.
For example, in the context at hand, you might expect that each of the four speeches contains the words freedom or people. And while it is certainly interesting which president used them the most, these words aren’t really distinctive, simply because each president uses them on this occasion. But it’s instructive to look at the words that are both frequent and unique to each speech.
Figure 2: Top 15 terms in each inaugural address by tf-idf.