Text as Data Case Study: Presidential Inauguration Addresses

Possible questions of interest: * Trends over time: word length, sentences, paragraphs, or special words like “America”. * Vocabulary size, average sentence length * Topics over time, such as religion or patriotism

R setup

require(tm)

Loading required package: tm
Loading required package: NLP

require(topicmodels)

Loading required package: topicmodels

require(stringr)

Loading required package: stringr

require(tidyverse)

Loading required package: tidyverse
Loading tidyverse: ggplot2
Loading tidyverse: tibble
Loading tidyverse: tidyr
Loading tidyverse: readr
Loading tidyverse: purrr
Loading tidyverse: dplyr
Conflicts with tidy packages ----------------------------------------------------------------------
annotate(): ggplot2, NLP
filter():   dplyr, stats
lag():      dplyr, stats

require(lubridate)  # date handling

Loading required package: lubridate

Attaching package: ‘lubridate’

The following object is masked from ‘package:base’:

    date

US Presidential Inauguration Addresses

There have been 58 presidential inauguration addresses. These are available on the web site in the tar file \({\tt inauguration.tgz}\). Quite often in text analytics, each document will be kept in a separate file. You will need to join these files together to build an analysis.

A quick peek at one of these text files shows they are in the following simple format:

• First line: last name of president
• Second line: city where address was given
• Third line: date, as in April 30, 1789

Following these three lines, each of the following lines indicates a paragraph, which may be composed of one or more sentences. Blank lines separate the paragraphs.

path <- "~/data/text/inauguration"    # change to suit your computer; download from web site
files <- list.files(path)
files

 [1] "_source.txt"    "01_wash.txt"    "02_wash.txt"    "03_adams.txt"   "04_jeff.txt"   
 [6] "05_jeff.txt"    "06_mad.txt"     "07_mad.txt"     "08_mon.txt"     "09_mon.txt"    
[11] "10_adams.txt"   "11_jack.txt"    "12_jack.txt"    "13_vbur.txt"    "14_harr.txt"   
[16] "15_polk.txt"    "16_tay.txt"     "17_pierce.txt"  "18_buck.txt"    "19_linc.txt"   
[21] "20_linc.txt"    "21_grant.txt"   "22_grant.txt"   "23_hayes.txt"   "24_gar.txt"    
[26] "25_clev.txt"    "26_clev.txt"    "27_harr.txt"    "28_mack.txt"    "29_mack.txt"   
[31] "30_tr.txt"      "31_taft.txt"    "32_wilson.txt"  "33_wilson.txt"  "34_harding.txt"
[36] "35_cool.txt"    "36_hoover.txt"  "37_fdr.txt"     "38_fdr.txt"     "39_fdr.txt"    
[41] "40_fdr.txt"     "41_truman.txt"  "42_ike.txt"     "43_ike.txt"     "44_kenn.txt"   
[46] "45_lbj.txt"     "46_nixon.txt"   "47_nixon.txt"   "48_carter.txt"  "49_reagan.txt" 
[51] "50_reagan.txt"  "51_bush.txt"    "52_clinton.txt" "53_clinton.txt" "54_bush.txt"   
[56] "55_bush.txt"    "56_obama.txt"   "57_obama.txt"   "58_trump.txt"  

first_address <- read_lines(file.path(path,"01_wash.txt"))  # file.path joins path elements

length(first_address) # lines

[1] 16

first_address[1:3]

[1] "Washington"     "New York"       "April 30, 1789"

first_address[4:6]

[1] "Fellow-Citizens of the Senate and of the House of Representatives:"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
[2] ""                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
[3] "Among the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the 14th day of the present month. On the one hand, I was summoned by my Country, whose voice I can never hear but with veneration and love, from a retreat which I had chosen with the fondest predilection, and, in my flattering hopes, with an immutable decision, as the asylum of my declining years--a retreat which was rendered every day more necessary as well as more dear to me by the addition of habit to inclination, and of frequent interruptions in my health to the gradual waste committed on it by time. On the other hand, the magnitude and difficulty of the trust to which the voice of my country called me, being sufficient to awaken in the wisest and most experienced of her citizens a distrustful scrutiny into his qualifications, could not but overwhelm with despondence one who (inheriting inferior endowments from nature and unpracticed in the duties of civil administration) ought to be peculiarly conscious of his own deficiencies. In this conflict of emotions all I dare aver is that it has been my faithful study to collect my duty from a just appreciation of every circumstance by which it might be affected. All I dare hope is that if, in executing this task, I have been too much swayed by a grateful remembrance of former instances, or by an affectionate sensibility to this transcendent proof of the confidence of my fellow-citizens, and have thence too little consulted my incapacity as well as disinclination for the weighty and untried cares before me, my error will be palliated by the motives which mislead me, and its consequences be judged by my country with some share of the partiality in which they originated."

Building a corpus (with metadata)

Remove the blank lines. A small, utility function will be handy.

remove_blank_lines <- function(text) {
    n <- unname(sapply(text,str_count))
    return(text[0<n])
}

After removing the blank lines, we essentially have paragraphs.

first_address <- remove_blank_lines(first_address)
length(first_address)

[1] 10

Now read all of the inaugural addresses into a list, skipping the first which isn’t relevant. (You can use the DirSource method from tm to read all of the files into a corpus instead.)

addresses <- lapply(files[-1], 
                    function(file) 
                        remove_blank_lines(
                            read_lines(file.path(path,file))
                        ))
length(addresses)

[1] 58

The number of paragraphs in an address has grown over time, but maybe not the number of words!

n.paragraphs <- sapply(addresses, length)-3   # 3 prefix lines
plot( 1:58, n.paragraphs, type='b', xlab="Address")

Now combine these files into a data frame with the text and metadata (president’s name, date, and whatever other variable we want to keep track of for each address). Let’s figure out how to handle one document, then map a function that does these chores over the corpus. Getting the name of the president and location of address are easy: these are just the first two lines.

typeof(addresses)

[1] "list"

To handle the date, I will use the lubridate package; it is quite nice for handling dates in a variety of formats.

addresses[[1]][3]   # [[ ]] indexes a list element

[1] "April 30, 1789"

mdy(addresses[[1]][3])   # mdy from lubridate

[1] "1789-04-30"

I’ll also count the number of paragraphs, so I will add a separator (delimiter).

str_c(addresses[[1]][-(1:3)], collapse=' <paragraph> ')  # put in a paragraph delimiter

[1] "Fellow-Citizens of the Senate and of the House of Representatives: <paragraph> Among the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the 14th day of the present month. On the one hand, I was summoned by my Country, whose voice I can never hear but with veneration and love, from a retreat which I had chosen with the fondest predilection, and, in my flattering hopes, with an immutable decision, as the asylum of my declining years--a retreat which was rendered every day more necessary as well as more dear to me by the addition of habit to inclination, and of frequent interruptions in my health to the gradual waste committed on it by time. On the other hand, the magnitude and difficulty of the trust to which the voice of my country called me, being sufficient to awaken in the wisest and most experienced of her citizens a distrustful scrutiny into his qualifications, could not but overwhelm with despondence one who (inheriting inferior endowments from nature and unpracticed in the duties of civil administration) ought to be peculiarly conscious of his own deficiencies. In this conflict of emotions all I dare aver is that it has been my faithful study to collect my duty from a just appreciation of every circumstance by which it might be affected. All I dare hope is that if, in executing this task, I have been too much swayed by a grateful remembrance of former instances, or by an affectionate sensibility to this transcendent proof of the confidence of my fellow-citizens, and have thence too little consulted my incapacity as well as disinclination for the weighty and untried cares before me, my error will be palliated by the motives which mislead me, and its consequences be judged by my country with some share of the partiality in which they originated. <paragraph> Such being the impressions under which I have, in obedience to the public summons, repaired to the present station, it would be peculiarly improper to omit in this first official act my fervent supplications to that Almighty Being who rules over the universe, who presides in the councils of nations, and whose providential aids can supply every human defect, that His benediction may consecrate to the liberties and happiness of the people of the United States a Government instituted by themselves for these essential purposes, and may enable every instrument employed in its administration to execute with success the functions allotted to his charge. In tendering this homage to the Great Author of every public and private good, I assure myself that it expresses your sentiments not less than my own, nor those of my fellow- citizens at large less than either. No people can be bound to acknowledge and adore the Invisible Hand which conducts the affairs of men more than those of the United States. Every step by which they have advanced to the character of an independent nation seems to have been distinguished by some token of providential agency; and in the important revolution just accomplished in the system of their united government the tranquil deliberations and voluntary consent of so many distinct communities from which the event has resulted can not be compared with the means by which most governments have been established without some return of pious gratitude, along with an humble anticipation of the future blessings which the past seem to presage. These reflections, arising out of the present crisis, have forced themselves too strongly on my mind to be suppressed. You will join with me, I trust, in thinking that there are none under the influence of which the proceedings of a new and free government can more auspiciously commence. <paragraph> By the article establishing the executive department it is made the duty of the President \"to recommend to your consideration such measures as he shall judge necessary and expedient.\" The circumstances under which I now meet you will acquit me from entering into that subject further than to refer to the great constitutional charter under which you are assembled, and which, in defining your powers, designates the objects to which your attention is to be given. It will be more consistent with those circumstances, and far more congenial with the feelings which actuate me, to substitute, in place of a recommendation of particular measures, the tribute that is due to the talents, the rectitude, and the patriotism which adorn the characters selected to devise and adopt them. In these honorable qualifications I behold the surest pledges that as on one side no local prejudices or attachments, no separate views nor party animosities, will misdirect the comprehensive and equal eye which ought to watch over this great assemblage of communities and interests, so, on another, that the foundation of our national policy will be laid in the pure and immutable principles of private morality, and the preeminence of free government be exemplified by all the attributes which can win the affections of its citizens and command the respect of the world. I dwell on this prospect with every satisfaction which an ardent love for my country can inspire, since there is no truth more thoroughly established than that there exists in the economy and course of nature an indissoluble union between virtue and happiness; between duty and advantage; between the genuine maxims of an honest and magnanimous policy and the solid rewards of public prosperity and felicity; since we ought to be no less persuaded that the propitious smiles of Heaven can never be expected on a nation that disregards the eternal rules of order and right which Heaven itself has ordained; and since the preservation of the sacred fire of liberty and the destiny of the republican model of government are justly considered, perhaps, as deeply, as finally, staked on the experiment entrusted to the hands of the American people. <paragraph> Besides the ordinary objects submitted to your care, it will remain with your judgment to decide how far an exercise of the occasional power delegated by the fifth article of the Constitution is rendered expedient at the present juncture by the nature of objections which have been urged against the system, or by the degree of inquietude which has given birth to them. Instead of undertaking particular recommendations on this subject, in which I could be guided by no lights derived from official opportunities, I shall again give way to my entire confidence in your discernment and pursuit of the public good; for I assure myself that whilst you carefully avoid every alteration which might endanger the benefits of an united and effective government, or which ought to await the future lessons of experience, a reverence for the characteristic rights of freemen and a regard for the public harmony will sufficiently influence your deliberations on the question how far the former can be impregnably fortified or the latter be safely and advantageously promoted. <paragraph> To the foregoing observations I have one to add, which will be most properly addressed to the House of Representatives. It concerns myself, and will therefore be as brief as possible. When I was first honored with a call into the service of my country, then on the eve of an arduous struggle for its liberties, the light in which I contemplated my duty required that I should renounce every pecuniary compensation. From this resolution I have in no instance departed; and being still under the impressions which produced it, I must decline as inapplicable to myself any share in the personal emoluments which may be indispensably included in a permanent provision for the executive department, and must accordingly pray that the pecuniary estimates for the station in which I am placed may during my continuance in it be limited to such actual expenditures as the public good may be thought to require. <paragraph> Having thus imparted to you my sentiments as they have been awakened by the occasion which brings us together, I shall take my present leave; but not without resorting once more to the benign Parent of the Human Race in humble supplication that, since He has been pleased to favor the American people with opportunities for deliberating in perfect tranquillity, and dispositions for deciding with unparalleled unanimity on a form of government for the security of their union and the advancement of their happiness, so His divine blessing may be equally conspicuous in the enlarged views, the temperate consultations, and the wise measures on which the success of this Government must depend."

Addresses <- bind_rows(
    lapply(addresses,
           function(addr) {
               return(tibble(president=addr[1],
                             site=addr[2],
                             date=mdy(addr[3]),
                             text=str_c(addr[-(1:3)], collapse=" <paragraph> ")  
               ))  } 
    ))

Addresses$text[1]

[1] "Fellow-Citizens of the Senate and of the House of Representatives: <paragraph> Among the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the 14th day of the present month. On the one hand, I was summoned by my Country, whose voice I can never hear but with veneration and love, from a retreat which I had chosen with the fondest predilection, and, in my flattering hopes, with an immutable decision, as the asylum of my declining years--a retreat which was rendered every day more necessary as well as more dear to me by the addition of habit to inclination, and of frequent interruptions in my health to the gradual waste committed on it by time. On the other hand, the magnitude and difficulty of the trust to which the voice of my country called me, being sufficient to awaken in the wisest and most experienced of her citizens a distrustful scrutiny into his qualifications, could not but overwhelm with despondence one who (inheriting inferior endowments from nature and unpracticed in the duties of civil administration) ought to be peculiarly conscious of his own deficiencies. In this conflict of emotions all I dare aver is that it has been my faithful study to collect my duty from a just appreciation of every circumstance by which it might be affected. All I dare hope is that if, in executing this task, I have been too much swayed by a grateful remembrance of former instances, or by an affectionate sensibility to this transcendent proof of the confidence of my fellow-citizens, and have thence too little consulted my incapacity as well as disinclination for the weighty and untried cares before me, my error will be palliated by the motives which mislead me, and its consequences be judged by my country with some share of the partiality in which they originated. <paragraph> Such being the impressions under which I have, in obedience to the public summons, repaired to the present station, it would be peculiarly improper to omit in this first official act my fervent supplications to that Almighty Being who rules over the universe, who presides in the councils of nations, and whose providential aids can supply every human defect, that His benediction may consecrate to the liberties and happiness of the people of the United States a Government instituted by themselves for these essential purposes, and may enable every instrument employed in its administration to execute with success the functions allotted to his charge. In tendering this homage to the Great Author of every public and private good, I assure myself that it expresses your sentiments not less than my own, nor those of my fellow- citizens at large less than either. No people can be bound to acknowledge and adore the Invisible Hand which conducts the affairs of men more than those of the United States. Every step by which they have advanced to the character of an independent nation seems to have been distinguished by some token of providential agency; and in the important revolution just accomplished in the system of their united government the tranquil deliberations and voluntary consent of so many distinct communities from which the event has resulted can not be compared with the means by which most governments have been established without some return of pious gratitude, along with an humble anticipation of the future blessings which the past seem to presage. These reflections, arising out of the present crisis, have forced themselves too strongly on my mind to be suppressed. You will join with me, I trust, in thinking that there are none under the influence of which the proceedings of a new and free government can more auspiciously commence. <paragraph> By the article establishing the executive department it is made the duty of the President \"to recommend to your consideration such measures as he shall judge necessary and expedient.\" The circumstances under which I now meet you will acquit me from entering into that subject further than to refer to the great constitutional charter under which you are assembled, and which, in defining your powers, designates the objects to which your attention is to be given. It will be more consistent with those circumstances, and far more congenial with the feelings which actuate me, to substitute, in place of a recommendation of particular measures, the tribute that is due to the talents, the rectitude, and the patriotism which adorn the characters selected to devise and adopt them. In these honorable qualifications I behold the surest pledges that as on one side no local prejudices or attachments, no separate views nor party animosities, will misdirect the comprehensive and equal eye which ought to watch over this great assemblage of communities and interests, so, on another, that the foundation of our national policy will be laid in the pure and immutable principles of private morality, and the preeminence of free government be exemplified by all the attributes which can win the affections of its citizens and command the respect of the world. I dwell on this prospect with every satisfaction which an ardent love for my country can inspire, since there is no truth more thoroughly established than that there exists in the economy and course of nature an indissoluble union between virtue and happiness; between duty and advantage; between the genuine maxims of an honest and magnanimous policy and the solid rewards of public prosperity and felicity; since we ought to be no less persuaded that the propitious smiles of Heaven can never be expected on a nation that disregards the eternal rules of order and right which Heaven itself has ordained; and since the preservation of the sacred fire of liberty and the destiny of the republican model of government are justly considered, perhaps, as deeply, as finally, staked on the experiment entrusted to the hands of the American people. <paragraph> Besides the ordinary objects submitted to your care, it will remain with your judgment to decide how far an exercise of the occasional power delegated by the fifth article of the Constitution is rendered expedient at the present juncture by the nature of objections which have been urged against the system, or by the degree of inquietude which has given birth to them. Instead of undertaking particular recommendations on this subject, in which I could be guided by no lights derived from official opportunities, I shall again give way to my entire confidence in your discernment and pursuit of the public good; for I assure myself that whilst you carefully avoid every alteration which might endanger the benefits of an united and effective government, or which ought to await the future lessons of experience, a reverence for the characteristic rights of freemen and a regard for the public harmony will sufficiently influence your deliberations on the question how far the former can be impregnably fortified or the latter be safely and advantageously promoted. <paragraph> To the foregoing observations I have one to add, which will be most properly addressed to the House of Representatives. It concerns myself, and will therefore be as brief as possible. When I was first honored with a call into the service of my country, then on the eve of an arduous struggle for its liberties, the light in which I contemplated my duty required that I should renounce every pecuniary compensation. From this resolution I have in no instance departed; and being still under the impressions which produced it, I must decline as inapplicable to myself any share in the personal emoluments which may be indispensably included in a permanent provision for the executive department, and must accordingly pray that the pecuniary estimates for the station in which I am placed may during my continuance in it be limited to such actual expenditures as the public good may be thought to require. <paragraph> Having thus imparted to you my sentiments as they have been awakened by the occasion which brings us together, I shall take my present leave; but not without resorting once more to the benign Parent of the Human Race in humble supplication that, since He has been pleased to favor the American people with opportunities for deliberating in perfect tranquillity, and dispositions for deciding with unparalleled unanimity on a form of government for the security of their union and the advancement of their happiness, so His divine blessing may be equally conspicuous in the enlarged views, the temperate consultations, and the wise measures on which the success of this Government must depend."

View(Addresses)

Add two descriptive statistics, the number of the address and the number of characters.

Addresses <- bind_cols(Addresses,
                       address =1:nrow(Addresses),            # an id for later
                       n.char = str_length(Addresses$text)
                       )

A simple example of the next construction is useful to appreciate what is happening.

str_locate_all(c("apple apple", "banana app", "pear", "pineapple app"),
              'ap')

[[1]]
     start end
[1,]     1   2
[2,]     7   8

[[2]]
     start end
[1,]     8   9

[[3]]
     start end

[[4]]
     start end
[1,]     5   6
[2,]    11  12

Addresses <- bind_cols(Addresses, 
                       n.paragraph = sapply(str_locate_all(Addresses$text, '<paragraph>'),
                                             nrow))

Addresses$text[1]

[1] "Fellow-Citizens of the Senate and of the House of Representatives: <paragraph> Among the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order, and received on the 14th day of the present month. On the one hand, I was summoned by my Country, whose voice I can never hear but with veneration and love, from a retreat which I had chosen with the fondest predilection, and, in my flattering hopes, with an immutable decision, as the asylum of my declining years--a retreat which was rendered every day more necessary as well as more dear to me by the addition of habit to inclination, and of frequent interruptions in my health to the gradual waste committed on it by time. On the other hand, the magnitude and difficulty of the trust to which the voice of my country called me, being sufficient to awaken in the wisest and most experienced of her citizens a distrustful scrutiny into his qualifications, could not but overwhelm with despondence one who (inheriting inferior endowments from nature and unpracticed in the duties of civil administration) ought to be peculiarly conscious of his own deficiencies. In this conflict of emotions all I dare aver is that it has been my faithful study to collect my duty from a just appreciation of every circumstance by which it might be affected. All I dare hope is that if, in executing this task, I have been too much swayed by a grateful remembrance of former instances, or by an affectionate sensibility to this transcendent proof of the confidence of my fellow-citizens, and have thence too little consulted my incapacity as well as disinclination for the weighty and untried cares before me, my error will be palliated by the motives which mislead me, and its consequences be judged by my country with some share of the partiality in which they originated. <paragraph> Such being the impressions under which I have, in obedience to the public summons, repaired to the present station, it would be peculiarly improper to omit in this first official act my fervent supplications to that Almighty Being who rules over the universe, who presides in the councils of nations, and whose providential aids can supply every human defect, that His benediction may consecrate to the liberties and happiness of the people of the United States a Government instituted by themselves for these essential purposes, and may enable every instrument employed in its administration to execute with success the functions allotted to his charge. In tendering this homage to the Great Author of every public and private good, I assure myself that it expresses your sentiments not less than my own, nor those of my fellow- citizens at large less than either. No people can be bound to acknowledge and adore the Invisible Hand which conducts the affairs of men more than those of the United States. Every step by which they have advanced to the character of an independent nation seems to have been distinguished by some token of providential agency; and in the important revolution just accomplished in the system of their united government the tranquil deliberations and voluntary consent of so many distinct communities from which the event has resulted can not be compared with the means by which most governments have been established without some return of pious gratitude, along with an humble anticipation of the future blessings which the past seem to presage. These reflections, arising out of the present crisis, have forced themselves too strongly on my mind to be suppressed. You will join with me, I trust, in thinking that there are none under the influence of which the proceedings of a new and free government can more auspiciously commence. <paragraph> By the article establishing the executive department it is made the duty of the President \"to recommend to your consideration such measures as he shall judge necessary and expedient.\" The circumstances under which I now meet you will acquit me from entering into that subject further than to refer to the great constitutional charter under which you are assembled, and which, in defining your powers, designates the objects to which your attention is to be given. It will be more consistent with those circumstances, and far more congenial with the feelings which actuate me, to substitute, in place of a recommendation of particular measures, the tribute that is due to the talents, the rectitude, and the patriotism which adorn the characters selected to devise and adopt them. In these honorable qualifications I behold the surest pledges that as on one side no local prejudices or attachments, no separate views nor party animosities, will misdirect the comprehensive and equal eye which ought to watch over this great assemblage of communities and interests, so, on another, that the foundation of our national policy will be laid in the pure and immutable principles of private morality, and the preeminence of free government be exemplified by all the attributes which can win the affections of its citizens and command the respect of the world. I dwell on this prospect with every satisfaction which an ardent love for my country can inspire, since there is no truth more thoroughly established than that there exists in the economy and course of nature an indissoluble union between virtue and happiness; between duty and advantage; between the genuine maxims of an honest and magnanimous policy and the solid rewards of public prosperity and felicity; since we ought to be no less persuaded that the propitious smiles of Heaven can never be expected on a nation that disregards the eternal rules of order and right which Heaven itself has ordained; and since the preservation of the sacred fire of liberty and the destiny of the republican model of government are justly considered, perhaps, as deeply, as finally, staked on the experiment entrusted to the hands of the American people. <paragraph> Besides the ordinary objects submitted to your care, it will remain with your judgment to decide how far an exercise of the occasional power delegated by the fifth article of the Constitution is rendered expedient at the present juncture by the nature of objections which have been urged against the system, or by the degree of inquietude which has given birth to them. Instead of undertaking particular recommendations on this subject, in which I could be guided by no lights derived from official opportunities, I shall again give way to my entire confidence in your discernment and pursuit of the public good; for I assure myself that whilst you carefully avoid every alteration which might endanger the benefits of an united and effective government, or which ought to await the future lessons of experience, a reverence for the characteristic rights of freemen and a regard for the public harmony will sufficiently influence your deliberations on the question how far the former can be impregnably fortified or the latter be safely and advantageously promoted. <paragraph> To the foregoing observations I have one to add, which will be most properly addressed to the House of Representatives. It concerns myself, and will therefore be as brief as possible. When I was first honored with a call into the service of my country, then on the eve of an arduous struggle for its liberties, the light in which I contemplated my duty required that I should renounce every pecuniary compensation. From this resolution I have in no instance departed; and being still under the impressions which produced it, I must decline as inapplicable to myself any share in the personal emoluments which may be indispensably included in a permanent provision for the executive department, and must accordingly pray that the pecuniary estimates for the station in which I am placed may during my continuance in it be limited to such actual expenditures as the public good may be thought to require. <paragraph> Having thus imparted to you my sentiments as they have been awakened by the occasion which brings us together, I shall take my present leave; but not without resorting once more to the benign Parent of the Human Race in humble supplication that, since He has been pleased to favor the American people with opportunities for deliberating in perfect tranquillity, and dispositions for deciding with unparalleled unanimity on a form of government for the security of their union and the advancement of their happiness, so His divine blessing may be equally conspicuous in the enlarged views, the temperate consultations, and the wise measures on which the success of this Government must depend."

View(Addresses)

Plots are prettier now – and perhaps more informative.

Addresses %>% 
    mutate(`characters per paragraph`=n.char/n.paragraph) %>%    # `` allows improper R name
    ggplot(aes(x=date, y=`characters per paragraph`)) +
    geom_line() + geom_point()

Word frequencies and sentiments

Build a document term matrix from the text of the addresses. The steps are the same as those used in the example of wine tasting notes.

1 Convert the text into a tm corpus 2 Tokenize the text by combining tm_map with content_transformers 3 Build the document-term matrix using tm’s function

Several of these content transformers rely on regular expressions.

• Replace dollar amounts by the symbol “”. This will track mentions of money, without distinguising the amounts as separate types.
• Convert years (a four digit number starting with 17, 18, 19, or 20) into one symbol. This retains the notion of a date, but without introducing distinct types.
  
• Replace other numbers by the symbol “” (rather than removing them all).

The order of these transformations matters. You would not find a dollar amount if you got rid of the numbers first or if you removed punctuation first (which would knock out the “$” sign).

These transformations use regular expressions, as illustrated below.

str_replace("will cost $41,000,000 that ", "\\$[1-9][,0-9]*", "<money>")  # $ otherwise is EOL

[1] "will cost <money> that "

str_replace("the year 1789 in which we ",  "[1-9][0-9]{3}", "<year>") 

[1] "the year <year> in which we "

str_replace_all("one 1 two 33  555 digit", "[1-9][,0-9]*", "<number>")

[1] "one <number> two <number>  <number> digit"

AddressCorpus <- Corpus(VectorSource(Addresses$text))
replace <- content_transformer(function(text, from, to) str_replace_all(text, from, to))
toSpace <- content_transformer(function(text, pattern) str_replace_all(text, pattern, " "))
toLower <- content_transformer(function(text) tolower(text))
AddressCorpus <- tm_map(AddressCorpus, toLower)
AddressCorpus <- tm_map(AddressCorpus, toSpace, '-|/') 
AddressCorpus <- tm_map(AddressCorpus, replace, "\\$[1-9][,0-9]*", "<money>") 
AddressCorpus <- tm_map(AddressCorpus, replace, "[1-9][0-9]{3}",   "<year>") 
AddressCorpus <- tm_map(AddressCorpus, replace, "[1-9][,0-9]*",    "<number>") 
AddressCorpus <- tm_map(AddressCorpus, toSpace, "<paragraph>")
AddressCorpus <- tm_map(AddressCorpus, removePunctuation)
AddressCorpus <- tm_map(AddressCorpus, removeWords, stopwords("english"))  
AddressCorpus <- tm_map(AddressCorpus, stripWhitespace)

inspect(AddressCorpus[[1]])

<<PlainTextDocument>>
Metadata:  7
Content:  chars: 5500

fellow citizens senate house representatives among vicissitudes incident life event filled greater anxieties notification transmitted order received numberth day present month one hand summoned country whose voice can never hear veneration love retreat chosen fondest predilection flattering hopes immutable decision asylum declining years retreat rendered every day necessary well dear addition habit inclination frequent interruptions health gradual waste committed time hand magnitude difficulty trust voice country called sufficient awaken wisest experienced citizens distrustful scrutiny qualifications overwhelm despondence one inheriting inferior endowments nature unpracticed duties civil administration peculiarly conscious deficiencies conflict emotions dare aver faithful study collect duty just appreciation every circumstance might affected dare hope executing task much swayed grateful remembrance former instances affectionate sensibility transcendent proof confidence fellow citizens thence little consulted incapacity well disinclination weighty untried cares error will palliated motives mislead consequences judged country share partiality originated impressions obedience public summons repaired present station peculiarly improper omit first official act fervent supplications almighty rules universe presides councils nations whose providential aids can supply every human defect benediction may consecrate liberties happiness people united states government instituted essential purposes may enable every instrument employed administration execute success functions allotted charge tendering homage great author every public private good assure expresses sentiments less fellow citizens large less either people can bound acknowledge adore invisible hand conducts affairs men united states every step advanced character independent nation seems distinguished token providential agency important revolution just accomplished system united government tranquil deliberations voluntary consent many distinct communities event resulted can compared means governments established without return pious gratitude along humble anticipation future blessings past seem presage reflections arising present crisis forced strongly mind suppressed will join trust thinking none influence proceedings new free government can auspiciously commence article establishing executive department made duty president recommend consideration measures shall judge necessary expedient circumstances now meet will acquit entering subject refer great constitutional charter assembled defining powers designates objects attention given will consistent circumstances far congenial feelings actuate substitute place recommendation particular measures tribute due talents rectitude patriotism adorn characters selected devise adopt honorable qualifications behold surest pledges one side local prejudices attachments separate views party animosities will misdirect comprehensive equal eye watch great assemblage communities interests another foundation national policy will laid pure immutable principles private morality preeminence free government exemplified attributes can win affections citizens command respect world dwell prospect every satisfaction ardent love country can inspire since truth thoroughly established exists economy course nature indissoluble union virtue happiness duty advantage genuine maxims honest magnanimous policy solid rewards public prosperity felicity since less persuaded propitious smiles heaven can never expected nation disregards eternal rules order right heaven ordained since preservation sacred fire liberty destiny republican model government justly considered perhaps deeply finally staked experiment entrusted hands american people besides ordinary objects submitted care will remain judgment decide far exercise occasional power delegated fifth article constitution rendered expedient present juncture nature objections urged system degree inquietude given birth instead undertaking particular recommendations subject guided lights derived official opportunities shall give way entire confidence discernment pursuit public good assure whilst carefully avoid every alteration might endanger benefits united effective government await future lessons experience reverence characteristic rights freemen regard public harmony will sufficiently influence deliberations question far former can impregnably fortified latter safely advantageously promoted foregoing observations one add will properly addressed house representatives concerns will therefore brief possible first honored call service country eve arduous struggle liberties light contemplated duty required renounce every pecuniary compensation resolution instance departed still impressions produced must decline inapplicable share personal emoluments may indispensably included permanent provision executive department must accordingly pray pecuniary estimates station placed may continuance limited actual expenditures public good may thought require thus imparted sentiments awakened occasion brings us together shall take present leave without resorting benign parent human race humble supplication since pleased favor american people opportunities deliberating perfect tranquillity dispositions deciding unparalleled unanimity form government security union advancement happiness divine blessing may equally conspicuous enlarged views temperate consultations wise measures success government must depend

The document-term matrix is not nearly so sparse as was that derived from the wine tasting notes. Fully 8% of the counts are not zero. The vocabulary is considerably larger. (The wine data had about 5,600 types.)

dtm <- DocumentTermMatrix(AddressCorpus)
dtm

<<DocumentTermMatrix (documents: 58, terms: 8986)>>
Non-/sparse entries: 39252/481936
Sparsity           : 92%
Maximal term length: 17
Weighting          : term frequency (tf)

As in other examples, it ia useful to have the counts of tokens in documents and counts of the tokens for each type.

ni <- rowSums(as.matrix(dtm))
mj <- colSums(as.matrix(dtm))

Out of curiousity, the longest term is (now that I fixed an errors in the source text!)

j <- which.max(str_length(names(mj)))
j

[1] 3579

names(mj)[j]

[1] "misrepresentation"

Exploring the vocabulary

This plot shows the most common word types.

Freq <- data_frame(type = names(mj), count = mj)     
Freq %>%
    top_n(25, count)                         %>%
    mutate(type=reorder(type,count))         %>%     # rather than alphabetical order
    ggplot(aes(type,count)) + geom_col() + coord_flip()

Does this distribution change over time?

For example, here are distributions from around the start of the 21th century compared to those from the 100 years before. To get these, notice that we have a DTM for the whole collection. We just need to pull out the counts for the appropriate periods using the dates held in the metadata of the Addresses data frame. There are fancy ways to do this, but this approach is easy to check.

i.21 <- which(mdy("1/1/1990") < Addresses$date)  # lubridate
i.21

[1] 52 53 54 55 56 57 58

mj.21 <- colSums(as.matrix(dtm[i.21,]))
i.20 <- which((mdy("1/1/1890") < Addresses$date) & (Addresses$date < mdy("1/1/1920")))
i.20

[1] 26 28 29 30 31 32 33

mj.20 <- colSums(as.matrix(dtm[i.20,]))

Make a utility function for things you do frequently.

show_top_types <- function(named.counts, n) {
    tibble(type = names(named.counts), count = named.counts) %>%
    top_n(25, count)                         %>%
    mutate(type=reorder(type,count))         %>%     # rather than alphabetical order
    ggplot(aes(type,count)) + geom_col() + coord_flip()
}

show_top_types(mj.20)

show_top_types(mj.21)

Plots of the two sets of frequencies are interesting.

tibble(c21=mj.21, c20=mj.20, types=names(mj.20)) %>%
    ggplot(aes(c20, c21)) +
    geom_point(alpha=0.2)  +
    geom_abline(color = "gray40", lty = 2) +
    geom_text(aes(label = types), check_overlap=TRUE, vjust=1.5) 

How often does America (American) show up over time. Is there a trend?

american.words <- c('american', 'americans', 'america')
n.amer <- rowSums(as.matrix(dtm[,american.words]))
plot(Addresses$date,n.amer)

LSA and topic models

That’s a lot of types and not many documents. This is quite a small data set for these methods.

dim(dtm)

[1]   58 8986

Remove the words that are rare (rather than in this case replacing them with OOV). I will limit the analysis to words that appear in at least 5 addresses.

n.addr <- colSums(0 < as.matrix(dtm))

dtm.rs <- as.matrix(dtm[,5 < n.addr])
dim(dtm.rs)

[1]   58 1894

Add the scaling.

mj.rs <- colSums(dtm.rs)
ni.rs <- rowSums(dtm.rs)
dtm.lsa <- dtm.rs / sqrt(ni.rs)
dtm.lsa <- t( t(dtm.lsa)/sqrt(mj.rs) )
udv <- svd(dtm.lsa)

Again, not a clear cut off in the spectrum.

plot(udv$d, log='x')

row.names(udv$v) <- names(mj.rs)
plot_loadings(udv$v, 2,3, threshold=0.05, cex=0.7)

How about topic models? This looks promising.

n.topics <- 5
lda <- LDA(dtm.rs, n.topics, control = list(seed = 1234))
attributes(lda)$alpha

[1] 0.06071776

As in the wine example, common words that appear in most addresses show up in every topic.

terms(lda,15)

      Topic 1      Topic 2   Topic 3      Topic 4      Topic 5       
 [1,] "government" "will"    "will"       "will"       "government"  
 [2,] "will"       "upon"    "government" "world"      "will"        
 [3,] "people"     "can"     "people"     "new"        "states"      
 [4,] "upon"       "must"    "public"     "people"     "upon"        
 [5,] "can"        "people"  "great"      "can"        "constitution"
 [6,] "must"       "great"   "states"     "america"    "people"      
 [7,] "world"      "world"   "country"    "nation"     "may"         
 [8,] "peace"      "nations" "may"        "must"       "can"         
 [9,] "public"     "shall"   "every"      "let"        "union"       
[10,] "american"   "free"    "shall"      "freedom"    "one"         
[11,] "shall"      "peace"   "citizens"   "time"       "power"       
[12,] "justice"    "may"     "united"     "government" "shall"       
[13,] "nation"     "men"     "war"        "one"        "country"     
[14,] "national"   "nation"  "nation"     "every"      "citizens"    
[15,] "republic"   "freedom" "upon"       "today"      "now"         

And the composition over time is interesting. The summary provided by topics is a bit too vague, just providing the rank order. Topics 3 and 5 were “popular” in early addresses, now its topic 4.

t(topics(lda,5))

   [,1] [,2] [,3] [,4] [,5]
1     3    5    1    4    2
2     3    4    5    1    2
3     3    1    2    4    5
4     5    3    4    1    2
5     5    3    1    4    2
6     3    5    1    2    4
7     3    1    5    4    2
8     3    5    2    1    4
9     3    5    2    1    4
10    3    1    5    4    2
11    3    5    1    2    4
12    5    3    1    4    2
13    5    3    1    4    2
14    5    3    1    4    2
15    5    3    1    2    4
16    3    5    1    2    4
17    5    3    1    4    2
18    5    3    4    2    1
19    5    2    1    4    3
20    4    5    3    2    1
21    5    3    2    4    1
22    3    2    5    4    1
23    3    5    2    1    4
24    5    2    3    4    1
25    1    3    5    4    2
26    1    5    3    2    4
27    3    2    5    1    4
28    3    2    5    1    4
29    3    1    2    5    4
30    2    1    4    3    5
31    5    2    1    3    4
32    2    4    1    5    3
33    2    1    4    3    5
34    1    4    2    5    3
35    2    1    5    3    4
36    1    2    5    3    4
37    2    3    1    4    5
38    1    4    2    5    3
39    4    1    3    2    5
40    4    1    2    5    3
41    2    1    4    3    5
42    2    4    1    3    5
43    2    4    1    3    5
44    4    2    5    3    1
45    4    2    5    1    3
46    4    2    1    5    3
47    4    2    1    3    5
48    4    2    5    3    1
49    4    1    2    5    3
50    4    1    2    5    3
51    4    2    3    5    1
52    4    2    1    3    5
53    4    2    1    5    3
54    4    1    3    2    5
55    4    2    3    1    5
56    4    2    3    5    1
57    4    5    2    1    3
58    4    2    1    3    5

theta <- attributes(lda)$gamma
dim(theta)

[1] 58  5

The numerical proportions are more interesting.

round(theta,2)

      [,1] [,2] [,3] [,4] [,5]
 [1,] 0.00 0.00 0.86 0.00 0.14
 [2,] 0.00 0.00 1.00 0.00 0.00
 [3,] 0.00 0.00 1.00 0.00 0.00
 [4,] 0.08 0.00 0.33 0.23 0.36
 [5,] 0.11 0.00 0.43 0.00 0.46
 [6,] 0.02 0.00 0.85 0.00 0.13
 [7,] 0.23 0.00 0.61 0.00 0.16
 [8,] 0.00 0.00 1.00 0.00 0.00
 [9,] 0.00 0.00 1.00 0.00 0.00
[10,] 0.09 0.00 0.91 0.00 0.00
[11,] 0.00 0.00 1.00 0.00 0.00
[12,] 0.15 0.00 0.42 0.00 0.44
[13,] 0.11 0.00 0.38 0.00 0.52
[14,] 0.00 0.00 0.20 0.00 0.80
[15,] 0.00 0.00 0.00 0.00 1.00
[16,] 0.00 0.00 1.00 0.00 0.00
[17,] 0.20 0.00 0.21 0.08 0.51
[18,] 0.00 0.00 0.17 0.00 0.83
[19,] 0.00 0.00 0.00 0.00 1.00
[20,] 0.00 0.00 0.20 0.44 0.36
[21,] 0.00 0.16 0.24 0.00 0.60
[22,] 0.00 0.27 0.55 0.00 0.18
[23,] 0.05 0.11 0.60 0.00 0.24
[24,] 0.00 0.23 0.05 0.04 0.68
[25,] 0.68 0.00 0.32 0.00 0.00
[26,] 1.00 0.00 0.00 0.00 0.00
[27,] 0.00 0.34 0.36 0.00 0.30
[28,] 0.05 0.35 0.41 0.00 0.19
[29,] 0.27 0.24 0.48 0.00 0.01
[30,] 0.14 0.86 0.00 0.00 0.00
[31,] 0.00 0.37 0.00 0.00 0.63
[32,] 0.00 1.00 0.00 0.00 0.00
[33,] 0.38 0.59 0.00 0.03 0.00
[34,] 0.95 0.00 0.00 0.05 0.00
[35,] 0.44 0.45 0.00 0.00 0.12
[36,] 1.00 0.00 0.00 0.00 0.00
[37,] 0.04 0.88 0.08 0.00 0.00
[38,] 0.49 0.12 0.00 0.39 0.00
[39,] 0.19 0.00 0.04 0.77 0.00
[40,] 0.17 0.00 0.00 0.83 0.00
[41,] 0.13 0.83 0.00 0.04 0.00
[42,] 0.01 0.68 0.00 0.31 0.00
[43,] 0.00 0.80 0.00 0.20 0.00
[44,] 0.00 0.12 0.00 0.88 0.00
[45,] 0.00 0.00 0.00 1.00 0.00
[46,] 0.00 0.00 0.00 1.00 0.00
[47,] 0.00 0.00 0.00 1.00 0.00
[48,] 0.00 0.20 0.00 0.80 0.00
[49,] 0.12 0.06 0.00 0.83 0.00
[50,] 0.00 0.00 0.00 1.00 0.00
[51,] 0.00 0.00 0.00 1.00 0.00
[52,] 0.00 0.00 0.00 1.00 0.00
[53,] 0.00 0.00 0.00 1.00 0.00
[54,] 0.00 0.00 0.00 1.00 0.00
[55,] 0.00 0.05 0.00 0.95 0.00
[56,] 0.00 0.09 0.00 0.91 0.00
[57,] 0.00 0.00 0.00 1.00 0.00
[58,] 0.00 0.00 0.00 1.00 0.00

tibble(one=theta[,1], two=theta[,2], three=theta[,3], four=theta[,4], five=theta[,5],
       date=Addresses$date) %>%
    gather(one,two,three,four,five, key="Topic", value="Proportion") %>%
    ggplot(aes(date,Proportion)) +
    geom_point(aes(color=Topic, alpha=Proportion))