I, too, am tired of hearing about Hillary’s use of a private email server. On the other hand, it led to a pretty neat data set to unpack: a dump of emails she’s sent and received.

I played around with this data set a bit and was particularly interested in how different groups of people interacted with Hillary. Did men use shorter sentences than women, for example? Did her staffers send one-liners versus ambassadors who sent lengthy emails? Did she have interesting relationships with people we might not be familiar with?

I didn’t get a chance to answer all of these questions, but I ended up being interested in the way words in her email were clustered, and decided to come up with a visualization based on that.

For a simple representation to start, I created a scatter plot visualization using mpld3, which creates interactive matplotlib graphs for the browser. It’s clunky to navigate (you need to switch to a zoom-in mode, drag a rectangular portion of the graph to zoom in on, then switch again to the cursor mode to scroll over words), but it’s interesting to see which words appear together for a first step.

Isn’t it interesting that “bipartisan” appears well outside the main cluster of words?

Lesson learned along the way: visualizing text is hard. I found that the norm for text visualizations out there, such as word clouds or circle packing, was reductionist for some of the data I have, like topic models or k-means clustering.

While I didn’t create data visualizations for some of the questions I posed earlier, I do have some statistics:

For males:

6187 sentences
83764 word tokens
10762 word types
7.78 average tokens per type
13.54 average sentence length
5.01 average word token length
7.34 average word type length
Hapax legomena (words that appear only once – an indicator of vocabulary usage) comprise 49.60% of the types

For females:

22845 sentences
369517 word tokens
30386 word types
12.16 average tokens per type
16.17 average sentence length
4.94 average word token length
7.84 average word type length
Hapax legomena comprise 49.76% of the types