An Analysis of Sina Weibo Censorship Using WeiboScope Search Data

Starting at 4:28 PM May 19, 2012, I posted on my Sina Weibo account two names as well as the Chinese words for “Taiwanese independence.” The first name I posted was “Chen Guangcheng” (in English and Chinese), the blind lawyer who escaped house arrest in Shandong province and made his way to the U.S. Embassy in Beijing. The second name was “Bo Xilai” (in English and Chinese), the former Party Secretary of Chongqing who recently fell from power. Less than 14 hours later, I received a message from Sina Weibo’s system administrator informing me that my two posts on “Chen Guangcheng” were “inappropriate” and had been censored. While I can still see the two “Chen Guangcheng” posts on my Sina Weibo account page, no one else can. Surprisingly, my posts on “Bo Xilai” and “Taiwan independence” were not censored.

Herein lies the conundrum with censorship in China. We know that certain topics are censored from blogs hosted in China, Chinese search engines and Weibos. But we don’t know where the line lies. Part of the reason is because the line is constantly moving. Baidu, Sina and Tencent could help identify the line by publishing a list of banned topics or keywords, but they don’t. Rather, they hire “monitoring editors” and rely on self-censorship to ensure that user generated content does not run afoul of Chinese authorities.

Some computer scientists in academia have tried to make sense of censorship in Sina Weibo by analyzing the data. In March 2012, David Bamman, Brendan O’Connor and Noah Smith at Carnegie Mellon University published a paper entitled “Censorship and deletion practices in Chinese social media” in First Monday after analyzing 56 million Sina Weibo messages and found that more than 16% had been deleted. King-Wa Fu and Cedric Sam at the University of Hong Kong’s Journalism and Media Studies Centre have hacked the Weibo Scope Search that archives deleted posts on Sina Weibo.

For my MIT Media Lab final project, I’ve tried to build on King-Wa Fu and Cedric Sam‘s work by analyzing the data collected from the Weibo Scope Search to try to make some sense of Sina Weibo censorship. Since its inception February 1, 2012 to May 20, 2012, the Weibo Scope Search has collected 12,032 deleted messages from Sina Weibo. The first thing I did was to simply plot all the deleted messages on a timeline from February 1, 2012 to May 20, 2012 and this is what I got:

My findings were consistent with the Carnegie Mellon team’s findings. There are spikes in Sina Weibo censorship as a result of media reports and rumors. During the Carnegie Mellon survey duration from June 27, 2011 to September 30, 2011, there was a rumor that former President Jiang Zemin passed away causing a spike in Sina Weibo deletions. From February 1, 2012 to May 20, 2012, the following incidents in China caused in censors employed by Sina Weibo to work overtime:

• February 6, 2012 – Chongqing Public Security Bureau head Wang Lijun goes to U.S. Consulate in Chengdu with information about the death of British businessman Neil Heywood that implicates Chongqing Party Secretary Bo Xilai.
• March 8, 2012 – Chongqing Party Secretary Bo Xilai fails to show up at the National People’s Congress, sparking rumors that he has fallen from power.
• March 15, 2012 – Bo Xilai is removed from post as Chongqing Party Secretary.
• March 18, 2012 – A Ferrari crashed on Beijing’s Fourth Ring Road killing one and injuring two people.
• April 22, 2012 – Blind lawyer Chen Guangcheng escapes from house arrest in Shandong province and makes his way to U.S. Embassy in Beijing.
• May 14, 2012 – The Beijing Daily posts a message on its official Weibo charging that U.S. Ambassador to China Gary Locke is posing as an ordinary citizen and calls for Locke to disclose his wealth.

How to Analyze WeiboScope Search Data

King-Wa Fu and Cedric Sam at the University of Hong Kong’s Journalism and Media Studies Centre have built a WeiboScope Search that sends all of the deleted Weibo posts to a server in Hong Kong and stores them.  However, the data is in JSON format, which looks like this:

To make sense of the data collected, we need to first clean up the data. I used Google refine to clean up the data by:

1) Download + install Google refine
2) Click on “Create Project”
3) Click on “Web Addresses (URLs)”

4) Insert link http://research.jmsc.hku.hk/social/sinaweibo/lastpermissiondenied.all.json
5) Click “Next”

6) Highlight the fields you’re interested in and left click the mouse.

Google refine should automatically put all the fields into columns:

7) Click “Create Project”
8) Click “Export”
9) Click “Excel”

Now that we have the data formatted, we want to make sense of it.

The first project I did was to graph the deleted weibos on a timeline. My classmate Eugene Wu suggested that the best software to visualize the data is Tableau.

1) Download + install Tableau
2) Click on “Open Data”
3) Under “Connect to Data: In a file”, click on “Microsoft Excel”
4) When the “Excel Workbook Connection” window pops up, click “Ok”

5) Change the format for the “created at” and “deleted” columns from “text” to “Date & time” by right clicking the mouse, selecting “Change Data Type” then “Date & time.”

6) Go to the Dimensions box and drag the “deleted” data set to “Columns”
7) Go to the Measures box and drag the “Number of Records” data set to “Rows”
8) In the “Show Me” box, select the type of graph you want. Voila!

Powered by Tableau

Several hours before my classmate J. Nathan Matias first showed me his new tool Dataforager Sunday, Pakistan blocked Twitter apparently because some tweets were urging people to join the third “Everybody Draw Muhammad Day” campaign on May 20, 2012. While I was playing around with Dataforager, I applied it to five news reports about Pakistan blocking (then unblocking) Twitter and came up with the table below:

The Washington Post and Global Voices do the best job of coding links in the story to interviewees’ Twitter accounts. The New York Times and Guardian cite sources as having Twitter accounts, but forces the reader to search for each Twitter account.

One of J. Nathan Matias‘ original intentions for Dataforager was to help users compile a list of experts to learn more about a particular subject or topic. Since I know very little about Pakistan besides the fact that it borders India and it’s where Osama Bin Laden was living for the past six years, I tested J. Nathan Matias‘ theory to see if I could find enough information to write a story about Pakistan using the list of Twitterers compiled using Dataforager.

For this experiment, I used Dataforager to compile a list of experts on Pakistan on Twitter from Washington Post’s Pakistan blocks, then restores, Twitter access article by Richard Leiby and Storify’s Flurry of tweets in wake of Pakistani Twitter ban article by Annie Ali Khan.

From Annie Ali Khan‘s Storify article, Dataforager pulled up a list of 17 Pakistani’s on Twitter:

From Richard Leiby’s Washington Post article, Dataforager pulled up a list of 3 Pakistani’s on Twitter:

Going through the list of tweets compiled from Dataforager Tuesday morning, both mention some kind of unrest in Karachi. Entrepreneur Mohammed Sumair Kolia tweeted that 8 people were killed and 30 injured in the unrest in Karachi. Pakistan’s Ambassador to the U.S. and former journalist Sherry Rehman tweeted that two journalists were injured during the firing at the Awami Tehrik rally in Karachi.

By themselves, the tweets aren’t enough to piece together what happened in Karachi. We know how many people were injured, but we still don’t know what happened, who did it, how it happened, and why it happened. Fortunately, the tweets use a #Karachi hashtag. Doing a “#Karachi” search on Twitter, I get a list of tweets about what happened in Karachi.

A sports reporter with GEO News, Faizan Lakhani, was the first one to tweet “Reports of firing in Boltun Market and old city area. ‪#Karachi‬” with a “#Karachi” hashtag.

Two and a half hours later, the Express Tribune posted on its web site and Twitter a live blog/news article about the riots that unfolded in Karachi after unidentified gunmen open fire on a Awami Tehreek and Peoples Amn Committee rally.