Data to accompany the ICWSM 2015 paper "CREDBANK: A Large-scale Social Media Corpus With Associated Credibility Annotations"

View the Project on GitHub TanuM/CREDBANK-data

Downloading the Data using S3

You can use AWS CLI or the easier s3cmd to download the data. First configure the tool to use your AWS account credentials and then use the appropriate command to download the s3 links listed below.

To download a data file (e.g., to the current folder, use the following command:

aws s3api get-object --request-payer requester --bucket credbank --key


s3cmd get --requester-pays s3://credbank/

Note that you will be charged a small data transfer fee by AWS, which can be offset by their free tier.

Data Description

The CREDBANK corpus was collected between mid October 2014 and end of February 2015. It is a collection of streaming tweets tracked over this period, topics in this tweet stream, topics classified as events or non events, events annotated with credibility ratings. The data is spread across four files. The description of each file along with their location is listed below:

1. Streaming Tweet File

S3 Location: s3://credbank/

This file contains more than 169 million streaming tweets spread across multiple timebins when these were collected. The collection period span from 10-Oct-2014 to 26-Feb-2015. Each line in the file corresponds to a timebin and contains 3 fields:

A snippet:

timespan_key        tweet_count     ListOf_tweetid_author_createdAt_tuple 
20150114_044623-20150114_061420 92255   [('555299882668670977', 'ZaE_BeAtZBsM808', 'Wed Jan 14 09:46:23 +0000 2015'), ('555299882698035203', 'gothebroncos', 'Wed Jan 14 09:46:23 +0000 2015'),....]

2. Topic File (Event/Non-Event)

S3 Location: s3://credbank/

This file contains more than 62,000 topics. Each topic is represented by 3 terms, corresponding to the top three terms returned by running topic modeling (LDA) over the streaming tweets. It contains 3 fields:

A snippet:

time_key    topic_terms isEvent
20141024_170629-20141024_181626 louis,ebola,nurse       1
20141024_170629-20141024_181626 retweet,gain,retweets   0

3. Credibility Annotation File

S3 Location: s3://credbank/

This file contains more than 1300 events and their corresponding credibility ratings and reasons behind the ratings entered by Turkers. It contains 4 fields:

A snippet:

topic_key   topic_terms Cred_Ratings    Reasons
louis_ebola_nurse-20141024_170629-20141024_181626       [u'louis', u'ebola', u'nurse']  ['1', '-1', '2', '-2', '0', '2', '0',....]    ['Nurses union describes the procedures taken by nurse who now has Ebola from treating a patient., .....]

4. Searched Tweet File

S3 Location: s3://credbank/

This file contains more than 80 million tweets spread across event topics. The tweets are fetched using the Twitter search API. The search query is constructed by taking a boolean AND over all 3 terms of a topic. Each line in the file contains 4 fields:

A snippet:

topic_key   topic_terms tweet_count ListOf_tweetid_author_createdAt_tuple
louis_ebola_nurse-20141024_170629-20141024_181626       [u'louis', u'ebola', u'nurse']  13243 [('ID=522760301435830272', 'AUTHOR=iMhartyz', 'CreatedAt=2014-10-16 14:45:42'), ('ID=522760172872413184', 'AUTHOR=Tino_carter_v', 'CreatedAt=2014-10-16 14:45:12'), ......] 

Citation Information

If you use the dataset in your research, please cite the following paper:

Mitra, Tanushree, and Eric Gilbert. "CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations." Ninth International AAAI Conference on Web and Social Media. 2015.