|
| 1 | + |
| 2 | +# twarc2 |
| 3 | + |
| 4 | +twarc2 is a command line tool and Python library for archiving Twitter JSON |
| 5 | +data. Each tweet is represented as a JSON object that was returned from the |
| 6 | +Twitter API. Since Twitter's introduction of their [v2 |
| 7 | +API](https://developer.twitter.com/en/docs/twitter-api/api-reference-index#v2) |
| 8 | +the JSON representation of a tweet is conditional on the types of fields and |
| 9 | +expansions that are requested. twarc2 does the work of requesting the highest |
| 10 | +fidelity representation of a tweet by requesting all the available data for |
| 11 | +tweets. |
| 12 | + |
| 13 | +Tweets are streamed or stored as [line-oriented |
| 14 | +JSON](https://en.wikipedia.org/wiki/JSON_Streaming#Line-delimited_JSON). twarc2 |
| 15 | +will handle Twitter API's [rate |
| 16 | +limits](https://dev.twitter.com/rest/public/rate-limiting) for you. In addition |
| 17 | +to letting you collect tweets twarc can also help you collect users and hydrate |
| 18 | +tweet ids. It also has a collection of [plugins](plugins) you can use to do |
| 19 | +things with the collected JSON data (such as converting it to CSV). |
| 20 | + |
| 21 | +twarc2 was developed as part of the [Documenting the Now](http://www.docnow.io) |
| 22 | +project which was funded by the [Mellon Foundation](https://mellon.org/). |
| 23 | + |
| 24 | +## Install |
| 25 | + |
| 26 | +Before using twarc you will need to register an application at |
| 27 | +[apps.twitter.com](http://apps.twitter.com). Once you've created your |
| 28 | +application, note down the consumer key, consumer secret and then click to |
| 29 | +generate an access token and access token secret. With these four variables |
| 30 | +in hand you are ready to start using twarc. |
| 31 | + |
| 32 | +1. install [Python 3](http://python.org/download) |
| 33 | +2. [pip](https://pip.pypa.io/en/stable/installing/) install twarc: |
| 34 | + |
| 35 | +``` |
| 36 | + pip install --upgrade twarc |
| 37 | +``` |
| 38 | + |
| 39 | +### Homebrew (macOS only) |
| 40 | + |
| 41 | +For macOS users, you can also install `twarc` via [Homebrew](https://brew.sh/): |
| 42 | + |
| 43 | +```bash |
| 44 | +$ brew install twarc |
| 45 | +``` |
| 46 | + |
| 47 | +### Windows |
| 48 | + |
| 49 | +If you installed with pip and see a "failed to create process" when running twarc try reinstalling like this: |
| 50 | + |
| 51 | + python -m pip install --upgrade --force-reinstall twarc |
| 52 | + |
| 53 | +## Quickstart: |
| 54 | + |
| 55 | +First you're going to need to tell twarc about your application API keys and |
| 56 | +grant access to one or more Twitter accounts: |
| 57 | + |
| 58 | + twarc2 configure |
| 59 | + |
| 60 | +Then try out a search: |
| 61 | + |
| 62 | + twarc2 search blacklivesmatter search.jsonl |
| 63 | + |
| 64 | +Or maybe you'd like to collect tweets as they happen? |
| 65 | + |
| 66 | + twarc2 filter blacklivesmatter stream.jsonl |
| 67 | + |
| 68 | +See below for the details about these commands and more. |
| 69 | + |
| 70 | +## Configure |
| 71 | + |
| 72 | +Once you've got your Twitter developer access set up you can tell twarc what they are with the `configure` command. |
| 73 | + |
| 74 | + twarc2 configure |
| 75 | + |
| 76 | +This will store your credentials in your home directory so you don't have to |
| 77 | +keep entering them in. You can most of twarc's functionality by simply |
| 78 | +configuring the *bearer token*, but if you want it to be complete you can enter |
| 79 | +in the *API key* and *API secret*. |
| 80 | + |
| 81 | +You can also the keys in the system environment (`CONSUMER_KEY`, |
| 82 | +`CONSUMER_SECRET`, `ACCESS_TOKEN`, `ACCESS_TOKEN_SECRET`) or using command line |
| 83 | +options (`--consumer-key`, `--consumer-secret`, `--access-token`, |
| 84 | +`--access-token-secret`). |
| 85 | + |
| 86 | +## Search |
| 87 | + |
| 88 | +This uses Twitter's [tweets/search/recent](https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent) and [tweets/search/all](https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-all) endpoints to download *pre-existing* tweets matching a given query. This command will search for any tweets mentioning *blacklivesmatter* from the 7 days. |
| 89 | + |
| 90 | + twarc2 search blacklivesmatter tweets.jsonl |
| 91 | + |
| 92 | +If you have access to the [Academic Research Product Track](https://developer.twitter.com/en/products/twitter-api/academic-research) you can search the full archive of tweets by using the `--archive` option. |
| 93 | + |
| 94 | + twarc2 search --archive blacklivesmatter tweets.jsonl |
| 95 | + |
| 96 | +The queries can be a lot more expressive than matching a single term. For |
| 97 | +example this query will search for tweets containing either `blacklivesmatter` |
| 98 | +or `blm` that were sent to the user \@deray. |
| 99 | + |
| 100 | + twarc2 search 'blacklivesmatter OR blm to:deray' tweets.jsonl |
| 101 | + |
| 102 | +The best way to get familiar with Twitter's search syntax is to consult Twitter's [Building queries for Search Tweets](https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/build-a-query) documentation. |
| 103 | + |
| 104 | +You also should definitely check out Igor Brigadir's *excellent* reference guide |
| 105 | +to the Twitter Search syntax: |
| 106 | +[Advanced Search on Twitter](https://github.com/igorbrigadir/twitter-advanced-search/blob/master/README.md). |
| 107 | +There are lots of hidden gems in there that the advanced search form doesn't |
| 108 | +make readily apparent. |
| 109 | + |
| 110 | +### Limit |
| 111 | + |
| 112 | +Because there is a 500,000 tweet limit (5 million for Academic Research Track) |
| 113 | +you may want to limit the number of tweets you retrieve by using `--limit`: |
| 114 | + |
| 115 | + twarc2 search --limit 5000 blacklivesmatter tweets.jsonl |
| 116 | + |
| 117 | +### Time |
| 118 | + |
| 119 | +You can also limit to a particular time range using `--start-time` and/or |
| 120 | +`--end-time`, which can be especially useful in conjunction with `--archive` |
| 121 | +when you are searching for historical tweets. |
| 122 | + |
| 123 | + twarc2 search --start-time 2014-07-17 --end-time 2014-07-24 '"eric garner"' tweets.jsonl |
| 124 | + |
| 125 | +If you leave off --start-time or --end-time it will be open on that side. So |
| 126 | +for example to get all "eric garner" tweets before 2014-07-24 you would just |
| 127 | +leave off the `--start-time`: |
| 128 | + |
| 129 | + twarc2 search --end-time 2014-07-24 '"eric garner"' tweets.jsonl |
| 130 | + |
| 131 | +## Stream |
| 132 | + |
| 133 | +The `stream` command will use Twitter's API |
| 134 | +[tweets/search/stream](https://developer.twitter.com/en/docs/twitter-api/tweets/filtered-stream/api-reference/get-tweets-search-stream) |
| 135 | +endpoint to collect tweets as they happen. In order to use it you first need to |
| 136 | +create one or more [rules]. For example: |
| 137 | + |
| 138 | + twarc2 stream-rules add blacklivesmatter |
| 139 | + |
| 140 | +You can list your active stream rules: |
| 141 | + |
| 142 | + twarc2 stream-rules list |
| 143 | + |
| 144 | +And you can collect the data from the stream, which will bring down any tweets that match your rules: |
| 145 | + |
| 146 | + twarc2 stream stream.jsonl |
| 147 | + |
| 148 | +When you want to stop you use `ctrl-c`. This only stops the stream but doesn't delete your stream rule. To remove a rule you can: |
| 149 | + |
| 150 | + twarc2 stream-rules delete blacklivesmatter |
| 151 | + |
| 152 | +### Sample |
| 153 | + |
| 154 | +Use the `sample` command to listen to Twitter's [tweets/sample/stream](https://developer.twitter.com/en/docs/twitter-api/tweets/sampled-stream/api-reference/get-tweets-sample-stream) API for a "random" sample of recent public statuses. |
| 155 | + |
| 156 | + twarc2 sample sample.jsonl |
| 157 | + |
| 158 | +### Users |
| 159 | + |
| 160 | +If you have a file of user ids you can fetch the user metadata for them with |
| 161 | +the `users` command: |
| 162 | + |
| 163 | + twarc users users.txt users.jsonl |
| 164 | + |
| 165 | +If the file contains usernames instead of user ids you can use the `--usernames` option: |
| 166 | + |
| 167 | + twarc2 users --usernames users.txt users.jsonl |
| 168 | + |
| 169 | +### Followers |
| 170 | + |
| 171 | +You can fetch the followers of an account using the `followers` command: |
| 172 | + |
| 173 | + twarc2 followers deray users.jsonl |
| 174 | + |
| 175 | +### Following |
| 176 | + |
| 177 | +To get the users that a user is following you can use `following`: |
| 178 | + |
| 179 | + twarc2 following deray users.jsonl |
| 180 | + |
| 181 | +The result will include exactly one user id per line. The response order is |
| 182 | +reverse chronological, or most recent followers first. |
| 183 | + |
| 184 | +### Timeline |
| 185 | + |
| 186 | +The `timeline` command will use Twitter's [user timeline API](https://developer.twitter.com/en/docs/twitter-api/tweets/timelines/api-reference/get-users-id-tweets) to collect the most recent tweets posted by the user indicated by screen_name. |
| 187 | + |
| 188 | + twarc2 timeline deray tweets.jsonl |
| 189 | + |
| 190 | +### Conversation |
| 191 | + |
| 192 | +You can retrieve a conversation thread using the tweet ID at the head of the |
| 193 | +conversation: |
| 194 | + |
| 195 | + twarc2 conversation 266031293945503744 > conversation.jsonl |
| 196 | + |
| 197 | +## Dehydrate |
| 198 | + |
| 199 | +The `dehydrate` command generates an id list from a file of tweets: |
| 200 | + |
| 201 | + twarc2 dehydrate tweets.jsonl tweet-ids.txt |
| 202 | + |
| 203 | +## Hydrate |
| 204 | + |
| 205 | +twarc's `hydrate` command will read a file of tweet identifiers and write out the tweet JSON for them using Twitter's [tweets](https://developer.twitter.com/en/docs/twitter-api/tweets/lookup/api-reference/get-tweets) |
| 206 | +API endpoint: |
| 207 | + |
| 208 | + twarc2 hydrate ids.txt tweets.jsonl |
| 209 | + |
| 210 | +Twitter API's [Terms of Service](https://dev.twitter.com/overview/terms/policy#6._Be_a_Good_Partner_to_Twitter) discourage people from making large amounts of raw Twitter data available on the Web. The data can be used for research and archived for local use, but not shared with the world. Twitter does allow files of tweet identifiers to be shared, which can be useful when you would like to make a dataset of tweets available. You can then use Twitter's API to *hydrate* the data, or to retrieve the full JSON for each identifier. This is particularly important for [verification](https://en.wikipedia.org/wiki/Reproducibility) of social media research. |
| 211 | + |
| 212 | +# Command Line Usage |
| 213 | + |
| 214 | +Below is what you see when you run `twarc2 --help`. |
| 215 | + |
1 | 216 | ::: mkdocs-click: |
2 | 217 | :module: twarc.command2 |
3 | 218 | :command: twarc2 |
0 commit comments