Request more data per API call?

[igorbrigadir]

I'm new to R, so I don't know if i've missed something - but it looks like you could speed up data collection in a few places by setting a larger count parameter for endpoints:

https://dev.twitter.com/rest/reference/get/statuses/user_timeline - Defaults to 20, can request 200 tweets per request

https://dev.twitter.com/rest/reference/get/search/tweets - Can request 100 tweets per request - currently defaults to 25 is that right?

[MemoWalk]

From your second link:

„The number of tweets to return per page, up to a maximum of 100. Defaults to 15. This was formerly the “rpp” parameter in the old Search API.“

I think this is what you are asking about?

I am an R novice too, yet I have been querying some large amount of data since mid August - I can tell you, the parameter „number of tweets per page“ does not bother the API as programmed in R.

But you actually can speed up data collection by issuing your request via app auth.

I have already written some code that just does this - login as app, see my issue #97 .

Perhaps in half a year I have both time and ability to integrate this into the twitteR package, yet I still do not use this approach. At the moment I am delaying the single queries using the R command

Sys.sleep(22)

In my setting this works fairly well and the „... waiting and trying ... “ message from Twitter does not appear all too often. You just need some time (and probably you are already applying this).

Hope it helps.

[igorbrigadir]

Yep - for search API, definitely makes sense to use Application Auth - 450 Requests per 15 min window vs 180 https://dev.twitter.com/rest/public/rate-limits

[MemoWalk]

Are you able to implement it?

Because I am (at the moment) not but would be eager to learn and help wherever possible.

Some time ago I have already looked in the source of twitteR - I think the change had to be implemented in

comm.R

because there is the function

doRppAPICall

which is called in

search.R

-- If we could add some if/else clause kind of „app_auth = yes“ and supply it with the bearer token, we should be able to rise the number of requests.

[geoffjentry]

Most API calls - via doRppAPICall - fetch things in batches of 100. It would probably be bertter to have it tunable for specific endpoints. I'll try to look at that if I get a chance, or if an intrepid user wants to submit a PR so much the better.