Manage "Rate limited .... blocking for a minute and retrying up to 119 times ..."

[dcabanillas]

Hi I know that we have twitter limits but I want to know how we should work with these limits.

I'm working with
for(i in 1:length(query)){
search_twitter_and_store(query[i], geocode='41.4242,2.1705,6km')
}

But with i=35 I'm getting the first
"Rate limited .... blocking for a minute and retrying up to 119 times ..."
"Rate limited .... blocking for a minute and retrying up to 118 times ..."
...

"Rate limited .... blocking for a minute and retrying up to 113 times ..."

When i arrives to 114 I am getting the next error
Error in curl::curl_fetch_memory(url, handle = handle) :
Timeout was reached

First question, how I can avoid the error?

Second question, how I can know how far/near I'm to the limit?

Third question Should I use the param retryOnRateLimit?

Thanks a lot.

[MemoWalk]

I am simply using Sys.sleep(22) which is a pretty good approximation to spread the calls out over the time windows. An alternative would be using application-oauth, which allows more connections.

[geoffjentry]

@dcabanillas as @MemoWalk mentioned you'll want to take steps to slow yourself down. There's also a function getCurRateLimitInfo() which will show you where you stand.

As for retryOnRateLimit - the intention is to let you run something as long as possible in order to get your results, only you can decide if that's right for you. The timeout error is weird, if you have your exact code I can try to replicate it

[NesquikMike]

Hi, @geoffjentry & @MemoWalk I'm relatively new to twitter API and R I I'm doing a project in political psychology for my masters. I'm also being Rate Limited at the moment, and I'm just a bit worried that I will get blacklisted. I want to use the sys.sleep(22) to delay number of request so I don't get blacklisted, but I don't know how to incorporate it into my code. Here is an example of the code I'm running at the moment:

for(i in 1:nrow(huntsmanRandomUnpro)) { huntsmanRandomUnpro$accessible_tweets[i] <- try(length(userTimeline(user = huntsmanRandomUnpro[i, 2], n = 3200, includeRts = FALSE, retryOnRateLimit = 2000))) }

This is for about 1700 different users, how could I insert sys.sleep into that code to ensure I don't get blacklisted for making too many requests?

[NesquikMike]

Never mind, I managed to do it by just inserting Sys.sleep into the for loop:
for(i in 1:nrow(huntsmanRandomUnpro)) { huntsmanRandomUnpro$accessible_tweets[i] <- try(length(userTimeline(user = huntsmanRandomUnpro[i, 2], n = 3200, includeRts = FALSE, retryOnRateLimit = 2000))) sys.sleep(30) }