Accessing live matches 403 error on Heroku

[rakemen]

My telegram bot is deployed on Heroku. I always get a 403 Forbidden error when I try to get live matches info or access any links on mapps.cricbuzz.com/cbzios/. When I try to run locally everything works fine. Could it be that all Heroku IP addresses are blocked by the cricbuzz website? Any luck deploying your code on Heroku?

[imaginarystarlord]

I am also getting the same problem

[rakemen]

I am also getting the same problem

Do you have any workarounds in mind?

[imaginarystarlord]

Yes

[rakemen]

@imaginarystarlord Can you share the idea, so that it may be helpful to others. Thanks!

[codophobia]

@rakemen Yes, it looks like ip address of the cloud servers are blocked. One way to get around this is to use a proxy address and then make a request. You can find many proxy ip addresses online.

export http_proxy=http://68.183.108.113:8080/
curl mapps.cricbuzz.com/cbzios/match/livematches

The above request works on azure and should also work on other cloud instances. So, the solution will be to set up "http_proxy" environment variable before using the library.

[imaginarystarlord]

we have to rotate IP with rotating user agent

[swainsandeep]

I have tried with the above ip in heroku,it is not working can anyone suggest any other ip,or where can I get those ips.

[swainsandeep]

@rakemen Yes, it looks like ip address of the cloud servers are blocked. One way to get around this is to use a proxy address and then make a request. You can find many proxy ip addresses online.

export http_proxy=http://68.183.108.113:8080/
curl mapps.cricbuzz.com/cbzios/match/livematches

The above request works on azure and should also work on other cloud instances. So, the solution will be to set up "http_proxy" environment variable before using the library.

@codophobia I have tried with the ip provided by you it is not working in heroku,can u suggest any other ip adress,can u guide me where I can get those ips.

[imaginarystarlord]

What u want to do?

[codophobia]

@swainsandeep https://free-proxy-list.net/

[swainsandeep]

@codophobia thanks for Sharing free proxy list.

I have tried one of the proxy in the provided site.Still same issue facing in Pythonanywhere site.

export http_proxy=http://51.38.71.101:8080

from pycricbuzz import Cricbuzz
c = Cricbuzz()
c.matches()

Getting this error:-

Traceback (most recent call last):
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1254, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1036, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 974, in send
self.connect()
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connection.py", line 184, in connect
conn = self._new_conn()
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f334a927358>: Failed to establish a new connection: [Errno 111] Con
nection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='51.38.71.101', port=8080): Max retries exceeded with url: http://mapps.cricbuzz.com/cbzios/matc
h/livematches (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f334a927358>: Failed t
o establish a new connection: [Errno 111] Connection refused',)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/pycricbuzz/cricbuzz.py", line 73, in matches
crawled_content = self.crawl_url(url)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/pycricbuzz/cricbuzz.py", line 10, in crawl_url
r = requests.get(url).json()
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/cricdcar/.virtualenvs/cricdcar/lib/python3.6/site-packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPConnectionPool(host='51.38.71.101', port=8080): Max retries exceeded with url: http://mapps.cricbuzz.com/cbzios/match/
livematches (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f334a927358>: Failed to
establish a new connection: [Errno 111] Connection refused',)))

[imaginarystarlord]

You have to rotate ips with rotating user agent, continuously, because these are free ips, they work sometimes, they may not work some times

[swainsandeep]

@imaginarystarlord imaginarystarlord Thanks for the reply.Can you share some code snippet for this.How to rotate ips using rotating user agent.

[swainsandeep]

@codophobia @imaginarystarlord Hi All,Anyone has any official document for cricbuzz for the usinng the api.Actually I raised this issue in pythonanywhere,they are asking for official documentation to to whitelist the api in Pythonanwhere.

[rakemen]

@codophobia @imaginarystarlord Hi All,Anyone has any official document for cricbuzz for the usinng the api.Actually I raised this issue in pythonanywhere,they are asking for official documentation to to whitelist the api in Pythonanwhere.

As far as I am aware, I haven't seen any documentation of this api, just the blog post link which codophobia has on this repo, maybe you can forward them the blog link and see if they can close the issue?