API Key Discussion

[kylewest520]

Hi! This is a great package with outstanding documentation. Well done and thank you for making it public!

I am working on a similar project and am recently facing a new issue regarding connecting to the API. I am working with PHP, not directly your python package (though I have used your python library in the past). I'm hoping that you might have some insight to the API issue since you have been working with it quite a bit, and my issue isn't specific to PHP.

Have you explored ways to grab a user's SWID and espn_s2 cookies automatically based on their ESPN login username & password? I have code that allows me to connect to the API with username & password and I am able to fetch the 2 cookies for any user. The process first grabs and API key before grabbing the cookies. Just recently the code stopped working and I am receiving an error message related to "Authorization header missing or invalid" even though the authorization header is the username & password.

I don't mean for this to sound like I am asking for specific debugging help. Rather I hope to kickoff a discussion and pick your brain about if you have seen similar challenges, different approaches for grabbing a user's cookies, or any recent changes to the API that might have caused this.

Best,
Kyle

[cwendt94]

Hey Kyle, I'm glad you have found this package helpful and thank you for opening up a discussion!

I have recently ran into the same issue with ESPN's login API via username and password. It looks like ESPN has recently updated their login authentication and added extra security. I have done some investigation looking at the API calls when logging in via the web browser and documented it here #100. I'm going to copy my findings here as I think Discussions is a better platform to talk about it than a issue ticket.

I have been looking at how web authenticates and it looks like they are using google recaptcha for extra authentication. Here are the logs when I update using these endpoints

url_api_key = 'https://registerdisney.go.com/jgc/v6/client/ESPN-ONESITE.WEB-PROD/api-key?langPref=en-US'
url_login = 'https://registerdisney.go.com/jgc/v6/client/ESPN-ONESITE.WEB-PROD/guest/login?langPref=en-US'
{'data': None, 'error': {'keyCategory': 'FAILURE_BY_DESIGN', 'conversationId': None, 'correlationId': '88778c0d-30bc-493f-a651-f3887f2ca1fa', 
'errors': [{'code': 'AUTHORIZATION_CREDENTIALS', 'category': 'FAILURE_BY_DESIGN', 'inputName': None, 'errorId': '1c328375-ad5b-4e02-b6ac-b02f94b15d50', 
'timestamp': '2020-09-09T19:21:55.675-0700', 
'data': {'type': 'GenericReasonCodeErrorData', 'reasonCode': 'PALOMINO_CHECK_FAILED'}, 'developerMessage': None, 'content': None}]}}

Im not sure what 'PALOMINO_CHECK_FAILED means

On the web ESPN calls google API to get a key right before the login call. The call sequence looks like this

Get a API Key https://registerdisney.go.com/jgc/v6/client/ESPN-ONESITE.WEB-PROD/api-key?langPref=en-US
Get a recaptcha https://www.google.com/recaptcha/api2/reload?k=6LdIg6MZAAAAAFYsKbfC_YLPlx4DZ3U7IJMcOI67
Then login https://registerdisney.go.com/jgc/v6/client/ESPN-ONESITE.WEB-PROD/guest/login?langPref=en-US

I have also been trying to see which endpoint mobile authenticates too and see what authentication they are using. However they are now using https so I cannot see auth url path.

I would love some help if anyone has any ideas or experience with google recaptcha API

[kylewest520]

Looking at both of our error messages, I wonder if I need to add a parameter from the recaptcha and send that to the login URL along with username & password.

[cwendt94]

I have tried that endpoint as well and have seen the same issue.

The recaptcha step is definitely new and I see them pass it in the header now for the login API g-recaptcha-v3-response
They are making a POST call to this endpoint https://www.google.com/recaptcha/api2/reload?k=6LdIg6MZAAAAAFYsKbfC_YLPlx4DZ3U7IJMcOI67 to get that key.

However I have not been able to crack what information they are passing in the POST to get the key back. In the call the request payload type is content-type: application/x-protobuffer

[cwendt94]

I also wonder if the ESPN-FANTASYLM-PROD API might accept different authentication then web login API ESPN-ONESITE.WEB-PROD. One might be for web access and another might be for mobile/app. I just don't know of a way to see how mobile/app authenticates.

[kylewest520]

Do you think it's as simple as grabbing a new key from the reCAPTCHA step and passing that (along with username & password) to the login URL? I thought reCAPTCHA was for verifying "I am a human, not a robot". Is there a way to bypass those checks with a reCAPTCHA key?

[cwendt94]

It might be that simple, I just don't know how their API works. I thought the same thing for reCAPTCHA but I guess they also have an API for authentication.

When I have some time I am going to dig into https://www.google.com/recaptcha/api2/reload?k=6LdIg6MZAAAAAFYsKbfC_YLPlx4DZ3U7IJMcOI67 API call. If you open developer tools on a web browser and login into ESPN fantasy you should also see the reCAPTCHA reload call before the ESPN login API call.

[xpose2000]

I wanted to update here since no one seemed to follow up. The problem was the captcha solving. ESPN just recently added it and all you had to do was pass it along with the request just like before. However, they changed something else as of a few days ago and it is back to saying PALOMINO_CHECK_FAILED. Not quite sure what it is.

[cwendt94]

Were you able to programmatically generate a captcha token? I remember trying to use this endpoint https://www.google.com/recaptcha/api2/reload?k=6LdIg6MZAAAAAFYsKbfC_YLPlx4DZ3U7IJMcOI67 but seemed to require more input to generate the token.

When I was playing around with passing the captcha token, I got PALOMINO_CHECK_FAILED as well.

[xpose2000]

[cwendt94]

Thanks for the update and information! I am going to try to dig back into this over the weekend.

[kylewest520]

I'm glad to know that it sounds like there was a workaround for the orignal reCAPTCHA challenge. @xpose2000 do you mind sharing which service you were using to solve the captcha? If I can get that part working I can help with working through the new issue. Thanks!

[DesiPilla]

I was also struggling with this, but I believe I have found a work-around for it. I wrote a function get_credentials() that uses Selenium to open up a browser where a user can log in (the default chrome profile is used, which may log the user in automatically if they have used espn.com recently). The swid and espn_s2 cookies are then fetched automatically and can be stored for future use.

I tried getting selenium to locate the login fields, but wasn't having much luck. Ideally, this could automate this process without requiring user interaction. This could potentially improve the function.

I hope this helps you. I'll leave it up to you on how/where would be best to integrate this into your repo, but this was how I tackled the issue. I've linked the corresponding code in my repo, but here it is for ease:

def get_credentials(user=None):
    '''
    This function identifies the SWID and ESPN_S2 for a user using Selenium.
    
    The function will identify the default Chrome profile for a user on the
    machine, and open up a browser using this account.
    
    The user will be prompted to sign in to their ESPN account if they are
    not auto-signed in by their browser.
    
    The swid and espn_s2 cookies are identified and returned.
    '''
    # Get list of users on computer
    users = os.listdir('C://Users')
    for i in ['All Users', 'Default', 'Default User', 'desktop.ini', 'Public']:
        users.remove(i)
    print("[FETCHING CREDENTIALS] All users: ", users)
    
    if user is None:
        # Select first user
        user = users[0]
        print("[FETCHING CREDENTIALS] Using user: {}".format(users[0]))
    
    # Locate and use the default Chrome profile for the user
    print("[FETCHING CREDENTIALS] Locating default Chrome profile...")
    options = webdriver.ChromeOptions() 
    options.add_argument(r'--user-data-dir=C:/Users/{}/AppData/Local/Google/Chrome/User Data'.format(user))
    options.add_argument('--profile-directory=Default')

    # Instantiate Chrome instance using Selenium and the default Chrome profile
    print("[FETCHING CREDENTIALS] Instantiating Chrome browser...")
    DRIVER_PATH = 'C://chromedriver.exe'
    try: 
        driver = webdriver.Chrome(executable_path=DRIVER_PATH, chrome_options=options)
    except InvalidArgumentException as e:
        if "user data directory is already in use" in str(e):
            #driver.close()  # Close window
            raise Exception("Chrome is already open in another window. Please close all other Chrome windows and re-launch.")
    
    # Navigate to ESPN website for league and login
    driver.get('https://fantasy.espn.com/')

    # Check if user is logged in automatically by browser
    cookies = [cookie["name"] for cookie in driver.get_cookies()] # Get list of cookie names
    if ("SWID" not in cookies) or ("espn_s2" not in cookies):
        # Wait for user to log in maunally
        print("[FETCHING CREDENTIALS] Login to ESPN account in browser.")
        driver.find_element_by_xpath('//*[@id="global-user-trigger"]').click()
        #driver.find_element_by_xpath('//*[@id="global-viewport"]/div[3]/div/ul[1]/li[7]/a').click()
        while True:
            time.sleep(5)
            cookies = [cookie["name"] for cookie in driver.get_cookies()] # Get updated list of cookie names
            if ("SWID" in cookies) and ("espn_s2" in cookies):
                print("[FETCHING CREDENTIALS] Login detected.")
                break;            
            print("[FETCHING CREDENTIALS] Login not detected... waiting 5 seconds...")
    else:
        print("[FETCHING CREDENTIALS] Login detected.")
        pass
           
    # Identify cookies for user
    swid, espn_s2 = None, None
    cookies = driver.get_cookies()
    for cookie in cookies:
        if cookie['name'] == 'SWID':
            swid = cookie['value'][1:-1]
        if cookie['name'] == 'espn_s2':
            espn_s2 = cookie['value']
    
    #return driver
    if swid is None:
        raise Exception("[FETCHING CREDENTIALS] ERROR: SWID cookie not found.")
    if espn_s2 is None:
        raise Exception("[FETCHING CREDENTIALS] ERROR: SWID cookie not found.")    
    
    # Close the browser        
    driver.close()
    
    print("[FETCHING CREDENTIALS] ESPN Credenitals:\n[FETCHING CREDENTIALS] ---------------------")
    print("[FETCHING CREDENTIALS] swid: {}\n[FETCHING CREDENTIALS] espn_s2: {}".format(swid, espn_s2))
    return swid, espn_s2

For storing credentials:

try:
    login = pd.read_csv('login.csv').iloc[:, 1:]   # Load cached credentials if file exists
except:
    login = pd.DataFrame(columns=['manager', 'league_name', 'league_id', 'swid', 'espn_s2'])
    creds = {'manager':self.teams[1].owner, 'league_name':self.settings['name'], 'league_id':self.league_id, 'swid':self.swid, 'espn_s2':self.espn_s2}
    login = login.append(creds, ignore_index=True)
    login.to_csv('login.csv', index=False) 
    print("[BUILDING LEAGUE] League credentials saved.")

[cwendt94]

This is a great alternative to paying for a captcha solving service!

I am planning on removing the username and password feature because there is no easy way to programmatically retrieve a user swid and espn_s2. Leaving it up to the caller to get their swid and espn_s2 for private leagues I think is the best option.