[bigtent] Add Mailstrom like email provider detection

[ozten]

https://mailstrom.co/ is quite clever about being able to detect if a domain name is backed by gmail.com.

Dave Troy, one of the Mailstrom wizards, was kind enough to share their dark magick:

1. Is the domain name gmail (gmail.com, googlemail.com)?
2. Does the MX resolve to a google mail host?
3. If still not sure, we connect to Gmail via SMTP to see if it will accept an email to that address

Currently we do #1 to hand off to BigTent, but we should implement 2 and 3 to extend the % of email addresses which can be delegated to BigTent for authentication.

Potential issues:

1. Can we cache this for all email addresses for a single domain
2. How long should the cache live
3. If BigTent has an error, do we need a way to tell Persona to clear it's cache for that domain?

[ozten]

Also note: jaredhanson/passport-openid#11

[quack1]

I'm interested to work on this issue, but I'm a new contributor, so I don't really know where to start. Do you have some more infos please ?

[shane-tomlinson]

@ozten, @callahad, or @seanmonstar - can you guys fill @quack1 in with some context? It's always awesome to have contributors!

[seanmonstar]

If I understand correctly, this is to support Google Apps for your Domain address. Such that, if I use [email protected], it will detect that I use Google Apps, and defer to BigTent to use the Google Identity Bridge. Sound right?

[callahad]

@seanmonstar That's correct. I'm still not sure if implicitly supporting Google Apps is a good thing, but I've generally been ignoring this can of worms until we get Gmail and Hotmail out the door.

[seanmonstar]

Yea, I couldn't recommend a place to start in our code, because our code for Gmail is in big flux. Maybe in a week or 2...

[jaredhirsch]

@quack1 Maybe the best answer, for now, is to keep a close eye on the Gmail-related code :-)

Failing that, we currently have 431 other bugs, perhaps one of those is enticing?

[quack1]

Thanks for your answers. I had choosen this one because it was marked as 'good-first-bug', but now I'll look at another simple one!

[ozten]

@quack1 I can mentor you on this bug, as I filed it and I do think it is a good first bug. I'll coordinate with @seanmonstar and @callahad to make sure it works with what they are planning.

Warning - This code wouldn't be used right away. It also might never get used, if it turns out that we can't do Identity Bridging on Google Apps for your Domain, for whatever reason.

If you're still game...

This library can be built independently of current work. I'd recommend a new file server/lib/google_apps_discovery.js

[or a stand-alone node module...]

The interface would look like this:

var googleApps = require('./lib/google_apps_discovery');

googleApps('[email protected]', function(err, isGoogleDomain) {
    if (err) {
        ...
    } else if (googleDomain === true) {
        // This email addresses is backed by Google Apps for your domain, we can do Gmail Identity Bridging
    }
});

This is assuming you own quackrules.org and it's setup to be a Google Apps for your domain website. You'll need one of those two test this, or atleast find someone else with an email address that is backed by Google Apps for your domain.

So the one function in the module takes an email address and a callback function. The callback will be called with an error as well as a boolean. The boolean indicates if the domain name is backed by Google Apps.

This bug is a dependency for supporting people's personal domain names via Gmail OpenID. There are other pieces that need to be tested and figured out, but this portion is pretty independent of those others.

[quack1]

@ozten Thank you for mentoring me on this bug! I'm working on a script to automate the setup of an environment developement on ubuntu, and I'll look at this bug much closer just after that and I'll come back to you for more informations!

[quack1]

Hi @ozten, I've just finished my work on the script, so I will start working on this bug, and I contact you when I have a "plan" to see with you if I go in the right direction!

[ozten]

That sounds great!

[quack1]

@ozten Hi! I look at this a little and I would like to clarify everything! :)

We need to create a simple function, which can said if a given email address if backed by Google Apps or not.
This can be done in a new file (/lib/google_apps_discovery.js ?) in the browserid repository.

This function will do the previous steps @ozten mentionned in the first post of this thread:

1. Check if the domain is 'google.com' or 'gmail.com'
2. Check if the MX resolves to a 'google.com' or 'gmail.com' host
3. Connect to one of the smtp server of gmail.com and try to send a mail to the address

Finally, a callback function is returned with :

• err == null and isGoogleDomain == True if the address mail is backed by Google
• err == null and isGoogleDomain == False if the address mail is not backed by Google
• err != null and isGoogleDomain == False if the function cannot verify if the domain is backed or not by Google Apps

Am I correct with this ? I setup a fake email on Google Apps with the free month offer to test this, I will purchase some month pass if I need more time to work on!

FIY, I have a 2h1/2 train travel tomorrow night, so I think I'll work on this during the trip!

[ozten]

Howdy @quack1, yes I think you've got it.

I think you'll want to look at using DNS as well as how the SMTP protocol works. It's possible that you don't have to send an email, but instead start the SMTP transmission and get far enough to see if it is a recognized email address.

I'd recommend starting with Steps 1 and 2, before tackling Step 3, which might be harder.

Happy hacking!

[jjulian]

Here's an example SMTP session that can be used for step 3:

-> "220 mx.google.com ESMTP s6si1988268eew.153 - gsmtp\r\n"
<- "EHLO \r\n"
-> "250-mx.google.com at your service, [10.0.0.1]\r\n"
-> "250-SIZE 35882577\r\n"
-> "250-8BITMIME\r\n"
-> "250-STARTTLS\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "MAIL FROM:<[email protected]>\r\n"
-> "250 2.1.0 OK s6si1988268eew.153 - gsmtp\r\n"
<- "RCPT TO:<[email protected]>\r\n"
-> "250 2.1.5 OK s6si1988268eew.153 - gsmtp\r\n"
<- "QUIT\r\n"
-> "221 2.0.0 closing connection s6si1988268eew.153 - gsmtp\r\n"

You'll have to sign in with and send a valid Google email address in the MAIL FROM. The response to the RCPT is the key: you'll probably see a 250 or a 550.

[quack1]

OK, that's great!

Yes, I've test directly from the cli to dig MX and start an SMTP session with telnet to see if it was possible to do that this way and to see how I can detect both cases.

[ozten]

Read these to see how awesome this feature can be:

• https://www.tbray.org/ongoing/When/201x/2013/06/07/Why-findIDP
• http://blog.romeda.org/2013/06/thoughts-on-signin.html

[blaine]

Thanks, Austin! What I've failed to mention in the blog post, but is obvious when you use our sign in, is that we support not just Google sign-in, but Hotmail and Yahoo with MX detection, and Facebook and Twitter sign-in as fallback. I want to add support for loads more providers, too, including Persona!

The one thing that's critical for me is that I as the web site developer control the whole UX and UI for sign-in. I can't / won't rely on something like BigTent unless it's strictly providing back-end support for my sign-in flow. Likewise Persona; if I know that my user is already familiar with Persona and has an account setup there, I can send them through the Persona flow – if not, I'll use something they are familiar with. Getting the UI / UX right is absolutely the most important thing in all this. :-)

[quack1]

@ozten I have a (dummy) question. I have write some code to do the first two steps, but how can I test it locally ? I mean, without hacking all the code to make it use my function to know which server contact.

I would like to only start "this" function... I wrote a sample test file but it doesn't look like it works well either.. (BTW, I'm quite a noob with Node.js).

lib/google_apps_discovery.js --> https://gist.github.com/quack1/5817334
tests/google_apps_discovery_tests.js --> https://gist.github.com/quack1/5817352
$ npm test --> https://gist.github.com/quack1/5817359

[chemistrydioxide]

Option 2 is something that I have already proposed as well. However, some people think that Google Apps for your Domain users should rather run their own IdP instead of using BigTent.

Option 3 is a very bad thing to do. It allows Google to usurp any random email address they want and steal accounts. Another reason why 3 is bad is that there are several domains in Google's database which Google thinks to belong to Google when in fact, they have been moved to a different mail service. I also don't see which use cases should be covered by 3. Either an email address belongs to Google - then we should be able to tell that by DNS lookup. Or the email address doesn't live on Google's servers, then we can tell that by DNS as well.

[callahad]

Hi! To help us better focus, I'm "closing" all issues that have been open for more than six months. These have been tagged "cleanup-2014" so that we can go back and review them in the future.

For more information, check out this thread: http://thread.gmane.org/gmane.comp.mozilla.identity.devel/7394

If you believe this bug is still a major issue for you, please comment, submit a pull request, or discuss it on our mailing list: https://lists.mozilla.org/listinfo/dev-identity

Sorry for GitHub notification spam!