Lack of support for Private and Restricted Google APIs

[husseyd]

---

Hi,

I'm working at an org where all traffic to Google needs to use Private or Restricted Google APIs. This involves DNS config with CNAMEs that point services like bigquery.googleapis.com to restricted.googleapis.com. This is for VPC Service Controls, and ensures all traffic stays on off of the internet and within client networks/VPCs.

After a lot of headscratching about why this was not working with bigrquery (I received VPC SC errors, and digging into GCP logs saw that requests were coming over the internet from on-prem and thus being blocked Googleside), I dug into the source code on this repo and found the issue.

Generally client libraries and SDKs talk to GCP services at [service].googleapis.com - eg https://bigquery.googleapis.com.
At R/bq-request.R, the URLs/hosts used to reach BQ are hardcoded:

base_url <- "https://www.googleapis.com/bigquery/v2/"
upload_url <- "https://www.googleapis.com/upload/bigquery/v2/"

Due to the generic 'www.googleapis.com', the Google-provided processes for using private/restricted googleapis are blocked when using bigrquery. googleapis.com itself can't really be CNAMED to [restricted || private].googleapis.com as not all services (maps, analytics) are suitable for this.

Anyway this was a long winded way of asking - please consider one of the following options:

1. Update base_url and upload_url globally to:

    base_url <- "https://bigquery.googleapis.com/bigquery/v2/"
    upload_url <- "https://bigquery.googleapis.com/upload/bigquery/v2/"

2. Or - provide a mechanism (like a boolean flag) for users to use the 'bigquery.googleapis' urls above instead of the current generic 'www.googleapis.com'

Reference:

• BigQuery API ref
• VPC Service Controls Overview
• Private and Restricted Google APIs networking and DNS

[craigcitro]

FWIW I think switching the defaults to https://bigquery.googleapis.com/... is the right move; these days all the discovery docs point to that as the base URL, and (as is the case here) there's functionality that only works against the new root URL.

[husseyd]

FWIW I think switching the defaults to https://bigquery.googleapis.com/... is the right move; these days all the discovery docs point to that as the base URL, and (as is the case here) there's functionality that only works against the new root URL.

That would be awesome. If I can do anything to help that happen please let me know.

[hadley]

@husseyd do you want to do a PR?

[husseyd]

Opened PR #542 to address.