Skip to content

Commit

Permalink
fix: Exclude location fields when api_type is BULK (MeltanoLabs#51)
Browse files Browse the repository at this point in the history
In the current implementation of the tap, fields described with
`type=address` are correctly excluded when using `api_type=BULK`.

Although, this is not the case for geolocation fields (described with
`type=location`).

I'm not a Salesforce specialist per se, but know that the changes I'm
applying on this branch solved the issue described
[here](MeltanoLabs#42).

In my specific case I had a field like this:

```
                    "Geolocation__c": {
                        "type": [
                            "number",
                            "object",
                            "null"
                        ],
                        "properties": {
                            "longitude": {
                                "type": [
                                    "null",
                                    "number"
                                ]
                            },
                            "latitude": {
                                "type": [
                                    "null",
                                    "number"
                                ]
                            }
                        }
                    },
```

As it's a compound field, split over multiple properties (`longitude`
and `latitude`), the parent field `Geolocation__c` does not have much
value as the relevant information is propagated in the "sub-fields"
(`latitude` and `longitude`).

So what I'm doing in this PR is to ensure that:
- `Geolocation__c` is marked with `unsupported` in the schema metadata
(snippet below) while still ensuring that the "sub-fields" are correctly
queried.

Snippet of the excluded schema after this PR:
```
                {
                    "breadcrumb": [
                        "properties",
                        "Geolocation__c"
                    ],
                    "metadata": {
                        "inclusion": "unsupported",
                        "unsupported-description": "cannot query compound address fields with bulk API"
                    }
                },
```

Snippet of the "sub-fields" in the schema:

```
                    "Geolocation__Latitude__s": {
                        "type": [
                            "null",
                            "number"
                        ]
                    },
                    "Geolocation__Longitude__s": {
                        "type": [
                            "null",
                            "number"
                        ]
                    },
```
  • Loading branch information
dlouseiro authored and Daniela Angelova committed Sep 26, 2024
1 parent 9ef5c4d commit 48a5f42
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion tap_salesforce/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ def do_discover(sf: Salesforce, streams: list[str]):
f, mdata)

# Compound Address fields cannot be queried by the Bulk API
if f['type'] == "address" and sf.api_type == tap_salesforce.salesforce.BULK_API_TYPE:
if f['type'] in ("address", "location") and sf.api_type == tap_salesforce.salesforce.BULK_API_TYPE:
unsupported_fields.add(
(field_name, 'cannot query compound address fields with bulk API'))

Expand Down

0 comments on commit 48a5f42

Please sign in to comment.