Skip to content

Commit

Permalink
Implement where function for selecting columns by type
Browse files Browse the repository at this point in the history
  • Loading branch information
emilyriederer committed Sep 5, 2021
2 parents a81f6ed + 51f4330 commit a19222f
Show file tree
Hide file tree
Showing 12 changed files with 91 additions and 19 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ The complete list of macros included are:
- `not_one_of(string_list, relation or list)`
- `matches(string, relation)`
- `everything(relation)`
- `where(fn, relation)` where `fn` is the string name of a [Column type-checker](https://docs.getdbt.com/reference/dbt-classes/#column) (e.g. "is_number")

Note that all of the select-helper functions that take a relation as an argument can optionally be passed a list of names instead.

Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'dbtplyr'
version: '0.1.1'
version: '0.2.0'
config-version: 2
require-dbt-version: ">=0.19.0"

Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json", "dbt_version": "0.19.0", "generated_at": "2021-04-10T16:37:46.220893Z", "invocation_id": "02c1793a-12e3-403e-94f3-88d1e09499b8", "env": {}}, "nodes": {}, "sources": {}, "errors": null}
{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json", "dbt_version": "0.20.1", "generated_at": "2021-09-05T16:01:55.753633Z", "invocation_id": "5fde9619-1fac-4388-b0f3-13a0a7d66850", "env": {}}, "nodes": {}, "sources": {}, "errors": null}
24 changes: 12 additions & 12 deletions docs/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions integration_tests/data/data_types.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
n_a,cat_b
1,a
3 changes: 3 additions & 0 deletions integration_tests/models/as_values.sql
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@ select
-- everything
cast({{ "'" ~ dbtplyr.everything(cols) | join(',') ~ "'"}} as {{cast_to}}) as everything, -- no items

-- where
cast({{ "'" ~ dbtplyr.where("is_number", ref('data_types')) | join(',') ~ "'"}} as {{cast_to}}) as where_number,

1 as x -- dummy to allow trailing commas


6 changes: 5 additions & 1 deletion integration_tests/models/schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -72,4 +72,8 @@ models:
- name: ends_with_z
tests:
- accepted_values:
values: ['']
values: ['']
- name: where_number
tests:
- accepted_values:
values: ['n_a']
30 changes: 29 additions & 1 deletion macros/macro.yml
Original file line number Diff line number Diff line change
Expand Up @@ -148,4 +148,32 @@ macros:
arguments:
- name: relation
type: 'Relation'
description: 'Relation passed with Jinja ref syntax or list of column namess'
description: 'Relation passed with Jinja ref syntax or list of column names'

- name: where
description: >
Returns list of string column names from relation
which are identified as a specific type
arguments:
- name: fn
type: 'String'
description: >
String with name of function for dbt Column class which identifies type.
Currently supports 'is_numeric', 'is_number', 'is_float', 'is_string'
- name: relaiton
type: 'Relation'
description: >
Relation passed with Jinja ref syntax. Note that unlike other macros,
a list of column names is not supported for this function because
metadata from the underlying Column class is required
- name: map
description: >
Utility function for applying a function to a list.
This function is intended for internal use only.
The primary use is currently to support the where function
arguments:
- name: input_list
description: List of objects
- name: fn
description: Function to apply
12 changes: 12 additions & 0 deletions macros/map.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{% macro map(input_list, fn) %}

{% set results_list = [] %}
{% for l in input_list %}
{% if fn(l) %}
{{ results_list.append(l) or "" }}
{% endif %}
{% endfor %}

{{ return(results_list) }}

{% endmacro %}
23 changes: 22 additions & 1 deletion macros/select_helpers.sql
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,25 @@
{%set cols = dbtplyr.get_column_names(relation) %}
{{return(cols)}}

{% endmacro %}
{% endmacro %}

{% macro where(fn, relation) %}

{% set cols = adapter.get_columns_in_relation(relation) %}
{% set results_list = [] %}

{% for c in cols %}
{% if fn == "is_character" and c.is_character() %}
{{ results_list.append(c.name) or "" }}
{% elif fn == "is_number" and c.is_number() %}
{{ results_list.append(c.name) or "" }}
{% elif fn == "is_float" and c.is_float() %}
{{ results_list.append(c.name) or "" }}
{% elif fn == "is_numeric" and c.is_numeric() %}
{{ results_list.append(c.name) or "" }}
{% endif %}
{% endfor %}

{{return(results_list)}}

{% endmacro %}
3 changes: 2 additions & 1 deletion models/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,9 @@ The complete list of macros included are:
- `not_one_of(string_list, relation or list)`
- `matches(string, relation)`
- `everything(relation)`
- `where(fn, relation)` where `fn` is the string name of a [Column type-checker](https://docs.getdbt.com/reference/dbt-classes/#column) (e.g. "is_number")

Note that all of the select-helper functions that take a relation as an argument can optionally be passed a list of names instead.
Note that all of the select-helper functions (except `where`) that take a relation as an argument can optionally be passed a list of names instead.

You may access documentation for each individual macro using the navigation bar on the left-hand side of the screen.

Expand Down

0 comments on commit a19222f

Please sign in to comment.