cannot get import to work

[davisford]

i'm using the docker container from here: https://hub.docker.com/r/transferwiseworkspace/pipelinewise -- with latest tag.

root@2f87f552bd60:/app# ./entrypoint.sh --version
PipelineWise 0.48.5 - Command Line Interface

i have mounted some tap/target files into the container via a volume. any attempt at running the import --dir /path/to/dir command results in the following:

root@2f87f552bd60:/app# ./entrypoint.sh import --dir /etc/pipelinewise
time=2022-10-03 19:14:19 logger_name=pipelinewise log_level=INFO message=Profiling mode not enabled
time=2022-10-03 19:14:19 logger_name=pipelinewise.cli.config log_level=INFO message=Searching YAML config files in /app
time=2022-10-03 19:14:19 logger_name=pipelinewise.cli.config log_level=INFO message=SAVING CONFIG
time=2022-10-03 19:14:19 logger_name=pipelinewise.cli.config log_level=INFO message=SAVING MAIN CONFIG JSON to /root/.pipelinewise/config.json
time=2022-10-03 19:14:19 logger_name=pipelinewise.cli.pipelinewise log_level=INFO message=ACTIVATING TAP STREAM SELECTIONS...
time=2022-10-03 19:14:19 logger_name=pipelinewise.cli.pipelinewise log_level=INFO message=
            -------------------------------------------------------
            IMPORTING YAML CONFIGS FINISHED
            -------------------------------------------------------
                Total targets to import        : 0
                Total taps to import           : 0
                Taps imported successfully     : 0
                Taps failed to import          : []
                Runtime                        : 0:00:00.000229
            -------------------------------------------------------

I even tried the sample files mentioned here: https://transferwise.github.io/pipelinewise/installation_guide/creating_pipelines.html because I thought maybe there was something wrong with my YAML, or it was expecting .yml vs .yaml extension or something...nope, still imports zero.

root@2f87f552bd60:/app# ls -alh /etc/pipelinewise/
total 28K
drwxr-xr-x 7 root root  224 Oct  3 19:03 .
drwxr-xr-x 1 root root 4.0K Oct  3 19:09 ..
-rw-r--r-- 1 root root 1.3K Oct  3 19:03 tap-mysql.yml
-rw-r--r-- 1 root root 6.3K Oct  3 13:55 tap-postgres.yaml
-rw-r--r-- 1 root root 3.1K Oct  3 13:55 target-snowflake.yaml

Here is the contents of tap-mysql.yml

root@2f87f552bd60:/app# cat /etc/pipelinewise/tap-mysql.yml
---
id: "fx"
name: "FX (Monolith)"
type: "tap-mysql"
owner: "[email protected]"

# Source connection details
db_conn:
  host: "localhost"
  port: 10602
  user: "my_user"
  password: "<PASSWORD>"                  # Plain string or Vault Encrypted password

target: "snowflake_test"                  # Target ID, should match the id from target_snowflake.yml
batch_size_rows: 100000                   # Batch size for the stream to optimise load performance

# Source to Destination Schema mapping
schemas:
  - source_schema: "fx"                   # You can replicate from multiple schemas
      target_schema: "fx_clear"           # Target schema in snowflake
      target_schema_select_permissions:   # Grant permission once the table created
        - grp_power
      tables:                             # List Tables to replicate
        - table_name: "table_one"
          replication_method: FULL_TABLE  # 1) FULL_TABLE replication
        - table_name: "table_two"         #
          replication_method: LOG_BASED   # 2) LOG_BASED replication
        - table_name: "table_three"       #
          replication_method: INCREMENTAL # 3) INCREMENTAL replication
          replication_key: "updated_at"   #    Incremental load needs replication key

Can someone explain to me what I'm missing? Validate also produces no errors:

root@2f87f552bd60:/app# ./entrypoint.sh validate --dir /etc/pipelinewise/
time=2022-10-03 19:23:47 logger_name=pipelinewise log_level=INFO message=Profiling mode not enabled
time=2022-10-03 19:23:47 logger_name=pipelinewise.cli.pipelinewise log_level=INFO message=Searching YAML config files in /etc/pipelinewise/
time=2022-10-03 19:23:47 logger_name=pipelinewise.cli.pipelinewise log_level=INFO message=Detected taps: set()
time=2022-10-03 19:23:47 logger_name=pipelinewise.cli.pipelinewise log_level=INFO message=Detected targets: set()
time=2022-10-03 19:23:47 logger_name=pipelinewise.cli.pipelinewise log_level=INFO message=Validation successful

[Samira-El]

Hi Davis,
Can you share the docker command or docker-compose config you're using to create this container?

[davisford]

docker run --entrypoint /bin/bash -it --volume $(pwd):/etc/pipelinewise transferwiseworkspace/pipelinewise:latest

where $(pwd) is the host dir that contains yaml specs for taps/targets mounted into the container at /etc/pipelinewise