Schemachange version 4.0.0 not picking up variable User from created connections.toml file #317

BlessingZondi · 2025-01-17T22:25:35Z

With the recent update of schemachange to version 4 and requiring to use a connections.toml file for authentication, it seems as though schemachange is not picking up variables created and populated in the connections.toml file.

My yml script creates the connections.toml file in the current working directory of the pipeline in azure devops and within the same script I have tried to validate if the file is created and what is the current working dir of where the file is created and also the contents of the file, all of this is handled by the script and the script displays all these details (the displaying of contents of the file was a way of testing and validating if the file is indeed created and populated).

I am not sure if the manner at which I pass these variables to schemachange is incorrect or if there is any other thing that is incorrect in my yml script. If anyone can assist? Maybe I could be missing something!

Below is the yml script that I have tried to put together:
trigger:
branches:
include:
- main
paths:
include:
- /Migrations
pool:
vmImage: 'ubuntu-latest'

variables:

group: sf_insure_var

steps:

task: UsePythonVersion@0
inputs:
versionSpec: '3.x'
addToPath: true
script: |
python -m pip install --upgrade pip
pip install schemachange
displayName: 'Install schemachange'
script: |
echo "Creating connections.toml file"
echo "[connections.sta_prod]" > connections.toml
echo "user = '$(user)'" >> connections.toml
echo "password = '$(password)'" >> connections.toml
echo "account = '$(account)'" >> connections.toml
echo "warehouse = '$(warehouse)'" >> connections.toml
echo "database = '$(database)'" >> connections.toml
echo "schema = '$(schema)'" >> connections.toml
echo "role = '$(role)'" >> connections.toml

echo "Current working directory:"
pwd
echo "Listing directory contents:"
ls -al
echo "Contents of connections.toml:"
cat connections.toml
displayName: 'Validate connections.toml file creation and directory'
script: |
schemachange -f $(System.DefaultWorkingDirectory) -c $(System.DefaultWorkingDirectory)/connections.toml -a sta_prod
displayName: 'Run schemachange'

pearsall-will · 2025-01-17T22:33:02Z

@BlessingZondi Do you have any errors you could share from schemachange logs?

BlessingZondi · 2025-01-18T12:34:59Z

@pearsall-will Good day,

Error logs below:

2025-01-17T21:48:28.5325707Z ##[section]Starting: Run schemachange
2025-01-17T21:48:28.5330400Z ==============================================================================
2025-01-17T21:48:28.5330545Z Task : Command line
2025-01-17T21:48:28.5330611Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2025-01-17T21:48:28.5330724Z Version : 2.246.1
2025-01-17T21:48:28.5330806Z Author : Microsoft Corporation
2025-01-17T21:48:28.5330878Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2025-01-17T21:48:28.5330985Z ==============================================================================
2025-01-17T21:48:28.7731158Z Generating script.
2025-01-17T21:48:28.7732263Z Script contents:
2025-01-17T21:48:28.7732792Z schemachange -f /home/vsts/work/1/s -c /home/vsts/work/1/s/connections.toml -a stage_insure_prod
2025-01-17T21:48:28.7733164Z ========================== Starting Command Output ===========================
2025-01-17T21:48:28.7747773Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/307cb437-615a-4875-8769-74d69080f808.sh
2025-01-17T21:48:29.8867554Z �[2m2025-01-17T21:48:29.885588Z�[0m [�[32m�[1minfo �[0m] �[1mschemachange version: 4.0.0 �[0m
2025-01-17T21:48:29.8879070Z -a, --snowflake-account is deprecated. It will be ignored in future versions.
2025-01-17T21:48:29.8879703Z [DEPRECATED - Set in connections.toml instead.] The name of the snowflake account (e.g. xy12345.east-us-2.azure, xy12345.east-us-2.azure.privatelink, org-accountname, org-accountname.privatelink)
2025-01-17T21:48:29.8884239Z �[2m2025-01-17T21:48:29.887906Z�[0m [�[32m�[1minfo �[0m] �[1mUsing root folder �[0m �[36mroot_folder�[0m=�[35m/home/vsts/work/1/s�[0m
2025-01-17T21:48:29.8884944Z �[2m2025-01-17T21:48:29.888101Z�[0m [�[32m�[1minfo �[0m] �[1mUsing variables �[0m �[36mvars�[0m=�[35m{}�[0m
2025-01-17T21:48:29.8905905Z Traceback (most recent call last):
2025-01-17T21:48:29.8906479Z File "/opt/hostedtoolcache/Python/3.12.8/x64/bin/schemachange", line 8, in
2025-01-17T21:48:29.8907133Z sys.exit(main())
2025-01-17T21:48:29.8907340Z ^^^^^^
2025-01-17T21:48:29.8907545Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/schemachange/cli.py", line 65, in main
2025-01-17T21:48:29.8907775Z session = SnowflakeSession(
2025-01-17T21:48:29.8907928Z ^^^^^^^^^^^^^^^^^
2025-01-17T21:48:29.8908161Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/schemachange/session/SnowflakeSession.py", line 85, in init
2025-01-17T21:48:29.8908428Z self.con = snowflake.connector.connect(**connect_kwargs)
2025-01-17T21:48:29.8908606Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-17T21:48:29.8908821Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/snowflake/connector/init.py", line 55, in Connect
2025-01-17T21:48:29.8909069Z return SnowflakeConnection(**kwargs)
2025-01-17T21:48:29.8909257Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-17T21:48:29.8909486Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/snowflake/connector/connection.py", line 457, in init
2025-01-17T21:48:29.8909723Z self.connect(**kwargs)
2025-01-17T21:48:29.8909969Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/snowflake/connector/connection.py", line 733, in connect
2025-01-17T21:48:29.8911186Z self.__config(**kwargs)
2025-01-17T21:48:29.8914721Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/snowflake/connector/connection.py", line 1227, in __config
2025-01-17T21:48:29.8917668Z Error.errorhandler_wrapper(
2025-01-17T21:48:29.8918106Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/snowflake/connector/errors.py", line 284, in errorhandler_wrapper
2025-01-17T21:48:29.8918559Z handed_over = Error.hand_to_other_handler(
2025-01-17T21:48:29.8918844Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-17T21:48:29.8919707Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/snowflake/connector/errors.py", line 342, in hand_to_other_handler
2025-01-17T21:48:29.8926276Z connection.errorhandler(connection, cursor, error_class, error_value)
2025-01-17T21:48:29.8926757Z File "/opt/hostedtoolcache/Python/3.12.8/x64/lib/python3.12/site-packages/snowflake/connector/errors.py", line 215, in default_errorhandler
2025-01-17T21:48:29.8927125Z raise error_class(
2025-01-17T21:48:29.8927526Z snowflake.connector.errors.ProgrammingError: 251005: User is empty
2025-01-17T21:48:29.9626339Z
2025-01-17T21:48:29.9668527Z ##[error]Bash exited with code '1'.
2025-01-17T21:48:29.9684097Z ##[section]Finishing: Run schemachange

pearsall-will · 2025-01-18T14:31:39Z

@BlessingZondi You're using the wrong command line arguments for what you're wanting to do.

-c should be --connections-file-path
-a should be --connection-name

Like below

schemachange -f (System.DefaultWorkingDirectory) \
--connections-file-path (System.DefaultWorkingDirectory)/connections.toml \
--connection-name sta_prod

You can read through the arguments the CLI parses in this file

BlessingZondi · 2025-01-20T12:28:53Z

@pearsall-will thanks for the feedback, I will have a look at the script and make necessary changes

eugeneburd · 2025-01-27T17:02:28Z

we are also getting the same User is empty error:

jobs:
  deploy-snowflake-changes-job:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Use Python 3.8.x
        uses: actions/[email protected]
        with:
          python-version: 3.8.x

      - name: Create connections.toml
        run: |
          cat << EOF > connections.toml
          [snowflake]
          account = "${{ vars.SNOWFLAKE_ACCOUNT }}"
          user = "${{ secrets.SNOWFLAKE_USERNAME }}"
          password = "${{ secrets.SNOWFLAKE_PASSWORD }}"
          role = "${{ vars.SNOWFLAKE_ROLE }}"
          warehouse = "${{ vars.SNOWFLAKE_WAREHOUSE }}"
          database = "${{ vars.SNOWFLAKE_DATABASE }}"
          EOF

      - name: Run schemachange
        env:
          SNOWFLAKE_DEFAULT_CONNECTION_NAME: "snowflake"
        run: |
          echo "GITHUB_WORKSPACE: $GITHUB_WORKSPACE"
          python --version
          echo "Step 1: Installing schemachange"
          pip install schemachange
          
          echo "Step 2: Running schemachange"
          schemachange \
            -f $GITHUB_WORKSPACE/migrations/AIRBYTE_WAREHOUSE/BORDEREAUX \
            -c ${{ vars.SNOWFLAKE_DATABASE }}.SCHEMACHANGE.CHANGE_HISTORY \
            --create-change-history-table \
            --connections-file-path connections.toml

Version 3.7.0 does not have this issue (although the config is different):


jobs:
  deploy-snowflake-changes-job:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Use Python 3.8.x
        uses: actions/[email protected]
        with:
          python-version: 3.8.x

      - name: Run schemachange
        env:
          SNOWFLAKE_ACCOUNT: ${{ vars.SNOWFLAKE_ACCOUNT }}
          SNOWFLAKE_USERNAME: ${{ secrets.SNOWFLAKE_USERNAME }}
          SNOWFLAKE_ROLE: ${{ vars.SNOWFLAKE_ROLE }}
          SNOWFLAKE_WAREHOUSE: ${{ vars.SNOWFLAKE_WAREHOUSE }}
          SNOWFLAKE_DATABASE: ${{ vars.SNOWFLAKE_DATABASE }}
          SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}
        run: |
          echo "GITHUB_WORKSPACE: $GITHUB_WORKSPACE"
          python --version
          echo "Step 1: Installing schemachange"
          pip install schemachange==3.7.0
          
          echo "Step 2: Running schemachange"
          schemachange -f $GITHUB_WORKSPACE/migrations/AIRBYTE_WAREHOUSE/BORDEREAUX -a $SNOWFLAKE_ACCOUNT -u $SNOWFLAKE_USERNAME -r $SNOWFLAKE_ROLE -w $SNOWFLAKE_WAREHOUSE -d $SNOWFLAKE_DATABASE -c $SNOWFLAKE_DATABASE.SCHEMACHANGE.CHANGE_HISTORY --create-change-history-table

sanelson · 2025-01-30T15:57:42Z

@eugeneburd, see the comment by @pearsall-will as well as my comment on #320 for the solution.

TLDR; Make sure you provide the --connection-name option. In your case specifically:

          echo "Step 2: Running schemachange"
          schemachange \
            -f $GITHUB_WORKSPACE/migrations/AIRBYTE_WAREHOUSE/BORDEREAUX \
            -c ${{ vars.SNOWFLAKE_DATABASE }}.SCHEMACHANGE.CHANGE_HISTORY \
            --create-change-history-table \
            --connections-file-path connections.toml \
            --connection-name snowflake

BlessingZondi added question Further information is requested Workaround Issues that can be addressed via a workaround labels Jan 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Schemachange version 4.0.0 not picking up variable User from created connections.toml file #317

Schemachange version 4.0.0 not picking up variable User from created connections.toml file #317

BlessingZondi commented Jan 17, 2025

pearsall-will commented Jan 17, 2025

BlessingZondi commented Jan 18, 2025

pearsall-will commented Jan 18, 2025 •

edited

Loading

BlessingZondi commented Jan 20, 2025

eugeneburd commented Jan 27, 2025

sanelson commented Jan 30, 2025

Schemachange version 4.0.0 not picking up variable User from created connections.toml file #317

Schemachange version 4.0.0 not picking up variable User from created connections.toml file #317

Comments

BlessingZondi commented Jan 17, 2025

pearsall-will commented Jan 17, 2025

BlessingZondi commented Jan 18, 2025

pearsall-will commented Jan 18, 2025 • edited Loading

BlessingZondi commented Jan 20, 2025

eugeneburd commented Jan 27, 2025

sanelson commented Jan 30, 2025

pearsall-will commented Jan 18, 2025 •

edited

Loading