feat: [SQL Query transformer] Improve attribute definitions in yaml templates #528
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added the following features to logger_adaptor : - convert file to parquet - store to objectstore as parquet
Updated logger_adaptor logic
added route in application-sdk called observability which would decode the logs and dispaly as parquet.
updated Dapr client to use objectstore binding
- Added logic to check if duckdb is already up and running or not. If yes, then connect to the already running server and use that connection to connect to duckdb ui - added batching logic which will flush logs to dapr objectstore either per 100 log records or at every 10 seconds asynchronously - these constants are defined in constants.py and are configurable - the /observability api endpoint will redirect to the duckdb ui automatically which by default is accessible on 0.0.0.0:9000
…ervability-logging
- Added purge logic which runs once in a day in a rolling manner. It will purge all log records which are older than the set LOG_RETENTION_DAYS const. - Redirected to 4213 instead of 9000 which is the correct UI port for Duckdb.
- Added Unit tests to check if the logs are going to the dapr's object store or not - Added documentation for the enhanced functionality of LoggerAdaptor
- Created a Observability superclass which would be inherited by all the Adaptors. It will have function to push to objectStore, log retention capabilities etc.
Updated documentation
- Added a MetricsAdaptor on the lines of LoggerAdaptor - It uses functionality from Observability as its super class - Added tests for this - Added sample metrics in __init__.py for testing - Added relevant documentation for this - Made observability a generic superclass which can be used by any adaptor
- Added TraceAdaptor - Added corresponding tests - Added corresponding documentation
…ervability-metrics
- Added the following metrics to application-sdk : 1. Metrics which calculate time for whole workflow run 2. Metrics to calculate number of records / chunks written and in what format 3. Metrics to calculate time of steps like pre-flight check, authentication etc.
- added hierarchical file structure for observability - added signal capturing and flushing buffer capability in case any error is encountered.
- Fixed unit tests
Co-authored-by: Copilot <[email protected]> Signed-off-by: Abhishek Agrawal <[email protected]>
feat: add logging adapter with objectstore sink
Signed-off-by: Abhishek Agrawal <[email protected]>
- fixed redeclaration of methods
- Updated log directory to /tmp/observability - Updated extra fields model
- Added metric adaptor
feat : Add minimal set of metrics for all apps
🎉 Snyk checks have passed. No issues have been found so far. |
OnkarVO7
changed the title
📜 Docstring Coverage ReportRESULT: PASSED (minimum: 30.0%, actual: 72.9%) Detailed Coverage Report |
|
🛠 Docs available at: https://k.atlan.dev/application-sdk/sql_tranformer_impr_attr |
📦 Trivy Vulnerability Scan Results
Report Summary
Scan Result Details✅ No vulnerabilities found during the scan for |
📦 Trivy Secret Scan Results
Report Summary
Scan Result Details✅ No secrets found during the scan for |
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
|
🛠 Full Test Coverage Report: https://k.atlan.dev/coverage/application-sdk/pr/528 |
📦 Example workflows test results
|
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 5 to 6. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@v5...v6) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
- added _send_to_otel method to logger_adaptor
- fixed pre-commit related errors
* fix diff issues * view defn fixes
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog
Improve attribute definitions in yaml templates
Before we required the attribute definitions in the yaml templates to be in flat structure with dot separator for nested columns
e.g
After this change, the nested attributes can be defined normally as per yaml structures.
E.g
Additional context (e.g. screenshots, logs, links)
https://atlanhq.atlassian.net/browse/APP-6727
Checklist
Copyleft License Compliance