Expanded testing, plus new models for AWS Config, SQS, SNS (#16)

Adds a number of improvements. Tests will need to be revisited, but I've
added a decent base.

Bumps to v0.1.1

### Added

- Adds support for generating a Graphviz diagram of an Organization with the new
  `OrganizationDataBuilder.to_dot()` function
- Adds `DOT` as a supported output format for the `organization dump-all` command
- Adds models for AWS Config notifications
- Adds models for SQS and SNS messages
- Adds methods to ModelBase to allow (de)serializing JSON or YAML strings
- Adds ModelBase.from_dict() to initialize a model from a dict using dacite
- Adds CodeQL analysis workflow for GitHub Actions

### Changed

- breaking: Renames `organization dump-json` CLI command to `organization dump-all`
- Moves buiders into the models namespace

.github/CODE_OF_CONDUCT.md

@@ -117,11 +117,11 @@ This Code of Conduct is adapted from the [Contributor Covenant][homepage],
 version 2.0, available at
 [https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
 
-Community Impact Guidelines were inspired by 
+Community Impact Guidelines were inspired by
 [Mozilla's code of conduct enforcement ladder][Mozilla CoC].
 
 For answers to common questions about this code of conduct, see the FAQ at
-[https://www.contributor-covenant.org/faq][FAQ]. Translations are available 
+[https://www.contributor-covenant.org/faq][FAQ]. Translations are available
 at [https://www.contributor-covenant.org/translations][translations].
 
 [homepage]: https://www.contributor-covenant.org

.github/workflows/ci.yml

@@ -127,3 +127,41 @@ jobs:
 
       - name: Run tests
         run: make test
+
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+    permissions:
+      actions: read
+      contents: read
+      security-events: write
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.9
+
+      - name: Install Poetry
+        uses: snok/[email protected]
+        with:
+          virtualenvs-create: false
+          virtualenvs-in-project: true
+
+      - name: Restore cache venv
+        uses: syphar/restore-virtualenv@v1
+        id: cache-venv
+
+      - name: Initialize CodeQL
+        uses: github/codeql-action/[email protected]
+        with:
+          languages: python
+          setup_python_dependencies: false
+
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/[email protected]

.gitignore

@@ -308,7 +308,7 @@ dist
 .pnp.*
 
 # -------------------------------------------------------------------------------------
-# Python 
+# Python
 # -------------------------------------------------------------------------------------
 
 # Byte-compiled / optimized / DLL files
@@ -1101,3 +1101,7 @@ $RECYCLE.BIN/
 
 # Windows shortcuts
 *.lnk
+
+# Misc
+
+*.bak

.pre-commit-config.yaml

@@ -1,32 +1,32 @@
 # pre-commit run --all-files
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v3.4.0
+    rev: v4.0.1
     hooks:
       - id: check-ast
       - id: check-json
       - id: check-merge-conflict
       - id: check-toml
       - id: check-yaml
       - id: debug-statements
-      - id: detect-aws-credentials
-        args: ["--allow-missing-credentials"]
       - id: detect-private-key
       - id: end-of-file-fixer
       - id: mixed-line-ending
       - id: no-commit-to-branch
       - id: trailing-whitespace
   - repo: https://github.com/psf/black
-    rev: 21.6b0
+    rev: 21.11b1
     hooks:
       - id: black
   - repo: https://github.com/asottile/blacken-docs
-    rev: v1.10.0
+    rev: v1.12.0
     hooks:
       - id: blacken-docs
         additional_dependencies: [black==21.6b0]
-  - repo: https://github.com/flakehell/flakehell
-    rev: v.0.8.0
-    hooks:
-      - name: Run flakehell static analysis tool
-        id: flakehell
+  # Disable flakehell for now due to import bug with newer flake8
+  # https://github.com/flakehell/flakehell/issues/22
+  # - repo: https://github.com/flakehell/flakehell
+  #   rev: v.0.9.0
+  #   hooks:
+  #     - name: Run flakehell static analysis tool
+  #       id: flakehell

CHANGELOG.md

@@ -14,15 +14,23 @@ the unreleased section to the section for the new release.
 
 No unreleased changes.
 
+## [0.1.1] - 2021-11-23
+
 ### Added
 
 - Adds support for generating a Graphviz diagram of an Organization with the new
   `OrganizationDataBuilder.to_dot()` function
 - Adds `DOT` as a supported output format for the `organization dump-all` command
+- Adds models for AWS Config notifications
+- Adds models for SQS and SNS messages
+- Adds methods to ModelBase to allow (de)serializing JSON or YAML strings
+- Adds ModelBase.from_dict() to initialize a model from a dict using dacite
+- Adds CodeQL analysis workflow for GitHub Actions
 
 ### Changed
 
 - breaking: Renames `organization dump-json` CLI command to `organization dump-all`
+- Moves buiders into the models namespace
 
 ## [0.1.0-beta2] - 2021-06-16
 
@@ -78,7 +86,8 @@ Initial alpha release
 These Markdown anchors provide a link to the diff for each release. They should be
 updated any time a new release is cut.
 -->
-[Unreleased]: https://github.com/timoguin/aws-org-tools-py/compare/v0.1.0-beta-2...HEAD
+[Unreleased]: https://github.com/timoguin/aws-org-tools-py/compare/v0.1.1...HEAD
+[0.1.1]: https://github.com/timoguin/aws-org-tools-py/compare/v0.1.0-beta2...v0.1.1
 [0.1.0-beta2]: https://github.com/timoguin/aws-org-tools-py/compare/v0.1.0-beta1...v0.1.0-beta2
 [0.1.0-beta1]: https://github.com/timoguin/aws-org-tools-py/compare/v0.1.0-alpha4...v0.1.0-beta1
 [0.1.0-alpha4]: https://github.com/timoguin/aws-org-tools-py/releases/tag/v0.1.0-alpha4

Makefile

@@ -31,8 +31,9 @@ shellcmd: ${VENV_DIR}
 lint: ${VENV_DIR}
 	@echo "Running the black code formatter"
 	@poetry run black ${ARGS} .
-	@echo "Running flakehell plugins"
-	@poetry run flakehell lint aws_data_tools
+	# flakehell currently disabled, see .pre-commit-config.yaml notesk
+	# @echo "Running flakehell plugins"
+	# @poetry run flakehell lint aws_data_tools
 
 .PHONY: lint-docs
 lint-docs: ${VENV_DIR}
@@ -72,7 +73,7 @@ build: ${VENV_DIR}
 .PHONY: test ## Run the test suite
 test: ${VENV_DIR}
 	@echo "Running test suite"
-	@poetry run pytest --suppress-no-test-exit-code
+	@poetry run pytest --cov ${OPTS} ${ARGS}
 
 # Ensures the Python venv exists and has dependencies installed
 ${VENV_DIR}:

README.md

@@ -1,7 +1,8 @@
 # AWS Data Tools
 
 <!-- Badges -->
-[![Actions Status][gh-actions-badge]][gh-actions-link]
+[![Actions CI Status][gh-actions-ci-badge]][gh-actions-ci-link]
+[![Actions CodeQL Status][gh-actions-codeql-badge]][gh-actions-codeql-link]
 [![PyPI][pypi-badge]][pypi-link]
 [![License][license-badge]][license-link]
 
@@ -32,7 +33,7 @@ To dump a data representation of an AWS Organization, you can do the following u
 the builder:
 
 ```python
-from aws_data_tools.builders.organizations import OrganizationDataBuilder
+from aws_data_tools.models.organizations import OrganizationDataBuilder
 
 odb = OrganizationDataBuilder(init_all=True)
 organization = odb.as_json()
@@ -41,7 +42,7 @@ organization = odb.as_json()
 Here is how to do the same thing with the CLI:
 
 ```
-$ awsdata organization dump-json
+$ awsdata organization dump-all
 ```
 
 ## Usage
@@ -54,9 +55,9 @@ abstract some of these operations.
 ### Builders
 
 While it is possible to directly utilize and interact with the data models, probably
-the largest benefit is the [builders](aws_data_tools/builders) package. It abstracts
-any API operations and data transformations required to build data models. The models
-can then be serialized to dicts, as well as JSON or YAML strings.
+the largest benefit are the builders. They abstract any API operations and data
+transformations required to build data models. The models can then be serialized to
+dicts, as well as DOT, JSON, or YAML strings.
 
 A full model of an AWS Organization can be constructed using the
 `OrganizationDataBuilder` class. It handles recursing the organizational tree and
@@ -67,9 +68,9 @@ The simplest example pulls all supported organizational data and creates the rel
 data models:
 
 ```python
-from aws_data_tools.builders.organizations import OrganizationDataBuilder as odb
+from aws_data_tools.models.organizations import OrganizationDataBuilder
 
-org = odb(init_all=True)
+odb = OrganizationDataBuilder(init_all=True)
 ```
 
 Note that this makes many API calls to get this data. For example, every OU, policy,
@@ -84,9 +85,9 @@ requires 316 API calls! That's why this library was created.
 For more control over the process, you can init each set of components as desired:
 
 ```python
-from aws_data_tools.builders.organizations import OrganizationDataBuilder as odb
+from aws_data_tools.models.organizations import OrganizationDataBuilder
 
-org = odb()
+org = OrganizationDataBuilder()
 org.init_connection()
 org.init_organization()
 org.init_root()
@@ -274,8 +275,10 @@ View the [Contributing Guide](.github/CONTRIBUTING.md) to learn about giving bac
 
 
 <!-- Markown anchors -->
-[gh-actions-badge]: https://github.com/timoguin/aws-data-tools-py/actions/workflows/ci.yml/badge.svg
-[gh-actions-link]: https://github.com/timoguin/aws-data-tools-py/actions
+[gh-actions-ci-badge]: https://github.com/timoguin/aws-data-tools-py/actions/workflows/ci.yml/badge.svg
+[gh-actions-ci-link]: https://github.com/timoguin/aws-data-tools-py/actions/workflows/ci.yml
+[gh-actions-codeql-badge]: https://github.com/timoguin/aws-data-tools-py/actions/workflows/codeql-analysis.yml/badge.svg
+[gh-actions-codeql-link]: https://github.com/timoguin/aws-data-tools-py/actions/workflows/codeql-analysis.yml
 [license-badge]: https://img.shields.io/github/license/timoguin/aws-data-tools-py.svg
 [license-link]: https://github.com/timoguin/aws-data-tools-py/blob/main/LICENSE
 [pypi-badge]: https://badge.fury.io/py/aws-data-tools.svg

aws_data_tools/__init__.py

@@ -1,6 +1,9 @@
 """
 A library for working with data from AWS APIs
 """
+# flake8: noqa: F401
+
+from . import client, models, utils
 
 __VERSION__ = "0.1.0-beta2"