Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโ€™ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic outline for security issues tool & action #88

Merged
merged 61 commits into from
Oct 27, 2023
Merged
Show file tree
Hide file tree
Changes from 59 commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
c763552
Add basic outline for security issues tool & action
Nicoretti Oct 23, 2023
3f28860
Remove unused imports
Nicoretti Oct 23, 2023
25cd737
Cleanup secuirity issues action structure
Nicoretti Oct 23, 2023
b3b1e14
Update shell used in security action
Nicoretti Oct 23, 2023
565eab7
Update security issues command
Nicoretti Oct 23, 2023
6af52f8
Update file names in action
Nicoretti Oct 23, 2023
e242ff2
Add basic support for maven input format
Nicoretti Oct 23, 2023
a5626db
Various changes
Nicoretti Oct 24, 2023
f4693b0
fix secruity issue action to use the new cli commands
Nicoretti Oct 24, 2023
a0832d7
Restructure security module
Nicoretti Oct 24, 2023
f03d2d1
Add basic implementation for ticket creation
Nicoretti Oct 24, 2023
7554c2d
Replace rich console with std print
Nicoretti Oct 24, 2023
17c32f7
Add additional debug output
Nicoretti Oct 24, 2023
d369d45
Fix issue filter
Nicoretti Oct 24, 2023
5351461
replace sys.exit with raise typer.Exit
Nicoretti Oct 24, 2023
b319a11
WIP: Documenation
Nicoretti Oct 24, 2023
3015ac8
Bump version
Nicoretti Oct 24, 2023
b8ffaff
Update docs
Nicoretti Oct 24, 2023
c33f454
Add test for security issue creation
Nicoretti Oct 25, 2023
3422beb
Address various pylint warnings
Nicoretti Oct 25, 2023
a16d785
Change entry point for security command
Nicoretti Oct 25, 2023
16a9d23
Update cli
Nicoretti Oct 25, 2023
90670b1
Update action
Nicoretti Oct 25, 2023
c70fff9
Remove empty lines from input
Nicoretti Oct 25, 2023
6ba2dac
Update CLI
Nicoretti Oct 25, 2023
94ae3e1
Improve CLI
Nicoretti Oct 25, 2023
2eefbdf
Update github action
Nicoretti Oct 25, 2023
f618baf
Remove todo's from github action
Nicoretti Oct 25, 2023
236de49
Require input format for action
Nicoretti Oct 25, 2023
7ffb136
Fix github action
Nicoretti Oct 25, 2023
f400d75
Update github output for security action
Nicoretti Oct 25, 2023
b01f9b3
Set python version for action to 3.11
Nicoretti Oct 25, 2023
7a700a3
Filter newlines
Nicoretti Oct 25, 2023
4b5690e
Remove extra marker
Nicoretti Oct 25, 2023
7a9e19b
ouput filtered issues
Nicoretti Oct 25, 2023
abdcb25
Change output format
Nicoretti Oct 25, 2023
085473e
Adjust format once more
Nicoretti Oct 25, 2023
21ee87e
Update note in code
Nicoretti Oct 25, 2023
384e174
Update documentation
Nicoretti Oct 25, 2023
2bcb4a7
Adjust output format for summary
Nicoretti Oct 25, 2023
7893d69
Update branch/tag reference in action
Nicoretti Oct 26, 2023
de02702
Add integration test for security cve convert command
Nicoretti Oct 26, 2023
d0a1bbb
Add integration tests for secruity cve filter command
Nicoretti Oct 26, 2023
0a61b7e
Add integration test for security cve create command
Nicoretti Oct 26, 2023
d7fb2ab
Update doc/tools.rst
Nicoretti Oct 26, 2023
2a6d2fe
Update doc/tools.rst
Nicoretti Oct 26, 2023
639f00b
Update doc/tools.rst
Nicoretti Oct 26, 2023
b21b329
Update test/integration/cli/security-cve-create.t
Nicoretti Oct 26, 2023
4a530b0
Update test/integration/cli/security-cve-filter.t
Nicoretti Oct 26, 2023
ece835a
Update doc/github_actions/security_issues.rst
Nicoretti Oct 26, 2023
ad5cb68
Update .github/actions/security-issues/action.yml
Nicoretti Oct 26, 2023
2ca20fa
Update .github/actions/security-issues/action.yml
Nicoretti Oct 26, 2023
b226e45
Update doc/github_actions/security_issues.rst
Nicoretti Oct 26, 2023
1dede7a
Update doc/github_actions/security_issues.rst
Nicoretti Oct 26, 2023
081297e
Update exasol/toolbox/tools/security.py
Nicoretti Oct 26, 2023
33afe8f
Update exasol/toolbox/tools/security.py
Nicoretti Oct 26, 2023
43ae7b2
Update exasol/toolbox/tools/security.py
Nicoretti Oct 26, 2023
58b0fd5
Apply code review feedback
Nicoretti Oct 27, 2023
1e74e0c
Adjust action
Nicoretti Oct 27, 2023
88a5c0f
Fix typo
Nicoretti Oct 27, 2023
5a5a95d
Update pr template
Nicoretti Oct 27, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .github/actions/security-issues/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: 'SIA'
description: 'The Security Issues Action creates github issues for open security issues in the repository'

inputs:

command:
description: 'Command for generating a security report'
required: true

format:
description: 'Input format (e.g. "maven" or "pass-through")'
required: true

github-token:
description: 'Github Token'
required: true

runs:

using: "composite"
steps:

- name: Setup Python (${{ inputs.python-version}})
uses: actions/setup-python@v4
with:
python-version: 3.11
Nicoretti marked this conversation as resolved.
Show resolved Hide resolved

- name: Install Python Toolbox / Security tool
shell: bash
run: |
pip install exasol-toolbox==0.6.0

- name: Create Security Issue Report
shell: bash
run: |
${{ inputs.command }} | tee input

- name: Convert Report To Common Input Format
shell: bash
run: |
tbx security cve convert ${{inputs.format}} < input | tee cves.jsonl

- name: Filter Issues
env:
GH_TOKEN: ${{ inputs.github-token }}
shell: bash
run: |
tbx security cve filter github-issues < cves.jsonl 2> filtered.txt | tee issues.jsonl
cat filtered.txt

- name: Create Issues
env:
GH_TOKEN: ${{ inputs.github-token }}
shell: bash
run: |
tbx security cve create < issues.jsonl | tee created.txt

- name: Create Report
shell: bash
run: |
echo -e "# Summary\n" >> $GITHUB_STEP_SUMMARY
echo -e "## Created Security Issue\n" >> $GITHUB_STEP_SUMMARY
cat created.txt >> $GITHUB_STEP_SUMMARY
echo -e "## Filtered Security Issue\n" >> $GITHUB_STEP_SUMMARY
tail -n +2 filtered.txt | grep . >> $GITHUB_STEP_SUMMARY
6 changes: 6 additions & 0 deletions doc/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@
Unreleased
==========

โœจ Added
--------

* Added security command
* Added security-issues action

.. _changelog-0.5.0:

0.5.0 - 2023-10-12
Expand Down
7 changes: 2 additions & 5 deletions doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,12 @@

sys.path.insert(0, os.path.abspath("../"))


# -- Project information -----------------------------------------------------

project = "Exasol Toolbox"
copyright = "2022, Exasol"
copyright = "2022, Exasol" # pylint: disable=redefined-builtin
author = "Exasol"


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
Expand Down Expand Up @@ -60,7 +58,6 @@
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", ".build-docu"]


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages. See the documentation for
Expand All @@ -72,7 +69,7 @@
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ["_static"]
html_title = f"Toolbox"
html_title = "Toolbox"
html_theme_options = {
"light_logo": "light-exasol-logo.svg",
"dark_logo": "dark-exasol-logo.svg",
Expand Down
4 changes: 1 addition & 3 deletions doc/developer_guide/developer_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,4 @@

../design
development
todos


ideas
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
๐Ÿ“‹ Todo's
---------
๐Ÿ“‹ Ideas
--------
.. todolist::

- Add commit hooks (version check etc.) for the toolbox itself
Expand Down
7 changes: 7 additions & 0 deletions doc/github_actions/github_actions.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
๐Ÿฆบ Github Actions
=================

.. toctree::
:maxdepth: 2

security_issues
102 changes: 102 additions & 0 deletions doc/github_actions/security_issues.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
security-issues
===============

Example Usage
-------------

.. code-block:: yaml

name: Report Security Issues for Repository

on:
schedule:
# โ€œEvery day at 00:00.โ€ (https://crontab.guru)
- cron: "0 0 * * *"

jobs:

report_security_issues:

name: Report Security Issues

runs-on: ubuntu-latest

permissions:
issues: write
Nicoretti marked this conversation as resolved.
Show resolved Hide resolved

steps:
- name: SCM Checkout
uses: actions/checkout@v4

- name: Report Security Issues
uses: exasol/python-toolbox/.github/actions/[email protected]/security-issues-action
with:
format: "maven"
command: "cat maven-cve-report.json"
github-token: ${{ secrets.GITHUB_TOKEN }}

Configuration
-------------
This action exposes 3 configuration parameters `command`_, `format`_ and `github-token`_, for details see
the specific sections below.

command
+++++++

Workspace command which shall be executed in order to check the project's dependencies for CVEs.

.. note::

The calling workflow needs to make sure the specified command can be executed in the context of the workflow.


format
++++++

Specifies converter which needs to be applied on the output of the provided command.
Currently there are only two converters available

#. maven

Converts the output of mavens oss plugin into required input format.


#. pass-through

In case the command itself already outputs the expected input format, the format can be specified as code:`pass-through`.


Input Format
------------

The expect intput format is jsonl (line based json), of the following form:

.. code-block:: python

{ "cve": "<cve-id>", "cwe": "<cwe-id>", "description": "<multiline string>", "coordinates": "<string>", "references": ["<url>", "<url>", ...] }
Nicoretti marked this conversation as resolved.
Show resolved Hide resolved


.. attention::

The input format may change in the future. Therefore make sure to rather use or contribute a converter for
a specific format rather than outputting this format by your own tooling.


github-token
++++++++++++
The temporary GitHub token of the workflow needs to be passed into the action (:code:`${{ secrets.GITHUB_TOKEN }}`),
in order to enable the action to query and created GitHub issues.


Ideas
-----

.. todo::

Add additional details to the :code:`security.Issue` type


.. todo::

Consider adapting common CVE report format as input, for additional details
`see here <https://github.com/CVEProject/cve-schema/blob/master/schema/v5.0/CVE_JSON_5.0_schema.json>`_.
1 change: 1 addition & 0 deletions doc/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

user_guide/user_guide
tools
github_actions/github_actions
api
developer_guide/developer_guide
changelog
44 changes: 12 additions & 32 deletions doc/tools.rst
Original file line number Diff line number Diff line change
@@ -1,48 +1,28 @@
๐Ÿ’ป Tools
========

tbx
---
The :code:`tbx` is the main entry point for all of the toolbox specific tooling.
The python-toolbox ships with a set of command line tools, whose entry point always is the command :code:`tbx`.
The commands are structured in a *tree* manner, and help is provided along with the command(s) no matter the nesting.

How to get Help
---------------

.. code-block:: shell

$ tbx --help

Usage: tbx [OPTIONS] COMMAND [ARGS]...

โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ --install-completion Install completion for the current shell. โ”‚
โ”‚ --show-completion Show completion for the current shell, to copy it or โ”‚
โ”‚ customize the installation. โ”‚
โ”‚ --help Show this message and exit. โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
โ•ญโ”€ Commands โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ workflow โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

workflow
++++++++
The workflow command helps to install and maintain GitHub workflows provided by the toolbox.

.. code-block:: shell

$ tbx workflow --help
$ tbx command --help

Usage: tbx workflow [OPTIONS] COMMAND [ARGS]...

โ•ญโ”€ Options โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ --help Show this message and exit. โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
โ•ญโ”€ Commands โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”‚ diff Diff a specific workflow against the installed one. โ”‚
โ”‚ install Installs the requested workflow into the target directory. โ”‚
โ”‚ list List all available workflows. โ”‚
โ”‚ show Shows a specific workflow. โ”‚
โ”‚ update Similar to install but checks for existing workflows and shows diff โ”‚
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
.. code-block:: shell

$ tbx command subcommand --help

.. code-block:: shell

$ tbx command subcommand subsubcommand --help


If the details for a specific command are not sufficient checkout the according subsections bellow,
or `create an isssue <https://github.com/exasol/python-toolbox/issues/new?assignees=&labels=documentation&projects=&template=documentation.md&title=%F0%9F%93%9A+%3CInsert+Title%3E>`_ if nothing is avialable yet.
Loading