Skip to content

Latest commit

 

History

History
1862 lines (1470 loc) · 108 KB

CHANGELOG.md

File metadata and controls

1862 lines (1470 loc) · 108 KB
Raw

CHANGELOG

v2.0.0 (2024-12-03)

Chores

Continuous integration

Documentation

  • docs: Add real-world example (30bd774)

    Include the GMS system-level testing class hierarchy as an example of the significant flexibility afforded by the staged-script framework.

  • docs: Pin Sphinx version (3a1c80d)

    The Sphinx 8.0.0 release causes problems with sphinx-rtd-theme, so I'm pinning Sphinx below 8.0 to get things back up and running again.

  • docs: Fix lines included from examples (8dd8fb4)

    Should have been included in ccfa0d9a6cb7428d4ba4c7e38f7d07d02efe9a8c.

Testing

  • test: Remove unnecessary parentheses (b190ac6)

    To align with updated ruff rules.

v1.0.2 (2024-07-02)

Bug fixes

  • fix: Specify Poetry dependencies (d3e3b5c)

v1.0.1 (2024-07-02)

Chores

  • chore: Add CHANGELOG (c5ce0f4)

    Add a dummy CHANGELOG.md to be overwritten by Semantic Release.

  • chore: Add example requirements (343b8e7)

    Add a blank requirements file for the examples to facilitate adding requirements in the future.

Continuous integration

Documentation

  • docs: Slight tweaks to contributing guidelines (59325de)

Patch

  • patch: Indicate that the package is typed (55a20c9)

    According to PEP 561, we need to add this file to static type checkers can infer the types from the package.

Testing

  • test: Run the examples and check their output (ccfa0d9)

v1.0.0 (2024-06-25)

Bug fixes

  • fix: Retry group help text (d237e38)

    When I created the retry group in the argument parser, I didn't realize that I was using add_argument_group in a way in which it wasn't intended to be used. This commit brings things into conformity with how argparse is intended to be used, specifying a one-word group name, followed by a more detailed group description.

  • fix!: Handle --dry-run correctly (8f436af)

    When a DriverScript subclass is run in dry-run mode, instead of executing commands in the shell, print a message indicating what would have been run. Prior to this, DriverScript provided the flag, but relied on subclass developers to ensure commands weren't run. This was an oversight.

    Note that this is a breaking change, as behavior of the run() method has fundamentally changed for dry-run mode.

Chores

  • chore: Update .gitignore (815c8cd)

  • chore: Ignore mypy warnings (4dbf516)

    Temporarily disable these warnings until there's time to revisit and address them.

  • chore: Add version to init.py (f42f22d)

  • chore: Remove YAPF comments (86f5cfe)

    No longer needed after switching from YAPF to Ruff.

  • chore: Ignore mypy warnings (a21b0c2)

    Temporarily disable these warnings until there's time to revisit and address them.

  • chore: Ignore unchecked shell input (520b68e)

    When we built the precursor to staged-script, we didn't understand the security implications of having the user pass commands to the underlying shell as a string rather than a list of strings. It just seemed like a better interface, to make it easy for the user writing their Python scripts to simply wrap their bash commands. However, this opens up a danger for bad actors to cause problems via command injection. We can remove the vulnerability by switching to only allowing a list of strings as input to run(), but that breaking change will need to wait till another day when I have more time available.

  • chore: Use installed reverse_argparse (7a3d1b7)

  • chore: Move init.py (ba94a17)

  • chore: Remove unnecessary shebang lines (3516d84)

  • chore: Add requirements files (0dc5512)

  • chore: Add pyproject.toml (e81f2d2)

  • chore: Add GitHub issur/PR templates (017b544)

Code style

  • style: Automatically format the code base (2413186)
  • style: Pytest parametrize tuple (0211644)
  • style: Match exception message (c2353bd)
  • style: User iterable unpacking (9b18fac)
  • style: Use f-string conversion flag (56b2c45)

Continuous integration

Documentation

  • docs: Remove WIP banner (59bd6ce)

    In preparation for the initial release, remove the banner indicating that open sourcing is still a work in progress.

  • docs: Fix typo (e68b838)

  • docs: Create documentation and examples (5336421)

  • docs: Stub out Sphinx documentation (00bd9af)

  • docs: Add copyright/license text to source files (b0c22fc)

  • docs: Fix docstring issues (7365c87)

  • docs: Update OpenSSF Best Practices badge (1633c78)

  • docs: Add contributing guidelines (8c04e16)

  • docs: Add code of conduct (579786e)

  • docs: Add security guidelines (c66c1dd)

  • docs: Add README (29c2ed3)

  • docs: Add license file (4be6ebf)

  • docs: Update docstrings (aaccd8d)

    Provide additional clarity in DriverScript's documentation.

  • docs: Update syntax (5bb662a)

    Update the instructions for overriding print_script_execution_summary to use the |= syntax for updating dictionaries.

  • docs: Use "subclass" everywhere (c6fa97d)

    Instead of inconsistently switching between "subclass" and "child class" in docstrings, use "subclass" everywhere for consistency's sake.

  • docs: Add docstrings to stage inner functions (09b4797)

    The number and size of the inner functions for the stage decorator are getting substantial enough that they really need docstrings at this point.

  • docs: Update docstrings for stage methods (966103c)

    Fill in all the details regarding subclassing that were omitted in earlier commits.

Features

  • feat!: Create RetryStage exception (d5e8941)

    Rather than relying on stages raising a tenacity.TryAgain exception to trigger retrying a stage, add a RetryStage exception to the package. This makes things more explicit for subclass developers, and makes it such that something outside the subclass developer's control that happens to raise a TryAgain doesn't trigger retying the stage, unless of course the subclass developer catches it and then raises a RetryStage in its place.

    Note that this is a breaking change, because any subclasses relying on raising TryAgain to trigger a stage retry will need to be updated to raise the new exception.

  • feat: Add script name/stem attributes (74c387d)

    Capture the name and stem of the file being executed as attributes of DriverScript such that subclass developers can easily refer to them rather than determining them on the fly. This is useful for things like the script execution summary.

    Also add a unit test for raise_parser_error(), which was accidentally omitted in a prior commit.

  • feat: Add help formatter to package (b458d95)

    Add a class to the driver_script package to use as a formatter class for the DriverScript ArgumentParser. Make it such that the parser description is treated as raw text (no automatic formatting is done), and show the default values for all arguments.

  • feat: Add flag for printing commands (7a7a0e0)

    Add a flag when instantiating a DriverScript to govern whether commands should be printed immediately before running them. Default it to True for backwards compatibility. Also provide a keyword argument to the run() method that, if specified, will override the class flag. In this way users could, e.g., instantiate a DriverScript, telling it to print all commands, and then specify that certain commands should not be printed. The opposite of this is also possible (e.g., don't print any commands, except this one and that one).

  • feat: Add raise_parser_error (0f60f8d)

    Add a method to the DriverScript base class to raise a parser error when a user has done something wrong in specifying the command line arguments. This differs from argparse.ArgumentParser.error() in that it's a little more user friendly, printing the help text first, and then the particular error in yellow.

  • feat: Capture script success/failure (68ea1ae)

    Provide an instance attribute to allow subclass developers to toggle whether the script has succeeded or failed, and include the result in the script execution summary.

  • feat!: Add stage retry functionality (d27ce6f)

    Add the ability to automatically retry a stage that raises a tenacity.TryAgain exception:

    • Provide default implementations for methods to prepare to retry a stage and appropriately handle when retrying ultimately fails, while allowing subclass developers to override them or customize them on a stage-by-stage basis.
    • Provide documentation to instruct subclass developers on how to make use of the retry functionality.
    • Wrap the Begin-Stage Actions, Stage Body, and End-Stage Actions in the retrying loop.

    Note that this is marked as a breaking change, as it's unclear what will happen to any subclasses that implemented their own stage retry functionality outside of this. The safest thing will be for subclass developers to update their subclasses to use this capability now provided by the base class.

  • feat!: Provide retry args for every stage (281c376)

    Automatically add command line arguments to govern stage retry behavior (number of retry attempts, how long to wait between retries, total time to spend trying the stage) for every stage defined in a subclass.

    Note that this is a breaking change, because existing subclasses that currently define identical command line arguments will no longer work. Subclasses must be updated to remove their retry arguments and allow the base class to define them.

  • feat: Allow custom "skip stage" phases (7b27bfa)

    Provide subclass developers with the flexibility to specialize the "skip stage" actions on a stage by stage basis.

  • feat: Allow custom "end stage" phases (b30ea55)

    Provide subclass developers with the flexibility to specialize the "end stage" actions on a stage by stage basis.

  • feat: Add post-stage actions (8670752)

    Add the flexibility to allow subclass designers to specify post-stage actions to run after a stage ends. These can be specified for all stages, and they can also optionally be customized on a stage by stage basis.

  • feat: Allow custom "begin stage" phases (f05a5d4)

    Provide subclass developers with the flexibility to specialize the "begin stage" actions on a stage by stage basis.

  • feat: Add pre-stage actions (fb6d51e)

    Add the flexibility to allow subclass designers to specify pre-stage actions to run before a stage begins. These can be specified for all stages, and they can also optionally be customized on a stage by stage basis.

  • feat: Add script execution summary (689a84a)

    Add the ability to generate a summary of everything that was done by a derivative of DriverScript, including:

    • The effective command line invocation of the script.
    • Any commands executed in the underlying shell.
    • A timing report of the stages executed.
    • Any additional information the user wishes to pass in.

  • feat!: Allow stages to be run multiple times (e147b03)

    Convert the durations attribute from a dict to a list to allow for the possibility that one or more stages might be run multiple times (e.g., if something failed). Introduce a StageDuration class that inherits from NamedTuple, such that the entries added to the list must always be a (str, timedelta) tuple that will from then on be immutable and allow easy member access via dot notation.

Refactoring

  • refactor: Use functools.cached_property instead (3d91282)

    This package was originally developed for Python 3.6, before functools.cached_property was introduced. That should be a drop-in replacement for the home-grown lazy_property, though, so now that we just support Python 3.8+, we should use it instead.

  • refactor: Support down to Python 3.8 (0c69f03)

    • Switch type hinting to the older style.
    • Use dict.update() instead of the |= operator.

  • refactor: Reduce setup file (b233553)

    Remove the guts of the setup.py file so it just uses the pyproject.toml under the hood.

  • refactor: Specify UTC timezone (1a95cd0)

  • refactor: Explicitly don't check for errors (ed9d050)

  • refactor: Re-raise exception correctly (889645d)

  • refactor: Save exception messages to variable (2715390)

  • refactor!: Make boolean parameters keyword-only (2f3098f)

  • refactor!: Rename package (8f42c66)

    Rename driver-script to staged-script in preparation for a future decoupling of the stage behavior from the shell interaction.

  • refactor: Rearrange DriverScript class (e351982)

    Rearrange the methods in the DriverScript class to make the class easier to understand. No functional changes; just cut from here, paste to there. Also prefix some methods with _ to indicate they're only intended for internal use.

  • refactor!: Stages no longer registered by default (b3ac262)

    Since its inception, DriverScript's stage decorator has attempted to capture all the stages defined by DriverScript subclasses in the stages class variable automatically. The desire was to keep things as easy as possible for subclass developers, in that all they'd need to do was inherit from DriverScript, define some methods decorated with stage, and then DriverScript would automatically handle things like stage-specific arguments (retries, etc.) and methods (pre/post, begin/end, etc.). Unfortunately this feature has been a thorn in our side for development, due to, e.g., pytest multiple-loading modules (and therefore redefining stages), DriverScript subclass hierarchies, etc.

    In order to pave the way for the flexibility for one DriverScript subclass to use stages defined within a separate DriverScript subclass, this commit makes it such that stages are no longer automatically registered. Instead subclass developers must pass the set of stages to register into the constructor when instantiating an object. This is a minor inconvenience, but it removes a good deal of hacky code and makes it such that DriverScript subclasses are decoupled from one another, so we believe the change is well worth it in the long run.

    Note that this is a breaking change, as all subclasses must be update their constructor definitions and any instantiations.

  • refactor!: Switch stages to standard set (525fd79)

    Initially the stages attribute of a DriverScript class was implemented as an ordered set (which in Python is a value-less dict converted to a list). The thinking there was that a DriverScript subclass should define a series of stages in order and they would be executed in that order. However, this failed to consider the following use case: Class A inherits from DriverScript and defines a few stages, then class B inherits from A and defines few more, but the intended sequence of stages isn't all from A followed by all from B; instead, they're meant to be interspersed. Since this is a valid use case, it makes sense to switch stages over to a standard set.

    Another motivation for the stages being an ordered set was considering the possibility of wanting to return to a prior stage if a certain stage failed. Consider an example DriverScript subclass with stages to configure, build, and test a code. If the test stage fails, perhaps the failure was seemingly random, and it'd be worthwhile to automatically rewind and retry the build and test stages again before notifying anyone of the failure. Or perhaps a certain regex match in the test output indicates you should rewind further and reconfigure. However, after further reflection, allowing such flexibility would open up a can of worms that probably shouldn't be opened. DriverScript already allows a subclass developer to automatically retry a stage. If something's happening that would cause you to back up further than the current stage and try things again, that probably means there's flakiness either in your code base or its surrounding infrastructure that a human should look into, so DriverScript probably shouldn't allow you to hide that.

    Note that this might be a breaking change if any subclasses were relying on the ordering of the stages.

  • refactor: Use function from reverse_argparse (1c7fae0)

    Since reverse_argparse now provides quote_arg_if_necessary() as a public function, use this instead of reimplementing the functionality in DriverScript.

  • refactor!: Set current stage earlier (27d571f)

    Set the current_stage upon entering the stage wrapper, rather than in the _begin_stage method. Use current_stage in place of the stage_name passed into the wrapper, where appropriate.

    Note that this is a breaking change, as it changes the signature for the _begin_stage method.

  • refactor: Get phase method rather than running it (8af3f75)

    In preparation for adding automatic retry functionality to the stage decorator, rewrite the run_phase method such that instead of running the method corresponding to the phase, it simply returns the Callable, which is then run outside the function. This generalization will make it such that we can pass a phase method to the Retrying object when we're ready for that.

  • refactor: Wrap retryable phases into a function (345b77c)

    In preparation for adding automatic retry functionality to the stage decorator, wrap the phases that are to be retried into a function so they're easy to call together.

  • refactor!: Stages always return None (b1238be)

    In preparation for adding stage retry functionality, make it such that a function decorated with DriverScript.stage must return None. This is reasonable because a conceptual stage of a script does something, but doesn't return something.

    Note that this is a breaking change for DriverScript, as any subclasses relying on returning values from functions decorated with stage will no longer work. A follow-up commit will fix this problem for GMSSystemTest.

  • refactor: Change custom naming convention (e0b0f5f)

    For simplicity's sake, when specializing one of the stage phases for a particular stage, just append the stage name to the end of the default method name.

  • refactor: Consolidate helper functions (980dd7c)

    Replace the helper functions inside the stage decorator with a single one that executes the correct methods based on input arguments.

  • refactor: Use finally to end the stage (c8539ba)

    Rather than having _end_stage() called in two different places, consolidate those calls into one using a finally clause of the try block.

  • refactor: Add run method (2fe663a)

    Create a run method to wrap calls to subprocess.run and capture the command executed.

  • refactor: Add parse_args method to base class (c73e8eb)

    Add the parse_args method to the DriverScript base class to handle parsing the command line argument supplied by the base class. This will be overridden and extended by child classes.

  • refactor: Add parser property to base class (1c4f10f)

    Add a parser lazily-evaluated property to the DriverScript base class to handle parsing of arguments supplied by the base class. This will be overridden and extended by child classes.

  • refactor: Use Rich Table for timing report (1d24cd1)

    Rather than generating the timing report table manually, use rich.table.Table to handle the magic automatically. Also abandon the Markdown format for the table, as Rich's default styling looks nicer, and if a user needs to copy it into an issue, they can always place it in a code block.

  • refactor: Create DriverScript base class (bb8ea46)

    Create an initial cut of the DriverScript base class, such that any Python scripts designed to drive a series of commands in the underlying shell can inherit from it.

Testing

  • test: Cover run() in dry-run mode (c44d6ff)

  • test: Add coverage configuration (27bc6d6)

  • test: Fix imports for unit/integration tests (e1836e3)

  • test: Rename tests (2723351)

    The convention is to name tests test_<function_to_test>, but these tests accidentally omitted the leading underscore.

  • test: Rename test fixtures (5f7b063)

    Remove acronyms for clarity.

  • test: Disable warning on too many parametrizations (99e59d9)

  • test: Remove duplicate test case (176bb6f)

  • test: Reduce cognitive complexity (5153a1b)

    Extract a method from the test to reduce the amount of copy/paste/modify.

  • test: Complete testing of DriverScript (298b4a9)

    Create a basic subclass of DriverScript to test the functionality of the stage decorator.

  • test!: Cover _add_stage (e079a72)

    Ensure _add_stage results in the correct list of stages, and that invalid identifiers do indeed throw an exception.

    Note that this is a breaking change because it now disallows certain stage names that were previously allowable.

  • test: Cover additional case (0ede83f)

    Cover the base case when no additional sections are supplied when printing the script execution summary.

Unknown

  • Merge branch 'driver-script-switch-stages-to-set' into 'develop' (9157e38)

    refactor!(DriverScript): Switch stages to standard set

    See merge request gms/gms-common!10807

  • Merge branch 'driver-script-retry-stage-exception' into 'develop' (10783ae)

    DriverScript: Create RetryStage exception

    See merge request gms/gms-common!10742

  • Merge branch 'driver-script-custom-help-formatter' into 'develop' (928203c)

    DriverScript: Add help formatter to package

    See merge request gms/gms-common!10688

  • Merge branch 'driver-script-fix-help-text' into 'develop' (e04abe6)

    DriverScript: Fix retry group help text

    Closes #2272

    See merge request gms/gms-common!10685

  • Merge branch '2216-fix-how-driver-script-handles-dry-run' into 'develop' (f94c3fd)

    Fix how DriverScript handles --dry-run

    Closes #2216

    See merge request gms/gms-common!10576

  • Merge branch 'move-script-success-to-driver-script' into 'develop' (6e1f46e)

    Move script_success to DriverScript

    Closes #2215

    See merge request gms/gms-common!10574