Feature/acp 3507/acp 3746 persist webhooks and allow to run them later

.github/workflows/ci.yml

@@ -51,6 +51,11 @@ jobs:
             - name: Composer install
               run: composer install --optimize-autoloader
 
+            - name: Setup Database
+              run: |
+                  mkdir -p tests/_data
+                  touch tests/_data/app_webhook_db
+
             - name: Setup AppWebhook
               run: composer setup
 

.gitignore

@@ -20,6 +20,9 @@ composer.lock
 src/Generated
 src/Orm
 tests/_data/app_webhook_db
+data/DE/logs/ZED/propel.log
+
+data/cache/
 
 # tests
 tests/**/_generated/

README.md

@@ -1,8 +1,8 @@
-# AppWebhook Package
+# AppWebhook Module
 [![Latest Stable Version](https://poser.pugx.org/spryker/app-webhook/v/stable.svg)](https://packagist.org/packages/spryker/app-webhook)
-[![Minimum PHP Version](https://img.shields.io/badge/php-%3E%3D%208.1-8892BF.svg)](https://php.net/)
+[![Minimum PHP Version](https://img.shields.io/badge/php-%3E%3D%208.2-8892BF.svg)](https://php.net/)
 
-Provides SyncAPI and AsyncAPI schema files and the needed code to make the Mini-Framework an App.
+Provides SyncAPI and AsyncAPI schema files and the needed code to make the Mini-Framework an App with Webhook capabilities.
 
 ## Installation
 
@@ -36,3 +36,68 @@ With these commands you've set up the AppWebhook and can start the tests
 vendor/bin/codecept build
 vendor/bin/codecept run
 ```
+
+### Documentation
+
+### Webhook handling
+
+This package is responsible to receive and handle webhooks. The package provides a controller that can be used to handle incoming webhooks.
+
+The API endpoint is `/webhooks` and the controller is `WebhooksController` inside the Glue Application. This package is not handling webhooks on its own, you must implement the logic to handle the webhooks via the provided `\Spryker\Zed\AppWebhook\Dependency\Plugin\WebhookHandlerPluginInterface`, see the description down below.
+
+#### Process in a Nutshell
+
+- A webhook is received by the `WebhooksController` and the `WebhookRequestTransfer` is created.
+- The webhook content is mapped to a `WebhookRequestTransfer`.
+- The `WebhookRequestTransfer` is passed together with a `WebhookResponseTransfer` to the `\Spryker\Glue\AppWebhookBackendApi\Dependency\Facade\AppWebhookBackendApiToAppWebhookFacadeInterface::handleWebhook()` method.
+- The `handleWebhook` method does:
+  - Creates an identifier for this specific webhook.
+  - When it is not a retried webhook the webhook will be persisted in the database.
+  - In case of a retried webhook the webhook will be fetched from the database.
+  - When the number of retries exceeds the configurable allowed number of retries the webhook will be removed from the database and an exception will be thrown.
+  - Find the correct handler for the webhook and call the `handle` method of the handler.
+  - When the handler returns a failed `WebhookResponseTransfer` the response will be persisted in the database.
+  - When the handler returns a not handled `WebhookResponseTransfer` the response will be persisted in the database with a message that was provided by the implementation of the plugin.
+  - When the handler throws an exception the exception message will be persisted in the database.
+  - When the handler returns a successful `WebhookResponseTransfer` the webhook will be removed from the database.
+  - It returns the `WebhookResponseTransfer` to the controller.
+- The controller formats the `WebhookResponseTransfer` into a Glue response which will be either:
+  - 200 OK in case everything went well.
+  - 400 BAD REQUEST in case of a failed response.
+
+#### Retry Mechanism
+
+In a case when a webhook can not be handled it is persisted in the database and will be retried with the next incoming webhook. The number of retries is configurable and can be set in the `AppWebhookConfig::getAllowedNumberOfRetries()` method.
+
+There are numerous reasons why a webhook may fail. An exception is thrown, the plugin implementation returns a failed response or the plugin implementation returns a not handled response.
+
+Another case could be an event is sent to the application before it is ready to handle it. For example, in the PreOrder payment of a PSP the order has not persisted yet and has no order-reference, but the PSP sends a webhook request about a payment state, in this case, the system has to wait until it can process the webhook.
+
+##### Future improvements for the Retry mechanism
+
+It may be helpful in the future to provide a console command that can be used to retry failed webhooks. This command can be used to retry all failed webhooks or only a specific webhook.
+
+### Configuration
+
+Currently only the number of allowed retries can be configured. The configuration can be found in the `AppWebhookConfig` class.
+
+### Plugins
+
+### GlueApplication
+
+#### \Spryker\Glue\AppWebhookBackendApi\Plugin\GlueApplication\AppWebhookBackendApiRouteProviderPlugin
+
+This plugin provides the routes for the AppWebhookBackendApi module.
+
+### Extensions
+
+#### \Spryker\Zed\AppWebhook\Dependency\Plugin\WebhookHandlerPluginInterface
+
+This plugin can be implemented by any other module and has two methods:
+
+- `\Spryker\Zed\AppWebhook\Dependency\Plugin\WebhookHandlerPluginInterface::canHandle(WebhookRequestTransfer $webhookRequestTransfer): bool`
+- `\Spryker\Zed\AppWebhook\Dependency\Plugin\WebhookHandlerPluginInterface::handle(WebhookRequestTransfer $webhookRequestTransfer): WebhookResponseTransfer`
+
+The `canHandle()` method is used to check if a webhook can be handled by a specific module. F.e. you have two handlers one for `order.created` and one for `order.updated` you can check in the `canHandle()` method if the webhook can be handled by the module and return true or false.
+
+The `handle()` method is used to handle the webhook. The method is called if the `canHandle()` method returns true. The method should return a `WebhookResponseTransfer` with the status of the webhook handling.

architecture-baseline.json

@@ -0,0 +1 @@
+[]

codeception.yml

@@ -2,6 +2,7 @@ namespace: SprykerTest
 
 include:
     - tests/SprykerTest/Glue/AppWebhookBackendApi/
+    - tests/SprykerTest/Zed/AppWebhook/
 
 bootstrap: _data/Environment.php
 

composer.json

@@ -4,7 +4,7 @@
   "description": "AppWebhook module",
   "license": "proprietary",
   "require": {
-    "php": ">=8.1",
+    "php": ">=8.2",
     "spryker/glue-application-extension": "^1.0.0",
     "spryker/kernel": "^3.30.0",
     "spryker/log": "^3.0.0",
@@ -27,7 +27,7 @@
     "spryker/app-kernel": "^1.0.0",
     "spryker/code-sniffer": "*",
     "spryker/development": "^3.34.0",
-    "spryker/glue-application": "^1.64",
+    "spryker/glue-application": "^1.64.0",
     "spryker/message-broker-aws": "^1.7.0",
     "spryker/propel": "*",
     "spryker/testify": "*",
@@ -52,12 +52,13 @@
   "scripts": {
     "cs-check": "phpcs -p src/ tests/",
     "cs-fix": "phpcbf -p src/ tests/",
-    "setup": "tests/bin/console transfer:generate && tests/bin/console transfer:databuilder:generate && tests/bin/console dev:ide-auto-completion:zed:generate && tests/bin/console dev:ide-auto-completion:glue:generate",
+    "setup": "tests/bin/console app-webhook:setup && tests/bin/console transfer:generate && tests/bin/console transfer:databuilder:generate && tests/bin/console dev:ide-auto-completion:zed:generate && tests/bin/console dev:ide-auto-completion:glue:generate && tests/bin/console dev:ide-auto-completion:glue-backend:generate && tests/bin/console propel:install && vendor/bin/codecept build",
     "stan": "phpstan analyze src/Spryker/",
     "test": "codecept build && codecept run",
     "test-strict": "vendor/bin/infection --threads=max --min-msi=100 --min-covered-msi=100",
     "test-strict-ci": "vendor/bin/infection --threads=max --logger-github=true --min-msi=100 --min-covered-msi=100",
     "test-cover": "codecept build && codecept run --coverage-xml",
+    "test-cover-html": "codecept build && codecept run --coverage-html",
     "rector": "vendor/bin/rector process src/Spryker/ --config rector.php --ansi",
     "rector-ci": "vendor/bin/rector process src/Spryker/ --config rector.php --ansi --dry-run",
     "local-ci": "composer cs-fix && composer cs-check && composer stan && composer rector-ci && composer test"

config/Shared/config_default.php

@@ -8,6 +8,61 @@
 use Spryker\Shared\ErrorHandler\ErrorHandlerConstants;
 use Spryker\Shared\Kernel\KernelConstants;
 use Spryker\Shared\Log\LogConstants;
+use Spryker\Shared\Propel\PropelConstants;
+use Spryker\Zed\PropelOrm\Business\Builder\ExtensionObjectBuilder;
+use Spryker\Zed\PropelOrm\Business\Builder\ExtensionQueryBuilder;
+use Spryker\Zed\PropelOrm\Business\Builder\ObjectBuilder;
+use Spryker\Zed\PropelOrm\Business\Builder\QueryBuilder;
+
+$connections = [
+    'mysql' => [
+        'adapter' => 'sqlite',
+        'dsn' => 'sqlite:tests/_data/app_webhook_db',
+        'user' => '',
+        'password' => '',
+        'settings' => [],
+    ],
+];
+
+$config[PropelConstants::PROPEL] = [
+    'database' => [
+        'connections' => [],
+    ],
+    'runtime' => [
+        'defaultConnection' => 'default',
+        'connections' => ['default', 'zed'],
+    ],
+    'generator' => [
+        'defaultConnection' => 'default',
+        'connections' => ['default', 'zed'],
+        'objectModel' => [
+            'defaultKeyType' => 'fieldName',
+            'builders' => [
+                // If you need full entity logging on Create/Update/Delete, then switch to
+                // Spryker\Zed\PropelOrm\Business\Builder\ObjectBuilderWithLogger instead.
+                'object' => ObjectBuilder::class,
+                'objectstub' => ExtensionObjectBuilder::class,
+                'query' => QueryBuilder::class,
+                'querystub' => ExtensionQueryBuilder::class,
+            ],
+        ],
+    ],
+    'paths' => [
+        'phpDir' => APPLICATION_ROOT_DIR,
+        'sqlDir' => APPLICATION_SOURCE_DIR . '/Orm/Propel/Sql/',
+        'migrationDir' => APPLICATION_SOURCE_DIR . '/Orm/Propel/Migration_SQLite/',
+        'schemaDir' => APPLICATION_SOURCE_DIR . '/Orm/Propel/Schema/',
+    ],
+];
+
+$config[PropelConstants::ZED_DB_ENGINE] = 'mysql';
+$config[PropelConstants::ZED_DB_HOST] = 'localhost';
+$config[PropelConstants::ZED_DB_PORT] = 1234;
+$config[PropelConstants::ZED_DB_USERNAME] = 'catface';
+$config[PropelConstants::ZED_DB_PASSWORD] = 'catface';
+
+$config[PropelConstants::PROPEL]['database']['connections']['default'] = $connections['mysql'];
+$config[PropelConstants::PROPEL]['database']['connections']['zed'] = $connections['mysql'];
 
 $config[KernelConstants::PROJECT_NAMESPACE] = 'Spryker';
 $config[KernelConstants::PROJECT_NAMESPACES] = ['Spryker'];

phpcs.xml

@@ -36,4 +36,9 @@
         <exclude name="SlevomatCodingStandard.Classes.DisallowConstructorPropertyPromotion.DisallowedConstructorPropertyPromotion"/>
     </rule>
 
+    <!-- PHP 8.0 we will allow the null safe operator -->
+    <rule ref="SlevomatCodingStandard.ControlStructures.DisallowNullSafeObjectOperator">
+        <exclude name="SlevomatCodingStandard.ControlStructures.DisallowNullSafeObjectOperator"/>
+    </rule>
+
 </ruleset>

phpstan.neon

@@ -0,0 +1,4 @@
+parameters:
+    level: 8
+    ignoreErrors:
+          - '#Call to an undefined method Propel\\Runtime\\Collection\\Collection\:\:delete\(\).#'

src/Spryker/Shared/AppWebhook/Transfer/app_webhook.transfer.xml

@@ -4,14 +4,24 @@
     <transfer name="WebhookRequest" strict="true">
         <property name="content" type="string"/>
         <property name="mode" type="string"/>
+        <property name="identifier" type="string" description="This is the default identifier set for internal processing."/>
+        <property name="isRetry" type="bool" description="When this WebhookRequest was tried to be processed before we set this to not persist it again in the database."/>
+        <property name="retries" type="int" description="The number of how often this webhook was retried."/>
     </transfer>
 
     <transfer name="WebhookResponse" strict="true">
         <property name="isSuccessful" type="bool"/>
+        <property name="isHandled" type="bool" description="A Webhook can be accepted for later processing and setting this to false will keep it persisted. When true or null it will be removed from persistence."/>
+        <property name="identifier" type="string" description="The identifier will be set from outside and used to find previously unprocessed webhooks and execute them when possible."/>
+        <property name="sequenceNumber" type="int" description="This number will be set from outside. When there were already unprocessed webhooks for the same identifier this number will be increased."/>
         <property name="content" type="string"/>
         <property name="message" type="string"/>
     </transfer>
 
+    <transfer name="WebhookInboxCriteria" strict="true">
+        <property name="identifiers" type="array" singular="identifier"/>
+    </transfer>
+
     <transfer name="GlueError">
         <property name="message" type="string"/>
     </transfer>

src/Spryker/Zed/AppWebhook/AppWebhookConfig.php

@@ -0,0 +1,25 @@
+<?php
+
+/**
+ * Copyright © 2016-present Spryker Systems GmbH. All rights reserved.
+ * Use of this software requires acceptance of the Evaluation License Agreement. See LICENSE file.
+ */
+
+namespace Spryker\Zed\AppWebhook;
+
+use Spryker\Zed\Kernel\AbstractBundleConfig;
+
+class AppWebhookConfig extends AbstractBundleConfig
+{
+    /**
+     * @api
+     *
+     * @var int
+     */
+    protected const NUMBER_OF_ALLOWED_RETRIES = 9;
+
+    public function getAllowedNumberOfWebhookRetries(): int
+    {
+        return static::NUMBER_OF_ALLOWED_RETRIES;
+    }
+}

src/Spryker/Zed/AppWebhook/AppWebhookDependencyProvider.php

@@ -10,6 +10,9 @@
 use Spryker\Zed\Kernel\AbstractBundleDependencyProvider;
 use Spryker\Zed\Kernel\Container;
 
+/**
+ * @method \Spryker\Zed\AppWebhook\AppWebhookConfig getConfig()
+ */
 class AppWebhookDependencyProvider extends AbstractBundleDependencyProvider
 {
     /**

src/Spryker/Zed/AppWebhook/Business/AppWebhookBusinessFactory.php

@@ -8,18 +8,43 @@
 namespace Spryker\Zed\AppWebhook\Business;
 
 use Spryker\Zed\AppWebhook\AppWebhookDependencyProvider;
+use Spryker\Zed\AppWebhook\Business\Identifier\IdentifierBuilder;
+use Spryker\Zed\AppWebhook\Business\Identifier\IdentifierBuilderInterface;
 use Spryker\Zed\AppWebhook\Business\WebhookHandler\WebhookHandler;
+use Spryker\Zed\AppWebhook\Business\WebhookProcessor\WebhookProcessor;
 use Spryker\Zed\Kernel\Business\AbstractBusinessFactory;
 
+/**
+ * @method \Spryker\Zed\AppWebhook\Persistence\AppWebhookEntityManagerInterface getEntityManager()
+ * @method \Spryker\Zed\AppWebhook\Persistence\AppWebhookRepositoryInterface getRepository()
+ * @method \Spryker\Zed\AppWebhook\AppWebhookConfig getConfig()
+ */
 class AppWebhookBusinessFactory extends AbstractBusinessFactory
 {
     public function createWebhookHandler(): WebhookHandler
     {
         return new WebhookHandler(
             $this->getAppWebhookHandlerPlugins(),
+            $this->getConfig(),
+            $this->getRepository(),
+            $this->getEntityManager(),
+            $this->createIdentifierBuilder(),
         );
     }
 
+    public function createWebhookProcessor(): WebhookProcessor
+    {
+        return new WebhookProcessor(
+            $this->createWebhookHandler(),
+            $this->getRepository(),
+        );
+    }
+
+    public function createIdentifierBuilder(): IdentifierBuilderInterface
+    {
+        return new IdentifierBuilder();
+    }
+
     /**
      * @return array<\Spryker\Zed\AppWebhook\Dependency\Plugin\WebhookHandlerPluginInterface>
      */

src/Spryker/Zed/AppWebhook/Business/AppWebhookFacade.php

@@ -7,12 +7,15 @@
 
 namespace Spryker\Zed\AppWebhook\Business;
 
+use Generated\Shared\Transfer\WebhookInboxCriteriaTransfer;
 use Generated\Shared\Transfer\WebhookRequestTransfer;
 use Generated\Shared\Transfer\WebhookResponseTransfer;
 use Spryker\Zed\Kernel\Business\AbstractFacade;
 
 /**
  * @method \Spryker\Zed\AppWebhook\Business\AppWebhookBusinessFactory getFactory()
+ * @method \Spryker\Zed\AppWebhook\Persistence\AppWebhookEntityManagerInterface getEntityManager()
+ * @method \Spryker\Zed\AppWebhook\Persistence\AppWebhookRepositoryInterface getRepository()
  */
 class AppWebhookFacade extends AbstractFacade implements AppWebhookFacadeInterface
 {
@@ -25,4 +28,24 @@ public function handleWebhook(WebhookRequestTransfer $webhookRequestTransfer, We
     {
         return $this->getFactory()->createWebhookHandler()->handleWebhook($webhookRequestTransfer, $webhookResponseTransfer);
     }
+
+    /**
+     * @api
+     *
+     * @inheritDoc
+     */
+    public function processUnprocessedWebhooks(WebhookInboxCriteriaTransfer $webhookInboxCriteriaTransfer): void
+    {
+        $this->getFactory()->createWebhookProcessor()->processUnprocessedWebhooks($webhookInboxCriteriaTransfer);
+    }
+
+    /**
+     * @api
+     *
+     * @inheritDoc
+     */
+    public function deleteWebhooks(WebhookInboxCriteriaTransfer $webhookInboxCriteriaTransfer): void
+    {
+        $this->getEntityManager()->deleteWebhookRequests($webhookInboxCriteriaTransfer);
+    }
 }