refactor configuration and add router function (#131)

* poc

* create router

* update tests

* add createsnapshot

* add revoke

* each finding should export a method for automations it supports
- migrate createsnapshot

* fix deprecreated terraform string format

* add anomalous iam

* remove configs

* add badip to read cscc notifications

* move target/exclude to router

* fix typo in config

* use proto getter

* improve documentation
- move folder ids to variables
- move config to empty-config.yaml
- add deploy.sh for redploying gcfs

* badip should work from sd ot cscc notifications

* add logging and example in debug/main.go

* increase timeout for long runing create snapshot

* add documentation for removing stackdriver sink if using ncscc notifications

* simplify redeploy
- use a different filename per zip so an apply forces a redeploy

* more documentation edits

* incorporate CheckMatches into router

* remove config

* errors should not break exec if a single automation fails
- correct typo

* add iam finding to debug cli

* remove notification config editor from terraform
- this is granted during setup via gcloud

* correct ignore path

* enable required apis

* modify debug print

* address nits

.gitignore

@@ -10,4 +10,5 @@ deploy
 .history
 .vscode
 .idea
+cloudfunctions/router/config.yaml
 settings.json

README.md

@@ -1,98 +1,96 @@
 # Security Response Automation
 
-Setup automated actions to run on your security findings. You can use our predefined functions to auto-remediate findings as they come in or write and customize your own.
+Take automated actions on your Cloud Security Command Center findings:
 
-- Automatically create a disk snapshot to enable future forensic investigations.
+- Automatically create disk snapshots to enable forensic investigations.
 - Revoke IAM grants that violate your desired policy.
-- Notify other systems such as Turbinia, PagerDuty, Slack or just send an email.
+- Notify other systems such as PagerDuty, Slack or email.
+- See the full list of [automations](/automations.md) for more information.
 
-You can selectively control which resources are enforced by each function. Every action is logged and you can also run in **dry_run** mode where changes are not saved.
+You're in control:
 
-## Note
+- Service account runs with lowest permission needed granted at granularity you specify.
+- You control which projects are enforced by each automation. 
+- Every action is logged to StackDriver and is easy auditable.
+- Can be run in monitor mode where actions are logged only.
 
-This project is currently under development and is not yet ready for users. Stay tuned! If you do decide to use this library early and have any questions please reach out to [email protected] for help.
+### Configure automations
 
-## Getting Started
+Before installation we'll configure our automations, copy `./router/empty-config.yaml` to `./router/config.yaml`. Within this file we'll define a few steps to get started:
 
-This repository contains Cloud Functions to take automated actions on findings from Event Threat Detection and Security Health Analytics (SHA). For example, if SHA alerts you that a Google Cloud Storage bucket is open you may want to close it, or perhaps leave it alone if its meant to be public.
+- Which automations should apply to which findings. 
+- Which projects to target these automations with.
+- Which projects to exclude.
+- Fill in any needed variables.
 
-### Configuration
+## Restricting projects
 
-Before installation we'll configure our automations, rename or copy `empty-settings.json` to `settings.json`. This is done because `settings.json` is ignored by Git so your changes are kept out of our repository and you don't accidently lose your changes. Within this file we'll restrict our automations to only take actions if the affected resource are within a set of resource IDs we declare. For example, you may want to revoke IAM grants in your development environment but in your prod environment you may want to monitor only.
+Every automation accepts a `target` and `exclude` array that accepts an ancestry pattern that is compared against the incoming project. For example lets say you have a folder `424242424242` that contains sensitive projects that you want to enforce. However your developers use folder ID `5656565656` that you want to leave alone. If you have projects outside of folders you can specify them too.
 
-- For a full list of automations and their individual configurations see [automations](/automations.md).
-- For each resource ID (folder, project, or organization) you configure below you'll also need to modify (main.tf)[/main.tf] so Terraform can grant the required permissions.
+In this case your configuration could look like:
 
-Each automation that considers resources will support the following resources:
-
-#### Resources
-
-- Project IDs `folder_ids`: Take the action if the affected project ID is within this set.
-- Folder IDs `project_ids`: Take the action if the affected project ID has an ancestor of a folder ID within this set.
-- Organization ID `organization_id`: Take the action if the affected project ID is within this organization ID.
-
-Each automation will check if it's affected project is within the configured resources and only take an action if there's a match. Setting an `organization_id` in a automation's configuration will allow every project within the organization to affected by that automation.
-
-### Example
-
-In the [automations](/automations.md) documentation we see that this automation is configured in [settings.json](settings.json) under the `revoke_iam` key. Within this key we'll fill out which projects will be enforced, in this example we'll specify a folder along with an allow list of expected domains.
-
-```json
-{
-  "revoke_grants": {
-    "resources": {
-      "folder_ids": ["670032686187"],
-      "organization_id": "",
-      "project_ids": []
-    },
-    "allow_domains": ["google.com", "googleplex.com"]
-  }
-}
+```yaml
+target:
+  - organizations/1234567890/folders/424242424242/*
+  - organizations/1234567890/projects/77981237242
+excludes:
+  - organizations/1234567890/folders/5656565656/*
 ```
 
-Since we're using folders we'll also want to modify [main.tf](/main.tf) to inform Terraform which folders we're enforcing so the required roles are automatically granted. If you choose you can leave out this step but you must authorize the SRA service account to have the necessary roles to revoke the IAM grants. You could grant the account `Project IAM Admin` role on each project ID you want enforced then add the project IDs to the above `project_ids` key. You could also grant the role at the organization level and enter your organzation ID in the `organization_id`.
-
-```terraform
-module "revoke_iam_grants" {
-  source = "./terraform/automations/revoke-iam-grants"
-  setup  = "${module.google-setup}"
-  folder-ids = [
-    "670032686187",
-  ]
-}
+In the [automations](/automations.md) documentation we see that this automation is configured in [config.yaml](config.yaml) under the action name `revoke_iam`. In this example we'll configure Security Response Automation to apply this automation to Event Threat Detection's Anomalous IAM Grant findigns.
+
+It's important to note this automation requires the `allow_domains` to contain at least one valid domain. This is used to ensure SRA only removes domains not explictly allowed. It's also best practice to run SRA with `dry_run` enabled. This way you can let SRA generate StackDriver logs to see what actions it would have taken. Once you confirm this is as expected you can set `dry_run` to false and redeploy.
+
+```yaml
+apiVersion: security-response-automation.cloud.google.com/v1alpha1
+kind: Remediation
+metadata:
+  name: router
+spec:
+  parameters:
+    etd:
+      anomalous_iam:
+        - action: iam_revoke
+          target:
+            - organizations/1234567890/folders/424242424242/*
+          exclude:
+          properties:
+            dry_run: false
+            allow_domains:
+              - foo.com
 ```
 
-### Installation
+#### Configuring permissions
 
-Following these instructions will deploy all automations. Before you get started be sure
-you have (at least) **Go version 1.11 installed**.
+The service account is configured separately within [main.tf](/main.tf). Here we inform Terraform which folders we're enforcing so the required roles are automatically granted. You have a few choices for how to configure this step:
 
-```shell
-$ gcloud auth application-default login
-$ terraform init
+- **Recommended** Specify a list of folder IDs that SRA could grant its service account the necessary roles to. This ensures SRA only has the access it needs at the folders where it's being used. This list will be asked below in the **Installation** section.
+- Grant permissions on your own either per project or at the organizational level.
 
-// Install all Functions.
-$ terraform apply
+#### Forward findings to Pub/Sub
 
-// Install a single Function.
-$ terraform apply --target module.revoke_iam_grants
-```
+Currently Event Threat Detection publishes to StackDriver and CSCC, Security Health Analytics publishes to CSCC only. We're currently in the process of moving to CSCC notifications but for completeness sake we'll list instructions for StackDriver (legacy) and CSCC notifications.
 
-TIP: Instead of entering variables every time you can create `terraform.tfvars`
-file and input key value pairs there, i.e.
-`automation-project="aerial-jigsaw-235219"`.
+**StackDriver** 
 
-If at any point you want to revert the changes we've made just run `terraform destroy .`
+If you're only interested in processing ETD findings then your configuration is done for you automatically below using Terraform. You can skip the **Setup CSCC Notifications** section.
+
+**Setup CSCC Notifications**
 
-**CSCC Notifications**
+CSCC Notifications will enable you to receive SHA & ETD findings. 
 
-Security Health Analytics requires CSCC notifications to be setup. This requires your account to be added to a early access group, please ping [email protected] to be added. You can then create a new notification config that will send all CSCC findings to a Pub/Sub topic.
+Configure CSCC notifications
 
 ```shell
 $ export PROJECT_ID=<YOUR_AUTOMATION_PROJECT_ID>
 $ export SERVICE_ACCOUNT_EMAIL=automation-service-account@$PROJECT_ID.iam.gserviceaccount.com \
   ORGANIZATION_ID=<YOUR_ORGANIZATION_ID> \
-  TOPIC_ID=cscc-notifications-topic
+  TOPIC_ID=threat-findings
+
+$ gcloud beta organizations add-iam-policy-binding \
+	$ORGANIZATION_ID \
+	--member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
+	--role='roles/securitycenter.notificationConfigEditor'
 
 $ gcloud organizations add-iam-policy-binding $ORGANIZATION_ID \
   --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
@@ -101,7 +99,7 @@ $ gcloud organizations add-iam-policy-binding $ORGANIZATION_ID \
 $ go run ./local/cli/main.go \
   --command create \
   --org-id=$ORGANIZATION_ID \
-  --topic=projects/$PROJECT_ID/topics/cscc-notifications-topic12w
+  --topic=projects/$PROJECT_ID/topics/$TOPIC_ID
 
 // Note the output, specifically the generated `service_acount`:
 //
@@ -113,30 +111,57 @@ $ go run ./local/cli/main.go \
 // streaming_config:<filter:"state = \"ACTIVE\"" >
 //
 // Make sure to replace `SERVICE_ACCOUNT_FROM_ABOVE` with the generated service account.
-gcloud beta pubsub topics add-iam-policy-binding projects/$PROJECT_ID/topics/$TOPIC_ID \
-  --member="serviceAccount:<SERVICE_ACCOUNT_FROM_ABOVE>" \
+
+$ gcloud beta pubsub topics add-iam-policy-binding projects/$PROJECT_ID/topics/$TOPIC_ID \
+  --member="serviceAccount:service-459837319394@gcp-sa-scc-notification.iam.gserviceaccount.com" \
   --role="roles/pubsub.publisher"
 
-gcloud organizations remove-iam-policy-binding $ORGANIZATION_ID \
+$ gcloud organizations remove-iam-policy-binding $ORGANIZATION_ID \
   --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
   --role='roles/pubsub.admin'
 ```
 
-### Reinstalling a Cloud Function
+### Installation
 
-Terraform will create or destroy everything by default. To redeploy a single Cloud Function you can do:
+Following these instructions will deploy all automations. Before you get started be sure
+you have:
+
+- Go version 1.11
+- Terraform version 0.12.17
 
 ```shell
-$ zip -r ./deploy/functions.zip . -x *deploy* -x *.git* -x *.terraform*
-$ terraform apply .
+$ gcloud auth application-default login
+$ terraform init
+
+// Install all automations.
+$ terraform apply
+
+// Install a single automations.
+$ terraform apply --target module.revoke_iam_grants
 ```
 
-Then visit Cloud Console, Cloud Functions, click the Function name, edit then deploy.
+**NOTE**
 
-### Test
+If you setup CSCC notifications it's important to remove the StackDriver export so automations are not triggered twice. This is done by running:
 
 ```shell
-$ go test ./...
+$ gcloud logging sinks delete sink-threat-findings --project=$PROJECT_ID
+```
+
+TIP: Instead of entering variables every time you can create `terraform.tfvars`
+file and input key value pairs there, i.e.
+`automation-project="aerial-jigsaw-235219"`.
+
+If at any point you want to revert the changes we've made just run `terraform destroy .`
+
+### Reinstalling a Cloud Function
+
+Terraform will create or destroy everything by default. To redeploy a single Cloud Function you can do:
+
+```shell
+// revoke_iam_grants is the name of the Terraform module in `./main.tf`.
+// IAMRevoke is the exported Cloud Function name in `exec.go`.
+$ scripts/deploy.sh revoke_iam_grants IAMRevoke $PROJECT_ID
 ```
 
 ### Logging
@@ -145,5 +170,3 @@ Each Cloud Function logs its actions to the below log location. This can be acce
 StackDriver and clicking on the arrow on the right hand side then 'Convert to advanced filter'.
 Then paste in the below filter making sure to change the project ID to the project where your
 Cloud Functions are installed.
-
-`logName="projects/{{ project_id }}/logs/security-response-automation"`

automations.md

@@ -26,8 +26,7 @@ Removes members from an IAM policy.
 
 Configuration
 
-- Configured in settings.json under the `revoke_iam` key.
-- See general [resource list](/README.md#resources) options.
+- Action name `iam_revoke`
 
 Before a user is removed the user is checked against the below lists. These lists are meant to be mutually exclusive however this is not enforced. These lists allow you to specify exactly what domain names are disallowed or conversely which domains are allowed.
 
@@ -51,17 +50,19 @@ Automatically create a snapshot of all disks associated with a GCE instance.
 
 Configuration
 
-- Configured in settings.json under the `create_snapshot` key.
-- `snapshot_project_id` Optional project ID where disk snapshots should be sent to. If outputing to Turbinia this should be the same as `turbinia_project_id`.
-- `snapshot_zone` Optional zone where disk snapshots should be sent to. If outputing to Turbinia this should be the same as `turbinia_zone`.
-- `output_destinations` Repeated set of optional output destinations after the function has executed.
+- action `gce_create_disk_snapshot`
+- `target_snapshot_project_id` Project ID where disk snapshots should be sent to. If outputing to Turbinia this should be the same as `turbinia_project_id`.
+- `target_snapshot_project_zone` Zone where disk snapshots should be sent to. If outputing to Turbinia this should be the same as `turbinia_zone`.
+- `output` Repeated set of optional output destinations after the function has executed.
   - `turbinia` Will notify Turbinia when a snapshot is created.
 
 Required if output contains `turbinia`:
 
-- `turbinia_project_id` Project ID where Tubinia is installed.
-- `turbinia_topic_name` Pub/Sub topic where we should notify Turbinia.
-- `turbinia_zone` Zone where Turbinia disks are kept.
+The below keys are placed under the `turbinia` key:
+
+- `project_id` Project ID where Tubinia is installed.
+- `topic_name` Pub/Sub topic where we should notify Turbinia.
+- `zone` Zone where Turbinia disks are kept.
 
 #### Remove public IPs from an instance
 

clients/pubsub.go

@@ -43,5 +43,6 @@ func (p *PubSub) Topic(id string) *pubsub.Topic {
 
 // Publish will publish a message to a PubSub topic.
 func (p *PubSub) Publish(ctx context.Context, topic *pubsub.Topic, message *pubsub.Message) (string, error) {
+	defer topic.Stop()
 	return topic.Publish(ctx, message).Get(ctx)
 }