Merge pull request #16 from ScreamBun/master

Fall cleaning update for Yuuki development

Author: ScreamBun

README.md

@@ -1,44 +1,50 @@
 
-## OASIS TC Open Repository: openc2-yuuki
+<a href="https://openc2.org/" target="_blank">![OpenC2](https://github.com/ScreamBun/SB_Utils/blob/master/assets/images/openc2.png?raw=true)</a>
+
+[![Python 3.8+](https://img.shields.io/badge/Python-3.8%2B-yellow)](https://www.python.org/downloads/release/python-3100/) 
+[![Open2C Lang Spec](https://img.shields.io/badge/Open2C%20Lang%20Spec-1.0-brightgreen)](https://openc2.org/specifications)
 
-This GitHub public repository ( [https://github.com/oasis-open/openc2-yuuki](https://github.com/oasis-open/openc2-yuuki) ) was created at the request of the [OASIS Open Command and Control (OpenC2) TC](https://www.oasis-open.org/committees/openc2/) as an [OASIS TC Open Repository](https://www.oasis-open.org/resources/open-repositories/) to support development of open source resources related to Technical Committee work.
-While this TC Open Repository remains associated with the sponsor TC, its development priorities, leadership, intellectual property terms, participation rules, and other matters of governance are [separate and distinct](https://github.com/oasis-open/openc2-yuuki/blob/master/CONTRIBUTING.md#governance-distinct-from-oasis-tc-process) from the OASIS TC Process and related policies.
-All contributions made to this TC Open Repository are subject to open source license terms expressed in the [BSD-3-Clause License](https://www.oasis-open.org/sites/www.oasis-open.org/files/BSD-3-Clause.txt).  That license was selected as the declared ["Applicable License"](https://www.oasis-open.org/resources/open-repositories/licenses) when the TC Open Repository was created.
-As documented in ["Public Participation Invited"](https://github.com/oasis-open/openc2-yuuki/blob/master/CONTRIBUTING.md#public-participation-invited), contributions to this OASIS TC Open Repository are invited from all parties, whether affiliated with OASIS or not.  Participants must have a GitHub account, but no fees or OASIS membership obligations are required. Participation is expected to be consistent with the [OASIS TC Open Repository Guidelines and Procedures](https://www.oasis-open.org/policies-guidelines/open-repositories), the open source [LICENSE](https://github.com/oasis-open/openc2-yuuki/blob/master/LICENSE) designated for this particular repository, and the requirement for an [Individual Contributor License Agreement](https://www.oasis-open.org/resources/open-repositories/cla/individual-cla) that governs intellectual property.
-  
-[<img src="snow_yuuki.jpg" alt="Yuuki" title="Yuuki Image" width="224" height="104"/>](snow_yuuki.jpg)
-  
 ## Table of Contents  
 
-[Introduction](#introduction)  
-[Requirements and Setup](#requirements-and-setup)  
-[Yuuki's Consumer Components](#components-of-a-yuuki-consumer)  
-    * [Consumers](#consumers)  
-    * [Actuators](#actuators)  
-    * [Serializations](#serializations)  
-[Example Consumers](#examples)  
-[Transport Functions](#transport-functions)  
-    * [HTTP](#HTTP)  
-    * [MQTT](#MQTT)  
-    * [OpenDXL(experimental)](#opendxl)  
-[FAQ](#frequently-asked-questions)
+- [Introduction](#introduction)  
+- [Requirements and Setup](#requirements-and-setup)  
+- [Yuuki's Consumer Components](#components-of-a-yuuki-consumer)  
+  - [Consumers](#consumers)  
+  - [Actuators](#actuators)  
+  - [Serializations](#serializations)  
+- [Example Consumers](#examples)  
+- [Transport Functions](#transport-functions)  
+  - [HTTP](#HTTP)  
+  - [MQTT](#MQTT)   
+- [FAQ](#frequently-asked-questions)
 
 
 ## Introduction
 Yuuki is a tool for creating OpenC2 Consumers.
 Open Command and Control, or OpenC2, is a standardized language for the command and control of technologies that provide or support cyber defenses.
 OpenC2 Commands are sent by Producer devices to Consumers that receive and implement Commands.  
-OpenC2 is defined in the [OpenC2 Architecture Specification](https://docs.oasis-open.org/openc2/oc2arch/v1.0/csd02/oc2arch-v1.0-csd02.md) and [OpenC2 Language Specification](https://github.com/oasis-tcs/openc2-oc2ls/blob/published/oc2ls-v1.0-cs02.md)
+OpenC2 is defined in the [OpenC2 Architecture Specification](https://docs.oasis-open.org/openc2/oc2arch/v1.0/csd02/oc2arch-v1.0-csd02.md) and [OpenC2 Language Specification](https://github.com/oasis-tcs/openc2-oc2ls/blob/published/oc2ls-v1.0-cs02.md).
+
+### Background
+
+This GitHub public repository ( [https://github.com/oasis-open/openc2-yuuki](https://github.com/oasis-open/openc2-yuuki) ) was created at the request of the [OASIS Open Command and Control (OpenC2) TC](https://www.oasis-open.org/committees/openc2/) as an [OASIS TC Open Repository](https://www.oasis-open.org/resources/open-repositories/) to support development of open source resources related to Technical Committee work.
+
+While this TC Open Repository remains associated with the sponsor TC, its development priorities, leadership, intellectual property terms, participation rules, and other matters of governance are [separate and distinct](https://github.com/oasis-open/openc2-yuuki/blob/master/CONTRIBUTING.md#governance-distinct-from-oasis-tc-process) from the OASIS TC Process and related policies.
 
-#### Statement of Purpose
+All contributions made to this TC Open Repository are subject to open source license terms expressed in the [BSD-3-Clause License](https://www.oasis-open.org/sites/www.oasis-open.org/files/BSD-3-Clause.txt).  That license was selected as the declared ["Applicable License"](https://www.oasis-open.org/resources/open-repositories/licenses) when the TC Open Repository was created.
+
+As documented in ["Public Participation Invited"](https://github.com/oasis-open/openc2-yuuki/blob/master/CONTRIBUTING.md#public-participation-invited), contributions to this OASIS TC Open Repository are invited from all parties, whether affiliated with OASIS or not.  Participants must have a GitHub account, but no fees or OASIS membership obligations are required. Participation is expected to be consistent with the [OASIS TC Open Repository Guidelines and Procedures](https://www.oasis-open.org/policies-guidelines/open-repositories), the open source [LICENSE](https://github.com/oasis-open/openc2-yuuki/blob/master/LICENSE) designated for this particular repository, and the requirement for an [Individual Contributor License Agreement](https://www.oasis-open.org/resources/open-repositories/cla/individual-cla) that governs intellectual property.
+  
+
+### Statement of Purpose
 Statement of Purpose for this OASIS TC Open Repository (openc2-yuuki) as [proposed](https://drive.google.com/open?id=0B-FunCZrr-vtcUJTWVBNaFNlVUE) and [approved](https://www.oasis-open.org/committees/ballot.php?id=3115) [[bis]](https://issues.oasis-open.org/browse/TCADMIN-2746) by the OpenC2 TC:  
 The purpose of the openc2-yuuki GitHub repository is to   
     (a) demonstrate the implementation of OpenC2 via multiple dispatch on type, and    
     (b) provision a codebase to enable other prototype efforts.    
 The initial codebase for the openc2-yuuki repository is imported from the OpenC2 Forum's Github repository.  
 
-Yuuki is designed to be a good introduction to OpenC2, to facilitate experimentation with different Actuator profiles,  
-transfer protocols and message serializations, and to provide a simple OpenC2 Consumer for OpenC2 Producers to test against.  
+Yuuki is designed to be a good introduction to OpenC2, to facilitate experimentation with different Actuator profiles, transfer protocols, and message serializations, and to provide a simple OpenC2 Consumer for OpenC2 Producers to test against.  
+
 The three main components of Yuuki are the [Consumer](consumers), [Actuator](#actuators), and [Serialization](#serializations) classes, 
 defined respectively in the `consumer.py`, `actuator.py`, and `serialization.py` files.
 
@@ -126,7 +132,7 @@ An Actuator is identified by a string representing the namespace identifier (`ns
 Actuators consist of a number of action- target pairs, and inherit from the Actuator class, 
 giving them access to initialization, pair definition and registration, and some basic error handling.
 
-For example, see the sample implementation of an Actuator based on the [Stateless Packet Filtering](https://docs.oasis-open.org/openc2/oc2slpf/v1.0/oc2slpf-v1.0.html) Actuator profile in `examples/actuators/slpf.py`
+For example, see the sample implementation of an Actuator based on the [Stateless Packet Filtering](https://docs.oasis-open.org/openc2/oc2slpf/v1.0/oc2slpf-v1.0.html) Actuator profile in `examples/actuators/slpf.py`.
 [Stateless Packet Filtering](https://docs.oasis-open.org/openc2/oc2slpf/v1.0/oc2slpf-v1.0.html) is a standard Actuator profile with the nsid: `slpf`.
 nsids of nonstandard Actuator profiles are prefixed with `x-`.
 
@@ -236,8 +242,7 @@ Yuuki's Consumer functions require it has OpenC2 to read. Transport functions ar
 These are found under `/openc2_arch/transports` and have `__init__,` `config` and `transport` functions.  
 These were not listed with the other core parts of Yuuki only because they interact with its Consumer logic very little.  
 They are very important, but they deal with transporting serialized messages, not OpenC2 Commands.  
-This is where your connection info is sent to properly establish connections.  
-Tinker with caution!  
+This is where your connection info is sent to properly establish connections, so tinker with caution!  
 
 ### MQTT
 You can find the OpenC2 MQTT Transfer Specification [Here](https://github.com/oasis-tcs/openc2-transf-mqtt/blob/published/transf-mqtt-v1.0-cs01.md).
@@ -266,15 +271,6 @@ python examples/http_example.py
 python examples/producers/http_producer.py
 ```
 
-### OpenDXL
-
-| :warning:        | *Support for OpenDXL is experimental*|
-|------------------|:-------------------------------------|
-
-This example uses both the Event and Request/Response messaging capabilities of OpenDXL to send and receive OpenC2 Messages.
-
-An OpenDXL configuration file is required to run these examples.
-
 #### Start Consumer:
 ```sh
 python examples/opendxl_example.py PATH_TO_OPENDXL_CONFIG
@@ -310,10 +306,8 @@ Good Luck, and Have Fun!
 TC Open Repository [Maintainers](https://www.oasis-open.org/resources/open-repositories/maintainers-guide) are responsible for oversight of this project's community development activities, including evaluation of GitHub [pull requests]() and [preserving](https://www.oasis-open.org/policies-guidelines/open-repositories#repositoryManagement) open source principles of openness and fairness. Maintainers are recognized and trusted experts who serve to implement community goals and consensus design preferences.
 Initially, the associated TC members have designated one or more persons to serve as Maintainer(s); subsequently, participating community members may select additional or substitute Maintainers, per [consensus agreements](https://www.oasis-open.org/maintainers-guide/#additionalMaintainers).
 Current Maintainers of this TC Open Repository  
-  
-* [Dave Kemp]([email protected]); GitHub ID: [https://github.com/davaya](https://github.com/davaya); WWW: [Department of Defense](www.nsa.gov)  
-* [Joshua Brulé](mailto:[email protected]); GitHub ID: [https://github.com/jtcbrule](https://github.com/jtcbrule); WWW: [University of Maryland](https://umd.edu/)
-* [David Lemire](mailto:[email protected]); GitHub ID: [https://github.com/dlemire60](https://github.com/dlemire60); WWW: [National Security Agency](www.nsa.gov)
+
+* [David Lemire](mailto:[email protected]); Email: [email protected] 
 * The ScreamingBunny Development team; GitHub ID: [https://github.com/ScreamBun](https://github.com/ScreamBun)
 
 #### Where can I learn about OASIS TC Open Repositories?

examples/mqtt_consumer_full.py

@@ -11,12 +11,11 @@
 
 from actuators.slpf import slpf
 
-
 consumer = Consumer(rate_limit=60, versions=['1.0'], actuators=[slpf])
 
-host = "127.0.0.1"
+host = "test.mosquitto.org"
 port = 1883
-topics = ['oc2/cmd', 'oc2/cmd/ap/slpf', 'oc2/cmd/ap/database', 'oc2/cmd/ap/sbom']
+topics = ['oc2/cmd', 'oc2/cmd/ap/slpf']
 
 mqtt_config = MqttConfig(
     broker=BrokerConfig(
@@ -25,7 +24,7 @@
         client_id='',
         keep_alive=300,
         authorization=MQTTAuthorization(
-            enable=True,
+            enable=False,
             username='plug',
             password='fest'
         ),

examples/mqtt_consumer_er.py examples/mqtt_consumer_plugfest.py

@@ -1,23 +1,23 @@
 """
 Example Implementation of an OpenC2 MQTT Consumer
 """
-from yuuki.transports import (
+from oc2_arch.transports import (
     MqttTransport, MqttConfig, MQTTAuthorization, MQTTAuthentication, BrokerConfig, Publication, Subscription
 )
 
-from yuuki import Consumer
+from oc2_arch import Consumer
 # import actuator profiles for your consumer
 
-
-from actuators.er import er
+from actuators.database import database
+from actuators.sbom import sbom
 from actuators.slpf import slpf
 
 
-consumer = Consumer(rate_limit=60, versions=['1.0'], actuators=[er, slpf])
+consumer = Consumer(rate_limit=60, versions=['1.0'], actuators=[sbom, database, slpf])
 
-host = "127.0.0.1"
+host = '35.221.11.97'
 port = 1883
-topics = ['oc2/cmd', 'oc2/cmd/ap/er']
+topics = ['oc2/cmd', 'oc2/cmd/ap/slpf', 'oc2/cmd/ap/database', 'oc2/cmd/ap/sbom']
 
 mqtt_config = MqttConfig(
     broker=BrokerConfig(

examples/producers/openc2_command_er.py

@@ -1,4 +1,5 @@
 SB-Utils@git+https://github.com/ScreamBun/SB_utils.git#subdirectory=root
+OSQuery-ORM@git+https://github.com/ScreamBun/SB_utils.git#subdirectory=osquery_orm
 dxlclient
 elasticsearch
 flask

requirements.txt

@@ -22,7 +22,6 @@ classifiers =
 [options]
 include_package_data = true
 packages = find:
-package_dir = src
 python_requires = >=3.8
 setup_requires = setuptools_scm
 

setup.cfg

@@ -10,6 +10,8 @@ def get_requirements():
 
 setup(
     name='Yuuki',
-
+    package_data={
+        "yuuki": ["./src/yuuki/*"]
+    },
     install_requires=get_requirements()
 )

setup.py

@@ -4,7 +4,7 @@
 import json
 import logging
 from time import time
-from pprint import pformat  # properly prints JSON serielized text as part of return messages, useful for SBOM etc
+from pprint import pformat  # properly prints JSON serialized text as part of return messages, useful for SBOM etc
 from concurrent.futures import ThreadPoolExecutor
 from functools import partial
 from typing import Any, Callable, Dict, List, Union
@@ -53,7 +53,7 @@ def __init__(self, rate_limit: int, versions: List[str], actuators: List[Actuato
                 self.add_serialization(serialization)
         if actuators is not None:
             for actuator in actuators:
-                self.add_actuator_profile(actuator)
+                self.add_actuator(actuator)
                 print("Added actuator "+actuator.nsid)
         self.executor = ThreadPoolExecutor()
         print(r'''
@@ -88,6 +88,11 @@ def process_command(self, command, encode: str) -> Union[str, bytes, None]:
 
         try:
             openc2_msg = OpenC2Msg(**message)
+
+
+            print(openc2_msg)
+
+
         except ValidationError as e:
             #logging.error(e)
             openc2_rsp = OpenC2RspFields(status=StatusCode.BAD_REQUEST, status_text='Malformed OpenC2 message')
@@ -181,21 +186,21 @@ def _get_actuator_callable(self, oc2_msg: OpenC2Msg) -> Callable[[], OpenC2RspFi
         :return: The function with the received OpenC2 Command supplied as an argument.
         """
         oc2_cmd = oc2_msg.body.openc2.request
-        print(f"{oc2_cmd.action} {oc2_cmd.target_name} {oc2_cmd.actuator_name}")
+        print(f"{oc2_cmd.action} {oc2_cmd.target_name} {oc2_cmd.profile_name}")
         print(self.dispatch)
         print(self.understood)
         if oc2_cmd.action == 'query' and oc2_cmd.target_name == 'features':
             function = self.query_features
         elif oc2_cmd.action in self.dispatch and oc2_cmd.target_name in self.dispatch[oc2_cmd.action]:
-            if oc2_cmd.actuator_name is None:
+            if oc2_cmd.profile_name is None:
                 # Behavior of duplicate Action-Target pairs is currently undefined in the OpenC2 language.
                 # For the time being, this is handled by calling the function of the first matching pair.
                 function = next(iter(self.dispatch[oc2_cmd.action][oc2_cmd.target_name].values()))
             else:
-                if oc2_cmd.actuator_name in self.dispatch[oc2_cmd.action][oc2_cmd.target_name]:
-                    function = self.dispatch[oc2_cmd.action][oc2_cmd.target_name][oc2_cmd.actuator_name]
+                if oc2_cmd.profile_name in self.dispatch[oc2_cmd.action][oc2_cmd.target_name]:
+                    function = self.dispatch[oc2_cmd.action][oc2_cmd.target_name][oc2_cmd.profile_name]
                 else:
-                    raise TypeError(f'No Actuator: {oc2_cmd.actuator_name}')
+                    raise TypeError(f'No Actuator: {oc2_cmd.profile_name}')
         elif oc2_cmd.action in self.understood and oc2_cmd.target_name in self.understood[oc2_cmd.action]:
             function = self.unimplemented_command_function
         else:
@@ -255,7 +260,7 @@ def unimplemented_command_function(self, oc2_cmd: OpenC2CmdFields) -> OpenC2RspF
         print("Command Not Implemented")
         return OpenC2RspFields(status=StatusCode.NOT_IMPLEMENTED, status_text='Command Not Supported')
 
-    def add_actuator_profile(self, actuator: Actuator) -> None:
+    def add_actuator(self, actuator: Actuator) -> None:
         """
         Adds the Actuator's functions to the Consumer and adds the Actuator's namespace identifier (nsid) to the
         list of supported profiles

snow_yuuki.jpg

@@ -46,11 +46,12 @@ class OpenC2CmdFields(BaseModel, extra=Extra.forbid):
     action: str
     target: Dict[str, Any]
     args: Optional[Union[OpenC2CmdArgs, Dict[str, Any]]]
-    actuator: Optional[Dict[str, Dict[str, Any]]]
+    profile: Optional[Dict[str, Dict[str, Any]]]
     command_id: Optional[str]
 
-    @validator('target', 'actuator')
+    @validator('target', 'profile')
     def validate_choice_length(cls, choice: Dict):
+
         if len(choice) != 1:
             raise ValueError('Choice fields must have a length of one')
         return choice
@@ -60,11 +61,11 @@ def target_name(self):
         return next(iter(self.target))
 
     @property
-    def actuator_name(self):
-        if self.actuator is None:
+    def profile_name(self):
+        if self.profile is None:
             return None
         else:
-            return next(iter(self.actuator))
+            return next(iter(self.profile))
 
 
 class OpenC2Cmd(BaseModel, extra=Extra.forbid):

src/yuuki/consumer.py

@@ -2,4 +2,3 @@
 from .mqtt import (
     MqttTransport, MqttConfig, MQTTAuthorization, MQTTAuthentication, BrokerConfig, Subscription, Publication
 )
-from .opendxl import OpenDxlTransport, OpenDxlConfig