Introduction of Requirements in pumla docs-as-code approach.

First step with Weather Station example. Creating the reqs_repo file and an overview diagram.

arch/pumla_architecture.md

@@ -56,4 +56,42 @@ See the following image to get an idea of the big picture:
 
 ![](./00_pics/overview_internalstructure.png)
 
+## `pumla` Requirements as Code approach
+In systems engineering, architecture typically is highly connected to requirements engineering. The requirements
+are the base to tell you what is expected from the system. Tests are written against requirements (or are a means
+to describe a requirement). The traceability from architecture elements to requirements is crucial for knowing which
+part of the implementation is intended to do what and why at all. When product variants come into play, the
+start point of the variance is typically already on the requirements. So, not every product variant needs to have
+each requirement fulfilled.
 
+Therefore, bringing requirements engineering and architecture together into one docs-as-code/models-as-code approach
+has the potential to ease the implementation of traceability and therefore simplify systems engineering in software intense
+systems.
+
+### Design Principles & Ideas
+- [ ] requirements as YAML text: YAML better editable and readable for req. documentation as text than JSON
+  - transform internally to JSON to be able to use contents with PlantUML Preprocessing
+- [ ] traceability: in docs only one-directional trace from the lower requirement to the higher requirement (where it has been derived from)
+- [ ] generate bi-directional traceability in „pumla update“ step. create a reqrepo_json.puml file with all collected req from whole project
+    - [ ] have for each req „derived to“ and „derived from“ as keys. „derived to“ is created during the update step
+    - [ ] „derived from“ is documented as key in the YAML files
+- [ ] traceability to architecture artifacts. which arch element realizes a certain requirement.
+    - [ ] realizes is a part of the architecture model, not of the requirement.
+    - [ ] requirements are independent of their implementation
+    - [ ] a requirement belongs to a scope; within the scope it gets broken down
+- [ ] requirements can be structured in features.
+    - [ ] req that belong to a feature can have a loose or high coupling
+    - [ ] loose coupling will be realized with the „child injection“ mechanism
+    - [ ] high coupling by a feature table that directly names reqs that belong to the feature
+    - [ ] feature structure is kind of an architectural step with a composite breakdown view
+- [ ] automatic scope alignment if the req doc is next a single RUA arch spec? or manual scope connection? do we need the scope at all?
+- [ ] PUMLA macros to create diagrams that show all requirements fulfilled by a certain architecture element
+- [ ] PUMLA macros to create feature structure & breakdown views
+- [ ] PUMLA macros to create traceability views
+- [ ] status „aligned“, „new“, „decided“
+    - [ ] whether it is „realized/implemented“ will be decided with the respective test passing
+    - [ ] „rejected“ can be a commit message for deleting the req
+    - [ ] new: indicated a new unreviewed requirement which is there but not yet considered somewhere
+    - [ ] aligned: requirement is reviewed and aligned to be usable, but not yet decided how to be realized
+    - [ ] decided: there is a design decision done on to where and how the requirement shall be realized.
+

src/pumla/control/cmd_utils.py

@@ -254,26 +254,29 @@ def isInBlacklist(path, blacklist):
             retval = True
     return retval
 
-def findAllPUMLAFiles(path):
-    """" find all pumla files in given path """
-    pumlafiles = []
+def readBlacklist(path, pathfilename):
     blacklist = []
-
-    blacklistfilename = path + "/pumla_blacklist.txt"
-    #print(blacklistfilename)
-    if os.path.isfile(blacklistfilename):
-        #print("blacklist found\n")
-        file = open(blacklistfilename)
+    if os.path.isfile(pathfilename):
+        # print("readBlackList: blacklist found\n")
+        file = open(pathfilename)
         text = file.read()
-        #print(text)
         file.close()
         for li in text.splitlines():
             # comments in blacklist start with the hash,
             # they are ignored
-            if not li.startswith("#") and not li == "" :
+            if not li.startswith("#") and not li == "":
                 blacklist.append(path + li.strip("."))
-        #print(blacklist)
 
+    return blacklist
+
+def findAllPUMLAFiles(path):
+    """" find all pumla files in given path """
+    pumlafiles = []
+    blacklist = []
+
+    blacklistfilename = path + "/pumla_blacklist.txt"
+
+    blacklist = readBlacklist(path, blacklistfilename)
 
     # walk through the file and folder structure
     # and put all PUMLA files into a list

src/pumla/control/reqparse.py

@@ -0,0 +1,98 @@
+"""The pumla command line tool functions for requirements parsing."""
+
+import json
+import os
+import yaml
+from pathlib import Path
+from pumla.control.cmd_utils import isInBlacklist, readBlacklist
+
+
+def parsePUMLAReqFile(file):
+    """ parse the PUMLA Requirements file and read it
+        into a dictionary."""
+    reqsdict = yaml.safe_load(Path(file).read_text())
+    return reqsdict
+
+
+def findAllPUMLAReqFiles(path):
+    """" find all pumla files in given path """
+    pumlareqfiles = []
+    blacklist = []
+
+    blacklistfilename = path + "/pumla_blacklist.txt"
+
+    blacklist = readBlacklist(path, blacklistfilename)
+
+    # walk through the file and folder structure
+    # and put all PUMLA req files into a list
+    for dirpath, dirs, files in os.walk(path):
+        for filename in files:
+            if not isInBlacklist(dirpath, blacklist):
+                #print(dirpath)
+                fname = os.path.join(dirpath, filename)
+                # a PUMLA req. file must end with '.yml' (see Modelling Guideline)
+                if fname.endswith('.yaml'):
+                    with open(fname) as myfile:
+                        line = myfile.read()
+                        # a PUMLA file must have that first comment line (see Modelling Guideline)
+                        if line.startswith("#PUMLARR"):
+                            pumlareqfiles.append(fname)
+
+    #return the list of PUMLA req files found
+    return pumlareqfiles
+
+def updatePUMLAReqRepo(path, mrefilename):
+    """create, update/overwrite the PUMLA requirements repository json file
+        with current state of the source code repository"""
+    # traverse down the path and find all
+    # pumla req. files.
+    pumlareqfiles = findAllPUMLAReqFiles(path)
+
+    # parse each pumla file and create
+    # a PUMLA Elements, Connections and
+    # relations out of it, that
+    # get put into dict.
+    pumlareqslist = []
+
+    # sum up information from all files in common list
+    for f in pumlareqfiles:
+        reqs = parsePUMLAReqFile(f)
+        for r in reqs:
+            r.update({"derived to": "not yet derived"})
+            pumlareqslist.append(r)
+
+
+    # make it accessible from within PlantUML.
+    # $allreqs is the preprocessor variable that
+    # allows access by PlantUML pumla marcros.
+    pumlareqdict = {"reqsrepopath": path, "reqsrepofile": mrefilename, "reqs": pumlareqslist}
+    reqjsontxt = json.dumps(pumlareqdict)
+    jsontxt = "!$allreqs = " + reqjsontxt
+
+    pumltxt = "@startjson\n" + reqjsontxt + "\n@endjson\n"
+    pumltxtlines = pumltxt.split("},")
+
+    # split text to make the resulting file more readable;
+    # one element definition per line.
+    txt_lines = jsontxt.split("},")
+
+    # write the lines to the model element repo file
+    with open(mrefilename, "w") as fil:
+        for i in range(len(txt_lines) - 1):
+            fil.write(txt_lines[i] + "},\n")
+        fil.write(txt_lines[len(txt_lines) - 1] + "\n\n")
+    fil.close()
+
+    # write the lines to the model element repo file
+    with open("reqs_json_diagram.puml", "w") as fil:
+        for i in range(len(pumltxtlines) - 1):
+            fil.write(pumltxtlines[i] + "},\n")
+        fil.write(pumltxtlines[len(pumltxtlines) - 1] + "\n\n")
+    fil.close()
+
+    return True, mrefilename, pumlareqslist
+
+
+fn = "reqsrepo_json.puml"
+
+updatePUMLAReqRepo("../../../test/examples/WeatherStation/", fn)

src/pumla/main.py

@@ -23,6 +23,7 @@
 from pumla.control.cmd_utils import checkElsRelsConsForAliasExistence
 from pumla.control.cmd_utils import installPlantUMLJAR
 from pumla.control.cmd_utils import gendiagram
+from pumla.control.reqparse import updatePUMLAReqRepo
 
 parser = None
 parser_getjson = None
@@ -161,13 +162,21 @@ def cmdUpdate(args):
     """create/update/overwrite the pumla model repository"""
     identifyMe(parser)
     print("updating...")
-    (success, efn, a, b, c) = updatePUMLAMR(os.path.curdir, args.mrefilename)
-    if success:
+    (success_mr, efn, a, b, c) = updatePUMLAMR(os.path.curdir, args.mrefilename)
+    (success_rr, rfn, reqslist) = updatePUMLAReqRepo(os.path.curdir, args.rrefilename)
+
+    if success_mr:
         print("model repo file: " + efn)
         print("done.")
     else:
         print("failed.")
 
+    if success_rr:
+        print("reqs repo file: " + rfn)
+        print("done.")
+    else:
+        print("failed.")
+
 def cmdCheckAlias(args):
     """check whether a given alias name is already used in the
     current model repository"""
@@ -319,14 +328,21 @@ def main():
 
     parser_update = subparsers.add_parser(
         "update",
-        help="(re-)generate `modelrepo_json.puml` with updated info from `pumla` model elements found in repository.",
+        help="(re-)generate `modelrepo_json.puml` and `reqsrepo_json.puml` "
+             "with updated info from `pumla` model elements found in repository.",
     )
     parser_update.add_argument(
         "mrefilename",
         help="filename for model repo JSON file",
         nargs="?",
         default=os.path.curdir + "/modelrepo_json.puml",
     )
+    parser_update.add_argument(
+        "rrefilename",
+        help="filename for reqs repo JSON file",
+        nargs="?",
+        default=os.path.curdir + "/reqsrepo_json.puml",
+    )
     parser_update.set_defaults(func=cmdUpdate)
 
     parser_getjson = subparsers.add_parser(

test/examples/WeatherStation/features.yaml

@@ -0,0 +1,7 @@
+all features:
+  - HMI:
+    - display
+    - buttons
+  - Housing:
+    - Color
+  - Measurement

test/examples/WeatherStation/help

@@ -0,0 +1,51 @@
+!$allelems = {"modelrepopath": "/Users/mvoss/Desktop/git/github/pumla/test/examples/WeatherStation", "modelrepofile": "help", "elements": [{"name": "Temperature System", "alias": "tempSys", "type": "node", "stereotypes": ["block"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "-", "path": "./", "filename": "tempSys.puml", "taggedvalues": [{"tag": "Vendor", "values": ["C Ltd."]},
+ {"tag": "Arch Level", "values": ["0"]}]},
+ {"name": "MyNewClass1", "alias": "MyNewClass1", "type": "class", "stereotypes": ["Python"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "-", "path": "./", "filename": "reusableClass1.puml"},
+ {"name": "tSys1 professional", "alias": "tSys1", "type": "node", "stereotypes": ["instance", "block"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "tempSys", "path": "./", "filename": "tempSysInstances.puml"},
+ {"name": "tSys2", "alias": "tSys2", "type": "node", "stereotypes": ["instance", "block"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "tempSys", "path": "./", "filename": "tempSysInstances.puml"},
+ {"name": "Temperature Sensor B", "alias": "tempSensorB", "type": "component", "stereotypes": ["block", "external System"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "-", "path": "./tempSensorB/", "filename": "tempSensorB.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["B1", "B2"]},
+ {"tag": "Vendor", "values": ["B Inc."]},
+ {"tag": "Arch Level", "values": ["1"]}]},
+ {"name": "Public State", "alias": "publicState", "type": "state", "stereotypes": [], "ports": [], "typed_ifs": [], "kind": "dynamic", "parent": "tempSensorB", "instclassalias": "-", "path": "./tempSensorB/", "filename": "publicState.puml"},
+ {"name": "anotherClass", "alias": "anotherClass", "type": "class", "stereotypes": ["Python"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "-", "path": "./anotherClass/", "filename": "anotherClass.puml"},
+ {"name": "CWeather", "alias": "CWeather", "type": "class", "stereotypes": ["Python"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "-", "path": "./CWeather/", "filename": "CWeather.puml", "taggedvalues": [{"tag": "Vendor", "values": ["C Ltd."]}]},
+ {"name": "w3", "alias": "w3", "type": "class", "stereotypes": ["instance", "Python"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "CWeather", "path": "./CWeather/", "filename": "FurtherWeatherInstances.puml"},
+ {"name": "w4", "alias": "w4", "type": "class", "stereotypes": ["instance", "Python"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "CWeather", "path": "./CWeather/", "filename": "FurtherWeatherInstances.puml"},
+ {"name": "Weather Data Instance 1", "alias": "w1", "type": "class", "stereotypes": ["instance", "Python"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "CWeather", "path": "./CWeather/", "filename": "WeatherInstances.puml"},
+ {"name": "Weather Data Instance 2", "alias": "w2", "type": "class", "stereotypes": ["instance", "Python"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "CWeather", "path": "./CWeather/", "filename": "WeatherInstances.puml"},
+ {"name": "Wireless Unit", "alias": "wirelessUnit", "type": "rectangle", "stereotypes": ["block", "external System"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "-", "instclassalias": "-", "path": "./wirelessUnit/", "filename": "wirelessUnit.puml", "taggedvalues": [{"tag": "Vendor", "values": ["XY"]},
+ {"tag": "SecurityClass", "values": ["Alpha"]},
+ {"tag": "Frequency Range", "values": ["5 GHz"]},
+ {"tag": "Arch Level", "values": ["0"]}]},
+ {"name": "Temperature Sensor A", "alias": "tempSensorA", "type": "component", "stereotypes": ["block"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "tempSys", "instclassalias": "-", "path": "./tempSensorA/", "filename": "tempSensorA.puml", "taggedvalues": [{"tag": "Vendor", "values": ["A GmbH"]},
+ {"tag": "Arch Level", "values": ["1"]}]},
+ {"name": "**Internal Sequence**", "alias": "internalSequence", "type": "participant", "stereotypes": [], "ports": [], "typed_ifs": [], "kind": "dynamic", "parent": "tempSensorA", "instclassalias": "-", "path": "./tempSensorA/", "filename": "internalSequence.puml"},
+ {"name": "Temperature Sensor B (dC)", "alias": "tempSensorBdC", "type": "component", "stereotypes": ["block"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "tempSys", "instclassalias": "-", "path": "./tempSensorBdC/", "filename": "tempSensorBdC.puml", "taggedvalues": [{"tag": "Vendor", "values": ["C Ltd."]},
+ {"tag": "Arch Level", "values": ["1"]}]},
+ {"name": "displayTemp", "alias": "displayTemp", "type": "component", "stereotypes": ["block"], "ports": [{"name": "tempDc", "interfacetype": "tempinterface", "type": "inout", "alias": "porttempdc", "taggedvalues": [{"tag": "Vendor", "values": ["C Ltd."]},
+ {"tag": "Arch Level", "values": ["2", "3"]},
+ {"tag": "SysVariant", "values": ["B"]}]},
+ {"name": "tempdF", "interfacetype": "tempinterface", "type": "in", "alias": "porttempdF", "taggedvalues": []}], "typed_ifs": [{"name": "temp_dC_RUA", "interfacetype": "hello", "type": "inout", "alias": "temp_dC_displayTemp_RUA", "taggedvalues": []}], "kind": "static", "parent": "tempSys", "instclassalias": "-", "path": "./displayTemp/", "filename": "displayTemp.puml", "taggedvalues": [{"tag": "Vendor", "values": ["C Ltd."]},
+ {"tag": "Brightness", "values": ["300 Nits"]},
+ {"tag": "Arch Level", "values": ["1"]}]},
+ {"name": "Temp. Converter", "alias": "tempConverter", "type": "component", "stereotypes": ["block"], "ports": [], "typed_ifs": [], "kind": "static", "parent": "tempSys", "instclassalias": "-", "path": "./tempConv/", "filename": "tempConverter.puml", "taggedvalues": [{"tag": "Vendor", "values": ["C Ltd."]},
+ {"tag": "Arch Level", "values": ["1"]}]}]}
+
+!$allrelations = {"modelrelationrepopath": "/Users/mvoss/Desktop/git/github/pumla/test/examples/WeatherStation", "modelrelationrepofile": "help", "relations": [{"id": "REL#tSys1tempSys", "start": "tSys1", "end": "tempSys", "reltype": "..>", "reltxt": "instance of", "techntxt": "", "path": "./", "filename": "tempSysInstances.puml"},
+ {"id": "REL#tSys2tempSys", "start": "tSys2", "end": "tempSys", "reltype": "..>", "reltxt": "instance of", "techntxt": "", "path": "./", "filename": "tempSysInstances.puml"},
+ {"id": "REL#publicStateToIF", "start": "publicState", "end": "temp_dF_tempSensorB", "reltype": "..>", "reltxt": "provides", "techntxt": "", "path": "./tempSensorB/", "filename": "tempSensorB.puml"},
+ {"id": "REL#w3CWeather", "start": "w3", "end": "CWeather", "reltype": "..>", "reltxt": "instance of", "techntxt": "", "path": "./CWeather/", "filename": "FurtherWeatherInstances.puml"},
+ {"id": "REL#w4CWeather", "start": "w4", "end": "CWeather", "reltype": "..>", "reltxt": "instance of", "techntxt": "", "path": "./CWeather/", "filename": "FurtherWeatherInstances.puml"},
+ {"id": "REL#w1CWeather", "start": "w1", "end": "CWeather", "reltype": "..>", "reltxt": "instance of", "techntxt": "", "path": "./CWeather/", "filename": "WeatherInstances.puml"},
+ {"id": "REL#w2CWeather", "start": "w2", "end": "CWeather", "reltype": "..>", "reltxt": "instance of", "techntxt": "", "path": "./CWeather/", "filename": "WeatherInstances.puml"},
+ {"id": "tempSensorBdCRel#1", "start": "tempSensorBdC", "end": "tempSensorB", "reltype": "..>", "reltxt": "use", "techntxt": "", "path": "./tempSensorBdC/", "filename": "tempSensorBdC.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["B2"]}]},
+ {"id": "tempSensorBdCRel#2", "start": "tempSensorBdC", "end": "tempConverter", "reltype": "..>", "reltxt": "use", "techntxt": "", "path": "./tempSensorBdC/", "filename": "tempSensorBdC.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["B2"]}]},
+ {"id": "tempSensorBdCRel#3", "start": "tempSensorBdC", "end": "tempSensorB", "reltype": "..>", "reltxt": "useForB3", "techntxt": "", "path": "./tempSensorBdC/", "filename": "tempSensorBdC.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["B3"]}]},
+ {"id": "tempSensorBdCRel#4", "start": "tempSensorBdC", "end": "tempConverter", "reltype": "..>", "reltxt": "useForB3", "techntxt": "", "path": "./tempSensorBdC/", "filename": "tempSensorBdC.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["B3"]}]}]}
+
+!$allconnections = {"modelconnectionrepopath": "/Users/mvoss/Desktop/git/github/pumla/test/examples/WeatherStation", "modelconnectionrepofile": "help", "connections": [{"id": "CON#_SYS_VAR_B", "start": "temp_dC_tempSensorBdC", "end": "temp_dC_displayTemp", "contype": "--", "contxt": "", "path": "./connections/", "filename": "connections_tempSys_Var_B.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["B1", "B2"]}]},
+ {"id": "CON#_SYS_VAR_A", "start": "temp_dC_tempSensorA", "end": "temp_dC_displayTemp", "contype": "--", "contxt": "", "path": "./connections/", "filename": "connections_tempSys_Var_A.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["A"]}]},
+ {"id": "CON#_SYS_VAR_B2", "start": "temp_dC_tempSensorBdC", "end": "temp_dC_displayTemp", "contype": "--", "contxt": "", "path": "./connections/", "filename": "connections_tempSys_Var_B2.puml", "taggedvalues": [{"tag": "SysVariant", "values": ["B2"]}]},
+ {"id": "C#2", "start": "temp_dC_tempConverter", "end": "temp_dC_tempSensorBdC", "contype": "--", "contxt": "", "path": "./tempSensorBdC/", "filename": "tempSensorBdC.puml"},
+ {"id": "C#3", "start": "temp_dF_tempSensorB", "end": "temp_dF_tempConverter", "contype": "--", "contxt": "", "path": "./tempSensorBdC/", "filename": "tempSensorBdC.puml"}]}
+