Add experimental Dataservice

IoT Agent was so far operating independent of datamodel. This had seveal disadvantages: - No detection of mismatch between namespaces - No systematic normalization/contextualization of input data with respect to data model - No trust building/plausiblity checks of incoming data - No use of knowledge graphs to pre-process data at gateway This PR provides a first draft of the future Dataservice which uses the ontology not only for data modelling but also to determine contextualization of incoming data. It only contains a test binding which is creating random data to demonstrate the concept. In future it will contain modules to retrieve data from other protocols, mainly proxied by MQTT and REST. As first step the JSON-LD "@context" is becoming a central structure. Every ontology must contain a context which determines the prefixes and namespace binding. For instance, it is assumed that every ontology must contain a "base:" prefix for the namespace which contains all the base terms of the used ontology (such as Binding and Connector classes) An example ontology is provided here: https://industryfusion.github.io/contexts/staging/example/v0.1 including context: https://industryfusion.github.io/contexts/staging/example/v0.1/context.jsonld Related Epic: #514 Related User Story: #515 Signed-off-by: marcel <[email protected]>
IndustryFusion · Apr 3, 2024 · bfbbb6c · bfbbb6c
1 parent 58e845a
commit bfbbb6c
Show file tree

Hide file tree

Showing 68 changed files with 4,970 additions and 441 deletions.
diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml
@@ -18,6 +18,9 @@ jobs:
       run: |
         sudo  apt install sqlite3 sqlite3-pcre
         cd semantic-model/shacl2flink && make setup && make lint test build test-kms
+    - name: Build Dataservice
+      run: |
+        cd semantic-model/dataservice && make setup && make lint 
     - name: Build datamodel tools
       run: |
         cd semantic-model/datamodel/tools && make setup && make lint test
diff --git a/helm/common.yaml b/helm/common.yaml
@@ -16,5 +16,5 @@ db:
   oispdbUser: "oisp"
 
 ontology:
-  baseUri: https://industryfusion.github.io/contexts/staging/ontology/v0.1/
+  baseUri: https://industryfusion.github.io/contexts/staging/example/v0.1/
 
diff --git a/semantic-model/datamodel/Tutorial.md b/semantic-model/datamodel/Tutorial.md
@@ -6,7 +6,7 @@ A fundamental difference between JSON-LD and JSON is, that JSON describe generic
 ```json
 [
     {
-        "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+        "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
         "id": "urn:iff:cutter1",
         "type": "eclass:0173-1#01-AKJ975#017",
         "hasFilter": {
@@ -19,7 +19,7 @@ A fundamental difference between JSON-LD and JSON is, that JSON describe generic
         }
     },
     {
-        "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+        "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
         "id": "urn:iff:filter1",
         "type": "eclass:0173-1#01-ACK991#016",
         "machine_state": {
@@ -65,7 +65,7 @@ The same graph can, however, be represented by different JSON-LD structures. For
 ```json
 [
     {
-        "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+        "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
         "id": "urn:iff:cutter1",
         "type": "eclass:0173-1#01-AKJ975#017",
         "hasFilter": {
@@ -139,7 +139,7 @@ First exercise is to transform JSON-LD into RDF/turtle data:
 ```json
 # JSON-LD Data:
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:cutter1",
     "type": "eclass:0173-1#01-AKJ975#017"
 }
@@ -154,10 +154,10 @@ rdfpipe  -i json-ld -o ttl ../examples/simple_plasmacutter_data.json
 <urn:iff:abc123> a <https://industry-fusion.org/eclass#0173-1#01-AKJ975#017> .
 ```
 
-The result is showing a single triple line. `a` is an abbreviation for `rdf:type`. I.e. the triple defines an object with `id` equal to `<urn:iff:abc123>` and type equal to `https://industry-fusion.org/eclass#0173-1#01-AKJ975#017`. But why was `eclass:0173-1#01-AKJ975#017` translated to `https://industry-fusion.org/eclass#0173-1#01-AKJ975#017`? To answer the question, we take a closer look to the `@context` field. When we resolve the url `https://industryfusion.github.io/contexts/v0.1/context.jsonld`, we get the following result:
+The result is showing a single triple line. `a` is an abbreviation for `rdf:type`. I.e. the triple defines an object with `id` equal to `<urn:iff:abc123>` and type equal to `https://industry-fusion.org/eclass#0173-1#01-AKJ975#017`. But why was `eclass:0173-1#01-AKJ975#017` translated to `https://industry-fusion.org/eclass#0173-1#01-AKJ975#017`? To answer the question, we take a closer look to the `@context` field. When we resolve the url `https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld`, we get the following result:
 
 ```
-wget -O - https://industryfusion.github.io/contexts/v0.1/context.jsonld 2> /dev/null
+wget -O - https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld 2> /dev/null
 ```
 ```json
 # Result:
@@ -239,13 +239,13 @@ You can see that as a result, the @context field is missing and the `eclass:` pr
 
 The inverse operation of *expansion* is a *compaction*. One problem of the *compaction* is, that the *expanded Form* does no longer contain a @context. So the @context has to be added back explicitly. With a fixed @context, the *compaction* is uniquely defined, i.e. there is no other *compacted form*, given an *expanded form* and a @context.
 ```
-node ./jsonldConverter.js -n ../examples/simple_plasmacutter_data_expanded.json -c https://industryfusion.github.io/contexts/v0.1/context.jsonld
+node ./jsonldConverter.js -n ../examples/simple_plasmacutter_data_expanded.json -c https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld
 ```
 ```json
 # Result
 [
   {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:cutter1",
     "type": "eclass:0173-1#01-AKJ975#017"
   }
@@ -273,7 +273,7 @@ An example for the *concise* form is shown below:
 
 ```json
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:cutter1",
     "type": "eclass:0173-1#01-AKJ975#017",
     "hasFilter": {
@@ -292,7 +292,7 @@ node ./jsonldConverter.js -r ../examples/plasmacutter_data.json
 # Result:
 [
   {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:cutter1",
     "type": "eclass:0173-1#01-AKJ975#017",
     "hasFilter": {
@@ -316,7 +316,7 @@ node ./jsonldConverter.js -n ../examples/plasmacutter_data_normalized.json
 # Result
 [
   {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:cutter1",
     "type": "eclass:0173-1#01-AKJ975#017",
     "hasFilter": {
@@ -457,7 +457,7 @@ Basic SHACL shapes can be created from JSON-Schema by using the `jsonschema2shac
 For instance, the JSON-Schema for the plasmacutter above, can be converted by providing the schema to convert by option `-s`, the id of the single object by option '-i' and a context by option '-c':
 
 ```
-node ./jsonschema2shacl.js -s ../examples/plasmacutter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/v0.1/context.jsonld
+node ./jsonschema2shacl.js -s ../examples/plasmacutter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld
 ```
 ```
 # Result:
@@ -530,11 +530,11 @@ Therefore, the validation steps are as follows:
 
     For instance, the SHACL constraints for the plasmacutter and filter above can be generated by:
     ```
-    node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/v0.1/context.jsonld
+    node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld
     ```
     and
     ```
-    node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-ACK991#016 -c https://industryfusion.github.io/contexts/v0.1/context.jsonld
+    node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-ACK991#016 -c https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld
     ```
     respectively.
 4. Validate single objects with JSON-Schema and linked objects with SHACL
@@ -551,7 +551,7 @@ Therefore, the validation steps are as follows:
     Validation of single object with SHACL is possible but will always lead to an error when connected data is invovled. Let's look at the follwing example. Let's first create the SHACL file for the plasmacutter from the respective JSON-Schema and write it into `/tmp/shacl.ttl`:
 
     ```
-    node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/v0.1/context.jsonld >/tmp/shacl.ttl
+    node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld >/tmp/shacl.ttl
     ```
     Then convert the *concise* plasmacutter object into a *normalized* form and dump it into `/tmp/plasmacutter_normalized.json`
     ```
@@ -592,7 +592,7 @@ node ./jsonldConverter.js -r ../examples/plasmacutter_and_filter_data.json  > /t
 ```
 Then validate it with pyshacl and the plasmacutter only shema:
 ```
-node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/v0.1/context.jsonld >/tmp/shacl.ttl
+node ./jsonschema2shacl.js -s ../examples/plasmacutter_and_filter_schema.json -i  https://industry-fusion.org/eclass#0173-1#01-AKJ975#017 -c https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld >/tmp/shacl.ttl
 ```
 Finally, use `pyshacl` to validate:
 ```

diff --git a/semantic-model/datamodel/examples/filter_data.json b/semantic-model/datamodel/examples/filter_data.json
@@ -1,5 +1,5 @@
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:filter1",
     "type": "eeclass:0173-1#01-ACK991#016",
 

diff --git a/semantic-model/datamodel/examples/plasmacutter_and_filter_data.json b/semantic-model/datamodel/examples/plasmacutter_and_filter_data.json
@@ -1,6 +1,6 @@
 [
     {
-        "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+        "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
         "id": "urn:iff:cutter1",
         "type": "eclass:0173-1#01-AKJ975#017",
         "hasFilter": {
@@ -13,7 +13,7 @@
         }
     },
     {
-        "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+        "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
         "id": "urn:iff:filter1",
         "type": "eclass:0173-1#01-ACK991#016",
         "machine_state": {

diff --git a/semantic-model/datamodel/examples/plasmacutter_and_filter_data2.json b/semantic-model/datamodel/examples/plasmacutter_and_filter_data2.json
@@ -1,6 +1,6 @@
 [
     {
-        "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+        "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
         "id": "urn:iff:cutter1",
         "type": "eclass:0173-1#01-AKJ975#017",
         "hasFilter": {

diff --git a/semantic-model/datamodel/examples/plasmacutter_data.json b/semantic-model/datamodel/examples/plasmacutter_data.json
@@ -1,5 +1,5 @@
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:abc123",
     "type": "eclass:0173-1#01-AKJ975#017",
     "hasFilter": {

diff --git a/semantic-model/datamodel/examples/plasmacutter_data_normalized.json b/semantic-model/datamodel/examples/plasmacutter_data_normalized.json
@@ -1,6 +1,6 @@
 [
   {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:abc123",
     "type": "eclass:0173-1#01-AKJ975#017",
     "hasFilter": {

diff --git a/semantic-model/datamodel/examples/plasmacutter_datatype_unit_data.json b/semantic-model/datamodel/examples/plasmacutter_datatype_unit_data.json
@@ -1,5 +1,5 @@
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:abc123",
     "type": "eclass:0173-1#01-AKJ975#017",
     "hasFilter": {

diff --git a/semantic-model/datamodel/examples/plasmacutter_shacl.ttl b/semantic-model/datamodel/examples/plasmacutter_shacl.ttl
@@ -1,4 +1,4 @@
-@prefix iffb: <https://industry-fusion.org/base/v0.1/>.
+@prefix iffb: <https://industry-fusion.org/base/v0/>.
 @prefix sh: <http://www.w3.org/ns/shacl#>.
 @prefix ngsi-ld: <https://uri.etsi.org/ngsi-ld/>.
 

diff --git a/semantic-model/datamodel/examples/simple_plasmacutter2_data.json b/semantic-model/datamodel/examples/simple_plasmacutter2_data.json
@@ -1,5 +1,5 @@
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:abc123",
     "type": "Plasmacutter"
 }
diff --git a/semantic-model/datamodel/examples/simple_plasmacutter_data.json b/semantic-model/datamodel/examples/simple_plasmacutter_data.json
@@ -1,5 +1,5 @@
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:cutter1",
     "type": "eclass:0173-1#01-AKJ975#017"
 }
diff --git a/semantic-model/datamodel/examples/simple_plasmacutter_data2.json b/semantic-model/datamodel/examples/simple_plasmacutter_data2.json
@@ -1,5 +1,5 @@
 {
-    "@context": "https://industryfusion.github.io/contexts/v0.1/context.jsonld",
+    "@context": "https://industryfusion.github.io/contexts/tutorial/v0.1/context.jsonld",
     "id": "urn:iff:cutter1",
     "type": "Plasmacutter"
 }
diff --git a/semantic-model/dataservice/.flake8 b/semantic-model/dataservice/.flake8
@@ -0,0 +1,3 @@
+[flake8]
+max-line-length = 120
+ignore = E722, W504
diff --git a/semantic-model/dataservice/Makefile b/semantic-model/dataservice/Makefile
@@ -0,0 +1,15 @@
+PYTHON := python3
+LINTER := $(HOME)/.local/bin/flake8
+PIP := pip
+HELM_DIR := ../../helm/charts/shacl
+NAMESPACE := iff
+
+
+lint: requirements-dev.txt
+	$(LINTER)
+
+setup: requirements.txt setup-dev
+	$(PIP) install -r requirements.txt
+
+setup-dev: requirements-dev.txt
+	$(PIP) install -r requirements-dev.txt
diff --git a/semantic-model/dataservice/README.md b/semantic-model/dataservice/README.md
@@ -0,0 +1,13 @@
+# Dataservice setup
+
+## Setup and Activate Device
+
+## Use Dataservice to send testdata
+Start dataservice with `startDataservice.py`:
+    python3 ./startDataservice.py \<ontology-dir\> \<type\> \<binding-name\>
+
+*\<ontology\>* is supposed to be downloadable from a directory containing different *.ttl files, *\<type\>* is the (in the ontology context) namespaced class (e.g. `ex:MyClass` if `ex:` is defined in the ontologies context) and *\<binding-name\>* is the name of a *.ttl file in the *bindings* subdirectory of the ontoloy.
+
+Example:
+
+    python3 ./startDataservice.py  https://industryfusion.github.io/contexts/example/v0.1  iffBaseEntity:Cutter  base_test.ttl
diff --git a/semantic-model/dataservice/requirements-dev.txt b/semantic-model/dataservice/requirements-dev.txt
@@ -0,0 +1,5 @@
+flake8==5.0.4
+bandit==1.7.4
+black==22.8.0
+pytest==7.1.3
+pytest-cov==4.0.0
diff --git a/semantic-model/dataservice/requirements.txt b/semantic-model/dataservice/requirements.txt
@@ -0,0 +1,4 @@
+rdflib==6.2.0
+owlrl==6.0.2
+ruamel.yaml==0.17.21
+setuptools>=65.5.1 # not directly required, pinned by Snyk to avoid a vulnerability
diff --git a/semantic-model/dataservice/services/testConnector.py b/semantic-model/dataservice/services/testConnector.py
@@ -0,0 +1,40 @@
+#
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import asyncio
+import random
+from random import randint
+from rdflib import Literal
+from rdflib.namespace import XSD
+
+
+##########################################################################################
+# This function will receive an array of dictionaries containing the needed parameters
+# to read out data from machines or databases.
+# It will update the values in regular intervals
+##########################################################################################
+async def subscribe(connector_attribute_dict, firmware):
+    while True:
+        logic_var_type = connector_attribute_dict['logicVarType']
+        if logic_var_type == XSD.boolean:
+            connector_attribute_dict['value'] = Literal(random.choice([True, False]))
+        elif logic_var_type == XSD.integer:
+            connector_attribute_dict['value'] = Literal(randint(0, 1000))
+        elif logic_var_type == XSD.decimal or logic_var_type == XSD.float or logic_var_type == XSD.double:
+            connector_attribute_dict['value'] = Literal(float(randint(0, 100000)) / 100.0)
+        connector_attribute_dict['firmware'] = firmware
+
+        await asyncio.sleep(1)