Support for better type/model definition language for Sphinx-Needs

[twodrops]

Background

Sphinx-Needs is a tool which is currently used in our organization to handle safety critical artifacts following the docs-as-code methodology by 800+ developers. Firstly, kudos to the whole sphinx-needs team for this great tool 👍🏽

As Sphinx-Needs is getting used more and more at enterprises, some of its conceptual limitations in creating model/type definitions needs to addressed. I am starting a discussion so that we can try to arrive at a solution together. Perhaps as part of a Sphinx-Needs 2.0.

Currrent main short-comings in model/type definition

1. Sphinx-Needs currently uses a partially typed meta-model to create definitions
2. Sphinx-Needs uses conf.py as the model definition language

Partially typed meta-model

The three main definition elements in Sphinx-Needs (needs_types, needs_extra_options, needs_extra_links) are not related to each other.

For instance, the needs_extra_options are "global" and do not belong to any needs_types. This simplicty in sphinx-needs however results in several issues, mainly:

• Weaker model definitions, the consistency of which can only be checked in the lower meta-levels. For example, if only a "req" need shall have an option called "asil", this can only be checked with a needs_warning at runtime. This means sphinx-needs currently allows to creates need elements which does not conform to a schema and it can only be validated later. This is inefficient as a schema check can be done once (even statically before a build) and not "n" times at runtime for each instance as of now.
• A stricter model is needed not only within Sphinx-Needs but also for tools interfacing with it like VsCode Editor. As the information is not available within the model-definition such tools does not have access to this information. For example, a intellisense or a linter which checks which options are allowed for a need type.

A stricter meta-model could potentially look like this:

I was expecting that this would be different in the upcoming open-needs meta-model, but that does not seem to be the case. Therefore we need to check if we can arrive at a sphinx-needs solution.

GitHub Issue: #332

conf.py as model definition language

Sphinx-Needs currently uses conf.py for model/type definition. This information is needed not only for sphinx needs but for other tools/usecases as well.

• Visual Studio Code snippets to offer templates in VS Code editor
• Visualization of metamodel definition (for example in PlantUML) for reviewing meta-model, especially with focus on traceability
• Documentation of needs for users to know what each need type with its options and link means
• Editor IntelliSense and Linter configurations

Information in conf.py cannot be easily reused by other tools. A neutral master definition language has the advantage of being available for other tools and also allows to generate any of the other needed artifacts above (even a conf.py).

Similar discussion here: #287
(The discussion focuses on reusing rST for model definitions with ConfAsDocs, but I find it not the correct metameta language for this usecase).

[StephanKuempel]

Using jsonschema for defining needs types (like already mentioned in #332) should solve most of the problems with the currrent needs_types, needs_extra_options, needs_extra_links based definition.
e.g.

Schemas:

{
    "$id": "file:///e:/schemas/sw_component_schema.json",
  
    "type": "object",
    "properties": {
      "title": {"type": "string"},
      "description": {"type": "string"},
      "status": {"type": "string","enum": ["finished","in work","backlog"]},
      "implements": {"$ref": "requirement_schema.json"}
    },
    "required": ["title", "status", "implements"]
}

{
    "$id": "file:///e:/schemas/req_sw_schema.json",
    "type": "object",
    "properties": {
        "title" :{"type": "string"},
        "status": {"type": "string","enum": ["draft","submitted","negotiate","agreed", "obsolete","rejected", "n/a"]}
    },
    "required": ["title", "status"]
}

Instances:

sw_component_1.json

{
    "title": "Example SW Component",
    "implements": {"$ref": "req_sw_1.json"}
}

req_sw_1.json

{
    "title": "Example Req",
    "status": "draft"
}

[twodrops]

@StephanKuempel Thanks for the first good example with json schema. It looks like a good candidate to specify the structure than inventing something on own own.

To keep the example simple, probably we could use a single json schema/ json instance to start with. Splitting the contents into multiple JSONs and having cross references could be perhaps the a second step.

I did not fully understand the $ref usage above. In our case we need a $ref between types/elements right?

[twodrops]

A json schema source for inspiration;. https://raw.githubusercontent.com/frontaid/schema/master/frontaid-schema.json with $ref element usage.

[Rubyfi]

I tried to recreate your example in VSCode, but the reference in sw_component_1.json isn't resolved.
Instead IntelliSense marks the entry as invalid, since it does not contain the "title" and "status" entries.
I have attached my minimal example for reference.

To my knowledge, jsonschema does not support references to other objects out-of-the-box, but only references to other schemas, which are used to validate the children of an entry.
json_schema_test.zip

[StephanKuempel]

@twodrops you need the $ref in both, schema and instances. you can also use a $ref to refer a local element in the same file.
@Rubyfi I used jsonref.loads with specifiying a base_uri instead of json.loads
with json.loads you have to specify the full path in the $refs

[twodrops]

@StephanKuempel Additionally, if we use jsonschema as the definition language, it shall not be to validate needs.jsons only, but to satisfy the usecases mentioned above as well. That is,

• To offer IntelliSense, Templates in Visual Studio Code
• Linter configuration
• To generate sphinx-needs conf.py, including need_warnings (based on model contraints) automatically.
• Visualization of metamodel definition (for example in PlantUML) for reviewing meta-model, especially with focus on traceability
• Documentation of needs for users to know what each need type with its options and link means