Automate Creation of EA Artifacts from GC policy sources

[samperd]

User Story

As a GC EA Architect
I want to generate EA artifacts automatically from authoritative sources
So that EA components can be automatically generated, and updated instead of each EA Architect needing to re-model these components

EA Artifact Sources

• Web Sources

  • Parliamentary Bills: All Bills for the Current Session (42nd Parliament, 1st Session)
  • Laws and Legislation: eg Security of Information Act
  • Regulations: eg Grosse Isle, P.Q., Prohibited Place Order
  • Codes: Values and Ethics Code for the Public Sector
  • Policy Frameworks: eg Policy Framework for Information and Technology
  • Policies: Eg Policy on Management of Information Technology
  • Directives: eg Directive on Management of Information Technology
  • standards: eg Standard on Web Interoperability
  • Guidelines: Eg Guideline on Service Management
  • Tools: Related policies, directives, standards, guidelines, etc.

• Open Data Sources:

  • Departments:
    • List of legal department names and their respective numbers
  • Legislation:
    • List of Acts and Regulations
    • Consolidated Acts and regulations in XML
    • Consolidated federal Acts and regulations - Bulk XML file
  • policies:
    • All policies are linked together by way of an HTML encoded policy hierarchy

Background

*It should be possible to generate a new baseline GC EA architecture at the business and motivation level on a daily basis. This is one proposal how

• Each law has a HTML representation (eg https://laws-lois.justice.gc.ca/eng/acts/C-15.31/)
• For each law with an HTML representation there is a machine readable XML version (eg. https://laws-lois.justice.gc.ca/eng/XML/C-15.31.xml)
• XSLT is a transformation language to convert XML into various others forms including other XML formats. (https://www.w3schools.com/xml/xsl_intro.asp)
• Archimate is an openly documented modeling language specification
• The Archimate Model Exchange File Format is an XML representation of an archimate model
• software compliant to the Archimate exchange format can import and export Archimate exchange files
• Archi is an example of such an open source compliant tool
• Archi has a collaboration extension that allows architects to use Git as a version controlled EA repository Repository to manage EA elements
• canada-ca/architecture repository leverages GitHub as an example of a public implementation of Git
• GCCode Internal Link is an internal Git repository that uses [GitLab Community Edition]( as the interface
• The Archimate collaboration plugin converts each EA object into a mini Archimate Model Exchange file format. For example this is what a Requirement Element in the canada-ca/architecture model looks like:

<archimate:Requirement
    xmlns:archimate="http://www.archimatetool.com/archimate"
    name="C.2.3.3.2"
    id="02ff40fd-da1f-4970-955a-4937850196a8"
documentation="Make an audit trail available for all transactions to ensure accountability and non repudiation"/>

• converting various EA elements from their native XML representation (eg policies and laws) to the archimate exchange format can aid with easily updating EA elements
• These processes, if automated can run regularly (eg daily)

Therefore, it should be possible to generate a new baseline GC EA architecture at the business and motivation level on a daily basis.

Initial Implementation Details

• Start with Values and Ethics Code for the Public Sector and then work down each level of the hierarchy
• Test with a small sample of elements
• Use Python as a human friendly scripting language that has a lower barrier to entry to handle the management of the creation of these elements
• Develop an XSLT transformation for the EA elements
• Run the transfomation
• Create separate elements that can be added to the canada-ca/architecture repository
• Create a single Archimate exchange file containing ALL the elements for easy import outside of the canada-ca/architecture repository

Future upgrades

• add relationships to linked elements within a "policy"
  • "Related Laws"
  • "Supporting Tools"
  • "More information"
  • "Hierarchy"
• break large elements (eg a policy) into smaller components (eg sections and sub-sections)
• provide XSLT transformations to the managers of the web locations and assets so that canada.ca can provide native XML as well as Archimate compliant XML representations of all of GC EA elements

Future work

• Create a plugin for Archi that can Create, Read, Update, Delete elements in a model based on various "Policy" sources (as listed above)
  • [ ]scan internet sources
  • search for changes,
  • read sources,
  • transform from native formats to archimate,
  • import one or more elements into the model,
  • update the model,

[SteveSavage]

Hi Dave, FYI, John and I took a look at auto converting the XML versions in to Archimate elements.
Unfortunately they use XML to denote the structure of the document, not to describe the content.
Because I have to read the document to actually create meaningful elements anyway, I use a feature in Sparx EA that lets me import the entire document as a document, and then select text in the document to create elements. The document is updated with hyperlinks to the newly created elements. I think this is a Sparx EA specific feature, but greatly simplifies working with these documents.

I can open the document, read it, select a hyperlinked statement, which loads the linked element (typically a requirement), then I can use Sparx EA traceability window to see all the elements I created that "realize" that requirement.

Once I have everything in I have a script in Sparx EA that loops through all the created elements and updates the notes to provide full context for the statement.

E.g. for the requirement element with the name:
4.4.1.2. Adopting, as applicable, enterprise solutions within their respective departments.

The full context to make it readable (in the notes) is:
Directive on the integrated management of service, information, information technology, and cyber security
4. Requirements
4.4 Leveraging technology in the digital era
4.4.1 Departmental CIO is responsible for:
4.4.1.2. Adopting, as applicable, enterprise solutions within their respective departments.

I could query the DB to get this hierarchy dynamically, but to simplify my reporting templates I add this static information to the notes.

[samperd]

I noticed that there was a fair bit of formatting in there too. I am not yet convinced the XML to Archimate approach is not possible, or could not someday be possible, however I think it will require some collaboration across a few different groups.

I will maintain this pipe dream of using open sources and open data to auto generate an EA model, however I get the Sparx is serving you well.

I hope that some day we have an EA ecosystem that is actually compliant to some of our policy, for example I hope to see Archi become the tools of choice and Archimate model exchange become the default excahnge format and git become the repository of choice. Other options present large barriers to entry, however I understand this is our current siloed world.

I encourage you to share all of your hard work of dissecting documents with the canada-ca/architecture repository so others can re-use these components.

Perhaps we can move closer towards automation as time progresses. I think the perfect use case are all those policies you mentioned that will change. So I assume this means work on your part to update even your scripts.

Cheers

[SteveSavage]

I think an automatic conversion will be possible in the future, if they add contextual information to the policy XML. The best way would be to create the model first to capture everything, then create the document based on the model vs. the other way around.... I've said the same thing for regulations.

Will definitely continue to share my work. Both through my blog and through whatever official repositories the group decides. The Archimate Exchange Format seems to be our best option at the moment for sharing information, but this doens't let us share not Archimate models. So I also plan to share Sparx EA XMI files and generate CSV files: three, one for elements, one for connections, one for diagrams. We may have to use our collection of tools to publish the information in a set of native formats to optimize reuse. E.g. Sparx, Qualiware, etc.

For Sparx, personally I wish we had an enterprise version of it's Reusable Assess Service that I could just link to. https://sparxsystems.com/enterprise_architect_user_guide/14.0/model_repository/reuseable_asset_service.html

For Change management, I don't update. I set the status of the old policy instrument to "deprecated then replaced/retired" and note what it has been replaced by if replaced.

That way we can keep track of anything linked to the old policies that will need to be reviewed for changes. This will also let me explicitly cross link new and old policies to show and comment on specific changes. I plan to write some blog posts about all of this as I have time.

[SteveSavage]

I've been thinking about this some more, and I think I can use Sparx's API to automate this work a bit more.

Pass 1: Import

1. • API connects to page containing list of Policy Instruments (https://www.tbs-sct.gc.ca/pol/hierarch-eng.aspx)
2. • API downloads the XML version for each Policy instrument
3. • API saves each XML locally
4. • API for each XML, parses the XML
5. • API creates a package for the XML named by the Policy Instrument, with a ID of GCPI-{documentID from XML), auto generates and Alias based on the caps (this will need manual tweaking I'm sure)
6. • API creates sub-packages based on the chapter titles and appendix titles
7. • API creates sub-packages based on the headings within chapters
8. • API creates Archimate Requirement elements based on paragraph nodes, these "paragraph" requirements are created within the appropriate chapter, appending or heading package
9. • API creates Archimate Requirement elements based on some list nodes (those that are preceded by a paragraph, this filters out lists of links etc.), these requirements elements are created as children of the preceding "paragraph" requirement.
10. • API creates Archimate Meaning elements based on dt and dl notes

Pass 2 - Enumerate
The API attempts to assign unique numbers to all requirements based on the heiarchy, e.g.
6.1.2 Authorized individuals are informed of the following:

• Expectations for acceptable use of Government of Canada electronic networks and devices per Appendices B and C;

Becomes
6.1.2 Authorized individuals are informed of the following:
6.1.2.1 Expectations for acceptable use of Government of Canada electronic networks and devices per Appendices B and C;

After Phase 1 and 2 are complete for all Policy instruments.

Pass 3: Identify Actors / Roles

1. The API looks for the phrase "are responsible for"
2. The API generates a master set of elements for of potential Roles/Actors, default as Role.
3. A human reviews the list and the elements and converts elements to Actors if needed, or deletes
4. The API analyzing the requirements from each Policy Instrument, if the Policy Instrument has a requirement that references the Role/Actor followed by the phrase "are responsible for", a copy of the Role/Actor element is made in the same package as the policy instrument. The local copy of the Role/Actor element "realizes" the requirement. The local copy of the Role/Actor element is "generalized" by the master copy of the element.

I'll probably be able to do something similar for Business Objects, and perhaps Business Functions etc. But those might have to be manually created "master" Elements.

Dave, I know you and some others are Phython wizards, perhaps once I get the initial structure in place you can use some of it's AI type features to do some more analysis?

[orthosys]

Hi.
We have developed a python-based engine that does exactly that : extracting data from multiple sources and integrating the model update into the EA repository

[SteveSavage]

I've pretty much parked this project due to a career change and language training, I'd love to see an example of what you've managed to extract if you can make it available as an XMI or Archimate Exchange file.