Break out current write a connector documentation, improve documentat…

…ion, etc. (#4)

* wip: moved general write a connector documentation out of cobie specific help into its own markdown

* fixed to run on windows and added notes to readme

* wip: Removed COBie xls and provide curr. dl instr.

* removed inadvertent file

* wip: fixed some bad links

* wip: corrected markdown problems

* corrected link from "tree" to "blob"

* wip: fixed remaining broken links

COBie-connector/README.md

@@ -16,17 +16,23 @@ In the second phase, the COBie Connector reads the intermediary SQLite database
 
 ## How to Run It
 
-You may find the sample COBie Excel data under COBie-extractor/extractor/input/*. [Sample Data Source](https://www.nibs.org/page/bsa_commonbimfiles)
+COBie stands for "Construction-Operations Building Information Exchange". For background on COBie, please checkout [COBie](https://www.wbdg.org/bim/cobie).  Unfortunately, this site has a bad link to the COBie Excel data used for the extraction step for this example.  To run this the extractor, you must download the COBie example file(s) for the "Duplex Apartment" model into the COBie-extractor/extractor/input/* folder. Two different states of the Duplex Design Data are available for download: [Design](https://portal.nibs.org/files/wl/?id=QG4uSz2tnqDpgx661pXF329h0KMgnG23) and [Handover](https://portal.nibs.org/files/wl/?id=oy5MyBRPiLx7ZmAomBRMgL62o1hi3YLk) 
 
-1. Execute COBie-extractor (see see how to inside COBie-extractor/README.md) 
+The makefile is hardcoded to extract four versions of the file: COBieSampleSheetV1.xlsx, COBieSampleSheetV2.xlsx, COBieSampleSheetV3.xlsx, COBieSampleSheetV4.xlsx.  You'll need to make sure the names of the files match the makefile by either editing the makefile or renaming the spreadsheets (easier of two choices).  You'll want to make incremental changes between V1 and V2, V2 and V3 and so on.  Copy V1 to V2 and change the contents of V2 to emulate a change that may happen between states of a project, then copy V2 to V3 and make further changes.
+
+
+1. Execute COBie-extractor (see see how to inside COBie-extractor/README.md)
 2. Move the output of COBie extractor (intermediary SQLite DB's) to COBie-connector/test/assets/ (or execute "sh transferdb" if you are on Linux / WSL)
 3. Run "npm run test:unit" (output iModel will be COBie-connector/test/output/final.db) This produces the iModel snapshot that can be used locally.
-4. To run the connector against a live iModel, run "npm run build" and "npm run test:integration" that will test if the Connector updates iModel data and schema. 
+
+   ![npm run test:unit](./CobieConnectorPassed.png)
+
+4. To run the connector against a live iModel, run "npm run build" and "npm run test:integration" that will test if the Connector updates iModel data and schema.
    Note: You must set the environment variable imjs_config_dir = path/to/imodeljs-config (or the path to the directory that contains your default.json credential file) to successfully run the integration test.
 
 ## Architecture
 
-Mapping of data into requires The COBie connector uses the following parts to map data from the intermediary database into an iModel. 
+Mapping of data into requires The COBie connector uses the following parts to map data from the intermediary database into an iModel.
 
 ### DataAligner (Reusable Parser)
 
@@ -35,7 +41,7 @@ It immediately creates the entity it encounters.
 
 ### DataFetcher
 
-DataFetcher dynamically joins tables and return values from the intermediary SQLite database. 
+DataFetcher dynamically joins tables and return values from the intermediary SQLite database.
 
 ### EC Schema Generation
 
@@ -47,15 +53,15 @@ Notes:
 
 ### Configuration Files
 
-**ElementTree.ts**: a dictionary that dictates the order in which partitions/models/elements should be created/updated. 
+**ElementTree.ts**: a dictionary that dictates the order in which partitions/models/elements should be created/updated.
 
 **COBieElements.ts**: subclasses of EC Elements.
 
 **COBieRelatedElements.ts**: subclasses of EC RelatedElement.
 
 **COBieRelationships.ts**: subclasses of EC relationships.
 
-**COBieSchemaConfig.ts**: configuration for schema mapping. 
+**COBieSchemaConfig.ts**: configuration for schema mapping.
 
 ## Walk-through
 
@@ -64,5 +70,5 @@ Given a COBie Excel file, A,
 1. Execute COBie-extractor module to produce an intermediary database, DB_A.
 2. Execute COBie-connector module on DB_A
    a. DataFetcher provides an abstraction layer to read data from DB_A
-   b. DynamicSchemaGenerator uses DataFetcher to dynamically generate an EC Schema and compares this newly generated EC Schema with the existing EC Schema stored in the current iModel. If the EC Schemas are different, it bumps up the minor version of EC Schema and imports it into the iModel to replace the deprecated schema. 
+   b. DynamicSchemaGenerator uses DataFetcher to dynamically generate an EC Schema and compares this newly generated EC Schema with the existing EC Schema stored in the current iModel. If the EC Schemas are different, it bumps up the minor version of EC Schema and imports it into the iModel to replace the deprecated schema.
    c. DataAligner takes in the Schema object generated by DynamicSchemaGenerator and uses the Schema to align all the data fetched from DataFetcher to the iModel. A change detection algorithm is in place to properly update iModel Elements after the first execution of the Connector.

COBie-connector/package.json

@@ -8,7 +8,7 @@
         "copyFiles": "npx babel src --out-dir lib --copy-files",
         "pretest": "cpx ./src/test/logging.config.json ./lib/test",
         "test": "npm run test:unit ",
-        "test:unit": "mocha --opts ./src/test/unit/mocha.opts \"./src/test/unit/**/*.test.ts*\"",
+        "test:unit": "mocha \"./src/test/unit/**/*.test.ts*\"",
         "test:integration": "npm run pretest && betools test --testDir=\"./lib/test/integration\"",
         "build": "tsc && npm run copyFiles",
         "clean": "rimraf lib",
@@ -63,5 +63,14 @@
         "nock": "^12.0.3",
         "tslint": "^5.20.1",
         "typescript": "^3.9.7"
+    },
+    "mocha": {
+        "require": [
+            "ts-node/register",
+            "jsdom-global/register",
+            "ignore-styles"
+        ],
+        "check-leaks": true,
+        "no-timeouts": true
     }
 }

COBie-extractor/README.md

@@ -4,9 +4,27 @@ A Python script that dumps COBie Excel data into an intermediary SQLite database
 
 ## Instructions
 
-1. run "pip install -r requirements.txt" to install dependencies
-2. run "make all" to create all intermediary databases
-3. Linux / WSL: run "sh transferdb.sh" to move the newly created to COBie-connector folder as the input for COBie connector. Other OS: manually move the intermediary databases to COBie-connector/test/assets/.
+1. Requires Python 3 (e.g. 3.9.5) [Python Downloads](https://www.python.org/downloads/)
+2. To install dependencies,
+
+        run "pip install -r requirements.txt" 
+
+        ![pip install command](./pip_install_requirements.png)
+
+3. Create all intermediary databases
+
+        a. Windows cmd shell: RunExtractorCmd.cmd --all, or ...
+
+        b. Linux / WSL: run "make all"
+        c. Other OS: manually run python commands
+4. Move the newly created to COBie-connector folder as the input for COBie connector.
+
+        a. Windows: RunExtractorCmd.cmd --copy
+
+![RunExtractor --copy](./RunExtractorCmdCopy.png)
+
+        b. Linux / WSL: run "sh transferdb.sh"
+        c. Other OS: manually move the intermediary databases to COBie-connector/test/assets/.
 
 ## Allowed Schema Changes
 

COBie-extractor/extractor/RunExtractor.cmd

@@ -0,0 +1,47 @@
+echo off
+
+SET ARG=%1
+
+IF NOT DEFINED ARG goto usage
+
+IF %ARG%==--clean goto clean
+IF %ARG%==--all goto all
+IF %ARG%==--copy goto copyToConnector
+goto usage
+
+:all
+	IF NOT DEFINED pythonCmd Set pythonCmd=python.exe
+	echo extracting all
+	IF NOT EXIST output mkdir output
+	%pythonCmd% extractor.py input/COBieSampleSheetV1.xlsx output/intermediary_v1.db # create
+	%pythonCmd% extractor.py input/COBieSampleSheetV2.xlsx output/intermediary_v2.db # data change
+	%pythonCmd% extractor.py input/COBieSampleSheetV3.xlsx output/intermediary_v3.db # schema change (addition)
+	%pythonCmd% extractor.py input/COBieSampleSheetV4.xlsx output/intermediary_v4.db # schema change (deletion)
+
+goto end
+
+:clean
+	echo cleaning
+	rmdir /s /q output
+
+goto end
+
+:copyToConnector
+set assetsDir=..\..\COBie-connector\src\test\assets\
+IF NOT EXIST %assetsDir% mkdir %assetsDir%
+echo copying databases from extractor output to connector assets
+copy .\output\*.db %assetsDir%
+set assetsDir=
+goto end
+
+:usage
+	echo RunExtractor usage
+	echo -----------------------------
+	echo RunExtractor "<option>"
+	echo e.g. RunExtractor --all
+	echo options:
+	echo --all - extracts all sample sheets
+	echo --clean - removes output from previous extractions
+	echo --copy - copies databases from extractor output to connector assets
+:end
+	echo runextractor completed!