EN: Initial draft of sample COBie connector.

COBie-connector/.gitignore

@@ -0,0 +1,7 @@
+*.db
+*.db-journal
+*.bim*
+lib/
+node_modules/
+**/package-lock.json
+.prettierrc

COBie-connector/README.md

@@ -0,0 +1,68 @@
+
+# [COBie](https://en.wikipedia.org/wiki/COBie#:~:text=Construction%20Operations%20Building%20Information%20Exchange,COBie%20was%20designed%20by%20Dr.)
+
+COBie is an international standard for building data exchange. Its most common use is in product data handover from construction to operations. The COBie specifications and guidelines capture industry knowledge and best practices. The COBie standards do not dictate what information is required for a specific project handover. That responsibility still lies with the owner. The COBie data model is a subset (“smart filter”) of the buildingSMART data model, more commonly known as IFC (Industry Foundation Classes). COBie is part of the openBIM movement to collaboratively design, build and operate buildings. It is also part of the UK Building Information Modelling (BIM) Task Group level 2 initiative. The most common representation of COBie is the COBie spreadsheet, but it is important to note that the data format can be represented in multiple ways according to the requirements and needs of the specific data transfer.
+
+![COBie sheet](./cobie_sheet.png)
+
+## COBie Sample Connector
+
+COBie iModel connector is a component that is capable of reading COBie sheets and incrementally create or update iModels with that data. It has two parts COBie Connector and COBie-extractor.
+
+![Run Sample Process](./how_to_run_sample.png)
+
+In the first phase, the COBie-extractor converts the COBie Excel file into an intermediary SQLite database.  To do so please run the provided python script available in this repository. Copy the result to the COBie-connector/src/test/assets directory so that COBie Connector can consume it.
+In the second phase, the COBie Connector reads the intermediary SQLite database created by COBie-extractor and creates/updates an iModel from it.
+
+## How to Run It
+
+You may find the sample COBie Excel data under COBie-extractor/extractor/input/*. [Sample Data Source](https://www.nibs.org/page/bsa_commonbimfiles)
+
+1. Execute COBie-extractor (see see how to inside COBie-extractor/README.md) 
+2. Move the output of COBie extractor (intermediary SQLite DB's) to COBie-connector/test/assets/ (or execute "sh transferdb" if you are on Linux / WSL)
+3. Run "npm run test:unit" (output iModel will be COBie-connector/test/output/final.db) This produces the iModel snapshot that can be used locally.
+4. To run the connector against a live iModel, run "npm run build" and "npm run test:integration" that will test if the Connector updates iModel data and schema. 
+   Note: You must set the environment variable imjs_config_dir = path/to/imodeljs-config (or the path to the directory that contains your default.json credential file) to successfully run the integration test.
+
+## Architecture
+
+Mapping of data into requires The COBie connector uses the following parts to map data from the intermediary database into an iModel. 
+
+### DataAligner (Reusable Parser)
+
+A DataAligner takes in an ElementTree and always walks down the tree in the same order as [Depth First Search](https://en.wikipedia.org/wiki/Depth-first_search#:~:text=a%20depth%2Dfirst%20search%20starting,%2C%20E%2C%20C%2C%20G.).
+It immediately creates the entity it encounters.
+
+### DataFetcher
+
+DataFetcher dynamically joins tables and return values from the intermediary SQLite database. 
+
+### EC Schema Generation
+
+COBie Connector creates and imports the entire EC Schema in memory and dynamically generates new EC properties/classes if a new column/table is added to the intermediary SQLite database. The schema of the intermediary database is completely dependent on the schema of the COBie Excel file. Each sheet (tab) in the COBie Excel file and each row is converted into a database table and database row, respectively.
+
+Notes:
+
+1. Referenced EC Schemas must be included in schema/ directory but do not need to be explicitly imported into the iModel.
+
+### Configuration Files
+
+**ElementTree.ts**: a dictionary that dictates the order in which partitions/models/elements should be created/updated. 
+
+**COBieElements.ts**: subclasses of EC Elements.
+
+**COBieRelatedElements.ts**: subclasses of EC RelatedElement.
+
+**COBieRelationships.ts**: subclasses of EC relationships.
+
+**COBieSchemaConfig.ts**: configuration for schema mapping. 
+
+## Walk-through
+
+Given a COBie Excel file, A,
+
+1. Execute COBie-extractor module to produce an intermediary database, DB_A.
+2. Execute COBie-connector module on DB_A
+   a. DataFetcher provides an abstraction layer to read data from DB_A
+   b. DynamicSchemaGenerator uses DataFetcher to dynamically generate an EC Schema and compares this newly generated EC Schema with the existing EC Schema stored in the current iModel. If the EC Schemas are different, it bumps up the minor version of EC Schema and imports it into the iModel to replace the deprecated schema. 
+   c. DataAligner takes in the Schema object generated by DynamicSchemaGenerator and uses the Schema to align all the data fetched from DataFetcher to the iModel. A change detection algorithm is in place to properly update iModel Elements after the first execution of the Connector.

COBie-connector/package.json

@@ -0,0 +1,67 @@
+{
+    "name": "@bentley/cobie-connector",
+    "version": "1.0.12",
+    "description": "iModel Connector to push Sample COBie Data to iModelHub",
+    "main": "./lib/Main.js",
+    "typings": "./lib/Main",
+    "scripts": {
+        "copyFiles": "npx babel src --out-dir lib --copy-files",
+        "pretest": "cpx ./src/test/logging.config.json ./lib/test",
+        "test": "npm run test:unit ",
+        "test:unit": "mocha --opts ./src/test/unit/mocha.opts \"./src/test/unit/**/*.test.ts*\"",
+        "test:integration": "npm run pretest && betools test --testDir=\"./lib/test/integration\"",
+        "build": "tsc && npm run copyFiles",
+        "clean": "rimraf lib",
+        "lint": "tslint --project .",
+        "blint": "npm run build && npm run lint",
+        "start": "node ./node_modules/@bentley/imodeljs-backend/lib/iModelBridgeFwkMain.js"
+    },
+    "author": {
+        "name": "Bentley Systems, Inc.",
+        "url": "http://www.bentley.com"
+    },
+    "license": "ISC",
+    "dependencies": {
+        "@bentley/backend-itwin-client": "2.4.0",
+        "@bentley/bentleyjs-core": "2.4.0",
+        "@bentley/config-loader": "^1.14.1",
+        "@bentley/ecschema-metadata": "2.4.0",
+        "@bentley/frontend-authorization-client": "2.4.0",
+        "@bentley/geometry-core": "2.4.0",
+        "@bentley/imodel-bridge": "2.4.0",
+        "@bentley/imodelhub-client": "2.4.0",
+        "@bentley/imodeljs-backend": "2.4.0",
+        "@bentley/imodeljs-common": "2.4.0",
+        "@bentley/imodeljs-i18n": "2.4.0",
+        "@bentley/itwin-client": "2.4.0",
+        "@bentley/logger-config": "2.4.0",
+        "@bentley/rbac-client": "",
+        "@types/sqlite3": "^3.1.6",
+        "@types/xmldom": "^0.1.30",
+        "bunyan": "^1.8.13",
+        "bunyan-seq": "^0.2.0",
+        "draco3d": "^1.3.6",
+        "open": "^7.1.0",
+        "request-promise": "^4.2.6",
+        "sqlite": "^4.0.12",
+        "sqlite3": "^5.0.0",
+        "three": "^0.116.1",
+        "username": "^5.1.0",
+        "xmldom": "^0.3.0"
+    },
+    "devDependencies": {
+        "@bentley/build-tools": "^1.14.1",
+        "@bentley/oidc-signin-tool": "2.4.0",
+        "@types/chai": "^4.2.12",
+        "@types/jquery": "^3.5.0",
+        "@types/mocha": "^5.2.6",
+        "@types/node": "^10.17.28",
+        "@types/object-hash": "^1.3.3",
+        "babel-cli": "^6.26.0",
+        "chai": "^4.2.0",
+        "mocha": "^5.2.0",
+        "nock": "^12.0.3",
+        "tslint": "^5.20.1",
+        "typescript": "^3.9.7"
+    }
+}

COBie-connector/src/COBieConnector.ts

@@ -0,0 +1,119 @@
+/*---------------------------------------------------------------------------------------------
+ * Copyright (c) Bentley Systems, Incorporated. All rights reserved.
+ * See LICENSE.md in the project root for license terms and full copyright notice.
+ *--------------------------------------------------------------------------------------------*/
+
+import { BentleyStatus, ClientRequestContext, IModelStatus, Logger } from "@bentley/bentleyjs-core";
+import { AuthorizedClientRequestContext } from "@bentley/itwin-client";
+import { CodeSpec, CodeScopeSpec, IModelError } from "@bentley/imodeljs-common";
+import { IModelBridge, loggerCategory } from "@bentley/imodel-bridge";
+import { IModelDb, IModelJsFs } from "@bentley/imodeljs-backend";
+import { Schema } from "@bentley/ecschema-metadata";
+import { ItemState, SourceItem, SynchronizationResults } from "@bentley/imodel-bridge/lib/Synchronizer";
+import { DataFetcher } from "./DataFetcher";
+import { DataAligner } from "./DataAligner";
+import { SAMPLE_ELEMENT_TREE } from "./COBieElementTree";
+import { DynamicSchemaGenerator, SchemaSyncResults } from "./DynamicSchemaGenerator";
+import { CodeSpecs } from "./COBieElements";
+import { COBieSchema } from "./COBieSchema";
+import * as path from "path";
+
+export class COBieConnector extends IModelBridge {
+  public sourceDataState: ItemState = ItemState.New;
+  public sourceDataPath?: string;
+  public dataFetcher?: DataFetcher;
+  public schemaGenerator?: DynamicSchemaGenerator;
+  public dynamicSchema?: Schema;
+
+  public initialize(_params: any) {}
+  public async initializeJob(): Promise<void> {}
+
+  public async openSourceData(sourcePath: string): Promise<BentleyStatus> {
+    this.sourceDataPath = sourcePath;
+    const sourceDataStatus = this.getSourceDataStatus();
+    this.sourceDataState = sourceDataStatus.itemState;
+    if (this.sourceDataState === ItemState.Unchanged) return BentleyStatus.SUCCESS;
+    this.dataFetcher = new DataFetcher(sourcePath);
+    await this.dataFetcher.initialize();
+    return BentleyStatus.SUCCESS;
+  }
+
+  public async importDomainSchema(_requestContext: AuthorizedClientRequestContext | ClientRequestContext): Promise<any> {
+    if (this.sourceDataState === ItemState.New) {
+      const functionalSchemaPath = path.join(__dirname, "./schema/Functional.ecschema.xml");
+      const spatialCompositionSchemaPath = path.join(__dirname, "./schema/SpatialComposition.ecschema.xml");
+      const buildingSpatialSchemaPath = path.join(__dirname, "./schema/BuildingSpatial.ecschema.xml");
+      await this.synchronizer.imodel.importSchemas(_requestContext, [functionalSchemaPath, spatialCompositionSchemaPath, buildingSpatialSchemaPath]);
+    }
+  }
+
+  public async importDynamicSchema(requestContext: AuthorizedClientRequestContext | ClientRequestContext): Promise<any> {
+    if (this.sourceDataState === ItemState.Unchanged) return;
+    if (this.sourceDataState === ItemState.New) COBieSchema.registerSchema();
+
+    const schemaGenerator = new DynamicSchemaGenerator(this.dataFetcher!);
+    this.schemaGenerator = schemaGenerator;
+    const results: SchemaSyncResults = await schemaGenerator.synchronizeSchema(this.synchronizer.imodel);
+    if (results.schemaState !== ItemState.Unchanged) {
+      const schemaString = await schemaGenerator.schemaToString(results.dynamicSchema);
+      await this.synchronizer.imodel.importSchemaStrings(requestContext, [schemaString]);
+    }
+    this.dynamicSchema = results.dynamicSchema;
+  }
+
+  public async importDefinitions(): Promise<any> {
+    if (this.sourceDataState === ItemState.New) this.insertCodeSpecs();
+  }
+
+  public async updateExistingData() {
+    if (this.sourceDataState === ItemState.Unchanged) return;
+    if (!this.dataFetcher) throw new Error("No DataFetcher available for DataAligner.");
+    if (!this.schemaGenerator) throw new Error("No DynamicSchemaGenerator available for DataAligner.");
+
+    const aligner = new DataAligner(this);
+    await aligner.align(SAMPLE_ELEMENT_TREE);
+    this.dataFetcher.close();
+  }
+
+  public insertCodeSpecs() {
+    const insert = (codeSpec: CodeSpecs) => {
+      if (this.synchronizer.imodel.codeSpecs.hasName(codeSpec)) return;
+      const newCodeSpec = CodeSpec.create(this.synchronizer.imodel, codeSpec, CodeScopeSpec.Type.Model);
+      const codeSpecId = this.synchronizer.imodel.codeSpecs.insert(newCodeSpec);
+    };
+    insert(CodeSpecs.COBie);
+  }
+
+  public getSourceDataStatus(): SynchronizationResults {
+    let timeStamp = Date.now();
+    if (!this.sourceDataPath) throw new Error("we should not be in this method if the source file has not yet been opened");
+    const stat = IModelJsFs.lstatSync(this.sourceDataPath);
+    if (undefined !== stat) timeStamp = stat.mtimeMs;
+    const sourceItem: SourceItem = {
+      id: this.sourceDataPath!,
+      version: timeStamp.toString(),
+    };
+    const sourceDataStatus = this.synchronizer.recordDocument(IModelDb.rootSubjectId, sourceItem);
+    if (undefined === sourceDataStatus) {
+      const error = `Failed to retrieve a RepositoryLink for ${this.sourceDataPath}`;
+      throw new IModelError(IModelStatus.BadArg, error, Logger.logError, loggerCategory);
+    }
+    return sourceDataStatus;
+  }
+
+  public getApplicationId(): string {
+    return "Test-Cobie";
+  }
+
+  public getApplicationVersion(): string {
+    return "1.0.0.0";
+  }
+
+  public getBridgeName(): string {
+    return "COBieConnector";
+  }
+}
+
+export function getBridgeInstance() {
+  return new COBieConnector();
+}