diff --git a/COBie-connector/.gitignore b/COBie-connector/.gitignore new file mode 100644 index 0000000..fec4c0d --- /dev/null +++ b/COBie-connector/.gitignore @@ -0,0 +1,7 @@ +*.db +*.db-journal +*.bim* +lib/ +node_modules/ +**/package-lock.json +.prettierrc diff --git a/COBie-connector/README.md b/COBie-connector/README.md new file mode 100644 index 0000000..431626f --- /dev/null +++ b/COBie-connector/README.md @@ -0,0 +1,68 @@ + +# [COBie](https://en.wikipedia.org/wiki/COBie#:~:text=Construction%20Operations%20Building%20Information%20Exchange,COBie%20was%20designed%20by%20Dr.) + +COBie is an international standard for building data exchange. Its most common use is in product data handover from construction to operations. The COBie specifications and guidelines capture industry knowledge and best practices. The COBie standards do not dictate what information is required for a specific project handover. That responsibility still lies with the owner. The COBie data model is a subset (“smart filter”) of the buildingSMART data model, more commonly known as IFC (Industry Foundation Classes). COBie is part of the openBIM movement to collaboratively design, build and operate buildings. It is also part of the UK Building Information Modelling (BIM) Task Group level 2 initiative. The most common representation of COBie is the COBie spreadsheet, but it is important to note that the data format can be represented in multiple ways according to the requirements and needs of the specific data transfer. + +![COBie sheet](./cobie_sheet.png) + +## COBie Sample Connector + +COBie iModel connector is a component that is capable of reading COBie sheets and incrementally create or update iModels with that data. It has two parts COBie Connector and COBie-extractor. + +![Run Sample Process](./how_to_run_sample.png) + +In the first phase, the COBie-extractor converts the COBie Excel file into an intermediary SQLite database. To do so please run the provided python script available in this repository. Copy the result to the COBie-connector/src/test/assets directory so that COBie Connector can consume it. +In the second phase, the COBie Connector reads the intermediary SQLite database created by COBie-extractor and creates/updates an iModel from it. + +## How to Run It + +You may find the sample COBie Excel data under COBie-extractor/extractor/input/*. [Sample Data Source](https://www.nibs.org/page/bsa_commonbimfiles) + +1. Execute COBie-extractor (see see how to inside COBie-extractor/README.md) +2. Move the output of COBie extractor (intermediary SQLite DB's) to COBie-connector/test/assets/ (or execute "sh transferdb" if you are on Linux / WSL) +3. Run "npm run test:unit" (output iModel will be COBie-connector/test/output/final.db) This produces the iModel snapshot that can be used locally. +4. To run the connector against a live iModel, run "npm run build" and "npm run test:integration" that will test if the Connector updates iModel data and schema. + Note: You must set the environment variable imjs_config_dir = path/to/imodeljs-config (or the path to the directory that contains your default.json credential file) to successfully run the integration test. + +## Architecture + +Mapping of data into requires The COBie connector uses the following parts to map data from the intermediary database into an iModel. + +### DataAligner (Reusable Parser) + +A DataAligner takes in an ElementTree and always walks down the tree in the same order as [Depth First Search](https://en.wikipedia.org/wiki/Depth-first_search#:~:text=a%20depth%2Dfirst%20search%20starting,%2C%20E%2C%20C%2C%20G.). +It immediately creates the entity it encounters. + +### DataFetcher + +DataFetcher dynamically joins tables and return values from the intermediary SQLite database. + +### EC Schema Generation + +COBie Connector creates and imports the entire EC Schema in memory and dynamically generates new EC properties/classes if a new column/table is added to the intermediary SQLite database. The schema of the intermediary database is completely dependent on the schema of the COBie Excel file. Each sheet (tab) in the COBie Excel file and each row is converted into a database table and database row, respectively. + +Notes: + +1. Referenced EC Schemas must be included in schema/ directory but do not need to be explicitly imported into the iModel. + +### Configuration Files + +**ElementTree.ts**: a dictionary that dictates the order in which partitions/models/elements should be created/updated. + +**COBieElements.ts**: subclasses of EC Elements. + +**COBieRelatedElements.ts**: subclasses of EC RelatedElement. + +**COBieRelationships.ts**: subclasses of EC relationships. + +**COBieSchemaConfig.ts**: configuration for schema mapping. + +## Walk-through + +Given a COBie Excel file, A, + +1. Execute COBie-extractor module to produce an intermediary database, DB_A. +2. Execute COBie-connector module on DB_A + a. DataFetcher provides an abstraction layer to read data from DB_A + b. DynamicSchemaGenerator uses DataFetcher to dynamically generate an EC Schema and compares this newly generated EC Schema with the existing EC Schema stored in the current iModel. If the EC Schemas are different, it bumps up the minor version of EC Schema and imports it into the iModel to replace the deprecated schema. + c. DataAligner takes in the Schema object generated by DynamicSchemaGenerator and uses the Schema to align all the data fetched from DataFetcher to the iModel. A change detection algorithm is in place to properly update iModel Elements after the first execution of the Connector. diff --git a/COBie-connector/cobie_sheet.png b/COBie-connector/cobie_sheet.png new file mode 100644 index 0000000..a4d3f01 Binary files /dev/null and b/COBie-connector/cobie_sheet.png differ diff --git a/COBie-connector/how_to_run_sample.png b/COBie-connector/how_to_run_sample.png new file mode 100644 index 0000000..de236e9 Binary files /dev/null and b/COBie-connector/how_to_run_sample.png differ diff --git a/COBie-connector/package.json b/COBie-connector/package.json new file mode 100644 index 0000000..f991561 --- /dev/null +++ b/COBie-connector/package.json @@ -0,0 +1,67 @@ +{ + "name": "@bentley/cobie-connector", + "version": "1.0.12", + "description": "iModel Connector to push Sample COBie Data to iModelHub", + "main": "./lib/Main.js", + "typings": "./lib/Main", + "scripts": { + "copyFiles": "npx babel src --out-dir lib --copy-files", + "pretest": "cpx ./src/test/logging.config.json ./lib/test", + "test": "npm run test:unit ", + "test:unit": "mocha --opts ./src/test/unit/mocha.opts \"./src/test/unit/**/*.test.ts*\"", + "test:integration": "npm run pretest && betools test --testDir=\"./lib/test/integration\"", + "build": "tsc && npm run copyFiles", + "clean": "rimraf lib", + "lint": "tslint --project .", + "blint": "npm run build && npm run lint", + "start": "node ./node_modules/@bentley/imodeljs-backend/lib/iModelBridgeFwkMain.js" + }, + "author": { + "name": "Bentley Systems, Inc.", + "url": "http://www.bentley.com" + }, + "license": "ISC", + "dependencies": { + "@bentley/backend-itwin-client": "2.4.0", + "@bentley/bentleyjs-core": "2.4.0", + "@bentley/config-loader": "^1.14.1", + "@bentley/ecschema-metadata": "2.4.0", + "@bentley/frontend-authorization-client": "2.4.0", + "@bentley/geometry-core": "2.4.0", + "@bentley/imodel-bridge": "2.4.0", + "@bentley/imodelhub-client": "2.4.0", + "@bentley/imodeljs-backend": "2.4.0", + "@bentley/imodeljs-common": "2.4.0", + "@bentley/imodeljs-i18n": "2.4.0", + "@bentley/itwin-client": "2.4.0", + "@bentley/logger-config": "2.4.0", + "@bentley/rbac-client": "", + "@types/sqlite3": "^3.1.6", + "@types/xmldom": "^0.1.30", + "bunyan": "^1.8.13", + "bunyan-seq": "^0.2.0", + "draco3d": "^1.3.6", + "open": "^7.1.0", + "request-promise": "^4.2.6", + "sqlite": "^4.0.12", + "sqlite3": "^5.0.0", + "three": "^0.116.1", + "username": "^5.1.0", + "xmldom": "^0.3.0" + }, + "devDependencies": { + "@bentley/build-tools": "^1.14.1", + "@bentley/oidc-signin-tool": "2.4.0", + "@types/chai": "^4.2.12", + "@types/jquery": "^3.5.0", + "@types/mocha": "^5.2.6", + "@types/node": "^10.17.28", + "@types/object-hash": "^1.3.3", + "babel-cli": "^6.26.0", + "chai": "^4.2.0", + "mocha": "^5.2.0", + "nock": "^12.0.3", + "tslint": "^5.20.1", + "typescript": "^3.9.7" + } +} diff --git a/COBie-connector/src/COBieConnector.ts b/COBie-connector/src/COBieConnector.ts new file mode 100644 index 0000000..eb8cdce --- /dev/null +++ b/COBie-connector/src/COBieConnector.ts @@ -0,0 +1,119 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Bentley Systems, Incorporated. All rights reserved. + * See LICENSE.md in the project root for license terms and full copyright notice. + *--------------------------------------------------------------------------------------------*/ + +import { BentleyStatus, ClientRequestContext, IModelStatus, Logger } from "@bentley/bentleyjs-core"; +import { AuthorizedClientRequestContext } from "@bentley/itwin-client"; +import { CodeSpec, CodeScopeSpec, IModelError } from "@bentley/imodeljs-common"; +import { IModelBridge, loggerCategory } from "@bentley/imodel-bridge"; +import { IModelDb, IModelJsFs } from "@bentley/imodeljs-backend"; +import { Schema } from "@bentley/ecschema-metadata"; +import { ItemState, SourceItem, SynchronizationResults } from "@bentley/imodel-bridge/lib/Synchronizer"; +import { DataFetcher } from "./DataFetcher"; +import { DataAligner } from "./DataAligner"; +import { SAMPLE_ELEMENT_TREE } from "./COBieElementTree"; +import { DynamicSchemaGenerator, SchemaSyncResults } from "./DynamicSchemaGenerator"; +import { CodeSpecs } from "./COBieElements"; +import { COBieSchema } from "./COBieSchema"; +import * as path from "path"; + +export class COBieConnector extends IModelBridge { + public sourceDataState: ItemState = ItemState.New; + public sourceDataPath?: string; + public dataFetcher?: DataFetcher; + public schemaGenerator?: DynamicSchemaGenerator; + public dynamicSchema?: Schema; + + public initialize(_params: any) {} + public async initializeJob(): Promise {} + + public async openSourceData(sourcePath: string): Promise { + this.sourceDataPath = sourcePath; + const sourceDataStatus = this.getSourceDataStatus(); + this.sourceDataState = sourceDataStatus.itemState; + if (this.sourceDataState === ItemState.Unchanged) return BentleyStatus.SUCCESS; + this.dataFetcher = new DataFetcher(sourcePath); + await this.dataFetcher.initialize(); + return BentleyStatus.SUCCESS; + } + + public async importDomainSchema(_requestContext: AuthorizedClientRequestContext | ClientRequestContext): Promise { + if (this.sourceDataState === ItemState.New) { + const functionalSchemaPath = path.join(__dirname, "./schema/Functional.ecschema.xml"); + const spatialCompositionSchemaPath = path.join(__dirname, "./schema/SpatialComposition.ecschema.xml"); + const buildingSpatialSchemaPath = path.join(__dirname, "./schema/BuildingSpatial.ecschema.xml"); + await this.synchronizer.imodel.importSchemas(_requestContext, [functionalSchemaPath, spatialCompositionSchemaPath, buildingSpatialSchemaPath]); + } + } + + public async importDynamicSchema(requestContext: AuthorizedClientRequestContext | ClientRequestContext): Promise { + if (this.sourceDataState === ItemState.Unchanged) return; + if (this.sourceDataState === ItemState.New) COBieSchema.registerSchema(); + + const schemaGenerator = new DynamicSchemaGenerator(this.dataFetcher!); + this.schemaGenerator = schemaGenerator; + const results: SchemaSyncResults = await schemaGenerator.synchronizeSchema(this.synchronizer.imodel); + if (results.schemaState !== ItemState.Unchanged) { + const schemaString = await schemaGenerator.schemaToString(results.dynamicSchema); + await this.synchronizer.imodel.importSchemaStrings(requestContext, [schemaString]); + } + this.dynamicSchema = results.dynamicSchema; + } + + public async importDefinitions(): Promise { + if (this.sourceDataState === ItemState.New) this.insertCodeSpecs(); + } + + public async updateExistingData() { + if (this.sourceDataState === ItemState.Unchanged) return; + if (!this.dataFetcher) throw new Error("No DataFetcher available for DataAligner."); + if (!this.schemaGenerator) throw new Error("No DynamicSchemaGenerator available for DataAligner."); + + const aligner = new DataAligner(this); + await aligner.align(SAMPLE_ELEMENT_TREE); + this.dataFetcher.close(); + } + + public insertCodeSpecs() { + const insert = (codeSpec: CodeSpecs) => { + if (this.synchronizer.imodel.codeSpecs.hasName(codeSpec)) return; + const newCodeSpec = CodeSpec.create(this.synchronizer.imodel, codeSpec, CodeScopeSpec.Type.Model); + const codeSpecId = this.synchronizer.imodel.codeSpecs.insert(newCodeSpec); + }; + insert(CodeSpecs.COBie); + } + + public getSourceDataStatus(): SynchronizationResults { + let timeStamp = Date.now(); + if (!this.sourceDataPath) throw new Error("we should not be in this method if the source file has not yet been opened"); + const stat = IModelJsFs.lstatSync(this.sourceDataPath); + if (undefined !== stat) timeStamp = stat.mtimeMs; + const sourceItem: SourceItem = { + id: this.sourceDataPath!, + version: timeStamp.toString(), + }; + const sourceDataStatus = this.synchronizer.recordDocument(IModelDb.rootSubjectId, sourceItem); + if (undefined === sourceDataStatus) { + const error = `Failed to retrieve a RepositoryLink for ${this.sourceDataPath}`; + throw new IModelError(IModelStatus.BadArg, error, Logger.logError, loggerCategory); + } + return sourceDataStatus; + } + + public getApplicationId(): string { + return "Test-Cobie"; + } + + public getApplicationVersion(): string { + return "1.0.0.0"; + } + + public getBridgeName(): string { + return "COBieConnector"; + } +} + +export function getBridgeInstance() { + return new COBieConnector(); +} diff --git a/COBie-connector/src/COBieElementTree.ts b/COBie-connector/src/COBieElementTree.ts new file mode 100644 index 0000000..4e456c6 --- /dev/null +++ b/COBie-connector/src/COBieElementTree.ts @@ -0,0 +1,233 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { Subject, DefinitionPartition, DefinitionModel, PhysicalPartition, PhysicalModel, SpatialLocationPartition, SpatialLocationModel, SpatialCategory, + InformationRecordPartition, InformationRecordModel, DocumentPartition, DocumentListModel, GroupInformationPartition, GroupModel } from "@bentley/imodeljs-backend"; +import * as COBieElements from "./COBieElements"; +import * as COBieRelationships from "./COBieRelationships"; +import * as COBieRelatedElements from "./COBieRelatedElements"; + +/* + +ElementTree Syntax: + +subjects: { : { partitions ... }, ... } +partitions: { : { models ... }, ... } +models: { : { elements | elementClasses ... }, ... } +elements: { : { element specifications ... }, ... } (# of elements created = Object.keys(elements).length) +elementClasses: { : { element specifications ... }, ... } (# of elements created = # of rows in a database table) +relationshipClasses: { : { relationship specifications ... }, ... } (# of elements created = # of rows in a database table) + +elementClasses | elements: { + | : { + ref: reference to the element's corresponding TypeScript class + (categoryName): globally unique name for a category. It is used to get the category + typeDefinition: { + ref: reference to the TypeDefinition Element's TS class + modelName: the name of the model that contains the TypeDefinition element + key: the foreign key of a table in the intermediary SQLite database. It is used to get the BIS codeValue of the related Element. + } + } +} + +relationshipClasses: { + : { + ref: reference to the relationship's corresponding TypeScript class + sourceRef: reference to the source element's corresponding TypeScript class + sourceModelName: the name of the model that contains the source element + sourceKey: the foreign key value used to find source element + targetRef: reference to the target element's corresponding TypeScript class + targetModelName: the name of the model that contains the target element + targetKey: the foreign key value used to find target element + } +} + +*/ + +export const SAMPLE_ELEMENT_TREE: any = { + subjects: { + Subject1: { + ref: Subject, + partitions: { + DefinitionPartition1: { + ref: DefinitionPartition, + models: { + DefinitionModel1: { + ref: DefinitionModel, + elements: { + SpatialCategory1: { + ref: SpatialCategory, + }, + }, + elementClasses: { + Type: { + ref: COBieElements.Type, + }, + }, + }, + }, + }, + PhysicalPartition1: { + ref: PhysicalPartition, + models: { + PhysicalModel1: { + ref: PhysicalModel, + elementClasses: { + Component: { + ref: COBieElements.Component, + categoryName: "SpatialCategory1", + typeDefinition: { + ref: COBieElements.Type, + modelName: "DefinitionModel1", + key: "Component.typename", + }, + }, + }, + relationshipClasses: { + ComponentConnectsToComponent: { + ref: COBieRelationships.ComponentConnectsToComponent, + sourceRef: COBieElements.Component, + sourceModelName: "PhysicalModel1", + sourceKey: "Connection.rowname1", + targetRef: COBieElements.Component, + targetModelName: "PhysicalModel1", + targetKey: "Connection.rowname2", + }, + ComponentAssemblesComponents: { + ref: COBieRelatedElements.ComponentAssemblesComponents, + sourceRef: COBieElements.Component, + sourceModelName: "PhysicalModel1", + sourceKey: "Assembly.parentname", + targetRef: COBieElements.Component, + targetModelName: "PhysicalModel1", + targetKey: "Assembly.childnames", + }, + }, + }, + }, + }, + SpatialLocationPartition1: { + ref: SpatialLocationPartition, + models: { + SpatialLocationModel1: { + ref: SpatialLocationModel, + elementClasses: { + Facility: { + ref: COBieElements.Facility, + categoryName: "SpatialCategory1", + }, + Floor: { + ref: COBieElements.Floor, + categoryName: "SpatialCategory1", + }, + Space: { + ref: COBieElements.Space, + categoryName: "SpatialCategory1", + }, + }, + relationshipClasses: { + FloorComposesSpaces: { + ref: COBieRelatedElements.FloorComposesSpaces, + sourceRef: COBieElements.Floor, + sourceModelName: "SpatialLocationModel1", + sourceKey: "Space.floorname", + targetRef: COBieElements.Space, + targetModelName: "SpatialLocationModel1", + targetKey: "Space.name", + }, + }, + }, + }, + }, + InformationRecordPartition1: { + ref: InformationRecordPartition, + models: { + InformationRecordModel1: { + ref: InformationRecordModel, + elementClasses: { + Assembly: { + ref: COBieElements.Assembly, + }, + Attribute: { + ref: COBieElements.Attribute, + }, + Contact: { + ref: COBieElements.Contact, + }, + Connection: { + ref: COBieElements.Connection, + }, + Resource: { + ref: COBieElements.Resource, + }, + Spare: { + ref: COBieElements.Spare, + }, + Job: { + ref: COBieElements.Job, + }, + Issue: { + ref: COBieElements.Issue, + }, + Impact: { + ref: COBieElements.Impact, + }, + }, + }, + }, + }, + GroupInformationPartition1: { + ref: GroupInformationPartition, + models: { + GroupInformationModel1: { + ref: GroupModel, + elementClasses: { + Zone: { + ref: COBieElements.Zone, + }, + System: { + ref: COBieElements.System, + }, + }, + relationshipClasses: { + ZoneIncludesSpaces: { + ref: COBieRelationships.ZoneIncludesSpaces, + sourceRef: COBieElements.Zone, + sourceModelName: "GroupInformationModel1", + sourceKey: "Zone.id", + targetRef: COBieElements.Space, + targetModelName: "SpatialLocationModel1", + targetKey: "Zone.spacenames", + }, + SystemGroupsComponents: { + ref: COBieRelationships.SystemGroupsComponents, + sourceRef: COBieElements.System, + sourceModelName: "GroupInformationModel1", + sourceKey: "System.id", + targetRef: COBieElements.Component, + targetModelName: "PhysicalModel1", + targetKey: "System.componentnames", + }, + }, + }, + }, + }, + DocumentPartition1: { + ref: DocumentPartition, + models: { + DocumentListModel1: { + ref: DocumentListModel, + elementClasses: { + Document: { + ref: COBieElements.Document, + }, + }, + }, + }, + }, + }, + }, + }, +}; diff --git a/COBie-connector/src/COBieElements.ts b/COBie-connector/src/COBieElements.ts new file mode 100644 index 0000000..81b274c --- /dev/null +++ b/COBie-connector/src/COBieElements.ts @@ -0,0 +1,326 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { IModelDb, InformationRecordElement, PhysicalType, GroupInformationElement, FunctionalElement, ElementMultiAspect, Document as BisCoreDocument, ElementAspect, GeometricModel3d, PhysicalElement, SpatialElement, GeometricElement3d } from "@bentley/imodeljs-backend"; +import { Code, Placement3d } from "@bentley/imodeljs-common"; +import { Point3d, YawPitchRollAngles, Range3d } from "@bentley/geometry-core"; +import { Id64String } from "@bentley/bentleyjs-core"; + +export enum CodeSpecs { + COBie = "COBieConnectorDynamicCOBie", +} + +function addPlacement(props: any, elementData: any) { + const placement = new Placement3d(new Point3d(), new YawPitchRollAngles(), new Range3d()); + if (elementData["Coordinate.id"] !== null) { + placement.origin.x = elementData["Coordinate.coordinatexaxis"]; + placement.origin.y = elementData["Coordinate.coordinateyaxis"]; + placement.origin.z = elementData["Coordinate.coordinatezaxis"]; + } + props.placement = placement; +} + +/** + * PhysicalElement + */ + +export class Component extends PhysicalElement { + public static get className(): string { return "Component"; } + public static get tableName(): string { return "Component"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Component"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + category: categoryId, + model: modelId, + classFullName: this.classFullName, + }; + addPlacement(props, elementData); + return props; + } +} + +/** + * SpatialLocationElement + */ + +export class Space extends SpatialElement { + public static get className(): string { return "Space"; } + public static get tableName(): string { return "Space"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Space"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + category: categoryId, + model: modelId, + classFullName: this.classFullName, + }; + addPlacement(props, elementData); + props.footprintArea = elementData["Space.grossarea"]; + return props; + } +} + +export class Floor extends SpatialElement { + public static get className(): string { return "Floor"; } + public static get tableName(): string { return "Floor"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Floor"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + category: categoryId, + model: modelId, + classFullName: this.classFullName, + }; + addPlacement(props, elementData); + return props; + } +} + +export class Facility extends SpatialElement { + public static get className(): string { return "Facility"; } + public static get tableName(): string { return "Facility"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Facility"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + category: categoryId, + model: modelId, + classFullName: this.classFullName, + }; + addPlacement(props, elementData); + return props; + } +} + +/** + * FunctionalElement + */ + +/** + * DefinitionElement + */ + +export class Type extends PhysicalType { + public static get className(): string { return "Type"; } + public static get tableName(): string { return "Type"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Type"; } + public constructor(props: any, iModel: IModelDb) { + super(props, iModel); + } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId?: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +/** + * GroupInformationElement + */ + +export class Zone extends GroupInformationElement { + public static get className(): string { return "Zone"; } + public static get tableName(): string { return "Zone"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Zone"; } + public constructor(props: any, iModel: IModelDb) { + super(props, iModel); + } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId?: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class System extends GroupInformationElement { + public static get className(): string { return "System"; } + public static get tableName(): string { return "System"; } + public static get classFullName(): string { return "COBieConnectorDynamic:System"; } + public constructor(props: any, iModel: IModelDb) { + super(props, iModel); + } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId?: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +/** + * InformationRecordElement + */ + +export class Connection extends InformationRecordElement { + public static get className(): string { return "Connection"; } + public static get tableName(): string { return "Connection"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Connection"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Assembly extends InformationRecordElement { + public static get className(): string { return "Assembly"; } + public static get tableName(): string { return "Assembly"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Assembly"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Attribute extends InformationRecordElement { + public static get className(): string { return "Attribute"; } + public static get tableName(): string { return "Attribute"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Attribute"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Contact extends InformationRecordElement { + public static get className(): string { return "Contact"; } + public static get tableName(): string { return "Contact"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Contact"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Impact extends InformationRecordElement { + public static get className(): string { return "Impact"; } + public static get tableName(): string { return "Impact"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Impact"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Issue extends InformationRecordElement { + public static get className(): string { return "Issue"; } + public static get tableName(): string { return "Issue"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Issue"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Spare extends InformationRecordElement { + public static get className(): string { return "Spare"; } + public static get tableName(): string { return "Spare"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Spare"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Job extends InformationRecordElement { + public static get className(): string { return "Job"; } + public static get tableName(): string { return "Job"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Job"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +export class Resource extends InformationRecordElement { + public static get className(): string { return "Resource"; } + public static get tableName(): string { return "Resource"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Resource"; } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} + +/** + * Document + */ +export class Document extends BisCoreDocument { + public static get className(): string { return "Document"; } + public static get tableName(): string { return "Document"; } + public static get classFullName(): string { return "COBieConnectorDynamic:Document"; } + public constructor(props: any, iModel: IModelDb) { + super(props, iModel); + } + public static createProps(modelId: Id64String, code: Code, elementClass: any, elementData: any, categoryId?: Id64String) { + const props: any = { + code, + userLabel: elementData[`${this.className}.name`], + model: modelId, + classFullName: this.classFullName, + }; + return props; + } +} diff --git a/COBie-connector/src/COBieRelatedElements.ts b/COBie-connector/src/COBieRelatedElements.ts new file mode 100644 index 0000000..3def6ea --- /dev/null +++ b/COBie-connector/src/COBieRelatedElements.ts @@ -0,0 +1,36 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { PhysicalElementAssemblesElements } from "@bentley/imodeljs-backend"; +import { RelatedElement, RelatedElementProps } from "@bentley/imodeljs-common"; +import { Id64String } from "@bentley/bentleyjs-core"; +import * as COBieElements from "./COBieElements"; +import * as COBieRelatedElements from "./COBieRelatedElements"; + +export class ComponentAssemblesComponents extends PhysicalElementAssemblesElements { + public static get className(): string { return "ComponentAssemblesComponents"; } + public static get tableName(): string { return "Assembly"; } + public static get classFullName(): string { return "COBieConnectorDynamic:ComponentAssemblesComponents"; } + public static addRelatedElement(parentComponentElement: COBieElements.Component, childComponentElement: COBieElements.Component, relatedElement: COBieRelatedElements.ComponentAssemblesComponents) { + (childComponentElement as any).parent = relatedElement; + return childComponentElement; + } + constructor(parentId: Id64String, childId: Id64String, relClassName: string) { + super(parentId, relClassName); + } +} + +export class FloorComposesSpaces extends RelatedElement { + public static get className(): string { return "FloorComposesSpaces"; } + public static get tableName(): string { return "Space"; } + public static get classFullName(): string { return "COBieConnectorDynamic:FloorComposesSpaces"; } + public static addRelatedElement(floorElement: COBieElements.Floor, spaceElement: COBieElements.Space, relatedElement: COBieRelatedElements.FloorComposesSpaces) { + (spaceElement as any).composingElement = relatedElement; + return spaceElement; + } + constructor(sourceId: Id64String, targetId: Id64String, relClassName: string) { + super({ id: sourceId, relClassName } as RelatedElementProps); + } +} diff --git a/COBie-connector/src/COBieRelationships.ts b/COBie-connector/src/COBieRelationships.ts new file mode 100644 index 0000000..f2ae093 --- /dev/null +++ b/COBie-connector/src/COBieRelationships.ts @@ -0,0 +1,34 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { ElementRefersToElements, ElementGroupsMembers } from "@bentley/imodeljs-backend"; +import { Id64String } from "@bentley/bentleyjs-core"; + +export class ComponentConnectsToComponent extends ElementRefersToElements { + public static get className(): string { return "ComponentConnectsToComponent"; } + public static get tableName(): string { return "Connection"; } + public static get classFullName(): string { return "COBieConnectorDynamic:ComponentConnectsToComponent"; } + public static createProps(sourceId: Id64String, targetId: Id64String) { + return { sourceId, targetId, classFullName: ComponentConnectsToComponent.classFullName }; + } +} + +export class SystemGroupsComponents extends ElementGroupsMembers { + public static get className(): string { return "SystemGroupsComponents"; } + public static get tableName(): string { return "System"; } + public static get classFullName(): string { return "COBieConnectorDynamic:SystemGroupsComponents"; } + public static createProps(sourceId: Id64String, targetId: Id64String) { + return { sourceId, targetId, classFullName: SystemGroupsComponents.classFullName }; + } +} + +export class ZoneIncludesSpaces extends ElementGroupsMembers { + public static get className(): string { return "ZoneIncludesSpaces"; } + public static get tableName(): string { return "Zone"; } + public static get classFullName(): string { return "COBieConnectorDynamic:ZoneIncludesSpaces"; } + public static createProps(sourceId: Id64String, targetId: Id64String) { + return { sourceId, targetId, classFullName: ZoneIncludesSpaces.classFullName }; + } +} diff --git a/COBie-connector/src/COBieSchema.ts b/COBie-connector/src/COBieSchema.ts new file mode 100644 index 0000000..11899e5 --- /dev/null +++ b/COBie-connector/src/COBieSchema.ts @@ -0,0 +1,23 @@ +/*--------------------------------------------------------------------------------------------- + * Copyright (c) Bentley Systems, Incorporated. All rights reserved. + * See LICENSE.md in the project root for license terms and full copyright notice. + *--------------------------------------------------------------------------------------------*/ + +import * as path from "path"; +import { ClassRegistry, Schema, Schemas } from "@bentley/imodeljs-backend"; +import * as elementsModule from "./COBieElements"; +import * as relationshipsModule from "./COBieRelationships"; + +export class COBieSchema extends Schema { + public static get schemaName(): string { + return "COBieConnectorDynamic"; + } + public static registerSchema() { + if (this !== Schemas.getRegisteredSchema(this.schemaName)) { + Schemas.unregisterSchema(this.schemaName); + Schemas.registerSchema(this); + ClassRegistry.registerModule(elementsModule, this); + ClassRegistry.registerModule(relationshipsModule, this); + } + } +} diff --git a/COBie-connector/src/DataAligner.ts b/COBie-connector/src/DataAligner.ts new file mode 100644 index 0000000..8a36992 --- /dev/null +++ b/COBie-connector/src/DataAligner.ts @@ -0,0 +1,195 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { Id64String } from "@bentley/bentleyjs-core"; +import { Code, CodeSpec, Placement3d, AxisAlignedBox3d, RelatedElement } from "@bentley/imodeljs-common"; +import { IModelDb, SpatialCategory, DrawingCategory } from "@bentley/imodeljs-backend"; +import { ItemState, SourceItem, ChangeResults, SynchronizationResults } from "@bentley/imodel-bridge/lib/Synchronizer"; +import * as COBieElements from "./COBieElements"; +import * as COBieRelationships from "./COBieRelationships"; +import * as COBieRelatedElements from "./COBieRelatedElements"; +import { COBieConnector } from "./COBieConnector"; +import { DataFetcher } from "./DataFetcher"; +import { DynamicSchemaGenerator } from "./DynamicSchemaGenerator"; +import * as hash from "object-hash"; +import { PropertyRenameReverseMap } from "./schema/COBieSchemaConfig"; + +export class DataAligner { + + public imodel: IModelDb; + public connector: COBieConnector; + public dataFetcher: DataFetcher; + public schemaGenerator: DynamicSchemaGenerator; + public schemaItems: {[className: string]: any}; + public categoryCache: {[categoryName: string]: Id64String}; + public modelCache: {[modelName: string]: Id64String}; + public elementCache: {[identifier: string]: Id64String}; + + constructor(connector: COBieConnector) { + this.connector = connector; + this.dataFetcher = connector.dataFetcher!; + this.imodel = connector.synchronizer.imodel; + this.schemaGenerator = connector.schemaGenerator!; + this.schemaItems = connector.dynamicSchema!.toJSON().items!; + this.categoryCache = {}; + this.modelCache = {}; + this.elementCache = {}; + } + + public async align(elementTree: any) { + const partitions = elementTree.subjects.Subject1.partitions; + const partitionNames = partitions ? Object.keys(partitions) : []; + for (const partitionName of partitionNames) { + const partition = partitions[partitionName]; + const models = partition.models; + const modelNames = models ? Object.keys(models) : []; + + for (const modelName of modelNames) { + const model = models[modelName]; + const modelId = this.updateModel(partition, model, modelName); + + const elements = model.elements; + const elementNames = elements ? Object.keys(elements) : []; + for (const elementName of elementNames) { + const element = elements[elementName]; + this.updateElement(modelId, element, elementName); + } + + const elementClasses = model.elementClasses; + const elementClassNames = elementClasses ? Object.keys(elementClasses) : []; + for (const elementClassName of elementClassNames) { + const elementClass = elementClasses[elementClassName]; + await this.updateElementClass(modelId, elementClass); + } + + const relationshipClasses = model.relationshipClasses; + const relationshipClassNames = relationshipClasses ? Object.keys(relationshipClasses) : []; + for (const relationshipClassName of relationshipClassNames) { + const relationshipClass = relationshipClasses[relationshipClassName]; + await this.updateRelationshipClass(relationshipClass); + } + } + } + } + + public updateModel(partition: any, model: any, modelName: string) { + const jobSubjectId = this.connector.jobSubject.id; + const existingModelId = this.imodel.elements.queryElementIdByCode(partition.ref.createCode(this.imodel, jobSubjectId, modelName)); + if (existingModelId) { + this.modelCache[modelName] = existingModelId; + return existingModelId; + } + const newModelId = model.ref.insert(this.imodel, jobSubjectId, modelName); + this.modelCache[modelName] = newModelId; + return newModelId; + } + + public updateElement(modelId: any, element: any, elementName: string) { + if (element.ref === SpatialCategory || element.ref === DrawingCategory) { + const existingCategoryId = element.ref.queryCategoryIdByName(this.imodel, modelId, elementName); + if (existingCategoryId) { + this.categoryCache[elementName] = existingCategoryId; + return existingCategoryId; + } + const newCategoryId = element.ref.insert(this.imodel, modelId, elementName); + this.categoryCache[elementName] = newCategoryId; + return newCategoryId; + } + } + + public async updateRelationshipClass(relationshipClass: any) { + const tableName = relationshipClass.ref.tableName; + const tableData = await this.dataFetcher.fetchTableData(tableName); + for (const elementData of tableData) { + const sourceModelId = this.modelCache[relationshipClass.sourceModelName]; + const targetModelId = this.modelCache[relationshipClass.targetModelName]; + const sourceCode = this.getCode(relationshipClass.sourceRef.className, sourceModelId, elementData[relationshipClass.sourceKey]); + const targetCode = this.getCode(relationshipClass.targetRef.className, targetModelId, elementData[relationshipClass.targetKey]); + const sourceId = this.imodel.elements.queryElementIdByCode(sourceCode)!; + const targetId = this.imodel.elements.queryElementIdByCode(targetCode)!; + + if (relationshipClass.ref.className in COBieRelatedElements) { + const sourceElement = this.imodel.elements.getElement(sourceId); + const targetElement = this.imodel.elements.getElement(targetId); + const relatedElement = new relationshipClass.ref(sourceId, targetId, relationshipClass.ref.classFullName); + const updatedElement = relationshipClass.ref.addRelatedElement(sourceElement, targetElement, relatedElement); + updatedElement.update(); + } else if (relationshipClass.ref.className in COBieRelationships) { + if (!sourceId || !targetId) continue; + const relationship = this.imodel.relationships.tryGetInstance(relationshipClass.ref.classFullName, { sourceId, targetId }); + if (relationship) continue; + const relationshipProps = relationshipClass.ref.createProps(sourceId, targetId); + const relationshipId = this.imodel.relationships.insertInstance(relationshipProps); + } + } + } + + public async updateElementClass(modelId: any, elementClass: any) { + const tableName = elementClass.ref.tableName; + const tableData = await this.dataFetcher.fetchTableData(tableName); + const categoryId = this.categoryCache[elementClass.categoryName]; + const primaryKey = this.dataFetcher.getTablePrimaryKey(tableName); + const codeSpec: CodeSpec = this.imodel.codeSpecs.getByName(COBieElements.CodeSpecs.COBie); + + for (const elementData of tableData) { + const guid = tableName + elementData[`${tableName}.${primaryKey}`]; + const code = new Code({ spec: codeSpec.id, scope: modelId, value: guid }); + const sourceItem: SourceItem = { id: guid, checksum: hash.MD5(JSON.stringify(elementData)) }; + const changeResults: ChangeResults = this.connector.synchronizer.detectChanges(modelId, tableName, sourceItem); + + if (changeResults.state === ItemState.Unchanged) { + this.connector.synchronizer.onElementSeen(changeResults.id!); + continue; + } + + const props = elementClass.ref.createProps(modelId, code, elementClass, elementData, categoryId); + this.addForeignProps(props, elementClass, elementData); + if (props.placement) this.updateExtent(props.placement); + + const existingElementId = this.imodel.elements.queryElementIdByCode(code); + const element = this.imodel.elements.createElement(props); + + if (existingElementId) element.id = existingElementId; + const syncResults: SynchronizationResults = { element, itemState: changeResults.state }; + this.connector.synchronizer.updateIModel(syncResults, modelId, sourceItem, tableName); + this.elementCache[guid] = element.id; + + if (elementClass.typeDefinition && changeResults.state === ItemState.New) + this.updateTypeDefinition(element, elementClass.typeDefinition, elementData); + + } + } + + public addForeignProps(props: any, elementClass: any, elementData: any) { + const { className } = elementClass.ref; + const { properties } = this.schemaItems[className]; + for (const prop of properties) { + const attribute = prop.name in PropertyRenameReverseMap ? PropertyRenameReverseMap[prop.name] : prop.name; + props[prop.name] = elementData[`${className}.${attribute}`]; + } + } + + public updateTypeDefinition(element: any, typeClass: any, elementData: any) { + const typeCode = this.getCode(typeClass.ref.className, this.modelCache[typeClass.modelName], elementData[typeClass.key]); + const typeDef = this.imodel.elements.getElement(typeCode); + element.typeDefinition = typeDef; + element.update(); + } + + public updateExtent(placement: Placement3d) { + const targetPlacement: Placement3d = Placement3d.fromJSON(placement); + const targetExtents: AxisAlignedBox3d = targetPlacement.calculateRange(); + if (!targetExtents.isNull && !this.imodel.projectExtents.containsRange(targetExtents)) { + targetExtents.extendRange(this.imodel.projectExtents); + this.imodel.updateProjectExtents(targetExtents); + } + } + + public getCode(tableName: string, modelId: Id64String, keyValue: string) { + const codeValue = `${tableName}${keyValue}`; + const codeSpec: CodeSpec = this.imodel.codeSpecs.getByName(COBieElements.CodeSpecs.COBie); + return new Code({spec: codeSpec.id, scope: modelId, value: codeValue}); + } +} diff --git a/COBie-connector/src/DataFetcher.ts b/COBie-connector/src/DataFetcher.ts new file mode 100644 index 0000000..87b9761 --- /dev/null +++ b/COBie-connector/src/DataFetcher.ts @@ -0,0 +1,118 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { IModelStatus } from "@bentley/bentleyjs-core"; +import { IModelError } from "@bentley/imodeljs-common"; +import * as sqlite3 from "sqlite3"; +import * as sqlite from "sqlite"; + +export class DataFetcher { + public sourcePath: string; + public sourceDb?: any; + + constructor(sourcePath: string) { + this.sourcePath = sourcePath; + this.sourceDb = undefined; + } + + public async initialize() { + this.sourceDb = await sqlite.open({ filename: this.sourcePath, driver: sqlite3.Database }); + if (!this.sourceDb) throw new IModelError(IModelStatus.BadArg, "Source database not found."); + } + + public close() { + this.sourceDb.close(); + } + + public async fetchTables() { + const tables = await this.sourceDb.all("select * from sqlite_master where type='table'"); + return tables; + } + + public async fetchColumns(tableName: string) { + const cols = await this.sourceDb.all(`PRAGMA table_info(${tableName})`); + return cols; + } + + public async fetchTableData(tableName: string) { + const colBuilder = []; + const queryBuilder = []; + const fkeyObject: any = TABLE_JOIN_MAP[tableName]; + + const createColString = async (_tableName: string) => { + const cols: [any] = await this.fetchColumns(_tableName); + return cols.map((col: any) => `${_tableName}.${col.name} as "${_tableName}.${col.name}"`).join(", "); + }; + + for (const fkey in fkeyObject) { + if (fkeyObject.hasOwnProperty(fkey)) { + const referenced = fkeyObject[fkey]; + const colString = await createColString(referenced.tableName); + colBuilder.push(colString); + const join = `left outer join ${referenced.tableName} on ${tableName}.${fkey} = ${referenced.tableName}.${referenced.colName}`; + queryBuilder.push(join); + } + } + + const baseColString = await createColString(tableName); + colBuilder.unshift(baseColString); + + const selectedCols = colBuilder.join(", "); + queryBuilder.unshift(`select ${selectedCols} from ${tableName}`); + queryBuilder.push(`group by "${tableName}.id"`); + + const query = queryBuilder.join(" "); + const tableData = await this.sourceDb.all(query); + return tableData; + } + + public async fetchAllData() { + const allData: { [tableName: string]: any } = {}; + const tables = await this.fetchTables(); + for (const table of tables) { + allData[table.name] = await this.fetchTableData(table.name); + } + return allData; + } + + public getTablePrimaryKey(tableName: string) { + switch (tableName) { + case "Contact": + return "email"; + case "Facility": + case "Floor": + case "Space": + case "Type": + case "Component": + case "Spare": + case "Resource": + return "name"; + default: + return "id"; + } + } + +} + +const TABLE_JOIN_MAP: any = { + Component: { + name: { + tableName: "Coordinate", + colName: "name", + }, + }, + Space: { + name: { + tableName: "Coordinate", + colName: "name", + }, + }, + Floor: { + name: { + tableName: "Coordinate", + colName: "name", + }, + }, +}; diff --git a/COBie-connector/src/DynamicSchemaGenerator.ts b/COBie-connector/src/DynamicSchemaGenerator.ts new file mode 100644 index 0000000..a4d2a82 --- /dev/null +++ b/COBie-connector/src/DynamicSchemaGenerator.ts @@ -0,0 +1,143 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { IModelDb } from "@bentley/imodeljs-backend"; +import { Schema, PrimitiveType, SchemaContextEditor, SchemaContext, SchemaComparer, ISchemaCompareReporter, SchemaChanges, ISchemaChanges, AnyDiagnostic } from "@bentley/ecschema-metadata"; +import { IModelSchemaLoader } from "@bentley/imodeljs-backend/lib/IModelSchemaLoader"; +import { MutableSchema } from "@bentley/ecschema-metadata/lib/Metadata/Schema"; +import { ItemState } from "@bentley/imodel-bridge/lib/Synchronizer"; +import { DOMParser, XMLSerializer } from "xmldom"; +import { DataFetcher } from "./DataFetcher"; +import { PropertyRenameMap, PropertyTypeMap, COBieBaseEntityProps, COBieEntityPropMap, COBieRelationshipProps } from "./schema/COBieSchemaConfig"; + +export class DynamicSchemaGenerator { + public dataFetcher: DataFetcher; + + constructor(dataFetcher: DataFetcher) { + this.dataFetcher = dataFetcher; + } + + public async synchronizeSchema(imodel: IModelDb): Promise { + const createBaseClasses = async (editor: SchemaContextEditor, schema: Schema) => { + for (const entityProp of COBieBaseEntityProps) { + const baseInsertResult = await editor.entities.createFromProps(schema.schemaKey, entityProp); + } + }; + + const createProperties = async (editor: SchemaContextEditor, table: any, entityInsertResult: any) => { + const cols = await this.dataFetcher.fetchColumns(table.name); + for (const col of cols) { + const propertyName: string = PropertyRenameMap.hasOwnProperty(col.name) ? PropertyRenameMap[col.name] : col.name; + const propertyType: any = PropertyTypeMap.hasOwnProperty(propertyName) ? PropertyTypeMap[propertyName] : { typeName: "string", typeValue: PrimitiveType.String }; + const property = { name: propertyName, type: "PrimitiveProperty", typeName: propertyType.typeName }; + const propertyInsertResult = await editor.entities.createPrimitivePropertyFromProps( + entityInsertResult.itemKey!, + propertyName, + propertyType.typeValue, + property, + ); + } + }; + + const createEntityclasses = async (editor: SchemaContextEditor, schema: Schema) => { + const tables = await this.dataFetcher.fetchTables(); + for (const table of tables) { + const entityClassProps = COBieEntityPropMap[table.name]; + if (!entityClassProps) continue; + const entityInsertResult = await editor.entities.createFromProps(schema.schemaKey, entityClassProps); + await createProperties(editor, table, entityInsertResult); + } + }; + + const createRelationshipClasses = async (editor: SchemaContextEditor, schema: Schema) => { + for (const relationshipClassProps of COBieRelationshipProps) { + const relationshipInsertResult = await editor.relationships.createFromProps(schema.schemaKey, relationshipClassProps); + } + }; + + const createSchema = async (increaseVersion: boolean) => { + const schemaVersion = imodel.querySchemaVersion("COBieConnectorDynamic"); + let [readVersion, writeVersion, minorVersion] = [1, 0, 0]; + if (increaseVersion) { + [readVersion, writeVersion, minorVersion] = schemaVersion!.split(".").map((version) => parseInt(version, 10)); + minorVersion += 1; + } + + const context = new SchemaContext(); + const editor = new SchemaContextEditor(context); + const newSchema = new Schema(context, "COBieConnectorDynamic", "cbd", readVersion, writeVersion, minorVersion); + + const bisSchema = loader.getSchema("BisCore"); + const funcSchema = loader.getSchema("Functional"); + const buildingSpatialSchema = loader.getSchema("BuildingSpatial"); + const spatialComppositionSchema = loader.getSchema("SpatialComposition"); + + await context.addSchema(newSchema); + await context.addSchema(bisSchema); + await context.addSchema(funcSchema); + await context.addSchema(buildingSpatialSchema); + await context.addSchema(spatialComppositionSchema); + await (newSchema as MutableSchema).addReference(bisSchema); // TODO remove this hack later + await (newSchema as MutableSchema).addReference(funcSchema); + await (newSchema as MutableSchema).addReference(buildingSpatialSchema); + await (newSchema as MutableSchema).addReference(spatialComppositionSchema); + + await createBaseClasses(editor, newSchema); + await createEntityclasses(editor, newSchema); + await createRelationshipClasses(editor, newSchema); + return newSchema; + }; + + const loader = new IModelSchemaLoader(imodel); + const existingSchema = loader.tryGetSchema("COBieConnectorDynamic"); + const latestSchema = await createSchema(false); + + let schemaState: ItemState = ItemState.New; + let dynamicSchema = latestSchema; + + if (existingSchema) { + const reporter = new DynamicSchemaCompareReporter(); + const comparer = new SchemaComparer(reporter); + await comparer.compareSchemas(latestSchema, existingSchema); + const schemaIsChanged = reporter.diagnostics.length > 0; + if (schemaIsChanged) { + schemaState = ItemState.Changed; + dynamicSchema = await createSchema(true); + } else { + schemaState = ItemState.Unchanged; + dynamicSchema = existingSchema; + } + } + return { schemaState, dynamicSchema } as SchemaSyncResults; + } + + public async schemaToString(schema: Schema): Promise { + let xmlDoc = new DOMParser().parseFromString(``); + xmlDoc = await schema.toXml(xmlDoc); + const xmlString = new XMLSerializer().serializeToString(xmlDoc); + return xmlString; + } +} + +export interface SchemaSyncResults { + schemaState: ItemState; + dynamicSchema: Schema; +} + +class DynamicSchemaCompareReporter implements ISchemaCompareReporter { + public changes: SchemaChanges[] = []; + + public report(schemaChanges: ISchemaChanges): void { + this.changes.push(schemaChanges as SchemaChanges); + } + + public get diagnostics(): AnyDiagnostic [] { + let diagnostics: AnyDiagnostic [] = []; + for (const changes of this.changes) { + diagnostics = diagnostics.concat(changes.allDiagnostics); + } + return diagnostics; + } +} diff --git a/COBie-connector/src/schema/AecUnits.ecschema.xml b/COBie-connector/src/schema/AecUnits.ecschema.xml new file mode 100644 index 0000000..8a8af16 --- /dev/null +++ b/COBie-connector/src/schema/AecUnits.ecschema.xml @@ -0,0 +1,89 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/COBie-connector/src/schema/BuildingSpatial.ecschema.xml b/COBie-connector/src/schema/BuildingSpatial.ecschema.xml new file mode 100644 index 0000000..5f06d22 --- /dev/null +++ b/COBie-connector/src/schema/BuildingSpatial.ecschema.xml @@ -0,0 +1,44 @@ + + + + + + + spcomp:CompositeElement + spcomp:ICompositeVolume + + + + + + + spcomp:CompositeElement + spcomp:ICompositeVolume + + + + + + + spcomp:CompositeElement + spcomp:ICompositeVolume + + + + Story + + + + ElevationStory + + + + + + diff --git a/COBie-connector/src/schema/COBieSchemaConfig.ts b/COBie-connector/src/schema/COBieSchemaConfig.ts new file mode 100644 index 0000000..4996984 --- /dev/null +++ b/COBie-connector/src/schema/COBieSchemaConfig.ts @@ -0,0 +1,253 @@ +/* tslint:disable */ +import { EntityClassProps, RelationshipClassProps, PrimitiveType } from "@bentley/ecschema-metadata"; + +export const PropertyTypeMap: {[propertyName: string]: any} = { + elevation: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + height: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + usableheight: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + grossarea: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + netarea: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + nominallength: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + nominalwidth: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + nominalheight: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + duration: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + coordinatexaxis: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + coordinateyaxis: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + coordinatezaxis: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + clockwiserotation: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + elevationalrotation: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, + yawrotation: { + typeName: "double", + typeValue: PrimitiveType.Double, + }, +}; + +function reverseMap(map: {[key: string]: string}) { + const reverse: {[ecPropertyName: string]: string} = {}; + const keys = Object.keys(map); + for (const key of keys) { + reverse[PropertyRenameMap[key]] = key; + } + return reverse; +} + +export const PropertyRenameMap: {[propertyName: string]: string} = { id: "rowid", category: "cobiecategory", name: "cobiename" }; +export const PropertyRenameReverseMap = reverseMap(PropertyRenameMap); + +export const COBieBaseEntityProps: EntityClassProps[] = []; +export const COBieEntityPropMap: { [className: string]: EntityClassProps } = { + Assembly: { + name: "Assembly", + baseClass: "BisCore:InformationRecordElement", + }, + Attribute: { + name: "Attribute", + baseClass: "BisCore:InformationRecordElement", + }, + Component: { + name: "Component", + baseClass: "BisCore:PhysicalElement", + }, + Connection: { + name: "Connection", + baseClass: "BisCore:InformationRecordElement", + }, + Contact: { + name: "Contact", + baseClass: "BisCore:InformationRecordElement", + }, + Document: { + name: "Document", + baseClass: "BisCore:Document", + }, + Facility: { + name: "Facility", + baseClass: "BuildingSpatial:Building", + }, + Floor: { + name: "Floor", + baseClass: "BuildingSpatial:RegularStory", + }, + Impact: { + name: "Impact", + baseClass: "BisCore:InformationRecordElement", + }, + Issue: { + name: "Issue", + baseClass: "BisCore:InformationRecordElement", + }, + Spare: { + name: "Spare", + baseClass: "BisCore:InformationRecordElement", + }, + Job: { + name: "Job", + baseClass: "BisCore:InformationRecordElement", + }, + Resource: { + name: "Resource", + baseClass: "BisCore:InformationRecordElement", + }, + Space: { + name: "Space", + baseClass: "BuildingSpatial:Space", + }, + System: { + name: "System", + baseClass: "BisCore:GroupInformationElement", + }, + Type: { + name: "Type", + baseClass: "BisCore:PhysicalType", + }, + Zone: { + name: "Zone", + baseClass: "BisCore:GroupInformationElement", + // NOT AVAILABLE YET: baseClass: "SpatialComposition:SpatialLocationGroup", + }, +}; + +export const COBieRelationshipProps: RelationshipClassProps[] = [ + { + name: "ComponentConnectsToComponent", + baseClass: "BisCore:ElementRefersToElements", + strength: "Referencing", + strengthDirection: "Forward", + source: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "From Component", + abstractConstraint: "BisCore.PhysicalElement", + constraintClasses: ["COBieConnectorDynamic.Component"], + }, + target: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "To Component", + abstractConstraint: "BisCore.PhysicalElement", + constraintClasses: ["COBieConnectorDynamic.Component"], + }, + }, + { + name: "ComponentAssemblesComponents", + baseClass: "BisCore:PhysicalElementAssemblesElements", + strength: "Embedding", + strengthDirection: "Forward", + source: { + polymorphic: true, + multiplicity: "(0..1)", + roleLabel: "assmbles", + abstractConstraint: "BisCore.PhysicalElement", + constraintClasses: ["COBieConnectorDynamic.Component"], + }, + target: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "is assembled by", + abstractConstraint: "BisCore.PhysicalElement", + constraintClasses: ["COBieConnectorDynamic.Component"], + }, + }, + { + name: "SystemGroupsComponents", + baseClass: "BisCore:ElementGroupsMembers", + strength: "Referencing", + strengthDirection: "Forward", + source: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "System", + abstractConstraint: "BisCore.GroupInformationElement", + constraintClasses: ["COBieConnectorDynamic.System"], + }, + target: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "Physical Component", + abstractConstraint: "BisCore.PhysicalElement", + constraintClasses: ["COBieConnectorDynamic.Component"], + }, + }, + { + name: "ZoneIncludesSpaces", + baseClass: "BisCore:ElementGroupsMembers", + strength: "Referencing", + strengthDirection: "Forward", + source: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "Zone", + abstractConstraint: "BisCore.GroupInformationElement", + constraintClasses: ["COBieConnectorDynamic.Zone"], + }, + target: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "Spaces", + abstractConstraint: "BisCore.SpatialLocationElement", + constraintClasses: ["COBieConnectorDynamic.Space"], + }, + }, + { + name: "FloorComposesSpaces", + baseClass: "SpatialComposition:CompositeComposesSubComposites", + strength: "Embedding", + strengthDirection: "Forward", + source: { + polymorphic: true, + multiplicity: "(0..1)", + roleLabel: "is composed by", + abstractConstraint: "SpatialComposition.CompositeElement", + constraintClasses: ["COBieConnectorDynamic.Floor"], + }, + target: { + polymorphic: true, + multiplicity: "(0..*)", + roleLabel: "composes", + abstractConstraint: "SpatialComposition.CompositeElement", + constraintClasses: ["COBieConnectorDynamic.Space"], + }, + }, +]; \ No newline at end of file diff --git a/COBie-connector/src/schema/Functional.ecschema.xml b/COBie-connector/src/schema/Functional.ecschema.xml new file mode 100644 index 0000000..a58bf16 --- /dev/null +++ b/COBie-connector/src/schema/Functional.ecschema.xml @@ -0,0 +1,110 @@ + + + + + + + + + bis:InformationPartitionElement + + + + + + + bis:RoleModel + + + + + + + bis:RoleElement + + + 32 + True + + + + + + + + + + + + FunctionalElement + bis:IParentElement + + + + + + + FunctionalBreakdownElement + + + + The best practice is now to inherit from a FunctionalBreakdownElement instead. + + + + + + FunctionalElement + + + + + + + + + + The best practice is now to inherit from a FunctionalComponentElement subclass and mix in ISubModeledElement when a breakdown concept is needed. + + + FunctionalComponentElement + bis:ISubModeledElement + + + + bis:TypeDefinitionElement + + + + + + + + + + + + + + + + bis:ElementRefersToElements + + + + + + + + + + bis:DrawingGraphicRepresentsElement + + + + + + + + + diff --git a/COBie-connector/src/schema/SpatialComposition.ecschema.xml b/COBie-connector/src/schema/SpatialComposition.ecschema.xml new file mode 100644 index 0000000..e14b46b --- /dev/null +++ b/COBie-connector/src/schema/SpatialComposition.ecschema.xml @@ -0,0 +1,55 @@ + + + + + + + + + + + + + + + + + + bis:SpatialLocationElement + + + + + + + + CompositeElement + + + + + + + + CompositeElement + + + + + + bis:ElementRefersToElements + + + + + + + + + diff --git a/COBie-connector/src/test/ConnectorTestUtils.ts b/COBie-connector/src/test/ConnectorTestUtils.ts new file mode 100644 index 0000000..0139405 --- /dev/null +++ b/COBie-connector/src/test/ConnectorTestUtils.ts @@ -0,0 +1,176 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { AuthorizedClientRequestContext, ITwinClientLoggerCategory } from "@bentley/itwin-client"; +import { BentleyLoggerCategory, Config, DbResult, Id64, Id64String, Logger, LogLevel } from "@bentley/bentleyjs-core"; +import { IModelJsConfig } from "@bentley/config-loader/lib/IModelJsConfig"; +import { ChangeSet, IModelHubClientLoggerCategory } from "@bentley/imodelhub-client"; +import { BackendLoggerCategory, BriefcaseManager, ECSqlStatement, ExternalSourceAspect, IModelDb, IModelHost, IModelHostConfiguration, IModelJsFs, NativeLoggerCategory, PhysicalPartition, Subject } from "@bentley/imodeljs-backend"; +import { BridgeJobDefArgs } from "@bentley/imodel-bridge"; +import * as path from "path"; +import { assert } from "chai"; +import { KnownTestLocations } from "./KnownTestLocations"; +import { IModelBankArgs, IModelBankUtils } from "@bentley/imodel-bridge/lib/IModelBankUtils"; +import { IModelHubUtils } from "@bentley/imodel-bridge/lib/IModelHubUtils"; +import { HubUtility } from "./HubUtility"; +import { BridgeLoggerCategory } from "@bentley/imodel-bridge/lib/BridgeLoggerCategory"; +import * as COBieElement from "../COBieElements"; +import { IModel } from "@bentley/imodeljs-common"; + +export class TestIModelInfo { + private _name: string; + private _id: string; + private _localReadonlyPath: string; + private _localReadWritePath: string; + private _changeSets: ChangeSet[]; + + constructor(name: string) { + this._name = name; + this._id = ""; + this._localReadonlyPath = ""; + this._localReadWritePath = ""; + this._changeSets = []; + } + + get name(): string { return this._name; } + set name(name: string) { this._name = name; } + get id(): string { return this._id; } + set id(id: string) { this._id = id; } + get localReadonlyPath(): string { return this._localReadonlyPath; } + set localReadonlyPath(localReadonlyPath: string) { this._localReadonlyPath = localReadonlyPath; } + get localReadWritePath(): string { return this._localReadWritePath; } + set localReadWritePath(localReadWritePath: string) { this._localReadWritePath = localReadWritePath; } + get changeSets(): ChangeSet[] { return this._changeSets; } + set changeSets(changeSets: ChangeSet[]) { this._changeSets = changeSets; } +} + +function getCount(imodel: IModelDb, className: string) { + let count = 0; + imodel.withPreparedStatement("SELECT count(*) AS [count] FROM " + className, (stmt: ECSqlStatement) => { + assert.equal(DbResult.BE_SQLITE_ROW, stmt.step()); + const row = stmt.getRow(); + count = row.count; + }); + return count; +} + +export class ConnectorTestUtils { + public static setupLogging() { + Logger.initializeToConsole(); + Logger.setLevelDefault(LogLevel.Error); + + if (process.env.imjs_test_logging_config === undefined) { + // tslint:disable-next-line:no-console + console.log(`You can set the environment variable imjs_test_logging_config to point to a logging configuration json file.`); + } + const loggingConfigFile: string = process.env.imjs_test_logging_config || path.join(__dirname, "logging.config.json"); + + if (IModelJsFs.existsSync(loggingConfigFile)) { + // tslint:disable-next-line:no-console + console.log(`Setting up logging levels from ${loggingConfigFile}`); + // tslint:disable-next-line:no-var-requires + Logger.configureLevels(require(loggingConfigFile)); + } + } + + private static initDebugLogLevels(reset?: boolean) { + Logger.setLevelDefault(reset ? LogLevel.Error : LogLevel.Warning); + Logger.setLevel(BentleyLoggerCategory.Performance, reset ? LogLevel.Error : LogLevel.Info); + Logger.setLevel(BackendLoggerCategory.IModelDb, reset ? LogLevel.Error : LogLevel.Trace); + Logger.setLevel(BridgeLoggerCategory.Framework, reset ? LogLevel.Error : LogLevel.Trace); + Logger.setLevel(ITwinClientLoggerCategory.Clients, reset ? LogLevel.Error : LogLevel.Warning); + Logger.setLevel(IModelHubClientLoggerCategory.IModelHub, reset ? LogLevel.Error : LogLevel.Warning); + Logger.setLevel(ITwinClientLoggerCategory.Request, reset ? LogLevel.Error : LogLevel.Warning); + + Logger.setLevel(NativeLoggerCategory.DgnCore, reset ? LogLevel.Error : LogLevel.Warning); + Logger.setLevel(NativeLoggerCategory.BeSQLite, reset ? LogLevel.Error : LogLevel.Warning); + Logger.setLevel(NativeLoggerCategory.Licensing, reset ? LogLevel.Error : LogLevel.Warning); + Logger.setLevel(NativeLoggerCategory.ECDb, LogLevel.Trace); + Logger.setLevel(NativeLoggerCategory.ECObjectsNative, LogLevel.Trace); + Logger.setLevel(NativeLoggerCategory.UnitsNative, LogLevel.Trace); + } + + // Setup typical programmatic log level overrides here + // Convenience method used to debug specific tests/fixtures + public static setupDebugLogLevels() { + ConnectorTestUtils.initDebugLogLevels(false); + } + + public static resetDebugLogLevels() { + ConnectorTestUtils.initDebugLogLevels(true); + } + + public static async getTestModelInfo(requestContext: AuthorizedClientRequestContext, testProjectId: string, iModelName: string): Promise { + const iModelInfo = new TestIModelInfo(iModelName); + iModelInfo.id = await HubUtility.queryIModelIdByName(requestContext, testProjectId, iModelInfo.name); + + iModelInfo.changeSets = await BriefcaseManager.imodelClient.changeSets.get(requestContext, iModelInfo.id); + return iModelInfo; + } + + public static async startBackend(clientArgs?: IModelBankArgs): Promise { + const result = IModelJsConfig.init(true /* suppress exception */, false /* suppress error message */, Config.App); + const config = new IModelHostConfiguration(); + config.concurrentQuery.concurrent = 4; // for test restrict this to two threads. Making closing connection faster + config.cacheDir = KnownTestLocations.outputDir; + config.imodelClient = (undefined === clientArgs) ? IModelHubUtils.makeIModelClient() : IModelBankUtils.makeIModelClient(clientArgs); + await IModelHost.startup(config); + } + + public static async shutdownBackend(): Promise { + await IModelHost.shutdown(); + } + + public static verifyIModel(imodel: IModelDb, bridgeJobDef: BridgeJobDefArgs, isUpdate: boolean = false, isSchemaUpdate: boolean = false) { + assert.isDefined(imodel.getMetaData("COBieConnectorDynamic:Contact"), "Schema is imported."); + assert.equal(isUpdate ? 56 : 58, getCount(imodel, "COBieConnectorDynamic:Contact")); + assert.equal(1, getCount(imodel, "COBieConnectorDynamic:Facility")); + assert.equal(4, getCount(imodel, "COBieConnectorDynamic:Floor")); + assert.equal(22, getCount(imodel, "COBieConnectorDynamic:Space")); + assert.equal(20, getCount(imodel, "COBieConnectorDynamic:Zone")); + assert.equal(43, getCount(imodel, "COBieConnectorDynamic:Type")); + assert.equal(isUpdate ? 233 : 232, getCount(imodel, "COBieConnectorDynamic:Component")); + assert.equal(36, getCount(imodel, "COBieConnectorDynamic:System")); + assert.equal(3, getCount(imodel, "COBieConnectorDynamic:Spare")); + assert.equal(10, getCount(imodel, "COBieConnectorDynamic:Resource")); + assert.equal(94, getCount(imodel, "COBieConnectorDynamic:Job")); + assert.equal(0, getCount(imodel, "COBieConnectorDynamic:Impact")); + assert.equal(48, getCount(imodel, "COBieConnectorDynamic:Document")); + assert.equal(94, getCount(imodel, "COBieConnectorDynamic:Attribute")); + assert.equal(0, getCount(imodel, "COBieConnectorDynamic:Issue")); + assert.equal(1, getCount(imodel, "BisCore:SpatialCategory")); + assert.equal(22, getCount(imodel, "COBieConnectorDynamic:FloorComposesSpaces")); + assert.equal(36, getCount(imodel, "COBieConnectorDynamic:SystemGroupsComponents")); + assert.equal(20, getCount(imodel, "COBieConnectorDynamic:ZoneIncludesSpaces")); + assert.equal(2, getCount(imodel, "COBieConnectorDynamic:Connection")); + assert.equal(2, getCount(imodel, "COBieConnectorDynamic:ComponentConnectsToComponent")); + assert.equal(2, getCount(imodel, "COBieConnectorDynamic:Assembly")); + assert.equal(2, getCount(imodel, "COBieConnectorDynamic:ComponentAssemblesComponents")); + // assert.equal(704, getCount(imodel, "BisCore:ExternalSourceAspect")); + + assert.isTrue(imodel.codeSpecs.hasName(COBieElement.CodeSpecs.COBie)); + const jobSubjectName = `COBieConnector:${bridgeJobDef.sourcePath!}`; + const subjectId: Id64String = imodel.elements.queryElementIdByCode(Subject.createCode(imodel, IModel.rootSubjectId, jobSubjectName))!; + assert.isTrue(Id64.isValidId64(subjectId)); + + const spatialLocationModel = imodel.elements.queryElementIdByCode(PhysicalPartition.createCode(imodel, subjectId, "SpatialLocationModel1")); + assert.isTrue(spatialLocationModel !== undefined); + assert.isTrue(Id64.isValidId64(spatialLocationModel!)); + + const ids = ExternalSourceAspect.findBySource(imodel, spatialLocationModel!, "Space", "SpaceSite"); + assert.isTrue(Id64.isValidId64(ids.aspectId!)); + assert.isTrue(Id64.isValidId64(ids.elementId!)); + const spaceElement = imodel.elements.getElement(ids.elementId!); + assert.equal((spaceElement as any).floorname, isUpdate ? "Level 2" : "Level 1", "Floorname was updated."); + + if (isSchemaUpdate) { + const ids = ExternalSourceAspect.findBySource(imodel, spatialLocationModel!, "Floor", "FloorLevel 1"); + assert.isTrue(Id64.isValidId64(ids.aspectId!)); + assert.isTrue(Id64.isValidId64(ids.elementId!)); + const floorElement = imodel.elements.getElement(ids.elementId!); + assert.equal((floorElement as any).buildingname, "B1"); + } + } +} diff --git a/COBie-connector/src/test/HubUtility.ts b/COBie-connector/src/test/HubUtility.ts new file mode 100644 index 0000000..8392a33 --- /dev/null +++ b/COBie-connector/src/test/HubUtility.ts @@ -0,0 +1,139 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { AuthorizedClientRequestContext } from "@bentley/itwin-client"; +import { GuidString, Logger } from "@bentley/bentleyjs-core"; +import { BriefcaseManager } from "@bentley/imodeljs-backend"; +import { Briefcase as HubBriefcase, BriefcaseQuery, HubIModel, IModelHubClient, IModelQuery } from "@bentley/imodelhub-client"; +import { Project } from "@bentley/context-registry-client"; + +export class HubUtility { + public static logCategory = "HubUtility"; + + public static async queryIModelByName(requestContext: AuthorizedClientRequestContext, projectId: string, iModelName: string): Promise { + const iModels = await getIModelProjectAbstraction().queryIModels(requestContext, projectId, new IModelQuery().byName(iModelName)); + if (iModels.length === 0) + return undefined; + if (iModels.length > 1) + throw new Error(`Too many iModels with name ${iModelName} found`); + return iModels[0]; + } + + /** + * Queries the project id by its name + * @param requestContext The client request context + * @param projectName Name of project + * @throws If the project is not found, or there is more than one project with the supplied name + */ + public static async queryProjectIdByName(requestContext: AuthorizedClientRequestContext, projectName: string): Promise { + const project: Project | undefined = await HubUtility.queryProjectByName(requestContext, projectName); + if (!project) + throw new Error(`Project ${projectName} not found`); + return project.wsgId; + } + + /** + * Queries the iModel id by its name + * @param requestContext The client request context + * @param projectId Id of the project + * @param iModelName Name of the iModel + * @throws If the iModel is not found, or if there is more than one iModel with the supplied name + */ + public static async queryIModelIdByName(requestContext: AuthorizedClientRequestContext, projectId: string, iModelName: string): Promise { + const iModel: HubIModel | undefined = await HubUtility.queryIModelByName(requestContext, projectId, iModelName); + if (!iModel || !iModel.id) + throw new Error(`IModel ${iModelName} not found`); + return iModel.id!; + } + + private static async queryProjectByName(requestContext: AuthorizedClientRequestContext, projectName: string): Promise { + const project: Project = await getIModelProjectAbstraction().queryProject(requestContext, { + $select: "*", + $filter: "Name+eq+'" + projectName + "'", + }); + return project; + } + + /** + * Purges all acquired briefcases for the specified iModel (and user), if the specified threshold of acquired briefcases is exceeded + */ + public static async purgeAcquiredBriefcasesById(requestContext: AuthorizedClientRequestContext, iModelId: GuidString, onReachThreshold: () => void, acquireThreshold: number = 16): Promise { + const briefcases: HubBriefcase[] = await BriefcaseManager.imodelClient.briefcases.get(requestContext, iModelId, new BriefcaseQuery().ownedByMe()); + if (briefcases.length > acquireThreshold) { + onReachThreshold(); + + const promises = new Array>(); + briefcases.forEach((briefcase: HubBriefcase) => { + promises.push(BriefcaseManager.imodelClient.briefcases.delete(requestContext, iModelId, briefcase.briefcaseId!)); + }); + await Promise.all(promises); + } + } + + /** + * Purges all acquired briefcases for the specified iModel (and user), if the specified threshold of acquired briefcases is exceeded + */ + public static async purgeAcquiredBriefcases(requestContext: AuthorizedClientRequestContext, projectName: string, iModelName: string, acquireThreshold: number = 16): Promise { + const projectId: string = await HubUtility.queryProjectIdByName(requestContext, projectName); + const iModelId: GuidString = await HubUtility.queryIModelIdByName(requestContext, projectId, iModelName); + + return this.purgeAcquiredBriefcasesById(requestContext, iModelId, () => { + Logger.logInfo(HubUtility.logCategory, `Reached limit of maximum number of briefcases for ${projectName}:${iModelName}. Purging all briefcases.`); + }, acquireThreshold); + } + /** Create */ + public static async recreateIModel(requestContext: AuthorizedClientRequestContext, projectId: GuidString, iModelName: string): Promise { + // Delete any existing iModel + try { + const deleteIModelId: GuidString = await HubUtility.queryIModelIdByName(requestContext, projectId, iModelName); + await BriefcaseManager.imodelClient.iModels.delete(requestContext, projectId, deleteIModelId); + } catch (err) { + } + + // Create a new iModel + const iModel: HubIModel = await BriefcaseManager.imodelClient.iModels.create(requestContext, projectId, iModelName, { description: `Description for ${iModelName}` }); + return iModel.wsgId; + } +} + +class TestIModelHubProject { + public get isIModelHub(): boolean { return true; } + public terminate(): void { } + + public get iModelHubClient(): IModelHubClient { + return BriefcaseManager.imodelClient as IModelHubClient; + } + + public async queryProject(requestContext: AuthorizedClientRequestContext, query: any | undefined): Promise { + const client = BriefcaseManager.connectClient; + return client.getProject(requestContext, query); + } + public async createIModel(requestContext: AuthorizedClientRequestContext, projectId: string, params: any): Promise { + const client = this.iModelHubClient; + return client.iModels.create(requestContext, projectId, params.name, { path: params.seedFile, description: params.description, progressCallback: params.tracker }); + } + public async deleteIModel(requestContext: AuthorizedClientRequestContext, projectId: string, iModelId: GuidString): Promise { + const client = this.iModelHubClient; + return client.iModels.delete(requestContext, projectId, iModelId); + } + public async queryIModels(requestContext: AuthorizedClientRequestContext, projectId: string, query: IModelQuery | undefined): Promise { + const client = this.iModelHubClient; + return client.iModels.get(requestContext, projectId, query); + } +} + +let projectAbstraction: any; +const usingMocks = false; + +export function getIModelProjectAbstraction(): any { + if (projectAbstraction !== undefined) + return projectAbstraction; + + if ((process.env.IMODELJS_CLIENTS_TEST_IMODEL_BANK === undefined) || usingMocks) { + return projectAbstraction = new TestIModelHubProject(); + } + + throw new Error("WIP"); +} diff --git a/COBie-connector/src/test/KnownTestLocations.ts b/COBie-connector/src/test/KnownTestLocations.ts new file mode 100644 index 0000000..ddd755c --- /dev/null +++ b/COBie-connector/src/test/KnownTestLocations.ts @@ -0,0 +1,33 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import * as path from "path"; +import { Platform } from "@bentley/imodeljs-backend"; + +export class KnownTestLocations { + + /** The directory where test assets are stored. Keep in mind that the test is playing the role of the app. */ + public static get assetsDir(): string { + const imodeljsMobile = Platform.imodeljsMobile; + if (imodeljsMobile !== undefined) { + return path.join(process.execPath!, "Assets", "assets"); + } + + // Assume that we are running in nodejs + return path.join(__dirname, "assets"); + } + + /** The directory where tests can write. */ + public static get outputDir(): string { + const imodeljsMobile = Platform.imodeljsMobile; + if (imodeljsMobile !== undefined) { + return imodeljsMobile.knownLocations.tempDir; + } + + // Assume that we are running in nodejs + return path.join(__dirname, "output"); + } + +} diff --git a/COBie-connector/src/test/integration/COBieBridge.test.ts b/COBie-connector/src/test/integration/COBieBridge.test.ts new file mode 100644 index 0000000..1b2ca1a --- /dev/null +++ b/COBie-connector/src/test/integration/COBieBridge.test.ts @@ -0,0 +1,115 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { expect } from "chai"; +import * as path from "path"; +import { TestUsers, TestUtility } from "@bentley/oidc-signin-tool"; +import { BridgeJobDefArgs, BridgeRunner } from "@bentley/imodel-bridge"; +import { ServerArgs } from "@bentley/imodel-bridge/lib/IModelHubUtils"; +import { ConnectorTestUtils, TestIModelInfo } from "../ConnectorTestUtils"; +import { BriefcaseDb, BriefcaseManager, IModelJsFs } from "@bentley/imodeljs-backend"; +import { AccessToken, AuthorizedClientRequestContext } from "@bentley/itwin-client"; +import { BentleyStatus, ClientRequestContext, Logger, OpenMode } from "@bentley/bentleyjs-core"; +import { KnownTestLocations } from "../KnownTestLocations"; +import { HubUtility } from "../HubUtility"; + +describe("COBie Sample Connector Integration Test (Online)", () => { + + let testProjectId: string; + let requestContext: AuthorizedClientRequestContext; + let sampleIModel: TestIModelInfo; + const fs = require("fs"); + // let managerRequestContext: AuthorizedClientRequestContext; + + before(async () => { + // ConnectorTestUtils.setupLogging(); + // ConnectorTestUtils.setupDebugLogLevels(); + await ConnectorTestUtils.startBackend(); + + if (!IModelJsFs.existsSync(KnownTestLocations.outputDir)) + IModelJsFs.mkdirSync(KnownTestLocations.outputDir); + + try { + requestContext = await TestUtility.getAuthorizedClientRequestContext(TestUsers.regular); + } catch (error) { + Logger.logError("Error", `Failed with error: ${error}`); + } + testProjectId = await HubUtility.queryProjectIdByName(requestContext, "imodeljs_sampleConnector_test"); + const targetIModelId = await HubUtility.recreateIModel(requestContext, testProjectId, "TestSampleConnector"); + expect(undefined !== targetIModelId); + sampleIModel = await ConnectorTestUtils.getTestModelInfo(requestContext, testProjectId, "TestSampleConnector"); + + // Purge briefcases that are close to reaching the acquire limit + // managerRequestContext = await TestUtility.getAuthorizedClientRequestContext(TestUsers.manager); + // await HubUtility.purgeAcquiredBriefcases(managerRequestContext, "imodeljs_sampleConnector_test", "TestSampleConnector"); + }); + + after(async () => { + await ConnectorTestUtils.shutdownBackend(); + IModelJsFs.purgeDirSync(KnownTestLocations.outputDir); + IModelJsFs.unlinkSync(path.join(KnownTestLocations.assetsDir, "test.db")); + }); + + const runConnector = async (bridgeJobDef: BridgeJobDefArgs, serverArgs: ServerArgs, isUpdate: boolean = false, isSchemaUpdate: boolean = false) => { + const runner = new BridgeRunner(bridgeJobDef, serverArgs); + const status = await runner.synchronize(); + expect(status === BentleyStatus.SUCCESS); + const briefcases = BriefcaseManager.getBriefcases(); + const briefcaseEntry = BriefcaseManager.findBriefcaseByKey(briefcases[0].key); + expect(briefcaseEntry !== undefined); + let imodel: BriefcaseDb; + imodel = await BriefcaseDb.open(new ClientRequestContext(), briefcases[0].key, { openAsReadOnly: true }); + ConnectorTestUtils.verifyIModel(imodel, bridgeJobDef, isUpdate, isSchemaUpdate); + briefcaseEntry!.openMode = OpenMode.ReadWrite; + imodel.close(); + }; + + const getEnv = async () => { + const bridgeJobDef = new BridgeJobDefArgs(); + const testSourcePath = path.join(KnownTestLocations.assetsDir, "test.db"); + bridgeJobDef.sourcePath = testSourcePath; + bridgeJobDef.bridgeModule = path.join(__dirname, "..\\..\\COBieBridge.js"); + const serverArgs = new ServerArgs(); + serverArgs.contextId = testProjectId; + serverArgs.iModelId = sampleIModel.id; + serverArgs.getToken = async (): Promise => { + return requestContext.accessToken; + }; + return { testSourcePath, bridgeJobDef, serverArgs }; + }; + + it("should create an iModel", async () => { + const { testSourcePath, bridgeJobDef, serverArgs } = await getEnv(); + const sourcePath = path.join(KnownTestLocations.assetsDir, "intermediary_v1.db"); + IModelJsFs.copySync(sourcePath, testSourcePath, { overwrite: true }); + await runConnector(bridgeJobDef, serverArgs, false, false); + }); + + it("should not update an unchanged iModel", async () => { + const { testSourcePath, bridgeJobDef, serverArgs } = await getEnv(); + const sourcePath = path.join(KnownTestLocations.assetsDir, "intermediary_v1.db"); + IModelJsFs.unlinkSync(testSourcePath); + IModelJsFs.copySync(sourcePath, testSourcePath, { overwrite: true }); + await runConnector(bridgeJobDef, serverArgs, false, false); + }); + + /* + it("should update the data in an iModel", async () => { + const { testSourcePath, bridgeJobDef, serverArgs } = await getEnv(); + const sourcePath = path.join(KnownTestLocations.assetsDir, "intermediary_v2.db"); + IModelJsFs.unlinkSync(testSourcePath); + IModelJsFs.copySync(sourcePath, testSourcePath, { overwrite: true }); + await runBridge(bridgeJobDef, serverArgs, true, false); + }); + */ + + it("should update both data and schema of an iModel", async () => { + const { testSourcePath, bridgeJobDef, serverArgs } = await getEnv(); + const sourcePath = path.join(KnownTestLocations.assetsDir, "intermediary_v3.db"); + IModelJsFs.unlinkSync(testSourcePath); + IModelJsFs.copySync(sourcePath, testSourcePath, { overwrite: true }); + await runConnector(bridgeJobDef, serverArgs, true, true); + }); +}); diff --git a/COBie-connector/src/test/logging.config.json b/COBie-connector/src/test/logging.config.json new file mode 100644 index 0000000..2511e25 --- /dev/null +++ b/COBie-connector/src/test/logging.config.json @@ -0,0 +1,17 @@ +{ + "defaultLevel": "None", + "categoryLevels": [ + { + "category": "BeSQLite", + "logLevel": "None" + }, + { + "category": "imodeljs-backend.BriefcaseManager", + "logLevel": "Error" + }, + { + "category": "imodel-bridge.Framework", + "logLevel": "Error" + } + ] +} \ No newline at end of file diff --git a/COBie-connector/src/test/unit/COBieConnector.test.ts b/COBie-connector/src/test/unit/COBieConnector.test.ts new file mode 100644 index 0000000..0b09148 --- /dev/null +++ b/COBie-connector/src/test/unit/COBieConnector.test.ts @@ -0,0 +1,67 @@ +/*--------------------------------------------------------------------------------------------- +* Copyright (c) Bentley Systems, Incorporated. All rights reserved. +* See LICENSE.md in the project root for license terms and full copyright notice. +*--------------------------------------------------------------------------------------------*/ + +import { IModelJsFs, IModelHost, SnapshotDb, Subject, IModelDb } from "@bentley/imodeljs-backend"; +import { AuthorizedClientRequestContext, AccessToken } from "@bentley/itwin-client"; +import { Synchronizer } from "@bentley/imodel-bridge/lib/Synchronizer"; +import * as path from "path"; +import { ConnectorTestUtils } from "../ConnectorTestUtils"; +import { KnownTestLocations } from "../KnownTestLocations"; +import { COBieConnector } from "../../COBieConnector"; +import { BridgeJobDefArgs } from "@bentley/imodel-bridge"; + +describe("COBie Sample Connector Unit Tests", () => { + before(async () => { + // ConnectorTestUtils.setupLogging(); + // ConnectorTestUtils.setupDebugLogLevels(); + // if (!IModelJsFs.existsSync(KnownTestLocations.outputDir)) IModelJsFs.mkdirSync(KnownTestLocations.outputDir); + // await ConnectorTestUtils.startBackend(); + }); + + after(async () => { + // await ConnectorTestUtils.shutdownBackend(); + }); + + beforeEach(async () => { + await IModelHost.startup(); + }); + + afterEach(async () => { + await IModelHost.shutdown(); + }); + + it("Should create empty snapshot and synchronize source data", async () => { + const sourcePath = path.join(KnownTestLocations.assetsDir, "intermediary_v1.db"); + const targetPath = path.join(KnownTestLocations.outputDir, "final.db"); + if (IModelJsFs.existsSync(targetPath)) IModelJsFs.unlinkSync(targetPath); + + const connector = new COBieConnector(); + const targetDb = SnapshotDb.createEmpty(targetPath, { rootSubject: { name: "COBieConnector" }}); + const requestContext = new AuthorizedClientRequestContext(AccessToken.fromTokenString("Bearer test")); + const sync = new Synchronizer(targetDb, false, requestContext); + connector.synchronizer = sync; + + const jobSubject = Subject.create(targetDb, IModelDb.rootSubjectId, `COBieConnector:${sourcePath}`); + jobSubject.insert(); + connector.jobSubject = jobSubject; + + await connector.openSourceData(sourcePath); + await connector.onOpenIModel(); + + await connector.importDomainSchema(requestContext); + await connector.importDynamicSchema(requestContext); + targetDb.saveChanges(); + + await connector.importDefinitions(); + targetDb.saveChanges(); + + await connector.updateExistingData(); + targetDb.saveChanges(); + const bridgeJobDef = new BridgeJobDefArgs(); + bridgeJobDef.sourcePath = sourcePath; + ConnectorTestUtils.verifyIModel(targetDb, bridgeJobDef); + targetDb.close(); + }); +}); diff --git a/COBie-connector/src/test/unit/mocha.opts b/COBie-connector/src/test/unit/mocha.opts new file mode 100644 index 0000000..e56284b --- /dev/null +++ b/COBie-connector/src/test/unit/mocha.opts @@ -0,0 +1,5 @@ +--require ts-node/register +--require jsdom-global/register +--require ignore-styles +--check-leaks +--no-timeouts diff --git a/COBie-connector/tsconfig.json b/COBie-connector/tsconfig.json new file mode 100644 index 0000000..b6bd6fd --- /dev/null +++ b/COBie-connector/tsconfig.json @@ -0,0 +1,12 @@ +{ + "extends": "./node_modules/@bentley/build-tools/tsconfig-base.json", + "compilerOptions": { + "target": "es2017", + "module": "commonjs", + "baseUrl": "./node_modules", + "outDir": "./lib", + "noUnusedParameters": false + }, + "include": ["./src/**/*"], + "exclude": ["lib", "node_modules"] +} diff --git a/COBie-connector/tslint.json b/COBie-connector/tslint.json new file mode 100644 index 0000000..e455047 --- /dev/null +++ b/COBie-connector/tslint.json @@ -0,0 +1,3 @@ +{ + "extends": "@bentley/build-tools/tslint.json" +} \ No newline at end of file diff --git a/COBie-extractor/.gitignore b/COBie-extractor/.gitignore new file mode 100644 index 0000000..807b45e --- /dev/null +++ b/COBie-extractor/.gitignore @@ -0,0 +1,6 @@ +*.db +lib/ +bin/ +include/ +lib64* +share/ diff --git a/COBie-extractor/README.md b/COBie-extractor/README.md new file mode 100644 index 0000000..d81f1e6 --- /dev/null +++ b/COBie-extractor/README.md @@ -0,0 +1,14 @@ +# COBie Extractor + +A Python script that dumps COBie Excel data into an intermediary SQLite database consumed by COBie-connector. + +## Instructions + +1. run "pip install -r requirements.txt" to install dependencies +2. run "make all" to create all intermediary databases +3. Linux / WSL: run "sh transferdb.sh" to move the newly created to COBie-connector folder as the input for COBie connector. Other OS: manually move the intermediary databases to COBie-connector/test/assets/. + +## Allowed Schema Changes + +1. Add New Column +2. Add New Table diff --git a/COBie-extractor/extractor/Makefile b/COBie-extractor/extractor/Makefile new file mode 100644 index 0000000..7c57314 --- /dev/null +++ b/COBie-extractor/extractor/Makefile @@ -0,0 +1,10 @@ +all: + python3 extractor.py input/COBieSampleSheetV1.xlsx output/intermediary_v1.db # create + python3 extractor.py input/COBieSampleSheetV2.xlsx output/intermediary_v2.db # data change + python3 extractor.py input/COBieSampleSheetV3.xlsx output/intermediary_v3.db # schema change (addition) + python3 extractor.py input/COBieSampleSheetV4.xlsx output/intermediary_v4.db # schema change (deletion) + +clean: + rm output/* + + diff --git a/COBie-extractor/extractor/extractor.py b/COBie-extractor/extractor/extractor.py new file mode 100644 index 0000000..ad871fe --- /dev/null +++ b/COBie-extractor/extractor/extractor.py @@ -0,0 +1,85 @@ +import sys +import re +import xlrd +import sqlite3 + +from sqlalchemy.ext.declarative import declarative_base +from sqlalchemy.orm import sessionmaker +from sqlalchemy import create_engine, Column, String, Integer, Table, MetaData + +def clean_value(value): + return str(value).split(':')[1].strip('\'') + +def create_db_base(): + wb = xlrd.open_workbook(INPUT_PATH) + sheets = wb.sheets() + db_base = {"schema": {}, "objects": {}} + exclude = {"Instruction", "PickLists"} + for sheet in sheets: + if sheet.name in exclude: + continue + + header_row = sheet.row(0) + db_base["schema"][sheet.name] = [] + for header in header_row: + db_base["schema"][sheet.name].append(clean_value(header)) + + db_base["objects"][sheet.name] = create_objects(sheet) + + return db_base + + +def create_objects(sheet): + header_row = sheet.row(0) + objects = [] + + for nrow in range(1, sheet.nrows): + row = sheet.row(nrow) + obj = {} + for cell, header in zip(row, header_row): + key = clean_value(header) + value = clean_value(cell) + obj[key] = str(value) + objects.append(obj) + + return objects + +def dump_schema(schema): + for table_name, col_names in schema.items(): + cols = [Column("id", Integer, primary_key=True)] + cols.extend([Column(name.lower(), String) for name in col_names]) + table = Table(table_name, metadata) + for col in cols: + table.append_column(col) + + metadata.create_all(engine) + +def dump_objects(objects): + try: + conn = engine.connect() + for (table_name, table) in metadata.tables.items(): + rows = objects[table_name] + for row in rows: + colstring = ", ".join(row.keys()) + valarr = [val.replace('"', '""') for val in row.values()] + valstring = ", ".join([f'\"{val}\"' for val in valarr]) + stmt = f'insert into {table_name} ({colstring}) values ({valstring})' + conn.execute(stmt) + except err: + print(err) + +if __name__ == '__main__': + + INPUT_PATH = sys.argv[1] + OUTPUT_PATH = sys.argv[2] + + engine = create_engine(f'sqlite:///{OUTPUT_PATH}') + metadata = MetaData() + metadata.reflect(engine) + Session = sessionmaker(bind=engine) + + db_base = create_db_base() + dump_schema(db_base["schema"]) + dump_objects(db_base["objects"]) + + diff --git a/COBie-extractor/extractor/input/COBieSampleSheetV1.xlsx b/COBie-extractor/extractor/input/COBieSampleSheetV1.xlsx new file mode 100644 index 0000000..bbb88ae Binary files /dev/null and b/COBie-extractor/extractor/input/COBieSampleSheetV1.xlsx differ diff --git a/COBie-extractor/extractor/input/COBieSampleSheetV2.xlsx b/COBie-extractor/extractor/input/COBieSampleSheetV2.xlsx new file mode 100644 index 0000000..676353d Binary files /dev/null and b/COBie-extractor/extractor/input/COBieSampleSheetV2.xlsx differ diff --git a/COBie-extractor/extractor/input/COBieSampleSheetV3.xlsx b/COBie-extractor/extractor/input/COBieSampleSheetV3.xlsx new file mode 100644 index 0000000..d02671e Binary files /dev/null and b/COBie-extractor/extractor/input/COBieSampleSheetV3.xlsx differ diff --git a/COBie-extractor/extractor/input/COBieSampleSheetV4.xlsx b/COBie-extractor/extractor/input/COBieSampleSheetV4.xlsx new file mode 100644 index 0000000..8457b80 Binary files /dev/null and b/COBie-extractor/extractor/input/COBieSampleSheetV4.xlsx differ diff --git a/COBie-extractor/extractor/transferdb.sh b/COBie-extractor/extractor/transferdb.sh new file mode 100644 index 0000000..c29d5f5 --- /dev/null +++ b/COBie-extractor/extractor/transferdb.sh @@ -0,0 +1 @@ +cp output/*.db ../../COBie-connector/src/test/assets/ diff --git a/COBie-extractor/pyvenv.cfg b/COBie-extractor/pyvenv.cfg new file mode 100644 index 0000000..4118bb9 --- /dev/null +++ b/COBie-extractor/pyvenv.cfg @@ -0,0 +1,3 @@ +home = /usr/bin +include-system-site-packages = false +version = 3.8.2 diff --git a/COBie-extractor/requirements.txt b/COBie-extractor/requirements.txt new file mode 100644 index 0000000..202b8a8 --- /dev/null +++ b/COBie-extractor/requirements.txt @@ -0,0 +1,2 @@ +SQLAlchemy==1.3.18 +xlrd==1.2.0 diff --git a/README.md b/README.md index cd3d2b5..755a6c7 100644 --- a/README.md +++ b/README.md @@ -1 +1,205 @@ -# imodel-connector-sample \ No newline at end of file +# iTwin Connector + +## What is an iTwin? + +An iTwin is an infrastructure digital twin. + +An iTwin incorporates different types of data repositories – including drawings, specifications, documents, analytical models, photos, reality meshes, IoT feeds, and enterprise resource and enterprise asset management data – into a living digital twin. Please Go [here](http://www.bentley.com/itwin) to get additional information about iTwins and Bentley iTwin Services + +## What is an iModel? +Overview +* Contains digital components assembled from many sources +* Based on open source SQLite relational database format +* Backbone for iTwins + +Detailed Look + +* An iModel is a specialized information container for exchanging data associated with the lifecycle of infrastructure assets. +* iModels are self-describing, geometrically precise, open, portable, and secure. +* iModels were created to facilitate the sharing and distribution of information regardless of the source and format of the information. +* iModels are an essential part of the digital twin world. But a digital twin means a lot more than just an iModel. + +## iTwin Connectors + +iTwin connectors play an important role in enabling a wide range of both Bentley and third-party design applications to contribute to an iTwin. + +Bentley iTwin Services provides connectors to support a wide array of design applications to ensure all of the engineering data can be aggregated into a single digital twin environment inside an iModel. + +A complete list of available connectors can be found in [iTwin Services Community Wiki](https://communities.bentley.com/products/digital-twin-cloud-services/itwin-services/w/synchronization-wiki/47595/supported-applications-and-file-formats) + +Examples of iTwin Connector include: + +![](https://communities.bentley.com/resized-image/__size/650x450/__key/communityserver-wikis-components-files/00-00-00-05-55/Bentley.png) +![](https://communities.bentley.com/resized-image/__size/650x450/__key/communityserver-wikis-components-files/00-00-00-05-55/3rdParty.PNG) + +See [Section on iTwin Synchronization](#ways-to-sync-data-to-an-itwin) for more details on existing connectors. + +However in certain cases, where a specific format is not covered, one can develop a new connector using [iModel.js SDK](https://www.itwinjs.org/) + +![](./imodel_connector_backend.png) + +The imodel-bridge package provided as part of the iModel.js SDK makes it easier to write an iTwin connector backend that brings custom data into a digital twin. To run this environment with the iModel.js library that this package depends on requires JavaScript engine with es2017 support. + +Note: Please keep in mind iModelBridge is sometimes used as a synonym for iTwin Connector since it bridges the gap between input data and a digital twin. + +## How to write an iTwin Connector + +### Connecting data to an iTwin + +![iTwin Connector Steps](./imodel_connector_steps.png) + +There are three main steps that a connector needs to undertake to bring data into a digital twin + +- Extract data from the input source +- Transform and align the data to the digital twin. +- Generate [changesets](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/IModelHub/index.md#the-timeline-of-changes-to-an-imodel) and load data into an iModel. + +**Cobie Connector** +As part of this [repository](https://github.com/imodeljs/imodel-connector-sample), an iTwin connector that synchronizes data from COBie sheets is provided as a sample. The sections below give a high level overview of the various parts that go into creating an iTwin Connector. More information about the implementation of the sample can be found inside COBie-connector [readme](./cOBie-connector/README.md). + +### Extract + +Extraction of data from the input depends on the source format and the availablity of a library capable of understanding it. There are two strategies typically employed for data extraction. + +1. If the extraction library is compatible with TypeScript, write an extraction module and use that to connect the input data with the alignment phase. +2. If a TypeScript binding is not available, extract the data into an intermediary format that can be then ingested by the alignment phase. + +In case of the cobie connector, the cobie-extractor module present in this repository demonstrates how COBie sheet data can be extracted into a sqlite database. + +### Align + +An iModel Connector must carefully transform the source data to BIS-based data in the iModel, and hence each connector is written for a specific data source. + +- Mappings of data are *from* source *into* an iModel. +- Typically, a connector stores enough information about source data to detect the differences in it between job-runs. In this manner tge connector generates *changesets* that are sent to iModelHub. This is the key difference between a connector and a one-time converter. +- Each job generates data in the iModel that is isolated from all other jobs' data. The resulting combined iModel is partitioned at the Subject level of the iModel; each connector job has its own Subject. + +For each iTwin Connector author, there will always be two conflicting goals: + +1. To transform the data in such a way that it appears logical and "correct" to the users of the authoring application. +2. To transform the data in such a way that data from disparate authoring applications appear consistent. + +The appropriate balancing of these two conflicting goals is not an easy task. However, where clear BIS schema types exist, they should always be used. + +**Dynamic Schemas** + +Sometimes BIS domain schemas are not adequate to capture all the data in the authoring application. To avoid losing data, iTwin Connector may dynamically create application-specific schemas whose classes descend from the most appropriate BIS domain classes. + +As an iModel Connector always runs multiple times to keep an iModel synchronized, the schemas created by previous executions limit the schemas that can be used by subsequent executions. To provide consistency and enable concise changesets, the Connector adds to the previously-defined schemas (creating new schema versions). This follows the general schema update strategy defined in [Schema Versioning and Generations](https://github.com/imodeljs/imodeljs/blob/master/docs/bis/intro/schema-versioning-and-generations.md) + +The `DynamicSchema` custom attribute should be set on customer-specific application schemas. This custom attribute can be found in the standard schema `CoreCustomAttributes` and it enables iModelHub to programmatically detect dynamic schemas. Dynamic schemas require special handling since their name and version are typically duplicated between iModels from different work sets. + +**Display Labels** + +Wherever practical, the Elements generated from an iModel Connector should be identifiable through an optimal "Display Label". + +As discussed in [Element Fundamentals](https://github.com/imodeljs/imodeljs/tree/master/docs/bis/intro/element-fundamentals.md), the Display Labels are created through the following logic: + +1. If the UserLabel property is set, it is taken as the Display Label. +2. If the CodeValue is set (and the UserLabel is not set), the CodeValue becomes the Display Label. +3. If neither UserLabel nor CodeValue is set, then a default Display Label is generated from the following data: + - Class Name + - Associated Type's Name (if any) + +iTwin Connector data transformations should be written considering the Display Label logic; UserLabel is the appropriate property for a connector to set to control the Display Label (CodeValue should never be set for anything other than coding purposes). + +*But what value should an iModel connector set UserLabel to?* There are two goals to consider in the generation of UserLabels. Those goals, in priority order, are: + +1. Consistency with source application label usage. +2. Consistency with BIS domain default labeling strategy. + +If the source application data has a property that conceptually matches the BIS UserLabel property, that value should always be transformed to UserLabel. + +## Sync + +Rather than starting over when the source data changes, a connector should be able to detect and convert only the changes. That makes for compact, meaningful changesets, which are added to the iModel's [timeline](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/IModelHub/index.md#the-timeline-of-changes-to-an-imodel). + +To do incremental updates, a connector must do Id mapping and change-detection. An iTwin Connector uses the ExternalSourceAspect class defined in the BIS schema to acheive both. The following sections describe how this is acheived. + +**Provenance** + +Id mapping is a way of looking up the data in the iModel that corresponds to a given piece of source data. If the source data has stable, unique IDs, then Id mapping could be straightforward. + +For COBie data, this sample connector uses a combination of sheet name and unique row name to map data into the iModel. See [updateElementClass](https://github.com/imodeljs/imodel-connector-sample/src/DataAligner.ts) function in the provided sample. When the identifier is provided to the synchronizer, it is stored inside the ExternalSourceAspect class, in the Identifier property. + +Note: If the source data does not have stable, unique IDs, then the connector will have to use some other means of identifying pieces of source data in a stable way. A cryptographic hash of the source data itself can work as a stable Id -- that is, it can be used to identify data that has not changed. + +**Change-detection** + +Change-detection is a way of detecting changes in the source data. + +If the source data is timestamped in some way, then the change-detection logic should be easy. The connector just has to save the highest timestamp at the end of the conversion and then look for source data with later timestamps the next time it runs. + +However, in case of COBIE sheet data timestamps are not available. So in cases like this the connector will have to use some other means of recording and then comparing the state of the source data from run to run. If conversion is cheap, then the source data can be converted again and the results compared to the previous results, as stored in the iModel. In the COBIE case, a cryptographic hash of the source data is used to represent the source data. The hash is stored inside the external source aspect by the synchronizer. + +The change-detection algorithm implemented is + +- For each source data item: + - add source item's Id to the *source_items_seen* set + - Look in the mappings for the corresponding data in the iModel (element, aspect, model) + - If found, + - Detect if the source item's current data has changed. If so, + - Convert the source item to BIS data. + - Update the corresponding data in the iModel + - Else, + - Convert the source data to BIS data + - Insert the new data into the iModel + - Add the source data item's Id to the mappings + +Infer deletions: + +- For each source data item Id previously converted + - if item Id is not in *source_items_seen* + - Find the the corresponding data in the iModel + - Delete the data in the iModel + - Remove the the source data item's Id from the mappings + +In case of cobie connector, the above algorithm implemented inside the [align method of DataAligner](https://github.com/imodeljs/itwin-connector-sample/src/DataAligner.ts) + +## Execution Sequence +The ultimate purpose of a connector is to synchronize an iModel with the data in one or more source documents. That involves not only converting data but also authroization, communicating with an iModel server, and concurrency control. iModel.js defines a framework in which the connector itself can focus on the tasks of extraction, alignment, and change-detection. The other tasks are handled by classes provided by iModel.js. The framework is implemented by the BridgeRunner class. A BridgeRunner conducts the overall synchronization process. It loads and calls functions on a connector at the appropriate points in the sequence. The process may be summarized as follows: + +- BridgeRunner: [Opens a local briefcase copy](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/IModelDb.md) of the iModel that is to be updated. +- Import or Update Schema + - Connector: Possibly [import an appropriate BIS schema into the briefcase](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/SchemasAndElementsInTypeScript.md#importing-the-schema) or upgrade an existing schema. + - BridgeRunner: [Push](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/IModelDbReadwrite.md#pushing-changes-to-imodelhub) the results to the iModelServer. +- Convert Changed Data + - Connector: + - Opens to the data source. + - Detect changes to the source data. + - [Transform](#data-alignment) the new or changed source data into the target BIS schema. + - Write the resulting BIS data to the local briefcase. + - Remove BIS data corresponding to deleted source data. + - BridgeRunner: Obtain required [Locks and Codes](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/ConcurrencyControl.md) from the iModel server and/or code server. +- BridgeRunner: [Push](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/IModelDbReadwrite.md#pushing-changes-to-imodelhub) changes to the iModel server. + + +## Ways to sync data to an iTwin + +[The iTwin Synchronizer portal](https://communities.bentley.com/products/digital-twin-cloud-services/itwin-services/w/synchronization-wiki/47606/itwin-synchronizer-portal) and [iTwin Sychronizer client](https://communities.bentley.com/products/digital-twin-cloud-services/itwin-services/w/synchronization-wiki/47597/itwin-synchronizer-client) provides synchronization mechanism to bring data into an iTwin through a connector + +The following are the various steps involved in that workflow. +![iTwin workflow](https://communities.bentley.com/resized-image/__size/650x340/__key/communityserver-wikis-components-files/00-00-00-05-55/pastedimage1591602805184v1.png) + +More on synchronization using connectors could be found [here](https://communities.bentley.com/products/digital-twin-cloud-services/itwin-services/w/synchronization-wiki/47596/ways-to-sync-your-data-to-an-itwin) + +## More information + +For more indepth information please see: + +- [Importing a schema and bootstrapping definitions](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/SchemasAndElementsInTypeScript.md#importing-the-schema) +- [AccessToken](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/common/AccessToken.md) +- [BriefcaseManager.create]($backend) +- [BriefcaseDb.open]($backend) +- [IModelDb.saveChanges]($backend) +- [BriefcaseDb.pullAndMergeChanges]($backend) +- [BriefcaseDb.pushChanges]($backend) +- [ConcurrencyControl](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/ConcurrencyControl.md) +- [DefinitionModel.insert]($backend) +- [PhysicalModel.insert]($backend) +- [Insert a Subject element](./backend/CreateElements.md#Subject) +- [Insert a ModelSelector element](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/CreateElements.md#ModelSelector) +- [Insert a CategorySelector element](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/CreateElements.md#CategorySelector) +- [Insert a DisplayStyle3d element](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/CreateElements.md#DisplayStyle3d) +- [Insert a OrthographicViewDefinition element](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/backend/CreateElements.md#OrthographicViewDefinition) +- [Logging](https://github.com/imodeljs/imodeljs/tree/master/docs/learning/common/Logging.md) diff --git a/align.png b/align.png new file mode 100644 index 0000000..de1c705 Binary files /dev/null and b/align.png differ diff --git a/extract.png b/extract.png new file mode 100644 index 0000000..53e6617 Binary files /dev/null and b/extract.png differ diff --git a/imodel_connector_backend.png b/imodel_connector_backend.png new file mode 100644 index 0000000..5d63767 Binary files /dev/null and b/imodel_connector_backend.png differ diff --git a/imodel_connector_steps.png b/imodel_connector_steps.png new file mode 100644 index 0000000..6a52cdf Binary files /dev/null and b/imodel_connector_steps.png differ diff --git a/sync.png b/sync.png new file mode 100644 index 0000000..8c1721a Binary files /dev/null and b/sync.png differ