Home
Welcome to the pcf wiki!
- A High-Level Compiler for Digital Twins
- Architecture
- Declarative Synchronization
- Minimized Runtime Error and Testing
- Single Source of Truth
- Empowering Lossless Synchronization
- Intermediate Representation
- What is the difference between a Connector and a Loader?
| PCF | Programming Language |
|---|---|
| PCF-based Connector | Source File |
| Nodes | Lexical Tokens |
| RepoTree | Abstract Syntax Tree |
| IRModel | Intermediate Representation |
| PCF Core | Compiler Program |
| Synchronized iModel | Executable |
PCF allows you to represent your external source data in an iModel using two constructs: DMOs and Nodes (see definitions below). It makes sure that every EC Entity corresponding to your source data are correctly inserted, updated, and deleted in your iModel. With PCF, you also gain the power to organize the hierarchy of your digital twin (iModel) so your end iTwin.js applications knows what to expect from your iModels.
PCF minimizes runtime errors in its connectors by maximizing the functionalities offered by TypeScript so that most errors can be caught at compile time and runtime before a connector job kicks off. As previously mentioned, PCF follows the declarative paradigm so a connector does not contain any custom logics rather it contains only object definitions, which can be linted strictly.
Given that object definitions (DMOs and Nodes) are the main inputs to your connector, so long as their definitions are correct, each synchronization job is guaranteed to succeed with PCF.
Functionalities such as code-completion and code-refactoring available in most modern IDE's (e.g Visual Studio Code) will help you to write the correct definitions for them. Since most runtime errors are avoided at compile time and the source code of a connector only contains a set of object definitions, unit/integration tests are no longer needed for connectors, making them much easier to maintain. (except for geometry transformations)
SSOT = Single Source of Truth
As both of your connector and source data evolve, it is often that two types of changes will need to be made:
- mapping changes between the source schema and EC schema.
- iModel hierarchy changes of the EC Entities created from the source data.
As we are constantly dealing with changes, it is important to have an easy way to precisely capture these changes and inform downstream iTwin.js applications about them. PCF solves this problem by making DMO the SSOT of the mappings between source schema and EC schema and making Node the SSOT of the hierarchy of mapped EC Entities within the source discipline.
Sometimes you may want to define and update your own schema (called "dynamic schema" in EC terms) to better represent your source schema in the EC world. With PCF, you no longer need to hand-write a xml EC Schema, it is auto-generated and imported into your iModel if you have defined your own EC classes in DMOs. If dynamic schema is not known before runtime and it's based on the external data, you can programatically generate DMOs.
The term “lossless” here means all the structures and properties of the source data are maintained, not purely bytes. Missing a few pieces of information is ok because we can always add them later, but missing constraints could corrupt things.
When one converts a data format to another, it is likely that not all the properties of the source data are maintained in the target format (considered as a lossy transformation in PCF). For example, referential integrity gets lost if a synchronization job missed a database relationship. This could cause a disastrous consequence by allowing the target data to be modified without the relationship constraint
This mistake cannot be avoided at the framework level, the person who's responsible for the mappings between the source and target format must always be extreme cautious in defining mappings. However, the way mappings are presented through DMO in PCF significantly makes the job of this person easier and allows someone without much programming expertise to inspect the mappings.
An IR Model is an intermediate representation of your source data. It is a simple virtual Entity-Relationship store generated by a Loader. A Loader is a simple class that handles the conversion from a source data format to IR Model. In PCF, Loader is the only construct that interacts with the outside world. There could be a Loader for any data format, currently supported ones are JSON Loader, XLSX (Excel) Loader, and SQLite Loader. So long as a data format has a Loader, it can be imported into an iModel through PCF.
We must not always assume that the source data are normalized or well-modeled. Having an IR Model safeguards against dirty source data, thus maintaining the data quality in your iModel. IR Model forces every external class to have a primary key and value so that one can always use information stored in the external source to query an iModel and be confident that only a single EC instance gets returned. This is just one kind of optimization. Other kinds of optimizations could also be implemented on IR Model such as dynamically inferring relationships in your data if they're not provided.
One reason that a highly configurable connector is needed is that it's not possible to have a single connector for a single data format. Data formats could have different schemas that cannot be understood by a single connector. For example, the databases of different users have completely different schemas and importing those data into iModels will always require configurations.
Without PCF: data source A == Connector for A => iModel
With PCF: data source A == Loader for A => IR Model == PConnector => iModel
This new architecture separates the concern of accessing an external data from the connector so that we can reuse a connector to populate iModels. It's much easier to write a Loader than to write a full-blown Connector as you don't have to deal with the intricacies in the iTwin.js world. It requires zero expertise in iTwin.js to write a Loader. As long as there's a Loader for a data format, its data can be imported into an iModel through PCF.