Scalability Evaluation Benchmark description

Back to Scalability Evaluation main page

There are two benchmark scenarios defined in the performance test:

• Simple run scenario
• Run with modification scenario

The scenarios consist of benchmark phases where every phase is a distinct part of the model compilation process. The performance metrics (execution time and memory) are measured for each phase during the test. The phases used by the scenarios are the following:

• Load and scale model
• Initialize xUML-RT transformation
• Initialize xUML-RT to CPP QRT transformation
• Execute xUML-RT to CPP component structure transformation
• Execute rALF action code compile
• Execute CPP code generation
• Execute makefile generation
• Execute runtime mapping
• Execute file synchronisation
• Initialize Change monitor
• Modify UML model

Simple run scenario

The Simple run scenario executes and measures the performance of the EMDW-Model Compiler on a single UML model. No changes (other than scaling) are made to the model during the test, and the xUML-RT change monitor is not initialized.

The simple run scenarion consists of the following phases:

• Load and scale model
• Initialize xUML-RT transformation
• Initialize xUML-RT to CPP QRT transformation
• Execute xUML-RT to CPP component structure transformation
• Execute rALF action code compile
• Execute CPP code generation
• Execute makefile generation
• Execute runtime mapping
• Execute file synchronisation

Run with modification scenario

The Run with modification scenario executes the same phases as the Simple run scenario, then modifies some model elements and runs the delta codegeneration using the xUML-rt change monitor. This scenario can be used to measure the performance of the delta-transformation, where only a part of the model is recompiled.

Phases

• Load and scale model
• Initialize xUML-RT transformation
• Initialize xUML-RT to CPP QRT transformation
• Execute xUML-RT to CPP component structure transformation
• Execute rALF action code compile
• Execute CPP code generation
• Execute makefile generation
• Execute runtime mapping
• Execute file synchronisation
• Initialize Change monitor
• Modification iteration (maxIteration = modification.times)
  • Modify UML model
  • Execute xUML-RT to CPP component structure transformation
  • Execute rALF action code compile
  • Execute CPP code generation
  • Execute makefile generation
  • Execute file synchronisation