User metadata annotations

[mattdailis]

• Tickets addressed: Pull Units from mission model annotations and store them in DB #1034
• Review: By commit
• Merge strategy: Merge (no squash)

Description

This is our second attempt. Our first attempt was #1055.

Short term goal: Allow users to provide unit information for parameters, computed attributes, and resources in their mission model, such that this information can be displayed in the UI and extracted via the API.

Broader goal: Set up a pattern that missions, and Aerie developers, can use to add more kinds of metadata in the future, e.g. Range.

There are two (mostly) independent problems to solve here:

1. How do mission modelers provide unit information, and define other kinds of custom metadata?
2. How is this custom metadata serialized and represented to downstream tools?

The following sections delve into these problems:

How to define custom metadata

We knew from the outset that we would eventually want to be able to annotate deeply nested aspects of a value schema with metadata. E.g. a rover's pose could be characterized by a compound piece of data:

pose:
  position:
    latitude: degrees
    longitude: degrees
    elevation: kilometers
  orientation: quaternion
    q1: unit vector component
    q2: unit vector component
    q3: unit vector component
    q4: scalar

We had originally descoped this capability, but then pivoted (i.e. we increased scope) when the path to implementing it became clear.

The two options to select between were Javadoc tags, and annotations. I will describe both approaches, and explain why we have settled on annotations.

Surface Syntax

This Javadoc tags approach amounted to parsing javadoc comments as strings, and extracting portions of them based on a set of tags, such as @aerie.unit.

It lended itself quite well to class style parameters, since it could be part of the javadoc comment attached to the parameter itself:

Javadoc tags approach

Example of using javadoc to add unit information to a parameter:

@ActivityType("RunHeater")
public class RunHeater {
  /**
   * @aerie.unit seconds
   */
  @Parameter
  public long duration;

  /**
   * @aerie.unit kWh
   */
  @Parameter
  public int energyConsumptionRate;
}

Example of using javadoc to add unit information to a computed attributes record:

  /**
   *
   * @aerie.computedAttribute durationInSeconds
   * @aerie.unit seconds
   * @aerie.computedAttribute energyConsumptionRate
   * @aerie.unit kWh
   */
  @AutoValueMapper.Record
  record ComputedAttributes(
    long durationInSeconds,
    int energyConsumptionRate
  ) {}

Example of using @aerie.resourceName to assign a unit to all resources that match a regular expression:

/*
 * @aerie.resourceName \/prefix/.*
 * @aerie.unit F resources
 */

A limitation that we discovered of parsing the javadocs is that our implementation would only parse javadocs in a known set of locations. This works fine for parameters and computed attributes, but it breaks down for resources, which in practice are defined in submodels.

The second approach we pursued was to introduce a @Unit annotation, which could be attached to the type of a parameter or computed attribute:

Annotation approach

Example of using an annotation to add unit information to a parameter:

@ActivityType("RunHeater")
public class RunHeater {
  @Parameter
  public @Unit("seconds") long duration;

  @Parameter
  public @Unit("kWh") int energyConsumptionRate;
}

Example of using an annotation to add unit information to a computed attributes record:

@AutoValueMapper.Record
record ComputedAttributes(
  @Unit("seconds") long durationInSeconds,
  @Unit("kWh") int energyConsumptionRate
) {}

What about resources? Those are defined through a different process from parameters and computed attributes. Resources get registered at runtime, in the mission model constructor, rather than at compile time like the other entities. The approach with javadocs would record a regular expression and do name matching at runtime. The current approach instead expects a mission model to provide the units to the registrar as runtime values. Two convenience methods have been defined:

discreteResource(registrar, "/peel", this.peel, new DoubleValueMapper(), "kg");
realResource(registrar, "/fruit", this.fruit, "bananas");

This approach allows us to avoid any custom handling of units in the annotation processor, which means that it will generalize to any future metadata we would like to include. If it turns out that the regular expressions would be a significant quality of life improvement, we can consider that a separate feature.

How is custom metadata serialized

In Java, there is now a new kind of ValueSchema: a MetaSchema, which carries a Map<String, SerializedValue>, as well as a nested ValueSchema. The interpretation is that all of the metadata in the MetaSchema applies to the nested ValueSchema.

The withMeta method, used to construct a MetaSchema, checks to see if the inner value is itself a MetaSchema, and if so merges the two, overlaying the outer one on top of the inner one.

When serializing to json, this Map<String, SerializedValue> is attached to the nested object, like so:

MetaSchema(Map.of("unit", "m"), IntSchema()) becomes

{
  "type": "int",
  "metadata": {
    "unit": "m"
  }
}

Future-proofing

One of the goals of this PR is to lay the groundwork for adding additional metadata to value schemas.

Part of this PR brings annotations into the Resolver and TypePattern machinery. This allows us to generate value mappers for annotated types.

This PR allows users to define their own annotations, and as long as a value mapper is provided for that annotation, Aerie will be able to serialize it and include it in the value schema.

One generalization that this PR leaves undone is to allow for using value mapper methods to define value mappers for annotated types. The nuance here is that, in this PR, we have users use the @WithMetadata(Unit.class) annotation to declare an annotation of interest, and we use that to generate a type rule, rather than the users writing the type rule as the return type from a static method, the way that other value mappers are defined. This is due to a limitation of type use annotations that we may be able to work around in the future.

Verification

The Banananation example model was updated to include some unit information for parameters, computed attributes, and resources.

Things to test manually:

• Do the value schemas look correct in the database?
• Do scheduling and constraint checking still work with banananation? (I.e. do they correctly parse the new value schemas during typescript code generation)

The second bullet may be handled by e2e tests

Documentation

• Using Added docs for the @unit tag aerie-docs#71 as an example, write a new page about metadata.

Future work

• Incorporate unit information in typescript code generation
• Provide a convenience @AutoValueMapper.Annotation to make adding your own metadata annotations less verbose
• Find a workaround that allows type use annotations to be used for value mapper methods
• // int[] repeated(); // TODO add support for arrays in annotations for which we generate value mappers

[mattdailis]

Fixed a broken test - valueSchemaIsNumeric was returning false for the /fruit resource, because it's wrapped in a meta schema. I added the case below:

aerie/constraints/src/main/java/gov/nasa/jpl/aerie/constraints/TypescriptCodeGenerationService.java

Lines 392 to 397 in 416b628

	private static boolean valueSchemaIsNumeric(final ValueSchema valueSchema) {
	return valueSchema.equals(ValueSchema.INT)
	|| valueSchema.equals(ValueSchema.REAL)
	|| valueSchema.equals(LINEAR_RESOURCE_SCHEMA)
	|| valueSchema.asMeta().map($ -> valueSchemaIsNumeric($.target())).orElse(false);
	}

[mattdailis]

New e2e tests should be passing now. Updates since the walkthrough:

• There is now an @AutoValueMapper.Annotation that can be used to generate a value mapper for a given annotation. It will generate a struct with the same keys as the annotation
  • We still need our own definition in BasicValueMappers, because our annotation processor doesn't apply to contrib. Perhaps a future work question of how we leverage our annotation processor in library code
• I looked into separating TypePattern from ConcreteType, but ran into difficulties deciding which of the two should be returned from substitute. I think keeping them together is prudent
• ParameterTest activity has been updated to use different units for the key and value of doubleMap, to allow us to ascertain that the annotations are applied in the right places
• I added :examples:banananation:assemble as a dependency to the e2e-tests, so that it always uploads the most up-to-date version

[mattdailis]

For ease of review, I've pushed all changes as fixup commits (and there's one revert commit) - no changes have been made to earlier commits.

I looked into whether .box() on arrays should return this, or new ArrayPattern(this.element.box()), and determined that actually List<int[]> is a valid type 😲. I also took a look at TypeName's box() definition, and it returns this in all cases except for primitives.