Investigate proposed alternatives to the existing message generation pipeline.

[maspe36]

In this months WG meeting, I discussed some painpoints with our current message generation pipeline, and some potential alternatives or changes.

The Problem

Rust does not have a stable ABI, and wants to compile all of the source code and dependencies source code, when compiling a crate. This is fundamentally what pushes us to have such a complex message generation pipeline.

Potential Solutions

I have some ideas that may help alleviate some of this complexity. These ideas are not listed in any order of preference.

Automatically upload ROS 2 messages to crates.io as semver'd packages.

Cargo bakes in semver for crates, but thankfully all ROS 2 message packages have semver versions. Perhaps we could automatically generate and upload crates when new versions of ROS 2 message packages have been published. If we are able to have letters before numbers, then we could in theory have versions like version = "jazzy.*", in our Cargo.toml files. This would alleviate some complexity when it came to upstream message packages, but doesn't change the user story for custom message packages.

Leverage a crate for "plugins"

The abi_stable seems to provide some Rust-to-Rust FFI plugin system. If memory serves this is similar to what we do for rclcpp messages where building the message package, generates a shared library, that downstream packages then dynamically link to.

Restricting a ros2 workspace to have a single cargo workspace

The additional constraints that a cargo workspace provides would potentially allow us to do things like generating message source code on demand from a build script. Of course, this reduces the flexibility of how an end user could organize their source code.

Collecting all upstream ROS 2 messages in a single crate gated with cargo features

This is not my idea, this is from @esteve. Tracking issue for this idea at #394.

---

None of these ideas are perfect, and we have still only done a very high level investigation into all of them. While this investigation is on going, we are in parallel working to improve documentation for our existing message generation pipeline #401.

[maspe36]

Automatically upload ROS 2 messages to crates.io as semver'd packages.

I took a look at the semver crate, which is the crate that implement Cargo's interpretation of Semantic Versioning. So the original idea I had proposed is unfortunately not supported.

This sample code will panic

mod tests {
    use semver::Version;

    #[test]
    fn ros_distro_ver() {
        // The rolling branch on common_interfaces uses this specific tag.
        // It may be helpful if users could specify `version = "rolling.*"`
        // This way, if we had the generated ros2_rust messages up in crates.io,
        // end users can just worry about distros. This is similar to what
        // ROS users do today. Track a distro, and occasionally update it with
        // `apt upgrade`.
        let rolling = Version::parse("rolling.5.4.1").unwrap();
    }
}

There is also the added complication of the generated code actually being a combination of two version numbers. One from rosidl_generator_rs and rosidl_runtime_rs, and the other being from upstream ROS, the version numbers associated with the actual .idl files.

I treat rosidl_generator_rs and rosidl_runtime_rs as having the same version because I do not think it is realistic at the moment to have code generated without also knowing the runtime it'll be used with.

We could represent this combination of versions with the semver crate,

let rolling = Version::parse("5.4.1+rust.0.4.1").unwrap();

but now we cannot upload another crate with version 5.4.1+rust.0.5.0.

There are other crates out there that deal with this kind of dual version dilema. For example, take a look at libgit2-sys. This crate maintains many different versions of libgit2 as build metadata.

Crate Version	libgit2 Version
0.17.0	1.8.1
0.16.2	1.7.2
0.15.2	1.6.4

For this reason, I don't think this approach alone, will yield the results we want, unless we are willing to introduce a custom semver schema for ROS distros. This would have to be something like a major version 0 semantically resolves to the rolling distro, etc.

# Gives us the latest rolling (and latest rosidl_*_rs bindings...)
[dependencies.std_msgs]
version = "0.*"

# Gives us the latest jazzy (and latest rosidl_*_rs bindings...)
[dependencies.std_msgs]
version = "1.*"

❌ This is an unintuitive approach for us, as we'd need users to be more aware of the rosidl_*_rs version or our own mapping from a major version to a distro, than the distro name, or the upstream version. ❌

---

There should still be more discussion around ways to improve the message generation story. There are more avenues to explore. I will continue to deep dive into this

[maspe36]

It is almost a year from when I first opened this issue and I think that calls for some updates. Listing out the potential solutions remaining for investigation with where we ended up on them.

1. Automatically upload ROS 2 messages to crates.io as semver'd packages. ❌

See my message above

2. Leverage a crate for "plugins" 🤷

If we go this route, significant changes would be required for rosidl_runtime_rs. The abi_stable crate, requires us to use their primitive types to facilitate Rust-to-Rust FFI. I haven't done much here besides reading the docs.

I am shying away from this approach as I'd really like to continue leaning into Rust's compiler diagnostics which we would lose a part of if we started dynamically linking to message crates.

3. Restricting a ros2 workspace to have a single cargo workspace ❌

We are close to early cargo workspace support in colcon (colcon/colcon-cargo#59 🎉), however we have decided to support any number of cargo workspaces within a single colcon workspace. Ultimately, this won't improve the end user experience for message crates.

4. Collecting all upstream ROS 2 messages in a single crate gated with cargo features ❌

This was originally @esteve's idea (#394). It does not seem like it will work (as is) because a dependency needs to explicitly declare its features, so a downstream can't pass arbitrary features to a dependency. Effectively making a single crate with all messages enabled by features infeasible.

---

With these options out, I've taken a few pieces from @esteve's idea and put together a full example using what I call "shim crates". These are simply crates that exist to try to find the generated message crates from rosidl_generator_rs and include!() them, and re-export any symbols in the shim crate's lib.rs.

Full discussion is here with a link to the example
#471
https://github.com/maspe36/ros2_rust_shim_message_example/tree/main

At this moment, I believe these to be an acceptable option, but there is still community alignment required.