Add support for custom encode/decode logic

[stalniy]

Why

For example, in CosmosSDK DecCoin amount uses custom type implementation (i.e., (gogoproto.customtype) = "Dec"). This custom type changes how field is pre-processed on wire level. Basically instead of encoding it as just a string they preprocess it, convert from decimal string to bigint string and then encode as a string on protobuf level. On decode, operation is reversed

What

I need to customize logic of toBinary and fromBinary functions to support custom encode/decode logic for some fields would be nice to have this out of the box or have possibility to implement this. This would allow me to reuse existing libraries like @connectrpc/connect to send grpc requests to my server.

Alternatives

Potentially I can implement a wrapper function on top of toBinary and fromBinary but then this will require me to change every dependent code and basically loose possibility to re-use existing open source libraries that rely on the same functions

Possible implementation

protogen-es plugin generate base64 encoded AST for proto messages which can be enhanced with special field (customType). This customType implementation can be provided in runtime for example like this:

import { BinaryWriter, BinaryReader } from "@bufbuild/protobuf"

BinaryWriter.registerCustomType('Dec', (writer, field, msg) =>  ...);
BinaryReader.registerCustomType('Dec', (writer, field, msg) =>  ...);

[timostamm]

Hey Serhii, do these custom types really change how fields are encoded on the wire?

string amount = 2 [
  (cosmos_proto.scalar) = "cosmos.Dec", 
  (gogoproto.customtype) = "Dec", 
  (gogoproto.nullable) = false
];

I expect that gogoproto generated code will represent the value as a Dec, but I'd expect the field to be serialized as a Protobuf string in the binary and the ProtoJSON formats. Otherwise, it wouldn't be possible to parse a message from the wire with any Protobuf implementation except gogoproto, which seems like a really big drawback.

To support something similar in Protobuf-ES, the generated type for the field amount would need to change to Dec:

+ import type { Dec } from "??";

  /**
   * @generated from field: string amount = 2;
   */
- amount: string;
+ amount: Dec;

So it's also necessary to support the custom type option in protoc-gen-es. Because the option doesn't provide any information where to import the type from, a plugin option is necessary to tell protoc-gen-es where to import Dec from.

In the runtime, it's already possible to get the custom type option, but something like BinaryWriter.registerCustomType wouldn't be sufficient. We also need to know how to initialize a Dec field, how to compare Dec values, and we need to know what a zero Dec value is. Implementing this is a very significant amount of work, and even for the most simple cases, users would need matching configuration at several places for it to work. It's also likely to introduce costs in perf, bundle size, and / or complexity. I'm afraid it seems unlikely that we can support this feature.

Provided that my assumption is correct that customtype = "Dec" doesn't change what's encoded on the wire, wouldn't it be possible to use something similar to math.NewDecFromString? After parsing a message from binary or JSON, you create a decimal object from the string value, and convert it back to a string before setting the new field value.

[stalniy]

You are right, the actual representation on a wire doesn't change. It's still a string but this string is pre-processed (i.e., formatted) in a special way. Basically Dec type just formats the number from decimal to bigInt and on decode it does reverse operation. So, this doesn't require any changes in types or generated code. I assume that the only change is needed in toBinary, fromBinary functions, to run this custom preprocessing before encoding this string to proto binary string format.

Basically, I'd say that you should not implement support for cosmos sdk Dec type but just implement extension point that would allow app or library developers to add this support without hassle.

I can't just do this in runtime manually, because I'm working on data level. And this DecCoin type may be anywhere in data structure, this means I will need to manually track this, update when proto files are updated manually as well.

So, maybe the solution can be much simpler like adding interceptors on value when it's encoded/decoded. Basically possibility to provide custom transformation functions for every type:

BinaryWriter.addIntercepter((field: DescField, value: unknown) => {
   if (field.options.some(....)) { //check whether it's a Dec field
     return decToBigInt(value);
   }
   return value;
});

and similar thing for BinaryReader

[stalniy]

Basically I need to do something similar as currently done for BigInt implementation. It can be either string or bigint type.

From another look I see this logic can be implemented on ReflectMessageImpl class which already has similar logic for bigint in JS.

UPD: it has both set and get methods which can be intercepted. The logic related to bigint can be extracted as a plugin as well. Very likely WKT and wrapper types also could be converted to pluggable custom types

[stalniy]

Well, I personally don’t like the idea of such transformations. And I think it should not spread over different protobuf implementations. Instead it needs to be properly done in cosmos-sdk. There is even a bug for this (2years old).

Though would be good to have at least extension point to implement it on my side.

[timostamm]

Yes, the reflection layer can potentially adapt to different representations of a message. Support for this hasn't been fully fleshed out though, and it will likely require significant effort to fully support different use cases.

But circling back to #1074 (comment): The generated type for a modified field needs to change as well. This isn't supported by protoc-gen-es, and I don't see this working out with decent DX without a bespoke separate plugin.

I wonder how Cosmos solves this problem for Go? Do they maintain their own generator and marshaller?

[stalniy]

I don’t know the exact implementation in Golang. My understanding (which may not correlate with reality) is that they do it in the way you described in your comment, with custom type representation. And yes, there is a separate generator. Probably it’s specifically designed for blockchain usage.

In JS, there is telescope project which supports DecCoin type and customtype option. But in telescope, they do conversion inside decode & encode methods of that specific type. To the end client developer it’s the same type as in protobuf. It’s just a string

[timostamm]

It looks like cosmos is using a fork of gogoprotobuf (deprecated, see gogo/protobuf#691), which is a fork of golang/protobuf.

Telescope appears to use a fork of ts-proto, with custom generated code that parses the Protobuf string field DecCoin .amount with Decimal from @interchainjs/math: https://github.com/hyperweb-io/telescope/blob/5432b0a227cf768d747fc3c7b4f02b657930f607/__fixtures__/v-next/outputicjs/cosmos/base/v1beta1/coin.ts#L203

[timostamm]

Custom field representations are a great idea. But let's take a look at the field option:

string amount = 2 [
  (cosmos_proto.scalar) = "cosmos.Dec", 
  (gogoproto.customtype) = "Dec", 
  (gogoproto.nullable) = false
];

The code generator needs to know what type to generate for this field (and where to import the type from), and the runtime needs to know how to convert it (the generator needs provide a piece of code to the runtime).

This means the option needs to provide more information specifically for TypeScript, for example:

string amount = 2 [
  (protobuf_es).type = {
     name: "Dec",
     adapter: "DecAdapter",
     from: "somepackage"
  }, 
];

This will still lead to unexpected behavior though. For example, you can provide a partial message to create(), but Dec must be passed whole. I'm certain that there are other similar edge cases that are difficult to solve. Maybe we can support such a feature at some point, but I'm not sure that it is possible with respect to DX and maintainability.

If my understanding from #1074 (comment) is correct, my recommendation is to use regular generated code from protobuf-es, then use Decimal from @interchainjs/math to parse the string into the desired representation.

[stalniy]

yes, telescope parses it with Decimal but then it converts it back to string :) That's why to the client developer it's still a string. And there is no need for generator to provide something custom. The only thing which needs to be changed is a runtime a possibility to hook into encode/decode logic for scalars or all types.

Having some hook on reflection level would allow me to write this same logic without introducing any issues.

Anyway, I plan to take a look into this more deeply in the next few days probably I will understand the complexity you are trying to avoid but from the bird flight point of view it seems not hard.

[timostamm]

telescope parses it with Decimal but then it converts it back to string

I'm confused 😆

Is this just for validation, or does it modify the string?

[stalniy]

It modifies the string. So, on clients it’s a string that represents double BUT when it’s sent over wire it’s a string that represents BigInt with 18 numbers.

So this is just a trick over wire, and I guess it’s specifically for GoLang 😄 so, yeah it’s super awkward