Skip to content
This repository has been archived by the owner on Jan 13, 2023. It is now read-only.

[substrait] Add support for unsigned primitive and literal #41

Open
waynexia opened this issue Dec 13, 2022 · 0 comments
Open

[substrait] Add support for unsigned primitive and literal #41

waynexia opened this issue Dec 13, 2022 · 0 comments

Comments

@waynexia
Copy link
Contributor

Literal Type

One gap is unsigned primitives are not in Substrait's definition. In this discussion they were cataloged as "third party extension defined types". But it's widely used in Arrow and DataFusion. I'd like to discuss how they would be integrated here.

For types we can use type extension, as I do in greptimedb https://github.com/GreptimeTeam/greptimedb/blob/8959dbcef83507ccd76aaaffd2b44cab6426e68f/src/common/substrait/src/types.rs, I occupy the "1" variations for those types (I8, I16, I32, I64) and translate them to the unsigned version. I think we can do it this way here.

        Kind::I8(desc) => substrait_kind!(desc, int8_datatype, uint8_datatype),
        Kind::I16(desc) => substrait_kind!(desc, int16_datatype, uint16_datatype),
        Kind::I32(desc) => substrait_kind!(desc, int32_datatype, uint32_datatype),
        Kind::I64(desc) => substrait_kind!(desc, int64_datatype, uint64_datatype),

For literals we can do it similarly, but one difference is they also ship the data values. I prepare to transmute between the signed and unsigned values. But not sure if this is the best way we can achieve it. As I investigated, duckdb also doesn't cover this part https://github.com/duckdblabs/substrait/blob/a08fe49de78926a3112daddcb64d45a12e43950a/substrait/from_substrait.cpp#L94-L102

  case substrait::Expression_Literal::LiteralTypeCase::kI8:
    dval = Value::TINYINT(slit.i8());
    break;
  case substrait::Expression_Literal::LiteralTypeCase::kI32:
    dval = Value::INTEGER(slit.i32());
    break;
  case substrait::Expression_Literal::LiteralTypeCase::kI64:
    dval = Value::BIGINT(slit.i64());
    break;
@andygrove andygrove changed the title Add support for unsigned primitive and literal [substrait] Add support for unsigned primitive and literal Jan 13, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant