RFC-025: Serialize and Deserialize #8185

jerrykingxyz · 2024-10-22T06:45:34Z

jerrykingxyz
Oct 22, 2024
Maintainer

Feature Name: Rspack support serialize and deserialize
Start Date: 2024/10/21

Summary

Design a serialization and deserialization implementation for rspack

Motivation

Persistent caching requires writing data from memory to storage media such as disk, which requires the data structures in rspack to support serialization and deserialization.

Glossary

Converter: In this page, the term "converter" specifically refers to a series of Traits. By implementing these Traits, one can perform Serialize/Deserialize operations.
Cacheable: Data structures that can be Serialized/Deserialized are called cacheable.
ArchivedStruct: The intermediate structure generated by rkyv, which is between bytes and user structure, can be easily converted to bytes.
StructResolver: The object generated by rkyv to parse the user structure into the corresponding ArchivedStruct.

User Guide

API

The entry points for serialize/deserialize are both pure functions, and their signatures like:

pub fn to_bytes<T, C>(data: &T, context: &C) -> Result<Vec<u8>, SerializeError> {}

pub fn from_bytes<T, C>(bytes: &[u8], context: &C) -> Result<T, DeserializeError> {}

It should be noted that both of these will accept parameters named context, context can be accessed in serialize/deserialize , a typical example is Arc<CompilerOption> this global object, in the serialize operation can be ignored, in the deserialize can be directly cloned from the context.

use rspack_cacheable::{cacheable, with::{AsConverter, As}};

struct CacheContext {
    compiler_options: Arc<CompilerOptions>
}

#[cacheable]
struct FromContext;
impl AsConverter<Arc<CompilerOptions>> for FromContext {
  fn serialize(_data: &Arc<CompilerOptions>, _ctx: &dyn Any) -> Result<Self, SerializeError> {
    // return a default value to ignore it.
    Ok(FromContext)
  }
  fn deserialize(self, ctx: &dyn Any) -> Result<Arc<CompilerOptions>, DeserializeError> {
    let Some(ctx) = ctx.downcast_ref::<CacheContext>() else {
      return Err(DeserializeError::MessageError("context not match"));
    };
    // clone from context
    Ok(ctx.compiler_options.clone())
  }
}

#[cacheable]
struct ModuleFactoryCreateData {
    // use it in the target field
    #[cacheable(with=As<FromContext>)]
    pub options: Arc<CompilerOptions>,
    pub context: Context,
}

let ctx = CacheContext { compiler_options: ... }
// calling to_bytes passes the constructed context into it
to_bytes(..., &CacheContext {})

Data Cacheable

Automatic Implementation

To use the above API, we need to make the target struct cacheable. We can automatically implement the default converter by adding #[cacheable] to it.

use rspack_cacheable::{cacheable, to_bytes, from_bytes};

#[cacheable]
#[derive(Debug, PartialEq, Eq)]
struct Person {
    name: String,
}

let a = Person {
    name: String::from("a")
}
let bytes = to_bytes(&a, &()).unwrap();
let deserialize_a = from_bytes(&bytes, &()).unwrap();
assert_eq!(a, deserialize_a);

For unsupported data struct, you can specify to use a converter by adding #[cacheable(with=...)].

Name	Description	Example
AsPreset	Default support for data struct in third-party crates	use rspack_cacheable::{cacheable, with::AsPreset}; #[cacheable] struct Test { #[cacheable(with=AsPreset)] word: serde_json::Value }
AsCacheable	The default converter	use rspack_cacheable::{cacheable, with::AsCacheable}; #[cacheable] struct Test { // use Default Converter, can ignore macro name: String, #[cacheable(with=AsCacheable)] content: String }
AsInner	Use `AsInnerConverter` Trait for conversion, OnceCell, Arc, etc. have implemented `AsInnerConverter`	use rspack_cacheable::{cacheable, with::{AsInner, AsCacheable}}; #[cacheable] struct Test { #[cacheable(with=AsInner<AsCacheable>)] name: once_cell::sync::OnceCell<String> }
AsMap	Using `AsMapConverter` Trait for conversion, HashMap, IndexMap, DashMap, etc. have implemented `AsMapConverter`	use rspack_cacheable::{cacheable, with::{AsString, AsMap, AsCacheable}}; #[cacheable] struct Test { // can ignore macro when inner struct are cacheable map1: std:collections::HashMap<String, String>, #[cacheable(with=AsMap<AsCacheable, AsString>)] map1: std:collections::HashMap<String, std::path::PathBuf> }
AsRefStr	Using `AsRefStrConverter` Trait for conversion, Cow<'static, str>, Arc , etc. have implemented `AsRefStrConverter`	use rspack_cacheable::{cacheable, with::AsRefStr}; #[cacheable] struct Test { #[cacheable(with=AsRefStr)] name: Cow<'static, str> }
AsString	Using `AsStringConverter` Trait for conversion, PathBuf, etc. have implemented `AsStringConverter`	use rspack_cacheable::{cacheable, with::AsString}; #[cacheable] struct Test { #[cacheable(with=AsString)] path: std::path::PathBuf }
AsTuple2	Convert two elements tuple	use rspack_cacheable::{cacheable, with::{AsString, AsTuple2, AsCacheable}}; #[cacheable] struct Test { // can ignore macro when inner struct are cacheable param1: (String, String), #[cacheable(with=AsTuple2<AsCacheable, AsString>)] param2: (String, std::path::PathBuf) }
AsTuple3	Convert three elements tuple	use rspack_cacheable::{cacheable, with::{AsString, AsTuple3, AsCacheable}}; #[cacheable] struct Test { // can ignore macro when inner struct are cacheable param1: (String, String, String), #[cacheable(with=AsTuple3<AsCacheable, AsString, AsCacheable>)] param2: (String, std::path::PathBuf, String) }
AsVec	Use `AsVecConverter` Trait for conversion, HashSet, Vec, IndexSet, etc. have implemented `AsVecConverter`	use rspack_cacheable::{cacheable, with::{AsString, AsVec, AsCacheable}}; #[cacheable] struct Test { // can ignore macro when inner struct are cacheable in Vec, HashSet set1: Vec<String>, #[cacheable(with=AsVec<AsString>)] set2: Vec<std::path::PathBuf> }
AsOption	Convert the Option type	use rspack_cacheable::{cacheable, with::{AsVec, AsOption, AsString}}; #[cacheable] struct Test { // can ignore macro when inner struct are cacheable param1: Vec<Option<String>>, #[cacheable(with=AsVec<AsOption<AsString>>)] param2: Vec<Option<std::path::PathBuf>> }
Unsupported	Unsupported types. It will return error during Serialize and Deserialize	use rspack_cacheable::{to_bytes, cacheable, with::Unsupported}; #[cacheable] struct Test { #[cacheable(with=Unsupported)] inner: Arc<std::time::SystemTime> } let bytes = to_bytes(Test { inner: Arc::new(std::time::SystemTime::now()) }, &()); assert!(bytes.is_err());
Skip	Skip Serialize, Deserialize use default value	use rspack_cacheable::{to_bytes, from_bytes, cacheable, with::Skip}; #[cacheable] struct Test { #[cacheable(with=Skip)] inner: Option<usize> } let bytes = to_bytes(Test { inner: Some(1) }, &()).unwrap(); let new_test: Test = from_bytes(&bytes, &()).unwrap(); assert!(new_test.inner.is_none());
As	Use `AsConverter` Trait for conversion	use rspack_cacheable::{cacheable, with::{AsVec, AsOption, AsString}, SerializeError, DeserializeError}; use std::path::PathBuf; struct UnCacheableData(PathBuf); #[cacheable] struct CacheableData(String); impl AsConverter<UnCacheableData> for CacheableData { fn serialize(data: &UnCacheableData, _ctx: &dyn Any) -> Result<Self, SerializeError> { Ok(Self(data.0.to_string_lossy().to_string())) } fn deserialize(self, _ctx: &dyn Any) -> Result<UnCacheableData, DeserializeError> { Ok(UnCacheableData(PathBuf::from(&self.0))) } } #[cacheable] struct Data { #[cacheable(with=As<CacheableData>)] inner: UnCacheableData, }
AsOwned	Convert `Cow`	use std::borrow::Cow; use rspack_cacheable::{cacheable, with::AsOwned}; #[cacheable] struct Data { #[cacheable(with=AsOwned)] inner: Cow<String>, }
Inline	Convert the cacheable data reference. Note that since the `Inline` does not implement the deserialize method, it cannot be used with `from_bytes`.	use rspack_cacheable::{cacheable, to_bytes, with::Inline}; #[cacheable] struct Data { block1: String } #[cacheable] struct DataRef<'a> { #[cacheable(with=Inline)] block1: &'a String } let data = Data { block1: "block1".into() } let data_ref = DataRef { block1: &data.block1 } assert_eq!(to_bytes(&data, &()), to_bytes(&data_ref, &()));

In theory, AsPreset needs to support all third-party crates which is used in rspack. Here are some examples of combinations:

#[cacheable]
struct A {
    // Types supported by default, no need to specify converters
    a: u32
    // All of inner data structures are cacheable, no need to specify converters
    b: Vec<HashMap<String, HashSet<(u32, u32, u32)>>>
    // Similar to Map, Vec, but no default support for cacheable, need to specify converter
    #[cacheable(with=AsMap<AsCacheable, AsVec>)]
    c: DashMap<String, DashSet<(u32, u32, u32)>>
    // Inner contains uncacheable structures, need to specify converters
    #[cacheable(with=AsMap<AsCacheable, AsVec<AsTuple3<AsCacheable, AsCacheable, AsPreset>>>)]
    b: Vec<HashMap<String, HashSet<(u32, u32, Atom)>>>
}

Manual Implementation

In addition to using #[cacheable] for automatic implementation, you can also customize the Serialize and Deserialize.

The simplest way to customize is to implement the converter-related Trait. Currently, AsInner, AsRefStr, AsString, etc. all provide relevant Converter Traits, which can be used in conjunction with #[cacheable(with=...)].

use rspack_cacheable::{cacheable, with::{AsRefStr, AsRefStrConverter}};

#[cacheable]
struct Module {
    #[cacheable(with=AsRefStr)]
    path: CustomPath
}

struct CustomPath {
    inner: String
}
impl AsRefStrConverter for CustomPath {
    fn as_str(&self) -> &str {
        &self.inner
    }
    fn from_str(s: &str) -> Self {
        Self { inner: String::from(s) }
    }
}

You can also use #[cacheable(with=...)] to replace the default converter with a specific converter.

use rspack_cacheable::{cacheable, with::{AsRefStr, AsRefStrConverter}};

#[cacheable]
struct Module {
    path: CustomPath
}

// set default converter
#[cacheable(with=AsRefStr)]
struct CustomPath {
    inner: String
}
impl AsRefStrConverter for CustomPath {
    fn as_str(&self) -> &str {
        &self.inner
    }
    fn from_str(s: &str) -> Self {
        Self { inner: String::from(s) }
    }
}

For complex data structures, they can be directly implemented using the underlying Serialize/Deserialize crate. For details, please refer to the implementation of AsPreset, AsRefStr, etc.

Macro API

#[cacheable] supports the following configuration when used in struct and enum:

#[cacheable(with=...)]: Configure the default converter
#[cacheable(hashable)]: Make the generated ArchiveData structure support Hash, which can be used in structures such as HashMap & HashSet.

#[cacheable(hashable)]
#[derive(Debug, Clone, Hash, Eq, PartialEq)]
struct ModuleId(u32);

#[cacheable]
struct A {
    map: HashMap<ModuleId, (u32, u32)>,
    set: HashSet<ModuleId>
}

#[cacheable(crate=...)]: Configure the rspack_cacheable crate path used after macro expansion, default value is rspack_cacheable.

// Cargo.toml
[dependencies]
test = { path="../rspack_cacheable" }

// main.rs
use test::cacheable;

#[cacheable(crate=test)]
struct A {}

#[cacheable(as=...)]: Configure the struct that deserialize converts to, requiring its memory layout to match the current struct.

use rspack_cacheable::{
  cacheable, from_bytes, to_bytes,
  with::{AsTuple2, Inline},
};

#[cacheable]
#[derive(Debug, PartialEq, Eq)]
struct Person {
  name: String,
  content: (String, String),
}

#[cacheable(as=Person)]
struct PersonRef<'a> {
  #[rkyv(with=Inline)]
  name: &'a String,
  #[rkyv(with=AsTuple2<Inline, Inline>)]
  content: (&'a String, &'a String),
}

#[test]
fn as_attr() {
  let a = Person {
    name: "abc".into(),
    content: ("a".into(), "b".into()),
  };
  let a_ref = PersonRef {
    name: &a.name,
    content: (&a.content.0, &a.content.1),
  };
  // use a_ref to serialize
  let bytes = to_bytes(&a_ref, &()).unwrap();
  // deserialize to Person
  let deserialize_a: Person = from_bytes(&bytes, &()).unwrap();
  assert_eq!(a, deserialize_a);
}

#[Cacheable] supports the following configuration when used in fields:

#[cacheable(with=...)]: Configure the converter separately for the current field
#[cacheable(omit_bounds)]: Omit the feature boundaries in the impl generated by the current field, generally used when there is a recursion structure in the current field

#[cacheable]
struct Value {
    id: String,
    #[cacheable(omit_bounds)]
    children: Vec<Value>
}

Trait Cacheable

For dyn Trait , add #[cacheable_dyn] when both defined trait and implement trait to make it cacheable, and of course, the corresponding struct also needs cacheable.

use rspack_cacheable::{cacheable, cacheable_dyn, to_bytes, from_bytes};

#[cacheable_dyn]
trait Animal {
    fn color(&self) -> &str;
    fn name(&self) -> &str;
}

#[cacheable]
struct Dog {
    color: String,
}
#[cacheable_dyn]
impl Animal for Dog {
    fn color(&self) -> &str {
      &self.color
    }
    fn name(&self) -> &str {
      "dog"
    }
}

#[cacheable]
struct Data {
    animal: Box<dyn Animal>,
}

let dog_data = Data {
    animal: Box::new(Dog {
      color: String::from("black"),
    }),
};
assert_eq!(dog_data.animal.name(), "dog");
assert_eq!(dog_data.animal.color(), "black");
let bytes = to_bytes(&dog_data, &()).unwrap();
let deserialize_data = from_bytes::<Data, ()>(&bytes, &()).unwrap();
assert_eq!(deserialize_data.animal.name(), "dog");
assert_eq!(deserialize_data.animal.color(), "black");

Detailed design

There are many Serialize/Deserialize crates in Rust. Through third-party benchmark data, we temporarily choose rkyv. Thanks to the independent design of the API, if we need to switch the underlying crate later, it will not have a big impact on the API.

Rkyv

A brief introduction to the implementation of Rkyv helps to understand the next implementation process. There are some core trait in Rkyv:

Serialize: Convert user Struct to StructResolver, it can return an Error.
Resolve: Use the StructResolver return from Serialize to generate ArchivedStruct.
Deserialize: Convert ArchivedStruct to Struct.

Their relationship is shown in the following, and Rkyv will implement the conversion between ArchivedStruct and Bytes.

flowchart LR
    A(Struct) -->|serialize| B(StructResolver)
    B -->|resolve| C(ArchivedStruct)
    C <--> D[Bytes]
    C -->|deserialize| E(struct)

Here is a simple example to show the code after expanding rkyv:

// source
use rkyv::{Archive, Deserialize, Serialize};

#[derive(Archive, Serialize, Deserialize)]
struct Person {
    name: String,
    age: usize,
}

// expand
use rkyv::{Archive, Deserialize, Serialize};

struct Person {
    name: String,
    age: usize,
}

struct ArchivedPerson
where
    String: rkyv::Archive,
    usize: rkyv::Archive,
{
    name: <String as rkyv::Archive>::Archived,
    age: <usize as rkyv::Archive>::Archived,
}

struct PersonResolver
where
    String: ::rkyv::Archive,
    usize: ::rkyv::Archive,
{
    name: <String as ::rkyv::Archive>::Resolver,
    age: <usize as ::rkyv::Archive>::Resolver,
}
impl rkyv::Archive for Person
where
    String: rkyv::Archive,
    usize: rkyv::Archive,
{
    type Archived = ArchivedPerson;
    type Resolver = PersonResolver;
    const COPY_OPTIMIZATION: ::rkyv::traits::CopyOptimization<Self> = unsafe {
        ::rkyv::traits::CopyOptimization::enable_if(
            0 + ::core::mem::size_of::<String>() + ::core::mem::size_of::<usize>()
                == ::core::mem::size_of::<Person>()
                && <String as ::rkyv::Archive>::COPY_OPTIMIZATION.is_enabled()
                && { builtin # offset_of(Person, name) }
                    == { builtin # offset_of(ArchivedPerson, name) }
                && <usize as ::rkyv::Archive>::COPY_OPTIMIZATION.is_enabled()
                && { builtin # offset_of(Person, age) }
                    == { builtin # offset_of(ArchivedPerson, age) },
        )
    };

    fn resolve(&self, resolver: Self::Resolver, out: rkyv::Place<Self::Archived>) {
        let field_ptr = unsafe { &raw mut (*out.ptr()).name };
        let field_out = unsafe { rkyv::Place::from_field_unchecked(out, field_ptr) };
        <String as ::rkyv::Archive>::resolve(&self.name, resolver.name, field_out);
        let field_ptr = unsafe { &raw mut (*out.ptr()).age };
        let field_out = unsafe { rkyv::Place::from_field_unchecked(out, field_ptr) };
        <usize as ::rkyv::Archive>::resolve(&self.age, resolver.age, field_out);
    }
}

impl<__S: ::rkyv::rancor::Fallible + ?Sized> ::rkyv::Serialize<__S> for Person
where
    String: rkyv::Serialize<__S>,
    usize: rkyv::Serialize<__S>,
{
    fn serialize(
        &self,
        serializer: &mut __S,
    ) -> Result<
        <Self as rkyv::Archive>::Resolver,
        <__S as rkyv::rancor::Fallible>::Error,
    > {
        let __this = self;
        Ok(PersonResolver {
            name: <String as ::rkyv::Serialize<
                __S,
            >>::serialize(&__this.name, serializer)?,
            age: <usize as ::rkyv::Serialize<__S>>::serialize(&__this.age, serializer)?,
        })
    }
}


unsafe impl<
    __C: rkyv::bytecheck::rancor::Fallible + ?Sized,
> rkyv::bytecheck::CheckBytes<__C> for ArchivedPerson
where
    String: ::rkyv::Archive,
    usize: ::rkyv::Archive,
    <__C as ::rkyv::bytecheck::rancor::Fallible>::Error: ::rkyv::bytecheck::rancor::Trace,
    <String as ::rkyv::Archive>::Archived: ::rkyv::bytecheck::CheckBytes<__C>,
    <usize as ::rkyv::Archive>::Archived: ::rkyv::bytecheck::CheckBytes<__C>,
{
    unsafe fn check_bytes(
        value: *const Self,
        context: &mut __C,
    ) -> Result<
        (),
        <__C as rkyv::bytecheck::rancor::Fallible>::Error,
    > {
        <<String as rkyv::Archive>::Archived as ::rkyv::bytecheck::CheckBytes<
            __C,
        >>::check_bytes(&raw const (*value).name, context)
            .map_err(|e| {
                <<__C as rkyv::bytecheck::rancor::Fallible>::Error as ::rkyv::bytecheck::rancor::Trace>::trace(
                    e,
                    rkyv::bytecheck::StructCheckContext {
                        struct_name: "ArchivedPerson",
                        field_name: "name",
                    },
                )
            })?;
        <<usize as rkyv::Archive>::Archived as ::rkyv::bytecheck::CheckBytes<
            __C,
        >>::check_bytes(&raw const (*value).age, context)
            .map_err(|e| {
                <<__C as rkyv::bytecheck::rancor::Fallible>::Error as ::rkyv::bytecheck::rancor::Trace>::trace(
                    e,
                    rkyv::bytecheck::StructCheckContext {
                        struct_name: "ArchivedPerson",
                        field_name: "age",
                    },
                )
            })?;
        Ok(())
    }
}

impl<__D: rkyv::rancor::Fallible + ?Sized> ::rkyv::Deserialize<Person, __D>
for rkyv::Archived<Person>
where
    String: rkyv::Archive,
    <String as rkyv::Archive>::Archived: rkyv::Deserialize<String, __D>,
    usize: rkyv::Archive,
    <usize as rkyv::Archive>::Archived: rkyv::Deserialize<usize, __D>,
{
    fn deserialize(
        &self,
        deserializer: &mut __D,
    ) -> Result<Person, <__D as rkyv::rancor::Fallible>::Error> {
        let __this = self;
        Ok(Person {
            name: <<String as rkyv::Archive>::Archived as ::rkyv::Deserialize<
                String,
                __D,
            >>::deserialize(&__this.name, deserializer)?,
            age: <<usize as rkyv::Archive>::Archived as ::rkyv::Deserialize<
                usize,
                __D,
            >>::deserialize(&__this.age, deserializer)?,
        })
    }
}

API

Context

There are Sharing and Pooling in Rkyv, which can be used for pointer sharing in the Serialize and Deserialize. With this, the Context function can be easily achieved. Here is an example of using Serialize Sharing to implement the Context function:

use std::{any::Any, ptr::NonNull};

use rkyv::{
  ser::{sharing::SharingState, Sharing},
};

const CONTEXT_ADDR: usize = 0;

pub struct ContextGuard<'a> {
  context: &'a dyn Any,
}

impl<'a> ContextGuard<'a> {
  pub fn new(context: &'a dyn Any) -> Self {
    Self { context }
  }

  pub fn add_to_sharing<S: Sharing<SerializeError>>(
    &self,
    sharing: &mut S,
  ) -> Result<(), SerializeError> {
    sharing.start_sharing(CONTEXT_ADDR);
    sharing.finish_sharing(CONTEXT_ADDR, self as *const _ as usize)
  }

  pub fn sharing_context<S: Sharing<SerializeError>>(
    sharing: &'a mut S,
  ) -> Result<&'a dyn Any, SerializeError> {
    match sharing.start_sharing(CONTEXT_ADDR) {
      SharingState::Finished(addr) => {
        let guard: &Self = unsafe { &*(addr as *const Self) };
        Ok(guard.context)
      }
      _ => Err(SerializeError::NoContext),
    }
  }
}

CustomError

Rkyv can specify the Error type on the function signature, so only the corresponding Serializer and Deserializer types need to be declared.

// for to_bytes
enum SerializeError {
    SerializeFailed(&'static str)
    ...
}

pub type CacheableSerializer<'a> = HighSerializer<
    AlignedVec,
    ArenaHandle<'a>,
    // set custom error here
    SerializeError
>;

pub fn to_bytes<T, C: Any>(value: &T, ctx: &C) -> Result<Vec<u8>, SerializeError> {
    let mut serializer = CacheableSerializer::new();
    let guard = ContextGuard::new(ctx);
    guard.add_to_sharing(&mut serializer);
    ...
}

Cacheable

Rkyv supports the use of macros to quickly implement related functions. We need to expose the rkyv crate through rspack_cacheable to facilitate the use of the correct rkyv lib after macro expansion.

pub mod __private {
    pub extern crate rkyv;
}

Macro type only API

we only need to expand it into rkyv macros.

// source
#[cacheable]
struct A {}






#[cacheable(crate=test)]
struct B {}






#[cacheable(hashable)]
struct C {}

// macro expand
#[derive(
    rspack_cacheable::__private::rkyv::Archive,
    rspack_cacheable::__private::rkyv::Deserialize,
    rspack_cacheable::__private::rkyv::Serialize
)]
#[rkyv(crate=rspack_cacheable::__private::rkyv)]
struct A {}

#[derive(
    test::__private::rkyv::Archive,
    test::__private::rkyv::Deserialize,
    test::__private::rkyv::Serialize
)]
#[rkyv(crate=test::__private::rkyv)]
struct B {}

#[derive(
    rspack_cacheable::__private::rkyv::Archive,
    rspack_cacheable::__private::rkyv::Deserialize,
    rspack_cacheable::__private::rkyv::Serialize
)]
#[rkyv(crate=rspack_cacheable::__private::rkyv)]
#[rkyv(derive(Hash, PartialEq, Eq))]
struct C {}

For #[cacheable(with=...)], we should implement rkyv::{Archive, Serialize, Deserialize} to proxy.

// source
#[cacheable(with=AsRefStr)]
struct A {}

// macro expand
struct A {}

// proxy to AsRefStr method
impl rkyv::Archive for A {
    fn resolve(...) {
        AsRefStr::resolve_with(...)
    }
}
impl rkyv::Serialize for A {
    fn serialize(...) {
        AsRefStr::serialize_with(...)
    }
}
impl rkyv::Deserialize for Archived<A> {
    fn deserialize(...) {
        AsRefStr::deserialize_with(...)
    }
}

Macro field only API

Rkyv supports configuring specific converters directly using with in the field.

// source
#[cacheable]
struct A {
    #[cacheable(with=AsString)]
    inner: std::path::PathBuf
}

// marco expand
#[derive(
    rspack_cacheable::__private::rkyv::Archive,
    rspack_cacheable::__private::rkyv::Deserialize,
    rspack_cacheable::__private::rkyv::Serialize
)]
#[rkyv(crate=rspack_cacheable::__private::rkyv)]
struct A {
    // replace cacheable with rkyv
    #[rkyv(with=AsString)]
    inner: std::path::PathBuf
}

Although Rkyv supports the direct use of omit_bounds in fields, but also need to configure the type of bounds to make the program run normally, the detailed implementation can refer to this example

// source
#[cacheable]
struct Value {
    #[cacheable(omit_bounds)]
    children: Vec<Value>
}

// macro expand
#[derive(
    rspack_cacheable::__private::rkyv::Archive,
    rspack_cacheable::__private::rkyv::Deserialize,
    rspack_cacheable::__private::rkyv::Serialize
)]
#[rkyv(crate=rspack_cacheable::__private::rkyv)]
// add extra bounds limit
#[rkyv(serialize_bounds(
    __S: rspack_cacheable::__private::rkyv::ser::Writer + rspack_cacheable::__private::rkyv::ser::Allocator + rspack_cacheable::__private::rkyv::rancor::Fallible<Error = rspack_cacheable::SerializeError>,
))]
#[rkyv(deserialize_bounds(
    __D: rspack_cacheable::__private::rkyv::rancor::Fallible<Error = rspack_cacheable::DeserializeError>
))]
#[rkyv(bytecheck(
    bounds(
        __C: rspack_cacheable::__private::rkyv::validation::ArchiveContext + rspack_cacheable::__private::rkyv::rancor::Fallible<Error = rspack_cacheable::DeserializeError>,
    )
))]
struct Value {
    // replace cacheable with rkyv
    #[rkyv(omit_bounds)]
    children: Vec<Value>
}

Cacheable_dyn

#[cacheable_dyn] The implementation reference rkyv_dyn, but at present rkyv_dyn has not released version 0.8.x, and rkyv_dyn 0.7.x can not be used with rkyv 0.8.x, so refer to its own implementation.

Serialize

#[cacheable_dyn] It is easy to implement in serialize, just add a super trait when defining the current trait.

trait SerializeDyn {
    fn serialize_dyn(&self, serializer: &mut Serializer) -> Result<usize, SerializeError>;
}

impl<T> SerializeDyn for T where T: for<'a> SerializeUnsized<Serializer<'a>> {
    fn serialize_dyn(&self, serializer: &mut Serializer) -> Result<usize, SerializeError> {
        self.serialize_unsized(serializer)
    }
}

// use macro to add super trait
// using a new trait instead of SerializeUnsized avoids declaring generic parameters
trait Animal: SerializeDyn {}

impl SerializeUnsized<Serializer<'_>> for dyn Animal {
    fn serialize_unsized(&self, serializer: &mut Serializer) -> Result<usize, SerializeError> {
        self.serialize_dyn(serializer)
    }
}

Deserialize

#[cacheable_dyn] deserialize requires use inventory crate for global collection. The simple implementation step as follows:

Add __dyn_id to the trait to distinguish different implementations and write it to the meta information of the serialize data.

trait Animal: SerializeDyn {
    // use macro to add __dyn_id method
    fn __dyn_id() -> u32;
}

#[cacheable]
struct Cat {}

impl Animal for Cat {
    const fn __dyn_id() -> u32 {
        // use macro to generate unique ID
        1
    }
}

impl ArchiveUnsized for dyn Animal {
    type Archived = dyn DeserializeAnimal;
    fn archived_metadata(&self) -> ArchivedMetadata<Self> {
        ArchivedDynMetadata::new(Animal::__dyn_id(self))
    }
}

Build a Map globally to save the deserialize implementation corresponding to each type.

pub struct DynEntry {
    dyn_id: u64,
    vtable: usize,
}
inventory::collect!(DynEntry);
static DYN_REGISTRY: std::sync::LazyLock<HashMap<u64, usize>> = std::sync::LazyLock::new(|| {
    let mut result = HashMap::default();
    for entry in inventory::iter::<DynEntry> {
        let old_value = result.insert(entry.dyn_id, entry.vtable);
        if old_value.is_some() {
            panic!("cacheable_dyn init global REGISTRY error, duplicate implementation.")
        }
    }
    result
})

Define the DeserializeDyn trait for ArchivedStruct to define deserialize methods and register them globally.

struct DeserializeDyn<T> {
    fn deserialize_dyn(&self, deserializer: &mut Deserializer, out: *mut T) -> Result<(), DeserializeError>;
}

impl DeserializeDyn<dyn Animal> for Archived<Cat> {
    fn deserialize_dyn(...) {
        <Self as DeserializeUnsized<Cat, _>>::deserialize_unsized(...)
    }
}

fn get_vtable() -> usize {
    unsafe {
        core::mem::transmute(ptr_meta::metadata(
            // deserialize 方法是实现在 Archived<Cat> 上的
            core::ptr::null::<Archived<Cat>>() as *const dyn DeserializeAnimal
        ))
    }
}
// use inventory to collect globally
inventory::submit! { DynEntry::new(Cat::__dyn_id(), get_vtable()) }

Through the __dyn_id in the meta information to find the corresponding deserialize implementation.

impl ArchivePointee for dyn DeserializeAnimal {
    fn pointer_metadata(&self, meta...) -> usize {
        DYN_REGISTRY.get(meta.__dyn_id).unwrap()
    }
}

impl DeserializeUnsized<dyn Animal, Deserializer> for dyn DeserializeAnimal {
    unsafe fn deserialize_unsized(...) {
        self.deserialize_dyn(deserializer, out)
    }
}

Preset Converter

In addition to the basic functions mentioned above, we also need to implement a series of preset converters for use in third-party crates and manual implementation scenarios. The implementation of this part mainly relies on rkyv::with. Below is a simple implementation example using AsPreset.

use rkyv::{
  rancor::Fallible,
  ser::Writer,
  string::{ArchivedString, StringResolver},
  with::{ArchiveWith, DeserializeWith, SerializeWith},
  Place,
};
use serde_json::Value;
use crate::{DeserializeError, SerializeError};

struct AsPreset;

pub struct SerdeJsonResolver {
  inner: StringResolver,
  value: String,
}

impl ArchiveWith<Value> for AsPreset {
  type Archived = ArchivedString;
  type Resolver = SerdeJsonResolver;

  #[inline]
  fn resolve_with(_field: &Value, resolver: Self::Resolver, out: Place<Self::Archived>) {
    let SerdeJsonResolver { inner, value } = resolver;
    ArchivedString::resolve_from_str(&value, inner, out);
  }
}

impl<S> SerializeWith<Value, S> for AsPreset
where
  S: Fallible<Error = SerializeError> + Writer,
{
  #[inline]
  fn serialize_with(field: &Value, serializer: &mut S) -> Result<Self::Resolver, SerializeError> {
    let value = serde_json::to_string(field)
      .map_err(|_| SerializeError::SerializeFailed("serialize serde_json value failed"))?;
    let inner = ArchivedString::serialize_from_str(&value, serializer)?;
    Ok(SerdeJsonResolver { value, inner })
  }
}

impl<D> DeserializeWith<ArchivedString, Value, D> for AsPreset
where
  D: Fallible<Error = DeserializeError>,
{
  #[inline]
  fn deserialize_with(field: &ArchivedString, _: &mut D) -> Result<Value, DeserializeError> {
    serde_json::from_str(field)
      .map_err(|_| DeserializeError::DeserializeFailed("deserialize serde_json value failed"))
  }
}

Refs

https://github.com/djkoloski/rust_serialization_benchmark
https://docs.rs/rkyv/0.8.8/rkyv/index.html
https://docs.rs/rkyv/0.8.8/rkyv/with/index.html

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC-025: Serialize and Deserialize #8185

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 0 comments

Select a reply

RFC-025: Serialize and Deserialize #8185

jerrykingxyz Oct 22, 2024 Maintainer

Summary

Motivation

Glossary

User Guide

API

Data Cacheable

Automatic Implementation

Manual Implementation

Macro API

Trait Cacheable

Detailed design

Rkyv

API

Context

CustomError

Cacheable

Macro type only API

Macro field only API

Cacheable_dyn

Serialize

Deserialize

Preset Converter

Refs

Replies: 0 comments

jerrykingxyz
Oct 22, 2024
Maintainer