Skip to content

Fallback Ident is not normalized #500

Open
@kennytm

Description

@kennytm

Starting from Rust 1.42.0 (rust-lang/rust#66670) Unicode identifiers are normalized in NFC form, that means the following proc-macro will produce true:

extern crate proc_macro;
use proc_macro::{Ident, Span, TokenStream};

#[proc_macro]
pub fn check(_: TokenStream) -> TokenStream {
    let decomposed = Ident::new("e\u{301}", Span::call_site());
    let composed = Ident::new("\u{e9}", Span::call_site());
    (composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}

However, the proc_macro2 fallback does not perform NFC normalization, meaning the equivalent code in proc_macro2 will produce false:

use proc_macro2::{Ident, Span, TokenStream};

pub fn check(_: TokenStream) -> TokenStream {
    let decomposed = Ident::new("e\u{301}", Span::call_site());
    let composed = Ident::new("\u{e9}", Span::call_site());
    (composed.to_string() == decomposed.to_string()).to_string().parse().unwrap()
}

fn main() {
    dbg!(check(TokenStream::default()));
    // Ident { sym: false, span: bytes(1..6) }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions