Add nvtext::tokenized_to_tensor API

[davidwendt]

Description

Adds new nvtest::tokenized_to_tensor API for compatibility of the existing subword tokenizer. The tokenizer is to be replaced with a normalizer, wordpiece tokenizer and this new API to provide the complete functions to match what the subword tokenizer currently produces.

The wordpiece tokenizer created here #17600 will produce tokens for an input strings column as a list column of integers. This list column can be provided to this new API to produce the existing tensor data structure containing token-ids truncated or strided as well as an attention mask and metadata.

nvtext::tokenizer_result nvtext::tokenized_to_tensor(
  cudf::lists_column_view const& input,
  cudf::size_type max_sequence_length,
  cudf::size_type stride,
  bool do_truncate,
  rmm::cuda_stream_view stream ,
  rmm::device_async_resource_ref mr);

The nvtext::tokenizer_result is defined here:

cudf/cpp/include/nvtext/subword_tokenize.hpp

Line 77 in 4323ae4

struct tokenizer_result {

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[davidwendt]

/ok to test