Is batch processing possible during inference?

[ereday]

Hi, thank you for providing a pre-trained model. I am wondering if it is possible to perform a batched prediction with the model at https://huggingface.co/RUCAIBox/mtl-data-to-text ? Something like below?:

inputs = tokenizer(
    ["Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man",
     "Describe the following data: Batman | instance of | Superhero",
    ]
    return_tensors="pt",
)

generated_ids = model.generate(**inputs)

tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
['Iron Man is a fictional superhero appearing in American comic books published by Marvel Comics.',
"Batman is a superhero"]

[StevenTang1998]

Sorry, I am a little confused about your requirements. Does this code have an issue and does not solve the batch prediction?

[ereday]

Ah, sorry for the confusion. I was wondering how I can use the pre-trained model with batch sizes bigger than 1. As an example, let's say I have these two texts as input:

t1='Describe the following data: Love Is Mighty Close | INSTANCE_OF | album [SEP]'
t2='Describe the following data: Iron Man | instance of | Superhero [SEP] Stan Lee | creator | Iron Man'

And I'd like to make MTL-data-to-text model describe both t1 and t2 at the same time instead of running the forward pass twice as shown below:

tokenizer = MvpTokenizer.from_pretrained("RUCAIBox/mvp")
model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mtl-data-to-text")
# T1
inputs = tokenizer(t1, return_tensors="pt")
generated_ids = model.generate(**inputs)
tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
# T2
inputs = tokenizer(t2, return_tensors="pt")
generated_ids = model.generate(**inputs)
tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

Does the current codebase support inferencing with bs>1?

[StevenTang1998]

inputs = tokenizer([t1, t2], return_tensors="pt")
generated_ids = model.generate(**inputs)
tokenizer.batch_decode(generated_ids, skip_special_tokens=True)

Does this satisfy your requirements?

[ereday]

Yes, it works. Oddly enough, I tried this before I created the issue, but it didn't work then. Anyway, thank you. I'm closing the issue.