Exported HF models contain SoftmaxCrossEntropyLoss node

[yuanyao-nv]

I notice that using the torchscript exporter, ie using the --torchscript-onnx flag, all the exported HF models contain a SoftmaxCrossEntropyLoss node at the end that compares the model output with the true labels. Why is training-related ops showing up in the exported model and is there a way to disable such ops?

The export command I'm using is:

python pytorch/benchmarks/dynamo/huggingface.py --performance --amp -dcuda --output=/workspace/output/dynamo-onnx_huggingface_amp_inference_cuda_performance.csv --inference --use-eval-mode -n1 --torchscript-onnx --no-skip --dashboard -k <model_name> 

[titaiwangms]

I think this is why https://github.com/pytorch/pytorch/blob/8184cd85fcfe663019edb3c1e502e03dcbaba4f0/benchmarks/dynamo/common.py#L3785-L3787C14

[titaiwangms]

Maybe you would want to hack around here to make it eval mode: https://github.com/pytorch/pytorch/blob/01694eaa56adb343f5d3d15b53d2962615dafe17/benchmarks/dynamo/huggingface.py#L513-L520

[yuanyao-nv]

Maybe you would want to hack around here to make it eval mode: https://github.com/pytorch/pytorch/blob/01694eaa56adb343f5d3d15b53d2962615dafe17/benchmarks/dynamo/huggingface.py#L513-L520

I think it is the eval path that's being run, since is_training is still False. One can also replace --performance with --accuracy so that args.use_eval_mode doesn't get overwritten to False. But the exported model still contains a SoftmaxCrossEntropyLoss node.

[titaiwangms]

We found it's this: https://github.com/pytorch/pytorch/blob/a1ca4dfe0ba65de3a8293aabd9e25504f4771e94/benchmarks/dynamo/huggingface.py#L435