distilbert-onnx-coreml.py "works" for BERT, but I get "Error computing NN outputs." when predicting

[aarmstrong78]

Hi,

I used distilbert-onnx-coreml.py to convert a custom PyTorch BertForSequenceClassification model to CoreML. The conversion finishes without error.

However I can't use the resulting CoreML model for prediction. The following code fails:

model = coremltools.models.MLModel(f"./path/to/model/model.mlmodel")

input_ids = np.zeros((1,64))
d = {}
d['input_ids'] = input_ids

predictions = model.predict(d, True)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-29-1c38a7b07949> in <module>
----> 1 predictions = model.predict(d, True)

~/anaconda3/lib/python3.7/site-packages/coremltools/models/model.py in predict(self, data, useCPUOnly, **kwargs)
    328 
    329         if self.__proxy__:
--> 330             return self.__proxy__.predict(data, useCPUOnly)
    331         else:
    332             if _macos_version() < (10, 13):

RuntimeError: {
    NSLocalizedDescription = "Error computing NN outputs.";
}

Note, my input dim is 64:

spec.description.input

[name: "input_ids"
type {
  multiArrayType {
    shape: 1
    shape: 64
    dataType: INT32
  }
}
]

When I try to substitute my model into the DistilBERT demo app, I get the following error in Xcode when predicting:

CoreMLBert.bert_transactions_64Input
2020-01-07 10:12:58.271435+1300 CoreMLBert[1044:35882] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Invalid state": Cannot squeeze a dimension whose value is not 1: shape[1]=64 status=-5
2020-01-07 10:12:58.272716+1300 CoreMLBert[1044:35882] [coreml] Error computing NN outputs -5

The only hint that something might have gone wrong in the onnx->coreml conversion is a note about a deleted node, however I'm struggling to find out whether this is just a red herring:

[Core ML Pass] 1 disconnected constants nodes deleted
Translation to CoreML spec completed. Now compiling the CoreML model.
Model Compilation done.

Are there any particular layers that need custom conversion in BERT into coreml? Any suggestions on further debugging?

Thanks.

[aarmstrong78]

Actually, I get the same error just trying to run distilbert-onnx-coreml.py unchanged. I had to update the "target_ios" param to "minimum_ios_deployment_target" to get the script to run, so perhaps the script isn't compatible with the current version of coremltools?

Quantizing layer 783
Quantizing layer 798
Quantizing layer 809
Quantizing layer 824
––––––

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-0d553d8b37ca> in <module>
     36 
     37 pred_coreml = mlmodel.predict(
---> 38     {"input_ids": input_ids.astype(np.float32)}, useCPUOnly=True
     39 )
     40 

~/anaconda3/lib/python3.7/site-packages/coremltools/models/model.py in predict(self, data, useCPUOnly, **kwargs)
    328 
    329         if self.__proxy__:
--> 330             return self.__proxy__.predict(data, useCPUOnly)
    331         else:
    332             if _macos_version() < (10, 13):

RuntimeError: {
    NSLocalizedDescription = "Error computing NN outputs.";
}

[julien-c]

That’s very possible.

What versions of XCode and Mac OS are you running?

Do you get the same error on an iOS device?

[aarmstrong78]

MacOS 10.15.2 and Xcode Version 11.3 (11C29).

Running on an Xcode simulator I go the same error on my custom model with a bit more detail: "Cannot squeeze a dimension whose value is not 1", with the relevant dimension being 64. So somehow the shape was wrong when it got to that operation.

In the interim I've moved on to translating the model from a TF version directly to coreml using tfcoreml.

[jbmaxwell]

@aarmstrong78 Did you ever solve this problem? I have the same error message (Cannot squeeze a dimension whose value is not 1) from BERT running in CoreML (via ONNX).

[aarmstrong78]

Hi @jbmaxwell , no I didn't. After a lot of effort managed to get the TensorFlow version of Bert to convert into CoreML instead.

[jbmaxwell]

Just as an update, the model I was using above was based on the Pytorch-pretrianed-BERT repo, so I tried the huggingface distilBert model, but I get exactly the same error. (It's seeming like the problem is coming from onnx.)

[jbmaxwell]

Another update on my Cannot squeeze error; I'm training BertForMaskedLM, which, looking at the code, shouldn't have any squeeze calls in it—BertForQuestionAnswering, however, does call squeeze. So I'm guessing that, for whatever reason, the model being saved (or converted by onnx) is the BertForQuestionAnswering one. Any obvious reason why that might be?

[hollance]

If you can share the mlmodel file with the error, I can take a look.

[jbmaxwell]

@hollance That would be extremely helpful, thanks!
bert-test-256_FP16.mlmodel.zip

The error for from CoreML (in Xcode) is:

2020-02-27 10:05:54.749888-0800 Spliqs[2968:1145298] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Invalid state": Cannot squeeze a dimension whose value is not 1: shape[1]=256 status=-5
2020-02-27 10:05:54.750039-0800 Spliqs[2968:1145298] [coreml] Error computing NN outputs -5
Error running prediction: Error Domain=com.apple.CoreML Code=0 "Error computing NN outputs." UserInfo={NSLocalizedDescription=Error computing NN outputs.}

Just to note, I just ran that through my training script for a couple of epochs, as a test.

[hollance]

Run this script to fix the issue:

import coremltools
import numpy as np

mlmodel = coremltools.models.MLModel("bert-test-256_FP16.mlmodel")
spec = mlmodel._spec

spec.neuralNetwork.layers[9].activation.linear.alpha = 1  # whereNonZero
spec.neuralNetwork.layers[11].activation.linear.alpha = 1   # squeeze

new_model = coremltools.models.MLModel(spec)
new_model.save("w00t.mlmodel")

new_model.predict({"input_ids": np.zeros((1, 256), dtype=np.int32)})

The issue was the squeeze and whereNonZero layers at the beginning of the model. This script replaces them with harmless linear activation layers.

[jbmaxwell]

Thanks so much, I'll give this a try!

Do you understand why the squeeze was there in the first place? Searching modeling_distilbert.py only reveals an explicit .squeeze call in BertForQuestionAnswering (I was only using MLM). Reading your CoreML Survival Techniques, it suggests that occasionally TensorFlow will insert layers that aren't obviously necessary. Do you suspect that's what's happened here?

ps - I naively tried removing the squeeze entirely, which (obviously enough, I suppose) just caused a shape error at the broadcastTo ... ;-) I'll keep this trick of replacing with benign layers in mind in future.

[hollance]

It probably got inserted in the ONNX conversion step. There are most likely a whole bunch of other layers that don't really need to be in there. :-D

[jbmaxwell]

Wow, okay. It really is the Wild West! It would be great if Apple would throw a little more of its multi-billion dollar steam behind this process. It's generally pretty painful getting from PyTorch to CoreML, in my experience.

Thanks again.

[julien-c]

To be fair @bhushan23 from onnx-coreml helped us a lot so far :)

[jbmaxwell]

Absolutely! ONNX is invaluable, since presumably this wouldn't be possible at all without it. But by taking a proprietary approach, Apple must surely have realized there would be a significant demand on resources/human-hours trying to stay current in such a rapidly evolving field. Anyway, I don't mean to complain, it's just that it can be tricky enough getting a model working, only to face another significant (and sometimes insurmountable) hurtle in trying to get it converted to an mlmodel.