Question: Issue using verify script to validate model

[miyunari]

Question

What exactly does Unknown in-toto predicate type sigstore.dev/model-transparency/manifest/v1 mean? 😄

I tried to validate the model from the oss-na24-slsa-workshop-model-integrity demo using the verfiy.py script. It seemed to work first and then crashed with the following stacktrace:

$ kubectl logs whatever-workload model-validation
INFO:__main__:Creating verifier for sigstore
INFO:tuf.api._payload:No signature for keyid f5312f542c21273d9485a49394386c4575804770667f2ddb59b3bf0669fddd2f
INFO:tuf.api._payload:No signature for keyid ff51e17fcf253119b7033f6f57512631da4a0969442afcf9fc8b141c7f2be99c
INFO:tuf.api._payload:No signature for keyid ff51e17fcf253119b7033f6f57512631da4a0969442afcf9fc8b141c7f2be99c
INFO:tuf.api._payload:No signature for keyid ff51e17fcf253119b7033f6f57512631da4a0969442afcf9fc8b141c7f2be99c
INFO:tuf.api._payload:No signature for keyid ff51e17fcf253119b7033f6f57512631da4a0969442afcf9fc8b141c7f2be99c
INFO:__main__:Verifying model signature from /data/tensorflow_saved_model/model.sig
Traceback (most recent call last):
  File "/src/verify.py", line 176, in <module>
    main()
    ~~~~^^
  File "/src/verify.py", line 162, in main
    model.verify(
    ~~~~~~~~~~~~^
        sig=sig,
        ^^^^^^^^
    ...<3 lines>...
        ignore_paths=[args.sig_path],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/src/model_signing/model.py", line 77, in verify
    peer_manifest = verifier.verify(sig)
  File "/src/model_signing/signing/sign_sigstore.py", line 371, in verify
    return in_toto.IntotoPayload.manifest_from_payload(payload)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/src/model_signing/signing/in_toto.py", line 85, in manifest_from_payload
    raise ValueError(f"Unknown in-toto predicate type {predicate_type}")
ValueError: Unknown in-toto predicate type sigstore.dev/model-transparency/manifest/v1

I started building a model-validation-controller for k8s, thats why the command (verify --model_path=/data/tensorflow_saved_model --sig_path=/data/tensorflow_saved_model/model.sig sigstore --identity [email protected] --identity-provider https://accounts.google.com) is executed on an initcontainer.

Name:         whatever-workload
Namespace:    model-validation-controller
Priority:     0
Node:         master3/100.117.254.19
Start Time:   Mon, 20 Jan 2025 12:57:51 +0100
Labels:       <none>
Annotations:  validation.rhtas.redhat.com/ml: true
Status:       Pending
IP:           10.244.4.180
IPs:
  IP:  10.244.4.180
Init Containers:
  model-validation:
    Container ID:  containerd://4b519b93ddbff81939e646745e91f90c11adaff0ab22bc52f4bd387f5b523578
    Image:         ghcr.io/miyunari/model-transparency-cli:latest
    Image ID:      ghcr.io/miyunari/model-transparency-cli@sha256:3ffaa0edb5f2c925b4b0ab1d029496b6631303e0ab898ebc58ff1e4c725ff6b5
    Port:          <none>
    Host Port:     <none>
    Command:
      verify
      --model_path=/data/tensorflow_saved_model
      --sig_path=/data/tensorflow_saved_model/model.sig
      sigstore
      --identity
      [email protected]
      --identity-provider
      https://accounts.google.com
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 20 Jan 2025 13:00:53 +0100
      Finished:     Mon, 20 Jan 2025 13:00:55 +0100
    Ready:          False
    Restart Count:  5
    Environment:    <none>
    Mounts:
      /data from model-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kxpfw (ro)
Containers:
  whatever-workload:

[spencerschrock]

The library has changed since OSS NA 2024, and the old signature can't be validated by the new API. So either the signature needs to be reproduced, or you check out a commit here from ~9 months ago it should verify.

The signature file (model.sig) has a DSSE envelope with an intoto statement inside.

For that older signature, it looks like this:

jq --raw-output .dsseEnvelope.payload /tmp/tensorflow_saved_model/model.sig | base64 -d | jq 
{
  "_type": "https://in-toto.io/Statement/v1",
  "predicateType": "sigstore.dev/model-transparency/manifest/v1",
  // abbreviated ...
}

The library currently expects the predicate to be one of the following:

• https://model_signing/Digest/v0.1
• https://model_signing/DigestOfDigests/v0.1
• https://model_signing/DigestOfShardDigests/v0.1
• https://model_signing/Digests/v0.1
• https://model_signing/ShardDigests/v0.1

[mihaimaruseac]

That's partially my fault. We have several experimental ways to build the digest so I created multiple predicates for this, but forgot to check that the old ones would still be able to verify.