Set device to maintenance mode if TPM error is detected

[shjala]

We have seen errors like "point is not on the required curve" in the past that prevents the device from upgrading because TPM fails to perform an ECDH key agreement to generate the KEK and decrypt the vault key. To catch these errors early, check if TPM is not functioning properly on a critical path and set the device to maintenance mode.

[shjala]

For now it looks like this :

[shjala]

What remained :

• wait for add MAINTENANCE_MODE_REASON_TPM_ENCRYPTION_FAILURE eve-api#78 and update
• remove the injected error
• remove comment https://github.com/lf-edge/eve-api/blob/1a8f9be485b1cb6e0509b4c56558980bd8be7592/proto/info/info.proto#L707
• update the controller to show the new maintenance mode reason.

[shjala]

Updating EVE-API updates protobuf as a side effect :

pillar git:(maintenance.mode.tpm.err) ✗ go get -u github.com/lf-edge/eve-api/go/info
go: upgraded github.com/lf-edge/eve-api/go v0.0.0-20241213165007-1a8f9be485b1 => v0.0.0-20250102213900-786246223024
go: upgraded google.golang.org/protobuf v1.33.0 => v1.36.1

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 20.90%. Comparing base (2e69dc3) to head (9ec1a4e).
Report is 25 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4462   +/-   ##
=======================================
  Coverage   20.90%   20.90%           
=======================================
  Files          13       13           
  Lines        2894     2894           
=======================================
  Hits          605      605           
  Misses       2163     2163           
  Partials      126      126           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shjala]

@eriknordmark for fault injection I exprimented with SWTPM https://github.com/shjala/swtpm-fault-injection, but this won't work because Qemu is using control socket of SWTPM and not the TPM socket, so I have to do some reverse engineering to figure out how to inject faults there (work in progress...).

[OhmSpectator]

@shjala, could you please take a look at Erik's comments? =)

[shjala]

@shjala, could you please take a look at Erik's comments? =)

I'm aware, I'm doing some preliminary work to address the comments.

[shjala]

@eriknordmark Testing still is a bit manual, but works, I'll create some scripts to automate it and later add it to Eden:

➜  swtpm-fault-injection git:(main) ✗ git clone [email protected]:shjala/swtpm-fault-injection.git
➜  swtpm-fault-injection git:(main) ✗ cd swtpm-fault-injection
➜  swtpm-fault-injection git:(main) ✗ which swtpm             
/usr/local/bin/swtpm
➜  swtpm-fault-injection git:(main) ✗ ldd /usr/local/bin/swtpm
        linux-vdso.so.1 (0x00007ffc6e38c000)
        libswtpm_libtpms.so.0 => /usr/local/lib/swtpm/libswtpm_libtpms.so.0 (0x00007b4d65723000)
        libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007b4d655ce000)
        libtpms.so.0 => /usr/local/lib/libtpms.so.0 (0x00007b4d6549c000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007b4d65200000)
        libjson-glib-1.0.so.0 => /lib/x86_64-linux-gnu/libjson-glib-1.0.so.0 (0x00007b4d6546f000)
        libgobject-2.0.so.0 => /lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007b4d651a0000)
        libseccomp.so.2 => /lib/x86_64-linux-gnu/libseccomp.so.2 (0x00007b4d6544d000)
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007b4d64c00000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007b4d6512a000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007b4d64b19000)
        /lib64/ld-linux-x86-64.so.2 (0x00007b4d65758000)
        libgio-2.0.so.0 => /lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007b4d6493f000)
        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007b4d6543e000)
        libgmodule-2.0.so.0 => /lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007b4d65437000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007b4d6510e000)
        libmount.so.1 => /lib/x86_64-linux-gnu/libmount.so.1 (0x00007b4d650ca000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007b4d6509e000)
        libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007b4d65067000)
        libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007b4d648a8000)
➜  swtpm-fault-injection git:(main) ✗ ./findoffset.sh /usr/local/lib/swtpm/libswtpm_libtpms.so.0 SWTPM_IO_Read
Function: SWTPM_IO_Read
Start Address:    0x0000000000013e84
End Address:    0x0000000000013f67
End Offset for uretprobe: 227 bytes
➜  swtpm-fault-injection git:(main) ✗ sudo ./main -swtpm -pid <SWTPM_PID> -libtpms "/usr/local/lib/swtpm/libswtpm_libtpms.so.0" -swtpm-io-read-offset 227 -inputs="CmdECDHZGen:RCFailure:3m:false"

[shjala]

@eriknordmark Now device can be in Maintenance Mode for multiple reasons, I'm testing this by simulating multiple reasons and see if it works as expected. I suspect controller has to change too?

[shjala]

When I simulate multiple maintenance mode reasons, mergeMaintenanceMode() somehow gets called and incorrectly sets the mode to false with log Changed maintenanceMode to false, with reason as MaintenanceModeReasonTpmEncFailure !

[eriknordmark]

But the MaintModeReason* are values 1,2,3,4 thus you can't use them as bits in a bit map.
If you set 1|2 you end up with the 3 in the field and String() reports as 1, 2, and 3 were set.

Thus to handle multiple we either need to change the enum values in the API to be bits (1<<0, 1<<1, etc), which would an incompatible value change for MaintenanceModeReasonNoDiskSpace (value would change from 3 to 1<<2 = 4), or add some "additionalMaintenceModeReasons" array to the info message.

[shjala]

right, I think I'll add a new field in the API then.