Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add basic zstd compression support #842

Merged
merged 1 commit into from
Mar 25, 2025

Conversation

stevebeattie
Copy link
Member

Malcontent was not properly scanning zstd compressed files e.g. kernel modules on modern Ubuntu systems. As an example, without this change:

$  mal --format=simple --verbose analyze /lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
time=2025-03-24T20:51:36.262-07:00 level=DEBUG source=$HOME/git/chainguard-dev/malcontent/pkg/action/scan.go:71 msg="skipping /usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst [<unknown>]: data file or empty" path=/usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst

With this patch applied:

$ ./mal --format=simple --verbose analyze /lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
time=2025-03-24T20:53:47.375-07:00 level=DEBUG source=$HOME/git/chainguard-dev/malcontent/pkg/archive/archive.go:110 msg="creating temp dir" path=/usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
time=2025-03-24T20:53:47.375-07:00 level=DEBUG source=$HOME/git/chainguard-dev/malcontent/pkg/archive/zstd.go:18 msg="extracting zstd" dir=$HOME/tmp/ksmbd.ko.zst439390431 file=/usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
c2/addr/ip: medium
crypto/aes: low
crypto/cipher: medium
fs/attributes/remove: medium
fs/attributes/set: medium
fs/directory/create: low
fs/directory/remove: low
fs/file/delete: low
fs/file/open: low
fs/lock_update: low
impact/remote_access/heartbeat: medium
net/ip/send_unicast: low
net/rpc/ntlm: medium
net/socket/listen: medium
net/socket/peer_address: low
net/socket/receive: low
net/socket/send: low
os/kernel/netlink: low
persist/daemon: medium
persist/kernel_module/module: medium
persist/kernel_module/name: medium
sus/exclamation: medium

This patch was mostly copy-wasting from the bz2 archive implementation and cherry-picking bits and bobs from the zstd support in the rpm.go implementation.

@stevebeattie stevebeattie requested a review from egibs March 25, 2025 03:58
@stevebeattie
Copy link
Member Author

A couple of more broad thoughts:

  • these single file compressed formats (bz2, gz, zstd, xz) feel like they could be abstracted into a common generic method that takes a data structure consisting of the type, standard suffixes, and decompression function as an argument, to extract out all the common file handling code into one place, so that it doesn't need to copy-pasta'd all over the place. (I didn't check whether the gz and xz implementations support an io.Reader interface.)

  • some simple test cases for all the different archive types would be nice.

@stevebeattie stevebeattie enabled auto-merge (squash) March 25, 2025 05:55
@egibs
Copy link
Member

egibs commented Mar 25, 2025

Method looks good to me -- can you add zst and zstd here?

func ExtractionMethod(ext string) func(context.Context, string, string) error {
// The ordering of these statements is important, especially for extensions
// that are substrings of other extensions (e.g., `.gz` and `.tar.gz` or `.tgz`)
switch ext {
// New cases should go below this line so that the lengthier tar extensions are evaluated first
case ".apk", ".gem", ".tar", ".tar.bz2", ".tar.gz", ".tgz", ".tar.xz", ".tbz", ".xz":
return ExtractTar
case ".gz", ".gzip":
return ExtractGzip
case ".jar", ".zip", ".whl":
return ExtractZip
case ".bz2", ".bzip2":
return ExtractBz2
case ".rpm":
return ExtractRPM
case ".deb":
return ExtractDeb
default:
return nil
}
}

Malcontent was not properly scanning zstd compressed files e.g. kernel
modules on modern Ubuntu systems. As an example, without this change:

```
$  mal --format=simple --verbose analyze /lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
time=2025-03-24T20:51:36.262-07:00 level=DEBUG source=$HOME/git/chainguard-dev/malcontent/pkg/action/scan.go:71 msg="skipping /usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst [<unknown>]: data file or empty" path=/usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
```

With this patch applied:

```
$ ./mal --format=simple --verbose analyze /lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
time=2025-03-24T20:53:47.375-07:00 level=DEBUG source=$HOME/git/chainguard-dev/malcontent/pkg/archive/archive.go:110 msg="creating temp dir" path=/usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
time=2025-03-24T20:53:47.375-07:00 level=DEBUG source=$HOME/git/chainguard-dev/malcontent/pkg/archive/zstd.go:18 msg="extracting zstd" dir=$HOME/tmp/ksmbd.ko.zst439390431 file=/usr/lib/modules/6.11.0-19-generic/kernel/fs/smb/server/ksmbd.ko.zst
c2/addr/ip: medium
crypto/aes: low
crypto/cipher: medium
fs/attributes/remove: medium
fs/attributes/set: medium
fs/directory/create: low
fs/directory/remove: low
fs/file/delete: low
fs/file/open: low
fs/lock_update: low
impact/remote_access/heartbeat: medium
net/ip/send_unicast: low
net/rpc/ntlm: medium
net/socket/listen: medium
net/socket/peer_address: low
net/socket/receive: low
net/socket/send: low
os/kernel/netlink: low
persist/daemon: medium
persist/kernel_module/module: medium
persist/kernel_module/name: medium
sus/exclamation: medium
```

This patch was mostly copy-wasting from the bz2 archive implementation
and cherry-picking bits and bobs from the zstd support in the rpm.go
implementation.

v2: pick up missed change to add the zst and zstd extensions to
    ExtractMethod().

Signed-off-by: Steve Beattie <[email protected]>
@stevebeattie
Copy link
Member Author

Method looks good to me -- can you add zst and zstd here?

Doh, I had made that change locally, but missed adding it to the commit; the dangers of late night hacking. Added and pushed.

@stevebeattie stevebeattie merged commit 0a16bd0 into chainguard-dev:main Mar 25, 2025
9 checks passed
@stevebeattie stevebeattie deleted the basic_zstd_support branch March 31, 2025 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants