tensorflow not supported on Apple M1 chips

[lassedochreden]

Hi @jwindhager

I tested the DeepCell implementation in steinbock on my Mac (OS Big Sur 11.6.2, Apple M1 chip). As reported here, tensorflow within docker is not supported on this system and I get the following error:

steinbock segment deepcell --minmax

2022-01-21 13:57:30.691090: F tensorflow/core/lib/monitoring/sampler.cc:42] Check failed: bucket_limits_[i] > bucket_limits_[i - 1] (0 vs. 10)
qemu: uncaught target signal 6 (Aborted) - core dumped
/app/entrypoint.sh: line 3:    17 Aborted                 python -m steinbock "$@"

I used this alias: docker run --platform linux/amd64 -v /Users/lasmey/Desktop/IMC\ processing/steinbock_IMMUCAN:/data -v /tmp/.X11-unix:/tmp/.X11-unix -v ~/.Xauthority:/home/steinbock/.Xauthority:ro -u $(id -u):$(id -g) -e DISPLAY=$(hostname):0 ghcr.io/bodenmillergroup/steinbock:0.12.0

[nilseling]

I discussed this with @lassedochreden. Not a huge issue since we can always use ubuntu systems - just so that you know that there are tensorflow/mac issues.

[jwindhager]

Thanks for reporting this - tensorflow by now should support M1, but it may be that deepcell still uses an old version. I'll check and report back here.

[jwindhager]

I was wrong, tensorflow within linux Docker on M1 doesn't seem to work at the moment, sorry.

[caucheteux]

Dear all, After many trials and errors, I am now able to run Steinbock and other tools that you made available on this website almost without errors. This is really awesome, thanks a lot! I am currently using iMac with Intel chip but not without performance issues. So since M1/M2 chips are almost the only available options now, I am wondering whether previous issues mentioned here with Docker/ Steinbock and tensor flow vs Apple chips are still a problem at the moment? Thank you!

[jwindhager]

Hi @caucheteux, thanks for pinging us here about this! We are currently testing a few routes how to make this work (emulating the amd64 platform on M1, providing containers specifically built for M1, ...), but haven't yet settled on a solution. I'll reopen this issue to keep track of our efforts here.

One thing you can try already, is to execute docker run with the --platform linux/amd64 argument, as described here (HT to @dnebdal and Marianna Vincenti from UiO). I'm not sure whether this will work with the tensorflow base image of the steinbock Docker container, since this one is built with AVX support, but it may be worth a shot.

[jwindhager]

Quick update: using --platform linux/amd64 works in principle, but - as expected - the tensorflow build is "broken " in this case (compiled with AVX support), so running deepcell segmentation (and possibly Ilastik/CellProfiler) won't work.

[caucheteux]

Thanks a lot for the update, I appreciate it. I'll give it a go with amd64 and will let you know asap.

[dnebdal]

Quick update: using --platform linux/amd64 works in principle, but - as expected - the tensorflow build is "broken " in this case (compiled with AVX support), so running deepcell segmentation (and possibly Ilastik/CellProfiler) won't work.

Right, I guess that makes sense. Is it easier to recompile an amd64-without-AVX version of tensorflow than to make it work on M1? And just how much performance does that give up real hardware, anyway?

[jwindhager]

Quick update: It has been confirmed that an amd64-without-AVX build such as this one works indeed, albeit very slow (HT to Will Chen from UCSF). The preferred solution would still be to build tensorflow for linux/arm64...

[jul1an-o]

Hi @jwindhager ,

I'm using the most current version of the Steinbock pipeline (0.16.0) and have gotten some of it to work with my M1 Mac. As others have described, I'm running into issues with the deepcell/mesmer segmentation that uses tensorflow. I've tried the -macos.whl and -linux_x86_64.whl but still haven't gotten the docker segmentation to work. It seems that tensor flow installs correctly when trying the macos and linux_x86_64 wheels, but the Steinbock docker image has arguments that use the AVX version of tensorflow? I saw this file that you uploaded recently with details on using tensor flow with M1 Mac's.

How would I modify the docker image with the correct tensorflow arguments? When I open the docker desktop and 'View packages and CVEs' for the Steinbeck image, I can I see tensorflow arguments are listed on layers:

10:
|2 TF_PACKAGE=tensorflow-cpu TF_PACKAGE_VERSION=2.8.4 /bin/sh -c python3 -m pip install --no-cache-dir ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}

12:
|2 TF_PACKAGE=tensorflow-cpu TF_PACKAGE_VERSION=2.8.4 /bin/sh -c chmod a+rwx /etc/bash.bashrc

15:
RUN |1 TENSORFLOW_VERSION=2.8.4 /bin/sh -c apt-get update && apt-get install -yqq python3 python3-pip && apt-get clean && python3 -m pip install --upgrade pip && python3 -m pip install "tensorflow==${TENSORFLOW_VERSION}" # buildkit

(layer numbers on my device when inspecting the image's 59 layers in the docker dashboard).

Trying to modify the container by image by using docker run --platform linux/amd64 ghcr.io/bodenmillergroup/steinbock /bin/bash produces the same output as the --help command. I'm trying to swap the argument for the tensorflow source in layer 15 to one of the macos or linux_x86_64 wheels. Is this the correct approach? And how would I go about modifying the docker image to do this?

I appreciate your time and help!

EDIT 03/08/23

I've figured out how to download and (attempt) to modify the Steinbock docker image, but still haven't had success. Here are my steps trying to troubleshoot the issue:

docker images produces
ghcr.io/bodenmillergroup/steinbock 0.16.0 a0f249686928 4 weeks ago 9.03GB

docker save a0f249686928 > steinbock.tar saves Steinbock image as tar file

docker rmi a0f249686928 removes local Steinbock image

docker load < steinbock.tar test to confirm ability to load unmodified Steinbock image produced in above step

docker rmi a0f249686928 removes test load of unmodified Steinbock image

mkdir steinbockextract create directory for extracting Steinbock .tar image
tar -xvf steinbock.tar -C steinbockextract extract Steinbock .tar image

tar -cvf steinbockrepackaged.tar steinbockextract regenerate .tar file from Steinbockextract folder created above (still no modification made to any of the files contained inside the extracted .tar file)

docker load < steinbockrepackaged.tar produces this following error
open /var/lib/docker/tmp/docker-import-1846628369/steinbockextract/json: no such file or directory

When inspecting the contents of the extracted tar, it seems that I should be able to modify the a0f2496869280e5b545fc32b44121bd4eaf4137d39b91d23151d64f2244ace65.json which contains the configuration for the image. There are arguments in this .json that specify which version of tensorflow to install:
python3 -m pip install \"tensorflow==${TENSORFLOW_VERSION}\"
I would like to substitute one of the silicon compatible tensorflow packages in this pip install, but so far I'm not able to even load the unmodified Steinbock image after regenerating it as a .tar file.

[jwindhager]

Hi @jul1an-o, sorry for the delay. Just briefly writing to let you know that I'm currently traveling and will respond in 2 weeks the latest. Meanwhile, maybe the current steinbock maintainer (@Milad4849) can help out.

[jul1an-o]

Hi @jwindhager, just wanted to follow up on this and see if you were in the process of working on a solution?

[jwindhager]

Hey @jul1an-o, apologies for the late reply!

Unfortunately, the current steinbock Dockerfile installs the official build of TensorFlow that requires CPU hardware supporting Advanced Vector Extensions (AVX), which is not the case for Apple's ARM platform. One way to get steinbock (specifically, the DeepCell functionality) to work on M1 Macs is to replace the official TensorFlow build with a version that was built for non-AVX platforms, see this code comment (first item). Non-AVX builds of TensorFlow are not officially provided by Google, which is why steinbock does not provide this out of the box. Additionally, running DeepCell in an emulated (linux/amd64 on mac/arm64) environment without GPU or even AVX support would likely be very slow.

Having said that, people have reportedly had success with non-AVX community builds of TensorFlow, such as the one mentioned in the code comment above. If you want to give this a try, you would need to manually modify the Dockerfile to install the custom wheel instead of the official version (e.g. by adding a tensorflow-macos target, or by adding an instruction installing the custom build on top of the official one) and then build a custom steinbock Docker image from the modified Dockerfile. This adds quite some hassle for very little gain (as the deepcell commands will be very slow, see above), so I tend to recommend running steinbock on a different system (e.g. virtual machine, cluster) instead or creating the segmentation masks outside of steinbock.

As to your question on where we stand with adding out-of-the-box support for Apple's ARM platform: unfortunately, as long as Google does not provide a TensorFlow Docker image for that platform or mac/arm64 wheels of the TensorFlow package, this will likely not be added anytime soon, sorry (please correct me if I'm wrong @Milad4849).

[caucheteux]

Thanks guys. All this info is very helpful indeed. I also havent been able to run it on M1. But as @jwindhager said well, you can run everything else but the deepcell regardless. I've been running the deep cell/ Mesmer masks separately still allowing me to use Steinbock and IMCSegPipe data for the IMCRtools or IMCDataAnalysis R packages.

[erikgerdtsson]

Has this been resolved? Can Deepcell/Mesmer based pipeline be compatible with mac. Is there a new version out soon that support Apple M chip series? Thanks for sharing your pipeline, works great on Intel/CUDA.