GPU docs

[mcabbott]

This re-writes the start of the GPU documentation page. It aims to use simpler examples, and stress that model |> cu just works, before talking about more exotic non-CUDA packages, and the automatic model |> gpu.

Rendered MD: https://github.com/FluxML/Flux.jl/blob/gpu_docs/docs/src/guide/gpu.md

Current docs: http://fluxml.ai/Flux.jl/stable/guide/gpu/

[github-actions]

Once the build has completed, you can preview any updated documentation at this URL: https://fluxml.ai/Flux.jl/previews/PR2510/ in ~20 minutes

In particular, this page: https://fluxml.ai/Flux.jl/previews/PR2510/guide/gpu/

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 33.20%. Comparing base (32db5d4) to head (d9c03b9).
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2510   +/-   ##
=======================================
  Coverage   33.20%   33.20%           
=======================================
  Files          31       31           
  Lines        1843     1843           
=======================================
  Hits          612      612           
  Misses       1231     1231           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[CarloLucibello]

removing the milestone as this shouldn't be blocking

[mcabbott]

The point is, in part, to think through whatever interface we're adopting by trying to explain it clearly. If it's a mess then 0.15 is when to fix it. That's why it was on the milestone.

[CarloLucibello]

Can we get this done so that it doesn't delay the 0.15 release then? I would like to tag in a few days

[mcabbott]

Sure. What do you think this lacks?

[CarloLucibello]

Suggested change

	!!! compat "Flux ≤ 0.13"
	Old versions of Flux automatically loaded CUDA.jl to provide GPU support. Starting from Flux v0.14, it has to be loaded separately. Julia's [package extensions](https://pkgdocs.julialang.org/v1/creating-packages/#Conditional-loading-of-code-in-packages-(Extensions)) allow Flux to automatically load some GPU-specific code when needed.

[CarloLucibello]

gpu_backend! affects the return from gpu_device like this:

• If no GPU is available, it returns a CPUDevice object.
• If a LocalPreferences file is present, then the backend specified in the file is used. If the trigger package corresponding to the device is not loaded, then a warning is displayed.
• If no LocalPreferences file is present, then the first working GPU with loaded trigger package is used.

This is already described in the docstring of gpu_device. I think we shouldn't mention gpu_backend! at all in this guide because it is useless in practice.

[CarloLucibello]

Maybe we should put a TLDR at the top just saying something like

using CUDA # or AMDGPU or Metal
device = gpu_device()
model = model |> device
for epoch in 1:num_epochs
    for (x, y) in dataloader
        x, y = device((x, y))
        ... compute gradients and update model ...
    end
end

[CarloLucibello]

I will finish this

[mcabbott]

put a TLDR at the top

This is what we're trying not to do. Not "here's the magic boilerplate you should copy", but instead the guide is "here's how things actually work".