[Driver][SYCL] Update preprocessed file generation #19849
Conversation
When generating preprocessed files for SYCL offloading, create a fully packaged file that contains both the HOST and DEVICE binaries. This will allow for consumption of these binary preprocessed files to be more useful, as opposed to only being able to preprocess and keep the host side of the offloading compilation When the driver encounters preprocessed (file.ii) files on the command line, these are processed in the following way: - Determines if the file is a packaged file (offload binary) - Extracts device side - Compiles device side, packages generated device into offload binary - Extracts host side - Compiles host side, embedding device binary Offload binary determination is performed by checking the magic number associated with the input file. The extraction is done via the clang-offload-packager using a new JobAction. When no output file is given, we will not package the preprocessed files but will just perform the host preprocessing. When output to a file (with an output file option), we will perform the host and device compilation, package and output to that file.
mdtoguchi
had a problem deploying
to
WindowsCILock
GitHub Actions
Error
mdtoguchi
had a problem deploying
to
WindowsCILock
GitHub Actions
Error
mdtoguchi
had a problem deploying
to
WindowsCILock
GitHub Actions
Error
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
| @@ -10389,6 +10391,50 @@ void OffloadPackager::ConstructJob(Compilation &C, const JobAction &JA, | |||
| CmdArgs, Inputs, Output)); | |||
| } | |||
|
|
|||
| // Use the clang-offload-packager to extract binaries from an packaged | |||
There was a problem hiding this comment.
| // Use the clang-offload-packager to extract binaries from an packaged | |
| // Use the clang-offload-packager to extract device binaries from a packaged |
There was a problem hiding this comment.
The binaries within a packaged file are not always device binaries in our usage, as they can be either host or device binaries. I'll update, but not add 'device'.
Earlier description says : |
srividya-sundaram
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
srividya-sundaram
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
srividya-sundaram
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
clang/lib/Driver/Action.cpp
Outdated
| @@ -86,6 +88,13 @@ void Action::propagateDeviceOffloadInfo(OffloadKind OKind, const char *OArch, | |||
| // Deps job uses the host kinds. | |||
| if (Kind == OffloadDepsJobClass) | |||
| return; | |||
| // Packaging actions can use host kinds for preprocessing. | |||
There was a problem hiding this comment.
I am not sure I follow this comment. Can you add a more descriptive comment.
| /// When generating preprocessed files, verify the compilation phases. | ||
| // RUN: %clangxx --target=x86_64-unknown-linux-gnu --offload-new-driver -fsycl -E %s -o %t.ii -ccc-print-phases 2>&1 \ | ||
| // RUN: | FileCheck %s -check-prefix PREPROC_PHASES | ||
| // RUN: %clang_cl --target=x86_64-unknown-linux-gnu --offload-new-driver -fsycl -P %s -Fi%t.ii -ccc-print-phases 2>&1 \ |
There was a problem hiding this comment.
Why is the target a Linux for clang_cl executable?
There was a problem hiding this comment.
It's more for lining up the host target with the check strings. I can update to use a Windows based target instead, expanding the test accordingly.
| /// output file name. | ||
| // RUN: %clangxx --offload-new-driver -fsycl -E %s -o sycl-preprocess.ii -### 2>&1 \ | ||
| // RUN: | FileCheck %s -check-prefix PREPROC_TOOLS | ||
| // RUN: %clang_cl --offload-new-driver -fsycl -P %s -Fisycl-preprocess.ii -### 2>&1 \ |
There was a problem hiding this comment.
Why are we passing -P here?
-P description reads: Disable linemarker output in -E mode
There was a problem hiding this comment.
Use of -P enables preprocessing information. It's just another way of covering enabling preprocessing only with the clang-cl driver.
| /// output file name. | ||
| // RUN: %clangxx --offload-new-driver -fsycl -E %s -o sycl-preprocess.ii -### 2>&1 \ | ||
| // RUN: | FileCheck %s -check-prefix PREPROC_TOOLS | ||
| // RUN: %clang_cl --offload-new-driver -fsycl -P %s -Fisycl-preprocess.ii -### 2>&1 \ |
There was a problem hiding this comment.
Is this some special file format - -Fisycl-preprocess.ii ?
There was a problem hiding this comment.
It's an MSVC compatible way of designating the output file from the preprocessing step.
(from the MSVC help output)
/Fi[file] name preprocessed file
Right. The file is still a packaged file that contains both the host and device binaries. But the representation of the file is an 'offload binary' that has a specific magic number that can be queried against. Akin to say an object or library file that have special magic number representations. |
- Update clang-cl based test to use Windows triple - Update some comments
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
mdtoguchi
temporarily deployed
to
WindowsCILock
GitHub Actions
Inactive
When generating preprocessed files for SYCL offloading, create a fully packaged file that contains both the HOST and DEVICE binaries. This will allow for consumption of these binary preprocessed files to be more useful, as opposed to only being able to preprocess and keep the host side of the offloading compilation
When the driver encounters preprocessed (file.ii) files on the command line, these are processed in the following way:
Offload binary determination is performed by checking the magic number associated with the input file. The extraction is done via the clang-offload-packager using a new JobAction.
When no output file is given, we will not package the preprocessed files but will just perform the host preprocessing. When output to a file (with an output file option), we will perform the host and device compilation, package and output to that file.