Agnostic way to call dub

[jmh530]

run.dlang.org currently has an issue with compiler flags being ignored when dependencies are added.

One simple issue is that the dflags are not passed to dub through the dub.sdl. However, the underlying issue is that even if you do manually write the dflags to dub.sdl, then if you have something like -unittest then you get a big message that you should be using dub test instead of dub run. This is because run.dlang.org uses dub run everywhere.

So one potential solution would be to have special casing in run.dlang.org so that it reads the compiler flags and if it sees a -unittest then it re-writes it into a dub test instead of dub run.

However, it seems silly to me to have this behavior only in run.dlang.org. It makes more sense for this to be in dub itself so that other libraries can take advantage of it.

My proposed solution is something like dub exec -dflags (ignoring the specific name), which will re-write a call to dub exec into dub run, dub test, etc in an agnostic fashion based on the flags that an agnostic fashion. So for instance, dub exec is equivalent to dub run and dub exec -unittest is equivalent to dub test. Commands passed on the compile-line will override anything in the dub.sdl, but otherwise treated the same.

This is a bit similar to the relation between ldc2 and ldmd2 in providing a consistent interface. However, it should also be possible to write dub exec --compiler=ldc2 -someldcspecificflag.

[Geod24]

Did you find a resolution for this ?

[jmh530]

@Geod24 I think I have a solution here for my particular problem, but I have never used docker and have no idea how to test it properly. I was a little nervous about creating a PR without doing that.

[Geod24]

Even WIP are welcome as PR. Feel free to ping a few contributors (me included) if you need advice.

In order to test it, all you should have to do would be:

• docker build . -t mytest (. being the directory where core-exec is): This will look for the Dockerfile in the directory, send the build context (everything in . and under) to the daemon and execute the RUN lines in the Dockerfile, step by step; The end result will be accessible via either the hash or the name mytest (-t mytest is optional, just simpler).
• docker run mytest ARGS: That will run the generated container with the provided ARGS.

Personally, I think of docker build as the compilation step, and docker run as running the binary. In your case, you modified the entrypoint, so your change will affect what happen when you docker run (entrypoint is essentially the single program that is being run inside the container, and it can take arguments).

If you want to debug a bit more (errors can be very user unfriendly), you can use docker run -it --entrypoint /bin/sh to get an interactive shell and then try things around (running the entrypoint from your shell instead).

[jmh530]

Thanks. Just created PR.

I only have access to a Windows machine right now. The only way to install docker right now is to do a non-standard OS upgrade. This makes everything more difficult, so I will probably wait until after next week when I will have more access.

Are you sure that last "docker run -it..." is right? I get errors with that (as in more basic ones than the ones I get from being on Windows and trying to us docker).

[Geod24]

Oh, the last command is missing the image to use! It should be docker run -it --entrypoint /bin/sh mytest, sorry.

I only have access to a Windows machine right now.

You can run docker in WSL2. I haven't used that myself though.

[jmh530]

@Geod24 I'll need to try that later. To get docker to run on Windows requires first getting WSL2 working, which is what required the non-standard OS update. You can't just can't to the Windows updater to get it. I found a link on superuser that took me to the MS site you could get it. It apparently only became available outside of a beta-like thing in May. I should be able to try it in WSL2 instead of Powershell. I was getting some errors about bash not being available that might go away in WSL2.

[jmh530]

I've got WSL2 working properly with docker. The mytest.sh file I set up is below, saved temporarily to the core-exec folder. I can run the docker build . -t mytest without issue. However, if I run docker run mytest then I get the message that /usr/bin/env: 'bash\r': No such file or directory. When I navigate to /usr/bin, I found an env, but it didn't seem to be a folder (not sure if that matters).

So I also tried docker run -it --entrypoint /bin/sh mytest, which starts a new $ prompt. I'm not exactly sure what I'm supposed to do here. I tried going through he mytest file line by line and copying the commands in, but I tended to get errors that way.

Do I need to have mytest.sh in /bin/sh instead of core-exec?

#!/usr/bin/env bash

set -euvo pipefail

dockerId=$1

err_report() {
    echo "Error on line $1"
}
trap 'err_report $LINENO' ERR

## dub file with unittest
source="/++dub.sdl: name\"foo\" \n dependency\"mir\" version=\"*\"+/ unittest { import mir.combinatorics, std.stdio; writeln([0, 1].permutations); } void main() {}"
bsource=$(echo -e "$source" | base64 -w0)
[ "$(docker run --rm "$dockerId" "$bsource")" == "[[0, 1], [1, 0]]" ]

[Geod24]

I get the message that /usr/bin/env: 'bash\r': No such file or directory.

Looks like you edited the script in a Windows editor and \r were added (Windows line endings are \r\n). Try running dos2unix on the script.

[jmh530]

@Geod24 Would have never thought of that! Thanks! I gotta start normal work now for today, but I'll try to get back to this.

[jmh530]

@Geod24 Hmm, it looks like that didn't work. I think I'm better off just waiting until I can use linux instead of trying to figure it out with WSL.

[jmh530]

I'm making an effort on this again. I was able to get my Linux up and running with the latest version. However, it seems as if I am having some issues with the dmd install script when running docker build.

Step 10/24 : RUN bash /tmp/install.sh -p /dlang install ${DLANG_VERSION}
 ---> Running in 7d2d107e2b01
/tmp/install.sh: line 178: USERPROFILE: unbound variable
The command '/bin/sh -c bash /tmp/install.sh -p /dlang install ${DLANG_VERSION}' returned a non-zero code: 1

I got the same issue when I ran the install script myself (before I just used the package). I am running Linux Mint 20.

[jmh530]

@Geod24 The issue above was due to problems with the install.sh file. This has since bee resolved. To use the fix, I compiled with --no-cache, as in ~/dlangFiles/core-exec$ sudo docker build --no-cache . -t mytest. There is a large number of lines of output, but starting from Step 10 where I had the issue above:

Step 10/24 : RUN bash /tmp/install.sh -p /dlang install ${DLANG_VERSION}
 ---> Running in 73593cad5bb2
Downloading https://dlang.org/d-keyring.gpg
######################################################################## 100.0%
Downloading https://dlang.org/install.sh
######################################################################## 100.0%
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
The latest version of this script was installed as /dlang/install.sh.
It can be used it to install further D compilers.
Run `/dlang/install.sh --help` for usage information.

Downloading and unpacking http://downloads.dlang.org/releases/2.x/2.093.0/dmd.2.093.0.linux.tar.xz
######################################################################## 100.0%
Using dub 1.22.0 shipped with dmd-2.093.0

Run `source /dlang/dmd-2.093.0/activate` in your shell to use dmd-2.093.0.
This will setup PATH, LIBRARY_PATH, LD_LIBRARY_PATH, DMD, DC, and PS1.
Run `deactivate` later on to restore your environment.
Removing intermediate container 73593cad5bb2
 ---> 7fcdf9af4ec5
Step 11/24 : COPY ./har /tmp/har/src
 ---> 0245ee2b6ac6
Step 12/24 : SHELL ["/bin/bash", "-c"]
 ---> Running in 7a42700c5112
Removing intermediate container 7a42700c5112
 ---> cc3df0e56e7a
Step 13/24 : RUN source /dlang/$(ls -tr /dlang | tail -n1)/activate;   echo $PATH &&   $DMD -of=/tmp/har/src/har -g -debug /tmp/har/src/harmain.d /tmp/har/src/archive/har.d
 ---> Running in 8074a3de2c29
/dlang/dmd-2.093.0/linux/bin64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Error: module `harmain` is in file '/tmp/har/src/harmain.d' which cannot be read
import path[0] = /dlang/dmd-2.093.0/linux/bin64/../../src/phobos
import path[1] = /dlang/dmd-2.093.0/linux/bin64/../../src/druntime/import
Error: module `har` is in file '/tmp/har/src/archive/har.d' which cannot be read
import path[0] = /dlang/dmd-2.093.0/linux/bin64/../../src/phobos
import path[1] = /dlang/dmd-2.093.0/linux/bin64/../../src/druntime/import
The command '/bin/bash -c source /dlang/$(ls -tr /dlang | tail -n1)/activate;   echo $PATH &&   $DMD -of=/tmp/har/src/har -g -debug /tmp/har/src/harmain.d /tmp/har/src/archive/har.d' returned a non-zero code: 1

It looks like the line in Dockerfile that fails is

RUN source /dlang/$(ls -tr /dlang | tail -n1)/activate; \
  echo $PATH && \
  $DMD -of=/tmp/har/src/har -g -debug /tmp/har/src/harmain.d /tmp/har/src/archive/har.d

[Geod24]

What's the failure ?

[jmh530]

Sorry, the failure is at the end:

Error: module `harmain` is in file '/tmp/har/src/harmain.d' which cannot be read
import path[0] = /dlang/dmd-2.093.0/linux/bin64/../../src/phobos
import path[1] = /dlang/dmd-2.093.0/linux/bin64/../../src/druntime/import
Error: module `har` is in file '/tmp/har/src/archive/har.d' which cannot be read
import path[0] = /dlang/dmd-2.093.0/linux/bin64/../../src/phobos
import path[1] = /dlang/dmd-2.093.0/linux/bin64/../../src/druntime/import
The command '/bin/bash -c source /dlang/$(ls -tr /dlang | tail -n1)/activate;   echo $PATH &&   $DMD -of=/tmp/har/src/har -g -debug /tmp/har/src/harmain.d /tmp/har/src/archive/har.d' returned a non-zero code: 1

[Geod24]

Oh my bad, didn't read the full log 🤦
Did you run git submodule update --init ? It cannot find the har file that was added at a previous step, so I suspect the submodule was not initialized.

[jmh530]

@Geod24 I will try that. If that doesn't work, then I might try to just delete the docker container and start a new one.

[jmh530]

@Geod24 Thanks, that helps with the build step. What I'm trying to do now is just run the test.sh file that I adjusted from this branch. So I ran
sudo docker run test
but I got an error about
/entrypoint.sh: line 7: $1: unbound variable
I imagine that there are some ARGS I needed. So I then tried
sudo docker run dlang/core-exec:dmd test
which downloads some stuff, but then I get an error at the end that

Status: Downloaded newer image for dlangtour/core-exec:dmd
onlineapp.d: Error: module `onlineapp` source file must start with BOM or ASCII character, not \xB5

[Geod24]

Look at the usage examples: https://github.com/dlang-tour/core-exec#usage
You need to base64 encode the source.

[wilzbach]

Thanks, that helps with the build step. What I'm trying to do now is just run the test.sh file that I adjusted from this branch. So I ran

The source of truth are the CI scripts.
In particular, this line:

https://github.com/dlang-tour/core-exec/blob/7e131fb299c59af6bfded8ea8e1e0d7db55bcef1/.travis.yml#L22

tl;dr: build a docker image and name it with -t, then pass this name to the test script:

docker build -t dlangtour/core-exec:dmd .
./test.sh dlangtour/core-exec:dmd

[wilzbach]

You need to base64 encode the source.

Yup, but the test.sh handles the base64 bit -> https://github.com/dlang-tour/core-exec/blob/master/test.sh
It just expects the name/id of the docker image to test as its (first and only) argument.

[jmh530]

@Geod24 @wilzbach Thanks for the assistance. This is literally my first time working with docker. I will try again later.

[jmh530]

@wilzbach It's now running using those commands (after adding sudo to both).

[wilzbach]

after adding sudo to both

FYI: you shouldn't need to use sudo. Typically all that's required is adding your user to the docker group (but might depend on your OS/distro).

Details: https://docs.docker.com/engine/install/linux-postinstall/

[jmh530]

@wilzbach Thanks, that works.

So now I'm trying to track down the source of the errors using the shell approach that @Geod24 mentioned, using the command
docker run -it --entrypoint /bin/sh dlangtour/core-exec:dmd
I wanted to try something that I know works in the test.sh file, so I tried putting in

dockerId=$1
source='/++dub.sdl: name"foo"+/ void main() { import std.stdio; writeln("Hello World"); }'
bsource=$(echo "$source" | base64 -w0)
[ "$(docker run --rm $dockerId $bsource)" == "Hello World" ]

and there is no issue with the first three lines, but the last line says

/bin/sh: 23: docker: not found
/bin/sh: 23: [: unexpected operator

Any idea what I need to do differently?

[wilzbach]

/bin/sh: 23: [: unexpected operator

The test script requires bash, so this error likely means that it wasn't executed with bash (/bin/sh != /bin/bash).

using the command docker run -it --entrypoint /bin/sh dlangtour/core-exec:dmd

You can't run the test script inside the built D container as there's no docker installed. Is this what you were trying to do?
If you want to test a command without the test script, it's sth. like:

docker run --rm dlangtour/core-exec:dmd "$( "import std; 2.writeln;" | base64 -w0)"

If you want to connect to the docker container (i.e. open up a shell to it):

docker run -it --entrypoint /bin/sh dlangtour/core-exec:dmd
$ ./entrypoint.sh " $(echo "import std; 2.writeln;" | base64 -w0)"

Does this help?

[jmh530]

@wilzbach I'd say that my underlying issue currently is to get a better error message than just "Error on line 15". I know line 15 of myTest.sh has the error, I need to trace it back a bit further than that.

Anyway, when I run your code above I get base64: invalid input. I tried making some modifications to it to make it a little simpler and ran

docker run -it --entrypoint /bin/sh dlangtour/core-exec:dmd
source='void main() { import std.stdio: writeln; writeln("here");'
bsource=$(echo "$source" | base64 -w0)
./entrypoint.sh $(bsource)

but I'm still getting errors. In this case it is that

/bin/sh: 6: bsource: not found
./entrypoint.sh: line 7: $1: unbound variable

If I instead change the last line to ./entrypoint.sh $bsource then it just hangs for a while and gets killed.

[wilzbach]

Anyway, when I run your code above I get base64: invalid input

Sorry, I didn't test it as I was replying from a laptop without my D setup.
It looks like an issue with the base64 binary in the Docker image, this works fine on my machine:

$ docker run --rm dlangtour/core-exec:dmd "$(echo "import std; void main() { \"Hello\".writeln;}" | base64 -w0)"
Hello

but I'm still getting errors. In this case it is that

./entrypoint.sh $(bsource) this means you're executing the command bsource which doesn't exist. You want to reference the variable "$bsource".

[jmh530]

Thanks a bunch for that. There were two issues that were tripping me up that you made clearer. The first was the order, if I put the source and bsource first, then that helps. Second, was putting quotes around things properly. Below works without error

source="import std; void main() { \"Hello\".writeln;}"
bsource=$(echo "$source" | base64 -w0)
docker run --rm dlangtour/core-exec:dmd "$bsource"

After doing this, I tried to add a dub.sdl file and that failed, but this was only on my branch, not on master. So this at least now gives me some direction on figuring out what the issue is. Thanks again!