CI to scalingo always pass even when build actually failed

[BaptisteRichard]

First, I'm really out of my league here so please disregard any comment/interpretation/idea that is just plain silly for you, I'm trying my best to help with my limited knowledge.

Using scalingo as provider, all my CI jobs are marked as passed even if something went wrong in the build.

I tried to find why, and I don't know much about Ruby, but it seems to me that the command return the last status code of anything executed, and in the case of the scalingo provider, it seems to be the "remove keys" script, which return 0 as it successfully remove keys as expected.

Looking at file https://github.com/travis-ci/dpl/blob/56e5a5c631da074a7aea7e9ecb23596745950edb/spec/dpl/providers/scalingo_spec.rb I think there should be some kind of control of the return codes of each part before continuing. Maybe "have_run" might be expanded to something along the lines of "have_run_without_errors" ?

As far as I looked, I've seen that for other providers, the build stage is the last command sent, so anything going wrong at that point would lead to a job failure, which is not the case with Scalingo.

Unfortunately I'm completely unable to assess if my theory is correct nor propose a PR to correct this.

I'll try to keep an eye on this issue and hopefully help resolve it in any way I can.