Latest version of distill not working properly with pandoc

[mondpanther]

I am having a problem with distill after upgrading to version 1.0
Here is the error message:

Unknown highlight-style C:\Users\RALFMA1\DOCUME1\R\WIN-LI1\4.0\distill\RMARKD1\TEMPLA1\DISTIL1\RESOUR1\A11Y1.THE
Error: pandoc document conversion failed with error 2
In addition: Warning message:
In (function (category = "LC_ALL", locale = "") :
OS reports request to set locale to "en_US.UTF-8" cannot be honored
Execution halted

Can you help? Because of this I am not able to compile any distill documents anymore. I tried to re-install everything (r and rstudio, also wiped any traces of pandoc) but the error remains.

The problem went away after I reverted to version 0.8

[cderv]

Can you share some more information :

• Which Rstudio version are you using ?
• Confirm the pandoc version you are using : rmarkdown::find_pandoc() ?
• The result of find.package("distill") ?

I am also on Windows and this is working for me with no issue with pandoc 2.11.1.1

In order to debug, can you try to compile this example :

---
title: "Untitled"
description: |
  A new article created using the Distill format.
author:
  - name: Nora Jones 
    url: https://example.com/norajones
    affiliation: Spacely Sprockets
    affiliation_url: https://example.com/spacelysprokets
date: "`r Sys.Date()`"
output: distill::distill_article
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = FALSE)
```

Distill is a publication format for scientific and technical writing, native to the web. 

Learn more about using Distill for R Markdown at <https://rstudio.github.io/distill>.


```{r}
x <- mean(1:20)
print(x)
```

• Save in a test.Rmd file.
• You knit with the button
• Then paste here the command line that it produced (in the Rmarkdown pane that opens when you knità just before the error, right after output file: test.knit.md. It will show us the command line used to run pandoc.

Thank you.

[cderv]

@mondpanther is this still an issue for you ?

[mondpanther]

@cderv thanks for looking into this....it was still an issue for me before I downgraded distill. I also tried running the default distill article template without success.
I can try once more to upgrade. In the meantime here is some of the info you were asking:

rmarkdown::find_pandoc()
$version
[1] ‘2.7.3’

$dir
[1] "C:/Program Files/RStudio/bin/pandoc"

find.package("distill")
[1] "C:/Users/Ralf Martin/Documents/R/win-library/4.0/distill"

[cderv]

it was still an issue for me before I downgraded distill.

If it is working correctly in the previous version but not in the last version, then we need to fix this.

Thanks for the information. I wanted to check something: You have a space in your username, and we know that sometimes in could cause issue on Windows.

Can you try to investigate with me ? We can try something to check if the space in the username is the culprit, but it would require you can write to a place not in your user folder.

Can you check if this exists on your windows computer ? It should but not 100% sure.

file.access("C:/Users/Public/Documents", 2) == 0

If so, I'll send you some step to do on your side. Thank you.

[mondpanther]

yep..I have and can access that directory...

[cderv]

Let's try to check if the culprit is the space in the path.

1. Be sure to have the distill version 1.0 with the issue
2. Run this to copy the theme file which pandoc does not found (in your error message)

public_dir <- "C:/Users/Public/Documents/"
file.copy(distill:::distill_resource("a11y.theme"), public_dir)
theme_file <- list.files(public_dir, "a11y.theme", full.names = TRUE)
theme_file

3. Run this example in a test.Rmd file

---
title: "Untitled"
description: |
  A new article created using the Distill format.
author:
  - name: Nora Jones 
    url: https://example.com/norajones
    affiliation: Spacely Sprockets
    affiliation_url: https://example.com/spacelysprokets
date: "`r Sys.Date()`"
output: 
  distill::distill_article:
    highlight: "C:/Users/Public/Documents/a11y.theme"
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

Distill is a publication format for scientific and technical writing, native to the web. 

Learn more about using Distill for R Markdown at <https://rstudio.github.io/distill>.


```{r}
x <- mean(1:20)
print(x)
```

4. Click the knit button. Does it work ? Paste the command line you have in the Rmarkdown pane below "output_file: test.knit.md"

Let see if this works.

[mondpanther]

thanks..will do later today

[cderv]

Hi @mondpanther, do you still have an issue with this ?
We would like to know before preparing a new release.

[mondpanther]

yes sorry....just struggling with various deadlines at the mo

[cderv]

For reference, another user had the issue in the community and ran the little test above.
https://community.rstudio.com/t/pandoc-error-trying-distill/90275/9

The rendering worked when the theme is moved in another directory without space in name.

It confirmed what I believe to be the issue: WINDOWS create short pathname (when there is a space somewhere in the PATH). It seems to create issue with --highlight-style argument only - with other arguments it seems to work.

I'll confirm that by trying to reproduce on my side tomorrow and then see what to do (on our side and upstream)

[cderv]

I isolated the bug to be with Windows and shortpath name transformation when there is a space in the path: It seems --highlight-style does not handle it.

Reprex

dummy proj for the example

dir.create(tmp_dir <- tempfile(tmpdir = "."))
owd <- setwd(tmp_dir)

create a dummy md file

xfun::write_utf8(c(
  "# A header"
), "test.md")

Get the distill theme file for example

packageVersion("distill")
#> [1] '1.1'
theme_file <- distill:::distill_resource("arrow.theme")

create a copy in a directory with space

dir.create(theme_dir <- "dir with space")
file.copy(theme_file, theme_dir)
#> [1] TRUE
theme_file_copy <- file.path(theme_dir, basename(theme_file))

helper function

convert <- function(theme) {
  rmarkdown::pandoc_convert("test.md", to = 'html', output = "test.html",
                            options = c("--highlight-style", theme),
                            verbose = TRUE)
}

With a path without space no issue

convert(theme_file)
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --highlight-style "C:/Users/chris/Documents/R/win-library/4.0/distill/rmarkdown/templates/distill_article/resources/arrow.theme"
convert(rmarkdown::pandoc_path_arg(theme_file))
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --highlight-style "C:\Users\chris\Documents\R\win-library\4.0\distill\rmarkdown\templates\distill_article\resources\arrow.theme"

With a space in name, it is working well

convert(theme_file_copy)
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --highlight-style "dir with space/arrow.theme"

unless path is modified by rmarkdown::pandoc_path_arg()

convert(rmarkdown::pandoc_path_arg(theme_file_copy))
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --highlight-style "DIRWIT~1\ARROW~1.THE"
Unknown highlight-style DIRWIT~1\ARROW~1.THE
#> Error: pandoc document conversion failed with error 6

which use utils::shortPathName - this is cause the issue

convert(utils::shortPathName(theme_file_copy))
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --highlight-style "DIRWIT~1\ARROW~1.THE"
Unknown highlight-style DIRWIT~1\ARROW~1.THE
#> Error: pandoc document conversion failed with error 6

Clean

setwd(owd)
unlink(tmp_dir, recursive = TRUE)

^{Created on 2020-12-09 by the reprex package (v0.3.0.9001)}

If a fix is to be made in our tool, this would be in rmarkdown not distill. I'll report upstream because it seems it is not causing any issue with other argument

# dummy proj for the example
dir.create(tmp_dir <- tempfile(tmpdir = "."))
owd <- setwd(tmp_dir)
# create a dummy md file
xfun::write_utf8(c(
  "---",
  "title: test",
  "---",
  "# A header"
), "test.md")

# create a copy in a directory with space 
dir.create(sub_dir <- "dir with space")
template_file <- file.path(sub_dir, "template.html")
xfun::write_utf8(c(
  "$body$"
), template_file)

# helper function
convert <- function(template) {
  rmarkdown::pandoc_convert("test.md", to = 'html', output = "test.html",
                            options = c("--template", template),
                            verbose = TRUE)
}

# No issues
convert(template_file)
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --template "dir with space/template.html"
convert(rmarkdown::pandoc_path_arg(template_file))
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --template "DIRWIT~1\TEMPLA~1.HTM"
convert(utils::shortPathName(template_file))
#> "C:/Users/chris/scoop/apps/rstudio-daily/current/bin/pandoc/pandoc" +RTS -K512m -RTS test.md --to html4 --output test.html --template "DIRWIT~1\TEMPLA~1.HTM"
# Clean
setwd(owd)
unlink(tmp_dir, recursive = TRUE)

[cderv]

Let's precise that it seems Pandoc on Windows no how to handle space in path, so maybe utils::shortPathName is nor more required.

[cderv]

Following the feedback we had (rstudio/rmarkdown#1976 (comment)), I pushed a fix in a PR

[ntthung]

Hi! I have the same problem. I can reproduce this by opening a fresh distill template and click "knit".

R 4.0.3
RStudio version 1.3.1073
distill version 1.1

Strangely, on another computer running an earlier RStudio version (1.3.1056), but still with distill 1.1, this is only a warning and not an error, so the document can still be rendered.

[cderv]

Hi @ntthung

Could you try with the PR #236 above ?

remotes::install_github("rstudio/distill#236")

Edit: Or with

remotes::install_github("rstudio/distill")

now that it is merged

[joshpk]

I have also had this issue and tried installing the latest update to fix it, which worked for an article. But I still get the same issue if I create a "Distill Website" R Project.

[cderv]

But I still get the same issue if I create a "Distill Website" R Project.

Do you have more details ? Is it exactly the same error message ? Can you share both command output when you render the article and the website ? Thank you.

[joshpk]

It was exactly the same error message but now on re-starting RStudio I cannot replicate it, so maybe just an issue with not restarting the session after installing the dev version of distill.

[dataandcrowd]

Yeah Same problem occurs here. I tried the github version on my newly installed R 4.0.3. and Rstduio 1.3 but it didn't seem to work. My Ubuntu, however, works fine.

[cderv]

@MrSensible so different than @joshpk ? Because last message before yours indicates it is working fine after a restart.

Can you check that you still have the issue after a restart of R and RStudio, and the last dev version of distill ?
This issue above with highlight-style should be resolved on Windows. It could be another one if you have still issues related to Pandoc. Thanks.

Also, the initial issue was with Windows only, so working on Ubuntu is expected.

[dataandcrowd]

Thanks for coming back so quickly @cderv.

I found out that my Windows and Mac machine complain because I had a README.md in the main folder. After reinstalling distill v1.1 (binary), R4.0.3, Rstudio1.3, and removing the README.md, it worked fine.

Ubuntu never warned me even if I had a readme.md in the folder when I had a lower version of distill of v0.7. However, it starts to complain after I upgraded it to v1.1.

So I guess the conclusion is not to have a README file.

[cderv]

I am not really sure to see how having a README.md can cause the issue describe here about highlight style.

If you can reproduce an issue on a demo distill website having a README.md, I would be interested to look into it. In that case you can open a new issue instead of continuing in this one. Thanks !