Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use pak when package installation is not found #284

Draft
wants to merge 32 commits into
base: main
Choose a base branch
from

Conversation

schloerke
Copy link
Contributor

@schloerke schloerke commented Mar 18, 2021

Fixes #234

This PR is to get the conversation going to hopefully remove maintenance requirements of shinyapps.io by offloading to pak which uses RSPM to query rstudio/r-system-requirements

Benefits:

This metric indicates how good the rules are for the majority of packages R users are likely to install
Ubuntu 18
No rules 90.1%
With rules 99.2%".

Setup:

  • If there is no install script, then pak is used.
    • This PR does not use the CRAN version of pak. Instead, it uses the hosted version provided by RStudio's r-lib team. This is done to update rapidly as bug fixes are found. I'd recommend moving to a CRAN installation once the dev version makes it to CRAN.
    • Disadvantage: This requires RSPM to work / function, whereas the static files are known at install time.
  • If there is a package install script, the install script is used.
    • Using the existing install script will not install dependencies for the required packages. I do not know which packages are supplied at install time. Ex: rgl originally only wanted to install libgl1-mesa-dev and libgl-mesa-dev, where pak says it requires:
❯❯ pak::pkg_system_requirements("rgl", "ubuntu", "18.04")
 [1] "apt-get install -y imagemagick"      "apt-get install -y libmagick++-dev"  "apt-get install -y make"
 [4] "apt-get install -y libicu-dev"       "apt-get install -y pandoc"           "apt-get install -y libfreetype6-dev"
 [7] "apt-get install -y libglu1-mesa-dev" "apt-get install -y libpng-dev"       "apt-get install -y libgl1-mesa-dev"
[10] "apt-get install -y zlib1g-dev"

Alterations:

  • I have removed simple installation files that immediately matched ubuntu 18.04. If there were more complicated ones, like sodium, I did not remove their install script.
  • I am working on making a GitHub Action to test all known packages. (bionic && xenial)

Future:

Game plan moving forward. I'd recommend spamming the few handful of authors for packages like av which require system installations, but do not report them so that rstudio/r-system-requirements picks them up automatically. (Curious as to why av by Jeroen doesn't have any pak sys reqs when its SystemRequirements looks like: FFmpeg (>= 3.2); with at least libx264 and lame (mp3) drivers. Debian/Ubuntu:libavfilter-dev, Fedora/CentOS: ffmpeg-devel (via https://rpmfusion.org), MacOS Homebrew: ffmpeg.)

pak system requirements for existing packages

ubuntu 16.04 sys requirements
❯❯ pkgs %>% setNames(., .)%>% lapply(pak::pkg_system_requirements, "ubuntu", "16.04")
$animation
[1] "apt-get install -y imagemagick"          "apt-get install -y libmagick++-dev"
[3] "apt-get install -y texlive"              "apt-get install -y swftools"
[5] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"

$av
character(0)

$convertGraph
character(0)

$docxtractr
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libxml2-dev"

$gdalUtils
[1] "apt-get install -y software-properties-common" "add-apt-repository -y ppa:ubuntugis/ppa"
[3] "apt-get update"                                "apt-get install -y libgdal-dev"
[5] "apt-get install -y gdal-bin"                   "apt-get install -y libproj-dev"

$gert
[1] "apt-get install -y software-properties-common" "add-apt-repository -y ppa:cran/libgit2"
[3] "apt-get update"                                "apt-get install -y libcurl4-openssl-dev"
[5] "apt-get install -y libssl-dev"                 "apt-get install -y git"
[7] "apt-get install -y libgit2-dev"

$gmp
[1] "apt-get install -y libgmp3-dev"

$gridGraphics
character(0)

$gsl
[1] "apt-get install -y libgsl0-dev"

$hunspell
character(0)

$igraph
[1] "apt-get install -y libglpk-dev" "apt-get install -y libgmp3-dev" "apt-get install -y libxml2-dev"

$jqr
character(0)

$keyring
[1] "apt-get install -y libsodium-dev"   "apt-get install -y libsecret-1-dev" "apt-get install -y libssl-dev"

$lwgeom
[1] "apt-get install -y software-properties-common" "add-apt-repository -y ppa:ubuntugis/ppa"
[3] "apt-get update"                                "apt-get install -y libgeos-dev"
[5] "apt-get install -y libproj-dev"                "apt-get install -y libgdal-dev"
[7] "apt-get install -y gdal-bin"                   "apt-get install -y libudunits2-dev"

$magick
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y imagemagick"          "apt-get install -y libmagick++-dev"

$ncdf4
[1] "apt-get install -y libnetcdf-dev"

$openbabel
character(0)

$pagedown
[1] "apt-get install -y libicu-dev" "apt-get install -y pandoc"     "apt-get install -y make"
[4] "apt-get install -y libssl-dev"

$pdftools
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libpoppler-cpp-dev"

$pensieve
character(0)

$ragg
[1] "apt-get install -y libfontconfig1-dev" "apt-get install -y libfreetype6-dev"
[3] "apt-get install -y libjpeg-dev"        "apt-get install -y libpng-dev"
[5] "apt-get install -y libtiff-dev"        "apt-get install -y libfribidi-dev"
[7] "apt-get install -y libharfbuzz-dev"

$RAmazonS3
character(0)

$rcbc
character(0)

$rcdd
[1] "apt-get install -y libgmp3-dev"

$rDEA
[1] "apt-get install -y libglpk-dev"

$redland
[1] "apt-get install -y libxml2-dev" "apt-get install -y libicu-dev"  "apt-get install -y librdf0-dev"
[4] "apt-get install -y pandoc"

$rgdal
[1] "apt-get install -y software-properties-common" "add-apt-repository -y ppa:ubuntugis/ppa"
[3] "apt-get update"                                "apt-get install -y libgdal-dev"
[5] "apt-get install -y gdal-bin"                   "apt-get install -y libproj-dev"

$rgeos
[1] "apt-get install -y software-properties-common" "add-apt-repository -y ppa:ubuntugis/ppa"
[3] "apt-get update"                                "apt-get install -y libgeos-dev"

$rgl
 [1] "apt-get install -y imagemagick"      "apt-get install -y libmagick++-dev"  "apt-get install -y make"
 [4] "apt-get install -y libicu-dev"       "apt-get install -y pandoc"           "apt-get install -y libfreetype6-dev"
 [7] "apt-get install -y libglu1-mesa-dev" "apt-get install -y libpng-dev"       "apt-get install -y libgl1-mesa-dev"
[10] "apt-get install -y zlib1g-dev"

$Rglpk
[1] "apt-get install -y libglpk-dev"

$rJava
[1] "apt-get install -y make"        "apt-get install -y default-jdk"

$Rmpfr
[1] "apt-get install -y libgmp3-dev" "apt-get install -y libmpfr-dev"

$Rpoppler
[1] "apt-get install -y libglib2.0-dev"     "apt-get install -y libpoppler-cpp-dev"

$rPython
character(0)

$RSelenium
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libxml2-dev"

$rsvg
[1] "apt-get install -y librsvg2-dev"

$Rsymphony
character(0)

$seewave
[1] "apt-get install -y libsndfile1-dev"

$sf
[1] "apt-get install -y software-properties-common" "add-apt-repository -y ppa:ubuntugis/ppa"
[3] "apt-get update"                                "apt-get install -y libgdal-dev"
[5] "apt-get install -y gdal-bin"                   "apt-get install -y libgeos-dev"
[7] "apt-get install -y libproj-dev"                "apt-get install -y libudunits2-dev"

$sodium
[1] "apt-get install -y libsodium-dev"

$ssh
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y git"                  "apt-get install -y libssh2-1-dev"

$staplr
[1] "apt-get install -y make"        "apt-get install -y default-jdk" "apt-get install -y libxml2-dev"
[4] "apt-get install -y libicu-dev"

$tensorflow
[1] "apt-get install -y python"

$tesseract
[1] "apt-get install -y libleptonica-dev"     "apt-get install -y libtesseract-dev"
[3] "apt-get install -y tesseract-ocr-eng"    "apt-get install -y libcurl4-openssl-dev"
[5] "apt-get install -y libssl-dev"           "apt-get install -y libpoppler-cpp-dev"

$tiff
[1] "apt-get install -y libjpeg-dev" "apt-get install -y libtiff-dev"

$tkrplot
character(0)

$tm
[1] "apt-get install -y libxml2-dev"

$topicmodels
[1] "apt-get install -y libxml2-dev" "apt-get install -y libgsl0-dev"

$udunits2
[1] "apt-get install -y libudunits2-dev"

$units
[1] "apt-get install -y libudunits2-dev"

$V8
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libv8-dev"

$websocket
[1] "apt-get install -y make"       "apt-get install -y libssl-dev"

$xslt
[1] "apt-get install -y libxml2-dev" "apt-get install -y libxslt-dev"

ubuntu 18.04 sys requirements
$animation
[1] "apt-get install -y imagemagick"          "apt-get install -y libmagick++-dev"
[3] "apt-get install -y texlive"              "apt-get install -y swftools"
[5] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"

$av
character(0)

$convertGraph
character(0)

$docxtractr
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libxml2-dev"

$gdalUtils
[1] "apt-get install -y libgdal-dev" "apt-get install -y gdal-bin"    "apt-get install -y libproj-dev"

$gert
[1] "apt-get install -y software-properties-common" "add-apt-repository -y ppa:cran/libgit2"
[3] "apt-get update"                                "apt-get install -y libcurl4-openssl-dev"
[5] "apt-get install -y libssl-dev"                 "apt-get install -y git"
[7] "apt-get install -y libgit2-dev"

$gmp
[1] "apt-get install -y libgmp3-dev"

$gridGraphics
character(0)

$gsl
[1] "apt-get install -y libgsl0-dev"

$hunspell
character(0)

$igraph
[1] "apt-get install -y libglpk-dev" "apt-get install -y libgmp3-dev" "apt-get install -y libxml2-dev"

$jqr
[1] "apt-get install -y libjq-dev"

$keyring
[1] "apt-get install -y libsodium-dev"   "apt-get install -y libsecret-1-dev" "apt-get install -y libssl-dev"

$lwgeom
[1] "apt-get install -y libgeos-dev"     "apt-get install -y libproj-dev"     "apt-get install -y libgdal-dev"
[4] "apt-get install -y gdal-bin"        "apt-get install -y libudunits2-dev"

$magick
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y imagemagick"          "apt-get install -y libmagick++-dev"

$ncdf4
[1] "apt-get install -y libnetcdf-dev"

$openbabel
character(0)

$pagedown
[1] "apt-get install -y libicu-dev" "apt-get install -y pandoc"     "apt-get install -y make"
[4] "apt-get install -y libssl-dev"

$pdftools
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libpoppler-cpp-dev"

$pensieve
character(0)

$ragg
[1] "apt-get install -y libfontconfig1-dev" "apt-get install -y libfreetype6-dev"
[3] "apt-get install -y libjpeg-dev"        "apt-get install -y libpng-dev"
[5] "apt-get install -y libtiff-dev"        "apt-get install -y libfribidi-dev"
[7] "apt-get install -y libharfbuzz-dev"

$RAmazonS3
character(0)

$rcbc
character(0)

$rcdd
[1] "apt-get install -y libgmp3-dev"

$rDEA
[1] "apt-get install -y libglpk-dev"

$redland
[1] "apt-get install -y libxml2-dev" "apt-get install -y libicu-dev"  "apt-get install -y librdf0-dev"
[4] "apt-get install -y pandoc"

$rgdal
[1] "apt-get install -y libgdal-dev" "apt-get install -y gdal-bin"    "apt-get install -y libproj-dev"

$rgeos
[1] "apt-get install -y libgeos-dev"

$rgl
 [1] "apt-get install -y imagemagick"      "apt-get install -y libmagick++-dev"  "apt-get install -y make"
 [4] "apt-get install -y libicu-dev"       "apt-get install -y pandoc"           "apt-get install -y libfreetype6-dev"
 [7] "apt-get install -y libglu1-mesa-dev" "apt-get install -y libpng-dev"       "apt-get install -y libgl1-mesa-dev"
[10] "apt-get install -y zlib1g-dev"

$Rglpk
[1] "apt-get install -y libglpk-dev"

$rJava
[1] "apt-get install -y make"        "apt-get install -y default-jdk"

$Rmpfr
[1] "apt-get install -y libgmp3-dev" "apt-get install -y libmpfr-dev"

$Rpoppler
[1] "apt-get install -y libglib2.0-dev"     "apt-get install -y libpoppler-cpp-dev"

$rPython
character(0)

$RSelenium
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libxml2-dev"

$rsvg
[1] "apt-get install -y librsvg2-dev"

$Rsymphony
character(0)

$seewave
[1] "apt-get install -y libsndfile1-dev"

$sf
[1] "apt-get install -y libgdal-dev"     "apt-get install -y gdal-bin"        "apt-get install -y libgeos-dev"
[4] "apt-get install -y libproj-dev"     "apt-get install -y libudunits2-dev"

$sodium
[1] "apt-get install -y libsodium-dev"

$ssh
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y git"                  "apt-get install -y libssh2-1-dev"

$staplr
[1] "apt-get install -y make"        "apt-get install -y default-jdk" "apt-get install -y libxml2-dev"
[4] "apt-get install -y libicu-dev"

$tensorflow
[1] "apt-get install -y python"

$tesseract
[1] "apt-get install -y libleptonica-dev"     "apt-get install -y libtesseract-dev"
[3] "apt-get install -y tesseract-ocr-eng"    "apt-get install -y libcurl4-openssl-dev"
[5] "apt-get install -y libssl-dev"           "apt-get install -y libpoppler-cpp-dev"

$tiff
[1] "apt-get install -y libjpeg-dev" "apt-get install -y libtiff-dev"

$tkrplot
character(0)

$tm
[1] "apt-get install -y libxml2-dev"

$topicmodels
[1] "apt-get install -y libxml2-dev" "apt-get install -y libgsl0-dev"

$udunits2
[1] "apt-get install -y libudunits2-dev"

$units
[1] "apt-get install -y libudunits2-dev"

$V8
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libv8-dev"

$websocket
[1] "apt-get install -y make"       "apt-get install -y libssl-dev"

$xslt
[1] "apt-get install -y libxml2-dev" "apt-get install -y libxslt-dev"
ubuntu 20.04 sys requirements
$animation
[1] "apt-get install -y imagemagick"          "apt-get install -y libmagick++-dev"
[3] "apt-get install -y texlive"              "apt-get install -y libcurl4-openssl-dev"
[5] "apt-get install -y libssl-dev"

$av
character(0)

$convertGraph
character(0)

$docxtractr
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libxml2-dev"

$gdalUtils
[1] "apt-get install -y libgdal-dev" "apt-get install -y gdal-bin"    "apt-get install -y libproj-dev"

$gert
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y git"                  "apt-get install -y libgit2-dev"

$gmp
[1] "apt-get install -y libgmp3-dev"

$gridGraphics
character(0)

$gsl
[1] "apt-get install -y libgsl0-dev"

$hunspell
character(0)

$igraph
[1] "apt-get install -y libglpk-dev" "apt-get install -y libgmp3-dev" "apt-get install -y libxml2-dev"

$jqr
[1] "apt-get install -y libjq-dev"

$keyring
[1] "apt-get install -y libsodium-dev"   "apt-get install -y libsecret-1-dev" "apt-get install -y libssl-dev"

$lwgeom
[1] "apt-get install -y libgeos-dev"     "apt-get install -y libproj-dev"     "apt-get install -y libgdal-dev"
[4] "apt-get install -y gdal-bin"        "apt-get install -y libudunits2-dev"

$magick
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y imagemagick"          "apt-get install -y libmagick++-dev"

$ncdf4
[1] "apt-get install -y libnetcdf-dev"

$openbabel
character(0)

$pagedown
[1] "apt-get install -y libicu-dev" "apt-get install -y pandoc"     "apt-get install -y make"
[4] "apt-get install -y libssl-dev"

$pdftools
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libpoppler-cpp-dev"

$pensieve
character(0)

$ragg
[1] "apt-get install -y libfontconfig1-dev" "apt-get install -y libfreetype6-dev"
[3] "apt-get install -y libjpeg-dev"        "apt-get install -y libpng-dev"
[5] "apt-get install -y libtiff-dev"        "apt-get install -y libfribidi-dev"
[7] "apt-get install -y libharfbuzz-dev"

$RAmazonS3
character(0)

$rcbc
character(0)

$rcdd
[1] "apt-get install -y libgmp3-dev"

$rDEA
[1] "apt-get install -y libglpk-dev"

$redland
[1] "apt-get install -y libxml2-dev" "apt-get install -y libicu-dev"  "apt-get install -y librdf0-dev"
[4] "apt-get install -y pandoc"

$rgdal
[1] "apt-get install -y libgdal-dev" "apt-get install -y gdal-bin"    "apt-get install -y libproj-dev"

$rgeos
[1] "apt-get install -y libgeos-dev"

$rgl
 [1] "apt-get install -y imagemagick"      "apt-get install -y libmagick++-dev"  "apt-get install -y make"
 [4] "apt-get install -y libicu-dev"       "apt-get install -y pandoc"           "apt-get install -y libfreetype6-dev"
 [7] "apt-get install -y libglu1-mesa-dev" "apt-get install -y libpng-dev"       "apt-get install -y libgl1-mesa-dev"
[10] "apt-get install -y zlib1g-dev"

$Rglpk
[1] "apt-get install -y libglpk-dev"

$rJava
[1] "apt-get install -y make"        "apt-get install -y default-jdk"

$Rmpfr
[1] "apt-get install -y libgmp3-dev" "apt-get install -y libmpfr-dev"

$Rpoppler
[1] "apt-get install -y libglib2.0-dev"     "apt-get install -y libpoppler-cpp-dev"

$rPython
character(0)

$RSelenium
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libxml2-dev"

$rsvg
[1] "apt-get install -y librsvg2-dev"

$Rsymphony
character(0)

$seewave
[1] "apt-get install -y libsndfile1-dev"

$sf
[1] "apt-get install -y libgdal-dev"     "apt-get install -y gdal-bin"        "apt-get install -y libgeos-dev"
[4] "apt-get install -y libproj-dev"     "apt-get install -y libudunits2-dev"

$sodium
[1] "apt-get install -y libsodium-dev"

$ssh
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y git"                  "apt-get install -y libssh2-1-dev"

$staplr
[1] "apt-get install -y make"        "apt-get install -y default-jdk" "apt-get install -y libxml2-dev"
[4] "apt-get install -y libicu-dev"

$tensorflow
[1] "apt-get install -y python"

$tesseract
[1] "apt-get install -y libleptonica-dev"     "apt-get install -y libtesseract-dev"
[3] "apt-get install -y tesseract-ocr-eng"    "apt-get install -y libcurl4-openssl-dev"
[5] "apt-get install -y libssl-dev"           "apt-get install -y libpoppler-cpp-dev"

$tiff
[1] "apt-get install -y libjpeg-dev" "apt-get install -y libtiff-dev"

$tkrplot
character(0)

$tm
[1] "apt-get install -y libxml2-dev"

$topicmodels
[1] "apt-get install -y libxml2-dev" "apt-get install -y libgsl0-dev"

$udunits2
[1] "apt-get install -y libudunits2-dev"

$units
[1] "apt-get install -y libudunits2-dev"

$V8
[1] "apt-get install -y libcurl4-openssl-dev" "apt-get install -y libssl-dev"
[3] "apt-get install -y libv8-dev"

$websocket
[1] "apt-get install -y make"       "apt-get install -y libssl-dev"

$xslt
[1] "apt-get install -y libxml2-dev" "apt-get install -y libxslt-dev"

GitHub Actions Testing

Testing all packages on focal and xenial. Packages that have an install file will use it, otherwise pak will be used.

https://github.com/schloerke/shinyapps-package-dependencies/runs/2143004651?check_suite_focus=true


cc @samperman

* master:
  Standardize on OS_CODENAME instead of OS_DISTRIBUTION
  update the CHANGELOG.md to reflect removed packages
  our base images no longer need extra things installed for rPython to work.
  neither focal nor xenial need this installed for this package to work
  remove vivid/trusty/precise, only worry about xenial and focal
  PR feedback
  rcbc has some different install requirements, and also needs -y in the install step.  It was never actually doing anything
  fix openbabel for focal, requires libeigen3-dev, which is a source package only on focal...
  python-software-properties doesn't exist in focal it seems
  do not support docxtractr on focal
  on travis this is prompting for user input, make sure it doesn't
  hopeful changelog updates
  add focal Dockerfile, and change Makefile to have focal tests instead of bionic
  still not working for focal
  update packages to support focal
  Update CHANGELOG.md
  add support for focal to animation
  add focal dockerfile and update the makefile to support it.  Remove bionic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

merge with r-system-requirements
1 participant