diff --git a/docs/404.html b/docs/404.html index 3db9ef5..01d70be 100644 --- a/docs/404.html +++ b/docs/404.html @@ -23,7 +23,7 @@ - + diff --git a/docs/AP.html b/docs/AP.html index 141de2f..303038c 100644 --- a/docs/AP.html +++ b/docs/AP.html @@ -23,7 +23,7 @@ - + diff --git a/docs/CC.html b/docs/CC.html index 60b0c61..d2231f6 100644 --- a/docs/CC.html +++ b/docs/CC.html @@ -23,7 +23,7 @@ - + @@ -147,17 +147,16 @@

4.1 About Coffee and Coding

4.2 Recorded presentations

Where possible, Coffee and Coding presentations are recorded to allow people to catch up on talks they may have missed and to provide a useful training resource for colleagues.

-

These recordings are available on the Coffee and Coding Microsoft Stream Channel. We regularly update the access list, but if you cannot access the videos please contact the Coffee and Coding mailbox.

+

These recordings are available on the Coffee and Coding Sharepoint page. We regularly update the access list, but if you cannot access the videos please contact the Coffee and Coding mailbox.

4.2.1 Some useful talks

If you are new to coding in MoJ, you might find the following recorded talks useful:-

-

The full catalogue of recorded talks can be found here.

+

The full archive of recorded talks can be found here.

diff --git a/docs/CTG.html b/docs/CTG.html index a90b4fc..1958842 100644 --- a/docs/CTG.html +++ b/docs/CTG.html @@ -23,7 +23,7 @@ - + diff --git a/docs/index.html b/docs/index.html index c4961d9..e43154a 100644 --- a/docs/index.html +++ b/docs/index.html @@ -23,7 +23,7 @@ - + @@ -133,7 +133,7 @@

1 Summary of available resources

diff --git a/docs/search_index.json b/docs/search_index.json index b2efb8d..04b2c69 100644 --- a/docs/search_index.json +++ b/docs/search_index.json @@ -1 +1 @@ -[["index.html", "Analytical Platform and related tools training 1 Summary of available resources 1.1 Analytical IT Tools Strategy 1.2 Analytical Platform 1.3 Coding Mentoring Scheme 1.4 Coding Training Groups 1.5 Coffee and Coding 1.6 Bite-sized sessions 1.7 Analytical Function training 1.8 DataCamp 1.9 Slack 1.10 Other assistance", " Analytical Platform and related tools training MoJ coding training leads (the Analytical Platform team is not responsible for the content on this page) 2024-05-08 1 Summary of available resources 1.1 Analytical IT Tools Strategy The MoJ Analytical IT Tools Strategy describes the analytical IT tools analysts are recommended to use along with the ways of working to be followed. 1.2 Analytical Platform The Analytical Platform is a data analysis environment, enabling the use of modern open source tools such as R and Python, and holding key datasets for MoJ analysts. To learn more about the Analytical Platform and to get up and running, go to the Analytical Platform chapter. More extensive information is provided by the Analytical Platform user guidance. 1.3 Coding Mentoring Scheme If you are new to coding (in any language or in a new coding language), or if you are new to Data and Analysis and your role involves coding, then it is recommended that you request a coding mentor. The purpose of the scheme is to provide a better on-the-job coding learning experience and raise awareness of the preferred D&A ways of working that will for instance enable people to get up to speed more quickly with others’ code. The scheme is also open to non-coders who need to use the Analytical Platform to advise them through the learning process, and for those needing help with developing the reproducibility of a coding product and/or pipeline or commencing a more complex coding project. To request a coding mentor please complete this mentee form. If you would like to become a coding mentor please complete this mentor form. For more information please contact Jose Vieira. 1.4 Coding Training Groups Those working in Data and Analysis are recommended to take the internal training courses in R, SQL, Python and Git/GitHub as they are run using the MoJ Analytical Platform. The main introductory R, SQL and Git/GitHub sessions are usually run live in February/March, June/July and October/November each year while you can also work through R, SQL, Git/GitHub and Python sessions yourself using the training material and/or recordings. To learn more about the sessions currently available and how to access the material and recordings, go to the Coding Training Groups chapter. 1.5 Coffee and Coding The internal training (see above) is complemented by Coffee and Coding presentations. These presentations usually take the form of a demonstration of a tool or technique and/or a show and tell of work done within the department using particular coding methods. For more information go to the Coffee and Coding chapter. 1.6 Bite-sized sessions The Data and Analysis bite-sized sessions are short sessions (generally talks of up to 15mins followed by Q/A) on a variety of topics including specialist as well as softer skills. Recent topics have included: New statistical methods guidance for MoJ analysts How to use the Evidence Library to support your work Prototyping - why, how and when? Key components of RAP in MoJ coding projects ChatGPT for coding - Know your Frenemy Hints and tips on public speaking What are faith and belief? You can find recordings of bite-sized sessions in the bite-sized session video library. For more information including if you are interested in presenting a session or joining the bite-sized session facilitation team please contact Aidan Mews or Edward Adams. 1.7 Analytical Function training There are many Analytical Function training opportunities for analysts including about specialist topics not presently covered internally. Examples useful for RAP practitioners include Best practice in programming – clean code and a more lengthy Introduction to unit testing than currently available internally. You can learn more about such opportunities via: The Analytical Function Learning Curriculam The Analytical Function Training Courses 1.8 DataCamp DataCamp licenses are beneficial to cover gaps in current training provision that are not currently picked up by either internal or Analytical Function training e.g. training in Power BI and more advanced or niche R, SQL, and Python skills. Read more about DataCamp and signup for a license (paid for by MoJ) via this landing page View this March 2024 DataCamp Licence: Onboarding and Q&A Session and associated onboarding slides To access DataCamp on your mobile see this internet link DataCamp courses currently in development are shown by this Trello board Reach out to the DataCamp support team for any technical requests via this internet link 1.9 Slack Technical help can be requested via the following Data and Analysis slack channels: ask-operations-engineering - this provides support to those with Analytical Platform issues intro_R - this provides support to those starting out in the world of R R - this is for beginners and experts alike sql git python RAP More information about Slack including how to get set up is available via the Analytical Platform User Guidance and the Data and Analysis Hive. 1.10 Other assistance You may also find useful: Statistical Methods Guidance on key topics for MoJ analysts with many of the signposted sources including R or Python code. a trello board providing links to further free online analytical training including in R or Python and R cheatsheets. There are also many useful free R and Python books on the web, for instance for Python: Automate The Boring Stuff with Python (Good for Python Fundamentals) Python Data Science Handbook "],["AP.html", "2 Analytical Platform 2.1 Introduction 2.2 Summary of key terms 2.3 Getting set up 2.4 Managing data", " 2 Analytical Platform 2.1 Introduction To gain an overview of the Analytical Platform watch this 2-3 min introductory video. For more technical details, you can also read the user guidance section About the Analytical Platform. 2.2 Summary of key terms It will help you to be familiar with the following key terms: Analytical Platform (AP): A data analysis environment providing modern tools and key datasets for MoJ analysts. AP contains training documents, resources, and access to various analytical software such as Rstudio and Jupyter. Control Panel: A place to navigate to Rstudio, Jupyter, S3 Buckets etc RStudio: Development environment for writing R code and R Shiny apps JupyterLab: Development environment for writing Python code including Python notebooks Git: Version control software that enables multiple people to make separate changes at the same time. GitHub: A web-based interface that uses Git and on which you publish and share your version-controlled code. You use Git locally (e.g. using RStudio) to track versions of your code, and then submit those changes to Github. GitHub Repositories (Repo): Similar to setting up a project folder on DOM1 shared drive to save work and share with others. Files on Github Repos represent the definitive version of the project. Everyone who works on the project makes contributions from their own personal versions. Amazon S3: A web-based cloud storage platform. Where your home directory stores working copies of code, data and final analytical outputs should be stored in Amazon S3. Access to amazon S3 buckets can be managed. Slack: Collaboration tool where you can get technical support for Analytical Platform tools such as R, Python, Git. You can share knowledge, submit admin requests and communicate quickly with other AP users. 2.3 Getting set up Follow the steps in the Getting Started section of the Analytical Platform User Guide. You need to: Set up a Slack account. Set up a GitHub account with two-factor authentication. Set up a Analytical Platform account. Set up RStudio to use R and/or JupyterLab to use Python. You can learn more about GitHub by viewing or attending the internal Introduction to Git/GitHub. For those that need to get set up to use Athena databases for SQL (in R or Athena) on the Analytical Platform, please follow the additional instructions in the “Training Requirements” section of the Introduction to SQL training repository. 2.4 Managing data Once you have got set up on the Analytical Platform, do read about the following data management/handling topics: How data are held on the Analytical Platform and finding the data you need. You can read about the three different data storage options (Amazon S3, Curated databases and home directories). Working with Amazon S3, data FAQ, the Data Uploader tool and interacting with Amazon S3 via the Analytical Platform. Information governance procedures to be followed. Data retention policies including when deleting data means they are permanently deleted. "],["CTG.html", "3 Coding Training Groups 3.1 R Training 3.2 Git/GitHub Training 3.3 SQL Training 3.4 Python Training 3.5 Get involved!", " 3 Coding Training Groups Those working in Data and Analysis are recommended to take the internal training courses as they are run using the MoJ Analytical Platform and focus on topics of relevance to MoJ analysts. The main introductory R, SQL and Git/GitHub sessions are usually run live in February/March, June/July and October/November each year while you can also work through R, SQL, Git/GitHub and Python sessions yourself using the training material and/or recordings. There is also Python training you can work through yourself. 3.1 R Training The following sessions are available; click on the links to view the material in the GitHub repositories. The first three are particularly recommended for new starters. Introduction: Using R on the Analytical Platform Introduction to R R Charting Introduction to R + R Markdown Interfacing Excel with R Writing Functions in R Developing R packages & RAP ways of working The recordings of these R training sessions are available in the R/SQL/Git Training Video Library. 3.2 Git/GitHub Training Whether seeking to use R, SQL or Python, it is recommended that all newcomers attend or work through themselves this Introduction to Git/GitHub (click on link to view material in GitHub repository) session. A recording can be viewed in the R/SQL/Git Training Video Library. 3.3 SQL Training All newcomers who are to use SQL should attend or work through themselves the Introduction to SQL (click on link to view material in GitHub repository) session. A recording of this session is available in the R/SQL/Git Training Video Library 3.4 Python Training These have not yet been run as live sessions, but you can work through the material at your own pace. Introduction to Python Iterables in Python Unit Testing and GitHub Actions AWS Tooling demos Pyspark in AP Prisoid Academy - Python Training (if you have access to NOMIS) Plotly Dash Coffee and Coding Session 3.5 Get involved! One great way of learning is by teaching. If you would be interested in playing an active role, whether delivering, updating or designing training, or co-ordinating the courses, please contact Aidan Mews (R), Georgina Eaton (SQL), or Mike Ratford (Python). If you have any questions please contact Aidan Mews, Georgina Eaton or Mike Ratford. "],["CC.html", "4 Coffee and Coding 4.1 About Coffee and Coding 4.2 Recorded presentations 4.3 Code repositories 4.4 Get involved!", " 4 Coffee and Coding 4.1 About Coffee and Coding The aim of the MoJ Coffee and Coding presentations is to nurture, enable and encourage a vibrant, supportive and inclusive coding community at MoJ. Coffee & Coding provides a regular opportunity for people within MoJ who are interested in coding to: Share skills and knowledge Showcase their work and projects Provide advice and guidance to each other Network and get to know each other Code can be shared from any language — whether R, Python, SQL, or other more specialised languages. The format is generally 30–40 minutes for a presentation, followed by time for questions. The presentations usually take the form of a demonstration of a tool or technique and/or a show and tell of work done within the department using these methods. 4.2 Recorded presentations Where possible, Coffee and Coding presentations are recorded to allow people to catch up on talks they may have missed and to provide a useful training resource for colleagues. These recordings are available on the Coffee and Coding Microsoft Stream Channel. We regularly update the access list, but if you cannot access the videos please contact the Coffee and Coding mailbox. 4.2.1 Some useful talks If you are new to coding in MoJ, you might find the following recorded talks useful:- An Introduction to GitHub Ten Top Tips for Data Presentation An Introduction to Airflow 2.0 Package Management in renv The full catalogue of recorded talks can be found here. 4.3 Code repositories You can find code examples to go along with past Coffee and Coding presentations in our private repo (only accessible by members of the MoJ Analytical Services GitHub organisation). 4.4 Get involved! We are always looking for new presentations. If you have an idea for a talk you would like to give, or if you want to showcase a coding project you have been working on, please contact the Coffee and Coding mailbox. If you have feedback or suggestions for the Coffee and Coding team, we would love to hear from you. You can respond to our open feedback form here. "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]] +[["index.html", "Analytical Platform and related tools training 1 Summary of available resources 1.1 Analytical IT Tools Strategy 1.2 Analytical Platform 1.3 Coding Mentoring Scheme 1.4 Coding Training Groups 1.5 Coffee and Coding 1.6 Bite-sized sessions 1.7 Analytical Function training 1.8 DataCamp 1.9 Slack 1.10 Other assistance", " Analytical Platform and related tools training MoJ coding training leads (the Analytical Platform team is not responsible for the content on this page) 2024-05-17 1 Summary of available resources 1.1 Analytical IT Tools Strategy The MoJ Analytical IT Tools Strategy describes the analytical IT tools analysts are recommended to use along with the ways of working to be followed. 1.2 Analytical Platform The Analytical Platform is a data analysis environment, enabling the use of modern open source tools such as R and Python, and holding key datasets for MoJ analysts. To learn more about the Analytical Platform and to get up and running, go to the Analytical Platform chapter. More extensive information is provided by the Analytical Platform user guidance. 1.3 Coding Mentoring Scheme If you are new to coding (in any language or in a new coding language), or if you are new to Data and Analysis and your role involves coding, then it is recommended that you request a coding mentor. The purpose of the scheme is to provide a better on-the-job coding learning experience and raise awareness of the preferred D&A ways of working that will for instance enable people to get up to speed more quickly with others’ code. The scheme is also open to non-coders who need to use the Analytical Platform to advise them through the learning process, and for those needing help with developing the reproducibility of a coding product and/or pipeline or commencing a more complex coding project. To request a coding mentor please complete this mentee form. If you would like to become a coding mentor please complete this mentor form. For more information please contact Jose Vieira. 1.4 Coding Training Groups Those working in Data and Analysis are recommended to take the internal training courses in R, SQL, Python and Git/GitHub as they are run using the MoJ Analytical Platform. The main introductory R, SQL and Git/GitHub sessions are usually run live in February/March, June/July and October/November each year while you can also work through R, SQL, Git/GitHub and Python sessions yourself using the training material and/or recordings. To learn more about the sessions currently available and how to access the material and recordings, go to the Coding Training Groups chapter. 1.5 Coffee and Coding The internal training (see above) is complemented by Coffee and Coding presentations. These presentations usually take the form of a demonstration of a tool or technique and/or a show and tell of work done within the department using particular coding methods. For more information go to the Coffee and Coding chapter. 1.6 Bite-sized sessions The Data and Analysis bite-sized sessions are short sessions (generally talks of up to 15mins followed by Q/A) on a variety of topics including specialist as well as softer skills. Recent topics have included: New statistical methods guidance for MoJ analysts How to use the Evidence Library to support your work Prototyping - why, how and when? Key components of RAP in MoJ coding projects ChatGPT for coding - Know your Frenemy Hints and tips on public speaking What are faith and belief? You can find recordings of bite-sized sessions in the bite-sized session video library. For more information including if you are interested in presenting a session or joining the bite-sized session facilitation team please contact Aidan Mews or Edward Adams. 1.7 Analytical Function training There are many Analytical Function training opportunities for analysts including about specialist topics not presently covered internally. Examples useful for RAP practitioners include Best practice in programming – clean code and a more lengthy Introduction to unit testing than currently available internally. You can learn more about such opportunities via: The Analytical Function Learning Curriculam The Analytical Function Training Courses 1.8 DataCamp DataCamp licenses are beneficial to cover gaps in current training provision that are not currently picked up by either internal or Analytical Function training e.g. training in Power BI and more advanced or niche R, SQL, and Python skills. Read more about DataCamp and signup for a license (paid for by MoJ) via this landing page View this March 2024 DataCamp Licence: Onboarding and Q&A Session and associated onboarding slides To access DataCamp on your mobile see this internet link DataCamp courses currently in development are shown by this Trello board Reach out to the DataCamp support team for any technical requests via this internet link 1.9 Slack Technical help can be requested via the following Data and Analysis slack channels: ask-operations-engineering - this provides support to those with Analytical Platform issues intro_R - this provides support to those starting out in the world of R R - this is for beginners and experts alike sql git python RAP More information about Slack including how to get set up is available via the Analytical Platform User Guidance and the Data and Analysis Hive. 1.10 Other assistance You may also find useful: Statistical Methods Guidance on key topics for MoJ analysts with many of the signposted sources including R or Python code. a trello board providing links to further free online analytical training including in R or Python and R cheatsheets. There are also many useful free R and Python books on the web, for instance for Python: Automate The Boring Stuff with Python (Good for Python Fundamentals) Python Data Science Handbook "],["AP.html", "2 Analytical Platform 2.1 Introduction 2.2 Summary of key terms 2.3 Getting set up 2.4 Managing data", " 2 Analytical Platform 2.1 Introduction To gain an overview of the Analytical Platform watch this 2-3 min introductory video. For more technical details, you can also read the user guidance section About the Analytical Platform. 2.2 Summary of key terms It will help you to be familiar with the following key terms: Analytical Platform (AP): A data analysis environment providing modern tools and key datasets for MoJ analysts. AP contains training documents, resources, and access to various analytical software such as Rstudio and Jupyter. Control Panel: A place to navigate to Rstudio, Jupyter, S3 Buckets etc RStudio: Development environment for writing R code and R Shiny apps JupyterLab: Development environment for writing Python code including Python notebooks Git: Version control software that enables multiple people to make separate changes at the same time. GitHub: A web-based interface that uses Git and on which you publish and share your version-controlled code. You use Git locally (e.g. using RStudio) to track versions of your code, and then submit those changes to Github. GitHub Repositories (Repo): Similar to setting up a project folder on DOM1 shared drive to save work and share with others. Files on Github Repos represent the definitive version of the project. Everyone who works on the project makes contributions from their own personal versions. Amazon S3: A web-based cloud storage platform. Where your home directory stores working copies of code, data and final analytical outputs should be stored in Amazon S3. Access to amazon S3 buckets can be managed. Slack: Collaboration tool where you can get technical support for Analytical Platform tools such as R, Python, Git. You can share knowledge, submit admin requests and communicate quickly with other AP users. 2.3 Getting set up Follow the steps in the Getting Started section of the Analytical Platform User Guide. You need to: Set up a Slack account. Set up a GitHub account with two-factor authentication. Set up a Analytical Platform account. Set up RStudio to use R and/or JupyterLab to use Python. You can learn more about GitHub by viewing or attending the internal Introduction to Git/GitHub. For those that need to get set up to use Athena databases for SQL (in R or Athena) on the Analytical Platform, please follow the additional instructions in the “Training Requirements” section of the Introduction to SQL training repository. 2.4 Managing data Once you have got set up on the Analytical Platform, do read about the following data management/handling topics: How data are held on the Analytical Platform and finding the data you need. You can read about the three different data storage options (Amazon S3, Curated databases and home directories). Working with Amazon S3, data FAQ, the Data Uploader tool and interacting with Amazon S3 via the Analytical Platform. Information governance procedures to be followed. Data retention policies including when deleting data means they are permanently deleted. "],["CTG.html", "3 Coding Training Groups 3.1 R Training 3.2 Git/GitHub Training 3.3 SQL Training 3.4 Python Training 3.5 Get involved!", " 3 Coding Training Groups Those working in Data and Analysis are recommended to take the internal training courses as they are run using the MoJ Analytical Platform and focus on topics of relevance to MoJ analysts. The main introductory R, SQL and Git/GitHub sessions are usually run live in February/March, June/July and October/November each year while you can also work through R, SQL, Git/GitHub and Python sessions yourself using the training material and/or recordings. There is also Python training you can work through yourself. 3.1 R Training The following sessions are available; click on the links to view the material in the GitHub repositories. The first three are particularly recommended for new starters. Introduction: Using R on the Analytical Platform Introduction to R R Charting Introduction to R + R Markdown Interfacing Excel with R Writing Functions in R Developing R packages & RAP ways of working The recordings of these R training sessions are available in the R/SQL/Git Training Video Library. 3.2 Git/GitHub Training Whether seeking to use R, SQL or Python, it is recommended that all newcomers attend or work through themselves this Introduction to Git/GitHub (click on link to view material in GitHub repository) session. A recording can be viewed in the R/SQL/Git Training Video Library. 3.3 SQL Training All newcomers who are to use SQL should attend or work through themselves the Introduction to SQL (click on link to view material in GitHub repository) session. A recording of this session is available in the R/SQL/Git Training Video Library 3.4 Python Training These have not yet been run as live sessions, but you can work through the material at your own pace. Introduction to Python Iterables in Python Unit Testing and GitHub Actions AWS Tooling demos Pyspark in AP Prisoid Academy - Python Training (if you have access to NOMIS) Plotly Dash Coffee and Coding Session 3.5 Get involved! One great way of learning is by teaching. If you would be interested in playing an active role, whether delivering, updating or designing training, or co-ordinating the courses, please contact Aidan Mews (R), Georgina Eaton (SQL), or Mike Ratford (Python). If you have any questions please contact Aidan Mews, Georgina Eaton or Mike Ratford. "],["CC.html", "4 Coffee and Coding 4.1 About Coffee and Coding 4.2 Recorded presentations 4.3 Code repositories 4.4 Get involved!", " 4 Coffee and Coding 4.1 About Coffee and Coding The aim of the MoJ Coffee and Coding presentations is to nurture, enable and encourage a vibrant, supportive and inclusive coding community at MoJ. Coffee & Coding provides a regular opportunity for people within MoJ who are interested in coding to: Share skills and knowledge Showcase their work and projects Provide advice and guidance to each other Network and get to know each other Code can be shared from any language — whether R, Python, SQL, or other more specialised languages. The format is generally 30–40 minutes for a presentation, followed by time for questions. The presentations usually take the form of a demonstration of a tool or technique and/or a show and tell of work done within the department using these methods. 4.2 Recorded presentations Where possible, Coffee and Coding presentations are recorded to allow people to catch up on talks they may have missed and to provide a useful training resource for colleagues. These recordings are available on the Coffee and Coding Sharepoint page. We regularly update the access list, but if you cannot access the videos please contact the Coffee and Coding mailbox. 4.2.1 Some useful talks If you are new to coding in MoJ, you might find the following recorded talks useful:- Ten top tips for data presentation Package management in renv Developing an Accessible and gov.uk styled dashboard The full archive of recorded talks can be found here. 4.3 Code repositories You can find code examples to go along with past Coffee and Coding presentations in our private repo (only accessible by members of the MoJ Analytical Services GitHub organisation). 4.4 Get involved! We are always looking for new presentations. If you have an idea for a talk you would like to give, or if you want to showcase a coding project you have been working on, please contact the Coffee and Coding mailbox. If you have feedback or suggestions for the Coffee and Coding team, we would love to hear from you. You can respond to our open feedback form here. "],["404.html", "Page not found", " Page not found The page you requested cannot be found (perhaps it was moved or renamed). You may want to try searching to find the page's new location, or use the table of contents to find the page you are looking for. "]] diff --git a/renv/activate.R b/renv/activate.R index 019b5a6..d13f993 100644 --- a/renv/activate.R +++ b/renv/activate.R @@ -2,10 +2,28 @@ local({ # the requested version of renv - version <- "0.16.0" + version <- "1.0.7" + attr(version, "sha") <- NULL # the project directory - project <- getwd() + project <- Sys.getenv("RENV_PROJECT") + if (!nzchar(project)) + project <- getwd() + + # use start-up diagnostics if enabled + diagnostics <- Sys.getenv("RENV_STARTUP_DIAGNOSTICS", unset = "FALSE") + if (diagnostics) { + start <- Sys.time() + profile <- tempfile("renv-startup-", fileext = ".Rprof") + utils::Rprof(profile) + on.exit({ + utils::Rprof(NULL) + elapsed <- signif(difftime(Sys.time(), start, units = "auto"), digits = 2L) + writeLines(sprintf("- renv took %s to run the autoloader.", format(elapsed))) + writeLines(sprintf("- Profile: %s", profile)) + print(utils::summaryRprof(profile)) + }, add = TRUE) + } # figure out whether the autoloader is enabled enabled <- local({ @@ -15,6 +33,14 @@ local({ if (!is.null(override)) return(override) + # if we're being run in a context where R_LIBS is already set, + # don't load -- presumably we're being run as a sub-process and + # the parent process has already set up library paths for us + rcmd <- Sys.getenv("R_CMD", unset = NA) + rlibs <- Sys.getenv("R_LIBS", unset = NA) + if (!is.na(rlibs) && !is.na(rcmd)) + return(FALSE) + # next, check environment variables # TODO: prefer using the configuration one in the future envvars <- c( @@ -34,9 +60,22 @@ local({ }) - if (!enabled) + # bail if we're not enabled + if (!enabled) { + + # if we're not enabled, we might still need to manually load + # the user profile here + profile <- Sys.getenv("R_PROFILE_USER", unset = "~/.Rprofile") + if (file.exists(profile)) { + cfg <- Sys.getenv("RENV_CONFIG_USER_PROFILE", unset = "TRUE") + if (tolower(cfg) %in% c("true", "t", "1")) + sys.source(profile, envir = globalenv()) + } + return(FALSE) + } + # avoid recursion if (identical(getOption("renv.autoloader.running"), TRUE)) { warning("ignoring recursive attempt to run renv autoloader") @@ -60,21 +99,90 @@ local({ # load bootstrap tools `%||%` <- function(x, y) { - if (is.environment(x) || length(x)) x else y + if (is.null(x)) y else x + } + + catf <- function(fmt, ..., appendLF = TRUE) { + + quiet <- getOption("renv.bootstrap.quiet", default = FALSE) + if (quiet) + return(invisible()) + + msg <- sprintf(fmt, ...) + cat(msg, file = stdout(), sep = if (appendLF) "\n" else "") + + invisible(msg) + + } + + header <- function(label, + ..., + prefix = "#", + suffix = "-", + n = min(getOption("width"), 78)) + { + label <- sprintf(label, ...) + n <- max(n - nchar(label) - nchar(prefix) - 2L, 8L) + if (n <= 0) + return(paste(prefix, label)) + + tail <- paste(rep.int(suffix, n), collapse = "") + paste0(prefix, " ", label, " ", tail) + + } + + heredoc <- function(text, leave = 0) { + + # remove leading, trailing whitespace + trimmed <- gsub("^\\s*\\n|\\n\\s*$", "", text) + + # split into lines + lines <- strsplit(trimmed, "\n", fixed = TRUE)[[1L]] + + # compute common indent + indent <- regexpr("[^[:space:]]", lines) + common <- min(setdiff(indent, -1L)) - leave + paste(substring(lines, common), collapse = "\n") + + } + + startswith <- function(string, prefix) { + substring(string, 1, nchar(prefix)) == prefix } bootstrap <- function(version, library) { + friendly <- renv_bootstrap_version_friendly(version) + section <- header(sprintf("Bootstrapping renv %s", friendly)) + catf(section) + # attempt to download renv - tarball <- tryCatch(renv_bootstrap_download(version), error = identity) - if (inherits(tarball, "error")) - stop("failed to download renv ", version) + catf("- Downloading renv ... ", appendLF = FALSE) + withCallingHandlers( + tarball <- renv_bootstrap_download(version), + error = function(err) { + catf("FAILED") + stop("failed to download:\n", conditionMessage(err)) + } + ) + catf("OK") + on.exit(unlink(tarball), add = TRUE) # now attempt to install - status <- tryCatch(renv_bootstrap_install(version, tarball, library), error = identity) - if (inherits(status, "error")) - stop("failed to install renv ", version) + catf("- Installing renv ... ", appendLF = FALSE) + withCallingHandlers( + status <- renv_bootstrap_install(version, tarball, library), + error = function(err) { + catf("FAILED") + stop("failed to install:\n", conditionMessage(err)) + } + ) + catf("OK") + + # add empty line to break up bootstrapping from normal output + catf("") + return(invisible()) } renv_bootstrap_tests_running <- function() { @@ -83,28 +191,32 @@ local({ renv_bootstrap_repos <- function() { + # get CRAN repository + cran <- getOption("renv.repos.cran", "https://cloud.r-project.org") + # check for repos override repos <- Sys.getenv("RENV_CONFIG_REPOS_OVERRIDE", unset = NA) - if (!is.na(repos)) + if (!is.na(repos)) { + + # check for RSPM; if set, use a fallback repository for renv + rspm <- Sys.getenv("RSPM", unset = NA) + if (identical(rspm, repos)) + repos <- c(RSPM = rspm, CRAN = cran) + return(repos) + } + # check for lockfile repositories repos <- tryCatch(renv_bootstrap_repos_lockfile(), error = identity) if (!inherits(repos, "error") && length(repos)) return(repos) - # if we're testing, re-use the test repositories - if (renv_bootstrap_tests_running()) - return(getOption("renv.tests.repos")) - # retrieve current repos repos <- getOption("repos") # ensure @CRAN@ entries are resolved - repos[repos == "@CRAN@"] <- getOption( - "renv.repos.cran", - "https://cloud.r-project.org" - ) + repos[repos == "@CRAN@"] <- cran # add in renv.bootstrap.repos if set default <- c(FALLBACK = "https://cloud.r-project.org") @@ -143,33 +255,34 @@ local({ renv_bootstrap_download <- function(version) { - # if the renv version number has 4 components, assume it must - # be retrieved via github - nv <- numeric_version(version) - components <- unclass(nv)[[1]] - - # if this appears to be a development version of 'renv', we'll - # try to restore from github - dev <- length(components) == 4L - - # begin collecting different methods for finding renv - methods <- c( - renv_bootstrap_download_tarball, - if (dev) - renv_bootstrap_download_github - else c( - renv_bootstrap_download_cran_latest, - renv_bootstrap_download_cran_archive + sha <- attr(version, "sha", exact = TRUE) + + methods <- if (!is.null(sha)) { + + # attempting to bootstrap a development version of renv + c( + function() renv_bootstrap_download_tarball(sha), + function() renv_bootstrap_download_github(sha) ) - ) + + } else { + + # attempting to bootstrap a release version of renv + c( + function() renv_bootstrap_download_tarball(version), + function() renv_bootstrap_download_cran_latest(version), + function() renv_bootstrap_download_cran_archive(version) + ) + + } for (method in methods) { - path <- tryCatch(method(version), error = identity) + path <- tryCatch(method(), error = identity) if (is.character(path) && file.exists(path)) return(path) } - stop("failed to download renv ", version) + stop("All download methods failed") } @@ -233,8 +346,6 @@ local({ type <- spec$type repos <- spec$repos - message("* Downloading renv ", version, " ... ", appendLF = FALSE) - baseurl <- utils::contrib.url(repos = repos, type = type) ext <- if (identical(type, "source")) ".tar.gz" @@ -251,13 +362,10 @@ local({ condition = identity ) - if (inherits(status, "condition")) { - message("FAILED") + if (inherits(status, "condition")) return(FALSE) - } # report success and return - message("OK (downloaded ", type, ")") destfile } @@ -314,8 +422,6 @@ local({ urls <- file.path(repos, "src/contrib/Archive/renv", name) destfile <- file.path(tempdir(), name) - message("* Downloading renv ", version, " ... ", appendLF = FALSE) - for (url in urls) { status <- tryCatch( @@ -323,14 +429,11 @@ local({ condition = identity ) - if (identical(status, 0L)) { - message("OK") + if (identical(status, 0L)) return(destfile) - } } - message("FAILED") return(FALSE) } @@ -344,8 +447,7 @@ local({ return() # allow directories - info <- file.info(tarball, extra_cols = FALSE) - if (identical(info$isdir, TRUE)) { + if (dir.exists(tarball)) { name <- sprintf("renv_%s.tar.gz", version) tarball <- file.path(tarball, name) } @@ -354,7 +456,7 @@ local({ if (!file.exists(tarball)) { # let the user know we weren't able to honour their request - fmt <- "* RENV_BOOTSTRAP_TARBALL is set (%s) but does not exist." + fmt <- "- RENV_BOOTSTRAP_TARBALL is set (%s) but does not exist." msg <- sprintf(fmt, tarball) warning(msg) @@ -363,10 +465,7 @@ local({ } - fmt <- "* Bootstrapping with tarball at path '%s'." - msg <- sprintf(fmt, tarball) - message(msg) - + catf("- Using local tarball '%s'.", tarball) tarball } @@ -393,8 +492,6 @@ local({ on.exit(do.call(base::options, saved), add = TRUE) } - message("* Downloading renv ", version, " from GitHub ... ", appendLF = FALSE) - url <- file.path("https://api.github.com/repos/rstudio/renv/tarball", version) name <- sprintf("renv_%s.tar.gz", version) destfile <- file.path(tempdir(), name) @@ -404,26 +501,105 @@ local({ condition = identity ) - if (!identical(status, 0L)) { - message("FAILED") + if (!identical(status, 0L)) return(FALSE) - } - message("OK") + renv_bootstrap_download_augment(destfile) + return(destfile) } + # Add Sha to DESCRIPTION. This is stop gap until #890, after which we + # can use renv::install() to fully capture metadata. + renv_bootstrap_download_augment <- function(destfile) { + sha <- renv_bootstrap_git_extract_sha1_tar(destfile) + if (is.null(sha)) { + return() + } + + # Untar + tempdir <- tempfile("renv-github-") + on.exit(unlink(tempdir, recursive = TRUE), add = TRUE) + untar(destfile, exdir = tempdir) + pkgdir <- dir(tempdir, full.names = TRUE)[[1]] + + # Modify description + desc_path <- file.path(pkgdir, "DESCRIPTION") + desc_lines <- readLines(desc_path) + remotes_fields <- c( + "RemoteType: github", + "RemoteHost: api.github.com", + "RemoteRepo: renv", + "RemoteUsername: rstudio", + "RemotePkgRef: rstudio/renv", + paste("RemoteRef: ", sha), + paste("RemoteSha: ", sha) + ) + writeLines(c(desc_lines[desc_lines != ""], remotes_fields), con = desc_path) + + # Re-tar + local({ + old <- setwd(tempdir) + on.exit(setwd(old), add = TRUE) + + tar(destfile, compression = "gzip") + }) + invisible() + } + + # Extract the commit hash from a git archive. Git archives include the SHA1 + # hash as the comment field of the tarball pax extended header + # (see https://www.kernel.org/pub/software/scm/git/docs/git-archive.html) + # For GitHub archives this should be the first header after the default one + # (512 byte) header. + renv_bootstrap_git_extract_sha1_tar <- function(bundle) { + + # open the bundle for reading + # We use gzcon for everything because (from ?gzcon) + # > Reading from a connection which does not supply a 'gzip' magic + # > header is equivalent to reading from the original connection + conn <- gzcon(file(bundle, open = "rb", raw = TRUE)) + on.exit(close(conn)) + + # The default pax header is 512 bytes long and the first pax extended header + # with the comment should be 51 bytes long + # `52 comment=` (11 chars) + 40 byte SHA1 hash + len <- 0x200 + 0x33 + res <- rawToChar(readBin(conn, "raw", n = len)[0x201:len]) + + if (grepl("^52 comment=", res)) { + sub("52 comment=", "", res) + } else { + NULL + } + } + renv_bootstrap_install <- function(version, tarball, library) { # attempt to install it into project library - message("* Installing renv ", version, " ... ", appendLF = FALSE) dir.create(library, showWarnings = FALSE, recursive = TRUE) + output <- renv_bootstrap_install_impl(library, tarball) + + # check for successful install + status <- attr(output, "status") + if (is.null(status) || identical(status, 0L)) + return(status) + + # an error occurred; report it + header <- "installation of renv failed" + lines <- paste(rep.int("=", nchar(header)), collapse = "") + text <- paste(c(header, lines, output), collapse = "\n") + stop(text) + + } + + renv_bootstrap_install_impl <- function(library, tarball) { # invoke using system2 so we can capture and report output bin <- R.home("bin") exe <- if (Sys.info()[["sysname"]] == "Windows") "R.exe" else "R" - r <- file.path(bin, exe) + R <- file.path(bin, exe) args <- c( "--vanilla", "CMD", "INSTALL", "--no-multiarch", @@ -431,19 +607,7 @@ local({ shQuote(path.expand(tarball)) ) - output <- system2(r, args, stdout = TRUE, stderr = TRUE) - message("Done!") - - # check for successful install - status <- attr(output, "status") - if (is.numeric(status) && !identical(status, 0L)) { - header <- "Error installing renv:" - lines <- paste(rep.int("=", nchar(header)), collapse = "") - text <- c(header, lines, output) - writeLines(text, con = stderr()) - } - - status + system2(R, args, stdout = TRUE, stderr = TRUE) } @@ -484,6 +648,9 @@ local({ # if the user has requested an automatic prefix, generate it auto <- Sys.getenv("RENV_PATHS_PREFIX_AUTO", unset = NA) + if (is.na(auto) && getRversion() >= "4.4.0") + auto <- "TRUE" + if (auto %in% c("TRUE", "True", "true", "1")) return(renv_bootstrap_platform_prefix_auto()) @@ -653,34 +820,61 @@ local({ } - renv_bootstrap_validate_version <- function(version) { + renv_bootstrap_validate_version <- function(version, description = NULL) { - loadedversion <- utils::packageDescription("renv", fields = "Version") - if (version == loadedversion) + # resolve description file + # + # avoid passing lib.loc to `packageDescription()` below, since R will + # use the loaded version of the package by default anyhow. note that + # this function should only be called after 'renv' is loaded + # https://github.com/rstudio/renv/issues/1625 + description <- description %||% packageDescription("renv") + + # check whether requested version 'version' matches loaded version of renv + sha <- attr(version, "sha", exact = TRUE) + valid <- if (!is.null(sha)) + renv_bootstrap_validate_version_dev(sha, description) + else + renv_bootstrap_validate_version_release(version, description) + + if (valid) return(TRUE) - # assume four-component versions are from GitHub; three-component - # versions are from CRAN - components <- strsplit(loadedversion, "[.-]")[[1]] - remote <- if (length(components) == 4L) - paste("rstudio/renv", loadedversion, sep = "@") + # the loaded version of renv doesn't match the requested version; + # give the user instructions on how to proceed + dev <- identical(description[["RemoteType"]], "github") + remote <- if (dev) + paste("rstudio/renv", description[["RemoteSha"]], sep = "@") else - paste("renv", loadedversion, sep = "@") + paste("renv", description[["Version"]], sep = "@") - fmt <- paste( - "renv %1$s was loaded from project library, but this project is configured to use renv %2$s.", - "Use `renv::record(\"%3$s\")` to record renv %1$s in the lockfile.", - "Use `renv::restore(packages = \"renv\")` to install renv %2$s into the project library.", - sep = "\n" + # display both loaded version + sha if available + friendly <- renv_bootstrap_version_friendly( + version = description[["Version"]], + sha = if (dev) description[["RemoteSha"]] ) - msg <- sprintf(fmt, loadedversion, version, remote) - warning(msg, call. = FALSE) + fmt <- heredoc(" + renv %1$s was loaded from project library, but this project is configured to use renv %2$s. + - Use `renv::record(\"%3$s\")` to record renv %1$s in the lockfile. + - Use `renv::restore(packages = \"renv\")` to install renv %2$s into the project library. + ") + catf(fmt, friendly, renv_bootstrap_version_friendly(version), remote) FALSE } + renv_bootstrap_validate_version_dev <- function(version, description) { + expected <- description[["RemoteSha"]] + is.character(expected) && startswith(expected, version) + } + + renv_bootstrap_validate_version_release <- function(version, description) { + expected <- description[["Version"]] + is.character(expected) && identical(expected, version) + } + renv_bootstrap_hash_text <- function(text) { hashfile <- tempfile("renv-hash-") @@ -700,6 +894,12 @@ local({ # warn if the version of renv loaded does not match renv_bootstrap_validate_version(version) + # execute renv load hooks, if any + hooks <- getHook("renv::autoload") + for (hook in hooks) + if (is.function(hook)) + tryCatch(hook(), error = warnify) + # load the project renv::load(project) @@ -839,26 +1039,78 @@ local({ } + renv_bootstrap_version_friendly <- function(version, shafmt = NULL, sha = NULL) { + sha <- sha %||% attr(version, "sha", exact = TRUE) + parts <- c(version, sprintf(shafmt %||% " [sha: %s]", substring(sha, 1L, 7L))) + paste(parts, collapse = "") + } + + renv_bootstrap_exec <- function(project, libpath, version) { + if (!renv_bootstrap_load(project, libpath, version)) + renv_bootstrap_run(version, libpath) + } + + renv_bootstrap_run <- function(version, libpath) { + + # perform bootstrap + bootstrap(version, libpath) + + # exit early if we're just testing bootstrap + if (!is.na(Sys.getenv("RENV_BOOTSTRAP_INSTALL_ONLY", unset = NA))) + return(TRUE) + + # try again to load + if (requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) { + return(renv::load(project = getwd())) + } + + # failed to download or load renv; warn the user + msg <- c( + "Failed to find an renv installation: the project will not be loaded.", + "Use `renv::activate()` to re-initialize the project." + ) + + warning(paste(msg, collapse = "\n"), call. = FALSE) + + } renv_json_read <- function(file = NULL, text = NULL) { + jlerr <- NULL + # if jsonlite is loaded, use that instead - if ("jsonlite" %in% loadedNamespaces()) - renv_json_read_jsonlite(file, text) + if ("jsonlite" %in% loadedNamespaces()) { + + json <- tryCatch(renv_json_read_jsonlite(file, text), error = identity) + if (!inherits(json, "error")) + return(json) + + jlerr <- json + + } + + # otherwise, fall back to the default JSON reader + json <- tryCatch(renv_json_read_default(file, text), error = identity) + if (!inherits(json, "error")) + return(json) + + # report an error + if (!is.null(jlerr)) + stop(jlerr) else - renv_json_read_default(file, text) + stop(json) } renv_json_read_jsonlite <- function(file = NULL, text = NULL) { - text <- paste(text %||% read(file), collapse = "\n") + text <- paste(text %||% readLines(file, warn = FALSE), collapse = "\n") jsonlite::fromJSON(txt = text, simplifyVector = FALSE) } renv_json_read_default <- function(file = NULL, text = NULL) { # find strings in the JSON - text <- paste(text %||% read(file), collapse = "\n") + text <- paste(text %||% readLines(file, warn = FALSE), collapse = "\n") pattern <- '["](?:(?:\\\\.)|(?:[^"\\\\]))*?["]' locs <- gregexpr(pattern, text, perl = TRUE)[[1]] @@ -906,14 +1158,14 @@ local({ map <- as.list(map) # remap strings in object - remapped <- renv_json_remap(json, map) + remapped <- renv_json_read_remap(json, map) # evaluate eval(remapped, envir = baseenv()) } - renv_json_remap <- function(json, map) { + renv_json_read_remap <- function(json, map) { # fix names if (!is.null(names(json))) { @@ -940,7 +1192,7 @@ local({ # recurse if (is.recursive(json)) { for (i in seq_along(json)) { - json[i] <- list(renv_json_remap(json[[i]], map)) + json[i] <- list(renv_json_read_remap(json[[i]], map)) } } @@ -960,35 +1212,9 @@ local({ # construct full libpath libpath <- file.path(root, prefix) - # attempt to load - if (renv_bootstrap_load(project, libpath, version)) - return(TRUE) - - # load failed; inform user we're about to bootstrap - prefix <- paste("# Bootstrapping renv", version) - postfix <- paste(rep.int("-", 77L - nchar(prefix)), collapse = "") - header <- paste(prefix, postfix) - message(header) - - # perform bootstrap - bootstrap(version, libpath) - - # exit early if we're just testing bootstrap - if (!is.na(Sys.getenv("RENV_BOOTSTRAP_INSTALL_ONLY", unset = NA))) - return(TRUE) - - # try again to load - if (requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) { - message("* Successfully installed and loaded renv ", version, ".") - return(renv::load()) - } - - # failed to download or load renv; warn the user - msg <- c( - "Failed to find an renv installation: the project will not be loaded.", - "Use `renv::activate()` to re-initialize the project." - ) + # run bootstrap code + renv_bootstrap_exec(project, libpath, version) - warning(paste(msg, collapse = "\n"), call. = FALSE) + invisible() }) diff --git a/renv/settings.json b/renv/settings.json new file mode 100644 index 0000000..ffdbb32 --- /dev/null +++ b/renv/settings.json @@ -0,0 +1,19 @@ +{ + "bioconductor.version": null, + "external.libraries": [], + "ignored.packages": [], + "package.dependency.fields": [ + "Imports", + "Depends", + "LinkingTo" + ], + "ppm.enabled": null, + "ppm.ignored.urls": [], + "r.version": null, + "snapshot.type": "implicit", + "use.cache": true, + "vcs.ignore.cellar": true, + "vcs.ignore.library": true, + "vcs.ignore.local": true, + "vcs.manage.ignores": true +}