From e4dd38b6487e6bf0ff9e604f9fe13bf1bd47fd5d Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Fri, 14 Aug 2020 12:38:03 -1000 Subject: [PATCH 01/33] first commit of chapter 1: intro to web archiving, autopilot, and installation --- chapter1.md | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 chapter1.md diff --git a/chapter1.md b/chapter1.md new file mode 100644 index 0000000..bf0fa7f --- /dev/null +++ b/chapter1.md @@ -0,0 +1,36 @@ +# Chapter 1 + + +## What is web archiving? + +Web archiving is the process of recording web resources. Various elements of the website such as HTML, scripts, images, videos, etc. can be recorded to preserve as much of the original resource as possible. By creating dynamic archives that provide a user with the same experience they would have if they accessed the original site, web archives can give much more information than a static screenshot. + +Web archives may be utilized by future researchers, historians and the general public. + +## Why Autopilot? + +Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. + +## Installing Autopilot +Both [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) must be installed to run Autopilot. + +To install Autopilot, use Terminal to clone the Webrecorder repository: +1. `git clone ` +2. `cd autopilot; bash init-default.sh.` +3. `docker-compose build` +4. `docker-compose up -d` + +The Autopilot instance can be accessed in the browser at `http://localhost:8089`. + +^^ EXPLAIN BETTER, also do I need to explain remote browsers/what Docker is? + +## Autopilot basics + Autopilot +## Status page: pre-made behaviors + + +# Chapter 2: Using Autopilot +## Creating your first behavior +## Testing your first behavior +## Fixing a broken behavior +## Checking behavior status \ No newline at end of file From 6177866dd306604dd57bac647a6ab755a160b472 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 17 Aug 2020 23:05:56 -1000 Subject: [PATCH 02/33] Added file for chapter 2 documentation --- chapter2.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 chapter2.md diff --git a/chapter2.md b/chapter2.md new file mode 100644 index 0000000..4ee9d9f --- /dev/null +++ b/chapter2.md @@ -0,0 +1,5 @@ +# Chapter 2: Using Autopilot +## Creating your first behavior +## Testing your first behavior +## Fixing a broken behavior +## Checking behavior status \ No newline at end of file From 9d52ef22af1c9be29638d60445d28f67eff8f432 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 18 Aug 2020 12:25:59 -1000 Subject: [PATCH 03/33] install section of chapter 1 modified to include a way to install autopilot without Docker --- chapter1.md | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/chapter1.md b/chapter1.md index bf0fa7f..d5c7689 100644 --- a/chapter1.md +++ b/chapter1.md @@ -12,7 +12,13 @@ Web archives may be utilized by future researchers, historians and the general p Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. ## Installing Autopilot -Both [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) must be installed to run Autopilot. + +To use this project you must first install its dependencies + +`$ yarn install +# or "npm install"` + +If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). To install Autopilot, use Terminal to clone the Webrecorder repository: 1. `git clone ` @@ -22,10 +28,13 @@ To install Autopilot, use Terminal to clone the Webrecorder repository: The Autopilot instance can be accessed in the browser at `http://localhost:8089`. -^^ EXPLAIN BETTER, also do I need to explain remote browsers/what Docker is? ## Autopilot basics - Autopilot + Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which + +##About Web Traffic + + ## Status page: pre-made behaviors From e986ba6b2147e738f64503d18d976217afef6f1c Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 18 Aug 2020 13:18:57 -1000 Subject: [PATCH 04/33] fixed in-line code to be code blocks and added some usage info --- chapter1.md | 49 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 34 insertions(+), 15 deletions(-) diff --git a/chapter1.md b/chapter1.md index d5c7689..fc977d6 100644 --- a/chapter1.md +++ b/chapter1.md @@ -9,37 +9,56 @@ Web archives may be utilized by future researchers, historians and the general p ## Why Autopilot? -Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. +Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. Furthermore, Autopilot is designed to be accessible for anyone to use, and anyone with little javascript knowledge to contribute to. This makes webarchiving available to everyone. ## Installing Autopilot -To use this project you must first install its dependencies +To use this project you must first install its dependencies. You can do this via a package manager like [Yarn](https://classic.yarnpkg.com/en/docs/install/#mac-stable) or [npm](https://www.npmjs.com/). -`$ yarn install -# or "npm install"` +``` +$ yarn install +# or "npm install" +``` If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). To install Autopilot, use Terminal to clone the Webrecorder repository: -1. `git clone ` -2. `cd autopilot; bash init-default.sh.` -3. `docker-compose build` -4. `docker-compose up -d` +``` +1. git clone https://github.com/webrecorder/behaviors.git +2. cd autopilot; bash init-default.sh. +3. docker-compose build +4. docker-compose up -d +``` The Autopilot instance can be accessed in the browser at `http://localhost:8089`. ## Autopilot basics - Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which +Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. -##About Web Traffic +Autopilot has a cli, which has the following commands available. + +``` +$ ./bin/cli --help + +Usage: cli [options] [command] + +Options: + -V, --version output the version number + -h, --help output usage information + +Commands: + api Start the behavior api sever + behaviors Build and or validate behaviors, or generate their metadata + help [cmd] display help for [cmd] +``` + +//I can't yet explain how to use autopilot in the browser since the extension isn't finished yet, right? + +## About Web Traffic ## Status page: pre-made behaviors -# Chapter 2: Using Autopilot -## Creating your first behavior -## Testing your first behavior -## Fixing a broken behavior -## Checking behavior status \ No newline at end of file + From f0becd08e7988de67dbbdb71bbaa27a6fc8dfa80 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 18 Aug 2020 15:23:37 -1000 Subject: [PATCH 05/33] Added documentation for the usage of autopilot --- chapter1.md | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/chapter1.md b/chapter1.md index fc977d6..f4ff8e2 100644 --- a/chapter1.md +++ b/chapter1.md @@ -53,6 +53,66 @@ Commands: help [cmd] display help for [cmd] ``` +The cli provides two commands API and behaviors and each command has its own options. + +# Behaviors command + +Execute `./bin/cli behaviors -b` to build the behaviors made available, using the config file located at the root of the project. + +The built behaviors, along with a behavior metadata file (`behaviorMetadata.js`), can be found in the `dist` directory which will be created for you if it does not exist in the root of this project. + +The following options are available to use with the behaviors command: + +``` + +$ ./bin/cli behaviors --help + +Usage: cli-behaviors [options] + +Options: + -V, --version output the version number + -v, --validate [fileOrDir] + -c, --config [configPath] Path to the behavior config file (default: "/behavior-config.yml") + -b, --build [fileOrDir] Build a behaviors or all behaviors contained within a directory (default: true) + -w, --watch [behaviorFileOrDir] Watch the files, and their imports, in the build directory for re-bundling on changes (placed in dist directory) + --metadata [dumpDir] Generate behavior metadata, optionally supplying a path to directory where metadata is to be placed. Defaults to current working directory + -h, --help output usage information +``` + + +# API command + +To run the behavior API server execute `./bin/cli api --build-behaviors`. + +This will start the API server after all behaviors provided by this project have been built. + +If you have already built the behaviors using the `behaviors` command provided by the cli then you may omit the `--build-behaviors` flag. + +The following options are available to use with the API command: + +``` +$ ./bin/cli api --help + +Usage: cli-api [options] + +Options: + -V, --version output the version number + -p, --port [port] The port the api server is to bind to (default: 3030) + -h, --host [host] The host address the server is listen on (default: "127.0.0.1") + -b, --behaviorDir [behaviorDir] The path to the directory containing the build behaviors (default: "/dist") + -m, --behaviorMetadata [medataPath] The path to the behavior metadata (default: "/dist/behaviorMetadata.js") + --build-behaviors Should the api server build the behaviors for starting up + -h, --help output usage information +``` + +Some configuration of the API server can be done via the environment variables listed below + +* `BEHAVIOR_API_HOST`: the host the api server will use (e.g. 127.0.0.1) +* `BEHAVIOR_API_PORT`: the port the api server will listen on (e.g. 3030) +* `WR_BEHAVIOR_DIR`: path to the directory containing the built behaviors +* `WR_BEHAVIOR_METADATA_PATH`: path to the behavior metadata file +* `BUILD_BEHAVIORS`: should the api server build the behaviors before starting + //I can't yet explain how to use autopilot in the browser since the extension isn't finished yet, right? ## About Web Traffic From c7b64e3b102992bc207660adf7d791f813fa5b6c Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 24 Aug 2020 10:43:31 -1000 Subject: [PATCH 06/33] Added headlines to chapter 2, some usage information for Docker (may not need this), and set up the pre-made behaviors sectin --- chapter1.md | 50 -------------------------------------------------- chapter2.md | 1 + 2 files changed, 1 insertion(+), 50 deletions(-) diff --git a/chapter1.md b/chapter1.md index f4ff8e2..e066fd6 100644 --- a/chapter1.md +++ b/chapter1.md @@ -36,22 +36,6 @@ The Autopilot instance can be accessed in the browser at `http://localhost:8089` ## Autopilot basics Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. -Autopilot has a cli, which has the following commands available. - -``` -$ ./bin/cli --help - -Usage: cli [options] [command] - -Options: - -V, --version output the version number - -h, --help output usage information - -Commands: - api Start the behavior api sever - behaviors Build and or validate behaviors, or generate their metadata - help [cmd] display help for [cmd] -``` The cli provides two commands API and behaviors and each command has its own options. @@ -61,24 +45,6 @@ Execute `./bin/cli behaviors -b` to build the behaviors made available, using th The built behaviors, along with a behavior metadata file (`behaviorMetadata.js`), can be found in the `dist` directory which will be created for you if it does not exist in the root of this project. -The following options are available to use with the behaviors command: - -``` - -$ ./bin/cli behaviors --help - -Usage: cli-behaviors [options] - -Options: - -V, --version output the version number - -v, --validate [fileOrDir] - -c, --config [configPath] Path to the behavior config file (default: "/behavior-config.yml") - -b, --build [fileOrDir] Build a behaviors or all behaviors contained within a directory (default: true) - -w, --watch [behaviorFileOrDir] Watch the files, and their imports, in the build directory for re-bundling on changes (placed in dist directory) - --metadata [dumpDir] Generate behavior metadata, optionally supplying a path to directory where metadata is to be placed. Defaults to current working directory - -h, --help output usage information -``` - # API command @@ -88,22 +54,6 @@ This will start the API server after all behaviors provided by this project have If you have already built the behaviors using the `behaviors` command provided by the cli then you may omit the `--build-behaviors` flag. -The following options are available to use with the API command: - -``` -$ ./bin/cli api --help - -Usage: cli-api [options] - -Options: - -V, --version output the version number - -p, --port [port] The port the api server is to bind to (default: 3030) - -h, --host [host] The host address the server is listen on (default: "127.0.0.1") - -b, --behaviorDir [behaviorDir] The path to the directory containing the build behaviors (default: "/dist") - -m, --behaviorMetadata [medataPath] The path to the behavior metadata (default: "/dist/behaviorMetadata.js") - --build-behaviors Should the api server build the behaviors for starting up - -h, --help output usage information -``` Some configuration of the API server can be done via the environment variables listed below diff --git a/chapter2.md b/chapter2.md index 4ee9d9f..338e461 100644 --- a/chapter2.md +++ b/chapter2.md @@ -1,4 +1,5 @@ # Chapter 2: Using Autopilot + ## Creating your first behavior ## Testing your first behavior ## Fixing a broken behavior From d29919eb9d79b55f16be08939c586cf5b05cfb21 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 24 Aug 2020 10:47:40 -1000 Subject: [PATCH 07/33] last commit didn't work right, trying again --- chapter1.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/chapter1.md b/chapter1.md index e066fd6..b3f522c 100644 --- a/chapter1.md +++ b/chapter1.md @@ -63,12 +63,29 @@ Some configuration of the API server can be done via the environment variables l * `WR_BEHAVIOR_METADATA_PATH`: path to the behavior metadata file * `BUILD_BEHAVIORS`: should the api server build the behaviors before starting -//I can't yet explain how to use autopilot in the browser since the extension isn't finished yet, right? + + +# Docker +To build the wr-behaviors docker image (`webrecorder/behaviors:latest`) execute `docker-compose build`. + +The image created is suitable for building behaviors and running the behavior api server. + +The default configuration of the image is to run the api server, however you can substitute the default command with any of the cli commands listed previously. + +For more information please consult the provided `Dockerfile` and `docker-compose.yml` files. ## About Web Traffic ## Status page: pre-made behaviors +# Autoscrool + +# Instagram + +# Twitter + +# Youtube Video + From aa36246ed3a8bd92744fb3d9bde62e70632a4c4b Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 25 Aug 2020 14:13:23 -1000 Subject: [PATCH 08/33] Started the guide on how to write a behavior for people with little js knowledge, still testing the premade behaviors for chapter 1 --- chapter1.md | 18 ++++++++++-------- chapter2.md | 25 +++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/chapter1.md b/chapter1.md index b3f522c..67b380d 100644 --- a/chapter1.md +++ b/chapter1.md @@ -3,17 +3,17 @@ ## What is web archiving? -Web archiving is the process of recording web resources. Various elements of the website such as HTML, scripts, images, videos, etc. can be recorded to preserve as much of the original resource as possible. By creating dynamic archives that provide a user with the same experience they would have if they accessed the original site, web archives can give much more information than a static screenshot. +Web archiving is the process of recording web resources. Various elements of the website such as HTML, scripts, images, videos, etc. can be recorded to preserve as much of the original resource as possible. By creating dynamic archives that provide a user with the same experience they would have if they accessed the original site, web archives can give much more information than a static screenshot. -Web archives may be utilized by future researchers, historians and the general public. +Web archives may be utilized by future researchers, historians and the general public. ## Why Autopilot? -Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. Furthermore, Autopilot is designed to be accessible for anyone to use, and anyone with little javascript knowledge to contribute to. This makes webarchiving available to everyone. +Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. Furthermore, Autopilot is designed to be accessible for anyone to use, and anyone with little javascript knowledge to contribute to. This makes webarchiving available to everyone. ## Installing Autopilot -To use this project you must first install its dependencies. You can do this via a package manager like [Yarn](https://classic.yarnpkg.com/en/docs/install/#mac-stable) or [npm](https://www.npmjs.com/). +To use this project you must first install its dependencies. You can do this via a package manager like [Yarn](https://classic.yarnpkg.com/en/docs/install/#mac-stable) or [npm](https://www.npmjs.com/). ``` $ yarn install @@ -22,7 +22,7 @@ $ yarn install If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). -To install Autopilot, use Terminal to clone the Webrecorder repository: +To install Autopilot, use Terminal to clone the Webrecorder repository: ``` 1. git clone https://github.com/webrecorder/behaviors.git 2. cd autopilot; bash init-default.sh. @@ -34,14 +34,14 @@ The Autopilot instance can be accessed in the browser at `http://localhost:8089` ## Autopilot basics -Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. +Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. The cli provides two commands API and behaviors and each command has its own options. # Behaviors command -Execute `./bin/cli behaviors -b` to build the behaviors made available, using the config file located at the root of the project. +Execute `./bin/cli behaviors -b` to build the behaviors made available, using the config file located at the root of the project. The built behaviors, along with a behavior metadata file (`behaviorMetadata.js`), can be found in the `dist` directory which will be created for you if it does not exist in the root of this project. @@ -77,9 +77,11 @@ For more information please consult the provided `Dockerfile` and `docker-compos ## About Web Traffic +//consider making the status page space a seperate chapter or a seperate section because it kind of dominates chapter 1. ## Status page: pre-made behaviors -# Autoscrool +# Autoscroll + # Instagram diff --git a/chapter2.md b/chapter2.md index 338e461..119dc9f 100644 --- a/chapter2.md +++ b/chapter2.md @@ -1,6 +1,31 @@ # Chapter 2: Using Autopilot + ## Creating your first behavior +A behavior is a JavaScript file which executes a series of actions that collect some information (metadata) about a webpage. When creating a behavior, you should first take a look at the Pre-Made Behaviors on the Status Page . This will give you a basic idea of what kind of behaviors already have been created. + +Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. + +1. `mkdir webrecorder` (or skip these two steps and use an already existing directory) +2. `cd webrecorder` +3. `git clone https://github.com/webrecorder/behaviors.git` + + + +Next, make a file in the appropriate directory (i.e. if the behavior is specific to Facebook, put the file under the Facebook directory). + +For example: +``` +touch webrecorder/facebook/myBehavior/ +``` + + + + + + ## Testing your first behavior + ## Fixing a broken behavior + ## Checking behavior status \ No newline at end of file From 34fb47ef2aaf419ffbc388b1fb577527eccc07f1 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 25 Aug 2020 14:37:37 -1000 Subject: [PATCH 09/33] Progress on the Format section for chapter 2 under Creating Your First Behavior --- chapter2.md | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/chapter2.md b/chapter2.md index 119dc9f..735bc1a 100644 --- a/chapter2.md +++ b/chapter2.md @@ -2,7 +2,7 @@ ## Creating your first behavior -A behavior is a JavaScript file which executes a series of actions that collect some information (metadata) about a webpage. When creating a behavior, you should first take a look at the Pre-Made Behaviors on the Status Page . This will give you a basic idea of what kind of behaviors already have been created. +A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. When creating a behavior, you should first take a look at the Pre-Made Behaviors on the Status Page . This will give you a basic idea of what kind of behaviors already have been created. Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. @@ -10,15 +10,26 @@ Once you have an idea of what you want your behavior to do, clone the webrecorde 2. `cd webrecorder` 3. `git clone https://github.com/webrecorder/behaviors.git` +Next, make a file in the appropriate directory. +For example, if the behavior is specific to Facebook, put the file under the Facebook directory: -Next, make a file in the appropriate directory (i.e. if the behavior is specific to Facebook, put the file under the Facebook directory). +`touch webrecorder/facebook/myBehavior.js` -For example: -``` -touch webrecorder/facebook/myBehavior/ -``` +### Format +Every behavior has: +1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). + + +2. named export metadata that is an object + +3. named export **isBehavior**. + +Details: **isBehavior** is a constant that you will flag as **true** when the behavior is complete and ready to be used. Otherwise, while the behavior is in progress, keep it as **false**. + +Example of expected format: +`export const isBehavior = true;` From a1644031cd1e409cdfc80e601d60093da5ba71e0 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 25 Aug 2020 15:22:30 -1000 Subject: [PATCH 10/33] Added more for format section, putting in code examples --- chapter2.md | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/chapter2.md b/chapter2.md index 735bc1a..3968d4a 100644 --- a/chapter2.md +++ b/chapter2.md @@ -21,15 +21,35 @@ Every behavior has: 1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). - -2. named export metadata that is an object +```js +export detault async function* myBehavior(cliAPI){...} +``` + +2. named export ***metadata*** that is an object + + Details: the ***metadata*** object gives information about the behavior. In this object you should include the data about the name, whether it's functional, the display name, whether it's a default behavior, a description about the behavior, and when it was last updated. + + Example of expected format, taken from the [autoscroll behavior](https://github.com/webrecorder/behaviors/blob/master/behaviors/autoscroll.js): +```js +export const metadata = { + name: 'autoScrollBehavior', + functional: true, + displayName: 'Default Scrolling', + defaultBehavior: true, + description: + 'Default behavior for any page. Automatically scrolls down the page as much as possible. If additional content loads that increases page height, scrolling will continue until autopilot is stopped by user. Any discovered audio/video is played, but no other interactions are performed.', + updated: '2019-08-21T14:52:23-07:00', +}; +``` 3. named export **isBehavior**. -Details: **isBehavior** is a constant that you will flag as **true** when the behavior is complete and ready to be used. Otherwise, while the behavior is in progress, keep it as **false**. + Details: **isBehavior** is a constant that you will flag as **true** when the file is complete and ready to be used. Otherwise, while the file is still in progress, keep it as **false**. -Example of expected format: -`export const isBehavior = true;` + Example of expected format: +```js +export const isBehavior = true; +``` From 9f978c92537500cb186ae5051732ff86809589eb Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Thu, 27 Aug 2020 09:03:49 -1000 Subject: [PATCH 11/33] fixing some stuff in format, adding some stuff in a new section explaining metadata --- chapter2.md | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/chapter2.md b/chapter2.md index 3968d4a..08860f3 100644 --- a/chapter2.md +++ b/chapter2.md @@ -19,13 +19,13 @@ For example, if the behavior is specific to Facebook, put the file under the Fac ### Format Every behavior has: -1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). +1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). This is the function that will retrieve the metadata we want from the webpage. ```js export detault async function* myBehavior(cliAPI){...} ``` -2. named export ***metadata*** that is an object +2. an export named ***metadata*** that is an object Details: the ***metadata*** object gives information about the behavior. In this object you should include the data about the name, whether it's functional, the display name, whether it's a default behavior, a description about the behavior, and when it was last updated. @@ -41,8 +41,9 @@ export const metadata = { updated: '2019-08-21T14:52:23-07:00', }; ``` +More details on metadata and how the metadata object should be written can be found under the [Metadata](#metadata-heading) section. -3. named export **isBehavior**. +3. an export named **isBehavior**. Details: **isBehavior** is a constant that you will flag as **true** when the file is complete and ready to be used. Otherwise, while the file is still in progress, keep it as **false**. @@ -50,10 +51,31 @@ export const metadata = { ```js export const isBehavior = true; ``` + +It's important to note that the tools will not recognize that the behavior is ready for use and valid if any of these three main components (the default export, **isBehavior**, and **metadata**) are missing. +The export `postStep` can be called after each action to convert the yielded results into the [expected format] (https://github.com/webrecorder/behaviors/blob/master/typedef/index.html#static-typedef-BehaviorStepResults). + +It is recommended that you use the library function [lib.buildCustomPostStepFn](https://github.com/webrecorder/behaviors/blob/master/function/index.html#static-function-buildCustomPostStepFn) if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. + +```js +export const postStep = lib.buildCustomPostStepFn(() => { ... }); +``` + +###Metadata {#metadata-heading} +A behavior's exported metadata is used to: + +- describe how the behavior should be matched to the pages it is written for +- provide an overview of what the behavior does +- have a more specific name associated with it when querying for it using the behavior api +- embed any additional information about the behavior ## Testing your first behavior From 1060548cf8b2a15b3fdf5a554ed76efb5f5ab0b5 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Thu, 27 Aug 2020 18:28:22 -1000 Subject: [PATCH 12/33] wrote some stuff on the expected format of the export metadata object --- chapter2.md | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/chapter2.md b/chapter2.md index 08860f3..c768263 100644 --- a/chapter2.md +++ b/chapter2.md @@ -25,23 +25,18 @@ Every behavior has: export detault async function* myBehavior(cliAPI){...} ``` +More details on how the default export should be written can be found under the [Default Export](#default-heading) section. + 2. an export named ***metadata*** that is an object - Details: the ***metadata*** object gives information about the behavior. In this object you should include the data about the name, whether it's functional, the display name, whether it's a default behavior, a description about the behavior, and when it was last updated. + Details: the ***metadata*** object gives information about the behavior, such as when it was last updated, whether it's functional, its display name, etc. - Example of expected format, taken from the [autoscroll behavior](https://github.com/webrecorder/behaviors/blob/master/behaviors/autoscroll.js): + Example of expected format: ```js -export const metadata = { - name: 'autoScrollBehavior', - functional: true, - displayName: 'Default Scrolling', - defaultBehavior: true, - description: - 'Default behavior for any page. Automatically scrolls down the page as much as possible. If additional content loads that increases page height, scrolling will continue until autopilot is stopped by user. Any discovered audio/video is played, but no other interactions are performed.', - updated: '2019-08-21T14:52:23-07:00', -}; + export const metadata = { ... }; ``` -More details on metadata and how the metadata object should be written can be found under the [Metadata](#metadata-heading) section. + +More details on how the metadata object should be written can be found under the [Metadata](#metadata-heading) section. 3. an export named **isBehavior**. @@ -59,7 +54,7 @@ to use. It's important to note that the tools will not recognize that the behavior is ready for use and valid if any of these three main components (the default export, **isBehavior**, and **metadata**) are missing. -The export `postStep` can be called after each action to convert the yielded results into the [expected format] (https://github.com/webrecorder/behaviors/blob/master/typedef/index.html#static-typedef-BehaviorStepResults). +The export `postStep` can be called after each action to convert the yielded results into the [expected format](https://github.com/webrecorder/behaviors/blob/master/typedef/index.html#static-typedef-BehaviorStepResults). It is recommended that you use the library function [lib.buildCustomPostStepFn](https://github.com/webrecorder/behaviors/blob/master/function/index.html#static-function-buildCustomPostStepFn) if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. @@ -70,13 +65,24 @@ export const postStep = lib.buildCustomPostStepFn(() => { ... }); ``` ###Metadata {#metadata-heading} -A behavior's exported metadata is used to: +A behavior's exported metadata object is used to: - describe how the behavior should be matched to the pages it is written for - provide an overview of what the behavior does -- have a more specific name associated with it when querying for it using the behavior api +- provide a more specific name associated with it when querying for it using the behavior api - embed any additional information about the behavior +With those usages in mind, every metadata object is expected to have the following properties: + +- name (string): the name for your behavior to be used when querying the behavior API for it by name +- description (string): a description of the behavior +- match (object): how the behavior will be matched to the page(s) it is written for + + + + +###Default Export (#default-heading) + ## Testing your first behavior ## Fixing a broken behavior From 200d60416afcc3295ff5405089e607240f5250d0 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 31 Aug 2020 10:16:03 -1000 Subject: [PATCH 13/33] added info on the match property in the metadata export --- chapter2.md | 42 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 39 insertions(+), 3 deletions(-) diff --git a/chapter2.md b/chapter2.md index c768263..6e4e7d4 100644 --- a/chapter2.md +++ b/chapter2.md @@ -74,10 +74,44 @@ A behavior's exported metadata object is used to: With those usages in mind, every metadata object is expected to have the following properties: -- name (string): the name for your behavior to be used when querying the behavior API for it by name -- description (string): a description of the behavior -- match (object): how the behavior will be matched to the page(s) it is written for +- ***name*** (string): the name for your behavior to be used when querying the behavior API for it by name +- ***description*** (string): a description of the behavior +- ***match*** (object): how the behavior will be matched to the page(s) it is written for +The ***match*** object has two variations and is shown below in the context of two valid metadata exports. + +```js +// variation 1 +export const metadata = { + name: 'the name of your behavior', + match: { + regex: /a regular expression dictating the URL the behavior will run on/, + }, + description: 'an description of what your behavior does', +}; +``` +The first variation, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. + + +The second variation, shown below, has two properties `base` (RegExp) and `sub` (Array). + +The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. + +```js +// variation 2 +export const metadata = { + name: 'the name of your behavior', + match: { + regex: { + base: /a regular expression dictating the base URL the behavior will run on/, + sub: [ + /an array of regular expressions dictating more specific parts of the base URL the behavior will run on/, + ], + }, + }, + description: 'a description of what your behavior does', +}; +``` @@ -87,4 +121,6 @@ With those usages in mind, every metadata object is expected to have the followi ## Fixing a broken behavior +When a behavior breaks, they click “fix this behavior,” and if someone has little js knowledge then they should be able to fix it + ## Checking behavior status \ No newline at end of file From a9677bafc6c454a70845610ae767c1ea2c18e8a2 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 1 Sep 2020 10:24:47 -1000 Subject: [PATCH 14/33] finished the tutorial on how to create a behavior in chapter 2 --- chapter2.md | 171 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 161 insertions(+), 10 deletions(-) diff --git a/chapter2.md b/chapter2.md index 6e4e7d4..9ce8199 100644 --- a/chapter2.md +++ b/chapter2.md @@ -64,7 +64,7 @@ It is recommended that you use the library function [lib.buildCustomPostStepFn]( export const postStep = lib.buildCustomPostStepFn(() => { ... }); ``` -###Metadata {#metadata-heading} +### Metadata {#metadata-heading} A behavior's exported metadata object is used to: - describe how the behavior should be matched to the pages it is written for @@ -74,11 +74,11 @@ A behavior's exported metadata object is used to: With those usages in mind, every metadata object is expected to have the following properties: -- ***name*** (string): the name for your behavior to be used when querying the behavior API for it by name -- ***description*** (string): a description of the behavior -- ***match*** (object): how the behavior will be matched to the page(s) it is written for +- **name** (string): the name for your behavior to be used when querying the behavior API for it by name +- **description** (string): a description of the behavior +- **match** (object): how the behavior will be matched to the page(s) it is written for -The ***match*** object has two variations and is shown below in the context of two valid metadata exports. +The **match** object has two variations and is shown below in the context of two valid metadata exports. ```js // variation 1 @@ -90,12 +90,14 @@ export const metadata = { description: 'an description of what your behavior does', }; ``` -The first variation, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. +The first variation of `match`, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. -The second variation, shown below, has two properties `base` (RegExp) and `sub` (Array). +The second variation of `match`, shown below, has the two properties `base` (RegExp) and `sub` (Array). + +The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. The behavior is considered matched to a URL when the `base` regular expression matches the URL and one of the `sub` regular expressions also matches the URL. + -The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. ```js // variation 2 @@ -115,12 +117,161 @@ export const metadata = { -###Default Export (#default-heading) +### Default Export {#default-heading} + +#### Asyncronous generator functions + +The purpose of an [asyncronous generator function](https://thecodebarbarian.com/async-generator-functions-in-javascript.html#:~:text=Async%20generator%20functions%20behave%20similarly,()%20function%20returns%20a%20promise.) is to collect data from a source that has too much data to return all at once (or would take too long to return all at once). Instead, the generator is a function that returns an object with a next() method on it. The programmer can keep calling next() until all of the data is received. An asynchronous generator function uses promises for the same purpose (promises will be expanded upon in the following section). + +The primary reasons that a behavior's **default export** is required to be an async generator function or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator) are: + + +- async generators are widely supported in Javascript +- they provide a simple way to run the behavior in the browser or through [browsertrix](https://github.com/webrecorder/browsertrix) +- they allow information about the behavior and its state to be easily retrieved by code executing the behavior + +Consider the following example: + +```javascript= +import * as lib from '../lib'; + +export default async function* myBehavior(cliAPI) { + // behavior code will be developed here +} + +export const metadata = { + name: 'myBehavior', + match: { + regex: /^(?:https?:\/\/(?:www\.)?)?:myAwesomeWebSite.com.*$/, + }, + description: 'It does really cool stuff', +}; + +export const isBehavior = true; +``` + +In this example, `myBehavior` is an asynchronous generator function that's being exported for our behavior and this behavior would work for any website with myAwesomeWebSite.com in its name (notice the regex for the matching above). Much of this boiler plate code shown above has been generated automatically by the cli command [insert command here]. + +#### Promises + +Next, you may have notticed looking over some of the behavior documentation that many of the functions return [Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise). + +Now, without turning into a JavaScript tutorial, the only thing you need to know about promises is that they can be awaited, that is to say you can await for their completion. + +This comes in handy when you want to wait for the the document's state to become ready since the pages of myAwesomeWebSite.com take a long time to load and can be done as easily as shown in `step 1`. This is the beginning of the code that will go in the default async function. + +```javascript= +// step 1, wait for the page to complete loading +await lib.domCompletePromise(); +``` + +Now that we know the browser has fully loaded the page, we can safely start executing behaviors on the page. + +The next step is to initialize our state and report it, which is done by yielding a value as shown in `step 2`, where state contains the number of videos played. + +```javascript= +// step 2, initialize state and return it +const state = { videosPlayed: 0 }; +yield lib.stateWithMsgNoWait('Document ready!', state); +``` + +The function `lib.stateWithMsgNoWait` indicates to the code executing the behavior that it has an update to report and that it does not have to wait. + +If the behavior was being run by [browsertrix](https://github.com/webrecorder/browsertrix) and the other function `lib.stateWithMsgWait` was used, browsertrix would have waited until the HTTP requests made by the page had significantly slowed down (no request made for set period of time) before executing further code in the function. However, since we use the *no wait* variant, the behavior will immediately return state. + +When you `yield` a value from the behavior you can consider the behavior paused until the runner initiates the `next()` action. + +Additionally, it should be noted that the second argument supplied to `lib.stateWithMsgNoWait` is optional but useful for reporting to yourself more detailed information about the state of your behavior. + +Continuing on with the creation of our behavior, let us assume that the page that the behavior is operating in has a list videos we want to play. Our behavior will generate the next video to play each iteration. + +We can accomplish this as shown in `step 3`. + +```javascript= +// step 3 +for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { + // videoListItem is a child of the element with id == 'videos' + // that has a video as child element + const videoWasPlayed = await lib.selectAndPlay('video', videoListItem); + if (videoWasPlayed) { + // increment our states counter for number of videos played + state.videosPlayed += 1; + // let ourselves know we played a video + yield lib.stateWithMsgNoWait('Played a video!', state); + } else { + yield lib.stateWithMsgNoWait('Failed to play a video using the standard DOM methods', state); + } +} +return lib.stateWithMsgNoWait('Done!', state); +``` + +In step three we use the function `childElementIterator` that returns an iterator over the child elements of the supplied parent element and then for each child of the element with `id="videos"` we: + +- select the video element that is a descendant of the `videoListItems` and play the video +- increment the number of videos played in our behavior's state +- let ourselves know that we played a video + +Also seen in `step 3` is the usage keyword `yield*`. + +`yield*` means that we are yielding another generator, that is to say all actions of the generator are to be treated as if we yielded them ourselves. + +In short, `step 3` can be described as playing a video contained in every child of the element with `id == video`, and once we have played all the videos on the page return a message with our final state from the behavior. + +The full behavior is shown below: + +```javascript= +import * as lib from '../lib'; + +export default async function* myBehavior(cliAPI) { + // behavior code + // step 1 + await lib.domCompletePromise(); + // step 2 + const state = { videosPlayed: 0 }; + yield lib.stateWithMsgNoWait('Document ready!', state); + // step 3 + for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { + // videoListItem is a child of the element with id == 'videos' + // that has a video as child element + const videoWasPlayed = await lib.selectAndPlay('video', videoListItem); + if (videoWasPlayed) { + // increment our states counter for number of videos played + state.videosPlayed += 1; + // let ourselves know we played a video + yield lib.stateWithMsgNoWait('Played a video!', state); + } else { + yield lib.stateWithMsgNoWait('Failed to play a video using the standard DOM methods', state); + } + } + return lib.stateWithMsgNoWait('Done!', state); +} + +export const metadata = { + name: 'myBehavior', + match: { + regex: /^(?:https?:\/\/(?:www\.)?)?:myAwesomeWebSite.com.*$/, + }, + description: 'It does really cool stuff', +}; + +export const isBehavior = true; +``` + + + ## Testing your first behavior +An example of the video playing thing embedded in a webpage. Show html code that was used for the website, js that was used to run a behavior. + +Tell them to write this test example (video recording thing) for me and I can link to it in github. + ## Fixing a broken behavior When a behavior breaks, they click “fix this behavior,” and if someone has little js knowledge then they should be able to fix it -## Checking behavior status \ No newline at end of file +## Checking behavior status + +how do you check behavior status and is this done in js + + From 6ec4f7503a7957842bbd18566d79bdb8097c2309 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 1 Sep 2020 14:07:41 -1000 Subject: [PATCH 15/33] added files for chapters 3, 4, and reference:API --- chapter2.md | 4 ++-- chapter3.md | 14 ++++++++++++++ chapter4.md | 5 +++++ reference.md | 28 ++++++++++++++++++++++++++++ 4 files changed, 49 insertions(+), 2 deletions(-) create mode 100644 chapter3.md create mode 100644 chapter4.md create mode 100644 reference.md diff --git a/chapter2.md b/chapter2.md index 9ce8199..6cef6f8 100644 --- a/chapter2.md +++ b/chapter2.md @@ -64,7 +64,7 @@ It is recommended that you use the library function [lib.buildCustomPostStepFn]( export const postStep = lib.buildCustomPostStepFn(() => { ... }); ``` -### Metadata {#metadata-heading} +### [Metadata](#metadata-heading) A behavior's exported metadata object is used to: - describe how the behavior should be matched to the pages it is written for @@ -117,7 +117,7 @@ export const metadata = { -### Default Export {#default-heading} +### [Default Export](#default-heading) #### Asyncronous generator functions diff --git a/chapter3.md b/chapter3.md new file mode 100644 index 0000000..778e06f --- /dev/null +++ b/chapter3.md @@ -0,0 +1,14 @@ +# Hacking Autopilot +## Contributing to Autopilot +## Tools of the Trade +## Creating a Behavior +What should be here that wasn't in chapter 2? +## Testing a behavior (Debugging) +This also seems like it should be in chapter 2 +## Behaviors +Also chapter 1 and 2 item +## Build System +## CLI +## Overview on Behaviors +## Provided CLI Commands +## Behavior Standard Library Reference \ No newline at end of file diff --git a/chapter4.md b/chapter4.md new file mode 100644 index 0000000..27c9233 --- /dev/null +++ b/chapter4.md @@ -0,0 +1,5 @@ +# Behind Autopilot +## Maintainers +## Contact Support +## Discussion FOrum +## Who Uses Autopilot? \ No newline at end of file diff --git a/reference.md b/reference.md new file mode 100644 index 0000000..31bc70a --- /dev/null +++ b/reference.md @@ -0,0 +1,28 @@ +# API +## CBehaviorRunner +## CMutationStream +## FcreateState +## FstateWithMsgNoWait +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## +## \ No newline at end of file From 959aa5aae078898cde097ca804b18726660de131 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 8 Sep 2020 20:37:12 -1000 Subject: [PATCH 16/33] added more steps into the "how to make a behavior" section of chapter2 so that it's easier to follow for a beginner, and some notes in chapter3 for how it should be written --- chapter2.md | 39 ++++++++++++++++++++++++++------------- chapter3.md | 6 +++++- 2 files changed, 31 insertions(+), 14 deletions(-) diff --git a/chapter2.md b/chapter2.md index 6cef6f8..13a0bbe 100644 --- a/chapter2.md +++ b/chapter2.md @@ -1,22 +1,35 @@ -# Chapter 2: Using Autopilot +# Chapter 2: Creating your first behavior +A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the Pre-Made Behaviors on the Status Page . This will give you a basic idea of what kind of behaviors already have been created. -## Creating your first behavior -A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. When creating a behavior, you should first take a look at the Pre-Made Behaviors on the Status Page . This will give you a basic idea of what kind of behaviors already have been created. - +## Setting up your file Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. 1. `mkdir webrecorder` (or skip these two steps and use an already existing directory) 2. `cd webrecorder` 3. `git clone https://github.com/webrecorder/behaviors.git` -Next, make a file in the appropriate directory. +Next, make a JavaScript file in the appropriate directory. For example, if the behavior is specific to Facebook, put the file under the Facebook directory: -`touch webrecorder/facebook/myBehavior.js` +`touch webrecorder/behaviors/facebook/myBehavior.js` + +You should also download a package manager like npm, and make sure that you are using the correct version. +``` +$ yarn install +# or "npm install" +``` + +To check your version of npm, type the command `npm --version`. You can check the [npm website](https://www.npmjs.com/package/npm?activeTab=versions) to see what the latest version is. + +### Opening the + ### Format + + + Every behavior has: 1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). This is the function that will retrieve the metadata we want from the webpage. @@ -25,7 +38,7 @@ Every behavior has: export detault async function* myBehavior(cliAPI){...} ``` -More details on how the default export should be written can be found under the [Default Export](#default-heading) section. +More details on how the default export should be written can be found under the [Default Export](#default-export) section. 2. an export named ***metadata*** that is an object @@ -36,7 +49,7 @@ More details on how the default export should be written can be found under the export const metadata = { ... }; ``` -More details on how the metadata object should be written can be found under the [Metadata](#metadata-heading) section. +More details on how the metadata object should be written can be found under the [Metadata](#metadata) section. 3. an export named **isBehavior**. @@ -64,7 +77,7 @@ It is recommended that you use the library function [lib.buildCustomPostStepFn]( export const postStep = lib.buildCustomPostStepFn(() => { ... }); ``` -### [Metadata](#metadata-heading) +### Metadata (#metadata-heading) A behavior's exported metadata object is used to: - describe how the behavior should be matched to the pages it is written for @@ -117,11 +130,11 @@ export const metadata = { -### [Default Export](#default-heading) +### Default Export(#default-heading) -#### Asyncronous generator functions +#### Asyncronous Generator Functions -The purpose of an [asyncronous generator function](https://thecodebarbarian.com/async-generator-functions-in-javascript.html#:~:text=Async%20generator%20functions%20behave%20similarly,()%20function%20returns%20a%20promise.) is to collect data from a source that has too much data to return all at once (or would take too long to return all at once). Instead, the generator is a function that returns an object with a next() method on it. The programmer can keep calling next() until all of the data is received. An asynchronous generator function uses promises for the same purpose (promises will be expanded upon in the following section). +The purpose of an [asyncronous generator function](https://thecodebarbarian.com/async-generator-functions-in-javascript.html#:~:text=Async%20generator%20functions%20behave%20similarly,()%20function%20returns%20a%20promise.) is to collect data from a source that has too much data to return all at once (or would take too long to return all at once. Instead, the generator is a function that returns an object with a next() method on it. The programmer can keep calling next() until all of the data is received. An asynchronous generator function uses promises for the same purpose (promises will be expanded upon in the following section). The primary reasons that a behavior's **default export** is required to be an async generator function or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator) are: @@ -205,7 +218,7 @@ for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { return lib.stateWithMsgNoWait('Done!', state); ``` -In step three we use the function `childElementIterator` that returns an iterator over the child elements of the supplied parent element and then for each child of the element with `id="videos"` we: +In `step 3` we use the function `childElementIterator` that returns an iterator over the child elements of the supplied parent element. Then for each child of the element with `id="videos"` we: - select the video element that is a descendant of the `videoListItems` and play the video - increment the number of videos played in our behavior's state diff --git a/chapter3.md b/chapter3.md index 778e06f..2b28ca7 100644 --- a/chapter3.md +++ b/chapter3.md @@ -1,10 +1,14 @@ # Hacking Autopilot +advanced chapter 2 basically +lib and metadata can go in here, should be able to control F to find what you're looking for in here ## Contributing to Autopilot ## Tools of the Trade -## Creating a Behavior +## Advanced Behavior API What should be here that wasn't in chapter 2? +all that advanced shit on metadata ## Testing a behavior (Debugging) This also seems like it should be in chapter 2 +how the testing infrastructure is working (behavior testing advanced) ## Behaviors Also chapter 1 and 2 item ## Build System From 41cc5e07dc6a0b87f5421c0370bd2102e43fe510 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 8 Sep 2020 20:46:11 -1000 Subject: [PATCH 17/33] changed the title of the section after "setting up your file" to "getting started using the CLI" --- chapter2.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/chapter2.md b/chapter2.md index 13a0bbe..8d0fa96 100644 --- a/chapter2.md +++ b/chapter2.md @@ -23,7 +23,9 @@ $ yarn install To check your version of npm, type the command `npm --version`. You can check the [npm website](https://www.npmjs.com/package/npm?activeTab=versions) to see what the latest version is. -### Opening the +### Getting started using the CLI + + ### Format From 8259197b71b53bc2554c0dd9f40718ccc861ec3b Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Fri, 11 Sep 2020 19:36:09 -1000 Subject: [PATCH 18/33] added stuff about using the `newBehavior` CLI in the getting started section --- chapter2.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/chapter2.md b/chapter2.md index 8d0fa96..c2fd241 100644 --- a/chapter2.md +++ b/chapter2.md @@ -5,16 +5,10 @@ A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web ## Setting up your file Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. -1. `mkdir webrecorder` (or skip these two steps and use an already existing directory) +1. `mkdir webrecorder` (or skip steps 1-2 and use an already existing directory) 2. `cd webrecorder` 3. `git clone https://github.com/webrecorder/behaviors.git` -Next, make a JavaScript file in the appropriate directory. - -For example, if the behavior is specific to Facebook, put the file under the Facebook directory: - -`touch webrecorder/behaviors/facebook/myBehavior.js` - You should also download a package manager like npm, and make sure that you are using the correct version. ``` $ yarn install @@ -25,7 +19,11 @@ To check your version of npm, type the command `npm --version`. You can check th ### Getting started using the CLI +You should now create a Javascript file for your new behavior. You can do this using the CLI. +The `newBehavior` command provides a simple way to create a new behavior by generating a new file in the behavior directory containing the required boiler plate. + +Executing `./bin/cli newBehavior awesomeBehavior` will create a new behavior file a`wesomeBehavior.js` located in the behavior directory. ### Format @@ -276,6 +274,9 @@ export const isBehavior = true; ## Testing your first behavior +Should mention the runner CLI in this section (this was found in the manual here https://github.com/webrecorder/behaviors/blob/master/manual/cli.md): + The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. + An example of the video playing thing embedded in a webpage. Show html code that was used for the website, js that was used to run a behavior. From 8a038f379f24040f013930b9535aef1c58943e63 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 14 Sep 2020 10:58:18 -1000 Subject: [PATCH 19/33] added directions for Testing your first behavior using the CLI --- chapter2.md | 37 +++++++++++++++++++++++++------------ chapter3.md | 1 + chapter4.md | 2 +- 3 files changed, 27 insertions(+), 13 deletions(-) diff --git a/chapter2.md b/chapter2.md index c2fd241..49ad388 100644 --- a/chapter2.md +++ b/chapter2.md @@ -9,7 +9,7 @@ Once you have an idea of what you want your behavior to do, clone the webrecorde 2. `cd webrecorder` 3. `git clone https://github.com/webrecorder/behaviors.git` -You should also download a package manager like npm, and make sure that you are using the correct version. +You should also download a package manager like yarn or npm, and make sure that you are using the correct version. ``` $ yarn install # or "npm install" @@ -67,11 +67,9 @@ to use. It's important to note that the tools will not recognize that the behavior is ready for use and valid if any of these three main components (the default export, **isBehavior**, and **metadata**) are missing. -The export `postStep` can be called after each action to convert the yielded results into the [expected format](https://github.com/webrecorder/behaviors/blob/master/typedef/index.html#static-typedef-BehaviorStepResults). - +The export `postStep` can be called after each action to convert the yielded results into the expected format. -It is recommended that you use the library function [lib.buildCustomPostStepFn](https://github.com/webrecorder/behaviors/blob/master/function/index.html#static-function-buildCustomPostStepFn) if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. - +It is recommended that you use the library function `lib.buildCustomPostStepFn`if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. ```js export const postStep = lib.buildCustomPostStepFn(() => { ... }); @@ -138,7 +136,6 @@ The purpose of an [asyncronous generator function](https://thecodebarbarian.com/ The primary reasons that a behavior's **default export** is required to be an async generator function or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator) are: - - async generators are widely supported in Javascript - they provide a simple way to run the behavior in the browser or through [browsertrix](https://github.com/webrecorder/browsertrix) - they allow information about the behavior and its state to be easily retrieved by code executing the behavior @@ -163,7 +160,8 @@ export const metadata = { export const isBehavior = true; ``` -In this example, `myBehavior` is an asynchronous generator function that's being exported for our behavior and this behavior would work for any website with myAwesomeWebSite.com in its name (notice the regex for the matching above). Much of this boiler plate code shown above has been generated automatically by the cli command [insert command here]. +In this example, `myBehavior` is an asynchronous generator function that's being exported for our behavior and this behavior would work for any website with myAwesomeWebSite.com in its name (notice the regex for the matching above). Much of this boiler plate code shown above has been generated automatically by the cli command +`./bin/cli newBehavior [behaviorName]`. #### Promises @@ -270,21 +268,36 @@ export const metadata = { export const isBehavior = true; ``` +## Testing your first behavior + +You should use the runner CLI to test your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. + +`$ ./bin/cli help runner` + +Please note that in order to provide automatic running of behaviors, this command must be able to launch the Chrome/Chromium browser. In other words, an already running instance of Chrome/Chromium can not be used. + +First, you will need to create a config file (yaml format) to use the runner. An example run config is provided for you and can found in the root of this project (`behavior-run-config.yml`). + +By using the example configuration file [`behavior-run-config.yml`](./behavior-run-config.yml) all that you have to do is change two fields: + +- behavior: the path to your new behavior in the behavior directory of this project +- url: the url of the page your behavior should be run in + + +The simplest way to use the runner is through the usage of a config file (yaml format) and can be supplied using the -c or --config flags like so: + +`./bin/cli runner -c `. -## Testing your first behavior Should mention the runner CLI in this section (this was found in the manual here https://github.com/webrecorder/behaviors/blob/master/manual/cli.md): - The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. -An example of the video playing thing embedded in a webpage. Show html code that was used for the website, js that was used to run a behavior. -Tell them to write this test example (video recording thing) for me and I can link to it in github. ## Fixing a broken behavior -When a behavior breaks, they click “fix this behavior,” and if someone has little js knowledge then they should be able to fix it +If a behavior is broken ## Checking behavior status diff --git a/chapter3.md b/chapter3.md index 2b28ca7..773b53b 100644 --- a/chapter3.md +++ b/chapter3.md @@ -3,6 +3,7 @@ advanced chapter 2 basically lib and metadata can go in here, should be able to control F to find what you're looking for in here ## Contributing to Autopilot ## Tools of the Trade + ## Advanced Behavior API What should be here that wasn't in chapter 2? all that advanced shit on metadata diff --git a/chapter4.md b/chapter4.md index 27c9233..9d19aea 100644 --- a/chapter4.md +++ b/chapter4.md @@ -1,5 +1,5 @@ # Behind Autopilot ## Maintainers ## Contact Support -## Discussion FOrum +## Discussion Forum ## Who Uses Autopilot? \ No newline at end of file From 223c244b45ef69220343c8ff48fbbcd86b02a3f3 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 14 Sep 2020 13:37:52 -1000 Subject: [PATCH 20/33] fixed some small formatting things --- chapter2.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/chapter2.md b/chapter2.md index 49ad388..7b88c6b 100644 --- a/chapter2.md +++ b/chapter2.md @@ -1,6 +1,6 @@ # Chapter 2: Creating your first behavior -A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the Pre-Made Behaviors on the Status Page . This will give you a basic idea of what kind of behaviors already have been created. +A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the Pre-Made Behaviors on the Status Page. This will give you a basic idea of what kind of behaviors have already been created. ## Setting up your file Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. @@ -8,6 +8,7 @@ Once you have an idea of what you want your behavior to do, clone the webrecorde 1. `mkdir webrecorder` (or skip steps 1-2 and use an already existing directory) 2. `cd webrecorder` 3. `git clone https://github.com/webrecorder/behaviors.git` +4. `cd behaviors` You should also download a package manager like yarn or npm, and make sure that you are using the correct version. ``` @@ -17,13 +18,15 @@ $ yarn install To check your version of npm, type the command `npm --version`. You can check the [npm website](https://www.npmjs.com/package/npm?activeTab=versions) to see what the latest version is. +You should also check to make sure that your version of node is 12 or above. + ### Getting started using the CLI You should now create a Javascript file for your new behavior. You can do this using the CLI. The `newBehavior` command provides a simple way to create a new behavior by generating a new file in the behavior directory containing the required boiler plate. -Executing `./bin/cli newBehavior awesomeBehavior` will create a new behavior file a`wesomeBehavior.js` located in the behavior directory. +Executing `./bin/cli newBehavior awesomeBehavior` will create a new behavior file `awesomeBehavior.js` located in the behavior directory. ### Format @@ -132,7 +135,7 @@ export const metadata = { #### Asyncronous Generator Functions -The purpose of an [asyncronous generator function](https://thecodebarbarian.com/async-generator-functions-in-javascript.html#:~:text=Async%20generator%20functions%20behave%20similarly,()%20function%20returns%20a%20promise.) is to collect data from a source that has too much data to return all at once (or would take too long to return all at once. Instead, the generator is a function that returns an object with a next() method on it. The programmer can keep calling next() until all of the data is received. An asynchronous generator function uses promises for the same purpose (promises will be expanded upon in the following section). +The purpose of an [asyncronous generator function](https://thecodebarbarian.com/async-generator-functions-in-javascript.html#:~:text=Async%20generator%20functions%20behave%20similarly,()%20function%20returns%20a%20promise.) is to collect data from a source that has too much data to return all at once (or would take too long to return all at once). Instead, the generator is a function that returns an object with a next() method on it. The programmer can keep calling next() until all of the data is received. An asynchronous generator function uses promises for the same purpose (promises will be expanded upon in the following section). The primary reasons that a behavior's **default export** is required to be an async generator function or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator) are: @@ -142,7 +145,7 @@ The primary reasons that a behavior's **default export** is required to be an as Consider the following example: -```javascript= +```js import * as lib from '../lib'; export default async function* myBehavior(cliAPI) { @@ -171,7 +174,7 @@ Now, without turning into a JavaScript tutorial, the only thing you need to know This comes in handy when you want to wait for the the document's state to become ready since the pages of myAwesomeWebSite.com take a long time to load and can be done as easily as shown in `step 1`. This is the beginning of the code that will go in the default async function. -```javascript= +```js // step 1, wait for the page to complete loading await lib.domCompletePromise(); ``` @@ -180,7 +183,7 @@ Now that we know the browser has fully loaded the page, we can safely start exec The next step is to initialize our state and report it, which is done by yielding a value as shown in `step 2`, where state contains the number of videos played. -```javascript= +```js // step 2, initialize state and return it const state = { videosPlayed: 0 }; yield lib.stateWithMsgNoWait('Document ready!', state); @@ -194,11 +197,11 @@ When you `yield` a value from the behavior you can consider the behavior paused Additionally, it should be noted that the second argument supplied to `lib.stateWithMsgNoWait` is optional but useful for reporting to yourself more detailed information about the state of your behavior. -Continuing on with the creation of our behavior, let us assume that the page that the behavior is operating in has a list videos we want to play. Our behavior will generate the next video to play each iteration. +Continuing on with the creation of our behavior, let us assume that the page that the behavior is operating in has a list of videos we want to play. Our behavior will generate the next video to play each iteration. We can accomplish this as shown in `step 3`. -```javascript= +```js // step 3 for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { // videoListItem is a child of the element with id == 'videos' @@ -230,7 +233,7 @@ In short, `step 3` can be described as playing a video contained in every child The full behavior is shown below: -```javascript= +```js import * as lib from '../lib'; export default async function* myBehavior(cliAPI) { @@ -270,6 +273,8 @@ export const isBehavior = true; ## Testing your first behavior +Blocked by [PR 63](https://github.com/webrecorder/behaviors/pull/63) + You should use the runner CLI to test your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. `$ ./bin/cli help runner` From edbfea46669831835bdb8fea27488f9de6f6215e Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 14 Sep 2020 13:58:19 -1000 Subject: [PATCH 21/33] moved chapters to manual --- chapter1.md => manual/chapter1.md | 6 +++--- chapter2.md => manual/chapter2.md | 4 ---- chapter3.md => manual/chapter3.md | 0 chapter4.md => manual/chapter4.md | 1 + 4 files changed, 4 insertions(+), 7 deletions(-) rename chapter1.md => manual/chapter1.md (98%) rename chapter2.md => manual/chapter2.md (98%) rename chapter3.md => manual/chapter3.md (100%) rename chapter4.md => manual/chapter4.md (98%) diff --git a/chapter1.md b/manual/chapter1.md similarity index 98% rename from chapter1.md rename to manual/chapter1.md index 67b380d..933d960 100644 --- a/chapter1.md +++ b/manual/chapter1.md @@ -33,20 +33,20 @@ To install Autopilot, use Terminal to clone the Webrecorder repository: The Autopilot instance can be accessed in the browser at `http://localhost:8089`. -## Autopilot basics +# Autopilot basics Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. The cli provides two commands API and behaviors and each command has its own options. -# Behaviors command +## Behaviors command Execute `./bin/cli behaviors -b` to build the behaviors made available, using the config file located at the root of the project. The built behaviors, along with a behavior metadata file (`behaviorMetadata.js`), can be found in the `dist` directory which will be created for you if it does not exist in the root of this project. -# API command +## API command To run the behavior API server execute `./bin/cli api --build-behaviors`. diff --git a/chapter2.md b/manual/chapter2.md similarity index 98% rename from chapter2.md rename to manual/chapter2.md index 7b88c6b..564a984 100644 --- a/chapter2.md +++ b/manual/chapter2.md @@ -295,10 +295,6 @@ The simplest way to use the runner is through the usage of a config file (yaml f -Should mention the runner CLI in this section (this was found in the manual here https://github.com/webrecorder/behaviors/blob/master/manual/cli.md): - - - ## Fixing a broken behavior diff --git a/chapter3.md b/manual/chapter3.md similarity index 100% rename from chapter3.md rename to manual/chapter3.md diff --git a/chapter4.md b/manual/chapter4.md similarity index 98% rename from chapter4.md rename to manual/chapter4.md index 9d19aea..7eae327 100644 --- a/chapter4.md +++ b/manual/chapter4.md @@ -1,5 +1,6 @@ # Behind Autopilot ## Maintainers + ## Contact Support ## Discussion Forum ## Who Uses Autopilot? \ No newline at end of file From d038ae7b5074850a9737f7a9f685a15df4ce832b Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 14 Sep 2020 15:37:30 -1000 Subject: [PATCH 22/33] wrote directions for how to set up config file in "testing your first behavior" o Please enter the commit message for your changes. Lines starting --- chapter2.md | 326 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 326 insertions(+) create mode 100644 chapter2.md diff --git a/chapter2.md b/chapter2.md new file mode 100644 index 0000000..5999564 --- /dev/null +++ b/chapter2.md @@ -0,0 +1,326 @@ +# Chapter 2: Creating your first behavior + +A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the [Pre-Made Behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/premade-behaviors.md). This will give you a basic idea of what kind of behaviors have already been created. + +## Setting up your file +Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. + +1. `mkdir webrecorder` (or skip steps 1-2 and use an already existing directory) +2. `cd webrecorder` +3. `git clone https://github.com/webrecorder/behaviors.git` +4. `cd behaviors` + +You should also download a package manager like yarn or npm, and make sure that you are using the correct version. +``` +$ yarn install +# or "npm install" +``` + +To check your version of npm, type the command `npm --version`. You can check the [npm website](https://www.npmjs.com/package/npm?activeTab=versions) to see what the latest version is. + +You should also check to make sure that your version of node is 12 or above. + +### Getting started using the CLI + +You should now create a Javascript file for your new behavior. You can do this using the CLI. + +The `newBehavior` command provides a simple way to create a new behavior by generating a new file in the behavior directory containing the required boiler plate. + +Executing `./bin/cli newBehavior awesomeBehavior` will create a new behavior file `awesomeBehavior.js` located in the behavior directory. + + +### Format + + + +Every behavior has: + +1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). This is the function that will retrieve the metadata we want from the webpage. + +```js +export detault async function* myBehavior(cliAPI){...} +``` + +More details on how the default export should be written can be found under the [Default Export](#default-export) section. + +2. an export named ***metadata*** that is an object + + Details: the ***metadata*** object gives information about the behavior, such as when it was last updated, whether it's functional, its display name, etc. + + Example of expected format: +```js + export const metadata = { ... }; +``` + +More details on how the metadata object should be written can be found under the [Metadata](#metadata) section. + +3. an export named **isBehavior**. + + Details: **isBehavior** is a constant that you will flag as **true** when the file is complete and ready to be used. Otherwise, while the file is still in progress, keep it as **false**. + + Example of expected format: +```js +export const isBehavior = true; +``` + + +It's important to note that the tools will not recognize that the behavior is ready for use and valid if any of these three main components (the default export, **isBehavior**, and **metadata**) are missing. + +The export `postStep` can be called after each action to convert the yielded results into the expected format. + +It is recommended that you use the library function `lib.buildCustomPostStepFn`if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. + +```js +export const postStep = lib.buildCustomPostStepFn(() => { ... }); +``` + +### Metadata (#metadata-heading) +A behavior's exported metadata object is used to: + +- describe how the behavior should be matched to the pages it is written for +- provide an overview of what the behavior does +- provide a more specific name associated with it when querying for it using the behavior api +- embed any additional information about the behavior + +With those usages in mind, every metadata object is expected to have the following properties: + +- **name** (string): the name for your behavior to be used when querying the behavior API for it by name +- **description** (string): a description of the behavior +- **match** (object): how the behavior will be matched to the page(s) it is written for + +The **match** object has two variations and is shown below in the context of two valid metadata exports. + +```js +// variation 1 +export const metadata = { + name: 'the name of your behavior', + match: { + regex: /a regular expression dictating the URL the behavior will run on/, + }, + description: 'an description of what your behavior does', +}; +``` +The first variation of `match`, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. + + +The second variation of `match`, shown below, has the two properties `base` (RegExp) and `sub` (Array). + +The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. The behavior is considered matched to a URL when the `base` regular expression matches the URL and one of the `sub` regular expressions also matches the URL. + + + +```js +// variation 2 +export const metadata = { + name: 'the name of your behavior', + match: { + regex: { + base: /a regular expression dictating the base URL the behavior will run on/, + sub: [ + /an array of regular expressions dictating more specific parts of the base URL the behavior will run on/, + ], + }, + }, + description: 'a description of what your behavior does', +}; +``` + + + +### Default Export(#default-heading) + +#### Asyncronous Generator Functions + +The purpose of an [asyncronous generator function](https://thecodebarbarian.com/async-generator-functions-in-javascript.html#:~:text=Async%20generator%20functions%20behave%20similarly,()%20function%20returns%20a%20promise.) is to collect data from a source that has too much data to return all at once (or would take too long to return all at once). Instead, the generator is a function that returns an object with a next() method on it. The programmer can keep calling next() until all of the data is received. An asynchronous generator function uses promises for the same purpose (promises will be expanded upon in the following section). + +The primary reasons that a behavior's **default export** is required to be an async generator function or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator) are: + +- async generators are widely supported in Javascript +- they provide a simple way to run the behavior in the browser or through [browsertrix](https://github.com/webrecorder/browsertrix) +- they allow information about the behavior and its state to be easily retrieved by code executing the behavior + +Consider the following example: + +```js +import * as lib from '../lib'; + +export default async function* myBehavior(cliAPI) { + // behavior code will be developed here +} + +export const metadata = { + name: 'myBehavior', + match: { + regex: /^(?:https?:\/\/(?:www\.)?)?:myAwesomeWebSite.com.*$/, + }, + description: 'It does really cool stuff', +}; + +export const isBehavior = true; +``` + +In this example, `myBehavior` is an asynchronous generator function that's being exported for our behavior and this behavior would work for any website with myAwesomeWebSite.com in its name (notice the regex for the matching above). Much of this boiler plate code shown above has been generated automatically by the cli command +`./bin/cli newBehavior [behaviorName]`. + +#### Promises + +Next, you may have notticed looking over some of the behavior documentation that many of the functions return [Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise). + +Now, without turning into a JavaScript tutorial, the only thing you need to know about promises is that they can be awaited, that is to say you can await for their completion. + +This comes in handy when you want to wait for the the document's state to become ready since the pages of myAwesomeWebSite.com take a long time to load and can be done as easily as shown in `step 1`. This is the beginning of the code that will go in the default async function. + +```js +// step 1, wait for the page to complete loading +await lib.domCompletePromise(); +``` + +Now that we know the browser has fully loaded the page, we can safely start executing behaviors on the page. + +The next step is to initialize our state and report it, which is done by yielding a value as shown in `step 2`, where state contains the number of videos played. + +```js +// step 2, initialize state and return it +const state = { videosPlayed: 0 }; +yield lib.stateWithMsgNoWait('Document ready!', state); +``` + +The function `lib.stateWithMsgNoWait` indicates to the code executing the behavior that it has an update to report and that it does not have to wait. + +If the behavior was being run by [browsertrix](https://github.com/webrecorder/browsertrix) and the other function `lib.stateWithMsgWait` was used, browsertrix would have waited until the HTTP requests made by the page had significantly slowed down (no request made for set period of time) before executing further code in the function. However, since we use the *no wait* variant, the behavior will immediately return state. + +When you `yield` a value from the behavior you can consider the behavior paused until the runner initiates the `next()` action. + +Additionally, it should be noted that the second argument supplied to `lib.stateWithMsgNoWait` is optional but useful for reporting to yourself more detailed information about the state of your behavior. + +Continuing on with the creation of our behavior, let us assume that the page that the behavior is operating in has a list of videos we want to play. Our behavior will generate the next video to play each iteration. + +We can accomplish this as shown in `step 3`. + +```js +// step 3 +for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { + // videoListItem is a child of the element with id == 'videos' + // that has a video as child element + const videoWasPlayed = await lib.selectAndPlay('video', videoListItem); + if (videoWasPlayed) { + // increment our states counter for number of videos played + state.videosPlayed += 1; + // let ourselves know we played a video + yield lib.stateWithMsgNoWait('Played a video!', state); + } else { + yield lib.stateWithMsgNoWait('Failed to play a video using the standard DOM methods', state); + } +} +return lib.stateWithMsgNoWait('Done!', state); +``` + +In `step 3` we use the function `childElementIterator` that returns an iterator over the child elements of the supplied parent element. Then for each child of the element with `id="videos"` we: + +- select the video element that is a descendant of the `videoListItems` and play the video +- increment the number of videos played in our behavior's state +- let ourselves know that we played a video + +Also seen in `step 3` is the usage keyword `yield*`. + +`yield*` means that we are yielding another generator, that is to say all actions of the generator are to be treated as if we yielded them ourselves. + +In short, `step 3` can be described as playing a video contained in every child of the element with `id == video`, and once we have played all the videos on the page return a message with our final state from the behavior. + +The full behavior is shown below: + +```js +import * as lib from '../lib'; + +export default async function* myBehavior(cliAPI) { + // behavior code + // step 1 + await lib.domCompletePromise(); + // step 2 + const state = { videosPlayed: 0 }; + yield lib.stateWithMsgNoWait('Document ready!', state); + // step 3 + for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { + // videoListItem is a child of the element with id == 'videos' + // that has a video as child element + const videoWasPlayed = await lib.selectAndPlay('video', videoListItem); + if (videoWasPlayed) { + // increment our states counter for number of videos played + state.videosPlayed += 1; + // let ourselves know we played a video + yield lib.stateWithMsgNoWait('Played a video!', state); + } else { + yield lib.stateWithMsgNoWait('Failed to play a video using the standard DOM methods', state); + } + } + return lib.stateWithMsgNoWait('Done!', state); +} + +export const metadata = { + name: 'myBehavior', + match: { + regex: /^(?:https?:\/\/(?:www\.)?)?:myAwesomeWebSite.com.*$/, + }, + description: 'It does really cool stuff', +}; + +export const isBehavior = true; +``` + +## Testing your first behavior + +Blocked by [PR 63](https://github.com/webrecorder/behaviors/pull/63) + +You should use the runner CLI to test your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. This will allow you to see if your behavior is working correctly. + +`$ ./bin/cli help runner` + +Please note that in order to provide automatic running of behaviors, this command must be able to launch the Chrome/Chromium browser. In other words, an already running instance of Chrome/Chromium can not be used. + +First, you will need to create a config file (yaml format) to use the runner. A run config file is provided for you and can found in the root of this project (`behavior-run-config.yml`). + +The file will look like this: +```yaml +behaviors: ./behaviors +lib: ./lib +build: ./build +dist: ./dist +tsconfig: ./tsconfig.json +metadata: ./dist/behaviorMetadata.js +``` + +By using the provided configuration file [`behavior-run-config.yml`](./behavior-run-config.yml) all that you have to do is change two fields: + +1. `behaviors`: the path to your new behavior in the behavior directory of this project +2. `url`: the url of the page your behavior should be run in + +Now you can start the build and run process by executing the following from the root directory of this project: + +`./bin/cli runner -c behavior-run-config.yml` + +The command will launch a Chrome/Chromium browser installed on your computer, build the behavior, and then run it until completion. + +If any changes are made to the behavior or any of the files it includes while the behavior is being run, it will be rebuilt and re-run automatically. + + + +## Fixing a broken behavior + +If you nottice that a behavior isn't working right, you can use these tips to troubleshoot what might be going wrong. + +First, go into the file for the behavior. Next, you should check for these easily fixable problems: + +1. Does the behavior have the 3 essential parts (the default export, **isBehavior**, and **metadata**)? The tools will not recognize that the behavior is ready for use and valid if any of these three components are missing. +2. Is the **isBehavior** export flagged as false? The tools will not recognize that the behavior is ready for use if **isBehavior** is marked as **false**. +3. + + + + + From 2fa30a06809dd3f60c1484df505c056184f34469 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 15 Sep 2020 13:54:13 -1000 Subject: [PATCH 23/33] wrote the "fixing a broken behavior" section --- chapter2.md | 47 ++++++++++++++--------------------------------- 1 file changed, 14 insertions(+), 33 deletions(-) diff --git a/chapter2.md b/chapter2.md index 5999564..645b2ea 100644 --- a/chapter2.md +++ b/chapter2.md @@ -31,8 +31,6 @@ Executing `./bin/cli newBehavior awesomeBehavior` will create a new behavior fil ### Format - - Every behavior has: 1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). This is the function that will retrieve the metadata we want from the webpage. @@ -78,7 +76,7 @@ It is recommended that you use the library function `lib.buildCustomPostStepFn`i export const postStep = lib.buildCustomPostStepFn(() => { ... }); ``` -### Metadata (#metadata-heading) +### Metadata A behavior's exported metadata object is used to: - describe how the behavior should be matched to the pages it is written for @@ -106,32 +104,9 @@ export const metadata = { ``` The first variation of `match`, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. +The second variation of `match` is a bit more advanced and lets you compare your URL to multiple sub URLs. You can get more information on that in the [next chapter](). -The second variation of `match`, shown below, has the two properties `base` (RegExp) and `sub` (Array). - -The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. The behavior is considered matched to a URL when the `base` regular expression matches the URL and one of the `sub` regular expressions also matches the URL. - - - -```js -// variation 2 -export const metadata = { - name: 'the name of your behavior', - match: { - regex: { - base: /a regular expression dictating the base URL the behavior will run on/, - sub: [ - /an array of regular expressions dictating more specific parts of the base URL the behavior will run on/, - ], - }, - }, - description: 'a description of what your behavior does', -}; -``` - - - -### Default Export(#default-heading) +### Default Export #### Asyncronous Generator Functions @@ -310,13 +285,19 @@ If any changes are made to the behavior or any of the files it includes while th ## Fixing a broken behavior -If you nottice that a behavior isn't working right, you can use these tips to troubleshoot what might be going wrong. +A behavior is broken when it doesn't work like it used to. Behaviors have to be fixed sometimes as websites can change at anytime. For example, a website might change its css and html names and classes, or how the Javascript behaves when you scroll. If a behavior depends on something that has changed, it may stop working. + +Here are the steps to fixing a broken behavior: -First, go into the file for the behavior. Next, you should check for these easily fixable problems: +1. Find the file of the behavior +2. Run the behavior +3. Note the output and check for console errors +4. Update the offending lines +5. [Create a fork](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) for your contributions +6. Commit and push your changes to your local branch +7. Create a pull request against the master branch -1. Does the behavior have the 3 essential parts (the default export, **isBehavior**, and **metadata**)? The tools will not recognize that the behavior is ready for use and valid if any of these three components are missing. -2. Is the **isBehavior** export flagged as false? The tools will not recognize that the behavior is ready for use if **isBehavior** is marked as **false**. -3. +You will then receive feedback on your work, which you can respond to in the pull request. From d1eb53c77acf1540f283b71a6ab0cdffb655b53b Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Tue, 15 Sep 2020 22:01:33 -1000 Subject: [PATCH 24/33] I don't remember what my changes were --- chapter2.md | 6 +++--- chapter4.md | 30 ++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 3 deletions(-) create mode 100644 chapter4.md diff --git a/chapter2.md b/chapter2.md index 645b2ea..55e5dd8 100644 --- a/chapter2.md +++ b/chapter2.md @@ -90,7 +90,7 @@ With those usages in mind, every metadata object is expected to have the followi - **description** (string): a description of the behavior - **match** (object): how the behavior will be matched to the page(s) it is written for -The **match** object has two variations and is shown below in the context of two valid metadata exports. +The **match** object has two variations, but you will probably only need to know the first. ```js // variation 1 @@ -104,7 +104,7 @@ export const metadata = { ``` The first variation of `match`, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. -The second variation of `match` is a bit more advanced and lets you compare your URL to multiple sub URLs. You can get more information on that in the [next chapter](). +The second variation of `match` is a bit more advanced and lets you compare your URL to multiple sub URLs. You can get more information on that in [chapter 3](./chapter3). ### Default Export @@ -285,7 +285,7 @@ If any changes are made to the behavior or any of the files it includes while th ## Fixing a broken behavior -A behavior is broken when it doesn't work like it used to. Behaviors have to be fixed sometimes as websites can change at anytime. For example, a website might change its css and html names and classes, or how the Javascript behaves when you scroll. If a behavior depends on something that has changed, it may stop working. +A behavior is broken when it doesn't work as expected or like it used to. Behaviors have to be fixed sometimes as websites can change at anytime. For example, a website might change its css and html names and classes, or how the Javascript behaves when you scroll. If a behavior depends on something that has changed, it may stop working. Here are the steps to fixing a broken behavior: diff --git a/chapter4.md b/chapter4.md new file mode 100644 index 0000000..75c7d63 --- /dev/null +++ b/chapter4.md @@ -0,0 +1,30 @@ +# Behind Autopilot +## Maintainers + +## Contact Support + + + +## Discussion Forum + +Use the [discussion forum](https://forum.webrecorder.net) to discuss ideas, questions, issues, etc. with the webrecorder community. + +## Who Uses Autopilot? +We are happy to have a growing community of academic and memory institutions that are using Webrecorder tools and software. + +Some of our colleagues that are using various Webrecorder tools: + +![DocNow](https://webrecorder.net/assets/logos/docnow-logo.png) +![ArQuivo.PT](https://webrecorder.net/assets/logos/arquivo-logo.png) +![UKWA](https://webrecorder.net/assets/logos/ukwa-logo.svg) +![LOCKSS](https://webrecorder.net/assets/logos/lockss-logo.png) +![Rhizome](https://webrecorder.net/assets/logos/rhizome-logo.png) +![ONF NFB](https://webrecorder.net/assets/logos/nfb-logo.svg) +![IIPC](https://webrecorder.net/assets/logos/iipc-logo.png) +![Stanford University Press](https://webrecorder.net/assets/logos/sup-logo.png) +![The National Archives](https://webrecorder.net/assets/logos/tna-logo.svg) +![National Library of Australia](https://webrecorder.net/assets/logos/nla-logo.png) +![Perma.cc](https://webrecorder.net/assets/logos/perma-logo.png) +![CLOCKSS](https://webrecorder.net/assets/logos/clockss-logo.png) + +*If your institution is using Webrecorder software and would like to be listed, please let us know!* \ No newline at end of file From ae14f2c437dc88dce7e85e7bff69cd76eb15a491 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Wed, 16 Sep 2020 11:02:52 -1000 Subject: [PATCH 25/33] started the FAQ document --- manual/FAQ.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 manual/FAQ.md diff --git a/manual/FAQ.md b/manual/FAQ.md new file mode 100644 index 0000000..2948e5b --- /dev/null +++ b/manual/FAQ.md @@ -0,0 +1,20 @@ +# FAQ +## Is Autopilot open source? + +Yes. All Webrecorder tools are licensed under open source licenses. + +From the beginning, the goal of Webrecorder has been to build quality open source tools enable ‘web archiving for all’ to allow anyone with a browser to create their own web archives, and to accurately replay them at a later time. + +## What does Autopilot cost? + +Autopilot is free to use. + +## What platforms does Autopilot run on? + + + +## How can I contribute to Autopilot? + +Read the chapter on [Hacking Autopilot](./chapter2.md) for a guide on contributing to Autopilot. + +##Glossary of Terms From 03c17d6712d20612095800c8d949c94e551f3cc8 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Wed, 16 Sep 2020 13:44:52 -1000 Subject: [PATCH 26/33] moved some advanced stuff from chapter 2 into chapter 3 --- chapter2.md | 8 -------- chapter4.md | 5 ++++- manual/FAQ.md | 4 +++- 3 files changed, 7 insertions(+), 10 deletions(-) diff --git a/chapter2.md b/chapter2.md index 55e5dd8..6cc8871 100644 --- a/chapter2.md +++ b/chapter2.md @@ -68,14 +68,6 @@ to use. It's important to note that the tools will not recognize that the behavior is ready for use and valid if any of these three main components (the default export, **isBehavior**, and **metadata**) are missing. -The export `postStep` can be called after each action to convert the yielded results into the expected format. - -It is recommended that you use the library function `lib.buildCustomPostStepFn`if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. - -```js -export const postStep = lib.buildCustomPostStepFn(() => { ... }); -``` - ### Metadata A behavior's exported metadata object is used to: diff --git a/chapter4.md b/chapter4.md index 75c7d63..235d609 100644 --- a/chapter4.md +++ b/chapter4.md @@ -1,9 +1,12 @@ # Behind Autopilot + ## Maintainers -## Contact Support +Autopilot is maintained by: +## Contact Support +Email [insert email] for support questions. ## Discussion Forum diff --git a/manual/FAQ.md b/manual/FAQ.md index 2948e5b..111af8b 100644 --- a/manual/FAQ.md +++ b/manual/FAQ.md @@ -15,6 +15,8 @@ Autopilot is free to use. ## How can I contribute to Autopilot? -Read the chapter on [Hacking Autopilot](./chapter2.md) for a guide on contributing to Autopilot. +Read the chapter on [Hacking Autopilot](./chapter3.md) for a guide on contributing to Autopilot. We suggest that beginner developers take a look at [Chapter 2](./chapter2.md) for a simpler overview of how Autopilot works. ##Glossary of Terms + + From 005fa31b66f4df8cda80a2bad8f7e5f732a8af8d Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Wed, 16 Sep 2020 17:35:23 -1000 Subject: [PATCH 27/33] added a code of conduct to the manual --- manual/code-of-conduct.md | 76 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 manual/code-of-conduct.md diff --git a/manual/code-of-conduct.md b/manual/code-of-conduct.md new file mode 100644 index 0000000..f6100db --- /dev/null +++ b/manual/code-of-conduct.md @@ -0,0 +1,76 @@ +[![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-v2.0%20adopted-ff69b4.svg)](code_of_conduct.md) + +## Code of Conduct + +### Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to making participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, gender identity and expression, level of experience, +nationality, personal appearance, race, religion, or sexual identity and +orientation. + +### Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or +advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic + address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +### Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +### Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +### Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at [INSERT EMAIL ADDRESS]. All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +### Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at [http://contributor-covenant.org/version/1/4][version] + +[homepage]: http://contributor-covenant.org +[version]: http://contributor-covenant.org/version/1/4/ \ No newline at end of file From ce81db2fa0a4a6d9bb5b5f149afc109a0904fdbb Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Mon, 21 Sep 2020 10:58:48 -1000 Subject: [PATCH 28/33] removed some stuff that was covered in chapter 2 --- chapter3.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100644 chapter3.md diff --git a/chapter3.md b/chapter3.md new file mode 100644 index 0000000..366e6fd --- /dev/null +++ b/chapter3.md @@ -0,0 +1,56 @@ +# Hacking Autopilot + +## Contributing to Autopilot + +We welcome contributions from anyone with beginning to advanced programming knowledge. + +Please be advised that we have a [code of conduct](./code-of-conduct.md) that we expect all contributors to follow. + +### Reporting bugs + + +## Tools of the Trade + +### Advanced options + +#### postStep +The export `postStep` can be called after each action to convert the yielded results into the expected format. + +It is recommended that you use the library function `lib.buildCustomPostStepFn`if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. + +```js +export const postStep = lib.buildCustomPostStepFn(() => { ... }); +``` + +#### Metadata export: match +If you want to compare your URL to multiple sub URLs, you can use a different version of `match`. This version is shown below, and has the two properties `base` (RegExp) and `sub` (Array). + +The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. The behavior is considered matched to a URL when the `base` regular expression matches the URL and one of the `sub` regular expressions also matches the URL. + +```js +// variation 2 +export const metadata = { + name: 'the name of your behavior', + match: { + regex: { + base: /a regular expression dictating the base URL the behavior will run on/, + sub: [ + /an array of regular expressions dictating more specific parts of the base URL the behavior will run on/, + ], + }, + }, + description: 'a description of what your behavior does', +}; +``` +## Testing a behavior (Debugging) +Blocked by PR 63 +## Behaviors +In pre made behaviors file +## Build System +## CLI +write docs on npm or yarn scripts +revamp the cli as scripts package.json scripts + +## Provided CLI Commands +## Behavior Standard Library Reference + \ No newline at end of file From 4436c7849e9b879f2cf0c06f22aa856710b31594 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Wed, 23 Sep 2020 10:36:51 -1000 Subject: [PATCH 29/33] updated chapter 1 --- chapter1.md | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 chapter1.md diff --git a/chapter1.md b/chapter1.md new file mode 100644 index 0000000..61fefe8 --- /dev/null +++ b/chapter1.md @@ -0,0 +1,84 @@ +# Chapter 1 + + +## What is web archiving? + +Web archiving is the process of recording web resources. Various elements of the website such as HTML, scripts, images, videos, etc. can be recorded to preserve as much of the original resource as possible. By creating dynamic archives that provide a user with the same experience they would have if they accessed the original site, web archives can give much more information than a static screenshot. + +Web archives may be utilized by future researchers, historians and the general public. + +## Why Autopilot? + +Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. Furthermore, Autopilot is designed to be accessible for anyone to use, and anyone with little javascript knowledge to contribute to. This makes webarchiving available to everyone. + +## Installing Autopilot + +To use this project you must first install its dependencies. You can do this via a package manager like [Yarn](https://classic.yarnpkg.com/en/docs/install/#mac-stable) or [npm](https://www.npmjs.com/). + +``` +$ yarn install +# or "npm install" +``` + +If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). + +To install Autopilot, use Terminal to clone the Webrecorder repository: +``` +1. git clone https://github.com/webrecorder/behaviors.git +2. cd autopilot; bash init-default.sh. +3. docker-compose build +4. docker-compose up -d +``` + +The Autopilot instance can be accessed in the browser at `http://localhost:8089`. + + +# Autopilot basics +Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. + + +The cli provides two commands API and behaviors and each command has its own options. + +## Behaviors command + +Execute `./bin/cli behaviors -b` to build the behaviors made available, using the config file located at the root of the project. + +The built behaviors, along with a behavior metadata file (`behaviorMetadata.js`), can be found in the `dist` directory which will be created for you if it does not exist in the root of this project. + + +## API command + +To run the behavior API server execute `./bin/cli api --build-behaviors`. + +This will start the API server after all behaviors provided by this project have been built. + +If you have already built the behaviors using the `behaviors` command provided by the cli then you may omit the `--build-behaviors` flag. + + +Some configuration of the API server can be done via the environment variables listed below + +* `BEHAVIOR_API_HOST`: the host the api server will use (e.g. 127.0.0.1) +* `BEHAVIOR_API_PORT`: the port the api server will listen on (e.g. 3030) +* `WR_BEHAVIOR_DIR`: path to the directory containing the built behaviors +* `WR_BEHAVIOR_METADATA_PATH`: path to the behavior metadata file +* `BUILD_BEHAVIORS`: should the api server build the behaviors before starting + + + +# Docker +If you would like, you can use Docker to run Autopilot. + +To build the wr-behaviors docker image (`webrecorder/behaviors:latest`) execute `docker-compose build`. + +The image created is suitable for building behaviors and running the behavior api server. + +The default configuration of the image is to run the api server, however you can substitute the default command with any of the cli commands listed previously. + +For more information please consult the provided `Dockerfile` and `docker-compose.yml` files. + +## About Web Traffic + + +## Status page: pre-made behaviors + + From 422c4c4bd8f96b2d7507dd2a16d71c09952193bb Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Wed, 23 Sep 2020 10:54:51 -1000 Subject: [PATCH 30/33] moved locations of the chapters --- manual/chapter1.md | 19 ++++------ manual/chapter2.md | 88 +++++++++++++++++++++------------------------- manual/chapter3.md | 58 ++++++++++++++++++++++++------ manual/chapter4.md | 29 ++++++++++++++- 4 files changed, 121 insertions(+), 73 deletions(-) diff --git a/manual/chapter1.md b/manual/chapter1.md index 933d960..031d55c 100644 --- a/manual/chapter1.md +++ b/manual/chapter1.md @@ -20,12 +20,14 @@ $ yarn install # or "npm install" ``` -If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). - To install Autopilot, use Terminal to clone the Webrecorder repository: ``` 1. git clone https://github.com/webrecorder/behaviors.git 2. cd autopilot; bash init-default.sh. +``` +If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). + +``` 3. docker-compose build 4. docker-compose up -d ``` @@ -66,6 +68,8 @@ Some configuration of the API server can be done via the environment variables l # Docker +If you would like, you can use Docker to run Autopilot. + To build the wr-behaviors docker image (`webrecorder/behaviors:latest`) execute `docker-compose build`. The image created is suitable for building behaviors and running the behavior api server. @@ -77,17 +81,6 @@ For more information please consult the provided `Dockerfile` and `docker-compos ## About Web Traffic -//consider making the status page space a seperate chapter or a seperate section because it kind of dominates chapter 1. ## Status page: pre-made behaviors -# Autoscroll - - -# Instagram - -# Twitter - -# Youtube Video - - diff --git a/manual/chapter2.md b/manual/chapter2.md index 564a984..6cc8871 100644 --- a/manual/chapter2.md +++ b/manual/chapter2.md @@ -1,6 +1,6 @@ # Chapter 2: Creating your first behavior -A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the Pre-Made Behaviors on the Status Page. This will give you a basic idea of what kind of behaviors have already been created. +A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the [Pre-Made Behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/premade-behaviors.md). This will give you a basic idea of what kind of behaviors have already been created. ## Setting up your file Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. @@ -31,8 +31,6 @@ Executing `./bin/cli newBehavior awesomeBehavior` will create a new behavior fil ### Format - - Every behavior has: 1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). This is the function that will retrieve the metadata we want from the webpage. @@ -70,15 +68,7 @@ to use. It's important to note that the tools will not recognize that the behavior is ready for use and valid if any of these three main components (the default export, **isBehavior**, and **metadata**) are missing. -The export `postStep` can be called after each action to convert the yielded results into the expected format. - -It is recommended that you use the library function `lib.buildCustomPostStepFn`if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. - -```js -export const postStep = lib.buildCustomPostStepFn(() => { ... }); -``` - -### Metadata (#metadata-heading) +### Metadata A behavior's exported metadata object is used to: - describe how the behavior should be matched to the pages it is written for @@ -92,7 +82,7 @@ With those usages in mind, every metadata object is expected to have the followi - **description** (string): a description of the behavior - **match** (object): how the behavior will be matched to the page(s) it is written for -The **match** object has two variations and is shown below in the context of two valid metadata exports. +The **match** object has two variations, but you will probably only need to know the first. ```js // variation 1 @@ -106,32 +96,9 @@ export const metadata = { ``` The first variation of `match`, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. +The second variation of `match` is a bit more advanced and lets you compare your URL to multiple sub URLs. You can get more information on that in [chapter 3](./chapter3). -The second variation of `match`, shown below, has the two properties `base` (RegExp) and `sub` (Array). - -The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. The behavior is considered matched to a URL when the `base` regular expression matches the URL and one of the `sub` regular expressions also matches the URL. - - - -```js -// variation 2 -export const metadata = { - name: 'the name of your behavior', - match: { - regex: { - base: /a regular expression dictating the base URL the behavior will run on/, - sub: [ - /an array of regular expressions dictating more specific parts of the base URL the behavior will run on/, - ], - }, - }, - description: 'a description of what your behavior does', -}; -``` - - - -### Default Export(#default-heading) +### Default Export #### Asyncronous Generator Functions @@ -275,33 +242,58 @@ export const isBehavior = true; Blocked by [PR 63](https://github.com/webrecorder/behaviors/pull/63) -You should use the runner CLI to test your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. +You should use the runner CLI to test your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. This will allow you to see if your behavior is working correctly. `$ ./bin/cli help runner` Please note that in order to provide automatic running of behaviors, this command must be able to launch the Chrome/Chromium browser. In other words, an already running instance of Chrome/Chromium can not be used. -First, you will need to create a config file (yaml format) to use the runner. An example run config is provided for you and can found in the root of this project (`behavior-run-config.yml`). +First, you will need to create a config file (yaml format) to use the runner. A run config file is provided for you and can found in the root of this project (`behavior-run-config.yml`). + +The file will look like this: +```yaml +behaviors: ./behaviors +lib: ./lib +build: ./build +dist: ./dist +tsconfig: ./tsconfig.json +metadata: ./dist/behaviorMetadata.js +``` -By using the example configuration file [`behavior-run-config.yml`](./behavior-run-config.yml) all that you have to do is change two fields: +By using the provided configuration file [`behavior-run-config.yml`](./behavior-run-config.yml) all that you have to do is change two fields: -- behavior: the path to your new behavior in the behavior directory of this project -- url: the url of the page your behavior should be run in +1. `behaviors`: the path to your new behavior in the behavior directory of this project +2. `url`: the url of the page your behavior should be run in +Now you can start the build and run process by executing the following from the root directory of this project: -The simplest way to use the runner is through the usage of a config file (yaml format) and can be supplied using the -c or --config flags like so: +`./bin/cli runner -c behavior-run-config.yml` -`./bin/cli runner -c `. +The command will launch a Chrome/Chromium browser installed on your computer, build the behavior, and then run it until completion. +If any changes are made to the behavior or any of the files it includes while the behavior is being run, it will be rebuilt and re-run automatically. ## Fixing a broken behavior -If a behavior is broken +A behavior is broken when it doesn't work as expected or like it used to. Behaviors have to be fixed sometimes as websites can change at anytime. For example, a website might change its css and html names and classes, or how the Javascript behaves when you scroll. If a behavior depends on something that has changed, it may stop working. + +Here are the steps to fixing a broken behavior: + +1. Find the file of the behavior +2. Run the behavior +3. Note the output and check for console errors +4. Update the offending lines +5. [Create a fork](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) for your contributions +6. Commit and push your changes to your local branch +7. Create a pull request against the master branch + +You will then receive feedback on your work, which you can respond to in the pull request. -## Checking behavior status -how do you check behavior status and is this done in js + diff --git a/manual/chapter3.md b/manual/chapter3.md index 773b53b..93171f8 100644 --- a/manual/chapter3.md +++ b/manual/chapter3.md @@ -1,19 +1,55 @@ # Hacking Autopilot -advanced chapter 2 basically -lib and metadata can go in here, should be able to control F to find what you're looking for in here + ## Contributing to Autopilot + +We welcome contributions from anyone with beginning to advanced programming knowledge. + +Please be advised that we have a [code of conduct](./code-of-conduct.md) that we expect all contributors to follow. + +### Reporting bugs + + ## Tools of the Trade -## Advanced Behavior API -What should be here that wasn't in chapter 2? -all that advanced shit on metadata +### Advanced options + +#### postStep +The export `postStep` can be called after each action to convert the yielded results into the expected format. + +It is recommended that you use the library function `lib.buildCustomPostStepFn`if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. + +```js +export const postStep = lib.buildCustomPostStepFn(() => { ... }); +``` + +#### Metadata export: match +If you want to compare your URL to multiple sub URLs, you can use a different version of `match`. This version is shown below, and has the two properties `base` (RegExp) and `sub` (Array). + +The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. The behavior is considered matched to a URL when the `base` regular expression matches the URL and one of the `sub` regular expressions also matches the URL. + +```js +// variation 2 +export const metadata = { + name: 'the name of your behavior', + match: { + regex: { + base: /a regular expression dictating the base URL the behavior will run on/, + sub: [ + /an array of regular expressions dictating more specific parts of the base URL the behavior will run on/, + ], + }, + }, + description: 'a description of what your behavior does', +}; +``` ## Testing a behavior (Debugging) -This also seems like it should be in chapter 2 -how the testing infrastructure is working (behavior testing advanced) -## Behaviors -Also chapter 1 and 2 item +Blocked by PR 63 (testing infrastructure) + ## Build System ## CLI -## Overview on Behaviors +write docs on npm or yarn scripts +revamp the cli as scripts package.json scripts + ## Provided CLI Commands -## Behavior Standard Library Reference \ No newline at end of file +## Behavior Standard Library Reference + \ No newline at end of file diff --git a/manual/chapter4.md b/manual/chapter4.md index 7eae327..235d609 100644 --- a/manual/chapter4.md +++ b/manual/chapter4.md @@ -1,6 +1,33 @@ # Behind Autopilot + ## Maintainers +Autopilot is maintained by: + ## Contact Support + +Email [insert email] for support questions. + ## Discussion Forum -## Who Uses Autopilot? \ No newline at end of file + +Use the [discussion forum](https://forum.webrecorder.net) to discuss ideas, questions, issues, etc. with the webrecorder community. + +## Who Uses Autopilot? +We are happy to have a growing community of academic and memory institutions that are using Webrecorder tools and software. + +Some of our colleagues that are using various Webrecorder tools: + +![DocNow](https://webrecorder.net/assets/logos/docnow-logo.png) +![ArQuivo.PT](https://webrecorder.net/assets/logos/arquivo-logo.png) +![UKWA](https://webrecorder.net/assets/logos/ukwa-logo.svg) +![LOCKSS](https://webrecorder.net/assets/logos/lockss-logo.png) +![Rhizome](https://webrecorder.net/assets/logos/rhizome-logo.png) +![ONF NFB](https://webrecorder.net/assets/logos/nfb-logo.svg) +![IIPC](https://webrecorder.net/assets/logos/iipc-logo.png) +![Stanford University Press](https://webrecorder.net/assets/logos/sup-logo.png) +![The National Archives](https://webrecorder.net/assets/logos/tna-logo.svg) +![National Library of Australia](https://webrecorder.net/assets/logos/nla-logo.png) +![Perma.cc](https://webrecorder.net/assets/logos/perma-logo.png) +![CLOCKSS](https://webrecorder.net/assets/logos/clockss-logo.png) + +*If your institution is using Webrecorder software and would like to be listed, please let us know!* \ No newline at end of file From 47ab4e8bd8a68df5d2327861c3160fb0972ba4c9 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Wed, 23 Sep 2020 10:57:19 -1000 Subject: [PATCH 31/33] moved locations of chapters --- chapter1.md | 84 --------------- chapter2.md | 299 ---------------------------------------------------- chapter3.md | 56 ---------- chapter4.md | 33 ------ 4 files changed, 472 deletions(-) delete mode 100644 chapter1.md delete mode 100644 chapter2.md delete mode 100644 chapter3.md delete mode 100644 chapter4.md diff --git a/chapter1.md b/chapter1.md deleted file mode 100644 index 61fefe8..0000000 --- a/chapter1.md +++ /dev/null @@ -1,84 +0,0 @@ -# Chapter 1 - - -## What is web archiving? - -Web archiving is the process of recording web resources. Various elements of the website such as HTML, scripts, images, videos, etc. can be recorded to preserve as much of the original resource as possible. By creating dynamic archives that provide a user with the same experience they would have if they accessed the original site, web archives can give much more information than a static screenshot. - -Web archives may be utilized by future researchers, historians and the general public. - -## Why Autopilot? - -Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. Furthermore, Autopilot is designed to be accessible for anyone to use, and anyone with little javascript knowledge to contribute to. This makes webarchiving available to everyone. - -## Installing Autopilot - -To use this project you must first install its dependencies. You can do this via a package manager like [Yarn](https://classic.yarnpkg.com/en/docs/install/#mac-stable) or [npm](https://www.npmjs.com/). - -``` -$ yarn install -# or "npm install" -``` - -If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). - -To install Autopilot, use Terminal to clone the Webrecorder repository: -``` -1. git clone https://github.com/webrecorder/behaviors.git -2. cd autopilot; bash init-default.sh. -3. docker-compose build -4. docker-compose up -d -``` - -The Autopilot instance can be accessed in the browser at `http://localhost:8089`. - - -# Autopilot basics -Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. - - -The cli provides two commands API and behaviors and each command has its own options. - -## Behaviors command - -Execute `./bin/cli behaviors -b` to build the behaviors made available, using the config file located at the root of the project. - -The built behaviors, along with a behavior metadata file (`behaviorMetadata.js`), can be found in the `dist` directory which will be created for you if it does not exist in the root of this project. - - -## API command - -To run the behavior API server execute `./bin/cli api --build-behaviors`. - -This will start the API server after all behaviors provided by this project have been built. - -If you have already built the behaviors using the `behaviors` command provided by the cli then you may omit the `--build-behaviors` flag. - - -Some configuration of the API server can be done via the environment variables listed below - -* `BEHAVIOR_API_HOST`: the host the api server will use (e.g. 127.0.0.1) -* `BEHAVIOR_API_PORT`: the port the api server will listen on (e.g. 3030) -* `WR_BEHAVIOR_DIR`: path to the directory containing the built behaviors -* `WR_BEHAVIOR_METADATA_PATH`: path to the behavior metadata file -* `BUILD_BEHAVIORS`: should the api server build the behaviors before starting - - - -# Docker -If you would like, you can use Docker to run Autopilot. - -To build the wr-behaviors docker image (`webrecorder/behaviors:latest`) execute `docker-compose build`. - -The image created is suitable for building behaviors and running the behavior api server. - -The default configuration of the image is to run the api server, however you can substitute the default command with any of the cli commands listed previously. - -For more information please consult the provided `Dockerfile` and `docker-compose.yml` files. - -## About Web Traffic - - -## Status page: pre-made behaviors - - diff --git a/chapter2.md b/chapter2.md deleted file mode 100644 index 6cc8871..0000000 --- a/chapter2.md +++ /dev/null @@ -1,299 +0,0 @@ -# Chapter 2: Creating your first behavior - -A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the [Pre-Made Behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/premade-behaviors.md). This will give you a basic idea of what kind of behaviors have already been created. - -## Setting up your file -Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. - -1. `mkdir webrecorder` (or skip steps 1-2 and use an already existing directory) -2. `cd webrecorder` -3. `git clone https://github.com/webrecorder/behaviors.git` -4. `cd behaviors` - -You should also download a package manager like yarn or npm, and make sure that you are using the correct version. -``` -$ yarn install -# or "npm install" -``` - -To check your version of npm, type the command `npm --version`. You can check the [npm website](https://www.npmjs.com/package/npm?activeTab=versions) to see what the latest version is. - -You should also check to make sure that your version of node is 12 or above. - -### Getting started using the CLI - -You should now create a Javascript file for your new behavior. You can do this using the CLI. - -The `newBehavior` command provides a simple way to create a new behavior by generating a new file in the behavior directory containing the required boiler plate. - -Executing `./bin/cli newBehavior awesomeBehavior` will create a new behavior file `awesomeBehavior.js` located in the behavior directory. - - -### Format - -Every behavior has: - -1. a **default export** that is an [async generator function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators) or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of#Iterating_over_async_generators). This is the function that will retrieve the metadata we want from the webpage. - -```js -export detault async function* myBehavior(cliAPI){...} -``` - -More details on how the default export should be written can be found under the [Default Export](#default-export) section. - -2. an export named ***metadata*** that is an object - - Details: the ***metadata*** object gives information about the behavior, such as when it was last updated, whether it's functional, its display name, etc. - - Example of expected format: -```js - export const metadata = { ... }; -``` - -More details on how the metadata object should be written can be found under the [Metadata](#metadata) section. - -3. an export named **isBehavior**. - - Details: **isBehavior** is a constant that you will flag as **true** when the file is complete and ready to be used. Otherwise, while the file is still in progress, keep it as **false**. - - Example of expected format: -```js -export const isBehavior = true; -``` - - -It's important to note that the tools will not recognize that the behavior is ready for use and valid if any of these three main components (the default export, **isBehavior**, and **metadata**) are missing. - -### Metadata -A behavior's exported metadata object is used to: - -- describe how the behavior should be matched to the pages it is written for -- provide an overview of what the behavior does -- provide a more specific name associated with it when querying for it using the behavior api -- embed any additional information about the behavior - -With those usages in mind, every metadata object is expected to have the following properties: - -- **name** (string): the name for your behavior to be used when querying the behavior API for it by name -- **description** (string): a description of the behavior -- **match** (object): how the behavior will be matched to the page(s) it is written for - -The **match** object has two variations, but you will probably only need to know the first. - -```js -// variation 1 -export const metadata = { - name: 'the name of your behavior', - match: { - regex: /a regular expression dictating the URL the behavior will run on/, - }, - description: 'an description of what your behavior does', -}; -``` -The first variation of `match`, shown above, defines a single property `regex` that is an JavaScript [RegExp](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp). The behavior using variation one is considered matched to an URL when the regular expression, defined in the `regex` property of match, matches the URL. - -The second variation of `match` is a bit more advanced and lets you compare your URL to multiple sub URLs. You can get more information on that in [chapter 3](./chapter3). - -### Default Export - -#### Asyncronous Generator Functions - -The purpose of an [asyncronous generator function](https://thecodebarbarian.com/async-generator-functions-in-javascript.html#:~:text=Async%20generator%20functions%20behave%20similarly,()%20function%20returns%20a%20promise.) is to collect data from a source that has too much data to return all at once (or would take too long to return all at once). Instead, the generator is a function that returns an object with a next() method on it. The programmer can keep calling next() until all of the data is received. An asynchronous generator function uses promises for the same purpose (promises will be expanded upon in the following section). - -The primary reasons that a behavior's **default export** is required to be an async generator function or a function returning an [async iterator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator) are: - -- async generators are widely supported in Javascript -- they provide a simple way to run the behavior in the browser or through [browsertrix](https://github.com/webrecorder/browsertrix) -- they allow information about the behavior and its state to be easily retrieved by code executing the behavior - -Consider the following example: - -```js -import * as lib from '../lib'; - -export default async function* myBehavior(cliAPI) { - // behavior code will be developed here -} - -export const metadata = { - name: 'myBehavior', - match: { - regex: /^(?:https?:\/\/(?:www\.)?)?:myAwesomeWebSite.com.*$/, - }, - description: 'It does really cool stuff', -}; - -export const isBehavior = true; -``` - -In this example, `myBehavior` is an asynchronous generator function that's being exported for our behavior and this behavior would work for any website with myAwesomeWebSite.com in its name (notice the regex for the matching above). Much of this boiler plate code shown above has been generated automatically by the cli command -`./bin/cli newBehavior [behaviorName]`. - -#### Promises - -Next, you may have notticed looking over some of the behavior documentation that many of the functions return [Promises](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise). - -Now, without turning into a JavaScript tutorial, the only thing you need to know about promises is that they can be awaited, that is to say you can await for their completion. - -This comes in handy when you want to wait for the the document's state to become ready since the pages of myAwesomeWebSite.com take a long time to load and can be done as easily as shown in `step 1`. This is the beginning of the code that will go in the default async function. - -```js -// step 1, wait for the page to complete loading -await lib.domCompletePromise(); -``` - -Now that we know the browser has fully loaded the page, we can safely start executing behaviors on the page. - -The next step is to initialize our state and report it, which is done by yielding a value as shown in `step 2`, where state contains the number of videos played. - -```js -// step 2, initialize state and return it -const state = { videosPlayed: 0 }; -yield lib.stateWithMsgNoWait('Document ready!', state); -``` - -The function `lib.stateWithMsgNoWait` indicates to the code executing the behavior that it has an update to report and that it does not have to wait. - -If the behavior was being run by [browsertrix](https://github.com/webrecorder/browsertrix) and the other function `lib.stateWithMsgWait` was used, browsertrix would have waited until the HTTP requests made by the page had significantly slowed down (no request made for set period of time) before executing further code in the function. However, since we use the *no wait* variant, the behavior will immediately return state. - -When you `yield` a value from the behavior you can consider the behavior paused until the runner initiates the `next()` action. - -Additionally, it should be noted that the second argument supplied to `lib.stateWithMsgNoWait` is optional but useful for reporting to yourself more detailed information about the state of your behavior. - -Continuing on with the creation of our behavior, let us assume that the page that the behavior is operating in has a list of videos we want to play. Our behavior will generate the next video to play each iteration. - -We can accomplish this as shown in `step 3`. - -```js -// step 3 -for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { - // videoListItem is a child of the element with id == 'videos' - // that has a video as child element - const videoWasPlayed = await lib.selectAndPlay('video', videoListItem); - if (videoWasPlayed) { - // increment our states counter for number of videos played - state.videosPlayed += 1; - // let ourselves know we played a video - yield lib.stateWithMsgNoWait('Played a video!', state); - } else { - yield lib.stateWithMsgNoWait('Failed to play a video using the standard DOM methods', state); - } -} -return lib.stateWithMsgNoWait('Done!', state); -``` - -In `step 3` we use the function `childElementIterator` that returns an iterator over the child elements of the supplied parent element. Then for each child of the element with `id="videos"` we: - -- select the video element that is a descendant of the `videoListItems` and play the video -- increment the number of videos played in our behavior's state -- let ourselves know that we played a video - -Also seen in `step 3` is the usage keyword `yield*`. - -`yield*` means that we are yielding another generator, that is to say all actions of the generator are to be treated as if we yielded them ourselves. - -In short, `step 3` can be described as playing a video contained in every child of the element with `id == video`, and once we have played all the videos on the page return a message with our final state from the behavior. - -The full behavior is shown below: - -```js -import * as lib from '../lib'; - -export default async function* myBehavior(cliAPI) { - // behavior code - // step 1 - await lib.domCompletePromise(); - // step 2 - const state = { videosPlayed: 0 }; - yield lib.stateWithMsgNoWait('Document ready!', state); - // step 3 - for (const videoListItem of lib.childElementIterator(lib.id('videos'))) { - // videoListItem is a child of the element with id == 'videos' - // that has a video as child element - const videoWasPlayed = await lib.selectAndPlay('video', videoListItem); - if (videoWasPlayed) { - // increment our states counter for number of videos played - state.videosPlayed += 1; - // let ourselves know we played a video - yield lib.stateWithMsgNoWait('Played a video!', state); - } else { - yield lib.stateWithMsgNoWait('Failed to play a video using the standard DOM methods', state); - } - } - return lib.stateWithMsgNoWait('Done!', state); -} - -export const metadata = { - name: 'myBehavior', - match: { - regex: /^(?:https?:\/\/(?:www\.)?)?:myAwesomeWebSite.com.*$/, - }, - description: 'It does really cool stuff', -}; - -export const isBehavior = true; -``` - -## Testing your first behavior - -Blocked by [PR 63](https://github.com/webrecorder/behaviors/pull/63) - -You should use the runner CLI to test your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. This will allow you to see if your behavior is working correctly. - -`$ ./bin/cli help runner` - -Please note that in order to provide automatic running of behaviors, this command must be able to launch the Chrome/Chromium browser. In other words, an already running instance of Chrome/Chromium can not be used. - -First, you will need to create a config file (yaml format) to use the runner. A run config file is provided for you and can found in the root of this project (`behavior-run-config.yml`). - -The file will look like this: -```yaml -behaviors: ./behaviors -lib: ./lib -build: ./build -dist: ./dist -tsconfig: ./tsconfig.json -metadata: ./dist/behaviorMetadata.js -``` - -By using the provided configuration file [`behavior-run-config.yml`](./behavior-run-config.yml) all that you have to do is change two fields: - -1. `behaviors`: the path to your new behavior in the behavior directory of this project -2. `url`: the url of the page your behavior should be run in - -Now you can start the build and run process by executing the following from the root directory of this project: - -`./bin/cli runner -c behavior-run-config.yml` - -The command will launch a Chrome/Chromium browser installed on your computer, build the behavior, and then run it until completion. - -If any changes are made to the behavior or any of the files it includes while the behavior is being run, it will be rebuilt and re-run automatically. - - - -## Fixing a broken behavior - -A behavior is broken when it doesn't work as expected or like it used to. Behaviors have to be fixed sometimes as websites can change at anytime. For example, a website might change its css and html names and classes, or how the Javascript behaves when you scroll. If a behavior depends on something that has changed, it may stop working. - -Here are the steps to fixing a broken behavior: - -1. Find the file of the behavior -2. Run the behavior -3. Note the output and check for console errors -4. Update the offending lines -5. [Create a fork](https://docs.github.com/en/github/getting-started-with-github/fork-a-repo) for your contributions -6. Commit and push your changes to your local branch -7. Create a pull request against the master branch - -You will then receive feedback on your work, which you can respond to in the pull request. - - - - - diff --git a/chapter3.md b/chapter3.md deleted file mode 100644 index 366e6fd..0000000 --- a/chapter3.md +++ /dev/null @@ -1,56 +0,0 @@ -# Hacking Autopilot - -## Contributing to Autopilot - -We welcome contributions from anyone with beginning to advanced programming knowledge. - -Please be advised that we have a [code of conduct](./code-of-conduct.md) that we expect all contributors to follow. - -### Reporting bugs - - -## Tools of the Trade - -### Advanced options - -#### postStep -The export `postStep` can be called after each action to convert the yielded results into the expected format. - -It is recommended that you use the library function `lib.buildCustomPostStepFn`if you want to perform some kind of action after each behavior step that is not directly tied to the running of the behavior. - -```js -export const postStep = lib.buildCustomPostStepFn(() => { ... }); -``` - -#### Metadata export: match -If you want to compare your URL to multiple sub URLs, you can use a different version of `match`. This version is shown below, and has the two properties `base` (RegExp) and `sub` (Array). - -The `base` regular expression is used as a generic test. If `base` matches a URL, the regular expressions in the `sub` array will be tested against the same URL. The behavior is considered matched to a URL when the `base` regular expression matches the URL and one of the `sub` regular expressions also matches the URL. - -```js -// variation 2 -export const metadata = { - name: 'the name of your behavior', - match: { - regex: { - base: /a regular expression dictating the base URL the behavior will run on/, - sub: [ - /an array of regular expressions dictating more specific parts of the base URL the behavior will run on/, - ], - }, - }, - description: 'a description of what your behavior does', -}; -``` -## Testing a behavior (Debugging) -Blocked by PR 63 -## Behaviors -In pre made behaviors file -## Build System -## CLI -write docs on npm or yarn scripts -revamp the cli as scripts package.json scripts - -## Provided CLI Commands -## Behavior Standard Library Reference - \ No newline at end of file diff --git a/chapter4.md b/chapter4.md deleted file mode 100644 index 235d609..0000000 --- a/chapter4.md +++ /dev/null @@ -1,33 +0,0 @@ -# Behind Autopilot - -## Maintainers - -Autopilot is maintained by: - -## Contact Support - -Email [insert email] for support questions. - -## Discussion Forum - -Use the [discussion forum](https://forum.webrecorder.net) to discuss ideas, questions, issues, etc. with the webrecorder community. - -## Who Uses Autopilot? -We are happy to have a growing community of academic and memory institutions that are using Webrecorder tools and software. - -Some of our colleagues that are using various Webrecorder tools: - -![DocNow](https://webrecorder.net/assets/logos/docnow-logo.png) -![ArQuivo.PT](https://webrecorder.net/assets/logos/arquivo-logo.png) -![UKWA](https://webrecorder.net/assets/logos/ukwa-logo.svg) -![LOCKSS](https://webrecorder.net/assets/logos/lockss-logo.png) -![Rhizome](https://webrecorder.net/assets/logos/rhizome-logo.png) -![ONF NFB](https://webrecorder.net/assets/logos/nfb-logo.svg) -![IIPC](https://webrecorder.net/assets/logos/iipc-logo.png) -![Stanford University Press](https://webrecorder.net/assets/logos/sup-logo.png) -![The National Archives](https://webrecorder.net/assets/logos/tna-logo.svg) -![National Library of Australia](https://webrecorder.net/assets/logos/nla-logo.png) -![Perma.cc](https://webrecorder.net/assets/logos/perma-logo.png) -![CLOCKSS](https://webrecorder.net/assets/logos/clockss-logo.png) - -*If your institution is using Webrecorder software and would like to be listed, please let us know!* \ No newline at end of file From 681d0c69c17cd466e9dac027b1e6f17fe2370c72 Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Thu, 12 Nov 2020 11:54:02 -1000 Subject: [PATCH 32/33] Moved the chapters to the correct locations --- manual/chapter1.md | 49 +++++++++++++++----- manual/chapter2.md | 16 +++++-- manual/chapter3.md | 112 ++++++++++++++++++++++++++++++++++++++++++++- manual/chapter4.md | 44 +++++++++--------- 4 files changed, 181 insertions(+), 40 deletions(-) diff --git a/manual/chapter1.md b/manual/chapter1.md index 031d55c..a0ad783 100644 --- a/manual/chapter1.md +++ b/manual/chapter1.md @@ -11,6 +11,9 @@ Web archives may be utilized by future researchers, historians and the general p Autopilot is a tool that navigates a website similarly to how a human would, doing things like scrolling, clicking buttons, and playing videos. These actions executed through functions called "behaviors." Because Autopilot can go through the technically complicated aspects of a website, it is useful for recording high-fidelity websites. Furthermore, Autopilot is designed to be accessible for anyone to use, and anyone with little javascript knowledge to contribute to. This makes webarchiving available to everyone. +# Autopilot basics +Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. + ## Installing Autopilot To use this project you must first install its dependencies. You can do this via a package manager like [Yarn](https://classic.yarnpkg.com/en/docs/install/#mac-stable) or [npm](https://www.npmjs.com/). @@ -21,9 +24,15 @@ $ yarn install ``` To install Autopilot, use Terminal to clone the Webrecorder repository: + +**You must be using node 12~** + ``` -1. git clone https://github.com/webrecorder/behaviors.git -2. cd autopilot; bash init-default.sh. +1. `git clone https://github.com/webrecorder/behaviors.git` +2. `cd behaviors` +3. `yarn build-watch` +4. `open new terminal window` +5. `bash scripts/init.sh` ``` If you wish to use this project via Docker, install [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/). @@ -34,20 +43,37 @@ If you wish to use this project via Docker, install [Docker](https://docs.docker The Autopilot instance can be accessed in the browser at `http://localhost:8089`. +# Running Behaviors +1. In order to run the behaviors you will first need to build them: +``` +npm run generate-runnable-behaviors + +``` -# Autopilot basics -Autopilot uses [behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/behaviors.md) to collect metadata from websites. Behaviors are Javascript modules which perform a series of actions on a webpage in order to collect information. +2. Then, go into the dist directory. +``` +cd dist +``` + +3. In the dist directory you will see behavior files. Open the behavior file you want and copy the code. +``` +open [filename] +``` + +4. Copy all of the behavior code and paste it into the developer console of a webpage. The console can be accessed in Chrome using the shortcut: Option + ⌘ + J (on macOS), or Shift + CTRL + J (on Windows/Linux). +You can also run behaviors using the CLI: -The cli provides two commands API and behaviors and each command has its own options. +## CLI and API + +The cli provides two commands `API` and `behaviors` and each command has its own options. ## Behaviors command -Execute `./bin/cli behaviors -b` to build the behaviors made available, using the config file located at the root of the project. +Execute `./bin/cli behaviors -b` to build the behaviors made available. This will build the behaviors using the behavior config file located in the root of this project. The built behaviors, along with a behavior metadata file (`behaviorMetadata.js`), can be found in the `dist` directory which will be created for you if it does not exist in the root of this project. - ## API command To run the behavior API server execute `./bin/cli api --build-behaviors`. @@ -78,9 +104,10 @@ The default configuration of the image is to run the api server, however you can For more information please consult the provided `Dockerfile` and `docker-compose.yml` files. -## About Web Traffic - - -## Status page: pre-made behaviors + + diff --git a/manual/chapter2.md b/manual/chapter2.md index 6cc8871..8a98adf 100644 --- a/manual/chapter2.md +++ b/manual/chapter2.md @@ -1,6 +1,6 @@ # Chapter 2: Creating your first behavior -A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the [Pre-Made Behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/premade-behaviors.md). This will give you a basic idea of what kind of behaviors have already been created. +A behavior is a [Javascript Module](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules) which executes a series of actions that collect some information (metadata) about a webpage. Before you create your first behavior, you should first take a look at the [Pre-Made Behaviors](https://github.com/webrecorder/behaviors/blob/master/manual/premade-behaviors.md). This will give you a basic idea of what kind of behaviors have already been created. ## Setting up your file Once you have an idea of what you want your behavior to do, clone the webrecorder repository so that you can work locally on your computer. @@ -238,11 +238,13 @@ export const metadata = { export const isBehavior = true; ``` -## Testing your first behavior +## Running your behavior +The easiest way to run your behavior is to copy all of the behavior code and paste it into the developer console of a webpage. -Blocked by [PR 63](https://github.com/webrecorder/behaviors/pull/63) +The console can be accessed in Chrome using the shortcut: Option + ⌘ + J (on macOS), or Shift + CTRL + J (on Windows/Linux). -You should use the runner CLI to test your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. This will allow you to see if your behavior is working correctly. +###Runner CLI +You can also use the runner CLI to run your behavior. The runner command allows you to automatically run a behavior on a specified URL using a Chrome/Chromium browser installed on your machine. `$ ./bin/cli help runner` @@ -273,6 +275,10 @@ The command will launch a Chrome/Chromium browser installed on your computer, bu If any changes are made to the behavior or any of the files it includes while the behavior is being run, it will be rebuilt and re-run automatically. +## Testing your first behavior + +Blocked by [PR 63](https://github.com/webrecorder/behaviors/pull/63) + ## Fixing a broken behavior @@ -295,5 +301,5 @@ You will then receive feedback on your work, which you can respond to in the pul + Blocked by website --> diff --git a/manual/chapter3.md b/manual/chapter3.md index 93171f8..bd0ccd0 100644 --- a/manual/chapter3.md +++ b/manual/chapter3.md @@ -7,9 +7,14 @@ We welcome contributions from anyone with beginning to advanced programming know Please be advised that we have a [code of conduct](./code-of-conduct.md) that we expect all contributors to follow. ### Reporting bugs +Bugs are tracked as [GitHub issues](https://guides.github.com/features/issues/). +Use these steps to report a bug: +1. Determine which repository under which to report the problem +2. Perform a [search](https://github.com/search?q=is%3Aissue+user%3Awebrecorder) to check if the issue has already been reported +3. Use the [bug report template](./bug-report.md) to report the bug. -## Tools of the Trade + ### Advanced options @@ -46,7 +51,110 @@ export const metadata = { Blocked by PR 63 (testing infrastructure) ## Build System -## CLI + +### Overview + +The behavior build process has three phases +- [Initialization](#initialization) +- [Collection](#collection) +- [Building](#building) + + +### Initialization + +Initialization has three steps +1. Ultimate build configuration +2. Building what resolution +3. Ensuring the necessary build directory structure exists + + +#### Ultimate build configuration + +The ultimate build configuration is created in combination with user supplied cli options using the following steps: + + 1. If a config was specified using `-c, --config`, the specified config will be used + 2. If the default config exists in the current working directory of the commands, the default config is used + 3. Otherwise the path to the behavior file or dir (`-b, --build [fileOrDir]`) is used and the six config values are set to the project defaults + +When a config file is used each of the six values from the config file are resolved as follows +- if the key exists and value is relative, make absolute by resolving it against the directory containing the config file +- if the key exists and is absolute, use key value +- if key does not exist use project default value + - build, dist: `/` + - lib, tsconfig: projects default value + - metadata: placed in current working directory + + +Once the ultimate build configuration has been created, the build process proceeds to the next step. + +#### Building what resolution + +The determination for what is being built is done using the value for the `-b, --build` cli option and that value can be one of two types: +- `boolean`: build all behaviors found in the value for the `behaviors` key from the supplied build config +- `string`: path to a directory containing behaviors or a single behavior to be built + +When the value for build is `boolean`: +- If the directory supplied via the `behaviors` config key exists, then what is being built is that directory and the initialization process contains to the next step +> [name=kyragaut] ^^ line 43 this doesn't make sense +- Otherwise, if the directory supplied via the `behaviors` config key does not exist the build process is ended + +When the value for build is a `string` and an absolute path: +- If the path exists, then what is being built is that directory or file and the initialization process contains to the next step +> [name=kyragaut] ^^ line 48 this doesn't make sense +- Otherwise, if the path does not exist the build process is ended + +When the value for build is a `string` and a relative path, it is resolved in the following order: +1. the value as is resolved using node's relative path resolution algorithm. Note this value is used by other steps if previous ones fail and is denoted as `resolvedPath` +2. the value as is joined with the supplied configs behavior dir or projects default behavior dir +3. the value as is joined with the the current working directory +4. `resolvedPath` is joined with the supplied configs behavior dir or projects default behavior dir +5. `resolvedPath` is joined with the current working directory + +If any of the absolute paths described above exist, then what is being built is the resolved path and the initialization process contains to the next step otherwise the build process is ended +> [name=kyragaut] ^^ line 59 this doesn't make sense + + +#### Ensuring the necessary build directory structure exists + +The final step in the initialization process is to ensure that the `build` and `dist` directories exist. + +These values may differ from the names used previously only when they are supplied by the user in a build config file. + +The `build` directory is used to hold intermediate files used by the build system in order to setup the behavior for final building and usage by other tools such as our own running system. + +Any setup in order to facilitate running the behavior is done here. + +The `dist` directory is where the built, bundled, behaviors are placed alongside their metadata if configured to do so. + + +### Collection + +The collection phase operates in one of two modes: +- `single behavior`: when the `what is being built` path resolves to file +- `multi-behavior`: when the `what is being built` path resolves to a directory + +The primary difference between modes is that `multi-behavior` mode considers every file contained in the directory and its descendant directories. + +Both modes use the same means in determining if a file is indeed a behavior which is as follows: +- the file is an es module +- has a `metadata` or `metaData` named export +- has an `isBehavior` named export + +Once the behavior(s) have been collected a report is printed stating how many behaviors were found and if any of the files considered partially met the requirements for collection. + +**Note**: Both the collection and building phases share modes with the mode operating under set by the collection phase. + +### Building + +The building phase can be described in the following steps: +1. Extract behaviors metadata +2. Create the behavior's intermediate file in the configured `build` directory +3. Use build behavior using rollup, built behavior placed in configured `dist` directory +4. Once all behaviors have been built generate behavior metadata. + + +The previous steps are applied to all behaviors returned by the collection phase +##CLI write docs on npm or yarn scripts revamp the cli as scripts package.json scripts diff --git a/manual/chapter4.md b/manual/chapter4.md index 235d609..cb59d56 100644 --- a/manual/chapter4.md +++ b/manual/chapter4.md @@ -1,33 +1,33 @@ # Behind Autopilot -## Maintainers + ## Maintainers -Autopilot is maintained by: + Autopilot is maintained by: -## Contact Support + ## Contact Support -Email [insert email] for support questions. + Email [insert email] for support questions. -## Discussion Forum + ## Discussion Forum -Use the [discussion forum](https://forum.webrecorder.net) to discuss ideas, questions, issues, etc. with the webrecorder community. + Use the [discussion forum](https://forum.webrecorder.net) to discuss ideas, questions, issues, etc. with the webrecorder community. -## Who Uses Autopilot? -We are happy to have a growing community of academic and memory institutions that are using Webrecorder tools and software. + ## Who Uses Autopilot? + We are happy to have a growing community of academic and memory institutions that are using Webrecorder tools and software. -Some of our colleagues that are using various Webrecorder tools: + Some of our colleagues that are using various Webrecorder tools: -![DocNow](https://webrecorder.net/assets/logos/docnow-logo.png) -![ArQuivo.PT](https://webrecorder.net/assets/logos/arquivo-logo.png) -![UKWA](https://webrecorder.net/assets/logos/ukwa-logo.svg) -![LOCKSS](https://webrecorder.net/assets/logos/lockss-logo.png) -![Rhizome](https://webrecorder.net/assets/logos/rhizome-logo.png) -![ONF NFB](https://webrecorder.net/assets/logos/nfb-logo.svg) -![IIPC](https://webrecorder.net/assets/logos/iipc-logo.png) -![Stanford University Press](https://webrecorder.net/assets/logos/sup-logo.png) -![The National Archives](https://webrecorder.net/assets/logos/tna-logo.svg) -![National Library of Australia](https://webrecorder.net/assets/logos/nla-logo.png) -![Perma.cc](https://webrecorder.net/assets/logos/perma-logo.png) -![CLOCKSS](https://webrecorder.net/assets/logos/clockss-logo.png) + ![DocNow](https://webrecorder.net/assets/logos/docnow-logo.png) + ![ArQuivo.PT](https://webrecorder.net/assets/logos/arquivo-logo.png) + ![UKWA](https://webrecorder.net/assets/logos/ukwa-logo.svg) + ![LOCKSS](https://webrecorder.net/assets/logos/lockss-logo.png) + ![Rhizome](https://webrecorder.net/assets/logos/rhizome-logo.png) + ![ONF NFB](https://webrecorder.net/assets/logos/nfb-logo.svg) + ![IIPC](https://webrecorder.net/assets/logos/iipc-logo.png) + ![Stanford University Press](https://webrecorder.net/assets/logos/sup-logo.png) + ![The National Archives](https://webrecorder.net/assets/logos/tna-logo.svg) + ![National Library of Australia](https://webrecorder.net/assets/logos/nla-logo.png) + ![Perma.cc](https://webrecorder.net/assets/logos/perma-logo.png) + ![CLOCKSS](https://webrecorder.net/assets/logos/clockss-logo.png) -*If your institution is using Webrecorder software and would like to be listed, please let us know!* \ No newline at end of file + *If your institution is using Webrecorder software and would like to be listed, please let us know!* \ No newline at end of file From 0ed0640cb0c503e82bea742a271db2645b7960cb Mon Sep 17 00:00:00 2001 From: Kyra GauthierDickey Date: Thu, 12 Nov 2020 11:54:51 -1000 Subject: [PATCH 33/33] added bug report format file --- manual/bug-report.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 manual/bug-report.md diff --git a/manual/bug-report.md b/manual/bug-report.md new file mode 100644 index 0000000..44545b7 --- /dev/null +++ b/manual/bug-report.md @@ -0,0 +1,15 @@ +#Bug Report Template +Fill out this template when reporting bugs. + +##Description + +##Stepts to reproduce +1. +2. +3. + +##Current behavior (bug) + +##Expected behavior (correct) + +##Additional information \ No newline at end of file