Feature/preprocess (#50)

* add source preprocessing script

* refactor quickstarts folder name for extraction

* update path to extract to

* add shortcode for code sections

* update netlify build

* add code section usage to readme

* refactor quickstart references

Author: dliuproduction

.gitignore

@@ -4,3 +4,6 @@ public/
 .idea/*
 .idea
 .DS_Store
+
+# Local Netlify folder
+.netlify

README.md

@@ -29,15 +29,45 @@ All content can be found in the `/content` directory. The structure of the websi
 
 ### Links
 
-  - [1.1](#types--primitives) **external**: When you insert an external link use the `external-link` shortcode
+- [1.1](#types--primitives) **external**: When you insert an external link use the `external-link` shortcode
 
-    ```markdown
-    {{< external-link href="https://app.dfuse.io" title="dfuseio" >}}
-    {{< external-link href="https://app.dfuse.io">}}
-    ```
+  ```markdown
+  {{< external-link href="https://app.dfuse.io" title="dfuseio" >}}
+  {{< external-link href="https://app.dfuse.io">}}
+  ```
 
 - [1.2](#types--primitives) **internal references**: Please put all _internal references_ as _full paths_, for greppability and refactoring.
 
+### Example code
+
+Use the following begin and end tags to indicate code sections that need to be referenced:
+
+```javascript
+// CODE:BEGIN:quickstarts_javascript_node_eos_section1
+function() {
+  var some = "code"
+}
+// CODE:END:quickstarts_javascript_node_eos_section1
+```
+
+Section names need to follow the path of the file from project root.
+For example the example snippet is from the file:
+
+```markdown
+├── quickstarts
+│   └── javascript
+│       └── node
+│           └── index.eos.js
+├── guides
+└── ...
+```
+
+When hugo builds the site, the code sections are extracted and stored in the `data` folder in project root. 
+Example code can then be referenced with the following shortcode:
+```go
+{{< code-section "quickstarts_javascript_node_eos_section1" >}}
+```
+
 ## Lexicon guide
 
 - web application
@@ -51,17 +81,19 @@ All content can be found in the `/content` directory. The structure of the websi
 
 Any commit to master automatically triggers a deployment. For this reason, the master branch is locked, requiring all changes to go through an approved pull request.
 
-
 Reference
 ETH -=> Endpoint ALPHA
 
 <new> side menu
 
-
 alpha (red)
 beta (yeallo-orangy)
 stage (light gray)
 add sub-title to graphql shortcode
 fix
 
 refactor pages
+
+```
+
+```

content/guides/eosio/getting-started/javascript-quickstart.md

@@ -46,10 +46,10 @@ the {{< external-link title="dfuse JS client library" href="https://github.com/d
 
 Here are a few of its key features:
 
-* Handles API token issuance
-* Refreshes your API token upon expiration
-* Automatically reconnects if the connection closes
-* Supports `Browsers` and `Node.js` environments
+- Handles API token issuance
+- Refreshes your API token upon expiration
+- Automatically reconnects if the connection closes
+- Supports `Browsers` and `Node.js` environments
 
 You can add it to your project using Yarn or NPM.
 
@@ -104,11 +104,16 @@ the dfuse client using the API key you created in the first step and the network
 connect to.
 
 Valid networks can be found at [EOSIO API Endpoints]({{< ref "reference/eosio/endpoints" >}})
-
 {{< tabs "create-client" >}}
-{{< tab-code title="Node.js" filename="./quickstarts/javascript/node.js/index.eosio.js" range="4:9" >}}
-{{< tab-code title="Bundler" filename="./quickstarts/javascript/bundler/index.eosio.js" range="1:6" >}}
-{{< tab-code title="Browser" filename="./quickstarts/javascript/browser/index.eosio.html" range="1:11" >}}
+{{< tab lang="javascript" title="Node.js" >}}
+{{< code-section "quickstarts_javascript_node_eos_section1" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Bundler" >}}
+{{< code-section "quickstarts_javascript_bundler_eos_section1" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Browser" >}}
+{{< code-section "quickstarts_javascript_browser_eos_section1" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 ## 4. Stream your first results
@@ -120,14 +125,20 @@ to you, you get to choose and pick only what you are interested in.
 {{< alert type="note" >}}
 Want to inspect the full set of available fields you can retrieve?
 
-* [GraphQL API Reference]({{< ref "/reference/eosio/graphql" >}})
-* {{< external-link href="https://mainnet.eos.dfuse.io/graphiql/?query=c3Vic2NyaXB0aW9uIHsKICBzZWFyY2hUcmFuc2FjdGlvbnNGb3J3YXJkKHF1ZXJ5OiJyZWNlaXZlcjplb3Npby50b2tlbiBhY3Rpb246dHJhbnNmZXIgLWRhdGEucXVhbnRpdHk6JzAuMDAwMSBFT1MnIikgewogICAgdW5kbyBjdXJzb3IKICAgIHRyYWNlIHsgaWQgbWF0Y2hpbmdBY3Rpb25zIHsganNvbiB9IH0KICB9Cn0=" title="GraphiQL, online query editor with completion and docs">}}
-{{< /alert >}}
+- [GraphQL API Reference]({{< ref "/reference/eosio/graphql" >}})
+- {{< external-link href="https://mainnet.eos.dfuse.io/graphiql/?query=c3Vic2NyaXB0aW9uIHsKICBzZWFyY2hUcmFuc2FjdGlvbnNGb3J3YXJkKHF1ZXJ5OiJyZWNlaXZlcjplb3Npby50b2tlbiBhY3Rpb246dHJhbnNmZXIgLWRhdGEucXVhbnRpdHk6JzAuMDAwMSBFT1MnIikgewogICAgdW5kbyBjdXJzb3IKICAgIHRyYWNlIHsgaWQgbWF0Y2hpbmdBY3Rpb25zIHsganNvbiB9IH0KICB9Cn0=" title="GraphiQL, online query editor with completion and docs">}}
+  {{< /alert >}}
 
 {{< tabs "define-query">}}
-{{< tab-code title="Node.js" filename="./quickstarts/javascript/node.js/index.eosio.js" range="11:17" >}}
-{{< tab-code title="Bundler" filename="./quickstarts/javascript/bundler/index.eosio.js" range="8:14" >}}
-{{< tab-code title="Browser" filename="./quickstarts/javascript/browser/index.eosio.html" range="13:21" >}}
+{{< tab lang="javascript" title="Node.js" >}}
+{{< code-section "quickstarts_javascript_node_eos_section2" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Bundler" >}}
+{{< code-section "quickstarts_javascript_bundler_eos_section2" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Browser" >}}
+{{< code-section "quickstarts_javascript_browser_eos_section2" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 Next, you create the GraphQL subscription to stream transfers as they come. You will use the `searchTransactionsForward` operation, with the `"receiver:eosio.token action:transfer -data.quantity:'0.0001 EOS'"` query (See the [Search Query Language reference here]({{< ref "/reference/eosio/search-terms" >}})). This basically means, give me all transactions containing one or more
@@ -137,16 +148,23 @@ You can combine the dfuse client instance we created in step 3 with the GraphQL
 a `main` function:
 
 {{< tabs "execute-query">}}
-{{< tab-code title="Node.js" filename="./quickstarts/javascript/node.js/index.eosio.js" range="19:43" >}}
-{{< tab-code title="Bundler" filename="./quickstarts/javascript/bundler/index.eosio.js" range="16:52" >}}
-{{< tab-code title="Browser" filename="./quickstarts/javascript/browser/index.eosio.html" range="23:60" >}}
+{{< tab lang="javascript" title="Node.js" >}}
+{{< code-section "quickstarts_javascript_node_eos_section3" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Bundler" >}}
+{{< code-section "quickstarts_javascript_bundler_eos_section3" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Browser" >}}
+{{< code-section "quickstarts_javascript_browser_eos_section3" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 The function passed as the 2nd parameter to `client.graphql()` will be called every time a new result is returned
 by the API. And here is a sample of the prints you will receive as a result of executing the streaming operation
 above:
 
 <!-- **Note** We use python for all languages for a nicer output rendering -->
+
 {{< highlight "python" >}}
 Transfer eosbetdice11 -> eosbetbank11 [0.0500 EOS]
 Transfer newdexpublic -> gq4tcnrwhege [2.8604 EOS]
@@ -161,19 +179,26 @@ Transfer bluebetproxy -> bluebetbulls [0.6000 EOS]
 Here the small glue code containing the `main` function, imports and other helper functions to run the example:
 
 {{< tabs "support-code">}}
-{{< tab-code title="Node.js" filename="./quickstarts/javascript/node.js/index.eosio.js" range="45:45" >}}
-{{< tab-code title="Bundler" filename="./quickstarts/javascript/bundler/index.eosio.js" range="54:54" >}}
-{{< tab-code title="Browser" filename="./quickstarts/javascript/browser/index.eosio.html" range="61:63" >}}
+{{< tab lang="javascript" title="Node.js" >}}
+{{< code-section "quickstarts_javascript_node_eos_section4" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Bundler" >}}
+{{< code-section "quickstarts_javascript_bundler_eos_section4" >}}
+{{< /tab >}}
+{{< tab lang="javascript" title="Browser" >}}
+{{< code-section "quickstarts_javascript_browser_eos_section4" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 {{< tabs "full-working">}}
 
 {{< tab lang="shell" title="Node.js">}}
 git clone https://github.com/dfuse-io/docs
-cd docs/quickstarts/javascript/node.js
+cd docs/quickstarts/javascript/node
 npm install
 
 # Replace 'server_abcdef12345678900000000000' with your own API key!
+
 DFUSE_API_KEY=server_abcdef12345678900000000000 node index.eosio.js
 {{< /tab >}}
 
@@ -183,34 +208,38 @@ cd docs/quickstarts/javascript/bundler
 npm install
 
 # Replace 'web_abcdef12345678900000000000' with your own API key!
+
 DFUSE_API_KEY=web_abcdef12345678900000000000 npm run build:eosio
 
 # Open `index.eosio.html` directly in your favorite Browser
-open index.eosio.html       # Mac
-xdg-open index.eosio.html   # Ubuntu
-start index.eosio.thml      # Windows
+
+open index.eosio.html # Mac
+xdg-open index.eosio.html # Ubuntu
+start index.eosio.thml # Windows
 {{< /tab >}}
 
 {{< tab lang="shell" title="Browser">}}
 git clone https://github.com/dfuse-io/docs
 cd docs/quickstarts/javascript/browser
+
 # Manually edit index.eosio.html changing `web_abcdef12345678900000000000` with your own API key
 
 # Open `index.eosio.html` directly in your favorite Browser
-open index.eosio.html       # Mac
-xdg-open index.eosio.html   # Ubuntu
-start index.eosio.thml      # Windows
+
+open index.eosio.html # Mac
+xdg-open index.eosio.html # Ubuntu
+start index.eosio.thml # Windows
 {{< /tab >}}
 
 {{< /tabs >}}
 
 ## 6. What's next?
 
-* [GraphQL API Reference]({{< ref "/reference/eosio/graphql" >}})
-* [REST API Reference]({{< ref "/reference/eosio/rest" >}})
-* [WebSocket API Reference]({{< ref "/reference/eosio/websocket" >}})
-* [Check dfuse Core Concepts]({{< ref "/guides/core-concepts" >}})
-* [Look at one of our tutorials]({{< ref "/guides/eosio/tutorials" >}})
-* {{< external-link title="The `@dfuse/client-js` overview document" href="https://github.com/dfuse-io/client-js/blob/master/README.md#dfuse-javascripttypescript-client-library" >}}
-* {{< external-link title="The `@dfuse/client-js` quick API reference" href="https://github.com/dfuse-io/client-js/blob/master/README.md#api" >}} ({{< external-link title="Full API reference" href="https://dfuse-io.github.io/client-js/" >}})
-* {{< external-link title="GraphiQL, online query editor with completion and docs" href="https://mainnet.eos.dfuse.io/graphiql/?query=c3Vic2NyaXB0aW9uIHsKICBzZWFyY2hUcmFuc2FjdGlvbnNGb3J3YXJkKHF1ZXJ5OiJyZWNlaXZlcjplb3Npby50b2tlbiBhY3Rpb246dHJhbnNmZXIgLWRhdGEucXVhbnRpdHk6JzAuMDAwMSBFT1MnIikgewogICAgdW5kbyBjdXJzb3IKICAgIHRyYWNlIHsgaWQgbWF0Y2hpbmdBY3Rpb25zIHsganNvbiB9IH0KICB9Cn0=" >}}
+- [GraphQL API Reference]({{< ref "/reference/eosio/graphql" >}})
+- [REST API Reference]({{< ref "/reference/eosio/rest" >}})
+- [WebSocket API Reference]({{< ref "/reference/eosio/websocket" >}})
+- [Check dfuse Core Concepts]({{< ref "/guides/core-concepts" >}})
+- [Look at one of our tutorials]({{< ref "/guides/eosio/tutorials" >}})
+- {{< external-link title="The `@dfuse/client-js` overview document" href="https://github.com/dfuse-io/client-js/blob/master/README.md#dfuse-javascripttypescript-client-library" >}}
+- {{< external-link title="The `@dfuse/client-js` quick API reference" href="https://github.com/dfuse-io/client-js/blob/master/README.md#api" >}} ({{< external-link title="Full API reference" href="https://dfuse-io.github.io/client-js/" >}})
+- {{< external-link title="GraphiQL, online query editor with completion and docs" href="https://mainnet.eos.dfuse.io/graphiql/?query=c3Vic2NyaXB0aW9uIHsKICBzZWFyY2hUcmFuc2FjdGlvbnNGb3J3YXJkKHF1ZXJ5OiJyZWNlaXZlcjplb3Npby50b2tlbiBhY3Rpb246dHJhbnNmZXIgLWRhdGEucXVhbnRpdHk6JzAuMDAwMSBFT1MnIikgewogICAgdW5kbyBjdXJzb3IKICAgIHRyYWNlIHsgaWQgbWF0Y2hpbmdBY3Rpb25zIHsganNvbiB9IH0KICB9Cn0=" >}}

content/guides/eosio/getting-started/other-languages.md

@@ -18,8 +18,12 @@ The code from the examples on this page can be found {{< external-link href="htt
 The JWT is a token with a short expiration period, used to communicate with dfuse services. You will have to implement token caching and manage renewal upon expiration. See [Authentication]({{< relref "/guides/core-concepts/authentication" >}}) for more details.
 
 {{< tabs "generate-jwt">}}
-{{< tab-code title="Go" filename="./quickstarts/go/main-eos.go" range="21:39" >}}
-{{< tab-code title="Python" filename="./quickstarts/python/main.py" range="17:28" >}}
+{{< tab title="Go" lang="go">}}
+{{< code-section "quickstarts_go_eos_section2" >}}
+{{< /tab >}}
+{{< tab title="Python" lang="python">}}
+{{< code-section "quickstarts_python_eos_section2" >}}
+{{< /tab >}}
 {{< tab title="Shell" lang="shell" >}}
 curl https://auth.dfuse.io/v1/auth/issue -s \
   --data-binary '{"api_key":"server_abcdef12345678900000000000"}'
@@ -79,8 +83,12 @@ requests and streams you need to do, it should be properly cached at the appropr
 level for your use case.
 
 {{< tabs "create-client" >}}
-{{< tab-code title="Go" filename="./quickstarts/go/main-eos.go" range="41:59" >}}
-{{< tab-code title="Python" filename="./quickstarts/python/main.py" range="30:41" >}}
+{{< tab title="Go" lang="go">}}
+{{< code-section "quickstarts_go_eos_section3" >}}
+{{< /tab >}}
+{{< tab title="Python" lang="python">}}
+{{< code-section "quickstarts_python_eos_section3" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 ## 5. Stream your first results
@@ -97,17 +105,25 @@ Want to inspect the full set of available fields you can retrieve?
 {{< /alert >}}
 
 {{< tabs "define-query">}}
-{{< tab-code title="Go" filename="./quickstarts/go/main-eos.go" range="126:144" >}}
-{{< tab-code title="Python" filename="./quickstarts/python/main.py" range="73:78" >}}
+{{< tab title="Go" lang="go">}}
+{{< code-section "quickstarts_go_eos_section4" >}}
+{{< /tab >}}
+{{< tab title="Python" lang="python">}}
+{{< code-section "quickstarts_python_eos_section4" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 And we can finally define the code needed to perform your first stream using
 dfuse Search. This snippet initiates the connection with dfuse servers and starts
 streaming transfers forever.
 
 {{< tabs "run-query">}}
-{{< tab-code title="Go" filename="./quickstarts/go/main-eos.go" range="146:180" >}}
-{{< tab-code title="Python" filename="./quickstarts/python/main.py" range="80:94" >}}
+{{< tab title="Go" lang="go">}}
+{{< code-section "quickstarts_go_eos_section5" >}}
+{{< /tab >}}
+{{< tab title="Python" lang="python">}}
+{{< code-section "quickstarts_python_eos_section5" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 And here is a sample of the prints you will receive from the standard output after running the example above:
@@ -127,8 +143,14 @@ Transfer bluebetproxy -> bluebetbulls [0.6000 EOS]
 Here is the small glue code containing the `main` function, imports and other helper functions to run the example:
 
 {{< tabs "support-code">}}
-{{< tab-code title="Go" filename="./quickstarts/go/main-eos.go" range="1:19,182:201" >}}
-{{< tab-code title="Python" filename="./quickstarts/python/main.py" range="1:15,96:104" >}}
+{{< tab title="Go" lang="go">}}
+{{< code-section "quickstarts_go_eos_section1" >}}
+{{< code-section "quickstarts_go_eos_section6" >}}
+{{< /tab >}}
+{{< tab title="Python" lang="python">}}
+{{< code-section "quickstarts_python_eos_section1" >}}
+{{< code-section "quickstarts_python_eos_section6" >}}
+{{< /tab >}}
 {{< /tabs >}}
 
 If you prefer, you can directly clone our ready-made repository with all the quick start