Information from Day 3

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test of a loop over an array\n",
    "\n",
    "In this notebook we will explore\n",
    "\n",
    "1. writing loops in Bash  \n",
    "    1.1. looping oveer an array  \n",
    "    1.2. looping over a file  \n",
    "2. using loops to collect data from an API  \n",
    "    2.1 Testing the API query  \n",
    "    Z.Z. Adding the query to the loop  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Writing loops in Bash\n",
    "### 1.1. Reading from an array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step is to assign values to an array. We anticipate the structure of the Nominatim API query, by writing each value as \"City,Country\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen,Norway Paris,France Turin,Italy Bordeaux,France Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "# assigning the values of the array to the variable cities_array\n",
    "cities_array=(Bergen,Norway Paris,France Turin,Italy Bordeaux,France Accra,Ghana)\n",
    "# check the array has been assigned properly\n",
    "echo \"${cities_array[@]}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now now test our loop who will use the array as an input, and will simply print the output to the notebook.\n",
    "A loop has three parts:\n",
    "* An intialisation, which can start with foor, while, or until depending on what we are iterating on.\n",
    "    * for is typically used for iterating on the values of an array\n",
    "    * while is commonly used to iterate over the lines of a file\n",
    "    * until is commonly associated with counter (until i < 5 for example)\n",
    "* A do section, where the function(s) that we want to execute every loop live\n",
    "* A done section which marks the end of a loop, but can also includees\n",
    "    * ```< file``` to add a file as an input to the loop\n",
    "    * ```<< EOF \\n line1 \\n line2 \\n EOF``` to have a \"while read line\" parse a multi-line string.\n",
    "    * ```<<< $result``` to have the result of a function be passed to the loop.\n",
    "\n",
    "**Note1:** using ```<< $result``` will not work, even if the assigned a multi-line string to $result. That is because Bash expects a delimiter when using here document \"<<\". If the multi-line string is inside a variable, the correct solution is to use \"<<<\"\n",
    "\n",
    "**Note2:** instead of writing on a single line with a semicolon between the parts of the loop (necessary when using the teminal, where everything has to be written on a singele line), we will write the diifferent parts of the loop on different lines, which has the same effect of using semi-colon, with the added benefit of making the code more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen,Norway\n",
      "Paris,France\n",
      "Turin,Italy\n",
      "Bordeaux,France\n",
      "Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "# create a loop that iterates over cities_array and print it\n",
    "for x in ${cities_array[@]}\n",
    "do \n",
    "    echo \"$x\"\n",
    "done "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To add the values of the array to a file, one basic way would be to simply echo each value and append them to the same file.  \n",
    "\n",
    "As a reminder, we use the \"greater than\" sign in various ways:  \n",
    "\n",
    "**Single sign**. \n",
    "* ```>``` is a file redirection. It redirects the output of a command to a file, overwriting the existing contents of the file if it exists.  \n",
    "    * **Example:** ```echo \"Hello\" > file.txt``` writes \"Hello\" to file.txt, replacing its contents. If the file already exists, it is overwritten.  \n",
    "* ```<``` Redirects the contents of a file to be used as the input to a command.  \n",
    "    * **Example:** ```grep \"text\" < file.txt``` searches for \"text\" in file.txt.  \n",
    "\n",
    "**Double sign**  \n",
    "* ```>>``` is an append redirection. It redirects the output of a command to a file, but if the file already exists, it _appends_ the output of the command to the file instead overwriting the file.\n",
    "    * **Example:** ```echo \"Hello again\" >> file.txt``` adds \"Hello again\" to the end of file.txt.\n",
    "* ```<<``` is called \"here document\". It is also used to input content to a command, but this time it is for a multiline string instead of a fiie.\n",
    "    * **Example:**  \n",
    "    ```\n",
    "    cat <<EOF\n",
    "    Line 1\n",
    "    Line 2\n",
    "    EOF\n",
    "    ```  \n",
    "    will give the the string with the contents \"Line 1\" and \"Line 2\" as an input to the command cat. Here, \"EOF\" is simply a marker (also called delimiteer) to tell Bash where the file starts and end. We would have used any other unique value, such as \"END_OF_FILE\", or \"@!\".  \n",
    "  \n",
    "**Triple sign**.  \n",
    "* ```<<<```  is called a \"here string\". It is used to redirect a single line or a formatted string directly into the standard input of a command. Unlike ```<<``` it does not need a delimiter, as it is focused on strings that take a single line.  \n",
    "    * **Example:** ```grep \"text\" <<< \"Here is some text to search through\"``` tells the comamnd grep to search the word \"test\" in the string \"Here is some text to search through\".\n",
    "\n",
    "* ```>>>``` doees not exist in Bash!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen,Norway\n",
      "Paris,France\n",
      "Turin,Italy\n",
      "Bordeaux,France\n",
      "Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "# the first echo command creates the file, so only one arrow is needeed\n",
    "echo \"Bergen,Norway\" > cities2.csv\n",
    "# the following echo commands have to append instead of creating the file, so we use >>\n",
    "echo \"Paris,France\" >> cities2.csv\n",
    "echo \"Turin,Italy\" >> cities2.csv\n",
    "echo \"Bordeaux,France\" >> cities2.csv\n",
    "echo \"Accra,Ghana\" >> cities2.csv\n",
    "\n",
    "#we verify the content of our cities2.csv file\n",
    "cat cities2.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But writing echo for each variable is not sustainable. We're going to use a loop instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen,Norway\n",
      "Paris,France\n",
      "Turin,Italy\n",
      "Bordeaux,France\n",
      "Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "# the loop will append the values of the array in a new file called cities3.csv\n",
    "for i in ${cities_array[@]}\n",
    "do \n",
    "    echo \"$i\" >> cities3.csv \n",
    "done\n",
    "\n",
    "#we check that the file has all the values that we expect\n",
    "cat cities3.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that when it comes to writing into files, we could have structured the loop a bit differently. The below loop gives the same result as the previoous one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen,Norway\n",
      "Paris,France\n",
      "Turin,Italy\n",
      "Bordeaux,France\n",
      "Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "# the loop will append the values of the array in a new file called cities3_2.csv\n",
    "for i in ${cities_array[@]}\n",
    "do \n",
    "    echo \"$i\" \n",
    "done >> cities3_2.csv\n",
    "\n",
    "#we check that the file has all the values that we expect\n",
    "cat cities3_2.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, the function within a loop can do more than simply print the values to a file. We can add some tweaks to see how we can apply some transformations to the values of our array. For example, we could add some text before each value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The student comes from Bergen,Norway\n",
      "The student comes from Paris,France\n",
      "The student comes from Turin,Italy\n",
      "The student comes from Bordeaux,France\n",
      "The student comes from Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "for i in ${cities_array[@]}\n",
    "do \n",
    "    echo \"The student comes from $i\"\n",
    "done  >> cities3_3.csv \n",
    "\n",
    "#we check that the file has all the values that we expect\n",
    "cat cities3_3.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another option is to substitute one value by another (the equivalent of =substitute() in Google Sheets). \n",
    "For that purpoose, we format our echo different:\n",
    "\n",
    "```echo \"${variable//sign_to_be_replaced/replacement_sign}\"```\n",
    "\n",
    "where the ```//``` is an operator indicating a replacement operation\n",
    "\n",
    "for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@rabic\n"
     ]
    }
   ],
   "source": [
    "word=\"Arabic\"\n",
    "echo \"${word//A/@}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this is case sensitive, which is why only the upperrcase A was replaced. If we want to replace the lowercase a then:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ar@bic\n"
     ]
    }
   ],
   "source": [
    "echo \"${word//a/@}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Used in a loop, the substitution looks like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen is located in Norway\n",
      "Paris is located in France\n",
      "Turin is located in Italy\n",
      "Bordeaux is located in France\n",
      "Accra is located in Ghana\n"
     ]
    }
   ],
   "source": [
    "for i in ${cities_array[@]}\n",
    "do \n",
    "# we want to replace the commas found in each line by some text\n",
    "    echo \"${i//,/ is located in }\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will note that when using the substitution, we need to wrap the whole command in curly brackets ```{}``` in order for it to work. \n",
    "\n",
    "An important note is that if the values you want to substitute includes a forward slash ```/``` then the program will be confused, as it is a standard operator. In those cases, you have to \"espace\" the value that you want to be treated as text and not as part of the command. \n",
    "\n",
    "Escaping is done with a backward slash placed before the value that confuses bash, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen/Norway\n",
      "Paris/France\n",
      "Turin/Italy\n",
      "Bordeaux/France\n",
      "Accra/Ghana\n"
     ]
    }
   ],
   "source": [
    "for i in ${cities_array[@]}\n",
    "do \n",
    "#we replace the commas by a forward slash but we have to escape it to avoid confusing Bash\n",
    "    echo \"${i//,/\\/}\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Reading from a file\n",
    "Reading from a file is done by changing our loop in three ways:\n",
    "* we initialise the loop with a ```while``` operator instead of a ```for```\n",
    "* we use the ```read``` command, which is designed for reading an input line by line\n",
    "* we add the input file with a redirection after the ```done```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen,Norway\n",
      "Paris,France\n",
      "Turin,Italy\n",
      "Bordeaux,France\n",
      "Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "while read line\n",
    "do\n",
    "    echo \"$line\"\n",
    "done < cities3.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to combine this loop format with a file output. this would then look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bergen,Norway\n",
      "Paris,France\n",
      "Turin,Italy\n",
      "Bordeaux,France\n",
      "Accra,Ghana\n"
     ]
    }
   ],
   "source": [
    "while read line\n",
    "do\n",
    "    echo \"$line\"\n",
    "done < cities3.csv >> cities3_4.csv\n",
    "\n",
    "#we verify the content of our created file\n",
    "cat cities3_4.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Using loops to collect data from an API\n",
    "### 2.1. Testing the API query\n",
    "\n",
    "We are using OpenStreetMap's [nominatim API](https://nominatim.org/release-docs/develop/api/Search/), which allows us to find the coordinates of any \"object\" matching the query. The object can be a city, but also bakeries etc. \n",
    "\n",
    "The simplest way to query an API using Bash is to use the command curl.\n",
    "\n",
    "For a city, the request looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{\"place_id\":252059258,\"licence\":\"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright\",\"osm_type\":\"relation\",\"osm_id\":105270,\"lat\":\"44.841225\",\"lon\":\"-0.5800364\",\"class\":\"boundary\",\"type\":\"administrative\",\"place_rank\":16,\"importance\":0.6740050666982947,\"addresstype\":\"city\",\"name\":\"Bordeaux\",\"display_name\":\"Bordeaux, Gironde, Nouvelle-Aquitaine, France métropolitaine, France\",\"boundingbox\":[\"44.8107826\",\"44.9161806\",\"-0.6386987\",\"-0.5336838\"]},{\"place_id\":250563408,\"licence\":\"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright\",\"osm_type\":\"relation\",\"osm_id\":1667452,\"lat\":\"44.79384015\",\"lon\":\"-0.6063085906819762\",\"class\":\"boundary\",\"type\":\"administrative\",\"place_rank\":14,\"importance\":0.3362245296868723,\"addresstype\":\"municipality\",\"name\":\"Bordeaux\",\"display_name\":\"Bordeaux, Gironde, Nouvelle-Aquitaine, France métropolitaine, France\",\"boundingbox\":[\"44.5463125\",\"45.0413552\",\"-0.9049215\",\"-0.2415648\"]}]\n"
     ]
    }
   ],
   "source": [
    "curl \"https://nominatim.openstreetmap.org/search?q=Bordeaux,France&format=json\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "source": [
    "We can combine the curl with a program called jq, which is very useful for displaying json files nicely but also for ng and tranforming them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100   953  100   953    0     0   4021      0 --:--:-- --:--:-- --:--:--  4021\n",
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"place_id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m252059258\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"licence\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"osm_type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"relation\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"osm_id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m105270\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"lat\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"44.841225\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"lon\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"-0.5800364\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"class\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"boundary\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"administrative\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"place_rank\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m16\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"importance\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m0.6740050666982947\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"addresstype\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"city\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Bordeaux\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"display_name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Bordeaux, Gironde, Nouvelle-Aquitaine, France métropolitaine, France\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"boundingbox\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
      "    \u001b[0;32m\"44.8107826\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0;32m\"44.9161806\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0;32m\"-0.6386987\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0;32m\"-0.5336838\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m]\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n",
      "\u001b[1;39m{\n",
      "  \u001b[0m\u001b[34;1m\"place_id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m250563408\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"licence\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"osm_type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"relation\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"osm_id\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m1667452\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"lat\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"44.79384015\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"lon\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"-0.6063085906819762\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"class\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"boundary\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"type\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"administrative\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"place_rank\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m14\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"importance\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;39m0.3362245296868723\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"addresstype\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"municipality\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Bordeaux\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"display_name\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[0;32m\"Bordeaux, Gironde, Nouvelle-Aquitaine, France métropolitaine, France\"\u001b[0m\u001b[1;39m,\n",
      "  \u001b[0m\u001b[34;1m\"boundingbox\"\u001b[0m\u001b[1;39m: \u001b[0m\u001b[1;39m[\n",
      "    \u001b[0;32m\"44.5463125\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0;32m\"45.0413552\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0;32m\"-0.9049215\"\u001b[0m\u001b[1;39m,\n",
      "    \u001b[0;32m\"-0.2415648\"\u001b[0m\u001b[1;39m\n",
      "  \u001b[1;39m]\u001b[0m\u001b[1;39m\n",
      "\u001b[1;39m}\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "curl \"https://nominatim.openstreetmap.org/search?q=Bordeaux,France&format=json\"  | jq '.[]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32m\"\\\"44.841225\\\",\\\"-0.5800364\\\",\\\"city\\\"\"\u001b[0m\n",
      "\u001b[0;32m\"\\\"44.79384015\\\",\\\"-0.6063085906819762\\\",\\\"municipality\\\"\"\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "curl -s \"https://nominatim.openstreetmap.org/search?q=Bordeaux,France&format=json\" | jq '.[] | [.lat, .lon, .addresstype] | @csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;32m\"\\\"44.841225\\\",\\\"-0.5800364\\\",\\\"city\\\"\"\u001b[0m\n",
      "\u001b[0;32m\"\\\"44.79384015\\\",\\\"-0.6063085906819762\\\",\\\"municipality\\\"\"\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "line=\"Bordeaux,France\"\n",
    "curl -s \"https://nominatim.openstreetmap.org/search?q=${line}&format=json\" | jq '.[] | [.lat, .lon, .addresstype] | @csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "echo \"lat,lon,type,name\" > cities4.csv\n",
    "while read line \n",
    "do\n",
    "    result=$(curl -s \"https://nominatim.openstreetmap.org/search?q=${line// /+}&format=json\" | jq '.[] | [.lat, .lon, .addresstype, .name] | @csv')\n",
    "    result2=$(echo \"${result//\\\"/}\")\n",
    "    echo \"${result2//\\\\/}\"\n",
    "done < cities3.csv >> cities4.csv\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "remote_csv=$(curl -s \"https://raw.githubusercontent.com/clombion/turin_crash_course/wip/cities3.csv\")\n",
    "\n",
    "echo \"lat,lon,type,name\" > cities4.csv\n",
    "while read line \n",
    "do\n",
    "    result=$(curl -s \"https://nominatim.openstreetmap.org/search?q=${line// /+}&format=json\" | jq '.[] | select(.addresstype==\"city\") | [.lat, .lon, .addresstype, .name] | @csv')\n",
    "    result2=$(echo \"${result//\\\"/}\")\n",
    "    echo \"${result2//\\\\/}\"\n",
    "done <<< $remote_csv >> cities4.csv\n",
    "\n",
    "sed -i '/^$/d' cities4.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accra,Ghana\n",
      "Bergen,Norway\n",
      "Paris,France\n",
      "Turin,Italy\n",
      "Bordeaux,France\n",
      "Accra,Ghana\n",
      "New York,USA\n"
     ]
    }
   ],
   "source": [
    "remote_csv=$(curl -s \"https://raw.githubusercontent.com/clombion/turin_crash_course/wip/cities3.csv\")\n",
    "\n",
    "echo \"lat,lon,type,name\" > cities4.csv\n",
    "while read line \n",
    "do\n",
    "    result=$(curl -s \"https://nominatim.openstreetmap.org/search?q=${line// /+}&format=json\" | jq '.[] | select(.addresstype==\"city\") | [.lat, .lon] | @csv')\n",
    "    result2=$(echo \"${result//\\\"/}\")\n",
    "    echo \"${result2//\\\\/}\"\n",
    "done <<< $remote_csv >> cities4.csv\n",
    "\n",
    "sed -i '/^$/d' cities4.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total precipitation on 2024-04-10: 0.4 mm\n"
     ]
    }
   ],
   "source": [
    "# API call to fetch precipitation data\n",
    "json_response=$(curl -s \"https://api.open-meteo.com/v1/forecast?latitude=52.52&longitude=13.4&hourly=precipitation&start_date=2024-04-10&end_date=2024-04-10&timezone=auto\")\n",
    "\n",
    "# Calculate the sum of the precipitation using jq\n",
    "total_precipitation=$(echo \"$json_response\" | jq '[.hourly.precipitation[]] | add')\n",
    "\n",
    "echo \"Total precipitation on 2024-04-10: $total_precipitation mm\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}