Session_5/(4) Neural Networks Notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "NN.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Imports"
      ],
      "metadata": {
        "id": "5lLJjkOoPd_X"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets start with imports "
      ],
      "metadata": {
        "id": "PfJFgTSoJ1Dr"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K2cq7ownJRj0"
      },
      "outputs": [],
      "source": [
        "import torch"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Pytorch is used thorughout the code. This library offers tensor capabilities that we will use in coding the NN."
      ],
      "metadata": {
        "id": "i4V76T6cPVmy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Analyzing the network "
      ],
      "metadata": {
        "id": "fO_4mJLrPiDb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets first have a look at what network we will be coding today. To focus more on the process, lets keep the code simpler. To ensure this I will use a network as shown :"
      ],
      "metadata": {
        "id": "JpZsO8spPrdx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA2wAAAImCAYAAAAv2AnvAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAFiUAABYlAUlSJPAAAEhVSURBVHhe7d29jhzJnTfq4rmAAwJyZKxJSoCgKyCvYEbOWmtK3tAcOustIAh4vdcZmqInmWuto+EVDK9AGEAiTRlyFsQerM/TQVYOq6szIiOz8iM+ngcoTJPTlZUZn/8fs7r60f/+7/9+PAEAAFCcR//zP/8jsAEAABTo0YcPHz4+fvz4/EcAAABK8f+c/wsAAEBhBDYAAIBCCWwAAACFEtgAAAAKJbABAAAUSmADAAAolMAGAABQKIENAACgUAIbAABAoQQ2AACAQglsAAAAhRLYAAAACiWwAQAAFEpgAwAAKJTABgAAUCiBDQAAoFACGwAAQKEENgAAgEIJbAAAAIUS2AAAAAolsAEAABRKYAMAACiUwAYAAFAogQ0AAKBQAhsAAEChBDYAAIBCCWwAAACFEtgAAAAKJbABAAAUSmADAAAolMAGAABQKIENAACgUAIbAABAoQQ2AACAQglsAAAAhRLYAAAACiWwAQAAFEpgAwAAKJTABgAAUCiBDQAAoFACGwAAQKEENgAAgEIJbAAAAIUS2AAAAAolsAEAABRKYAMAACiUwAYAAFAogQ0AAKBQAhsA7ODRo0dZDwC4JLABwAaWBrGlzwOgTQIbsLvrgjT2gJpsNX63Oi4AdRDYgM0tLTiXPg/2tPf4NB8A+iKwAZsYiso1C8stjglLHT0WzQXYnznHER59+PDh4+PHj89/BLjN3pvZx48fz1/BfpaM84+//+X5q7RHf/jb+at85gFAuwQ2YBVH/6ujgpU9zBnnuQFtypwAZx4AtEdgA26yKKj9+Z/nLyb89ufnL/IpWNlKzlhfK6TF5IQ3cwCgLQIbsNissJYb0mJmhDcFK2srIawNhDaAvghswCJZYe3WkBaTEd4UrKxlaqzvFdSuTQU3cwCgDQIbMNtkWNsqqF2bCG4KVm6VGutHBbVrqeBmDgDUT2ADZkmGtb2C2rVEcFOwslQNYW0gtAG0y+9hA7IVGdaCxGtP3g2EETWFtSB1TuYAQN0ENiBLsWFtILSxktrC2kBoA2iTwAZMKj6sDYQ2blRrWBsIbQDtEdiApGrC2kBoYwM1hLVBTecKwDSBDVimxLA2KPncKFYs0NcYgGLn7B8tAOojsAFR0eKuhkAUOUcFK3PUfLfKnTaANghswDw13b1yp41MPQV5/2gBUBeBDRjVclGnYCVHC3eo3GUDqJ/ABuSr8Y6Vu2xM6DHA+0cLgHoIbMADPRRzClZSWroz5S4bQN0ENiBPzXeq3GUDAColsAHQrZ7vtLrLDFAHgQ24p6ciTsHKmBbfQuhtkQD1evThw4ePjx8/Pv8R6N1oiGnlLYW//fn5iy8+fvx4/goBdlwrYefRH/52/uoL4x+gfO6wAV0LIcXj84NxY0EHAPYisAFAB8buFArqAOUT2ICfjBZvLX3Cok+LBAAqI7ABQELrH9jhLhtA2XzoCPCT5u+wBSMfPMJnvX0Axdh4bz6cJX4ezweQAJTJHTage6FQ9VCs986dNoAyCWwAwCdCG0B5BDYA4CdCG0BZBDYA4B6hDaAcAhsAAEChBDYAuuSDVj6LtYO7bABlENgA4Cz1sfe1G7u2qdAqtAEcT2ADfjJavLX0e8si1xKKUoUpPUsFN3MD4FgCG8CZwpSeCW0AZRLYAC4oTGnxbZG51yS0AZRHYAOmtfC2yBnXEApTxWkfpn6Gq2WxaxfaAMoisAH39FTATl2r4pReCW0A5RDYgK4JbYxp6W2RS69FaAMog8AG5Kn5bZET5y609W2q/1uUe81CG8DxBDbggR4K2OtrDH+eKk4VqH1p4S7bGtcgtAEcS2AD8tV4l23mOU+FVQVqm6b6vSVLrlVoAziOwAaMarmAnbq2qf+vQO1HzXfZ1j53oQ3gGAIbME9Nd9luONec0KZIbUusz2sMbbFznhrXU259PgDzCWxAVLQ4qyG0Rc5xTsEZvnfq+4W2PtQU2rY+19icMBcAtiGwAcuUHNpWPjehrR+pvq4htKXOcWocr8FcAFifwAZETRZfJYa2xDndUrDmhDbFavtKDm17hrVksDUPAFYlsAGjsouukkLbRmFtEI6RE9xoW4mh7Yg7a0IbwD4effjw4ePjx4/PfwRYWGz9+Z/nLw6ycVi7NtVGWxXJbGvO2P/4+1+ev7rvMjzFvmdNR4S1S6k2Mw8AbiewAfdMFV/JgvYytF0GqK3D3M5hbSC0tWVOWLt0GcrGwtNWoS3nTt9eY1BoA9iOwAZ8Mid8LCpstwhtE2/HLKFYDRSs5ZsKHJN9fA5lewW2qbB2xJibakMAlhHYgEWBY+o5D6wZ2CaCWrB3gbikDSlDbtCYPebP1gxsJd1VG5PblgDkE9igc7cEjVkF7K2BLSOkDY4qDG9pS44T67ex/po15u+sEdZyQtqghDGWaiNzAGA+gQ06tkbAmFvAfpIb3maEtMHRBWFOeyhayxHrr6k+WjLuc8PbnIB2qaRxlWof4x9gHoENOrV2QbWkgF1TaUXgVHsoWo8X66Pcvjl6zA9KHUtrrzEAvfJ72KBDWxRSRxdgpRTPg6n2KO18e7NG+4c+PnLcH/36U0o+N4CaCGzQma3+1VsAeWiqPUObabf9bTUH9hLOsZYwFDtP4x4gn8AGHTmiUF3zuEOhWlMRmDrfgeJ1P7WGtWEclXyOcxn3AHkENujEkWHtstic81qp5805Tgmmzlfxeqyl4ynWb8OYvXzkuuW5JUqdv3EPME1ggw5sHdbmFl2XhWjqsUTJBeDUNYVzV8BuJ9a2S8daTOx4w7ieerQodV3GPECawAYNmwoAWxaHexSesdcouQAM5zzVNgrY9WnT46XGvf4BiBPYoFFTBdBagUqhtYzQtp9UW94yD/TRfEIbwHwCGzRoqvBZK6zFbH38S7HXqqH4m2onBeztUm24xTjdc+zXKtVGxjzAQwIbNGbPsKa4ul3oj6kCVjuv79Z5oE9uI7QB5BPYoCGpQmcqGKxlj9e4FnvNmgq/qXZTxM4Xa7OtxugRY79mqfYy3gG+ENigEVNhbW2lFVQtFMtT16CIzaet6iC0AUwT2KABe4e1mBJDU21FX05oU8imbT0fxo5f4tivRartjHUAgQ2qt3VxOqbUIqqVojlcx9S1KGTHHTEfuF2qb4x1oHcCG1SspOK05GK41oJPaFvPWuNTm2+n5DUE4EgCG1TqqLBWesHaWtGXE9qEiM9i7bD1mBA01hNrS2Mc6JnABpWZKtAVj+NqLvhCn071a+8F7R7X33sbH037A70S2KAiUwXLUWGttJAYO5/aCz6hbVzqurcem6WN/Rak2lRoA3oksEElSghriqXj5YS2nvopda3CVL2ENoAvBDaowFRRemRhWmpRHDuvFoq9nD7vvahde1wKCftL9aH+AHoisEHhpsLaXhRI5ek9tMWub695sef865XQBiCwQdFKCWsxpRessfNrqdCb6oNWi9o9r0swOJbQBvROYINCpQqRvYNSzUVR6aFyDeEap4ralgrbEuZGD+OqJFPjG6BlAhsUqISCdErtBWuLRd5Un7RwzXvPDWGgHEIb0CuBDQqzd0E6pYVCqPZwOUcPoW3Mnn3c03gqTarthTagVQIbFCIUG6WFtZhWCtZew8vUWCtV7JwFqL4IbUBvBDYowFSRcVRB2lLx01tRH6536ppr6t8jzlXxXy6hDeiJwAYHmyouSgsarQWf1ou7FkJb6hz3Ho+tjf+a6QugFwIbHKjksNZikIm1p9BW7lskU+e15fxofUy0otc5DfRFYIODTBWiWxajS5V4TuTJGVM1FblHjEXjvy5CG9AKgQ0OMBXW2E6sfXsp7moKbbFz2XqOKPTrkhoP+hJogcAGO0sVEKWENUVO22oIbaWNwVLmJuOENqBlAhvsqIawFtNSwRq7lp4Ku9AGU0XuUe1R8zzhOFPjGaBWAhvspJYitJfCRuH/2VQ77D0ejp4nCvu6CW1AiwQ22MHRReitego3PRZ1pYW2MUeOwZ7GfwtS/SW0ATUS2GBDoTioKaz1VswoxL+YaoupsbyG2PH36qfexn/LhDagJQIbbGSqKKglLPQYanot6EJfT/X3Vm1Tapv3OP5bkeo7oQ2oicAGG6gxrPUcUrhv79CWOp7+4RZCG9ACgQ1WVmNYi+m5WO69mMsJbWu0UeoYe46/sfPoefy3JNWPvc9zoA4CG6xoqvgstQDsvWiJ9Yt2Oe4tkqXOFeqUGk9CG1A6gQ1WMhXWaqNgZrBVaIs9b++xp2DvgzUNqJXABiuoOawpVj+L9ZP2+SwntM1pq9j3ljJfFPdtMs+BGglscKPURl9r0adYZUwYF1NjI6fwLak4VqgzMBaAUglscIPaw5oC5b5Yn2mn+24JbTXMmRrmLsul+tdcB0oksMFCNRSesJVbQtsYc4Y9CW1ATQQ2mCls5i2Htd4L59j1K+IeCm01VfhetlusDY8ac/q0b1NjF6AUAhvMMLWJH1V4LqEgiaupH0sw1V5hrMXGW2ltre/7kupvayRQCoENMrUU1mIUq2kKuLjaxo6+ZCC0AaUT2CBDatMOm71itT3C63xz26y0Ntbn/Ur1vfUSOJrABhOmwlorFKt5FG9pYRzljKUjx5s+ZExqTBozwJEENkhoMawpPPLF+lgbTpuaH6W1Ya3zmXUJbUCJBDaIaDGsxShWWVtOcRu+RxFMaYQ2oDQCG4xoNawpNuaL9be2jJvbNnu2pX4jh3/EAkoisMGVVEHX4iauMGFNSwPRkUHKHGCMf6wBSiGwwVnYhFsOa4qM5RRutwttODWHpubgrfQXazGWgD0JbHBnavOtPazFtHpdHCM2jy7HWW5w24s5QEpqfAhtwF4ENrrXQ1hTWNwuNg607Wexdoi129S8Wrtd9RNLCW3A0QQ2uja12bYQ1liP8TBuadG6d2i7pj/JJbQBRxLY6FZqkw2bc+vFnGJ1PT0XbFPzaMrUXAvHVxBTgqlxCrAVgY0u3Vpk1kQhsa7WxsdW5rbT1PffMo7NAdYitAFHENjoTk9hLUboWF+PxVrsmpeOry1D2zVzgKVSY0doA7YgsNGV3sKa4mEbsbHSU3vHrvXWeTT1/PC6c9rZHGALQhuwJ4GNbqQ20RbDWkxP18o2ti5IwxjNCW5LmQOsITWOhDZgTQIbXegxrCkYthUbN623+55zaep4xjhHE9qAPQhsNC1smHsWmKXr7XrZz1ZjKye0xeb42N+bA6wtNaZS+w9ALoGNZk1tlC0XboqEfcTGUKvtH7uuredSOP7UaxjzHKnl/QQ4nsBGk3oOazEKCm5xVFi7NCe0CXDsLTY+jUXgVgIbzek9rCkO9tVDkVbStcwJbddan/uUy7oM3EJgoylTxVqvBZtCdVstt2+JAWhqLiuOOYpxCWxBYKMZJRaWe1MQlKXl/ihhTs05h17WAI4ntAFrE9hogrAWp1DdR4vtHJtXJV2r8U2JUuNSaAPmEtionrBGyWotzmoIa4Occ1IkszehDViLwEbVhLUvFADHa2XM1TiWQttPtb85wt6ENmANAhvVEtamaYcy1FSY1TyvctpZkczehDbgVgIb1QkbXM1F5RZs+uWIjb/a+6ileTW1hsDahDbgFgIbVZna2HoMazHagrli86uGsbSk6FUosyehDVhKYKMawto4G315YmOx5L6KnVvN8yqc+9T5mz/sKTUejUUgRmCjCqmNLKco6432YI7WC8Wc0KZYZi9CGzCXwEbxpsJaz2zu5YqNzdL6rIX5ldOm4VqmrmftvgnHy3nQn973LmAegY2ipYoZG9447VKOmvui9nEUO/+p67olQF2GsDnHuX7enOdSr9hY1P/ANYGNYqU2rdqLyTXY1OtVSt/FzqOm+bWkLdcKbeH7Lh9r2uq41EG/A5cENoqU2qyEtThtU55S+yQ2x1oYQznXEL4n9X2psJT6f1vY+/XYz9QYBAgENrIMBcPUYw2p47RQTK5hrbbmOEf2ofHzxdSactlW4euj+03ftUdoA6Y8+vDhw8fHjx+f/wifrbVJzAlYU68prH0x1lbap2yl9FlqntU2htZs06n1Z4mPv//l+au0R3/42/mrfOZ7e1qam8C6BDZ+skXBcim14Uy9ts3qi1hbaaOyldJvLY2fsWu55Tqm1qEcuSEtZU6AM+/bkhqD+hr6JbCxSpEyx/WmM/X6Nqn7Yu2lncp3dN+1NHa2vJapNenSGgEtJSe8mfttSY0/fQ198jNsHQubwpzCZC2Xrzv1+janPNqJKbG51tLYWetaco+zdVgLcl7jiHWc7aTGn76GPrnD1qElC/5//8uT81dpP/vH+/NXtxFAxvVQdLfuiD5Mzfkax87WbZizRu4R1q5N3W2zDrSltXkLLCewdSanEBnkhrSYpeHNRhQ31n/aqy5bh41rLRZ9W86D3DXyiMA2ENz60eL8BeYT2DqSU4jcGtJicsObDSgu1n/arD579mWL42bsmta4npw1MjgyrA1Soc2a0JbUuNTX0AeBrRNThchWQe3aVHCz+cRtVaRyjD36Mzbvax43W11Tao0sIaCNEdr6kRyf+hqaJ7B1ILXQ7xXUrqWCm83noRYL795t3aetjpmx67r1mlJrZKlh7VIsuFkf2mIfgH75lMjGpQqRo8JakHrt1DnzhU26blv2X6tzaIvrSh2zhrCWYi1tS2zN0M/QPoGtYalF/MiwNhDa8miLvtza36nntxjyt7qmmsJa6lytH33Qz9A2ga1RqcW7hLA2ENqWcXetDXv2Y+1jZov1IHbMGu+s1X43kDypeWzPhHYJbA2qJawNhLY4G3CflvZ77HmtBvxbrivaVhUHn9i5W0faIrRBfwS2jpQY1gYln1tpWi2+exXrz7mFV29hDXomtEFfBLbGxBbqGgJR7Bx73XxsuuRqfaysfX2x47XwtkJ32fohtEE/BLYO1HT3yp02enTLXbbU97R8d82dwzg/z9YPoQ36ILA1pOXF2cbzmSKVXK2MlbXnfux4PYQc62ibhDZon8DWuBrvWLnLZpPtUazoSo2FaPhoPNivfX0thjV32foitEHbBLZG9LAg977ptF6EM6+PewhrCk3IJ7RBuwS2htV8p6rnu2w2Vq5dj4mex4h/uLiN9aVtQhu0SWBr2M/+8f78VTt63XAUqf2Y6uvUHDBO0npbP7wtsk/WAWiPwNaAVouQHu+y+RdQYqbGRmtF2l5zQaihRbH1wB4DdRLYGtfiXbbe+NfS/swttnoZI+YC3E5og/oIbJWz8LZDX7JEiyHGXIDbpdYGcwzqIrBVrIcFd+xtkT1tNO4o9Evf33dre4ytGz28HXLsGhXr/RDaoA0CW6VyF1q/06wONk6WEOqAKUIb1E9gq9DUAhtC2vCgXopxUmOg1fExtr6ZC3AboQ3qJrBVxsL6WWiHVtpCnxJjbABrEdqgXgJbRSyoD4U20S60aGpctzjuzWXYltAGdRLYKmEhTQvt01IbeQtY38z3L8wFWJfQBvUR2CqQWkAVM/fVFtxsjlybMyZaGj/mAuxHaIO6CGwVCwuuhXVcze0ihDOmx3FhLsB2pkKb+gLKIbAVLrZg9l7I5Fx/6RuOzZBrS+a7cQQsNbWXWl+gDAJbwZYUbz0J7dBCcLukb/uVM99bHR+KQjjO1LpifsLxBLZC5RRvwdhC+7N/vD9/1YfcIja0aSkbjw2QS7eOhxbH09rhdOx4j/7wt/NXfVm7bamfMQFlE9gKpJj/Yix8jm0s4e9yN5zQviW2sQ2zT6mxGBvrY2pdN6x32+o1lLKuYZ6G/049gPUJbIVJLXYK+mmhjXLb6ajNxYZGjp7nu7UO9jfMu9j8y927hr11eAC3E9gKklrYFDDzhPbKbbMSNhT926fY2JsaD7H/X1txpJjblrtrzBXWlrXnZTje8ACWEdgKkVrIlhRvLfwc2xrXENpmqv2CvTYTGxaD2FjIGa8t2/v6Ww01qesa1ruxB/3aYwwYa7CMwFaA1MLVe/F2bWl7hOflPPeIjUQf92eNMRYbNwohbjGsgakHbTmqX40lyCewHSy1YN1ayNd8l22rc89t0y02MJsTwZZzviZ7zoct5nPPhvaMPajDkv76+P9NP+YwZiCPwHag1CLlrst2QtseGdwu6WcGS8ZC7Dm1FkBrz4fc+etnvdY1tHvswfFy+2FJGFvyHGMD0gS2g6QWpjULtxrvssXOee1iLhwv95g2E9YQG0O3jO2158UetpxLw1yd+xo9hLZSxsplH4092E5uG88JW1PmhDf9D+MEtgOkFqTeCrdcW15bOHbu8ZduJjYhYmNgq7Fd25i7tR3C9Zpnn6XC57DexR4lGPoy9mCZnLZbM6iNEdpgGYFtZ6mFaKvNsqa7bEeea27BslbRUEpxxPa2LkBqGktrt8Xc+Zia5y3cZYtdQ+4YGdon9Tja0OexBw9NtcvWQe1SzmvpR7hPYNtRagFaaxOMHaeG0LbXWyGn5BYlucWBjadve8z7mFrG3pJ2yJ1/g9x5zbShLWOPow1jI/bozdQ17xXUrgltkE9g20lq4dlrgys5tJV4brn9sqQIKKGo4Vhrj4HY8VoreubOt9AuY20Tba+K77LdendtLUObxx5HG8ZQ7NGSqes5KqwNpu62tdYfsJTAtoPUgrPF5pU6ZonBKHVOR2/ucwqMsc3eZtO3WP8fPa6Pcst8GJtfKTlzN/b/awxtNZ3z0Dexx9GGsZZ61G4qKO1NaIM0gW1jqYVmy40pdeySQlvJYe3SnEJiakMv6brYTmwMbNn/sWOXXPBMtcfUfLoWjrdGG9cUgFLnuuV428rQh7FHCYZxGXuUIHYeJQW1S6WeF5RAYNtQatHeY9NJvUYJoa2WsHZpTsFQyqbN/vT9Q3PbJHz/nOfMmZuX5j4nhKPhUbol7VGDoa9jjxIM4zf22FrsNUoPRbHz26PNoGQC20ZSi8ueG0rqtY4MbTWGtUtLC4Maro3bHD33Y69RYsEzdq7hPHPPNTx/eNwi2mZXwew6pB0d2q7P79KtbVKzy3Ex9ijBMM5jj1vc+vyjCW17e3968+rF6cXz56fnF2Pw0yP83YtXp1dvjv9H/t4JbBtILSpHbBap1wzBac/gtvfrba2kAoCyGSdxlwVCji3mXep4RwezMalzMtbShvETe5Tgck6MPZbwlkPuuwtqL57fjaenp69fvj69fvv29Pb8f34S/u71y9PLr5/ehbcXp1dbl29vXt0LjS9qCIo7nbPAtrLUQnrkRjD12lsHqZzjl7RZzpV73sOEpk2xvt17XMde76ixN/a64RznzofwnC3bMnXskkJbiQGyJcM4Sz2ONsydsceY2sKau2xbe3N68fwuqL1+ENHi3r4+vXwaAsn5z5v48V5o/Ovfz18UbZ9zFthWlFpISljgc85hCFZrhbfcY5XQPreas5GnNlbqFC2UDhrbpc+pOeN/ztza08ff//L81T5CUJsKa6X3eyuGMRl7lKTWO2vuCG7l/enV869P11nt2Xffn969uxzH707vvv/m9Oz8/wevv36+/Z02HhDYVlJ6WBvMOZcl4e3yObnPK21z25Pg1oaa+nDvc73l9YbCYU9HvGaOnKBW4nn3auiP2IPl7Jm3ef/qd6eX98Las9N3d0Hth2+/Oj15cv6rT56cnnz1x9MP7767Cm1vTy//76a32RghsK0gtXiUuDAv2TCug1jsMUeLG9fSjSQ8zyZUp5Lnf63zq4S1Ief193hrortqbRrGeOyxltrvUrnLtrb3p7/85/1ba8+++9Pp23tB7cqTb09/+u7qPtvr/3P/Ltv7N6cX5zrm0+PFm7tXuhZ+Zu7iex69OA2x7/2r8LN0d3/39evz33z29uXTi++/vLN3+Xp3f38+0Ps3rz59eMrlc56HcxkrT3c/59sIbDcKHRKz5qK7hbU3BuJy2zk1nlrzZUFLP2pV8tzaq13nvk5pa1LOuQyBas3wNueYJY8zlhvmQuzRu5r3hkO9/8vpfl57dvq336TS2mdPvv2P0zfnrz97e/rx3fnLT/5++uv5q08iP8j193vf9MW7H6/enznL3bl8ClbPT0+/fvnpw1O+CB+a8vXp6dOx4HTkOc8nsG2kpgXVBrCtoW1z27n2oDJmuKbLR65bnru12LmUNJ+OOJe5/ZQ7N/Y2d6xdBq2csHVpyfNKbDP2McyZ1Nxp5e6Uu2zref+X/7z3ARmnZ/92yshrd56efnV1k+2vf1/p1tGdp9cHn+n111MfoPL29PJ3r0buoC136znPJbDdoIZibY6pxX+J1PHmFkOly72e3PYNx6u5jYbz3+Iatjx2rthrrzl/trRlv+RKrQ9HW6N9LkPY1GOOktsNKNeDu0K//sUpK6/dfdcvfn3+8uzt/VtsN3ny7Q+f17Xv79/He/bdu5/Wu48ff0i/dfPOs2++P707f/+767dxvn15WvNH79Y651wC20K1F2tTvgy2z49csefFjrFF0ViS2HVft0/K3CL4aHuf7xHtU1N/BFvPvzl9MIz93PFfmiPPu+Z2Yzu1rUdr6fW6GReC0g9//OqnABoC1VWWOr3+r3o/LEVgWyC2SLS8kQ6FwtSjV0s3jjntFl6j5A3q6PPb6/VTr9HbHJjb5rW0T+yahvOfM2/XsPfrUb/W3kbobZEkPfvu9KeRW1lf/etVYvvr31d9W+SeBLaZ9igIWxUrOFpt0zkF1pyCbG6RvLXF5/Pnf+Y9ZjqqfUovqNeaf0P7HtHGpRnm7RZ9v+WxAW717FdPz1+xB4FthlSBYlPNE2unmou/Nc99ToFWQpvNOoelQWzh87Zon9gxe5j/4dpvadNa2mhpHw9z9/KR65bnAm0a1ty1HsGDD8rIvuP0PvppiUV7+qv7v0Pu7Y+n9X7ybl8CW6ZhsI+xuXLt1jGRW7RdLsR7y3rdBUFr0oxjrtk2sWPVNP9j55pqp/D/1mzHkq3dx8M8nnoAxxvWulIeawvHfPLwk0PyAszCXwdwuHc/Xn0q5q9Otd4XFNgypCaOzXa+WJttsUBtbetzzh1f4Tz2ar+s11o7pMVkvM4abbNX2+5hqzEVjhs7du5rAvOMzdFWf95r7LqGdWqNRxeu7zidXp9yPodj+a8DONb769uC2Z+KWR6BbUJqEitClou1XQuL5trjIhwv95hbbzyTx94rqF3LDG5LpJ7X0howXGf475y2mjM+Sxa75pb6GOjck29P/3H9yYlfvzi9Sb0v8v2r0+9e3v91AM/+7Tf3g8+TX5zu3bsbuXP3/tXvTleHmZT9qwPevjz97sFvxn5/+sv924L3f+7u6HOeSWBL6KVQY5k5Re0a5hTGc4vuVRwR1K5tFNrG1LwGxM59TvuMjcfdxxwAWYb1+qt//+7BXbavnz4/vbhLbfciz/v3pzcvnp8ePX15dXdt7BMZr3+x9t0xX7w5H+/uOK+en57OTT7B6//6KUy+vzufVK58+/Lp6fmri9d8cR22rt/Gefw5z/How4cPHx8/fnz+I4NU4VFzoVaaWDvX0MZj577nec8trm+VfL0Swtql3/78/MVDc9qi5vE5Zc74uZS69qPnxFIt9zNtG51zDX8E/qP/9/xFJ7Zag94vDSPPvjl9/8MfT1+d/3hp/jHvjvXx+lhvTi8efX0XncY8O333bvhF1Knvi/jm+9PHP95/tX3P+TbusI1IFTI28HXV2p5Li901hbbLbb9bzzf6/Iy3Ih4icV65bRH7vh7XgGGspa69hDmxhH4GBpdrXQmPrYRfKv3u++s7bWnP7gLPu0hYC558+6fTd6kDPvvu7jWv3o/5wFenf08eJOIuSH73TeJ54bWvwlpw6DnPJLBdSRUdW04e7qu1+DtC7sIe2nRJu0afU2JQu7YwtLU8/uaMg1uLBmsmsLVhnVrj0ZMnX317+uHumkNwC2Hn2YPMEf7um9M3390FtXcfTz/cBZ70jaInp29/eHf6Phzr/DefhDD1/bvTxx++vfuO6w89eehzmPzm6nzujvnNfyQ+6ORXp9/8ceR5d3/45u6133167TFHnvM83hJ5IVXE9DaR9xZr+1Lbfex8SzjX3EI8yDnf5PFqCGzBzLdHtrgOzBkXgznXGjt+6e1V63nDYGwM9/aWSPO1R1dvQwx3wqKhrA3usJ21WKTVpKY2XlL87iW0Y25bhutYfC21hLVgpXOtcR24qY9vZN0EgHUIbHdSBY2i41hHFZtzlTZOwvnknlOsqI+2fU1hbZD51sjYNde2DsT6dI5bn1+6VvoagPZ1H9hSRYmNe1+x9i6pcKytiA1tmjuOw7UN1xe9zhrD2mAitMWuuaZ14LIPc8wZHzFzXq90NfU1AP3oOrClCg0bN7lqGCtzCvOWCvBctV9zOP8513A9HmJjY2m75I61o/Q4xmnT2Fzr6aPvS19rYC3dBrbUhm0BOM7aheOaWijyQvsuHt81310bzLyG0teCMCbnjMtU/y+51hrnROycS+9r6F1vv4ONtHu/9PrXvzh/0a4uPyUyVWTYtMtQYlE1dk61j5fsgruFsHYp8cmRg5L7NrvfznKvZe68q3FOzL1GKN3oPGzwkyJ9QiQ96+4OW6rQMfGJSY2bmmWP+YyA05JS14IwDnPHYriG4ZGr1OteS6ztWr9uAOrWVWAT1uoR64+SglMrYyZch/H/RYltsSSorWnstcf+rsZxZOzTotbePujtkPSum8CWKnZs2GWK9Utu4bqWvV/vKOZBWcK4Gx45Qv+t0YetjoNe5jH96XXttmfRky4CW2qjNuFZortx09rPsEWU0K9zQloQznmP8748p9rCT+x8rf8A1KD5wJYqLGzW5Yv10V4FY22F6SY6CWtHC2NtznjbMqgtmXfWUyhLK28j9HZIaDywKS7aUFpfGTsNiITQIwJ6SUEtxxFtdIvY+ZrHtKS38Wz+0puuPnRkYKK3obbCsXSj7dny3bWDry2095wxHNatPdeuOa9V25pqD6AXtd+dcncNPms2sPlX1bbE+m3L0LblselX6UGtJeYwPellnbAe0qMmA1tskzbJuZUxxFI1BrWax7t9AD6r9S6Vu2vwRXOBzSbdrlgfzimCc21xTPoUxlLueApjfHiUYupcSjpX6F1sPtYWfmLna72hV00FNkV2+6Kb0Q59b6NgjiVBjXXE2l0bA1CjZgJbqjCySTPHHuGPNg0hLXcM1RLUYudY09pqH6AXsbFey102d9fgoSYCm7DWl+hmtGHQMo5ImRPSgjCeahpTsWvbcs4tVeI5wd6i+2ThoU1Yg3HVB7bU5myCtyu6Gd1YrCn2mCOMlzljJoxb69J2Yn2hzelRdJ8sNLTVcgcQjlB1YEsVSjZo1tLLWBq9zt/+/PwFlwS1L+a0A1CG0sJR6nzUc1BxYBPWiPXz0gJS4dmZkTA6tXaEMTJnnITj1b4e5VxvCXMndg72A3qWGv8hJJUQ3IQ1mFZlYEsVByZ3X7bub+OJIKw5c0JJGDfGzvH0AUzPg6NC21RgNH/hi+oCm7BGjjnFdTD3++lD70FtzrUfOYfMX0jLCW17Bbec11LPwX1VBbbUpmxy9yvW97cWccbUWYs/xzZxTWHs5I6fME6GRw9Ku85YP5m/cF/OnNg6tOUc39yFh6oJbMIaW7k12LWk57kUrn1JUOtR7LrNJShbzpo13AFbK7zNOV6vaypMqSKwCWtMUUByK0HtvrH2KO26Y31mX4C4OWvYnLB1ae7z5pwT9OjRhw8fPj5+/Pj8x/IIa8yxpICroTDdU3TO/fmf5y8qt/Atnr2NiZx5sWS+rcnchXzRtf1A5ivkqfJTIgOTnDWUuIEdLTq3WvhZtgXXENqjt/Umd14c2S7mLuRLzZej5nFv6yrcosrAZpITExsbc4o746tRM8NaGAfGwhdz2mLrMBU7vv6Ch3LC2p7r3Z6vBa0o/i2RweViY5KTI6egU/SlRTf52t4auSCo9WzJvBh7zpbtaO5Cnug6fmdqvqSeO5e5CbepIrDBXDkF3d5FZo2iG3YNoU1QW2TJvMiZb2vZ87WgVtG1+2zJfJk65iXzEdZV7c+wQUpssxg2nDkbDwkhFA2Pksw4nzBWFBe3mZpvW9N/8MUWYS0Y1sqcB7Aud9ho2tjGFTaT2N/z0Kyiu5Q7bxmBTX8/FOvrnLa65bm59ngNqNnUem2uQJ3cYaM7swIITW7wipZ8uW0V+7615puwBmnCGrRLYKNptxab0Av/kAH1EtagbQIbzbNR3S6rDSt5O6TxkG9uW8W+/9Yw6O4axAlr0D6Bje7ZzFZSwgePCGuL1Hh3TV9Ceu6GOWKeQBt86Ajd8K/0y80u6Pe+2+ZDRm4y1r+3tNeac828hXGpddn8gLbMusP2/s2LTwvE8Hjx5v35/wCtShUFUSFA7XHHLeN1QuGieNnXWu0dG3v6k96l1mXzA9oz7y2Rf//r+YvP/vr38xetevPq9PwcTgXUdi0KJOQVBUOgWjO8zTimwmXanuPfXIPbpeaRNQ/a5GfYkn48vT1/FTQfUBs2VSgqJMdNtcus4mBG0HpgwXMVLsut0Xa3HiM29vQrPRPWoE8CGzAqtzAIX88uFC4DWM5jhkXn06kj/qHiltfUr/RMWIN+CWw0L7dAPKJ4LdWSwuDooHT067dizTaMHWtqrpmLcN+SNRlox8qB7c3pxd2iEhaWR4+en169+fy379+8Or14/uVnwcL/e3H3P8d/Iix1jOf3jvH8xd0xxg7y/vIYd4/wfef/9cX705sXF9/z6MXdK5//z6vz63z9+vw3n719+fTi++/OzY+0VWtpIdmDVBvkFAbhe/YsIPZ+vVaUOtZj56WP6dWtazJQvw3vsL09/XgXil7dBbWnX788vb78YbC7//f65denp89fRULb4PMx3rx4fj7G5UHent6+vjvG07Hg9PfTvY9Hifzw2dVnqPzk3Y/3TpaKCWDzrFkYDEFKQVGPLfoqdkxzE6YJa0Cw6VsiX3/99PQylX3evjz9buI2VTjG1/fT3pW3p5e/mwp+8zz91bPzV7Ro2OQUkvdtWRiE518+1hTOW/Hfhlg/rj1moAZbrslAXXb5GbZn331/evepUHt3+v6b81+evf3Pv2SFrWffDMf4eHr33VWgugt+/3d4P+MKnnz7w6fX+Xh1ss++e/f57z89fjh9++T8PyhSThFv0/ts78LgyzzKe+QQ2vLt3VaxPjRHYdzeazJQts0DWwg5P3z71elztnly+uqP35/uxaC3/3n6y0Ri+3SMPw7H+ByoroPf6/9aMbHRLMX/QzUUBuE8cs4lXEtPfbemo/tav8H0GlbKmgzsa9vA9uy7058e3IZ6epr1jsPRY5xOX/3rVWL7699XfVskdZtT/MU2wB4KyNoKg3BOOec1VfT07Kh2Sc2z2DmVOAZhK1Nz03yAfu3ylkgogc3uvlRxUHpb5Z5fKgzwxV79bQ7COGENSKk3sD391enejbq3P57enb+kb0sK9Nhm2Gqxn7quWgqDcJ655xqut9W+bJUClV5MrU3mAlBvYHv34+neZ0c++9Xp6flLWCK2KbZW6LcQ1i6Fc849796D29i1793nNY4x2MrUemS+AEG1ge399S9R+/UvfvpQErhm05tWexuF88+9ht6DW+nMV3ogrAG5yg9so7+r7f3pL/95/3ezPfvVxf21J784/fr85Scjb5d8/+p36d8RN+Ltj950Wbpbi/DYBtlKcR+7jpYKg3AtudcT2qOVvp1S0nWm+qelsQgxqfk4Zw0D+lDFHba3L5+enr96c/4UyPenNy+uw9az07/95vL+2vUnUb4+ff3i4vmvnp+ezk1rwev/Or05Z8f379+fj0fp5m58se+vvbDvJZgM5hQ9oW16a59AUQj7mwprANeqeUvk25dfn55+Kqqenr5+fRW2vvmPq19i/eT0m3+7+t0Bry+enxvWvvrX+78zLgS/p58Lu6dPfzf5++PYV48Fd66eC4S5wa1FpV1X6nzMY1rW81oMLFd+YHv2zem7bxK/uO3Zd6d3f/zq/Icvnnz7p9N3qd/3Fp53/du3H/jq9O/Jg1C6pRtg7Hk1FpMKhM/CteZcb2ivHkJDT30PJbAWA0vNC2y/uPeTYeFzPh6491bEsW84Pbk6zK9PY9/1xa9Ov/njD5/C1bPLY9/94Zvv353e/fBt5MNGnpy+/eHd6fu7sHcvcoUAePe8j5+ed/WrAUY8+Xbkte+e9eyb/zjdexcmh9qiwG5hA1UgPJR73b0Et73ltKl2pzXWYuAWjz58+PDx8ePH5z+W4M3pxaOvT6/Pf/p0JywayuCzsc1wjU0wtsnWsMEqEKbNCQa1tllJY7iH9oZr1mLgVtX8DBvEzCkC54ptplu+5hoUCHlCW+S2R2jT0vs9Vw1joJW2pm/WYmANAhtN6nkjVCDMF9olt21qCm4lnWfsXOa0PdTEWgysRWCjansUpLGNtcSiXYFwmznhIbR1iWNgSk3joMb2hcBaDKxJYKM5W2yGsWOWVFAqENYT2iu3zUK7lxgsahibl21sjNIKazGwtiID2/QnTUJZBenRFAjbCG2X2341jMeSxkJL7QoDazGwhQI/JRLyjG2MW2+Isc34yI1YgbCf3PBQQrsfMT/GzJ0zpZw3zDG1NhjDwC28JRJmmFNkHk2BsL7QpjntGsbDkWOilPEYO4+5Y7PE+QWDqfFpLQZuJbBRJQXcZ2sVxMyT276hf0oZq7WMidh5mvOUaGpcWouBNQhsNGOvjbGUglIBe6wwDnLHXOirvfqrlHEROw8FLK2YmmvGOrAWgY3qlFCQHh3aUq+jSNhXaO/cNg/9dsT4LWVM5J5H7PtKmPsQTI3FUuYc0IZiA9tQ2NigydHT5piaE4qE44S2z23/rda2UtZL6zYtmxrf1mFgbUUGtuvFMPxZAUBQ0jiIbcpbnmPq2IqEMpQQ3C7tPS5i1zP3PGLfv3V7QcrUGrz3fAP6UNVbIm3UjDlyg9zztacKBcoyp3gLfWt9e8i4piTWYOAo1f0Mm8KmXzX1+9rnqlCo19zgtlQJ8yN2DmuP0RKulb5Yg4EjFRnYcha/sHjatClho4ydw1rjU6HQhtBXe69tJc+PXMY4R7MGA0cr9g5bWARzFsK1ChvK1ms/KxTak9tvoe9zx30J82PvcyjhmmlfapxZg4G9FP+WyNzQZvPuT0mbZexcbhmXCoV2hf7L7cOl69ueYyR2fmudg/HOEazBQCmq+Bm2sDDmLI5LCxvKVkufxsbokvNXKPQhd20LrG8PaQ+2Yg0GSlLVh47kFjc28fa1vGEqFPqTu7YFYXxcjpGx8bLnOImN17XPIXa81HyBJVJjas+5BTCoKrANchbMsODayOtXWx/eWlQqFPoW+ji3n0te44xVamUNBkpUZWALcgubkosa2hQbl7eMQ4VCX3LXt6PtvbZuMbdgkBpH1mDgSNUGtkFuYWNDb0erG6cxyrXc9e3SXvMjNl5bnZ+0TVgDSlZ9YBvkLKhhQVYU16PmvoqNx9g1KRZICWMgdxy0vsbNnVuQEsaN9RcoXTOBLcgtaqYWaMpV0+aZe66KBXKVssbFjr3XeDUvWMPUHDHOgFI0FdgGuYvslgUNt2m5by6vLXWdigVi5qxxe82lEsZry+sG65oaK9ZfoCRNBrYgLLY5C+6eBQ23qXEDjZ3z1LhTLJAyd81ac51b6zi3MkdYamoMG1tAaZoNbIOw8OYsvmsWNNym935QLLCVW9e52HNLGrPWcVKmxof1FyhR84FtkLsI2+zLVPMmOufcFQtMSYWmOetcC2ud+cIcwhpQq24CW5Bb0LRSzNSo1XbPGXeKBZa6HDu561wwZ62LfV+J49b6zbXUmJgzZwCO0FVgG+Quzjb9MvSwkSoWyDFnTcpd54Jw3CXrXQnjNnYO1m8GqbFg7QVq0GVgG+Qs1EsLGeZruZ2NIbYytY6F/59blMbGqfFLrVJjV1gDatF1YAtyi5mw6Cta9tfChpozbowttrZ0rYuNzZLmZuxczKu+pfq/hb0F6Ef3gW0wp5hhfa22q/HCWtYaS7mFang945dapcausAbURmC7khvaFDKszZhirqWFZ3jeLUVriQVv7JzMq/4Ia0BrBLYRucWM4Lat2jfW2NjIHV8w2GqdWTIWjV1KJqwBLRLYEnKLGaHtNi2239JrMpbItWbxmbvWBWGMljpOY9dgXvUh1c9rzheAvQlsGXIW+pKLmBrVvLnmFg2KS0oTxmTu3Ct1zat57WC53HUXoEYCW6bcQqbUIqZUrbWVooG1jY2pksZSLWtea2sNX1h3gdYJbDPNCW4sU+sGu6RoiP298cNRlo69ksZsrWsI8y1ZdwFqI7AtlLMRhI1E4R3XUtvcUjQIbcSUMgbCGG1hzTOn2jE11nLGK0AtBLYbtFLElKTGTVbRwJ62HFNT61RNa17sPK3F9csZpwAtEdhWMKeI4bNW2iJ1HXOKBsUl1/bu+9jrjY3N3LEdjmkMs6ap8TRn3QWohcC2opyNQgETV9tGm+rHJdcSe47xwqCkORLOJfd8jlr3zKm2TPVbSfMDYE0C28pyi5ijCpgStHDdqWtQNFCb2HjOGcu5a17Q87rHbabGjXUXaJnAtpHcIkbx8llNm22qz269jtjzjZO+lNDfc8dy+P7c54Tr2+sazan6TfXV3LEKUBuBbWM5G8mexcvRar/O1PmvVTQoMBmzVVG69rgK55l7ruG1jWtSptbcreYFQEkEth3kbiq9Fi8tbLiKBtay5xoQe601xnPuuhdsvfbFzqPH9bYmqf6x5gI9Edh2lFvAKCLKFOuXLQoHBSaXai5Oc9e9YMvxXXMb9khYA/hCYDtAzmYTNqvWivOar+eIc1eU9GfPcRZ7ra3GXThuiWvfEXObtFSfWBeBHglsBym1eNlbDZtvacWDArMvrRWoudezxdrXWlu2qLT1FqAEAtvBwgaUswnVXqTXev5HFw+x16h9PHCs2PjZqyDOXfeCcK5bj3fzqQypfthrbAKUSGArRM5mtEfhsqfSN2DFA3vaa27HXueIMR1eM/d111r/Yq+3V/sPhutJPXqSul7rLdA7ga0gucVLbZt5jYVHScVDKQUm++ulUM1d+4La1r9gOOfLR46lz6tN6rp6mQMAKQJbgXKLl5o375I34RKLh9jrtlrA9Wavfoy9TinzMZxH7rmEa1nabnvMp+H81jxmsNVxj5K6jlLGJcDRBLaC5WxWpW/ctRUVigdK0fN4C9eee/2lrYF7nk9p1z5X6tyttwBfCGyFyy1catq4S92ISy8eYudQc8HGfv0Xe51S52MQzi33/Oa249rzKTxvr768duRrL5U635LHJMARHn348OHj48ePz3+kZLkbcimbXex8S9yMayoeampXpo3159p92cqYSc3TS7nXtUa75J7TtY+//+X5q7hHf/jb+at5Su7XqfaqbUwC7EFgq9DaRctWxs6zxM041Z41na9Cp057zJPWxkxqzl7Kub5b2ib3PIKcgDZlToArsW+n2qvW8QiwNYGtYjnFwlEb4C1F0J5SbVhy8VBL+5K2Rz+2OlZSc/fa1LWOHWvJc66tEdJicsJbSX081V61j0eALQlslcstWvbeDJcUQHtLtV0NxUMNbUzaHn0YG+etjJXUPL4Wu+Y5bZTzelsGtWs1BLepNmtlLAJsRWBrRG7RssfGOKf4OVIt5xlT+/n3bo/+62mMxK51zNj1jz3/+vumXmPPoHZtKrgd1eeTbdbgWARYm0+JbETY9HI2vjlFTcti7VBT8RA7V31crz3GX01jfI5wXbnXFuZIzjy5/J6p7z8yrAVTr3/EujDZZo2ORYC1CWyNydkAwya69+Zd0sZ8ROECe+t1nC8NbqnnTLXl0WFtUFJoS73WnD4C4G5N9ZbIduVuzmtunLHXLGVznioialR6m/PQWJ/1NA/3lJrztyolqI1JvUVy63HQ4joLcCR32BoWNsaczXHLgiYoZYNutYiInfvW/Qo1yF0H5yo5rAVHnZ+wBrA+ga0DOZtk2GRvLfBLDgiKCEqx9TyJHb/3cR6uf602KD2sDWLnudUYtM4CbENg60RusRI23DU38xI26R6KiNh1bFWYsa6tx2Er43wNt7ZFLWFtsFdo62GdBTiKwNaZsHHmbJ5zN/NSg0FPRUTsekrtmx5t3Rf6Ok/uOti6tcZLT+sswBEEtk7lbKJhE87Z0FPfMxwj5zhrS72mIoJSrDUWY+PdWI8LbdND+2x5V9A6C7A9ga1jucXKdeAa/nz991OWPm+J1PFbLiJi17Z1e0PNctfCmm3x1she11mAvQlsZBcrYXNes/AfjrfmMYPU8XooIhRKZVp7nF+KHdtYmCdrHUx8XH7p1rzT1vs6C7AngY2fHLnJhs1/jYJWERG3Rvuyri3HpKJ5vl7nyNzrts4C7Etg456w2R654YZCYGnRpIj4Ina9S9uW22zZ7vqUOW69y2adBdjfow8fPnx8/Pjx+Y/0bknx99//8uT8VdrP/vH+/FW+OQVA7Nx7LSK0RznG+mKNftDH64m25V3AuXwb5JYf4LGX2Ns6U+Mm1j4DYw5gOwIbP5nakK/lBrUxc8JbTiGgcB2nXY63ZR/o3/WMtWUL4SxmLLTFxk1snA2MN4BteUskn0xtyGsLYW94TAnnljq/vc+9JksLMLYlrFGLqbXCeAPYnsDWubAZT23It9xJyzEnuF1LnbtCgt4Y86xJWAMog8DWsZygNgSp60CVE7Dmuny9mMtzFtbyxNpiqv+53VZtrO/W1WN7jr3d87IdptrEGguwH4GtU1Ob8VhwGgLVVKi6VU5oS52/QuIhoa0ct47PWJ8Z9+tq+efXpkytC8YawL4ENh7YOpDlWBoMFRKUQhimRlP/GGaNBdifwNah2Ia8NCRtac75KCTSYu0jWOzn1jEa6ytjn60ZYwDHEdg6kwprpco5N8VEHqFtH3u2p7G/vl7eDpl7ncYYwLEEto7UXJSnQptighrcOk6F6m1o1zTrK8DxBDaKvrvG+mIFmMK1XLG+UUyzJeMLoAwCWydiBV9NYS12roLGfAqx7RiPtMAaAVAOga1jNd5ZE9q2pR23cUvxG+sTBTVbMbYAyiKwdUARzphYUWa8LLdX2ymoAaAfAlvjYgVkzT+35i4bNdni7hoA0A+BjSrVHDhL4i5buWJ94O4aAPRFYOuQsMMloW0dY+0lXAEAtxLYaIqQQQvcXQMABgJbw8aKvpburrlTuB532W6zRzsJa9vQrgCUTmADPhHa1rU0CGjv4z36w9/OX7Vt7DoFWIDyCGwAN1gzYMWOpYgGgH4JbMBP3GVbh4AFAKxFYGvUWIHd4s98jV2TcHEbYeMY7q4BAGMENiCLIPzQ1m0irB2n9Z9j6+Xn9ABaILABD3hr5HJLQpZ2PUZod23/hX8gACiTwAawwFqFfuw4iudtDCFNUAOgFgIbMMpdtvmErHItCWmtvm3Q2yEB6iKwAVFC27i1rj92HMFvPUuCWo+MOYByCWwAK1ir4FU4r2NuUIv+40Rjd6PcXQOoj8AGJEULWXctbqL9trEkqA1jvJfQdi123QCUQWADJgltX6xxzbFjKJyXGUJabt+Edh4ePXF3DaBOAlujxgqRn/3j/fkrYE2C1jHmhLQgJ6TF/n/tYSd2/sYuQPkENqo2FkIVINuIFrIzCubarXGtsWMYt/m2CGo5ag1t7qwB1E1gozlziznyCRUPrdEm2jXPXkGtl/4w7gDqILDRLMFtPz208xrXaDwuM3cuLw1ql2LPr+1ulbdCAtRPYOtMSz/HlnstQ7GnWF5HtJDtsH3nFL2x9lE4j5s7b0M7Do+1xI5VS2gT1gDaILA1zKb80NwikHHGFluZOz/DWDxiPJYe2mq7EwhAnMBGtwS39bXanrdeV+z5gu8Xc+fjXkEt9RqlhqLUeRlzAPV59OHDh4+PHz8+/5HWxAqg//6XJ+ev6hR7O+RQjCwtsBUz8/QSRMauc8419tJOS8ydq0e12dR5fvz9L89fHWcqQBpvAHVyh61xsQ265p9lmwprQfh6SXESirLhAcGtYyH2/N6L57nzbOmcXsvUax99t01YA2iXO2wdiBVFtd5lywlsY5YW3gqdtNYDydj15V5b620z19w5WGI75VzDnnfbcoKiNQygbgJbJ2JFRm2hbWlYuza3cBwofMa1HEzGri33ulpulznmzrca2ifnmrYMbrl39KxZAPUT2DoSKzBqCW1rhbVLcwvJgSLovlaDydh15V5Tq20yx9z5VVvbzLm+NcLbnLddWqMA2iGwdSRWXPQc2C7NLS4HCqPPWgwoY9eUez0ttkeuuXOptTGSIzfALfnZOGsSQFsEts7EiovSQ9vWYe3a4iKs80KppZByy7W01A5zzJ03LbXH0jVjTb2vPwCt8imRnYlt6LFAVIK9w1oQjr3k+KFoK6FwYxs5Y6K3sDaM+Tnjfun8KtmR19RiewLwhcDGT0oMbUef01AIzS2GlhSxLYi1U23t0Fu/LTF3fC+dS7XZ8xp7aE8A7vZcb4ns01ShdfRbJKeC2pFFyi3FfC/F1Vgb1XTtS88/NjZa6ve547+XMZ9yy5pxTXsC9Mcdtk5NbfpH3tkqOawF4fWHx1yhcFuzeKtJr9fdirljd+kcadHlmjGnXZY+D4C2uMPWuZwCbK+7bTkhseSCZWkgabUIi7VH6de79Lxrvd4pc8d1q+MZAI4isPFJTlG2VXDLvZtXSyE4t8AdtFbo1hpgxs4755yXPq9ES8Zwa+MXAEohsPGT3CJtreDWWlAbs6TwDVopfmPXX+r1LT3f2q4zZu54bWWcAkDJBDbuWRIwcgPckp+La6kgXNK2Qe1tsPS6B3te/9i5Tr1+7Ppq6re5fdTSvASA0glsjLq1yL5VywXh0rattU22GEtbtMXS4LX0eSWY2zctz0sAKJVPiWRUKMyOKs5aLwqHtp17naG4Hh412PJc92qLqT6KvX7pY3hu2y0ZrwDAOgQ2kvYs0nosCpde8x5hZam9z620tih1DA/tNKetepyTAFAagY1JQ9G2RfG21XFrs7QdlhThWzn6PG55/SXPO/Ja55jbLkvHIgCwDT/Dxs3mFoPkWxoK9mznVYLLn/95/mLEb39+/mKeOW0wdg2p58euuaTxPbdfzE0AKJPABhUoNbjNOq8hlI0FsFRguzYjwOVc/5LwVXJgE9QAoC0CG1SmlPCWdR5jQezWwHYpI7xNXffYddQY1gQ1AGiTwAaVWhrcgluK9cVB7dJl0Foa1i7dENzWCGxHhZ8lY0BQA4C6CGzQgKXhbW7xPvk6a4SvW0wEt+vrXSOsBXuHoLn9LaQBQL18SiQ0IBTkw2OOUPgPj5sdHdaCiXO45Tpjz90zDM3tqyVjAgAoi8AGjVlapE+FgWRQKCGsDTLPZU7wOdpU31wT1ACgHd4SCR1YGk6Goj/6/JKC2pjIWyTDdY1dUyzkxK5/61A0t9+ENABojzts0IFQyA+POUJgqDasBZFznBuExmwVjoY2n3OOS/oWAKiDwAadUdyPi7XJGuEux9KQpi8BoG0CG3TqpoK/hrtrgxvONRag1gxJS4MaANAHgQ1oPwQUGDAFNQAgh8AG/KTpUJAIbWPXvNXdNUENAJhDYAPmqentkCu7JTgJagDAEgIbcE80VISg1mhYu77mOcEqZQhpuccbQpqgBgAMBDZgWkd31WLhak6ImhPSAiENAIgR2IB+bBw8BTUAYG0CG/CTOWGjWpHQlgpbU6FKUAMAtiKwAWktvh1yxjWlgpWgBgBsTWADmGEIablBbQhpghoAsITABhBxGbLcTQMAjiCwAX3KfFukoAYAHElgAz4ZDSUt/vzaDIIaAHA0gQ3gBoIaALAlgQ1gpiGkCWoAwNYENqBPv/35+Yt8QhoAsDeBDejPzLAmqAEARxHYgL7MCGuCGgBwNIEN4IqgBgCUQmADuCCoAQAlEdiAT0aDyoIP5ihe579bDgCoi8AG9CcS2txdAwBKI7ABAAAUSmADAAAolMAGpLX4c2wtXhMA0CSBDfhJzz/D5efXAIASCWwAAACFEtiAaS29hdDbIQGAighswD3Rtwa2EHQi1+DtkABAqQQ2AACAQglswANN3mVzdw0AqJDABoxqKrS18HZOAKBLAhvQLXfXAIDSCWxAVBN32bwVEgComMAGJFUd2oQ1AKByjz58+PDx8ePH5z8CPPTo0aPzVxF//uf5i0JMhEmBDQCohTtswKTJgFPS3TZhDQBoiMAGZKkitAlrAEBjvCUSmGXy7ZHB3m+RzAiLwhoAUCOBDVikiOAmqAEAjRPYgMWyQttgrfA2462XwhoAUDuBDbjJrNA2mBveFvx8nLAGALRAYANWsSi4bUBQAwBa4lMigVWEoHRkWDr69QEAtiCwAavaOzgJagBAywQ2YBNDkNoiTG15bACAkghswOYuA9aSkHXr8wEAaiWwAbu7DmBTDwCAXglsAAAAhRLYAAAACiWwAQAAFEpgAwAAKJTABgAAUCiBDQAAoFACGwAAQKEENgAAgEIJbAAAAIUS2AAAAAolsAEAABRKYAMAACiUwAYAAFAogQ0AAKBQAhsAAEChBDYAAIBCCWwAAACFEtgAAAAKJbABAAAUSmADAAAolMAGAABQKIENAACgUAIbAABAoQQ2AACAQglsAAAAhRLYAAAACiWwAQAAFEpgAwAAKJTABgAAUCiBDQAAoFACGwAAQKEENgAAgEIJbAAAAIUS2AAAAAolsAEAABRKYAMAACiUwAYAAFAogQ0AAKBIp9P/DwEHcXfLi1cMAAAAAElFTkSuQmCC)"
      ],
      "metadata": {
        "id": "DBMUSY9GP5t7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The network has 4 layers. An input layer, two hidden layers and an output layer. We will use ReLu activation for the first three layers and then sigmoid activation for the output neuron."
      ],
      "metadata": {
        "id": "WdtW7WFgUTT9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that we will refer to the layer neurons by colours."
      ],
      "metadata": {
        "id": "V3Pj1IjIVnCY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Weights and Biases "
      ],
      "metadata": {
        "id": "XEp_2vkQUOKB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Weights and Biases are what the model gets updated to train. They are integral to the functioning of the neural network.\n",
        "\n",
        "A **weight** is what decides the prominence of a particular edge of the network. The edges are shown as the black lines in the diagram above.\n",
        "\n",
        "A **bias** is an inherent property of a particular neuron. Biases are not shown for the neurons in the diagram but are understood to be present. "
      ],
      "metadata": {
        "id": "3dgQRd8TUyNQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# lets use random weights and biases \n",
        "\n",
        "blue_weights = torch.rand((3,2), requires_grad = True)\n",
        "orange_weights = torch.rand((3,3), requires_grad = True)\n",
        "yellow_weights = torch.rand((1,3), requires_grad = True)\n"
      ],
      "metadata": {
        "id": "Wx1VF8jtVbHW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we name the weights tensors as per the layer from which the edges **terminate**. \n",
        "\n",
        "The command torch.rand((x,y), requires_grad=\"True/False\") creates a tensor with random values of size (x,y) and keep track of whether or not to include it in gradient calculation. The gradient will be used for gradient descent. "
      ],
      "metadata": {
        "id": "girqLckYXCqF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# now for the biases\n",
        "\n",
        "blue_bias = torch.rand((1,3), requires_grad = True)\n",
        "orange_bias = torch.rand((1,3), requires_grad = True)\n",
        "yellow_bias = torch.rand((1,1), requires_grad = True)\n",
        "\n",
        "# Let us also define the inputs\n",
        "\n",
        "input = torch.rand((1,2), requires_grad = False)"
      ],
      "metadata": {
        "id": "6oEQ4xr7XwGb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Biases are labelled as per the layer they apply to."
      ],
      "metadata": {
        "id": "-P-0TvtIYnce"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Forward Pass"
      ],
      "metadata": {
        "id": "1LVbqnlHYwZ5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is the part of the network where the output value is calculated. This step follows matrix multiplication. \n",
        "\n",
        "The output of any neuron can be understood to be given by:\n",
        "\n",
        "$ input \\times weight + bias $\n",
        "\n",
        "since we are using tensors to store inputs, weights and biases, we see that we need not repeat the multiplication sepertely for each neuron as matrix multiplication generates a matrix of corresponding outputs. \n",
        "\n",
        "Further the output of one intermediate layer becomes input for the subsequent layer. So we will define a function for this purpose to enhance reusability."
      ],
      "metadata": {
        "id": "ecwy95XUY0r7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def forward(input, weights, biases):\n",
        "  output = input@weights.T + biases\n",
        "  output = torch.relu(output)\n",
        "  return output"
      ],
      "metadata": {
        "id": "dPPLIDN1bLvo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since we used relu activation we cannot use this function for the output layer. We will code that seperately."
      ],
      "metadata": {
        "id": "0ySbyj6gbrRu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "blue_forward = forward(input, blue_weights, blue_bias)\n",
        "orange_forward = forward(blue_forward, orange_weights, orange_bias)\n",
        "\n",
        "# now for the yellow layer's output\n",
        "\n",
        "yellow_forward = orange_forward@yellow_weights.T + yellow_bias\n",
        "yellow_forward = torch.sigmoid(yellow_forward)"
      ],
      "metadata": {
        "id": "tUlidSaSsLCI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# True Values "
      ],
      "metadata": {
        "id": "8RI9kgzwqwF5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now lets define what the true output for each layer should be. This is necessary to find the loss/cost value."
      ],
      "metadata": {
        "id": "UtEXp5HGqyYQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "blue_output_true = torch.rand((1,3), requires_grad = False)\n",
        "orange_output_true = torch.rand((1,3), requires_grad = False)\n",
        "yellow_output_true = torch.rand((1,1), requires_grad = False)"
      ],
      "metadata": {
        "id": "uFxoArmnq-U4"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we set requires_grad = False as we do not want the gradient to be included in out gradient descent.\n",
        "\n",
        "Also, the use of torch.rand means that the loss value will change if we re-run this code. This is however not an issue as we want to observe only the training which will reflect as a drop in loss value."
      ],
      "metadata": {
        "id": "Xx6l-EJIrbzs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Loss "
      ],
      "metadata": {
        "id": "-ZWpCezXrvcR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Broadly loss functions can be classified into two categories:\n",
        "\n",
        "\n",
        "\n",
        "*   Regression and Classification losses,  $MSE = \\frac{\\sum_{i=1}^n (y_i-y_{true})^2}{N}$\n",
        "*   Cross Entropy loss, $-(y_ilog(y_{true}) + (1-y_i)log(1-y_{true}))$\n",
        "\n"
      ],
      "metadata": {
        "id": "OtIePWIH25M6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use mean-squared error loss. This is defined as: \n",
        "\n",
        "$ (true_{value} - predicted_{value})^2 $ "
      ],
      "metadata": {
        "id": "OQ8k7JWor1ko"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "blue_loss = torch.mean((blue_forward-blue_output_true)**2)\n",
        "orange_loss = torch.mean((orange_forward-orange_output_true)**2)\n",
        "yellow_loss = torch.mean((yellow_forward-yellow_output_true)**2)"
      ],
      "metadata": {
        "id": "veF8-ePqtJ5a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# torch.grad"
      ],
      "metadata": {
        "id": "hUrXFeXQh3tm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets now take a look at the function that allows us to perform gradient descent. This is very useful function and we will attempt to understand it using an example"
      ],
      "metadata": {
        "id": "cmfyTFXJh8ih"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# lets take the polynomial y = x^2 + 3x + 5\n",
        "\n",
        "x = torch.rand(1, requires_grad = True)\n",
        "print(x)\n",
        "\n",
        "y = x**2 + 3*x + 5\n",
        "y.backward()\n",
        "print(x.grad) # finds dy/dx \n"
      ],
      "metadata": {
        "id": "WCqyPIiHiL6L"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We see that the differentiation of y gives 2x +3 and for the x value in the tensor that is generated randomly the x.grad gives $\\frac{dy}{dx}$ at that x value. \n",
        "\n",
        "y.backward() starts the gradient calculation process by telling pytorch to run back through all the tensors with requires_grad = True for y and x.grad tells pytorch to find $\\frac{dy}{dx}$."
      ],
      "metadata": {
        "id": "Zr3_Mos2kaqf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Back Propogation"
      ],
      "metadata": {
        "id": "XBYz5zn6tyJS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we will get into the part of the code that trains the model. The back propogation code:"
      ],
      "metadata": {
        "id": "arbaf0RubzZX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Back propogation (gradient calculation and updation of connections):\n",
        "# The learning rate is fixed at 0.01\n",
        "\n",
        "yellow_loss.backward(retain_graph=True)\n",
        "with torch.no_grad():\n",
        "  yellow_weights = yellow_weights - 0.01*yellow_weights.grad\n",
        "  yellow_bias = yellow_bias - 0.01*yellow_bias.grad\n",
        "  orange_weights = orange_weights - 0.01*orange_weights.grad\n",
        "  orange_bias = orange_bias - 0.01*orange_bias.grad\n",
        "  blue_weights = blue_weights - 0.01*blue_weights.grad\n",
        "  blue_bias = blue_bias - 0.01*blue_bias.grad\n"
      ],
      "metadata": {
        "id": "LIA43SxGuC--"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here the learning rate is the updation number. tensor.backward() is the pytorch command that initiates backpropogation. tensor.grad accesses the backpropogation tree for the specific gradient for that tensor."
      ],
      "metadata": {
        "id": "GislLudjup8k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# New Output"
      ],
      "metadata": {
        "id": "EkItSlv3u76s"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we will run forward pass again and see the drop in the loss value. We will consider the output layer's loss as generally this is the loss that means the most in applications."
      ],
      "metadata": {
        "id": "UARZhbvHu-5g"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Earlier loss = \", yellow_loss.item())\n",
        "\n",
        "# New forward pass : \n",
        "\n",
        "blue_forward = forward(input, blue_weights, blue_bias)\n",
        "orange_forward = forward(blue_forward, orange_weights, orange_bias)\n",
        "\n",
        "# now for the yellow layer's output\n",
        "\n",
        "yellow_forward = orange_forward@yellow_weights.T + yellow_bias\n",
        "yellow_forward = torch.sigmoid(yellow_forward)\n",
        "\n",
        "# New loss values : \n",
        "\n",
        "blue_loss = torch.mean((blue_forward-blue_output_true)**2)\n",
        "orange_loss = torch.mean((orange_forward-orange_output_true)**2)\n",
        "yellow_loss = torch.mean((yellow_forward-yellow_output_true)**2)\n",
        "\n",
        "print(\"New loss = \", yellow_loss.item())"
      ],
      "metadata": {
        "id": "d1yw0jxbvDzC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We see the loss value decreased... So it works !!"
      ],
      "metadata": {
        "id": "WfuV1khpwvp0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Pytorch Linear Layers (Further Reference)"
      ],
      "metadata": {
        "id": "wwxGNL-rw1XN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Coding from scratch is not always feasible when too many layers are involved. For this purpose pytorch offers functions to define linear layers of neurons in a single line. "
      ],
      "metadata": {
        "id": "4uyMVcxIw5Us"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# lets define the blue layer using nn.linear\n",
        "\n",
        "blue_layer = torch.nn.Linear(2, 3, bias = True)\n",
        "\n",
        "# The above line is all it took to define the layer\n",
        "\n",
        "input_tensor = torch.rand(2)\n",
        "output_tensor = blue_layer(input_tensor)\n",
        "print(output_tensor.size())"
      ],
      "metadata": {
        "id": "yGcEuCyPxGPo"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}