Dealing_with_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dealing with data\n",
    "\n",
    "- Passing data around\n",
    "  - recommendation: do NOT use pickle (Python) or RDS (R)\n",
    "    - can only be accessed with Python/R\n",
    "    - neither is backwards compatible by default\n",
    "    - cannot be diffed with git not viewed on GitHub\n",
    "  - do use plain tabular format (CSV/TSV) or any domain-specific format (e.g. VCF/FASTA/PDB):\n",
    "    - can be opened in any relevant software\n",
    "    - easy to track changes\n",
    "- Versioning data\n",
    "   - do not store data in the same git repository\n",
    "     - large files make git difficult to work with after some time\n",
    "     - you may want to keep data more privat than the analyses/summaries\n",
    "   - have a separate repository for data\n",
    "     - do not push it to external service provider (GitHub/GitLab etc)\n",
    "     - do push it to a dedicated server within your organization\n",
    "   - or, use artifact versioning\n",
    "     - [quilt](https://github.com/quiltdata/quilt) - versioned data portal for AWS, petabyte-scale, notebook-oriented\n",
    "     - [scrapbook](https://github.com/nteract/scrapbook) - records the cell outputs (graphics, tables) separately from the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dealing with patients data\n",
    "\n",
    "- You most likely work with the anonymized data to begin with\n",
    "  - you (of course) keep the data locally, probably password-protected and outside of the repository (/have a gitignore file), \n",
    "  - and you do not print out entire dataset at once, just to be certain\n",
    "- But would it be ok if the data became public at any point without proper vetting?\n",
    "  - anything you push to an online repository is at a higher risk, even if it's private\n",
    "- Case study: a patient from minority background + recruited at the other, smaller hospital you work with + after six pregnancies.\n",
    "  - could this patient flag up as outlier? If yes, why?\n",
    "     - possibly for any of the three characteristics, let alone the sum of them\n",
    "  - would this patient be easier to identify than others?\n",
    "     - how many persons with such characteristics **and** the disease you study live there?\n",
    "  - if both answers are yes, you may want to put an extra effort to avoid revealing any additional information, e.g. which metabolites they had up/down"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Mitigation strategies\n",
    "- Proper inspection of diffs is one way to control what is being committed to your repository - even annonymized data should be taken a with a great care.\n",
    "- If you want to check for any accidental change in the data without printing it out (or adding to the version control) it might be better to use a checksum as demonstrated with MD5 example\\*\n",
    "\n",
    "\\*) while cheksums are not unique, to get one collision you would need to generate random data in a rate of [\"6 billion per second for 100 years\"](https://stackoverflow.com/a/288519). It might be that there are more pressing issues to worry about in such time scales."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}