data_processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess data needed for LangChain and Retrieval Augment Generation & Question Answering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4532\n",
      "{'title': 'https://www.webmd.com/digestive-disorders/digestive-diseases-gastritis', 'paragraphs': [{'qas': [{'url': 'https://www.webmd.com/digestive-disorders/qa/what-are-the-symptoms-of-gastritis', 'id': '46a31525fae6a6b203c9be1e0eb7017d', 'answers': [{'text': 'However, the most common symptoms include: Nausea or recurrent upset stomach Abdominal bloating Abdominal pain Vomiting Indigestion Burning or gnawing feeling in the stomach between meals or at night Hiccups Loss of appetite Vomiting blood or coffee ground-like material Black, tarry stools To diagnose gastritis, your doctor will review your personal and family medical history, perform a thorough physical evaluation, and may recommend any of the following tests: Upper endoscopy.', 'answer_span': [7], 'answer_start': 1688, 'answer_starts': [[1688, 482]]}], 'question': 'What are the symptoms of gastritis?', 'is_impossible': False}, {'url': 'https://www.webmd.com/digestive-disorders/qa/what-does-the-treatment-for-gastritis-involve', 'id': '75f972ba8b72bcb5b2bd901bd7006767', 'answers': [{'text': 'Treatment for gastritis usually involves: Taking antacids and other drugs (such as proton pump inhibitors or H-2 blockers) to reduce stomach acid Avoiding hot and spicy foods For gastritis caused by H. pylori infection, your doctor will prescribe a regimen of several antibiotics plus an acid blocking drug (used for heartburn) If the gastritis is caused by pernicious anemia, B12 vitamin shots will be given. Eliminating irritating foods from your diet such as lactose from dairy or gluten from wheat Once the underlying problem disappears, the gastritis usually does, too. You should talk to your doctor before stopping any medicine or starting any gastritis treatment on your own.', 'answer_span': [15, 16, 17], 'answer_start': 2888, 'answer_starts': [[2888, 409], [3298, 164], [3463, 108]]}], 'question': 'What does the treatment for gastritis involve?', 'is_impossible': False}, {'url': 'https://www.webmd.com/digestive-disorders/qa/how-does-an-upper-endoscopy-help-diagnose-gastritis', 'id': '9871bb970b4c59aba85aecad486798f1', 'answers': [{'text': 'However, the most common symptoms include: Nausea or recurrent upset stomach Abdominal bloating Abdominal pain Vomiting Indigestion Burning or gnawing feeling in the stomach between meals or at night Hiccups Loss of appetite Vomiting blood or coffee ground-like material Black, tarry stools To diagnose gastritis, your doctor will review your personal and family medical history, perform a thorough physical evaluation, and may recommend any of the following tests: Upper endoscopy. An endoscope, a thin tube containing a tiny camera, is inserted through your mouth and down into your stomach to look at the stomach lining. The doctor will check for inflammation and may perform a biopsy, a procedure in which a tiny sample of tissue is removed and then sent to a laboratory for analysis.', 'answer_span': [7, 8, 9], 'answer_start': 1688, 'answer_starts': [[1688, 482], [2171, 140], [2312, 164]]}], 'question': 'How does an upper endoscopy help diagnose gastritis?', 'is_impossible': False}, {'url': 'https://www.webmd.com/digestive-disorders/qa/how-is-a-fecal-occult-blood-test-stool-test-used-to-diagnose-gastritis', 'id': 'de443e4a4712a0b6e2cc555cdb51afdc', 'answers': [{'text': 'This test checks for the presence of blood in your stool, a possible sign of gastritis.', 'answer_span': [14], 'answer_start': 2800, 'answer_starts': [[2800, 87]]}], 'question': 'How is a fecal occult blood test (stool test) used to diagnose gastritis?', 'is_impossible': False}, {'url': 'https://www.webmd.com/digestive-disorders/qa/what-is-gastritis', 'id': 'a141a0d225d94a318f264aac7b6c6832', 'answers': [{'text': \"!s_sensitive, chron ID: $('article embeded_module[type=video][align=top]:eq(0)').attr('chronic_id'), continuous Play: true, cp Options: { flyout: true }, display Ads: true, mode: 'in-article', sticky: true }) }); }); } else { $(function(){ $('.responsive-video-container').remove(); }); } Gastritis is an inflammation, irritation, or erosion of the lining of the stomach. It can occur suddenly (acute) or gradually (chronic).\", 'answer_span': [1, 2], 'answer_start': 485, 'answer_starts': [[485, 371], [857, 53]]}], 'question': 'What is gastritis?', 'is_impossible': False}, {'url': 'https://www.webmd.com/digestive-disorders/qa/what-causes-gastritis', 'id': 'c487f7f0d7c83b3f0b5f0dd6a4dd779b', 'answers': [{'text': 'It may also be caused by any of the following: Helicobacter pylori ( H. pylori): A bacteria that lives in the mucous lining of the stomach; without treatment, the infection can lead to ulcers, and in some people, stomach cancer. Bile reflux: A backflow of bile into the stomach from the bile tract (that connects to the liver and gallbladder) Infections caused by bacteria and viruses If gastritis is left untreated, it can lead to a severe loss of blood and may increase the risk of developing stomach cancer. Treatment for gastritis usually involves: Taking antacids and other drugs (such as proton pump inhibitors or H-2 blockers) to reduce stomach acid Avoiding hot and spicy foods For gastritis caused by H. pylori infection, your doctor will prescribe a regimen of several antibiotics plus an acid blocking drug (used for heartburn) If the gastritis is caused by pernicious anemia, B12 vitamin shots will be given.', 'answer_span': [4, 5, 15], 'answer_start': None, 'answer_starts': [[1089, 228], [1318, 281], [2888, 409]]}], 'question': 'What causes gastritis?', 'is_impossible': False}], 'sent_list': [\"var s_context; s_context= s_context || {}; s_context['wb.modimp'] = 'vidfloat'; if(webmd.useragent && webmd.useragent.ua.type === 'desktop'){ webmd.ads2.disable Initial Load(); webmd.ads2.disable Ads Init = true; $(function() { webmd.p.pim.increment(); $('.responsive-video-container').insert After('.module-social-share-container'); require(['video2/1/responsive-player/video-loader'], function(video Loader) { video Loader.init({ autoplay: webmd.useragent.ua.type === 'desktop' && !\", \"!s_sensitive, chron ID: $('article embeded_module[type=video][align=top]:eq(0)').attr('chronic_id'), continuous Play: true, cp Options: { flyout: true }, display Ads: true, mode: 'in-article', sticky: true }) }); }); } else { $(function(){ $('.responsive-video-container').remove(); }); } Gastritis is an inflammation, irritation, or erosion of the lining of the stomach.\", 'It can occur suddenly (acute) or gradually (chronic).', 'Gastritis can be caused by irritation due to excessive alcohol use, chronic vomiting, stress, or the use of certain medications such as aspirin or other anti-inflammatory drugs.', 'It may also be caused by any of the following: Helicobacter pylori ( H. pylori): A bacteria that lives in the mucous lining of the stomach; without treatment, the infection can lead to ulcers, and in some people, stomach cancer.', 'Bile reflux: A backflow of bile into the stomach from the bile tract (that connects to the liver and gallbladder) Infections caused by bacteria and viruses If gastritis is left untreated, it can lead to a severe loss of blood and may increase the risk of developing stomach cancer.', 'Symptoms of gastritis vary among individuals, and in many people there are no symptoms.', 'However, the most common symptoms include: Nausea or recurrent upset stomach Abdominal bloating Abdominal pain Vomiting Indigestion Burning or gnawing feeling in the stomach between meals or at night Hiccups Loss of appetite Vomiting blood or coffee ground-like material Black, tarry stools To diagnose gastritis, your doctor will review your personal and family medical history, perform a thorough physical evaluation, and may recommend any of the following tests: Upper endoscopy.', 'An endoscope, a thin tube containing a tiny camera, is inserted through your mouth and down into your stomach to look at the stomach lining.', 'The doctor will check for inflammation and may perform a biopsy, a procedure in which a tiny sample of tissue is removed and then sent to a laboratory for analysis.', 'Blood tests.', 'The doctor may perform various blood tests, such as checking your red blood cell count to determine whether you have anemia, which means that you do not have enough red blood cells.', 'He or she can also screen for H. pylori infection and pernicious anemia with blood tests.', 'Fecal occult blood test (stool test).', 'This test checks for the presence of blood in your stool, a possible sign of gastritis.', 'Treatment for gastritis usually involves: Taking antacids and other drugs (such as proton pump inhibitors or H-2 blockers) to reduce stomach acid Avoiding hot and spicy foods For gastritis caused by H. pylori infection, your doctor will prescribe a regimen of several antibiotics plus an acid blocking drug (used for heartburn) If the gastritis is caused by pernicious anemia, B12 vitamin shots will be given.', 'Eliminating irritating foods from your diet such as lactose from dairy or gluten from wheat Once the underlying problem disappears, the gastritis usually does, too.', 'You should talk to your doctor before stopping any medicine or starting any gastritis treatment on your own.', 'Most people with gastritis improve quickly once treatment has begun.'], 'context': \"var s_context; s_context= s_context || {}; s_context['wb.modimp'] = 'vidfloat'; if(webmd.useragent && webmd.useragent.ua.type === 'desktop'){ webmd.ads2.disable Initial Load(); webmd.ads2.disable Ads Init = true; $(function() { webmd.p.pim.increment(); $('.responsive-video-container').insert After('.module-social-share-container'); require(['video2/1/responsive-player/video-loader'], function(video Loader) { video Loader.init({ autoplay: webmd.useragent.ua.type === 'desktop' && ! !s_sensitive, chron ID: $('article embeded_module[type=video][align=top]:eq(0)').attr('chronic_id'), continuous Play: true, cp Options: { flyout: true }, display Ads: true, mode: 'in-article', sticky: true }) }); }); } else { $(function(){ $('.responsive-video-container').remove(); }); } Gastritis is an inflammation, irritation, or erosion of the lining of the stomach. It can occur suddenly (acute) or gradually (chronic). Gastritis can be caused by irritation due to excessive alcohol use, chronic vomiting, stress, or the use of certain medications such as aspirin or other anti-inflammatory drugs. It may also be caused by any of the following: Helicobacter pylori ( H. pylori): A bacteria that lives in the mucous lining of the stomach; without treatment, the infection can lead to ulcers, and in some people, stomach cancer. Bile reflux: A backflow of bile into the stomach from the bile tract (that connects to the liver and gallbladder) Infections caused by bacteria and viruses If gastritis is left untreated, it can lead to a severe loss of blood and may increase the risk of developing stomach cancer. Symptoms of gastritis vary among individuals, and in many people there are no symptoms. However, the most common symptoms include: Nausea or recurrent upset stomach Abdominal bloating Abdominal pain Vomiting Indigestion Burning or gnawing feeling in the stomach between meals or at night Hiccups Loss of appetite Vomiting blood or coffee ground-like material Black, tarry stools To diagnose gastritis, your doctor will review your personal and family medical history, perform a thorough physical evaluation, and may recommend any of the following tests: Upper endoscopy. An endoscope, a thin tube containing a tiny camera, is inserted through your mouth and down into your stomach to look at the stomach lining. The doctor will check for inflammation and may perform a biopsy, a procedure in which a tiny sample of tissue is removed and then sent to a laboratory for analysis. Blood tests. The doctor may perform various blood tests, such as checking your red blood cell count to determine whether you have anemia, which means that you do not have enough red blood cells. He or she can also screen for H. pylori infection and pernicious anemia with blood tests. Fecal occult blood test (stool test). This test checks for the presence of blood in your stool, a possible sign of gastritis. Treatment for gastritis usually involves: Taking antacids and other drugs (such as proton pump inhibitors or H-2 blockers) to reduce stomach acid Avoiding hot and spicy foods For gastritis caused by H. pylori infection, your doctor will prescribe a regimen of several antibiotics plus an acid blocking drug (used for heartburn) If the gastritis is caused by pernicious anemia, B12 vitamin shots will be given. Eliminating irritating foods from your diet such as lactose from dairy or gluten from wheat Once the underlying problem disappears, the gastritis usually does, too. You should talk to your doctor before stopping any medicine or starting any gastritis treatment on your own. Most people with gastritis improve quickly once treatment has begun.\", 'sent_starts': [[0, 484], [485, 371], [857, 53], [911, 177], [1089, 228], [1318, 281], [1600, 87], [1688, 482], [2171, 140], [2312, 164], [2477, 12], [2490, 181], [2672, 89], [2762, 37], [2800, 87], [2888, 409], [3298, 164], [3463, 108], [3572, 68]]}]}\n",
      "var s_context; s_context= s_context || {}; s_context['wb.modimp'] = 'vidfloat'; if(webmd.useragent && webmd.useragent.ua.type === 'desktop'){ webmd.ads2.disable Initial Load(); webmd.ads2.disable Ads Init = true; $(function() { webmd.p.pim.increment(); $('.responsive-video-container').insert After('.module-social-share-container'); require(['video2/1/responsive-player/video-loader'], function(video Loader) { video Loader.init({ autoplay: webmd.useragent.ua.type === 'desktop' && ! !s_sensitive, chron ID: $('article embeded_module[type=video][align=top]:eq(0)').attr('chronic_id'), continuous Play: true, cp Options: { flyout: true }, display Ads: true, mode: 'in-article', sticky: true }) }); }); } else { $(function(){ $('.responsive-video-container').remove(); }); } Gastritis is an inflammation, irritation, or erosion of the lining of the stomach. It can occur suddenly (acute) or gradually (chronic). Gastritis can be caused by irritation due to excessive alcohol use, chronic vomiting, stress, or the use of certain medications such as aspirin or other anti-inflammatory drugs. It may also be caused by any of the following: Helicobacter pylori ( H. pylori): A bacteria that lives in the mucous lining of the stomach; without treatment, the infection can lead to ulcers, and in some people, stomach cancer. Bile reflux: A backflow of bile into the stomach from the bile tract (that connects to the liver and gallbladder) Infections caused by bacteria and viruses If gastritis is left untreated, it can lead to a severe loss of blood and may increase the risk of developing stomach cancer. Symptoms of gastritis vary among individuals, and in many people there are no symptoms. However, the most common symptoms include: Nausea or recurrent upset stomach Abdominal bloating Abdominal pain Vomiting Indigestion Burning or gnawing feeling in the stomach between meals or at night Hiccups Loss of appetite Vomiting blood or coffee ground-like material Black, tarry stools To diagnose gastritis, your doctor will review your personal and family medical history, perform a thorough physical evaluation, and may recommend any of the following tests: Upper endoscopy. An endoscope, a thin tube containing a tiny camera, is inserted through your mouth and down into your stomach to look at the stomach lining. The doctor will check for inflammation and may perform a biopsy, a procedure in which a tiny sample of tissue is removed and then sent to a laboratory for analysis. Blood tests. The doctor may perform various blood tests, such as checking your red blood cell count to determine whether you have anemia, which means that you do not have enough red blood cells. He or she can also screen for H. pylori infection and pernicious anemia with blood tests. Fecal occult blood test (stool test). This test checks for the presence of blood in your stool, a possible sign of gastritis. Treatment for gastritis usually involves: Taking antacids and other drugs (such as proton pump inhibitors or H-2 blockers) to reduce stomach acid Avoiding hot and spicy foods For gastritis caused by H. pylori infection, your doctor will prescribe a regimen of several antibiotics plus an acid blocking drug (used for heartburn) If the gastritis is caused by pernicious anemia, B12 vitamin shots will be given. Eliminating irritating foods from your diet such as lactose from dairy or gluten from wheat Once the underlying problem disappears, the gastritis usually does, too. You should talk to your doctor before stopping any medicine or starting any gastritis treatment on your own. Most people with gastritis improve quickly once treatment has begun.\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import pandas as pd\n",
    "\n",
    "json_data = json.load(open('merged_file.json'))\n",
    "print(len(json_data['data']))\n",
    "print(json_data['data'][0])\n",
    "print(json_data['data'][0]['paragraphs'][0]['context'])\n",
    "new_d = []\n",
    "for i in range(len(json_data['data'])):\n",
    "  each_d = {}\n",
    "  each_d['context'] = json_data['data'][i]['paragraphs'][0]['context']\n",
    "  new_d.append(each_d)\n",
    "df = pd.DataFrame(new_d)\n",
    "# Save the DataFrame to a CSV file\n",
    "df.to_csv('merged_file.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home2/kolubex/.miniconda3/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from operator import itemgetter\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.embeddings import OpenAIEmbeddings\n",
    "from langchain.schema.output_parser import StrOutputParser\n",
    "from langchain.schema.runnable import RunnablePassthrough, RunnableLambda\n",
    "from langchain.vectorstores import FAISS\n",
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use langchain CSV loader to load the CSV file and use recursive character text splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use langchain to load csv file\n",
    "from langchain.document_loaders.csv_loader import CSVLoader\n",
    "\n",
    "loader = CSVLoader(file_path='./data_val_consec.csv', source_column=\"context\")\n",
    "\n",
    "data = loader.load()\n",
    "\n",
    "# Split documents\n",
    "\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size = 500, chunk_overlap = 0)\n",
    "splits = text_splitter.split_documents(loader.load())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Embed and store splits\n",
    "from langchain.vectorstores import Chroma\n",
    "from langchain.embeddings import OpenAIEmbeddings, HuggingFaceEmbeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No sentence-transformers model found with name /home2/kolubex/.cache/torch/sentence_transformers/alibidaran_medical_transcription_generator. Creating a new one with MEAN pooling.\n"
     ]
    }
   ],
   "source": [
    "#embeddings_model_name = \"sentence-transformers/all-MiniLM-L6-v2\"\n",
    "embeddings_model_name = \"alibidaran/medical_transcription_generator\"\n",
    "embeddings = HuggingFaceEmbeddings(model_name=embeddings_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorstore = Chroma.from_documents(documents=splits,embedding=embeddings, persist_directory=\"./vectorstore_val_consec\")\n",
    "retriever = vectorstore.as_retriever()\n",
    "# Prompt\n",
    "# https://smith.langchain.com/hub/rlm/rag-prompt\n",
    "\n",
    "from langchain import hub\n",
    "rag_prompt = hub.pull(\"rlm/rag-prompt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorstore.persist()\n",
    "vectorstore = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorstore = Chroma(persist_directory=\"./vectorstore_train\", embedding_function=embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.llms import HuggingFacePipeline\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n",
    "from transformers import OpenAIGPTTokenizer, OpenAIGPTModel\n",
    "import torch\n",
    "from langchain.llms import HuggingFaceHub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.llms import HuggingFacePipeline\n",
    "import torch\n",
    "from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSeq2SeqLM\n",
    "\n",
    "model_id = 'google/flan-t5-small'\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(model_id, load_in_8bit=False, device_map='auto')\n",
    "\n",
    "pipeline = pipeline(\n",
    "    \"text2text-generation\",\n",
    "    model=model,\n",
    "    tokenizer=tokenizer,\n",
    "    max_length=128\n",
    ")\n",
    "\n",
    "hf = HuggingFacePipeline(pipeline=pipeline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = vectorstore.as_retriever()\n",
    "# Prompt\n",
    "# https://smith.langchain.com/hub/rlm/rag-prompt\n",
    "\n",
    "from langchain import hub\n",
    "rag_prompt = hub.pull(\"rlm/rag-prompt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    " # RAG chain\n",
    "\n",
    "from langchain.schema.runnable import RunnablePassthrough\n",
    "rag_chain = (\n",
    "    {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
    "    | rag_prompt\n",
    "    | hf\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "systolic heart failure\n",
      "7.006139039993286\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "s = time.time()\n",
    "ress = rag_chain.invoke(\"What are the symptoms of ischemic heart disease?\")\n",
    "print(ress)\n",
    "print(time.time()-s)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}