Skip to content
Permalink
master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Personalized-Medicine-Redefining-Cancer-Treatment-Challenge/MSK_Final.ipynb

Go to file
 
 
Cannot retrieve contributors at this time
627 lines (627 sloc) 33.4 KB
Raw Blame
Open in GitHub Desktop
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# In this notebook we present a solution to the [Personalized Medicine: Redefining Cancer Treatment](https://www.kaggle.com/c/msk-redefining-cancer-treatment) competition on Kaggle\n",
"\n",
"## ** The problem: ** \n",
"\n",
" #### <ol> We are provided with \"training\" data which contains many genes, variants, and so called classes, and along with this information we are given a text file, which usually appears to be a scientific paper. We are then tasked with building a classifier which can determine the class of a given gene, mutation and corresponding text file. The trouble is that we are not told what the classes mean so we have to determine a classifier which can determine a likelihood of what the classes mean by looking at the provided data associated with each class. We are also given \"test\" data which consists of a gene, variation, and text. The test data provided is quite different from the training data. \n",
" \n",
"\n",
"## **Our Approach:**\n",
"\n",
" #### <ol> We approach the problem using a random forest classifier. For a discussion on how this classifier works please see [classifier summary](https://github.uconn.edu/cow17005/Personalized-Medicine-Redefining-Cancer-Treatment-Challenge/blob/master/classifier.pdf)\n",
" \n",
"## **Results:**\n",
"\n",
" #### <ol> After submitting our solution to Kaggle (which scored the submission based on a subset of the data) we received a log-loss score of 5.085, which is only slightly worse than we received on our \"test\" data, where we knew the answer beforehand. The log-loss is calculated by taking the average of the negative of the sum of the log values of the predicted probability in the correct class. That means that predicting a high probability for the correct class will add only a small amount to the log-loss whereas predicting a low probability in the correct class will add a larger amount to the log-loss. For example if in row 5 (out of 968) one were to predict the probability of class 3 to be 0.2 then -log(0.2) = 1.609 would be added to log-loss, and 1 would be added to the denominator, so predicting a low probability in the correct class will add a bigger impact to log-loss. "
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"import pandas as pd\n",
"import numpy as np\n",
"import sklearn"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tokenization took: 2.14 ms\n",
"Type conversion took: 21.38 ms\n",
"Parser memory cleanup took: 4.68 ms\n"
]
}
],
"source": [
"variants_train=pd.read_csv('/Users/cory/Desktop/Kaggle/msk/training_variants.txt',index_col=0,verbose=True)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tokenization took: 0.12 ms\n",
"Type conversion took: 0.65 ms\n",
"Parser memory cleanup took: 0.00 ms\n"
]
}
],
"source": [
"variants_test= pd.read_csv('/Users/cory/Desktop/Kaggle/all/stage2_test_variants.csv',index_col=0,verbose=True)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Filled 5 NA values in column ID,Text\n"
]
}
],
"source": [
"text_train = pd.read_table('/Users/cory/Desktop/Kaggle/msk/training_text.txt',sep='\\|\\|',engine='python',verbose=True)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"text_train.columns=['Text']\n",
"text_test= pd.read_table('/Users/cory/Desktop/Kaggle/all/stage2_test_text.csv',sep='\\|\\|',engine='python',verbose=True)\n",
"text_test.index.names=['Id']\n",
"text_test.columns = ['Text']"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"text_test = text_test.replace(np.nan,'',regex=True)\n",
"text_train = text_train.replace(np.nan,'',regex=True)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"Train=variants_train.join(text_train)\n",
"Test=variants_test.join(text_test)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We take a glance at the training data and observe that there are 3321 genes in the data but only 264 of them are unique. In comparison, of the 3321 variations 2996 of them are unique. We also observe that of the 3321 text articles only 1921 are unique. "
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Gene</th>\n",
" <th>Variation</th>\n",
" <th>Class</th>\n",
" <th>Text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>3321</td>\n",
" <td>3321</td>\n",
" <td>3321.000000</td>\n",
" <td>3321</td>\n",
" </tr>\n",
" <tr>\n",
" <th>unique</th>\n",
" <td>264</td>\n",
" <td>2996</td>\n",
" <td>NaN</td>\n",
" <td>1921</td>\n",
" </tr>\n",
" <tr>\n",
" <th>top</th>\n",
" <td>BRCA1</td>\n",
" <td>Truncating Mutations</td>\n",
" <td>NaN</td>\n",
" <td>The PTEN (phosphatase and tensin homolog) phos...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>freq</th>\n",
" <td>264</td>\n",
" <td>93</td>\n",
" <td>NaN</td>\n",
" <td>53</td>\n",
" </tr>\n",
" <tr>\n",
" <th>mean</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>4.365854</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>std</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2.309781</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>min</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>1.000000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25%</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>2.000000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>50%</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>4.000000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>75%</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>7.000000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" <tr>\n",
" <th>max</th>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>9.000000</td>\n",
" <td>NaN</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Gene Variation Class \\\n",
"count 3321 3321 3321.000000 \n",
"unique 264 2996 NaN \n",
"top BRCA1 Truncating Mutations NaN \n",
"freq 264 93 NaN \n",
"mean NaN NaN 4.365854 \n",
"std NaN NaN 2.309781 \n",
"min NaN NaN 1.000000 \n",
"25% NaN NaN 2.000000 \n",
"50% NaN NaN 4.000000 \n",
"75% NaN NaN 7.000000 \n",
"max NaN NaN 9.000000 \n",
"\n",
" Text \n",
"count 3321 \n",
"unique 1921 \n",
"top The PTEN (phosphatase and tensin homolog) phos... \n",
"freq 53 \n",
"mean NaN \n",
"std NaN \n",
"min NaN \n",
"25% NaN \n",
"50% NaN \n",
"75% NaN \n",
"max NaN "
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Train.describe(include='all')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### To see how the classes are distributed we make a bar chart and observe that class 7 appears the most frequently while class 8 appears the least frequently. Further we see that classes 1, 2, and 4 also appear relatively frequently and classes 3, 5, 6, and 9 appear much less frequently. This could play a large role in our classifier predicting the classes which appear frequently more often."
]
},
{
"cell_type": "code",
"execution_count": 103,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/plain": [
"<BarContainer object of 9 artists>"
]
},
"execution_count": 103,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjQAAAGiCAYAAADwXFzAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X1Q1WX+//HXCfSILJwUF44n73DXvKPMVUdBS3dUrLypcSe3MNa7zEZSSS0j2683m1Cm5qxMFm5jlqn90WJtTSi1Zet6RyQl6mqtltSKNCse1BQUrt8f/jzTCZsCD/vhwudj5jPj+ZzrHN+fkeI5F+dwXMYYIwAAAItd5/QAAAAAV4ugAQAA1iNoAACA9QgaAABgPYIGAABYj6ABAADWI2gAAID1CBoAAGA9ggYAAFiPoAEAANarc9B89NFHGj16tHw+n1wulzZv3hx0vzFGCxculM/nU0REhIYMGaL9+/cHrSkvL1dqaqo8Ho88Ho9SU1N16tSpoDX79u3T4MGDFRERoRtuuEGLFy8Wn9IAAACupM5Bc/bsWfXq1UvZ2dlXvH/p0qVasWKFsrOzVVBQIK/Xq+HDh+v06dOBNSkpKSoqKlJeXp7y8vJUVFSk1NTUwP0VFRUaPny4fD6fCgoKtGrVKi1btkwrVqyoxyUCAIAmz1wFSSY3Nzdwu6amxni9XvP0008Hzp0/f954PB7zwgsvGGOMOXDggJFkdu3aFVizc+dOI8n861//MsYY8/zzzxuPx2POnz8fWJOVlWV8Pp+pqam5mpEBAEATFB7KODp69KhKS0uVnJwcOOd2uzV48GDt2LFD06ZN086dO+XxeNS/f//AmgEDBsjj8WjHjh3q2rWrdu7cqcGDB8vtdgfWjBgxQhkZGfryyy8VHx9f6++urKxUZWVl4HZNTY1OnjypmJgYuVyuUF4mAABoIMYYnT59Wj6fT9dd9/N/kBTSoCktLZUkxcXFBZ2Pi4vTV199FVgTGxtb67GxsbGBx5eWlqpTp061nuPyfVcKmqysLC1atOiqrwEAADivpKRE7dq1+9nrQxo0l/1wR8QYE3TuSjsmP7XG/P8XBP/YbktGRoZmz54duO33+9WhQweVlJQoOjq67hcBAAD+5yoqKtS+fXtFRUXV6XEhDRqv1yvp0i5K27ZtA+fLysoCOyxer1cnTpyo9dhvv/02aM3l3ZrvP4dUe/fnMrfbHfQjqsuio6MJGgAALFPXl4uE9PfQxMfHy+v1Kj8/P3CuqqpK27ZtU1JSkiQpMTFRfr9fe/bsCazZvXu3/H5/0JqPPvpIVVVVgTVbt26Vz+er9aMoAACAOgfNmTNnVFRUpKKiIkmXXghcVFSkY8eOyeVyKT09XZmZmcrNzVVxcbEmTpyoli1bKiUlRZLUvXt33X777Zo6dap27dqlXbt2aerUqRo1apS6du0q6dLbut1utyZOnKji4mLl5uYqMzNTs2fP5gW+AACgtrq+LeqDDz4wkmodEyZMMMZceuv2ggULjNfrNW6329x2221m3759Qc/x3//+14wfP95ERUWZqKgoM378eFNeXh605rPPPjO33nqrcbvdxuv1moULF9bpLdt+v99IMn6/v66XCAAAHFLf798uY5rmr9+tqKiQx+OR3+/nNTQAAFiivt+/+SwnAABgPYIGAABYj6ABAADWI2gAAID1CBoAAGA9ggYAAFiPoAEAANYjaAAAgPUIGgAAYD2CBgAAWC/c6QEAAM5xuRY5PcJPMmaB0yPAAuzQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsF/KguXjxop588knFx8crIiJCnTt31uLFi1VTUxNYY4zRwoUL5fP5FBERoSFDhmj//v1Bz1NeXq7U1FR5PB55PB6lpqbq1KlToR4XAAA0ASEPmmeeeUYvvPCCsrOzdfDgQS1dulTPPvusVq1aFVizdOlSrVixQtnZ2SooKJDX69Xw4cN1+vTpwJqUlBQVFRUpLy9PeXl5KioqUmpqaqjHBQAATUB4qJ9w586duuuuuzRy5EhJUqdOnbRx40Z9/PHHki7tzqxcuVLz58/X2LFjJUnr1q1TXFycNmzYoGnTpungwYPKy8vTrl271L9/f0nSmjVrlJiYqEOHDqlr166hHhsAAFgs5Ds0gwYN0vvvv6/Dhw9Lkj799FNt375dd955pyTp6NGjKi0tVXJycuAxbrdbgwcP1o4dOyRdiiKPxxOIGUkaMGCAPB5PYM0PVVZWqqKiIugAAADXhpDv0MybN09+v1/dunVTWFiYqqurtWTJEt13332SpNLSUklSXFxc0OPi4uL01VdfBdbExsbWeu7Y2NjA438oKytLixYtCuWlAAAAS4R8h+b111/X+vXrtWHDBn3yySdat26dli1bpnXr1gWtc7lcQbeNMUHnfnj/ldZ8X0ZGhvx+f+AoKSkJwdUAAAAbhHyH5tFHH9Xjjz+ue++9V5J000036auvvlJWVpYmTJggr9cr6dIuTNu2bQOPKysrC+zaeL1enThxotZzf/vtt7V2di5zu91yu92hvhwAAGCBkO/QfPfdd7ruuuCnDQsLC7xtOz4+Xl6vV/n5+YH7q6qqtG3bNiUlJUmSEhMT5ff7tWfPnsCa3bt3y+/3B9YAAABcFvIdmtGjR2vJkiXq0KGDevbsqb1792rFihWaPHmypEs/SkpPT1dmZqa6dOmiLl26KDMzUy1btlRKSookqXv37rr99ts1depUvfjii5KkBx98UKNGjeIdTgAAoJaQB82qVav0xz/+UdOnT1dZWZl8Pp+mTZum//u//wuseeyxx3Tu3DlNnz5d5eXl6t+/v7Zu3aqoqKjAmtdee00zZ84MvBtqzJgxys7ODvW4AACgCXAZY4zTQzSEiooKeTwe+f1+RUdHOz0OADRKLlfjf3eoMQucHgH/Q/X9/s1nOQEAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALBeuNMDALiyRa5FTo/wkxaYBU6PAACS2KEBAABNAEEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrNUjQfPPNN7r//vsVExOjli1b6pZbblFhYWHgfmOMFi5cKJ/Pp4iICA0ZMkT79+8Peo7y8nKlpqbK4/HI4/EoNTVVp06daohxAQCA5UIeNOXl5Ro4cKCaNWumd999VwcOHNDy5ct1/fXXB9YsXbpUK1asUHZ2tgoKCuT1ejV8+HCdPn06sCYlJUVFRUXKy8tTXl6eioqKlJqaGupxAQBAExAe6id85pln1L59e61duzZwrlOnToE/G2O0cuVKzZ8/X2PHjpUkrVu3TnFxcdqwYYOmTZumgwcPKi8vT7t27VL//v0lSWvWrFFiYqIOHTqkrl27hnpsAABgsZDv0Lz11lvq27ev7rnnHsXGxqp3795as2ZN4P6jR4+qtLRUycnJgXNut1uDBw/Wjh07JEk7d+6Ux+MJxIwkDRgwQB6PJ7DmhyorK1VRURF0AACAa0PIg+bIkSNavXq1unTpoi1btuihhx7SzJkz9corr0iSSktLJUlxcXFBj4uLiwvcV1paqtjY2FrPHRsbG1jzQ1lZWYHX23g8HrVv3z6UlwUAABqxkAdNTU2NfvOb3ygzM1O9e/fWtGnTNHXqVK1evTponcvlCrptjAk698P7r7Tm+zIyMuT3+wNHSUlJCK4GAADYIORB07ZtW/Xo0SPoXPfu3XXs2DFJktfrlaRaOy1lZWWBXRuv16sTJ07Ueu5vv/221s7OZW63W9HR0UEHAAC4NoQ8aAYOHKhDhw4FnTt8+LA6duwoSYqPj5fX61V+fn7g/qqqKm3btk1JSUmSpMTERPn9fu3ZsyewZvfu3fL7/YE1AAAAl4X8XU6PPPKIkpKSlJmZqXHjxmnPnj3KyclRTk6OpEs/SkpPT1dmZqa6dOmiLl26KDMzUy1btlRKSoqkSzs6t99+u6ZOnaoXX3xRkvTggw9q1KhRvMMJAADUEvKg6devn3Jzc5WRkaHFixcrPj5eK1eu1Pjx4wNrHnvsMZ07d07Tp09XeXm5+vfvr61btyoqKiqw5rXXXtPMmTMD74YaM2aMsrOzQz0uAABoAlzGGOP0EA2hoqJCHo9Hfr+f19PASotci5we4SctMAucHgFXyWXB15nh6+yaUt/v33yWEwAAsB5BAwAArEfQAAAA6xE0AADAeiF/l9O14Ed+WXGj0jRf6g0AwJWxQwMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA64U7PQAaAZfL6Ql+mjFOTwAAaMTYoQEAANYjaAAAgPUIGgAAYD2CBgAAWI+gAQAA1iNoAACA9QgaAABgPYIGAABYj6ABAADWI2gAAID1CBoAAGA9ggYAAFiPoAEAANYjaAAAgPUIGgAAYL0GD5qsrCy5XC6lp6cHzlVWVmrGjBlq06aNIiMjNWbMGH399ddBjzt27JhGjx6tyMhItWnTRjNnzlRVVVVDjwsAACzUoEFTUFCgnJwc3XzzzUHn09PTlZubq02bNmn79u06c+aMRo0aperqaklSdXW1Ro4cqbNnz2r79u3atGmT3njjDc2ZM6chxwUAAJZqsKA5c+aMxo8frzVr1qhVq1aB836/Xy+99JKWL1+uYcOGqXfv3lq/fr327dun9957T5K0detWHThwQOvXr1fv3r01bNgwLV++XGvWrFFFRUVDjQwAACzVYEGTlpamkSNHatiwYUHnCwsLdeHCBSUnJwfO+Xw+JSQkaMeOHZKknTt3KiEhQT6fL7BmxIgRqqysVGFh4RX/vsrKSlVUVAQdAADg2hDeEE+6adMmffLJJyooKKh1X2lpqZo3bx60ayNJcXFxKi0tDayJi4sLur9Vq1Zq3rx5YM0PZWVladGiRSG6AgAAYJOQ79CUlJRo1qxZWr9+vVq0aPGzH2eMkcvlCtz+/p9/bM33ZWRkyO/3B46SkpK6Dw8AAKwU8qApLCxUWVmZ+vTpo/DwcIWHh2vbtm3685//rPDwcMXFxamqqkrl5eVBjysrKwvsyni93lo7MeXl5bpw4UKtnZvL3G63oqOjgw4AAHBtCHnQDB06VPv27VNRUVHg6Nu3r8aPHx/4c7NmzZSfnx94zPHjx1VcXKykpCRJUmJiooqLi3X8+PHAmq1bt8rtdqtPnz6hHhkAAFgu5K+hiYqKUkJCQtC5yMhIxcTEBM5PmTJFc+bMUUxMjFq3bq25c+fqpptuCryAODk5WT169FBqaqqeffZZnTx5UnPnztXUqVPZeQEAALU0yIuCf8pzzz2n8PBwjRs3TufOndPQoUP18ssvKywsTJIUFhamd955R9OnT9fAgQMVERGhlJQULVu2zIlxAQBAI+cyxhinh2gIFRUV8ng88vv9Id/V+ZHXJTcqdfpXbXIX1DQscjX+d+0tMAucHgFXyWXB15nh6+yaUt/v33yWEwAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA64U7PQAAWGeDy+kJflqKcXoC4H+KHRoAAGA9ggYAAFiPoAEAANYjaAAAgPUIGgAAYD2CBgAAWI+gAQAA1iNoAACA9QgaAABgPYIGAABYj6ABAADW47OcADQ8Cz76SHz0EWA1dmgAAID1CBoAAGA9ggYAAFiPoAEAANYjaAAAgPUIGgAAYD2CBgAAWI+gAQAA1iNoAACA9QgaAABgvZAHTVZWlvr166eoqCjFxsbq7rvv1qFDh4LWVFZWasaMGWrTpo0iIyM1ZswYff3110Frjh07ptGjRysyMlJt2rTRzJkzVVVVFepxAQBAExDyoNm2bZvS0tK0a9cu5efn6+LFi0pOTtbZs2cDa9LT05Wbm6tNmzZp+/btOnPmjEaNGqXq6mpJUnV1tUaOHKmzZ89q+/bt2rRpk9544w3NmTMn1OMCAIAmIOQfTpmXlxd0e+3atYqNjVVhYaFuu+02+f1+vfTSS3r11Vc1bNgwSdL69evVvn17vffeexoxYoS2bt2qAwcOqKSkRD6fT5K0fPlyTZw4UUuWLFF0dHSoxwYAABZr8NfQ+P1+SVLr1q0lSYWFhbpw4YKSk5MDa3w+nxISErRjxw5J0s6dO5WQkBCIGUkaMWKEKisrVVhYeMW/p7KyUhUVFUEHAAC4NjRo0BhjNHv2bA0aNEgJCQmSpNLSUjVv3lytWrUKWhsXF6fS0tLAmri4uKD7W7VqpebNmwfW/FBWVpY8Hk/gaN++fQNcEQAAaIwaNGgefvhhffbZZ9q4ceNPrjXGyOVyBW5//88/tub7MjIy5Pf7A0dJSUn9BwcAAFZpsKCZMWOG3nrrLX3wwQdq165d4LzX61VVVZXKy8uD1peVlQV2Zbxeb62dmPLycl24cKHWzs1lbrdb0dHRQQcAALg2hDxojDF6+OGH9de//lV///vfFR8fH3R/nz591KxZM+Xn5wfOHT9+XMXFxUpKSpIkJSYmqri4WMePHw+s2bp1q9xut/r06RPqkQEAgOVC/i6ntLQ0bdiwQW+++aaioqICOy0ej0cRERHyeDyaMmWK5syZo5iYGLVu3Vpz587VTTfdFHjXU3Jysnr06KHU1FQ9++yzOnnypObOnaupU6ey8wIAAGoJedCsXr1akjRkyJCg82vXrtXEiRMlSc8995zCw8M1btw4nTt3TkOHDtXLL7+ssLAwSVJYWJjeeecdTZ8+XQMHDlRERIRSUlK0bNmyUI8LAACagJAHjTHmJ9e0aNFCq1at0qpVq350TYcOHfT222+HcjQAANBE8VlOAADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsB5BAwAArEfQAAAA6xE0AADAegQNAACwHkEDAACsR9AAAADrETQAAMB6BA0AALAeQQMAAKxH0AAAAOsRNAAAwHoEDQAAsF640wMAoeRa5HJ6hJ9kFhinRwCAJocdGgAAYD2CBgAAWI+gAQAA1iNoAACA9QgaAABgPYIGAABYj6ABAADWI2gAAID1CBoAAGA9ggYAAFiPoAEAANYjaAAAgPUIGgAAYD2CBgAAWI+gAQAA1iNoAACA9QgaAABgPYIGAABYj6ABAADWC3d6AAAAEGzRokVOj/CTFixY4PQIQdihAQAA1iNoAACA9QgaAABgPYIGAABYj6ABAADW411OAIAmwuX0AD+DcXqAJosdGgAAYD2CBgAAWK9RB83zzz+v+Ph4tWjRQn369NE//vEPp0cCAACNUKMNmtdff13p6emaP3++9u7dq1tvvVV33HGHjh075vRoAACgkWm0QbNixQpNmTJFDzzwgLp3766VK1eqffv2Wr16tdOjAQCARqZRvsupqqpKhYWFevzxx4POJycna8eOHVd8TGVlpSorKwO3/X6/JKmioqLhBm3Emtxl/9wLOt+wY4TCz/2aPG/BxTSp/77qcinfNdgUofOz/234Ovvf+pn//Z+/dv9dLj+vMXV8R5hphL755hsjyfzzn/8MOr9kyRJz4403XvExCxYsMLr0fjgODg4ODg4Oy4+SkpI6tUOj3KG5zOUK/p0Cxpha5y7LyMjQ7NmzA7dramp08uRJxcTE/OhjGouKigq1b99eJSUlio6Odnqcq8K1NF5N6Xq4lsapKV2L1LSux6ZrMcbo9OnT8vl8dXpcowyaNm3aKCwsTKWlpUHny8rKFBcXd8XHuN1uud3uoHPXX399g83YEKKjoxv9F9rPxbU0Xk3periWxqkpXYvUtK7HlmvxeDx1fkyjfFFw8+bN1adPH+Xn5wedz8/PV1JSkkNTAQCAxqpR7tBI0uzZs5Wamqq+ffsqMTFROTk5OnbsmB566CGnRwMAAI1M2MKFCxc6PcSVJCQkKCYmRpmZmVq2bJnOnTunV199Vb169XJ6tAYRFhamIUOGKDy80Tbmz8a1NF5N6Xq4lsapKV2L1LSupyldy5W4jKnr+6IAAAAal0b5GhoAAIC6IGgAAID1CBoAAGA9ggYAAFiPoAEAANYjaBz00UcfafTo0fL5fHK5XNq8ebPTI9VbVlaW+vXrp6ioKMXGxuruu+/WoUOHnB6rXlavXq2bb7458Bs1ExMT9e677zo9VkhkZWXJ5XIpPT3d6VHqbOHChXK5XEGH1+t1eqyr8s033+j+++9XTEyMWrZsqVtuuUWFhYVOj1VnnTp1qvVv43K5lJaW5vRodXbx4kU9+eSTio+PV0REhDp37qzFixerpqbG6dHq5fTp00pPT1fHjh0VERGhpKQkFRQUOD1Wg2iab0a3xNmzZ9WrVy9NmjRJv/vd75we56ps27ZNaWlp6tevny5evKj58+crOTlZBw4cUGRkpNPj1Um7du309NNP69e//rUkad26dbrrrru0d+9e9ezZ0+Hp6q+goEA5OTm6+eabnR6l3nr27Kn33nsvcDssLMzBaa5OeXm5Bg4cqN/+9rd69913FRsbq3//+9/WfWSLdOlrq7q6OnC7uLhYw4cP1z333OPgVPXzzDPP6IUXXtC6devUs2dPffzxx5o0aZI8Ho9mzZrl9Hh19sADD6i4uFivvvqqfD6f1q9fr2HDhunAgQO64YYbnB4vtOr1cdgIOUkmNzfX6TFCpqyszEgy27Ztc3qUkGjVqpX5y1/+4vQY9Xb69GnTpUsXk5+fbwYPHmxmzZrl9Eh1tmDBAtOrVy+nxwiZefPmmUGDBjk9RoOYNWuW+dWvfmVqamqcHqXORo4caSZPnhx0buzYseb+++93aKL6++6770xYWJh5++23g8736tXLzJ8/36GpGg4/ckKD8Pv9kqTWrVs7PMnVqa6u1qZNm3T27FklJiY6PU69paWlaeTIkRo2bJjTo1yVzz//XD6fT/Hx8br33nt15MgRp0eqt7feekt9+/bVPffco9jYWPXu3Vtr1qxxeqyrVlVVpfXr12vy5MlyuVxOj1NngwYN0vvvv6/Dhw9Lkj799FNt375dd955p8OT1d3FixdVXV2tFi1aBJ2PiIjQ9u3bHZqq4fAjJ4ScMUazZ8/WoEGDlJCQ4PQ49bJv3z4lJibq/Pnz+sUvfqHc3Fz16NHD6bHqZdOmTfrkk0+s/7l5//799corr+jGG2/UiRMn9NRTTykpKUn79+9XTEyM0+PV2ZEjR7R69WrNnj1bTzzxhPbs2aOZM2fK7XbrD3/4g9Pj1dvmzZt16tQpTZw40elR6mXevHny+/3q1q2bwsLCVF1drSVLlui+++5zerQ6i4qKUmJiov70pz+pe/fuiouL08aNG7V792516dLF6fFCz+ktIlyiJvQjp+nTp5uOHTuakpISp0ept8rKSvP555+bgoIC8/jjj5s2bdqY/fv3Oz1WnR07dszExsaaoqKiwDlbf+T0Q2fOnDFxcXFm+fLlTo9SL82aNTOJiYlB52bMmGEGDBjg0EShkZycbEaNGuX0GPW2ceNG065dO7Nx40bz2WefmVdeecW0bt3avPzyy06PVi9ffPGFue2224wkExYWZvr162fGjx9vunfv7vRoIUfQNBJNJWgefvhh065dO3PkyBGnRwmpoUOHmgcffNDpMeosNzc38D+yy4ck43K5TFhYmLl48aLTI16VYcOGmYceesjpMeqlQ4cOZsqUKUHnnn/+eePz+Rya6Op9+eWX5rrrrjObN292epR6a9euncnOzg4696c//cl07drVoYlC48yZM+Y///mPMcaYcePGmTvvvNNanpgYAAACjElEQVThiUKPHzkhJIwxmjFjhnJzc/Xhhx8qPj7e6ZFCyhijyspKp8eos6FDh2rfvn1B5yZNmqRu3bpp3rx5Vr9LqLKyUgcPHtStt97q9Cj1MnDgwFq/2uDw4cPq2LGjQxNdvbVr1yo2NlYjR450epR6++6773TddcEvLw0LC7P2bduXRUZGKjIyUuXl5dqyZYuWLl3q9EghR9A46MyZM/riiy8Ct48ePaqioiK1bt1aHTp0cHCyuktLS9OGDRv05ptvKioqSqWlpZIkj8ejiIgIh6ermyeeeEJ33HGH2rdvr9OnT2vTpk368MMPlZeX5/RodRYVFVXrdUyRkZGKiYmx7vVNc+fO1ejRo9WhQweVlZXpqaeeUkVFhSZMmOD0aPXyyCOPKCkpSZmZmRo3bpz27NmjnJwc5eTkOD1avdTU1Gjt2rWaMGGCwsPt/dYyevRoLVmyRB06dFDPnj21d+9erVixQpMnT3Z6tHrZsmWLjDHq2rWrvvjiCz366KPq2rWrJk2a5PRooefwDtE17YMPPjCSah0TJkxwerQ6u9J1SDJr1651erQ6mzx5sunYsaNp3ry5+eUvf2mGDh1qtm7d6vRYIWPra2h+//vfm7Zt25pmzZoZn89nxo4da+Xrmr7vb3/7m0lISDBut9t069bN5OTkOD1SvW3ZssVIMocOHXJ6lKtSUVFhZs2aZTp06GBatGhhOnfubObPn28qKyudHq1eXn/9ddO5c2fTvHlz4/V6TVpamjl16pTTYzUIlzHGOJNSAAAAocHvoQEAANYjaAAAgPUIGgAAYD2CBgAAWI+gAQAA1iNoAACA9QgaAABgPYIGAABYj6ABAADWI2gAAID1CBoAAGC9/wfLeCRDGy9s0AAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt; plt.rcdefaults()\n",
"\n",
"plt.bar([1,2,3,4,5,6,7,8,9], c.values(), color=[\"blue\", \"red\", \"green\", \"purple\", \"magenta\", \"orange\", \"navy\", \"yellow\", \"grey\"], tick_label=[1,2,3,4,5,6,7,8,9])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We see with the test data that there are 986 genes, with 279 unique. 945 unique variations, and 874 unique text articles. This marks a difference from the training data set where about 7.9% of the genes were unique and only slightly more than half of the text articles were unique."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>Gene</th>\n",
" <th>Variation</th>\n",
" <th>Text</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>count</th>\n",
" <td>986</td>\n",
" <td>986</td>\n",
" <td>986</td>\n",
" </tr>\n",
" <tr>\n",
" <th>unique</th>\n",
" <td>279</td>\n",
" <td>945</td>\n",
" <td>874</td>\n",
" </tr>\n",
" <tr>\n",
" <th>top</th>\n",
" <td>TP53</td>\n",
" <td>Truncating Mutations</td>\n",
" <td>Among the best-studied therapeutic targets in ...</td>\n",
" </tr>\n",
" <tr>\n",
" <th>freq</th>\n",
" <td>40</td>\n",
" <td>18</td>\n",
" <td>24</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" Gene Variation \\\n",
"count 986 986 \n",
"unique 279 945 \n",
"top TP53 Truncating Mutations \n",
"freq 40 18 \n",
"\n",
" Text \n",
"count 986 \n",
"unique 874 \n",
"top Among the best-studied therapeutic targets in ... \n",
"freq 24 "
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Test.describe(include='all')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We now import most of the tools we use to build the classifier. Recall that we use a random forest classifier which is a part of pythons scikit-learn package."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"import sklearn.ensemble\n",
"\n",
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.datasets import make_classification\n",
"\n",
"from sklearn.feature_extraction.text import CountVectorizer\n",
"\n",
"from sklearn.feature_extraction.text import TfidfTransformer\n",
"\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"from sklearn import metrics\n",
"\n",
"np.set_printoptions(threshold=np.nan)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.pipeline import Pipeline\n",
"\n",
"text_clf = Pipeline([('vect', CountVectorizer()),\n",
" ('tfidf', TfidfTransformer()),\n",
" ('clf', RandomForestClassifier()),\n",
"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### We take a random 70% of the training data and use a grid search to find the best parameters to create our classifier. By taking a random 70% of the training data to build our classifier on we can use the remaining 30% of the data to test our classifier on to determine how good it is before we classify the actual test data. If we find that the classifier is not as good as we'd like it to be then we can fine-tune it and repeat."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"\n",
"X_train, X_test = train_test_split(Train, train_size=.7)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.grid_search import GridSearchCV\n",
"\n",
"parameters = {'vect__ngram_range': [(1, 1), (1, 2)],\n",
" 'tfidf__use_idf': (True, False),\n",
" 'clf__max_depth': (None, 11,12)\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [],
"source": [
"gs_clf = GridSearchCV(text_clf, parameters, n_jobs=1)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [],
"source": [
"gs_clf = gs_clf.fit(X_train['Text'], X_train['Class'])"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 81, 4, 2, 29, 7, 12, 26, 0, 0],\n",
" [ 10, 59, 0, 7, 4, 0, 60, 0, 0],\n",
" [ 2, 0, 6, 9, 2, 0, 12, 0, 0],\n",
" [ 34, 4, 1, 147, 5, 3, 18, 0, 0],\n",
" [ 19, 1, 3, 8, 28, 4, 12, 0, 0],\n",
" [ 9, 4, 0, 4, 5, 54, 10, 0, 0],\n",
" [ 6, 23, 2, 11, 5, 1, 230, 0, 0],\n",
" [ 1, 1, 0, 1, 0, 0, 0, 1, 1],\n",
" [ 1, 0, 0, 2, 0, 0, 3, 0, 3]])"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"predicted = gs_clf.predict(X_test['Text'])\n",
"\n",
"metrics.confusion_matrix(X_test['Class'], predicted)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### To interpret the confusion matrix we note that each row corresponds to a class. The rows are the actual classes and the columns are the predicted classes. That means that in row 3 column 7 we observe that there were 12 items in class 3 that the classifier predicted to be in class 7. We also have that every number along the diagonal means that the classifier guessed the class correctly. We see that classes 1, 2, 4, and 7 are predicted frequently compared to classes 3, 5, 6, 8, and 9. This should make sense since we saw that these are the classes that appear the most in the data. Further note that class 8 was only predicted once and it turned out to be correct. \n",
"\n",
"\n",
"\n",
"### As discussed in the introduction, Kaggle scores the predictions based on log-loss. We can calculate the log-loss on our \"test\" sample which we pulled from 30% of the training data. As can be seen below we have a log-loss of 4.065."
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4.065640641565067"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"metrics.log_loss(X_test['Class'], gs_clf.predict_proba(X_test['Text']))"
]
},
{
"cell_type": "code",
"execution_count": 101,
"metadata": {},
"outputs": [],
"source": [
"Answer = gs_clf.predict_proba(Test['Text'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### This was submitted on Kaggle to find that the log-loss on some subset of the data was calculated to be 5.085, which is only slightly higher than the log-loss we calculated on the known data. \n",
"\n",
"\n",
"# Conclusion\n",
"\n",
"#### <ol> By using a random forest classifier and a grid search consisting of 3 parameters with no more than 3 subparameters to choose from each we predicted the probability of classes of some subset of 986 genes / variations based on associated text data with a log-loss of 5.085. This is only slightly higher than our log-loss score on the testing subset of the training data. This of course is not a great log-loss score but considering the different nature of the actual test data it is good to see that it scores similar to the data where we did know the class. Read below for what we could have done to fine tune our algorithm if we were to work more on this problem.\n",
" \n",
"# **Alternative Approaches:**\n",
"\n",
" #### <ol> After reading more discussion of the problem it appeared as if the actual test data was different enough from the training data so as to make the problem not necessarily the most well-posed. For that reason we did not put into practice the following approaches. If we were to continue working on this problem some approaches that would be interesting to consider would be to look at several different classifiers such as a support vector machine, which is also very popular for text classification. We could also have fine tuned our classifier with a stronger gridsearch, which is computationally expensive. Further we could have taken stronger consideration of the gene / variation into our classifier. For example we could have cut each text document to contain only 100 words to the left and right of any given mention of the gene / variation in consideration for that article. This may have made the computations less expensive and cut out some less useful information."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}