{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Text Classification Application: Fake News detection\n", "* Author: Johannes Maucher\n", "* Last update: 24.11.2020\n", "\n", "In this notebook conventional Machine Learning algorithms are applied to learn a discriminator-model for distinguishing fake- and non-fake news.\n", "\n", "What you will learn:\n", "* Access text from .csv file\n", "* Preprocess text for classification\n", "* Calculate BoW matrix\n", "* Apply conventional machine learning algorithms for fake news detection\n", "* Evaluation of classifiers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Access Data\n", "In this notebook a [fake-news corpus from Kaggle](https://www.kaggle.com/c/fake-news/data) is applied for training and testing Machine Learning algorithms. Download the 3 files and save it in a directory. The path of this directory shall be assigned to the variable `path`in the following code-cell: " ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "pfad=\"/Users/johannes/DataSets/fake-news/\"\n", "train = pd.read_csv(pfad+'train.csv',index_col=0)\n", "test = pd.read_csv(pfad+'test.csv',index_col=0)\n", "test_labels=pd.read_csv(pfad+'submit.csv',index_col=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Data in dataframe `train` is applied for training. The dataframe `test`contains the texts for testing the model and the dataframe `test_labels` contains the true labels of the test-texts. " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of texts in train-dataframe: \t 20800\n", "Number of columns in train-dataframe: \t 4\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titleauthortextlabel
id
0House Dem Aide: We Didn’t Even See Comey’s Let...Darrell LucusHouse Dem Aide: We Didn’t Even See Comey’s Let...1
1FLYNN: Hillary Clinton, Big Woman on Campus - ...Daniel J. FlynnEver get the feeling your life circles the rou...0
2Why the Truth Might Get You FiredConsortiumnews.comWhy the Truth Might Get You Fired October 29, ...1
315 Civilians Killed In Single US Airstrike Hav...Jessica PurkissVideos 15 Civilians Killed In Single US Airstr...1
4Iranian woman jailed for fictional unpublished...Howard PortnoyPrint \\nAn Iranian woman has been sentenced to...1
\n", "
" ], "text/plain": [ " title author \\\n", "id \n", "0 House Dem Aide: We Didn’t Even See Comey’s Let... Darrell Lucus \n", "1 FLYNN: Hillary Clinton, Big Woman on Campus - ... Daniel J. Flynn \n", "2 Why the Truth Might Get You Fired Consortiumnews.com \n", "3 15 Civilians Killed In Single US Airstrike Hav... Jessica Purkiss \n", "4 Iranian woman jailed for fictional unpublished... Howard Portnoy \n", "\n", " text label \n", "id \n", "0 House Dem Aide: We Didn’t Even See Comey’s Let... 1 \n", "1 Ever get the feeling your life circles the rou... 0 \n", "2 Why the Truth Might Get You Fired October 29, ... 1 \n", "3 Videos 15 Civilians Killed In Single US Airstr... 1 \n", "4 Print \\nAn Iranian woman has been sentenced to... 1 " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(\"Number of texts in train-dataframe: \\t\",train.shape[0])\n", "print(\"Number of columns in train-dataframe: \\t\",train.shape[1])\n", "train.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Append the test-dataframe with the labels, which are contained in dataframe `test_labels`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "test[\"label\"]=test_labels[\"label\"]" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of texts in test-dataframe: \t 5200\n", "Number of columns in test-dataframe: \t 4\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titleauthortextlabel
id
20800Specter of Trump Loosens Tongues, if Not Purse...David StreitfeldPALO ALTO, Calif. — After years of scorning...0
20801Russian warships ready to strike terrorists ne...NaNRussian warships ready to strike terrorists ne...1
20802#NoDAPL: Native American Leaders Vow to Stay A...Common DreamsVideos #NoDAPL: Native American Leaders Vow to...0
20803Tim Tebow Will Attempt Another Comeback, This ...Daniel VictorIf at first you don’t succeed, try a different...1
20804Keiser Report: Meme Wars (E995)Truth Broadcast Network42 mins ago 1 Views 0 Comments 0 Likes 'For th...1
\n", "
" ], "text/plain": [ " title \\\n", "id \n", "20800 Specter of Trump Loosens Tongues, if Not Purse... \n", "20801 Russian warships ready to strike terrorists ne... \n", "20802 #NoDAPL: Native American Leaders Vow to Stay A... \n", "20803 Tim Tebow Will Attempt Another Comeback, This ... \n", "20804 Keiser Report: Meme Wars (E995) \n", "\n", " author \\\n", "id \n", "20800 David Streitfeld \n", "20801 NaN \n", "20802 Common Dreams \n", "20803 Daniel Victor \n", "20804 Truth Broadcast Network \n", "\n", " text label \n", "id \n", "20800 PALO ALTO, Calif. — After years of scorning... 0 \n", "20801 Russian warships ready to strike terrorists ne... 1 \n", "20802 Videos #NoDAPL: Native American Leaders Vow to... 0 \n", "20803 If at first you don’t succeed, try a different... 1 \n", "20804 42 mins ago 1 Views 0 Comments 0 Likes 'For th... 1 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(\"Number of texts in test-dataframe: \\t\",test.shape[0])\n", "print(\"Number of columns in test-dataframe: \\t\",test.shape[1])\n", "test.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Selection\n", "\n", "In the following code cells, first the number of missing-data fields is determined. Then the information in columns `author`, `title` and `text` are concatenated to a single string, which is saved in the column `total`. After this process, only columns `total` and `label` are required, all other columns can be removed in the `train`- and the `test`-dataframe. " ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "title 558\n", "author 1957\n", "text 39\n", "label 0\n", "dtype: int64" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train.isnull().sum(axis=0)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "title 122\n", "author 503\n", "text 7\n", "label 0\n", "dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test.isnull().sum(axis=0)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "train = train.fillna(' ')\n", "train['total'] = train['title'] + ' ' + train['author'] + ' ' + train['text']" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "train = train[['total', 'label']]" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
totallabel
id
0House Dem Aide: We Didn’t Even See Comey’s Let...1
1FLYNN: Hillary Clinton, Big Woman on Campus - ...0
2Why the Truth Might Get You Fired Consortiumne...1
315 Civilians Killed In Single US Airstrike Hav...1
4Iranian woman jailed for fictional unpublished...1
\n", "
" ], "text/plain": [ " total label\n", "id \n", "0 House Dem Aide: We Didn’t Even See Comey’s Let... 1\n", "1 FLYNN: Hillary Clinton, Big Woman on Campus - ... 0\n", "2 Why the Truth Might Get You Fired Consortiumne... 1\n", "3 15 Civilians Killed In Single US Airstrike Hav... 1\n", "4 Iranian woman jailed for fictional unpublished... 1" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train.head()" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "test = test.fillna(' ')\n", "test['total'] = test['title'] + ' ' + test['author'] + ' ' + test['text']\n", "test = test[['total', 'label']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Preprocessing\n", "The input texts in column `total` shall be preprocessed as follows:\n", "* stopwords shall be removed\n", "* all characters, which are neither alpha-numeric nor whitespaces, shall be removed\n", "* all characters shall be represented in lower-case.\n", "* for all words, the lemma (base-form) shall be applied" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "import nltk\n", "from nltk.corpus import stopwords\n", "from nltk.stem import WordNetLemmatizer\n", "import re" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "stop_words = stopwords.words('english')" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "lemmatizer = WordNetLemmatizer()\n", "for index in train.index:\n", " #filter_sentence = ''\n", " sentence = train.loc[index,'total']\n", " # Cleaning the sentence with regex\n", " sentence = re.sub(r'[^\\w\\s]', '', sentence)\n", " # Tokenization\n", " words = nltk.word_tokenize(sentence)\n", " # Stopwords removal\n", " words = [lemmatizer.lemmatize(w).lower() for w in words if not w in stop_words]\n", " filter_sentence = \" \".join(words)\n", " train.loc[index, 'total'] = filter_sentence" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First 5 cleaned texts in the training-dataframe:" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
totallabel
id
0house dem aide we didnt even see comeys letter...1
1flynn hillary clinton big woman campus breitba...0
2why truth might get you fired consortiumnewsco...1
315 civilians killed in single us airstrike hav...1
4iranian woman jailed fictional unpublished sto...1
\n", "
" ], "text/plain": [ " total label\n", "id \n", "0 house dem aide we didnt even see comeys letter... 1\n", "1 flynn hillary clinton big woman campus breitba... 0\n", "2 why truth might get you fired consortiumnewsco... 1\n", "3 15 civilians killed in single us airstrike hav... 1\n", "4 iranian woman jailed fictional unpublished sto... 1" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Clean data in the test-dataframe in the same way as done for the training-dataframe above:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "lemmatizer = WordNetLemmatizer()\n", "for index in test.index:\n", " #filter_sentence = ''\n", " sentence = test.loc[index,'total']\n", " # Cleaning the sentence with regex\n", " sentence = re.sub(r'[^\\w\\s]', '', sentence)\n", " # Tokenization\n", " words = nltk.word_tokenize(sentence)\n", " # Stopwords removal\n", " words = [lemmatizer.lemmatize(w).lower() for w in words if not w in stop_words]\n", " filter_sentence = \" \".join(words)\n", " test.loc[index, 'total'] = filter_sentence" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First 5 cleaned texts in the test-dataframe:" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
totallabel
id
20800specter trump loosens tongues not purse string...0
20801russian warship ready strike terrorist near al...1
20802nodapl native american leaders vow stay all wi...0
20803tim tebow will attempt another comeback this t...1
20804keiser report meme wars e995 truth broadcast n...1
\n", "
" ], "text/plain": [ " total label\n", "id \n", "20800 specter trump loosens tongues not purse string... 0\n", "20801 russian warship ready strike terrorist near al... 1\n", "20802 nodapl native american leaders vow stay all wi... 0\n", "20803 tim tebow will attempt another comeback this t... 1\n", "20804 keiser report meme wars e995 truth broadcast n... 1" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "test.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Determine Bag-of-Word Matrix for Training- and Test-Data\n", "In the code-cells below two different types of Bag-of-Word matrices are calculated. The first type contains the **term-frequencies**, i.e. the entry in row $i$, column $j$ is the frequency of word $j$ in document $i$. In the second type, the matrix-entries are not the term-frequencies, but the tf-idf-values. \n", "\n", "Note that for a given typ (term-frequency or tf-idf) a separate matrix must be calculated for training and testing. Since we always pretend, that only training-data is known in advance, the matrix-structure, i.e. the columns (= words) depends only on the training-data. This matrix structure is calculated in the row:\n", "\n", "```\n", "count_vectorizer.fit(X_train)\n", "```\n", "and\n", "```\n", "tfidf.fit(freq_term_matrix_train),\n", "```\n", "respectively. An important parameter of the `CountVectorizer`-class is `min_df`. The value, which is assigned to this parameter is the minimum frequency of a word, such that it is regarded in the BoW-matrix. Words, which appear less often are disregarded.\n", "\n", "The training data is then mapped to this structure by \n", "```\n", "count_vectorizer.transform(X_train)\n", "```\n", "and\n", "```\n", "tfidf.transform(X_train),\n", "```\n", "respectively.\n", "\n", "For the test-data, however, no new matrix-structure is calculated. Instead the test-data is transformed to the structure of the matrix, defined by the training data." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "X_train = train['total'].values\n", "y_train = train['label'].values" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "X_test = test['total'].values\n", "y_test = test['label'].values" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "from sklearn.feature_extraction.text import TfidfTransformer\n", "from sklearn.feature_extraction.text import CountVectorizer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Train BoW-models and transform training-data to BoW-matrix:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "count_vectorizer = CountVectorizer(min_df=4)\n", "count_vectorizer.fit(X_train)\n", "freq_term_matrix_train = count_vectorizer.transform(X_train)\n", "tfidf = TfidfTransformer(norm = \"l2\")\n", "tfidf.fit(freq_term_matrix_train)\n", "tf_idf_matrix_train = tfidf.transform(freq_term_matrix_train)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(20800, 55055)" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "freq_term_matrix_train.toarray().shape" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(20800, 55055)" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tf_idf_matrix_train.toarray().shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Transform test-data to BoW-matrix:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "freq_term_matrix_test = count_vectorizer.transform(X_test)\n", "tf_idf_matrix_test = tfidf.transform(freq_term_matrix_test)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Train a linear classifier\n", "Below a [Logistic Regression model](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression) is trained. This is just a linear classifier with a sigmoid- or softmax- activity-function. " ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "X_train=tf_idf_matrix_train\n", "X_test=tf_idf_matrix_test\n", "#X_train=freq_term_matrix_train\n", "#X_test=freq_term_matrix_test" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "LogisticRegression()" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "logreg = LogisticRegression()\n", "logreg.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Evaluate trained model\n", "First, the trained model is applied to predict the class of the training-samples:" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "y_pred_train = logreg.predict(X_train)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([1, 1, 1, ..., 0, 1, 1])" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_pred_train" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "from sklearn.metrics import classification_report" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model's prediction are compared with the true classes of the training-samples. The classification-report contains the common metrics for evaluating classifiers:" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " 0 0.99 0.98 0.98 10387\n", " 1 0.98 0.99 0.98 10413\n", "\n", " accuracy 0.98 20800\n", " macro avg 0.98 0.98 0.98 20800\n", "weighted avg 0.98 0.98 0.98 20800\n", "\n" ] } ], "source": [ "print(classification_report(y_train,y_pred_train))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The output of the classification report shows, that the model is well fitted to the training data, since it predicts training data with an accuracy of 98%.\n", "\n", "However, accuracy on the training-data, provides no information on the model's capability to classify new data. Therefore, below the model's prediction on the test-dataset is calculated:" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [], "source": [ "y_pred_test = logreg.predict(X_test)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " precision recall f1-score support\n", "\n", " 0 0.59 0.65 0.62 2339\n", " 1 0.69 0.63 0.66 2861\n", "\n", " accuracy 0.64 5200\n", " macro avg 0.64 0.64 0.64 5200\n", "weighted avg 0.64 0.64 0.64 5200\n", "\n" ] } ], "source": [ "print(classification_report(y_test,y_pred_test))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The model's accuracy on the test-data is weak. The model is overfitted on the training-data. It seems that the distribution of test-data is significantly different from the distribution of training-data. \n", "\n", "The main drawback in this experiment is possibly the application of the BoW-model to represent texts. BoW disregards word-order and semantic relations between words. The application of word-embeddings and neural networks like CNNs and LSTMs may perform much better." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
totallabel
id
0house dem aide we didnt even see comeys letter...1
1flynn hillary clinton big woman campus breitba...0
2why truth might get you fired consortiumnewsco...1
315 civilians killed in single us airstrike hav...1
4iranian woman jailed fictional unpublished sto...1
\n", "
" ], "text/plain": [ " total label\n", "id \n", "0 house dem aide we didnt even see comeys letter... 1\n", "1 flynn hillary clinton big woman campus breitba... 0\n", "2 why truth might get you fired consortiumnewsco... 1\n", "3 15 civilians killed in single us airstrike hav... 1\n", "4 iranian woman jailed fictional unpublished sto... 1" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train.head()" ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "from tensorflow.keras.preprocessing import text" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [], "source": [ "MAX_NB_WORDS=5000" ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [], "source": [ "tokenizer=text.Tokenizer(num_words=MAX_NB_WORDS)\n", "tokenizer.fit_on_texts(train[\"total\"])" ] }, { "cell_type": "code", "execution_count": 36, "metadata": {}, "outputs": [], "source": [ "trainSeq=tokenizer.texts_to_sequences(train[\"total\"])" ] }, { "cell_type": "code", "execution_count": 37, "metadata": {}, "outputs": [], "source": [ "testSeq=tokenizer.texts_to_sequences(test[\"total\"])" ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "5000" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "tokenizer.num_words" ] }, { "cell_type": "code", "execution_count": 39, "metadata": {}, "outputs": [], "source": [ "textlenghtsTrain=[len(t) for t in trainSeq]" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [], "source": [ "textlenghtsTest=[len(t) for t in testSeq]" ] }, { "cell_type": "code", "execution_count": 41, "metadata": {}, "outputs": [], "source": [ "from matplotlib import pyplot as plt" ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEWCAYAAABmE+CbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjHklEQVR4nO3de7hV1Xnv8e+vEBE1KMjWIlA3RnJBT2qUGmwutSGNtEmDbbQlJ0bSkFCNsc2tFpo2mpPQaJqTpLaVxKgBNQGpMRG1tnowanKC4PaKqFQiRLYQ2EZj0EQi+PaP8W6ZLNa+b/YFfp/nWc+e651zjDnGXGuvd80x55pTEYGZmdlv9HcDzMxsYHBCMDMzwAnBzMySE4KZmQFOCGZmlpwQzMwMcELYq0j6mqR/6KW6fkvSc5KG5PPbJX2oN+rO+m6WNLO36uvCej8v6SlJP+3rdddpywJJn++nda+X9PZeqOc5SUf1Rpv2FEkh6ej+bsdg4IQwSOQ/8K8kbZX0c0k/knSWpJdfw4g4KyI+18m62v0wiIgnIuKgiNjRC22/QNLVNfX/YUQs7GndXWzHeOCTwKSI+M0680+W1NxL6+q1unpqTyaefI88vifqtr7nhDC4/HFEvBI4ErgQ+Fvg8t5eiaShvV3nAHEk8LOI2NLfDbHetxe/b/tORPgxCB7AeuDtNbETgZeAY/P5AuDzOT0auBH4OfA08APKF4CrssyvgOeA84BGIIBZwBPAnZXY0KzvduALwErgWeB6YFTOOxlortdeYBrwa+DFXN8Dlfo+lNO/Afw98BNgC3AlcHDOa23HzGzbU8Cn29lOB2f5lqzv77P+t2efX8p2LKgpd2DN/OeAI7LsHODHwM+AJZV+zweurdRxEbCsrbrqtPXl1yufvwu4P1+zHwGvr9menwIezO1/DbB/Zf55wCZgI/Ch3GZHA7Nz2/8623FDR/XRxnunje0dwNGV/vwbcBOwFVgBvKqNcguBT+b02KznI/n86Fyv8vmHgbUZW1rdllnuHOAxYF3G/qayLT5Y08Y/Ah7O9j0JfKq//7cH0qPfG+BHJ1+oOgkh408AZ+f0yx8wlA/vrwGvyMdbKv9gu9TFzg/dK/PDbDj1E8KTwLG5zHeAq3PeybSREHL6gtZlK/NvZ2dC+GD+wx8FHARcB1xV07ZvZLt+G9gGvK6N7XQlJVm9Msv+NzCrrXbWlK3Xj48BdwHjgGHA14FFOe+ArP8DuX2fAsZ1Zl11Xq/jKcnwjcAQSgJcDwyrbM+VlCQ1CngEOCvnTQN+ChyTbbqK3T+oP1+z7vbqa/O9U6cPtet5mvJFZSjwLWBxG+U+yM7k9L8pCfeayrzrc/ptuV2Pz+3/L8CdNeu/NfswPLfFZna+T79d08ZNwFtyeiRwfH//bw+kh4eMBr+NlH+GWi8CY4AjI+LFiPhB5H9BOy6IiOcj4ldtzL8qIh6KiOeBfwD+rPWgcw+9D/hyRDweEc8Bc4EZNUMAn42IX0XEA8ADlMSwi2zLnwNzI2JrRKwH/i/w/h607S8peyTNEbGNktxOkzQ0In4JnAF8GbgaODciunvc4MPA1yNiRUTsiHJ8ZRswpbLMxRGxMSKeBm4Ajsv4nwHfjIjV2abPdnKdbdXXnfdOq+siYmVEbKckhOPaWO4O4C15DOytwBeBN+W838v5UN4bV0TEvbn95wInSWqs1PWFiHg637et26L1fXpBzXpfBCZJGhERz0TEvZ3s1z7BCWHwG0v5Vlbrnyjfum+R9LikOZ2oa0MX5v+E8u1xdKda2b4jsr5q3UOBwyux6llBv6TsSdQaDexXp66xPWjbkcB380D+zynfpHe0ti0iVgKPA6IMJ/VkPZ9sXU+uazxl27Rqaxscwa6vTUevY0f1dee901Gdu4iIH1OGsI6j7IHcCGyU9Bp2TQi7vDfyC8PP2PU1rfa3dltU3wsA76EMG/1E0h2STupUr/YRTgiDmKTfofxj/LB2Xn5D/mREHAX8MfAJSVNbZ7dRZUffAsdXpn+L8m3rKeB5ylBFa7uGAA1dqHcj5QOxWvd2yq5/VzyVbaqt68lOlq/Xzg3AH0bEIZXH/hHxJICkcyhDGRsp4/jt1dWeDcC8mvUcEBGLOlF2E2VIq9X4mvldaksH753edAdwGrBfbs87gDMpQzn35zK7vDckHQgcyq6vabV/m9j9fbpzwYi7I2I6cBjwPXqWxPc6TgiDkKQRkt4FLKaMza+qs8y7JB0tScAvKN9qW08h3UwZr++qMyRNknQA8H8oB1R3UMbR95f0TkmvoBzIHVYptxlorJ4iW2MR8HFJEyQdBPwjZTx5e1cal21ZAsyT9EpJRwKfoAzndMZm4FBJB1diX8v6jgSQ1CBpek6/Gvg8Zdjo/cB5ko5rp672fAM4S9IbVRyY2/OVnSi7BPgLSa/L1+YzdfrV6de7g/dOb7oD+CjlJAYox5XOBX4YO093/jalb8dJGkZ5b6zI4cB6lgAfqLxPz2+dIWk/Se+TdHBEvMjOvllyQhhcbpC0lfJt8tOUseu/aGPZicD/o+yWLwcuiYjbc94XgL/PoYlPdWH9V1EOHP4U2B/4K4CIeBb4CHAZ5Zvb80B1LP3f8+/PJNUbs70i674TWAe8QPlg6I5zc/2PU/acvp31dygiHqUkp8dz2xwB/DPlzJZbctvfBbwxj29cDVwUEQ9ExGPA3wFXSRrWRl3trbuJchzhX4FnKEM2H+hku28GLga+n+WW56xt+fdyyrj5zyV9rxNVtvfe6U13UA7+tyaEH1L2NFufExHLKMervkP59v8qYEZbFea2+CpwG2Vb3FazyPuB9ZJ+AZxFSeaWWs86MbO9hKTXAQ9RzlDq0l6W7du8h2C2F5D0JzkkMpLye4gbnAysq5wQzPYOf0n5Md6PKePiZ/dvc2ww8pCRmZkB3kMwM7M0aC8GNXr06GhsbOzvZpiZDSr33HPPUxHRUG/eoE0IjY2NNDU19XczzMwGFUm1v95+mYeMzMwMcEIwM7PkhGBmZoATgpmZJScEMzMDnBDMzCw5IZiZGeCEYGZmyQnBzMyAQfxL5Z5onHNTj8qvv/CdvdQSM7OBw3sIZmYGOCGYmVnqMCFIukLSFkkP1cTPlbRG0mpJX6zE50pam/NOqcRPkLQq512cN/BG0jBJ12R8haTGXuyfmZl1Umf2EBYA06oBSb8PTAdeHxHHAF/K+CTKDbCPyTKXSBqSxeYDsyk38J5YqXMW8ExEHA18hXL7PzMz62MdJoSIuBN4uiZ8NnBhRGzLZbZkfDqwOCK2RcQ6YC1woqQxwIiIWB7lFm1XAqdWyizM6WuBqa17D2Zm1ne6ewzh1cBbcojnDkm/k/GxwIbKcs0ZG5vTtfFdyuRNwZ8FDq23UkmzJTVJamppaelm083MrJ7uJoShwEhgCvA3wJL8Vl/vm320E6eDebsGIy6NiMkRMbmhoe4Nf8zMrJu6mxCageuiWAm8BIzO+PjKcuOAjRkfVydOtYykocDB7D5EZWZme1h3E8L3gLcBSHo1sB/wFLAUmJFnDk2gHDxeGRGbgK2SpuSexJnA9VnXUmBmTp8G3JbHGczMrA91+EtlSYuAk4HRkpqB84ErgCvyVNRfAzPzQ3y1pCXAw8B24JyI2JFVnU05Y2k4cHM+AC4HrpK0lrJnMKN3umZmZl3RYUKIiPe2MeuMNpafB8yrE28Cjq0TfwE4vaN2mJnZnuVfKpuZGeCEYGZmyQnBzMwAJwQzM0tOCGZmBjghmJlZckIwMzPACcHMzJITgpmZAU4IZmaWnBDMzAxwQjAzs+SEYGZmgBOCmZklJwQzMwOcEMzMLHWYECRdIWlL3h2tdt6nJIWk0ZXYXElrJa2RdEolfoKkVTnv4ryVJnm7zWsyvkJSYy/1zczMuqAzewgLgGm1QUnjgT8AnqjEJlFugXlMlrlE0pCcPR+YTbnP8sRKnbOAZyLiaOArwEXd6YiZmfVMhwkhIu6k3Ou41leA84CoxKYDiyNiW0SsA9YCJ0oaA4yIiOV57+UrgVMrZRbm9LXA1Na9BzMz6zvdOoYg6d3AkxHxQM2sscCGyvPmjI3N6dr4LmUiYjvwLHBoG+udLalJUlNLS0t3mm5mZm3ockKQdADwaeAz9WbXiUU78fbK7B6MuDQiJkfE5IaGhs4018zMOqk7ewivAiYAD0haD4wD7pX0m5Rv/uMry44DNmZ8XJ041TKShgIHU3+IyszM9qAuJ4SIWBURh0VEY0Q0Uj7Qj4+InwJLgRl55tAEysHjlRGxCdgqaUoeHzgTuD6rXArMzOnTgNvyOIOZmfWhzpx2ughYDrxGUrOkWW0tGxGrgSXAw8B/AudExI6cfTZwGeVA84+BmzN+OXCopLXAJ4A53eyLmZn1wNCOFoiI93Ywv7Hm+TxgXp3lmoBj68RfAE7vqB1mZrZn+ZfKZmYGOCGYmVlyQjAzM8AJwczMkhOCmZkBTghmZpacEMzMDHBCMDOz5IRgZmaAE4KZmSUnBDMzA5wQzMwsOSGYmRnghGBmZskJwczMACcEMzNLnblj2hWStkh6qBL7J0mPSnpQ0nclHVKZN1fSWklrJJ1SiZ8gaVXOuzhvpUnebvOajK+Q1Ni7XTQzs87ozB7CAmBaTexW4NiIeD3w38BcAEmTgBnAMVnmEklDssx8YDblPssTK3XOAp6JiKOBrwAXdbczZmbWfR0mhIi4E3i6JnZLRGzPp3cB43J6OrA4IrZFxDrK/ZNPlDQGGBERyyMigCuBUytlFub0tcDU1r0HMzPrO71xDOGDwM05PRbYUJnXnLGxOV0b36VMJplngUPrrUjSbElNkppaWlp6oelmZtaqRwlB0qeB7cC3WkN1Fot24u2V2T0YcWlETI6IyQ0NDV1trpmZtaPbCUHSTOBdwPtyGAjKN//xlcXGARszPq5OfJcykoYCB1MzRGVmZntetxKCpGnA3wLvjohfVmYtBWbkmUMTKAePV0bEJmCrpCl5fOBM4PpKmZk5fRpwWyXBmJlZHxna0QKSFgEnA6MlNQPnU84qGgbcmsd/74qIsyJitaQlwMOUoaRzImJHVnU25Yyl4ZRjDq3HHS4HrpK0lrJnMKN3umZmZl3RYUKIiPfWCV/ezvLzgHl14k3AsXXiLwCnd9QOMzPbs/xLZTMzA5wQzMwsOSGYmRnghGBmZskJwczMACcEMzNLTghmZgY4IZiZWXJCMDMzwAnBzMySE4KZmQFOCGZmlpwQzMwMcEIwM7PkhGBmZoATgpmZpQ4TgqQrJG2R9FAlNkrSrZIey78jK/PmSloraY2kUyrxEyStynkX5600ydttXpPxFZIae7mPZmbWCZ3ZQ1gATKuJzQGWRcREYFk+R9Ikyi0wj8kyl0gakmXmA7Mp91meWKlzFvBMRBwNfAW4qLudMTOz7uswIUTEnZR7HVdNBxbm9ELg1Ep8cURsi4h1wFrgREljgBERsTwiAriypkxrXdcCU1v3HszMrO909xjC4RGxCSD/HpbxscCGynLNGRub07XxXcpExHbgWeDQeiuVNFtSk6SmlpaWbjbdzMzq6e2DyvW+2Uc78fbK7B6MuDQiJkfE5IaGhm420czM6uluQticw0Dk3y0ZbwbGV5YbB2zM+Lg68V3KSBoKHMzuQ1RmZraHdTchLAVm5vRM4PpKfEaeOTSBcvB4ZQ4rbZU0JY8PnFlTprWu04Db8jiDmZn1oaEdLSBpEXAyMFpSM3A+cCGwRNIs4AngdICIWC1pCfAwsB04JyJ2ZFVnU85YGg7cnA+Ay4GrJK2l7BnM6JWemZlZl3SYECLivW3MmtrG8vOAeXXiTcCxdeIvkAnFzMz6j3+pbGZmgBOCmZklJwQzMwOcEMzMLDkhmJkZ4IRgZmbJCcHMzAAnBDMzS04IZmYGOCGYmVlyQjAzM8AJwczMkhOCmZkBTghmZpacEMzMDHBCMDOz1KOEIOnjklZLekjSIkn7Sxol6VZJj+XfkZXl50paK2mNpFMq8RMkrcp5F+dtNs3MrA91OyFIGgv8FTA5Io4FhlBufzkHWBYRE4Fl+RxJk3L+McA04BJJQ7K6+cBsyj2YJ+Z8MzPrQz0dMhoKDJc0FDgA2AhMBxbm/IXAqTk9HVgcEdsiYh2wFjhR0hhgREQsj4gArqyUMTOzPtLthBARTwJfAp4ANgHPRsQtwOERsSmX2QQclkXGAhsqVTRnbGxO18Z3I2m2pCZJTS0tLd1tupmZ1dGTIaORlG/9E4AjgAMlndFekTqxaCe+ezDi0oiYHBGTGxoautpkMzNrR0+GjN4OrIuIloh4EbgO+F1gcw4DkX+35PLNwPhK+XGUIabmnK6Nm5lZH+pJQngCmCLpgDwraCrwCLAUmJnLzASuz+mlwAxJwyRNoBw8XpnDSlslTcl6zqyUMTOzPjK0uwUjYoWka4F7ge3AfcClwEHAEkmzKEnj9Fx+taQlwMO5/DkRsSOrOxtYAAwHbs6HmZn1oW4nBICIOB84vya8jbK3UG/5ecC8OvEm4NietMXMzHrGv1Q2MzPACcHMzJITgpmZAU4IZmaWnBDMzAxwQjAzs+SEYGZmgBOCmZklJwQzMwOcEMzMLDkhmJkZ4IRgZmbJCcHMzAAnBDMzS04IZmYGOCGYmVnqUUKQdIikayU9KukRSSdJGiXpVkmP5d+RleXnSloraY2kUyrxEyStynkX5600zcysD/V0D+Gfgf+MiNcCv025p/IcYFlETASW5XMkTQJmAMcA04BLJA3JeuYDsyn3WZ6Y883MrA91OyFIGgG8FbgcICJ+HRE/B6YDC3OxhcCpOT0dWBwR2yJiHbAWOFHSGGBERCyPiACurJQxM7M+0pM9hKOAFuCbku6TdJmkA4HDI2ITQP49LJcfC2yolG/O2Nicro3vRtJsSU2SmlpaWnrQdDMzq9WThDAUOB6YHxFvAJ4nh4faUO+4QLQT3z0YcWlETI6IyQ0NDV1tr5mZtaMnCaEZaI6IFfn8WkqC2JzDQOTfLZXlx1fKjwM2ZnxcnbiZmfWhbieEiPgpsEHSazI0FXgYWArMzNhM4PqcXgrMkDRM0gTKweOVOay0VdKUPLvozEoZMzPrI0N7WP5c4FuS9gMeB/6CkmSWSJoFPAGcDhARqyUtoSSN7cA5EbEj6zkbWAAMB27Oh5mZ9aEeJYSIuB+YXGfW1DaWnwfMqxNvAo7tSVvMzKxn/EtlMzMDnBDMzCw5IZiZGeCEYGZmyQnBzMwAJwQzM0tOCGZmBjghmJlZckIwMzPACcHMzJITgpmZAU4IZmaWnBDMzAxwQjAzs+SEYGZmgBOCmZmlHicESUMk3Sfpxnw+StKtkh7LvyMry86VtFbSGkmnVOInSFqV8y7OW2mamVkf6o09hL8GHqk8nwMsi4iJwLJ8jqRJwAzgGGAacImkIVlmPjCbcp/liTnfzMz6UI8SgqRxwDuByyrh6cDCnF4InFqJL46IbRGxDlgLnChpDDAiIpZHRABXVsqYmVkf6ekewleB84CXKrHDI2ITQP49LONjgQ2V5ZozNjana+NmZtaHup0QJL0L2BIR93S2SJ1YtBOvt87ZkpokNbW0tHRytWZm1hk92UN4E/BuSeuBxcDbJF0NbM5hIPLvlly+GRhfKT8O2JjxcXXiu4mISyNickRMbmho6EHTzcysVrcTQkTMjYhxEdFIOVh8W0ScASwFZuZiM4Hrc3opMEPSMEkTKAePV+aw0lZJU/LsojMrZczMrI8M3QN1XggskTQLeAI4HSAiVktaAjwMbAfOiYgdWeZsYAEwHLg5HwNW45ybul12/YXv7MWWmJn1nl5JCBFxO3B7Tv8MmNrGcvOAeXXiTcCxvdEWMzPrHv9S2czMACcEMzNLTghmZgY4IZiZWXJCMDMzwAnBzMySE4KZmQFOCGZmlpwQzMwMcEIwM7PkhGBmZoATgpmZJScEMzMDnBDMzCw5IZiZGeCEYGZmyQnBzMyAHiQESeMlfV/SI5JWS/rrjI+SdKukx/LvyEqZuZLWSloj6ZRK/ARJq3LexXlvZTMz60M92UPYDnwyIl4HTAHOkTQJmAMsi4iJwLJ8Ts6bARwDTAMukTQk65oPzAYm5mNaD9plZmbd0O2EEBGbIuLenN4KPAKMBaYDC3OxhcCpOT0dWBwR2yJiHbAWOFHSGGBERCyPiACurJQxM7M+0ivHECQ1Am8AVgCHR8QmKEkDOCwXGwtsqBRrztjYnK6N11vPbElNkppaWlp6o+lmZpZ6nBAkHQR8B/hYRPyivUXrxKKd+O7BiEsjYnJETG5oaOh6Y83MrE09SgiSXkFJBt+KiOsyvDmHgci/WzLeDIyvFB8HbMz4uDpxMzPrQz05y0jA5cAjEfHlyqylwMycnglcX4nPkDRM0gTKweOVOay0VdKUrPPMShkzM+sjQ3tQ9k3A+4FVku7P2N8BFwJLJM0CngBOB4iI1ZKWAA9TzlA6JyJ2ZLmzgQXAcODmfJiZWR/qdkKIiB9Sf/wfYGobZeYB8+rEm4Bju9sWMzPrOf9S2czMACcEMzNLTghmZgY4IZiZWXJCMDMzwAnBzMySE4KZmQFOCGZmlpwQzMwMcEIwM7PkhGBmZoATgpmZJScEMzMDnBDMzCw5IZiZGdCzG+RYNzTOuanbZddf+M5ebImZ2a4GzB6CpGmS1khaK2lOf7fHzGxfMyD2ECQNAf4N+AOgGbhb0tKIeLh/WzaweO/CzPakAZEQgBOBtRHxOICkxcB0yv2XrRf0JJn0lJOR2eAwUBLCWGBD5Xkz8MbahSTNBmbn0+ckrenm+kYDT3Wz7EA2IPuli3pUfED2qYf2xj7B3tmvvbFPR7Y1Y6AkBNWJxW6BiEuBS3u8MqkpIib3tJ6BZm/sl/s0eOyN/dob+9SegXJQuRkYX3k+DtjYT20xM9snDZSEcDcwUdIESfsBM4Cl/dwmM7N9yoAYMoqI7ZI+CvwXMAS4IiJW78FV9njYaYDaG/vlPg0ee2O/9sY+tUkRuw3Vm5nZPmigDBmZmVk/c0IwMzNgH0wIg+kSGZLGS/q+pEckrZb01xkfJelWSY/l35GVMnOzb2sknVKJnyBpVc67WFK9U337jKQhku6TdGM+H9R9knSIpGslPZqv10l7QZ8+nu+7hyQtkrT/YOyTpCskbZH0UCXWa/2QNEzSNRlfIamxL/vXqyJin3lQDlj/GDgK2A94AJjU3+1qp71jgONz+pXAfwOTgC8CczI+B7gopydln4YBE7KvQ3LeSuAkym8+bgb+sJ/79gng28CN+XxQ9wlYCHwop/cDDhnMfaL8WHQdMDyfLwE+MBj7BLwVOB54qBLrtX4AHwG+ltMzgGv6633Y423V3w3o4zfGScB/VZ7PBeb2d7u60P7rKdd7WgOMydgYYE29/lDO2jopl3m0En8v8PV+7Mc4YBnwNnYmhEHbJ2BEfniqJj6Y+9R69YBRlLMRbwTeMVj7BDTWJIRe60frMjk9lPLLZu2pvuzJx742ZFTvEhlj+6ktXZK7oW8AVgCHR8QmgPx7WC7WVv/G5nRtvL98FTgPeKkSG8x9OgpoAb6Zw2CXSTqQQdyniHgS+BLwBLAJeDYibmEQ96lGb/bj5TIRsR14Fjh0j7V8D9rXEkKnLpEx0Eg6CPgO8LGI+EV7i9aJRTvxPifpXcCWiLins0XqxAZUnyjfCo8H5kfEG4DnKcMQbRnwfcox9emUYZMjgAMlndFekTqxAdWnTupOPwZbH9u0ryWEQXeJDEmvoCSDb0XEdRneLGlMzh8DbMl4W/1rzunaeH94E/BuSeuBxcDbJF3N4O5TM9AcESvy+bWUBDGY+/R2YF1EtETEi8B1wO8yuPtU1Zv9eLmMpKHAwcDTe6zle9C+lhAG1SUy8iyGy4FHIuLLlVlLgZk5PZNybKE1PiPPepgATARW5i7xVklTss4zK2X6VETMjYhxEdFI2f63RcQZDO4+/RTYIOk1GZpKuXT7oO0TZahoiqQDsi1TgUcY3H2q6s1+VOs6jfKeHpR7CP1+EKOvH8AfUc7W+THw6f5uTwdtfTNl1/NB4P58/BFlfHIZ8Fj+HVUp8+ns2xoqZ3MAk4GHct6/MgAOegEns/Og8qDuE3Ac0JSv1feAkXtBnz4LPJrtuYpy5s2g6xOwiHIc5EXKt/lZvdkPYH/g34G1lDORjuqv16ynD1+6wszMgH1vyMjMzNrghGBmZoATgpmZJScEMzMDnBDMzCw5IVi/kHS7pD1+83JJf5VXH/3Wnl5Xrm+BpNP6Yl0dtONk5ZVkB7ps6+/2dztsgNxC06wrJA2Ncs2YzvgI5VzydXugHUMiYkdv19vJdYtyHvxLHS488J0MPAf8qJ/bsc/zHoK1SVJjfrv+Rl4X/xZJw3Pey9/wJY3OS1Eg6QOSvifpBknrJH1U0ifyom93SRpVWcUZkn6kcr39E7P8gXn9+ruzzPRKvf8u6Qbgljpt/UTW85Ckj2Xsa5QLzy2V9PGa5f9D0utz+j5Jn8npz0n6kIp/yvpWSfrznH+yyj0qvg2syuX+VdLDkm5i50XSkHRhxh+U9KU6bb5A0lWSblO5Lv+HK/P+JrfBg5I+W/N6XALcy66XWGi918ejkn4I/GklPipfkwfzNWjt90GSvpn9e1DSezL+XKXsaZIW5PQCSfOz/49L+r18rR5pXSaXe4ek5ZLuzdfsoIyvl/TZjK+S9FqVizaeBXxc0v2S3lK7nawP9fcv4/wYuA/KJYO3A8fl8yXAGTl9OzA5p0cD63P6A5RfbL4SaKBc+fGsnPcVygX6Wst/I6ffSl6aGPjHyjoOofyq/MCst5nKL0or7TwBWJXLHQSsBt6Q89YDo+uUmQOcQ7l09d3kZdGB7wOvAd4D3Eq5h8bhlEs5jKF8m30emJDL/2lluSOAn1MuXzCK8kvX1h9/HlKnDRdQrr0/PLfhhqzjHZSbu4vype3G3EaNlCvETqlT1/5ZfmKWW8LOX4H/C3B+Tr8NuD+nLwK+WqljZP59rhI7DViQ0wso158S5cJ3vwD+V7bxHsqvtUcDdwIHZpm/BT5TeS3OzemPAJdVtsOn+vv97se+d/lr67p1EXF/Tt9D+VDqyPcjYmtEtFASwg0ZX1VTfhFARNwJjJB0COXDcI6k+ylJY3/gt3L5WyOi3kXD3gx8NyKej4jnKBdi6+ib5g8oH7JvBm4CDpJ0ANAYEWsyvigidkTEZuAO4Hey7MrYOQT11spyG4HbMv4L4AXgMkl/CvyyjXZcHxG/ioinKMnoxNwG7wDuo+wJvJbyQQ/wk4i4q049r6W8Vo9F+ZS9umb7XAUQEbcBh0o6mHIBu39rXSginmlne7W6IetfBWyOiFVRhq1WU17bKZSbzPz/fA1nAkdWyrdeoLGz7yXrQz6GYB3ZVpneQfk2C2XPofULxf7tlHmp8vwldn3P1V43pfUyw+/JD+WXSXoj5Zt5Pd25JePdlGvTPE75hj8a+DDlg6qjOmvbsdv1XyJiew6DTaVcxO+jlG/nuy1a57mAL0TE16szcnilrW1Qtx2tRdtYVm2Uqcbaem2rr2vr86GU98itEfHeNtrSWmYH/vwZcLyHYN21njJUA2VYoTtax+XfTLkBy7OUu0+dmwdNkfSGTtRzJ3CqypU5DwT+hLIH0KaI+DVliOXPgLty+U9Vyt0J/LnKvZ8bKHsCK9tY94xcbgzw+9nug4CDI+I/gI9RhlPqma5yr+JDKcNRd1O2wQcrY+9jJR3WRvlWjwITJL0qn1c/kO8E3pd1nQw8FeW+GrdQEhU5r/W+wpslvU7Sb1C2ZVfcBbxJ0tFZ5wGSXt1Bma2UIUbrZ04I1l1fAs6W9CPKt+vueCbLf41yBUqAzwGvAB5UuSn65zqqJCLupYxvr6TcUe6yiLivE+v/AWXY45c5PY6dCeG7lCuXPkAZBjovymWua32XcsXMVcB8ytASlA+4GyU9mLGP1ylLtvkmygfp5yJiY5Q7k30bWC5pFeX+Cu1+YEbEC8Bs4KY8qPyTyuwLgMnZlgvZeanmzwMj88D5A2QyoxxfuTH7vam99dZpRwvleM+iXN9dlOGs9twA/IkPKvc/X+3UrJ9IuoByAHe3M5DM+oP3EMzMDPAegpmZJe8hmJkZ4IRgZmbJCcHMzAAnBDMzS04IZmYGwP8A0Umc+CmwJAYAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "plt.hist(textlenghtsTrain,bins=20)\n", "plt.title(\"Distribution of text lengths in words\")\n", "plt.xlabel(\"number of words per document\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [], "source": [ "textlenghtsTrain.sort(reverse=True)" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[11420, 9852, 9250, 8712, 8365, 8189, 7403, 6643, 6321, 6295]" ] }, "execution_count": 44, "metadata": {}, "output_type": "execute_result" } ], "source": [ "textlenghtsTrain[:10]" ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": [ "MAX_SEQUENCE_LENGTH=800\n", "EMBEDDING_DIM=100" ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "from tensorflow.keras.preprocessing.sequence import pad_sequences\n", "from tensorflow.keras.utils import to_categorical" ] }, { "cell_type": "code", "execution_count": 47, "metadata": {}, "outputs": [], "source": [ "X_train = pad_sequences(trainSeq, maxlen=MAX_SEQUENCE_LENGTH)\n", "X_test = pad_sequences(testSeq, maxlen=MAX_SEQUENCE_LENGTH)" ] }, { "cell_type": "code", "execution_count": 48, "metadata": {}, "outputs": [], "source": [ "y_train = to_categorical(np.asarray(train[\"label\"]))\n", "y_test = to_categorical(np.asarray(test[\"label\"]))" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": [ "from tensorflow.keras.layers import Embedding, Dense, Input, Flatten, Conv1D, MaxPooling1D, Dropout, Concatenate, GlobalMaxPool1D\n", "from tensorflow.keras.models import Model" ] }, { "cell_type": "code", "execution_count": 50, "metadata": {}, "outputs": [], "source": [ "embedding_layer = Embedding(MAX_NB_WORDS,\n", " EMBEDDING_DIM,\n", " #weights=[embedding_matrix],\n", " input_length=MAX_SEQUENCE_LENGTH,\n", " trainable=True)" ] }, { "cell_type": "code", "execution_count": 51, "metadata": {}, "outputs": [], "source": [ "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n", "embedded_sequences = embedding_layer(sequence_input)\n", "l_cov1= Conv1D(32, 5, activation='relu')(embedded_sequences)\n", "l_pool1 = MaxPooling1D(2)(l_cov1)\n", "l_cov2 = Conv1D(64, 3, activation='relu')(l_pool1)\n", "l_pool2 = MaxPooling1D(5)(l_cov2)\n", "l_flat = Flatten()(l_pool2)\n", "l_dense = Dense(64, activation='relu')(l_flat)\n", "preds = Dense(2, activation='softmax')(l_dense)\n", "model = Model(sequence_input, preds)" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Model: \"model\"\n", "_________________________________________________________________\n", "Layer (type) Output Shape Param # \n", "=================================================================\n", "input_1 (InputLayer) [(None, 800)] 0 \n", "_________________________________________________________________\n", "embedding (Embedding) (None, 800, 100) 500000 \n", "_________________________________________________________________\n", "conv1d (Conv1D) (None, 796, 32) 16032 \n", "_________________________________________________________________\n", "max_pooling1d (MaxPooling1D) (None, 398, 32) 0 \n", "_________________________________________________________________\n", "conv1d_1 (Conv1D) (None, 396, 64) 6208 \n", "_________________________________________________________________\n", "max_pooling1d_1 (MaxPooling1 (None, 79, 64) 0 \n", "_________________________________________________________________\n", "flatten (Flatten) (None, 5056) 0 \n", "_________________________________________________________________\n", "dense (Dense) (None, 64) 323648 \n", "_________________________________________________________________\n", "dense_1 (Dense) (None, 2) 130 \n", "=================================================================\n", "Total params: 846,018\n", "Trainable params: 846,018\n", "Non-trainable params: 0\n", "_________________________________________________________________\n" ] } ], "source": [ "model.compile(loss='categorical_crossentropy',\n", " optimizer='rmsprop',\n", " metrics=['categorical_accuracy'])\n", "model.summary()" ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Epoch 1/6\n", "163/163 [==============================] - 27s 154ms/step - loss: 0.2283 - categorical_accuracy: 0.8963 - val_loss: 2.1035 - val_categorical_accuracy: 0.6313\n", "Epoch 2/6\n", "163/163 [==============================] - 25s 154ms/step - loss: 0.0552 - categorical_accuracy: 0.9823 - val_loss: 2.7110 - val_categorical_accuracy: 0.6369\n", "Epoch 3/6\n", "163/163 [==============================] - 25s 153ms/step - loss: 0.0316 - categorical_accuracy: 0.9903 - val_loss: 3.6822 - val_categorical_accuracy: 0.6377\n", "Epoch 4/6\n", "163/163 [==============================] - 25s 152ms/step - loss: 0.0180 - categorical_accuracy: 0.9951 - val_loss: 5.2826 - val_categorical_accuracy: 0.6352\n", "Epoch 5/6\n", "163/163 [==============================] - 25s 156ms/step - loss: 0.0084 - categorical_accuracy: 0.9975 - val_loss: 5.7444 - val_categorical_accuracy: 0.6310\n", "Epoch 6/6\n", "163/163 [==============================] - 26s 158ms/step - loss: 0.0054 - categorical_accuracy: 0.9983 - val_loss: 7.5507 - val_categorical_accuracy: 0.6423\n" ] } ], "source": [ "history=model.fit(X_train, y_train, validation_data=(X_test, y_test),epochs=6, verbose=True, batch_size=128)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6 (default, Sep 26 2022, 11:37:49) \n[Clang 14.0.0 (clang-1400.0.29.202)]" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false }, "vscode": { "interpreter": { "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" } } }, "nbformat": 4, "nbformat_minor": 4 }