Commit dbede7ba authored by Loïc Barrault's avatar Loïc Barrault
Browse files

Added SICK project

parent 2a64acab
This source diff could not be displayed because it is too large. You can view the blob instead.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
{\bf Natural Language Inference} \\[5mm]
\bf Projet Apprentissage Automatique en Langues \\
{\bf 2018/2019} \\[2mm]
\section{Project Description}
This project actually contains 2 tasks: sentence entailment (classification task) and sentence relatedness (regression task)
The task of sentence entailment (SICK-E) is to predict whether two sentences are \textbf{entailed}, \textbf{neutral} or \textbf{contradictory}.
The task of sentence relatedness (SICK-R) is to predict the \textbf{relatedness score} between two sentences. This score ranges from 0.0 to 5.0.
The goal of this project is to implement a machine learning model for SICK-E and/or SICK-R.
\section{Data Description }
Details for this dataset are available at the following address: \textrm{}
File format (tab separated):
{ \scriptsize
pair\_ID & sentence\_A & sentence\_B & relatedness\_score & entailment\_judgment \\
93 & A lone biker is jumping in the air & A man is jumping into a full pool & 1.7 & NEUTRAL
The provided files are described in Table~\ref{table:data}.
Name & File & \# Sent. pairs \\
Train & SICK\_train.txt & 4501 \\
Dev & SICK\_trial.txt & 501 \\
Test & SICK\_test.txt & 4928 \\
\caption{\label{table:data}Description of the data}
For SICK-E, systems are evaluated on classification accuracy (the percent of labels that are predicted correctly) for every sentence pairs.
We are also interested in the precision/recall scores for each class as well as a confusion matrix.
For SICK-R, systems are evaluated using the Pearson correlation coefficient: see scipy.stats.pearsonr.
\section{Project Roadmap}
\item Preprocess and prepare the training data
\item Train, optimize and evaluate a baseline deep recurrent neural network (RNN) using pytorch. \textbf{[one per group]}
\item Each student should propose \textbf{one} enhancement to the baseline model (additional data, regularization, network initialization, new architecture, etc.) \textbf{[one per student]}
\item Prepare the final defense: present your model and the obtained results
\item SICK webpage: \textrm{}
\item Conneau and Kiela, 2018 \textbf{SentEval: An Evaluation Toolkit for Universal Sentence Representations}
\item \textrm{}
\ No newline at end of file
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment