Commit 44256ace authored by Anthony Larcher's avatar Anthony Larcher
Browse files

update readme

parent dbf7c196
......@@ -133,12 +133,9 @@ Run the baseline system.
# Integrate your system
This section describes the different parts of the ``evallies`` package that are necessary to set up your system for both
tasks.
This section describes the different parts of the ``evallies`` package that are necessary to set up your system for both tasks.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Note that input datasets, user simulation and evaluation blocks are fixed and guaranty reproducibility of the experiments.
Participants are free to edit their own system but are not allowed to modify the parameters of the user simulation or database.
......@@ -146,11 +143,11 @@ Participants are free to edit their own system but are not allowed to modify the
Two datasets are available:
* __diarization train__: this dataset includes audio files together with information about the TV shows
* __diarization train__: this dataset includes audio files together with information about the TV shows
and a manual segmentation available for supervised training. This dataset is available for the initial training of the system
and later for system adaptation all along the system life-cycle.
Audio files from this dataset can be access anytime on-demand.
* __diarization lifelong__: This dataset is available in a sequential manner and used for evaluation of the system.
* __diarization lifelong__: This dataset is available in a sequential manner and used for evaluation of the system.
Each audio file is provided to the system without any other information. For each file, the system has to return an hypothesis
that will be evaluated. All information for the files of this dataset are available to the User Simulation to answer the questions.
......@@ -179,10 +176,10 @@ After processing a file, your system exchanges with the user simulation through
The UEM format is a format describing the time ranges in the source audio files the system should be working on. It is used to give the boundaries of the shows but also to exclude the zones with overlapping speakers. It's a space-separated columns format, with four columns:
* File name without the extension
* Channel number (always 1)
* Start time of zone to diarize
* End time of zone to diarize
* File name without the extension
* Channel number (always 1)
* Start time of zone to diarize
* End time of zone to diarize
Example extract:
......@@ -199,14 +196,14 @@ TV8_LaPlaceDuVillage_2011-03-14_172834 1 492.927 495.556
The MDTM format is a format describing the reference or an hypothesis for the speaker identity in a file. It's a space-separated format, with eight columns:
* File name without the extension
* Channel number (always 1)
* Start time of the speaker range
* Duration of the speaker range (beware, not end time)
* Event type (always "speaker")
* Event subtype (always "na")
* Gender ("adult_male" or "adult_female", "unknown" for hypothese, not evaluated in any case)
* Speaker id
* File name without the extension
* Channel number (always 1)
* Start time of the speaker range
* Duration of the speaker range (beware, not end time)
* Event type (always "speaker")
* Event subtype (always "na")
* Gender ("adult_male" or "adult_female", "unknown" for hypothese, not evaluated in any case)
* Speaker id
In the references, the speaker id is the speaker name of the form "Firstname\_LASTNAME", in the hypothesis it is a unique, space-less, identifier per speaker.
......@@ -225,10 +222,9 @@ TV8\_LaPlaceDuVillage_2011-03-14_172834 1 471.666 6.730 speaker na adult_male Ph
Each file from the lifelong learning dataset comes with a flag stored in the __file_info__ variable and named __supervision__,
that specifies the mode of human assisted learning for this file. The mode can be:
* __active__ the system is allowed to ask questions to the human in the loop;
* __interactive__ once the system produces a first hypothesis, the human in the loop provides corrections
to the system to improve the hypothesis;
* __none__ Human assisted learning is OFF for this file. The system can still adapt the model in an unsupervised manner.
* __active__ the system is allowed to ask questions to the human in the loop;
* __interactive__ once the system produces a first hypothesis, the human in the loop provides corrections to the system to improve the hypothesis;
* __none__ Human assisted learning is OFF for this file. The system can still adapt the model in an unsupervised manner.
While processing an audio file, the system can perform unsupervised learning
and goes through the the Human Assisted Learning process if supervision mode is either
......@@ -241,16 +237,18 @@ The code below shows how to interact with the user simulation:
# Create a fake request that is used to initiate interactive learning
# For the case of active learning, this request is overwritten by your system
request = {"request_type": "toto", "time_1": 0.0, "time_2": 0.0}
```
# A request is defined as follow:
# The system can send a question to the human in the loop
# by using an object of type request
# The request is the question asked to the system
# package the request to be sent to the user simulation together with
# the ID of the file of interest and the current hypothesis
The system can send a question to the human in the loop
by using an object of type request
The request is the question asked to the system
package the request to be sent to the user simulation together with
the ID of the file of interest and the current hypothesis
```python
message_to_user = {
"file_id": file_id, # ID of the file the question is related to
"hypothesis": current_hypothesis, # The current hypothesis in ALLIES format
......@@ -262,8 +260,11 @@ human_assisted_learning, user_answer = user.validate(message_to_user)
```
The user simulation returns two objects:
* __human_assisted_learning__ a boolean
* __user_answer__, the answer of the user simulation that is defined as follow:
* __human_assisted_learning__ a boolean, True if the system can ask more questions, False otherwise
* __user_answer__, the answer of the user simulation that is defined as follow:
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment