README.md 3.36 KB
Newer Older
Anthony Larcher's avatar
Anthony Larcher committed
1
# ALLIES Evaluation home page
Anthony Larcher's avatar
Anthony Larcher committed
2

Anthony Larcher's avatar
Anthony Larcher committed
3
4
5
6
Contact email: <lifelong-speaker-evaluation@univ-lemans.fr>

## Planned schedule

Anthony Larcher's avatar
Anthony Larcher committed
7
8
9
10
11
12
13
14
15
|       Milestones              | Date                  |
|-------------------------------|-----------------------|
| Registration open             |   September 2021      |
| Development set released      |   September 15, 2021  |
| Registration deadline         |   November 30, 2021   |
| Evaluation data available     |   December 1, 2021    |
| Evaluation period ends        |   December 31, 2021   |
| System descriptions due       |   February 2022       |
| Workshop                      |   Under discussion    |
Anthony Larcher's avatar
Anthony Larcher committed
16

Anthony Larcher's avatar
Anthony Larcher committed
17

Anthony Larcher's avatar
Anthony Larcher committed
18
19
## Challenge description

Anthony Larcher's avatar
Anthony Larcher committed
20
21
22
23
### Evaluation plan



Anthony Larcher's avatar
Anthony Larcher committed
24
25
### The tasks

Anthony Larcher's avatar
Anthony Larcher committed
26
![The ALLIES lifelong learning framework](./allies_baseline.png)
Anthony Larcher's avatar
Anthony Larcher committed
27

Anthony Larcher's avatar
Anthony Larcher committed
28
29
30
31
32
33
#### Human assisted learning

#### Lifelong learning and incremental processing

### The metrics

Anthony Larcher's avatar
Anthony Larcher committed
34
35
36
37
References:
* Prokopalo, Yevhenii, Sylvain Meignier, Olivier Galibert, Loïc Barrault, and Anthony Larcher. "[Evaluation of Lifelong Learning Systems.](https://hal.archives-ouvertes.fr/LIUM-LST/hal-02496971v1)" In International Conference on Language Resources and Evaluation. 2020.


Anthony Larcher's avatar
Anthony Larcher committed
38
39
## Baseline system

Anthony Larcher's avatar
Anthony Larcher committed
40
41
42
43
44
45
46
47
48
49
50
51
The LIUM Diarization system system is provided as baseline and to illustrate the integration of a diarization system within The
Lifelong learning processing loop.

The system relies on oepn-source Python modules such as :
* [SIDEKIT](https://lium.univ-lemans.fr/en/sidekit/) an open source framewok for speaker and language characterization
* [S4D](https://lium.univ-lemans.fr/en/s4d/) SIDEKIT for Diarization developed at [LIUM](https://lium.univ-lemans.fr/en)
* [evallies](https://git-lium.univ-lemans.fr/Larcher/evallies) a python module specially developed top support research on lifelong learning human assisted speaker diarization
* [PyTorch](https://pytorch.org/) a well known machine learning framework

### Description

LIUM baseline diarization system is based on previous publications
Anthony Larcher's avatar
Anthony Larcher committed
52
53
54
55
* Yevhenii Prokopalo, Meysam Shamsi, Loic Barrault, Sylvain Meignier, Anthony Larcher, **Active correction for speaker diarization with human in the loop**, in IberSpeech 2021
* Anthony Larcher, Ambuj Mehrish, Marie Tahon, Sylvain Meignier, Jean Carrive, David Doukhan, Olivier Galibert, Nicholas Evans, **Speaker Embedding for Diarization of Broadcast data in the ALLIES Challenge**, in 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021
* Pierre-Alexandre Broux, Florent Desnous, Anthony Larcher, Simon Petitrenaud, Jean Carrive, and Sylvain Meignier, **S4D: Speaker Diarization Toolkit in Python**, in Interspeech. 2018.
* Anthony Larcher, Kong Aik Lee, and Sylvain Meignier, **An extensible speaker identification sidekit in python**, in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5095-5099. IEEE, 2016
Anthony Larcher's avatar
Anthony Larcher committed
56

Anthony Larcher's avatar
Anthony Larcher committed
57
58
### Installation and first run

Anthony Larcher's avatar
Anthony Larcher committed
59
Install conda, create the **allies** environment with the following commands:
Anthony Larcher's avatar
Anthony Larcher committed
60
61

```python
Anthony Larcher's avatar
Anthony Larcher committed
62
63
64
conda create -n allies python=3.9
conda activate allies
pip install evallies
Anthony Larcher's avatar
Anthony Larcher committed
65
66
```

Anthony Larcher's avatar
Anthony Larcher committed
67
68
69
70
71
72
73
74
75
76
Download the archive ''allies.tar.gz'', untar it and move in the directory.
```python
tar -xvf allies.tar.gz
cd ./allies
```

Download the data (wav, mdtm and uem files).
```python
Download command...
```
Anthony Larcher's avatar
Anthony Larcher committed
77

Anthony Larcher's avatar
Anthony Larcher committed
78
79
80
81
82
83
84
85
86
Run the baseline system.
```python
./run_allies_1.py
```

To run the x-vector system, download MUSAN.
```python
./run_allies_2.py
```
Anthony Larcher's avatar
Anthony Larcher committed
87

Anthony Larcher's avatar
Anthony Larcher committed
88
## Integrate your system