Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
Anthony Larcher
ALLIES evaluation
Commits
ba38ae68
Commit
ba38ae68
authored
Oct 06, 2021
by
Olivier Galibert
Browse files
Add more documentation
parent
0e7c0237
Changes
1
Hide whitespace changes
Inline
Side-by-side
README.md
View file @
ba38ae68
...
...
@@ -9,8 +9,8 @@ Contact email: <lifelong-speaker-evaluation@univ-lemans.fr>
| Registration open | September 2021 |
| Development set released | September 15, 2021 |
| Registration deadline | December 15, 2021 |
| Evaluation data available | December 15, 2021
|
| Evaluation period ends | January 15, 2022 |
| Evaluation data available | December 15, 2021 |
| Evaluation period ends | January 15, 2022
|
| System descriptions due | February 28, 2022 |
| Workshop | Possibly June 2022 |
...
...
@@ -272,6 +272,24 @@ message_to_user = {
request
}
# Initialize the user simulation
user
=
UserSimulation
(
parameters
)
`parameters`
is
a
hash
of
values
/
booleans
:
*
float
`max_cost_per_file`
:
maximum
total
cost
allowed
for
a
show
before
the
user
simulation
refuses
to
answer
*
float
`request_collar_cost`
:
collar
size
used
for
computing
answer
costs
(
seconds
)
*
float
`minimal_validation_time`
:
minimal
speaker
time
to
validate
a
speaker
identification
(
seconds
)
*
float
`minimal_validation_purity`
:
minimal
speaker
purity
to
validate
a
speaker
identification
(
0
-
1
)
*
bool
`question_allow_boundaries`
:
allow
the
user
simulation
to
give
boundaries
of
a
segment
(
default
True
)
*
bool
`question_allow_same_intra_show`
:
allow
the
user
simulation
to
say
two
points
in
the
current
show
are
of
the
same
speaker
(
default
True
)
*
bool
`question_allow_different_intra_show`
:
allow
the
user
simulation
to
say
two
points
in
the
current
show
are
of
different
speakers
(
default
True
)
*
bool
`question_allow_same_inter_show`
:
allow
the
user
simulation
to
say
two
points
in
the
different
shows
are
of
the
same
speaker
(
default
True
)
*
bool
`question_allow_different_inter_show`
:
allow
the
user
simulation
to
say
two
points
in
the
different
shows
are
of
different
speakers
(
default
True
)
*
bool
`question_allow_name`
:
allow
the
user
simulation
to
provide
the
name
of
a
speaker
(
default
True
)
Every
question
type
is
allowed
in
__active__
mode
,
the
restrictions
are
only
taken
into
account
in
__interactive__
mode
.
# Send the request to the user simulation and receive the answer
human_assisted_learning
,
user_answer
=
user
.
validate
(
message_to_user
)
```
...
...
@@ -285,6 +303,15 @@ The user simulation returns two objects:
*
`human_assisted_learning`
a boolean, True if the system can ask more questions, False otherwise
*
`user_answer`
, the answer of the user simulation that is defined as follow:
# Final validation of the speakers at the end of handling a show
speakers = user.final_commit(hypothesis)
Returns a list of pairs [hypothesis speaker name, status], both strings, where status is, in order of priority:
*
`known`
the speaker is already mapped from previous shows
*
`short`
there is not enough signal to validate the speaker (speech duration < minimal_validation_time)
*
`bad`
the speech of the speaker is not pure enough to be validated (speech purity < minimal_validation_purity)
*
`good`
the speaker is validated and that name should be used when present in future shows
### Evallies API
This section describes the different objects provided as part of the evallies API.
...
...
Write
Preview
Supports
Markdown
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment