DAVINCIS@IberLEF 2022

Organized by hugojair - Current server time: May 2, 2025, 8:36 a.m. UTC

Previous

Final (Subtask 1)
May 10, 2022, midnight UTC

Current

Final (Subtask 1)
May 10, 2022, midnight UTC

End

Competition Ends
June 10, 2022, 11 p.m. UTC

About the task

Violence has obvious negative effects on those who witness or experience it, including a higher incidence of depression, anxiety, post-traumatic stress disorder, among others. In addition, violence events have a high impact for governments, as they are in charge of guaranteeing security to their population. Therefore, the detection and tracking of violence related events is critical. In this context, social networks comprise a valuable information source for the detection and monitoring of violent events, as people very often post publications notifying the occurrence of violent events in real time. This represents an important opportunity for IT researchers that can provide solutions based on natural language processing for the timely detection of violent incidents in social networks. Solutions of this kind could be used by authorities to respond more efficiently to events happening in real time, and to develop crime prevention policies according to geographical zones and types of events. Likewise, such solutions would be very helpful to the population, as one could know what violent events are happening in which zones in real time.

We are organizing a task for IberLEF2022 called DA-VINCIS. The task focuses on the detection of violent incidents on Twitter. It challenges participants to develop methods able to classify tweets as reporting a violent event or not. For this first edition, the shared task will target Spanish. This is motivated by the lack of resources in Spanish for approaching the task. We are releasing a novel corpus carefully labeled according to violent event categories.

Subtasks

The shared task will feature two tracks: (1) violent event identification and (2) violent event category recognition, see below.

  • Subtask 1: Violent event identification. Determine whether a given tweet is associated with a violent incident or not (binary classification).
  • Subtask 2: Violent event category recognition. Recognize the crime category (see above) to which a given tweet belongs (multi-class multi-label classification). The considered categories are: Accident, Homicide, Non-Violent-incident, Robbery, Kidnapping.

Both subtasks will rely on the DA-VINCIS corpus, and participants can approach either or both tasks.

During the development phase, submissions will be evaluated on the validation partition. Participants will receive immediate feedback on the performance of their submissions. During the final phase submissions will be evaluated on the test partition. Results in the final phase will be used to determine the final and official ranking. These are the evaluation measures computed:

The following evaluation measures will be used in each track

  • Subtask 1. Precision, recall and f1-score with respect to the violent event class. The leading evaluation measure being f1-score.
  • Subtask 2. Macro-average of Precision, recall and f1-score, macro-average of f1-score will be the leading evaluation measure for this subtask.

By registering to this competition you agree to use the data exclusively for the purpose of participation to this competition. Data cannot be shared or distributed under any condition.

By submitting results to this competition, you consent to the public release of your scores at the IberLEF workshop and in the associated proceedings, at the task organizers' discretion. Scores may include, but are not limited to, automatically and manually calculated quantitative judgements, qualitative judgements, and such other metrics as the task organizers see fit. You accept that the ultimate decision of metric choice and score value is that of the task organizers.

You further agree that if your team has several members, each of them will register to the competition and build a competition team (as described on the 'Overview' page) and that if you are a single participant you will build a team with a single member.

You further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgement that the submission was incomplete, erroneous, deceptive, or violated the letter or spirit of the competition's rules. Inclusion of a submission's scores is not an endorsement of a team or individual's submission, system, or science.

Each team can participate with up to five submissions for the final phase. During the validation phase a maximum of 100 submissions are allowed, 5 per day. Files to be uploaded must be compressed in a .zip file. This is the expected format for submissions:

Substask 1

Format for predictions is a CSV file with one prediction per line (0 for non-violent or 1 for violent) in the same order as the corresponding data files (i.e., line 1 in the prediction file must correspond to user 1 in the data file). The file must be in a .zip file for submission, please do not use folders.

Example:

0
1
0
1
1
1
...

Substask 2

Format for predictions is a CSV file with five predictions per line, each prediction is associated to a single category (1 for the presence of the category and 0 otherwise) use a comma "," separator between predictions in the same line. The order of categories is as follows: Accident, Homicide, Non-Violent-incident, Robbery, Kidnapping, for columns 1 to 5, respectively.  This order corresponds to the order available in trial and training data.  You should use the same order as the corresponding data files i.e., line 1 in the prediction file must correspond to user 1 in the data file). The file must be in a .zip file for submission, please do not use folders..

Example:

0,1,1,0,1
0,0,0,0,1
1,0,1,0,1
1,0,0,1,0
1,1,1,0,1
1,0,0,0,1
...

You can check development data to see the format for both data and submissions by looking into reference (ground-truth) files.

Organization team

  • Luis Joaquin Arellano (INAOE, Mexico)
  • Hugo Jair Escalante (INAOE, Mexico)
  • Luis Villaseñor (INAOE, Mexico)
  • Manuel Montes (INAOE, Mexico)
  • Fernando Sánchez (Colegio de Mexico, Mexico)

Contact: da-vincis@googlegroups.com

Please note you can also use the forum to contact organizers and other participants

Paper submission

Format details will be communicated shortly, according to the specifications of IberLEF organizers.

 

Schedule

  • March 11th, 2022: Release of trial corpora
  • Mar 15th, 2022: Release of training corpora.
  • May 10th, 2022: Release of test corpora and start of evaluation campaign
  • June 7th, 2022: Deadline for paper submission
  • Jun 21th, 2022: Acceptance notification
  • Jun 28th, 2022: Camera ready submission deadline
  • September 2022: Publication of proceedings
  • October 2022: Workshop with SEPLN 2022

Development (Subtask 2)

Start: March 15, 2022, midnight

Development (Subtask 1)

Start: March 15, 2022, midnight

Final (Subtask 2)

Start: May 10, 2022, midnight

Final (Subtask 1)

Start: May 10, 2022, midnight

Competition Ends

June 10, 2022, 11 p.m.

You must be logged in to participate in competitions.

Sign In
# Username Score
1 LuisArellano 0.7817
2 EstebanPonce3 0.7804
3 danielvallejo237 0.7759