Competition

SemEval-2024 Task 3: The Competition of Multimodal Emotion Cause Analysis in Conversations (ECAC)

The ability to understand emotions is an essential component of human-like artificial intelligence, as emotions greatly influence human cognition, decision-making, and social interactions. Emotion Cause Analysis, the task of identifying the potential causes behind an individual’s emotional state, is of great importance.

Based on the multimodal conversational emotion cause dataset we built, we define the following two subtasks:

Subtask 1: Textual Emotion-Cause Pair Extraction in Conversations

• Task definition: Extracting all emotion-cause pairs from the given conversation solely based on text, where the emotion cause is defined and annotated as a textual span.
• Input: a conversation containing the speaker and the text of each utterance
• Output: all emotion-cause pairs, where each pair contains an emotion utterance along with its emotion category and the textual cause span in a specific cause utterance, e.g., (3_joy, 2_You made up!). The emotion category should be one of Ekman’s six basic emotions including Anger, Disgust, Fear, Joy, Sadness and Surprise. * Note: There may be multiple cause spans corresponding to the same emotion, thus forming multiple pairs.
• Task example:

{
"conversation_ID": 5,
"conversation": [
	{
		"utterance_ID": 1,
		"text": "Oh , look , wish me luck !",
		"speaker": "Rachel",
		"emotion": "joy"
	},
	{
		"utterance_ID": 2,
		"text": "What for ?",
		"speaker": "Monica",
		"emotion": "neutral"
	},
	{
		"utterance_ID": 3,
		"text": "I am gonna go get one of those job things .",
		"speaker": "Rachel",
		"emotion": "joy"
	}
	],
"emotion-cause_pairs": [
	[
		"1_joy",
		"3_I am gonna go get one of those job things ."
	],
	[
		"3_joy",
		"3_I am gonna go get one of those job things ."
	]
	]
}

• Evaluation metrics: In addition to the micro F1 score, we also calculate a weighted average of F1 scores across the six emotion categories. For the textual cause span, we adopt two strategies to determine whether the span is extracted correctly: Strict Match (the predicted span should be exactly the same as the annotated span) and Proportional Match (considering the overlap proportion of the predicted span and the annotated one).
  • w-avg. Strict F1
  • w-avg. Proportional F1 (main)
  • Strict F1
  • Proportional F1

Subtask 2: Multimodal Emotion-Cause Pair Extraction in Conversations

• Task definition: In consideration of three modalities, extracting all emotion-cause pairs in the conversation, where the emotion cause is defined and annotated at the utterance level and each utterance is represented by text, audio and video.
• Input: a conversation including the speaker, text, and audio-visual clip for each utterance
• Output: all emotion-cause pairs, where each pair contains an emotion utterance along with its emotion category and a cause utterance, e.g., (5_disgust, 5).
• Task example:

{
"conversation_ID": 5,
"conversation": [
	{
		"utterance_ID": 1,
		"text": "Oh , look , wish me luck !",
		"speaker": "Rachel",
		"emotion": "joy",
		"video_name": "dia5utt1.mp4"
	},
	{
		"utterance_ID": 2,
		"text": "What for ?",
		"speaker": "Monica",
		"emotion": "neutral",
		"video_name": "dia5utt2.mp4"
	},
	{
		"utterance_ID": 3,
		"text": "I am gonna go get one of those job things .",
		"speaker": "Rachel",
		"emotion": "joy",
		"video_name": "dia5utt3.mp4"
	}
	],
"emotion-cause_pairs": [
	[
		"1_joy",
		"3"
	],
	[
		"3_joy",
		"3"
	]
	]
}

• Evaluation metrics: In addition to the micro F1 score, we also calculate a weighted average of F1 scores across the six emotion categories.
  • w-avg. F1 (main)
  • F1