Competition

Evaluation Criteria

UCF MAMA is part of the detection track of the Tiny Actions Challenge. Each video clip in the test set has multiple actions occuring [out of 35 classes]. Therefore, the participants are required to predict per-frame bounding boxes for a particular action, as well as the corresponding action label. The leaderboard is evaluated at 0.5 ioU threshold for the fmap/vmap. Moreover, balanced accuracy is estimated over all action [label] predictions by averaging the recall of all action classes.
For computational efficiency, the leaderboard is evaluated at a subset of 2000 samples from the test set. The video ids on which our evaluation protocol runs are mentioned on GitHub.,

Evaluation Process

Note: A mini evaluation test set has been released here. Use only those test ids to make a submission to this leaderboard. Inference over entire test set shall be performed on 1 best performing submission per team at the end of the competition.

We will provide an evaluation server where the performers can submit their results. A single zipped folder containing the individual test samples and their predictions [pkl] format can be uploaded on the leaderboard.

Submission Format

Please submit a zip file containing several folders like sample1/ sample2/ etc. Each of these sample/ folders consists of a SINGLE .pkl file.
The pickle format is essentially a dictionary containing 35 keys (corresponding to 35 action classes in the MAMA dataset) Each key of a dictionary contains a numpy array of shape (n_tubes, 4*num_frames+1). In case a particular class is not present in the prediction, please create a numpy array of shape (0,4*num_frames+1) All the rows of the numpy arrays are of the form x1,y1,x2,y2,......,l etc. where each set of 4 coordintes represents the bbox around an actor in a particular frame In a single frame, the detection (x1,y1) and (x2,y2) represent the top left and bottom right of the bbox respectively. Note that the coordinates follow the opencv label format, i.e. x1 represents the offset along the width dimensions of the image. Also, all the detections are expected to be normalized b/w [0-1] for ease of the evaluation following the standard protocol of the AVA dataset. The last label [l] of a row in the numpy array represents a single value b/w [0-1] illustrating the models confidence for the presence of a particular actor/tube.
Please note that zip should follow the precise structure mentioned above. Sample submission files for the leaderboard have been released here

Terms and Conditions

Use of external data is permitted in this challenge.