Time Taken runs from the submission's first connection to the gateway until it calls finish() (or, if it never does, until its process exits), so model loading is not included.
Evaluation Queue
Wall Time is the elapsed time since the evaluation started (or since it was queued, while waiting), including container start-up and model loading.
Submission History
Each team's latest submissions, newest first.
Each team's row also shows how its ranking submission (submission_ranking.yaml) fared at intake. It is not run before the deadline: the one on main at the deadline is marked "fixed at the deadline" and is evaluated once afterwards.
Status Annotations
done
finished all benchmarks
running
being evaluated now
waiting
queued; the number is its place in line
deferred
a newer push on hold until the team's running submission finishes
cancelled
replaced by a newer push before it started, or cut off at the deadline (no result)
over-time
a benchmark ran out of time; scored on what finished, 0 for tasks never reached
rejected
refused before running (invalid submission.yaml, missing or oversized image, bad weights)
failed
the evaluation could not finish. If the submission's process exited early, or was stopped for writing more than allowed, it is scored and ranked on what was recorded until then
best
the submission the team is ranked by
37%
share of the pack's trials that finished, all benchmarks together, rounded down; shown after the run ends
Error Codes
Why a submission was rejected or failed. Its row shows the code beside the status; hover it for the meaning.