The daily evaluation aims to guaranty the proper functioning of SUP by detecting crashes and bugs due to last commits and showing if the results of the different algorithms are improving day after day.
Evaluation is performed as follows :
- update of SUP core and SUP plugins to the latest version
- full compilation of SUP core and SUP plugins
- processing of the reference sequences
- computation of the results using ViSEvAl
- an e-mail is sent the the sup_dlevel team with the results
The results are computed by comparing today metrics with the reference metrics computed from a default XML1 result file.
At the moment, only sequences are processed : one from ETISEO (ETI-BC-11-C1) to evaluate SUP standard processing chain (MOG segmentation, physical object constructor, F2F tracking and LTT) and one from Nice CHU (2011-01-11a) to evaluate events detection. The next sequence to be added will test person detection.
If you are working on a new plugin or improving an old one, you may want to add a sequence to the daily evaluation. In that case, you should send me :
- the SUP parameter file
- the context file (camera calibration)
- the ground-truth file
- the ViSEvAl configuration file with the metrics to be tested (optional)
Daily evaluation is usually performed everyday of the week at 0:55 AM. If you do not want to be spammed by the e-mails sent everyday, I suggest you add a filter in zimbra for that.
