Darkflow: Test it on VOC2007 and get 60% mAP

Created on 7 Apr 2017 · 4Comments · Source: thtrieu/darkflow

Hi,

Nice work!
But I wrote some codes to test it on VOC2007 test set with configuration '--model cfg/yolo-voc.cfg --load models/yolo-voc.weights' and got 60% mAP, while the YOLO 9000 paper reported 78.6%.
Then I changed input resolution in yolo-voc.cfg from 416416 to 544544 and improved mAP by 2%.
I also tested on a few images and compared them with outputs from darknet and saw significant differences.
I used these testing codes a few times before and am sure they are bug-free.
So I think there must be some bugs...

Thanks

help wanted

Source

ShawnDing1994

👍1

Most helpful comment

@ShawnDing1994 cloud you contribute your code for computing mAP of darkflow？

TaihuLight on 17 May 2017

👍3

All 4 comments

This is a nice work for tensorflow developer, and cloud you fix the code of darkflow according to project https://github.com/leetenki/YOLOv2 and https://github.com/hizhangp/yolo_tensorflow or https://github.com/gliese581gg/YOLO_tensorflow

TaihuLight on 8 Apr 2017

❤1

@ShawnDing1994 cloud you contribute your code for computing mAP of darkflow？

TaihuLight on 17 May 2017

👍3

When building the repo, I see minor numerical differences happens after each operation (e.g. convolution in TF produces result different from Darknet within 1e-4 or so), they get amplified after each layer and eventually results in (catastrophically) inferior mAP in this Tensorflow port. Pretty sure the high-level implementation in this repo is correct (reasonable detection), but lower level (opKernels in Tensorflow and C implementation in Darknet) may have technical mismatch.

It is important to understand how severe it is even when numerical difference is small (< 1e-7). Each trained layer in Darknet expects a different result than what it obtains from Tensorflow, thus produce an even worse result for the next layer. This error accumulates as the net gets deeper.

So, fine-tuning from Darknet weights is a cheap and direct solution to what I described above. Hope this helps.

thtrieu on 19 May 2017

@ShawnDing1994 how did you implement mAP for darkflow, could you contribute your code so I can use it? or could you guide me on how to implement it?