Darkflow: YOLO 9000

The configuration is here, It seems the author does not provide YOLO9000 weight file.

Thanks for the quick answer! I'll try to find the weights9000 and will post it here to help others.

Weights file is here: http://pjreddie.com/media/files/yolo9000.weights

Did you manage to get this working? I put in the weights file and well as the config file and I downloaded the 9k.names file as well. Processing didn't take too much longer but post processing took around 10x as long as with the normal yolo.cfg and yolo.weights setup. At the end no bounding boxes showed up in any of the return images. Am I doing something wrong?

I looked at the yolo9000.cfg, turns out there is more work to do. yolo9000 does not do the usual softmax but a grouped one. Very busy right now, I'll try to work on that but at least not in this week and the next.

Yeah no worries at all. Your response times to these issues is crazy fast :) thanks so much for all the work you've put into this - it's a really neat project. I just wanted to make sure I wasn't doing something wrong that was causing the issue - no rush at all getting this sorted out.

Are there any news?
I have the same problem as abagshaw.
Post processing 10x slower and no bounding boxes.
Do i have to change the cfg file and how?

Thanks!

@thtrieu Can we reopen this issue until is it fixed because at the moment I don't think darkflow has YOLO 9000 functionality?

I am likely going to be using YOLO9000 for my thesis project (although I will be heavily adapting it), and as this is the only tensorflow implementation of YOLOv2 I can find I suppose I will use this as a base. I will add my implementation when it is finished, but that will likely take a couple more weeks, so if anyone has more time sooner, it would help me tremendously aswell! ;)

I do have some time on my hands in the next few days - however my understanding of ML and what's actually going on behind the scenes here is pretty shallow. I'm not exactly sure what needs to be done to get YOLO 9000 working - if it's not too complicated could someone briefly outline what needs to be done and I'll give it a shot (if it is pretty complex don't bother as I doubt I'll get very far :) )

@abagshaw what needs to be done is to understand the following parameters in .cfg

random=1
tree=data/9k.tree
map = data/coco9k.map

These will be read into the meta dict by the current code, the only code that needs modifying is the post process function, which, currently does not take into account meta['random'], meta['tree'], meta['map'] to interpret the output tensor & draw bounding boxes.

But wouldn't it need a whole different implementation of the softmax (should be easy, just split it up over the different layers of the tree) and of the backpropagation over this softmax? (as the backprop should only run over the softmax nodes involved?)

Yeah sorry guys - I think this is a little over my head. I was combing through https://github.com/pjreddie/darknet/commit/d2dece3df743c97f2cfbb9bbf0dd0449a8730cec and there's a lot going on there that I don't understand. I'll keep poking around but I don't think I'm going to get very far.

Any updates on this?

Yeah sorry, I decided it would be faster to just stick the yolo9000 classification layer on faster-RCNN in just tensorflow. I might look into hobbying something together when my thesis is finished, but that won't be for another 4 months ;)

Drat - that's too bad. I'm trying to wrap my head around how this whole WordNet tree classification thing works in yolo9000. I wish I understood this stuff better so I could help. Hopefully someone has time to add this functionality soon :)

Guys, I created a repository for YOLO 9000. Everything is explained here. Check it out:

https://github.com/philipperemy/yolo-9000

It uses darknet instead of darkflow but I think it should be quite easy to port it.

@abagshaw @TheLaurens @saiprabhakar @thtrieu @frey123

@philipperemy Thanks, maybe I'm missing something - but I'm not sure how this gets us any closer to implementing YOLO9000 in darkflow?

If you feel like porting YOLO9000 to darkflow that would be awesome :) :)

@abagshaw having it working for darknet was the first step. Now I'm going to focus on how to make it work for darkflow!

@philipperemy Great! Keep us posted :)

@philipperemy Any progress on getting YOLO9000 to work in darkflow? :)

Sorry still no big progress! I'm busy on my work now :)

Thanks All.

May I ask some questions related to this topic?
Current I change the yolo.cfg to classify 2 classes. For the weight I use yolo.weights. After I train the model, it works pretty well.

Q1: From this issue, it seems I need different weight for different number of classes. I remember yolo.weights is at least used for 20 classes. Am I right?

Q2: How many classes can I use this yolo.weights for?

Q3: Is it possible to used the checkpoint of a model for 2 classes as the start weight for another model, say 3 classes?

Thanks again.

@chengs2000 Your questions aren't really related to YOLO9000 - maybe post your question in a new issue and it can be addressed there.

Hi, unless someone's doing this already (@philipperemy?) I'm going to try. Will report back in a few days with results.

@relh go ahead please! I didn't have much time to focus on this.

@relh Any progress? 😃

I got it working! :)

I'm right now just trying to figure out how best to structure my changes for a PR. I see that a lot of things moved around in the last few days.

I currently have most of it as a separate darkflow/cython_utils/cy_yolo9000_findboxes.pyx but I could probably roll it into the cy_yolo2_findboxes.pyx and incorporate another meta key in the if statement of the labels method of darkflow/darkflow/yolo/misc.py (and then do a split between softmax technique in findboxes before jumping into the loop).

I already went into the if statement in the labels method in misc.py to add a hyponym_map which is a dict mapping a parent index to a list of its child nodes per the data files.

Also, I currently have a data/ folder in my cfg/ folder and am just joining the path of the config and meta['tree'] and meta['map'] for the file paths.

If y'all have no input I'll make the PR with what makes sense to me, else I am all ears!

@relh Awesome! Thanks so much for your work on this!!

For my 2 cents: If your cy_yolo9000_findboxes.pyx code is entirely different than the one already written in cy_yolo2_findboxes.pyx than creating a new file isn't a problem - but if they share most of the same code it would probably be a better idea to try to roll in your changes into the existing file (to avoid adding duplicate code). It's hard for me to give much more input without seeing the changes you've made (I'm pretty unfamiliar with exactly how YOLO9000 differs from YOLOv2) - so I'd go with your best judgement on how to roll these changes in.

Looking forward to the PR!

@relh thanks a lot!! Sorry I was very busy recently and did not have the time to work on this!

@relh this is great news. Looking forward to the PR.

Great! (Looking forward to close this issue).

Is this issue closed? Do we have a tensorflow implementation for YOLO9000? Please point me to the same. Thanks.

Question in this issue: pjreddie refers to YOLO9000 as YOLOv2 on his YOLO website. If you go to the "What's new in Version 2" section and click the link to the paper, it opens the YOLO9000 paper.

So if there's no YOLO9000 implementation in Darkflow, what exactly is YOLOv2 in the context of Darkflow?

Yolo9k is the model trained on imagenet with the bounding box learning method whereas the default yolov2 is trained on COCO.

@philipperemy

1) seems your code only work on CPU, do you know why?
2) whats the difference with your project and darknet now?

Thanks!

@moskiteau code works well on GPU
My project is based on darknet and contains everything to run YOLO9000.

compiled with CPU flag:

seb@PHQ-4035-En:~/projects/stockshot/darknet$ ./darknet detector test cfg/combine9k.data cfg/yolo9000.cfg yolo9000.weights data/person.jpg
layer     filters    size              input                output
    0 conv     32  3 x 3 / 1   544 x 544 x   3   ->   544 x 544 x  32
    1 max          2 x 2 / 2   544 x 544 x  32   ->   272 x 272 x  32
    2 conv     64  3 x 3 / 1   272 x 272 x  32   ->   272 x 272 x  64
    3 max          2 x 2 / 2   272 x 272 x  64   ->   136 x 136 x  64
    4 conv    128  3 x 3 / 1   136 x 136 x  64   ->   136 x 136 x 128
    5 conv     64  1 x 1 / 1   136 x 136 x 128   ->   136 x 136 x  64
    6 conv    128  3 x 3 / 1   136 x 136 x  64   ->   136 x 136 x 128
    7 max          2 x 2 / 2   136 x 136 x 128   ->    68 x  68 x 128
    8 conv    256  3 x 3 / 1    68 x  68 x 128   ->    68 x  68 x 256
    9 conv    128  1 x 1 / 1    68 x  68 x 256   ->    68 x  68 x 128
   10 conv    256  3 x 3 / 1    68 x  68 x 128   ->    68 x  68 x 256
   11 max          2 x 2 / 2    68 x  68 x 256   ->    34 x  34 x 256
   12 conv    512  3 x 3 / 1    34 x  34 x 256   ->    34 x  34 x 512
   13 conv    256  1 x 1 / 1    34 x  34 x 512   ->    34 x  34 x 256
   14 conv    512  3 x 3 / 1    34 x  34 x 256   ->    34 x  34 x 512
   15 conv    256  1 x 1 / 1    34 x  34 x 512   ->    34 x  34 x 256
   16 conv    512  3 x 3 / 1    34 x  34 x 256   ->    34 x  34 x 512
   17 max          2 x 2 / 2    34 x  34 x 512   ->    17 x  17 x 512
   18 conv   1024  3 x 3 / 1    17 x  17 x 512   ->    17 x  17 x1024
   19 conv    512  1 x 1 / 1    17 x  17 x1024   ->    17 x  17 x 512
   20 conv   1024  3 x 3 / 1    17 x  17 x 512   ->    17 x  17 x1024
   21 conv    512  1 x 1 / 1    17 x  17 x1024   ->    17 x  17 x 512
   22 conv   1024  3 x 3 / 1    17 x  17 x 512   ->    17 x  17 x1024
   23 conv  28269  1 x 1 / 1    17 x  17 x1024   ->    17 x  17 x28269
   24 detection
mask_scale: Using default '1.000000'
Loading weights from yolo9000.weights...Done!
data/person.jpg: Predicted in 13.577125 seconds.
Tuareg: 25%
wild horse: 27%
goat herder: 82%
Shetland pony: 86%
German shepherd: 48%
Gordon setter: 51%
seb@PHQ-4035-En:~/projects/stockshot/darknet$

and compiled with GPU flag:

seb@PHQ-4035-En:~/projects/stockshot/darknet$ ./darknet detector test cfg/combine9k.data cfg/yolo9000.cfg yolo9000.weights data/person.jpg
layer     filters    size              input                output
    0 conv     32  3 x 3 / 1   544 x 544 x   3   ->   544 x 544 x  32
    1 max          2 x 2 / 2   544 x 544 x  32   ->   272 x 272 x  32
    2 conv     64  3 x 3 / 1   272 x 272 x  32   ->   272 x 272 x  64
    3 max          2 x 2 / 2   272 x 272 x  64   ->   136 x 136 x  64
    4 conv    128  3 x 3 / 1   136 x 136 x  64   ->   136 x 136 x 128
    5 conv     64  1 x 1 / 1   136 x 136 x 128   ->   136 x 136 x  64
    6 conv    128  3 x 3 / 1   136 x 136 x  64   ->   136 x 136 x 128
    7 max          2 x 2 / 2   136 x 136 x 128   ->    68 x  68 x 128
    8 conv    256  3 x 3 / 1    68 x  68 x 128   ->    68 x  68 x 256
    9 conv    128  1 x 1 / 1    68 x  68 x 256   ->    68 x  68 x 128
   10 conv    256  3 x 3 / 1    68 x  68 x 128   ->    68 x  68 x 256
   11 max          2 x 2 / 2    68 x  68 x 256   ->    34 x  34 x 256
   12 conv    512  3 x 3 / 1    34 x  34 x 256   ->    34 x  34 x 512
   13 conv    256  1 x 1 / 1    34 x  34 x 512   ->    34 x  34 x 256
   14 conv    512  3 x 3 / 1    34 x  34 x 256   ->    34 x  34 x 512
   15 conv    256  1 x 1 / 1    34 x  34 x 512   ->    34 x  34 x 256
   16 conv    512  3 x 3 / 1    34 x  34 x 256   ->    34 x  34 x 512
   17 max          2 x 2 / 2    34 x  34 x 512   ->    17 x  17 x 512
   18 conv   1024  3 x 3 / 1    17 x  17 x 512   ->    17 x  17 x1024
   19 conv    512  1 x 1 / 1    17 x  17 x1024   ->    17 x  17 x 512
   20 conv   1024  3 x 3 / 1    17 x  17 x 512   ->    17 x  17 x1024
   21 conv    512  1 x 1 / 1    17 x  17 x1024   ->    17 x  17 x 512
   22 conv   1024  3 x 3 / 1    17 x  17 x 512   ->    17 x  17 x1024
   23 conv  28269  1 x 1 / 1    17 x  17 x1024   ->    17 x  17 x28269
   24 detection
mask_scale: Using default '1.000000'
Loading weights from yolo9000.weights...Done!
data/person.jpg: Predicted in 0.060738 seconds.
African: 25%
worker: 82%
horse: 86%
working dog: 48%
hunting dog: 50%

./darknet detector test cfg/combine9k.data cfg/yolo9000.cfg yolo9000.weights data/person.jpg -thresh .25 -hier .001

can somebody explain what is the structure of 9000 label tree? In the file 9k.tree, for example the n0000245 -1, n0566538625 4, what does the -1 and 4 mean? How can I build a tree structure with this file 9k.tree?

@yaxiongchi
-1 means the root of tree
4: the index of father node