Detectron: Various errors when training scales=320

I have the same problem with COCO dataset and sometimes it happens in test time too, not only while training.

And what's more when I reduced the TEST.RPN_PRE_NMS_TOP_N from 1000 to 100, sometimes (~1 in 5 times) the test runs fine, sometimes models (including my own model and the faster-rcnn-R-50-FPN_1x model in model zoo) crashed in test time when I ran test_net.py. The error messages are also various.

I am quite confused about ROI_CANONICAL_SCALE: 90，how can i get it ?

@moyans You can use grep -rn <detectron directory> -e "ROI_CANONICAL_SCALE" --include=*.py to get all lines containing "ROI_CANONICAL_SCALE" in .py files

@daquexian Sorry, I didn't explain it clearly. I know where it is. The original value is 224. I'm just wondering how to calculate this number.

@moyans I calculated it by 224 * 320 / 800.

Did you find a solution for your problem? I got something similar when I reduce the image-scale for my own dataset to 360x480.. I think it is related to the cython_nms, since if I configure TRAIN/TEST.RPN_NMS_THRESH: 0.0 The model trains and evaluates (but with worse results of course..).
Another workaround was to use a larger TRAIN/TEST.SCALE (e.g. 800, with TRAIN/TEST.MAX_SIZE: 1333)

I also tried to debug this, but I still couldn't figure out the problem... using the CPU NMS (without cython) from the old py-faster-rcnn repo I found that sometimes you divide by zero inside the NMS (when the two boxes both have area 0). This should be fixed by setting TRAIN/TEST.RPN_MIN_SIZE > 0. But it seems that this is not the only problem...

Could you please try to switch off the RPN_NMS (set RPN_NMS_THRESH to 0.0) and see if it works then?

Maybe it could also be a problem that the number of anchors/proposals is too small when we use NMS on the regressed boxes that were generated on very small feature maps (due to the reduced input image-size)

@pfollmann Thanks for your information! I may try it tomorrow. Does the bug still exist even though TRAIN/TEST.RPN_MIN_SIZE > 0?

Yes, unfortunately with TRAIN/TEST.RPN_MIN_SIZE I still got errors at random iterations in your above-mentioned style..

I think that I found the problem: It is in detectron/utils/cython_nms.pyx:

cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1]

The numpy.argsort function seems to be buggy at this point (no clue why). I replaced it with the cython argsort implementation from https://github.com/jcrudy/cython-argsort/blob/master/cyargsort/argsort.pyx. To make it work the following changes are necessary:

• Place argsort.pyx in detectron/utils
• Change line 13 in argsort.pyx to
  ctypedef cnp.float32_t FLOAT_t
• register the file in setup.py (similar to cython_nms.pyx and cython_bbox.pyx)
• include it in detectron/utils/cython_nms.pyx, i.e. change the file as follows:
  import utils.argsort as argsort
  ...
  cdef np.ndarray[np.int_t, ndim=1] order = np.empty((ndets), dtype=np.intp)
argsort.argsort(-scores, order)
• and run 'make' in detectron to compile the cython-modules again.

For me the training works fine now for 20k iterations and also the inference had no more seg-faults (including a little speed-up ;-) )

The open question still is why cython_nms.pyx worked fine for other configurations of the TRAIN/TEST.SCALE? In my experience not the image-scale was the problem but the size of objects that are getting very small in the case of rescaled images to small sizes.

Hope that helps!

PS: My current Detectron-version is quite far from the master, therefore I'm not sure if I find time to launch a PR soon..

@pfollmann Wow! So coool! I will try it soon. Thanks!

On Thu, May 17, 2018, 10:57 PM pfollmann notifications@github.com wrote:

I think that I found the problem: It is in detectron/utils/cython_nms.pyx:

cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1]

The numpy.argsort function seems to be buggy at this point (no clue why).
I replaced it with the cython argsort implementation from
https://github.com/jcrudy/cython-argsort/blob/master/cyargsort/argsort.pyx.
To make it work the following changes are necessary:

• Place argsort.pyx in detectron/utils
• Change line 13 in argsort.pyx to
  ctypedef cnp.float32_t FLOAT_t
• register the file in setup.py (similar to cython_nms.pyx and
  cython_bbox.pyx)
• include it in detectron/utils/cython_nms.pyx, i.e. change the file
  as follows:
  import utils.argsort as argsort ... cdef np.ndarray[np.int_t, ndim=1]
  order = np.empty((ndets), dtype=np.intp) argsort.argsort(-scores, order)
• and run 'make' in detectron to compile the cython-modules again.

For me the training works fine now for 20k iterations and also the
inference had no more seg-faults (including a little speed-up ;-) )

The open question still is why cython_nms.pyx worked fine for other
configurations of the TRAIN/TEST.SCALE? In my experience not the
image-scale was the problem but the size of objects that are getting very
small in the case of rescaled images to small sizes.

Hope that helps!

PS: My current Detectron-version is quite far from the master, therefore
I'm not sure if I find time to launch a PR soon..

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebookresearch/Detectron/issues/415#issuecomment-389896464,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALEcn-6IxvfvqGsyY2sjINVV_0D92lXdks5tzY_cgaJpZM4Tzn30
.

@pfollmann It works! Thanks! I'd like to remain this issue open because the patch has not been merged into master.

Looking forward to your PR :) You might want to fetch the master and modify the corresponding files. The steps on master are nothing different from those you pointed out above.

@pfollmann Thanks, save my day!

hi,i meet this problem when use the above method,do you have any methhods to solve it,thank you very much

import detectron.utils.cython_nms as cython_nms
File "detectron/utils/cython_nms.pyx", line 27, in init detectron.utils.cython_nms
import utils.argsort as argsort
ImportError: No module named utils.argsort

@shenghsiaowong I think you should use import detectron.utils.argsort as argsort because the project structure changed after @pfollmann posted his solution.

，，，i have change it，but it does not work，i know thisis a small issue ，but ihave no idea
File "detectron/utils/cython_nms.pyx", line 27, in init detectron.utils.cython_nms
#import detectron.utils.argsort as argsort
ImportError: No module named utils.argsort

what is meaning of this？thank you
import detectron.utils.cython_nms as cython_nms
File "detectron/utils/cython_nms.pyx", line 28, in init detectron.utils.cython_nms
import detectron.utils.argsort as argsort
ImportError: dynamic module does not define init function (initargsort)

@shenghsiaowong sorry I haven't met this error. @pfollmann do you have time to send a PR for your excellent solution so that every user can benefit from it seamlessly? :)

@pfollmann @daquexian Hi, thanks for the solution, but when I try your advice, other error happened: Error in 'python': free() invalid next size (fast)
Could you give any appreciate advice about it?

@shenghsiaowong
try to paste "import detectron.utils.argsort as argsort"
after
cimport cython
import numpy as np
cimport numpy as np

@karenyun I am meeting same problem here, did you figure it out?

@karenyun @StepOITD I met the same problem and solve it as follows:
(1) changing the file detectron/utils/cython_nms.pyx as pfollmann suggested
(2) putting the following sentences after the definition of ndets. In other words, change the original code snippet:

    cdef np.ndarray[np.int_t, ndim=1] order = scores.argsort()[::-1]

    cdef int ndets = dets.shape[0]
    cdef np.ndarray[np.int_t, ndim=1] suppressed = \
            np.zeros((ndets), dtype=np.int)

to

    cdef int ndets = dets.shape[0]
    cdef np.ndarray[np.int_t, ndim=1] suppressed = \
            np.zeros((ndets), dtype=np.int)

    cdef np.ndarray[np.int_t, ndim=1] order = np.empty((ndets), dtype=np.intp)
    argsort.argsort(-scores, order) 

Hope it helps.