Detectron: संदर्भ_gpu.cu स्मृति समस्याओं का कारण बनता है

मेरे लिए यह समस्या बहुत बेतरतीब ढंग से होती है। नेटवर्क (इस मामले में रेटिनानेट) ठीक प्रशिक्षण दे रहा है, जब यादृच्छिक संख्या में पुनरावृत्तियों पर context_gpu.cu आग लगती है और ऐसा लगता है कि जीपीयू मेमोरी को खा रहा है जैसे कि प्रशिक्षण स्मृति त्रुटि से बाहर हो गया है।

हम पास्कल जीपीयू के साथ उबंटू 16.04 का उपयोग कर रहे हैं। कई मशीनों पर होता है, विभिन्न संख्या में GPU (1-4) के साथ और विभिन्न नेटवर्क आर्किटेक्चर को प्रशिक्षित करते समय।

कोई विचार?

json_stats: {"eta": "17:31:05", "fl_fpn3": 0.000720, "fl_fpn4": 0.001108, "fl_fpn5": 0.004955, "fl_fpn6": 0.020188, "fl_fpn7": 0.019255, "iter": 137400, "loss": 0.174941, "lr": 0.001250, "mb_qsize": 64, "mem": 7110, "retnet_bg_num": 31416503.000000, "retnet_fg_num": 63.500000, "retnet_loss_bbox_fpn3": 0.000000, "retnet_loss_bbox_fpn4": 0.000000, "retnet_loss_bbox_fpn5": 0.000000, "retnet_loss_bbox_fpn6": 0.011040, "retnet_loss_bbox_fpn7": 0.004171, "time": 0.614669}
json_stats: {"eta": "17:30:52", "fl_fpn3": 0.000606, "fl_fpn4": 0.001287, "fl_fpn5": 0.004547, "fl_fpn6": 0.027491, "fl_fpn7": 0.009270, "iter": 137420, "loss": 0.137118, "lr": 0.001250, "mb_qsize": 64, "mem": 7110, "retnet_bg_num": 31416766.000000, "retnet_fg_num": 65.000000, "retnet_loss_bbox_fpn3": 0.000000, "retnet_loss_bbox_fpn4": 0.000000, "retnet_loss_bbox_fpn5": 0.000000, "retnet_loss_bbox_fpn6": 0.015492, "retnet_loss_bbox_fpn7": 0.005272, "time": 0.614670}
I0225 23:55:34.962450 20807 context_gpu.cu:321] GPU 0: 7179 MB
I0225 23:55:34.962478 20807 context_gpu.cu:325] Total: 7179 MB
I0225 23:55:34.972862 20810 context_gpu.cu:321] GPU 0: 7323 MB
I0225 23:55:34.972884 20810 context_gpu.cu:325] Total: 7323 MB
I0225 23:55:34.987242 20807 context_gpu.cu:321] GPU 0: 7467 MB
I0225 23:55:34.987257 20807 context_gpu.cu:325] Total: 7467 MB
I0225 23:55:35.004983 20807 context_gpu.cu:321] GPU 0: 7611 MB
I0225 23:55:35.005004 20807 context_gpu.cu:325] Total: 7611 MB
I0225 23:55:35.019520 20807 context_gpu.cu:321] GPU 0: 7755 MB
I0225 23:55:35.019529 20807 context_gpu.cu:325] Total: 7755 MB
I0225 23:55:35.033624 20807 context_gpu.cu:321] GPU 0: 7899 MB
I0225 23:55:35.033632 20807 context_gpu.cu:325] Total: 7899 MB
I0225 23:55:35.048848 20808 context_gpu.cu:321] GPU 0: 8043 MB
I0225 23:55:35.048869 20808 context_gpu.cu:325] Total: 8043 MB
I0225 23:55:35.065871 20807 context_gpu.cu:321] GPU 0: 8187 MB
I0225 23:55:35.065881 20807 context_gpu.cu:325] Total: 8187 MB
I0225 23:55:35.082967 20807 context_gpu.cu:321] GPU 0: 8331 MB
I0225 23:55:35.082975 20807 context_gpu.cu:325] Total: 8331 MB
I0225 23:55:35.102628 20810 context_gpu.cu:321] GPU 0: 8467 MB
I0225 23:55:35.102646 20810 context_gpu.cu:325] Total: 8467 MB
I0225 23:55:35.123090 20807 context_gpu.cu:321] GPU 0: 8607 MB
I0225 23:55:35.123100 20807 context_gpu.cu:325] Total: 8607 MB
I0225 23:55:35.145066 20807 context_gpu.cu:321] GPU 0: 8739 MB
I0225 23:55:35.145074 20807 context_gpu.cu:325] Total: 8739 MB
I0225 23:55:35.166004 20807 context_gpu.cu:321] GPU 0: 8871 MB
I0225 23:55:35.166013 20807 context_gpu.cu:325] Total: 8871 MB
I0225 23:55:35.187448 20807 context_gpu.cu:321] GPU 0: 9003 MB
I0225 23:55:35.187456 20807 context_gpu.cu:325] Total: 9003 MB
I0225 23:55:35.208040 20807 context_gpu.cu:321] GPU 0: 9135 MB
I0225 23:55:35.208050 20807 context_gpu.cu:325] Total: 9135 MB
I0225 23:55:35.229956 20807 context_gpu.cu:321] GPU 0: 9267 MB
I0225 23:55:35.229964 20807 context_gpu.cu:325] Total: 9267 MB
I0225 23:55:35.251646 20807 context_gpu.cu:321] GPU 0: 9399 MB
I0225 23:55:35.251655 20807 context_gpu.cu:325] Total: 9399 MB
I0225 23:55:35.273802 20807 context_gpu.cu:321] GPU 0: 9531 MB
I0225 23:55:35.273811 20807 context_gpu.cu:325] Total: 9531 MB
I0225 23:55:35.294629 20807 context_gpu.cu:321] GPU 0: 9660 MB
I0225 23:55:35.294638 20807 context_gpu.cu:325] Total: 9660 MB
I0225 23:55:35.320922 20808 context_gpu.cu:321] GPU 0: 9795 MB
I0225 23:55:35.320945 20808 context_gpu.cu:325] Total: 9795 MB
I0225 23:55:35.346731 20809 context_gpu.cu:321] GPU 0: 9934 MB
I0225 23:55:35.346740 20809 context_gpu.cu:325] Total: 9934 MB
I0225 23:55:35.430550 20807 context_gpu.cu:321] GPU 0: 10068 MB
I0225 23:55:35.430560 20807 context_gpu.cu:325] Total: 10068 MB
I0225 23:55:35.566123 20809 context_gpu.cu:321] GPU 0: 10200 MB
I0225 23:55:35.566140 20809 context_gpu.cu:325] Total: 10200 MB
I0225 23:55:35.958365 20807 context_gpu.cu:321] GPU 0: 10332 MB
I0225 23:55:35.958379 20807 context_gpu.cu:325] Total: 10332 MB
terminate called after throwing an instance of 'caffe2::EnforceNotMet'
  what():  [enforce fail at context_gpu.cu:343] error == cudaSuccess. 2 vs 0. Error at: /home/user/workspace/caffe2/caffe2/core/context_gpu.cu:343: out of memory Error from operator: 
input: "gpu_0/res2_2_sum" input: "gpu_0/res3_0_branch1_w" input: "gpu_0/__m14_shared" output: "gpu_0/res3_0_branch1_w_grad" output: "gpu_0/__m13_shared" name: "" type: "ConvGradient" arg { name: "no_bias" i: 1 } arg { name: "stride" i: 2 } arg { name: "exhaustive_search" i: 0 } arg { name: "order" s: "NCHW" } arg { name: "kernel" i: 1 } device_option { device_type: 1 cuda_gpu_id: 0 } engine: "CUDNN" is_gradient_op: true
*** Aborted at 1519599336 (unix time) try "date -d @1519599336" if you are using GNU date ***
PC: @     0x7fd60edf8428 gsignal
*** SIGABRT (@0x3ea0000510d) received by PID 20749 (TID 0x7fd4b7fff700) from PID 20749; stack trace: ***
    @     0x7fd60f19e390 (unknown)
    @     0x7fd60edf8428 gsignal
    @     0x7fd60edfa02a abort
    @     0x7fd60c66484d __gnu_cxx::__verbose_terminate_handler()
    @     0x7fd60c6626b6 (unknown)
    @     0x7fd60c662701 std::terminate()
    @     0x7fd60c68dd38 (unknown)
    @     0x7fd60f1946ba start_thread
I0225 23:55:36.066072 20807 context_gpu.cu:321] GPU 0: 10524 MB
I0225 23:55:36.066085 20807 context_gpu.cu:325] Total: 10524 MB
    @     0x7fd60eeca41d clone
    @                0x0 (unknown)