Detectron: AVX2 not compiled on Caffe2 binary

Created on 27 Jun 2018  ·  13Comments  ·  Source: facebookresearch/Detectron

Expected results

Not warning about AVX, AVX2

Actual results

Found Detectron ops lib: /usr/local/caffe2_build/lib/libcaffe2_detectron_ops_gpu.so
E0626 23:41:21.145236   347 init_intrinsics_check.cc:54] CPU feature avx is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0626 23:41:21.145258   347 init_intrinsics_check.cc:54] CPU feature avx2 is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.
E0626 23:41:21.145263   347 init_intrinsics_check.cc:54] CPU feature fma is present on your machine, but the Caffe2 binary is not compiled with it. It means you may not get the full speed of your CPU.

Detailed steps to reproduce

Infersimple

`#!bin/bash
python tools/infer_simple.py \
    --cfg configs/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_1x.yaml \
    --output-dir results/ \
    --image-ext jpg \
    --wts models/model_final.pkl \
    test`

System information

  • Operating system: Ubuntu 16.04
  • Compiler version: 5.4.1
  • CUDA version: 8.0.61
  • cuDNN version: 6.0.21
  • NVIDIA driver version: 384.130
  • GPU models (for all devices if they are not all the same): GTX 1080Ti
  • PYTHONPATH environment variable: ?
  • python --version output: 2.7.12
  • Anything else that seems relevant: I used Dockerfile to create a Docker Image and then use nvidia-docker to use a container.
picture ambigus9

Most helpful comment

Hi @ambigus9, these warnings mean that your Caffe2 binary is not compiled with support for vector instructions. You should be able to enable them by specifying -DUSE_NATIVE_ARCH=ON when running cmake (disabled by default; see here).

picture ir413 on 27 Jun 2018
🎉3

All 13 comments

Hi @ambigus9, these warnings mean that your Caffe2 binary is not compiled with support for vector instructions. You should be able to enable them by specifying -DUSE_NATIVE_ARCH=ON when running cmake (disabled by default; see here).

ir413 picture ir413 on 27 Jun 2018
🎉3

@ir413 Thanks. I built a docker image using the Dockerfile that is provided here. As you can see it uses another image provided by Caffe2 So i think the original issue is in the compiled image by Caffe2. Is there any other available image that I can use on the Dockerfile and create a docker image that has AVX2 activated?

ambigus9 picture ambigus9 on 27 Jun 2018

I'm not sure if there are such Caffe2 images readily available. You can build your own Caffe2 image using the dockerfiles from here and make Detectron dockerfile use your custom Caffe2 image.

ir413 picture ir413 on 27 Jun 2018

@ir413 Thanks. I tried with the images available by Caffe2 here and at least caffe2ai/caffe2:latest gives me this error:
imagen
There is a conflict between Detectron and that image, isn't it?

ambigus9 picture ambigus9 on 27 Jun 2018

Caffe2 images from the docker hub are quite old (Last pushed: a year ago) so they may not be compatible with Detectron. As noted in my previous comment, I suggest building your own Caffe2 image which would give you full control over cmake flags used to build Caffe2.

ir413 picture ir413 on 27 Jun 2018

Thanks @ir413 , I would like to confirm if this is the message that appear when Caffe2 has been built with AVX2 correctly.

Infering an image using Detectron built on a Caffe2 Image with AVX2:

imagen

Infering same image using Detectron built on a Caffe2 Image without AVX2:

imagen

It's correct supose that the infer_time has improved when AVX2 is used?

Infer_time: 1.793s vs 2.348s

Thanks.

ambigus9 picture ambigus9 on 30 Jun 2018

@ambigus9 I am having the same issue as you. You seem to have got this resolved. Which Caffe2 Dockerfile did you use and which changes did you make to it? I tried out several things but I can't seem to get this running on my machine...

michael230779 picture michael230779 on 27 Nov 2018

Hi @michael230779 I recommend you to use this Dockerfile and after this line you should add this flag:

-DUSE_NATIVE_ARCH=ON

I hope to help you. Please let me know anything you need.

ambigus9 picture ambigus9 on 27 Nov 2018

@ambigus9 sorry for getting back to you so late. Did not have a chance to look at this before today. Thanks a lot for your help! I finally figured out a way that works for me. I am running the whole thing on a remote machine where I have limited permissions. Here are the changes I had to make in addition to the line you suggested. Maybe it can be of help for someone else.

In my own caffe2 dockerfile I also changed the cuda base image to be version 9.0

FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

And for some yet unknown reason, I also had to add the package "typing" to the pip install section

RUN pip install --no-cache-dir --upgrade pip==9.0.3 setuptools wheel && \ pip install --no-cache-dir \ flask \ future \ graphviz \ hypothesis \ jupyter \ matplotlib \ numpy \ protobuf \ pydot \ python-nvd3 \ pyyaml \ requests \ scikit-image \ scipy \ setuptools \ six \ tornado \ typing

Finally, in the densepose dockerfile I had to change the Caffe2_DIR like this:

RUN mv /usr/local/caffe2 /usr/local/caffe2_build #<-- remove this
ENV Caffe2_DIR /pytorch/caffe2 #<-- change to this

I got rid of the avx errors now. However, the processing time improvement is not as significant in my case as in your case @ambigus9 which makes me wonder if I got everything right. If anybody has any further suggestions, I am happy to hear them!

michael230779 picture michael230779 on 6 Dec 2018

I am also having trouble compiling with the cuda8 base image @michael230779 .

I have just built the the new caffe2 image with AVX and now it's not using the GPU. So something got screwed up.. back to the drawing board.

bafonso picture bafonso on 19 Dec 2018

@ambigus9 I understand that quite some time has passed but I need to follow these steps to build a cpu only docker image for caffe2. But I still get the error about the FindCmake when building the detectron docker image (after changing Detectron/docker/Dockerfile to use the caffe2 image I built). Can you please share what additional steps you took? Thanks in advance.

manyaafonso picture manyaafonso on 20 Nov 2019

@manyaafonso Do you want to make it work only on cpu? If it's true, you only need to consider:

docker pull caffe2ai/caffe2:c2v0.8.1.full.ubuntu16.04
https://hub.docker.com/r/caffe2ai/caffe2/

ambigus9 picture ambigus9 on 20 Nov 2019

@ambigus9 Thanks for your quick reply. It seems the docker image you are suggesting is no longer available. I get: Error response from daemon: manifest for caffe2ai/caffe2:c2v0.8.1.full.ubuntu16.04 not found
Did you do any changes to the CMakeLists.txt or other place so that FindCmake worked?

manyaafonso picture manyaafonso on 21 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

743341 picture 743341  ·  4Comments
baristahell picture baristahell  ·  3Comments
partnercloudsupport picture partnercloudsupport  ·  3Comments
olgaliak picture olgaliak  ·  4Comments
Hwang-dae-won picture Hwang-dae-won  ·  3Comments