I solved the problem. There was no issue, I simply and stupidly had a typo in my PYTHONPATH. I fixed that and it now works properly. Thanks for this great resource!

I wrote all commands during my installation down so I thought this might help with you're bug research. I've seen that you solved it. But maybe this can help others as a guideline. Good luck to everyone!

System information

• Amazon AWS Instance g3.4xlarge (NVIDIA Tesla M60)
• Ubuntu 16.04

Installation

I used the Caffe2 documentation and the Detectron documentation as a guideline

Update NVIDIA driver

Download driver

sudo dpkg -i nvidia-diag-driver-local-repo-ubuntu1604_375.66-1_amd64.deb
sudo apt-get update && sudo apt-get install wget -y --no-install-recommends

Installing Dependencies

sudo apt-get update
sudo apt-get install -y --no-install-recommends build-essential cmake git libgoogle-glog-dev libgtest-dev libiomp-dev libleveldb-dev liblmdb-dev libopencv-dev libopenmpi-dev libsnappy-dev libprotobuf-dev openmpi-bin openmpi-doc protobuf-compiler python-dev python-pip
sudo pip install setuptools future numpy protobuf enum networkx
sudo apt-get install -y --no-install-recommends libgflags-dev

Install CUDA

Link: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/
Search for the newest version. Use if possible cuda 8. I used: cuda-repo-ubuntu1604_8.0.61-1_amd64.deb

wget "http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/..." //add selected file name
sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda-8-0

Install cuDNN

Search for the newest version: https://developer.nvidia.com/cudnn

wget http://developer.download.nvidia.com/compute/redist/cudnn/v5.1/cudnn-8.0-linux-x64-v5.1.tgz
sudo tar -xzf cudnn-8.0-linux-x64-v5.1.tgz -C /usr/local
rm cudnn-8.0-linux-x64-v5.1.tgz && sudo ldconfig

Install Caffe2

Because of the rapid development of Caffe2, you might need to reset the repository to the last build passing version. You can check this on the Caffe2 GitHub repository. Click on the build passing/failing icon in the top of the README.md. On left you can check the last build and select the last passing build. Click on the build and copy the commit number.

Go to your selected installation directory: cd directory
git clone --recursive https://github.com/caffe2/caffe2.git && cd caffe2
git reset --hard $COMMIT_NUMBER //unnecessary when builds passing
sudo make -j16 //exchange 16 by your number of cores
cd build && sudo make install

Set environment variables

add these lines to .bashrc:

export PYTHONPATH=/usr/local
export PYTHONPATH=$PYTHONPATH:/home/ubuntu/caffe2/build
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

Test Caffe2 & GPU

Test your Caffe2 installation:

python2 -c 'from caffe2.python import core' 2>/dev/null && echo "Success" || echo "Failure"

Test the GPU Installation. The return value needs to be bigger than 0:

python2 -c 'from caffe2.python import workspace; print(workspace.NumCudaDevices())'

Install Detectron dependencies

sudo pip install numpy>=1.13 pyyaml>=3.12 matplotlib opencv-python>=3.2 setuptools Cython mock scipy
sudo git clone https://github.com/cocodataset/cocoapi.git $WANTED-DIRECTORY/cocoapi
cd $WANTED-DIRECTORY/cocoapi/PythonApi
sudo -H make install
sudo python setup.py install

add line to .bashrc:

export COCOAPI=$WANTED-DIRECTORY/cocoapi

Don't forget to download the data from the coco website in this directory. Structure as described in the data README.md

Install Detectron

git clone https://github.com/facebookresearch/detectron detectron
cd detectron/lib && sudo make -j16

check Installation

python2 detectron/tests/test_spatial_narrow_as_op.py

Hope I could help some people :)

@genekogan

I have the same issue when I try to build detectron on the platform of AMI of AWS.
I'm not sure whether my PYTHONPATH is correct
here is my PYTHONPATH:/home/ec2-user/src/caffe2/build

I am wondering what should the "build" directory include?
Here are the files that included in my "build" directory:

Are they as same as yours?

Thanks!

set PYTHONPATH or pth file to include ${CAFFE2_ROOT}/build

custom Caffe2 CPU or GPU operator lib not found: AssertionError: Custom ops lib not found at '/path/to/to/detectron/build/libcaffe2_detectron_custom_ops_gpu.so'Found when run the test_zero_even_op.py but other tests can pass, Detectron ops lib: /usr/local/lib/libcaffe2_detectron_ops_gpu.so found. Could you give me some idea?

@genekogan can you help me? The general tests all pass with the information“ /usr/local/lib/libcaffe2_detectron_ops_gpu.so found”. However, when i test the custom op and run the test_zero_even_op.py, it throws the error"Custom ops lib not found at '/path/to/to/detectron/build/libcaffe2_detectron_custom_ops_gpu".

@ym547559398 i'm not sure of the specific error you are getting. i solved my problem with exactly what @huhai463127310 said: make sure you set PYTHONPATH to include ${CAFFE2_ROOT}/build

@genekogan Thank you! I have solved it. A very simple conflict~

I still get the same problem now that caffe2 is integrated in pytorch, I compiled pytorch from latest source but running detetron simple_infer still complains about not finding detectron_ops (no gpu in my case)

@mehditlili Did you set the PYTHONPATH? Or else, I think we are lacking of some exteded ops, and there's a guide for us to help how to use the ops folder and build extended ops?

@mattifrind

Since pytorch/caffe2 distribution through conda is official, there should be no additional setup to locate those ops modules. While libcaffe2_detectron_ops_gpu.so can be located through the standard PATH env, the pytorch installation path should be searched by default w/o user intervention. Simply inserting the installation path of pytorch sys.path.insert(0, torch.__path__[0] at run time would ensure to find the detectron ops (latest release).

My problem is that Detectron apparently wants the header files in Caffe2/core, but the official build of Pytorch in Anacdona doesn’t include Caffe2/core. Do I need to do an additional pytorch install and append that path to Caffe2_Dir?

I sloved this problem by add "torch.pth" to ~/.conda/envs/dl/lib/python2.7/site-packages" directory. Because "libcaffe2_detectron_ops_gpu.so" is in "site-packages/torch" directory.

torch.pth contains only one path:
/home/ruixin/.conda/envs/dl/lib/python2.7/site-packages/torch

PS: this error will run the below function. You can see "ops_path = os.path.join(prefix, 'lib/libcaffe2_detectron_ops_gpu.so')" is to get the .so file.

def get_detectron_ops_lib():
    """Retrieve Detectron ops library."""
    # Candidate prefixes for the detectron ops lib path
    # print("=> path list = :\n", _CMAKE_INSTALL_PREFIX, "\n", sys.prefix, '\n', sys.exec_prefix, '\n', sys.path)
    prefixes = [_CMAKE_INSTALL_PREFIX, sys.prefix, sys.exec_prefix] + sys.path
    # Search for detectron ops lib
    # print("=> prefixes = {}".format(prefixes))
    for prefix in prefixes:
        ops_path = os.path.join(prefix, 'lib/libcaffe2_detectron_ops_gpu.so')
        # print("=> ops_path = {}".format(ops_path))
        if os.path.exists(ops_path):
            # TODO(ilijar): Switch to using a logger
            print('Found Detectron ops lib: {}'.format(ops_path))
            break
    assert os.path.exists(ops_path), \
        ('Detectron ops lib not found; make sure that your Caffe2 '
         'version includes Detectron module')
    return ops_path

@farleylai

I have tried your suggestion but it didn't work and I'm getting the same error.
Does anyone have another workaround to suggest?

@zabatani

Do you make sure libcaffe2_detectron_ops_gpu.so is in the path lib/ under torch.__path__[0]?
Depending on your installation, it might be somewhere else or you may have multiple entries in torch.__path__.
Print out and list the content to locate where it is.
Alternatively, you may add it to the env variable PATH.

If you are using caffe2 by installing pytorch-nightly, you can copy or link libcaffe2_detectron_ops_gpu.so (it's in somewhere like /home/ruixin/.conda/envs/dl/lib/python2.7/site-packages/torch) to /usr/lib or the virtual environment's lib folder.
It works for me.

After add PYTHONPATH environment variable: /path/to/pytorch/build, the test passed

@suica That worked for me. I couldn't put the link in /ur/lib and I am using conda instead of virtualenv. I put it in /path/to/conda/env/lib and it works. Thanks!

@genekogan谢谢！我已经解决了 一个非常简单的冲突〜

How did you solve the problem you encountered?

I have the same problem but in colab, some idea?

@genekogan你能帮帮我吗？一般测试全部通过信息“/usr/local/lib/libcaffe2_detectron_ops_gpu.so found”。但是，当我测试自定义操作并运行test_zero_even_op.py时，它会抛出错误“未在'/ path / to / to / detectron / build / libcaffe2_detectron_custom_ops_gpu中找到自定义操作库”。
Can you tell me how you solved this problem? I have encountered the same error

@genekogan你能帮帮我吗？一般测试全部通过信息“/usr/local/lib/libcaffe2_detectron_ops_gpu.so found”。但是，当我测试自定义操作并运行test_zero_even_op.py时，它会抛出错误“未在'/ path / to / to / detectron / build / libcaffe2_detectron_custom_ops_gpu中找到自定义操作库”。
Can you tell me how you solved this problem? I have encountered the same error

I can´t solved, sorry.

I too have the same error when I try to train the network using:
python2 tools/train_net.py \
--cfg configs/DensePose_ResNet50_FPN_single_GPU.yaml \
OUTPUT_DIR /tmp/detectron-output
Detectron and caffe2 installation tests both pass successfully.
The error output:
Found Detectron ops lib: /usr/local/lib/python2.7/site-packages/torch/lib/libcaffe2_detectron_ops_gpu.so
Traceback (most recent call last):
File "/usr/local/densepose/tools/train_net.py", line 35, in
c2_utils.import_custom_ops()
File "/usr/local/densepose/detectron/utils/c2.py", line 39, in import_custom_ops
custom_ops_lib = envu.get_custom_ops_lib()
File "/usr/local/densepose/detectron/utils/env.py", line 85, in get_custom_ops_lib
'Custom ops lib not found at \'{}\''.format(custom_ops_lib)
AssertionError: Custom ops lib not found at '/usr/local/densepose/build/libcaffe2_detectron_custom_ops_gpu.so'

@genekogan Thank you! I have solved it. A very simple conflict~

@genekogan你能帮帮我吗？一般测试全部通过信息“/usr/local/lib/libcaffe2_detectron_ops_gpu.so found”。但是，当我测试自定义操作并运行test_zero_even_op.py时，它会抛出错误“未在'/ path / to / to / detectron / build / libcaffe2_detectron_custom_ops_gpu中找到自定义操作库”。
Can you tell me how you solved this problem? I have encountered the same error

And I can't seem to find the 'libcaffe2_detectron_custom_ops_gpu.so' file. I did verify that caffe2/pytorch are in the PYTHONPATH environment variable.

found the solution

in file /path to library/densepose/detectron/utils/env.py
edit line 19
_CMAKE_INSTALL_PREFIX = '/usr/lib'
to
_CMAKE_INSTALL_PREFIX = 'path to lib folder of pytorch that you built using cmake or the setup.py file/'
and run again:)

I don't have detectron module in caffe2. i have installed caffe2 through pytorch. It is installing properly but it doesn't have detectron module. can i get any help.

Thanks in advance

Hey guys, the goal of this code block is to find the file "libcaffe2_detectron_ops_gpu.so
", so just do this:

1. find the file youself: find / -name libcaffe2_detectron_ops_gpu.so
2. add the search result to the search list

@genekogan Thank you! I have solved it. A very simple conflict~

how to solve it,i have the same problem,thank you