Caffe: Creating leveldB Issue

Created on 2 Aug 2014  ·  7Comments  ·  Source: BVLC/caffe

Hello! I am a new user of Caffe.
I made my own dataset, and I want to convert my dataset into leveldB format.
By referencing Imagenet's create_imagenet.sh file, I created create_navigation.sh file.
The code is:
TOOLS=/home/dkkim930122/caffe-master/build/tools
DATA=/home/dkkim930122/caffe-master/data/navigation

echo "Creating leveldb..."

GLOG_logtostderr=1 $TOOLS/convert_imageset.bin \
/home/dkkim930122/caffe-master/data/navigation/ \
$DATA/train.txt \
navigation_train_leveldb 0

echo "Done."

As I run the file from terminal, it says:
dkkim930122@DKK:~/caffe-master/examples/navigation$ ./create_navigation.sh
Creating leveldb...
I0801 17:52:19.638675 9643 convert_imageset.cpp:54] A total of 10 images.
I0801 17:52:19.638782 9643 convert_imageset.cpp:61] Opening leveldb navigation_train_leveldb
E0801 17:52:19.764928 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0001.JPEG
E0801 17:52:19.764960 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0002.JPEG
E0801 17:52:19.764974 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0003.JPEG
E0801 17:52:19.764987 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0004.JPEG
E0801 17:52:19.765000 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0005.JPEG
E0801 17:52:19.765013 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0007.JPEG
E0801 17:52:19.765025 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0009.JPEG
E0801 17:52:19.765038 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0010.JPEG
E0801 17:52:19.765053 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0011.JPEG
E0801 17:52:19.765065 9643 io.cpp:84] Could not open or find file /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_0013.JPEG
Done.

Could anyone help me how I could solve "Could not open or find file ..." error?
Thank you!

downstream problem?

Most helpful comment

Hello!
I apologize for my late reply.
I...just solved the problem!
And, it was super simple...my bad!

So, in "train.txt file, I had files listed as:
bedroom/aa041717.JPEG 0
bedroom/AA041511.JPEG 1

But, all my image files had file extensions as .jpg not .JPEG.
As I changed all image file extensions to .JPEG...it magically worked!
So, for new users of Caffe like me, make sure to use SAME file extension for images and images listed in text file!

I appreciate all the comments and helping me!

All 7 comments

My guess is that /home/dkkim930122/caffe-master/data/navigation/airport_inside/airport_inside_*.JPEG is not there?

(by the way, is there some symbolic link somewhere along the path? I can't remember exactly but recall having problems with symlinked files back in October)

Thank you, @Yangqing for your reply!!
I double checked the files, but the files are in the corresponding directories.
I don't think I have a symbolic link along the path...

Seems like you are using either SUNS or MIT-Indoor dataset. A lot of images in these datasets are actually gifs, but are renamed to jpeg/jpg :(. That _might_ be the issue. If that's the case, you can either run do a dry run to see which images the system can't load, and convert those images explicitly using an image editor; or do something like this in MATLAB:

for i=1:numel(files)
  if mod(i, 1000) == 0
    fprintf(1, '%d/%d\n', i, numel(files));
  end
  [im, cmap] = imread(files{i});
  if size(cmap,2) > 0
    fprintf(1, 'fixing %s\n', files{i});
    newim = cell(1, size(cmap,2));
    for j=1:size(cmap,2)
      ch = cmap(:,j);
      newim{j} = ch(im+1);
    end
    newim = cat(3, newim{:});
    imwrite(newim, files{i}, 'Quality', 100);
  end
end

If you prefer to do a dry run of the system, you can do something like this:

try {
... // opencv code to read image.

} catch ( cv::Exception & e) {
... // print image name, so that you can fix it.
}

Hope it helps!

I appreciate your help!
Yes, I am using the MIT-indoor set for my task!
I will definitely try the MATLAB & dry run!

For a small update...I tried again by using the "cat.jpg" image.
This image is used in the Imagenet Classification example, and I believe this is jpg/jpeg image (I hope!).
But it still gave me the same error that it could not load properly...

To make sure whether I am doing correctly:
All my training images are in "/home/dkkim930122/caffe-master/data/navigation" with various sub-folders (including "airport_inside").

"train.txt" is defined as:
airport_inside/airport_inside_0001.JPEG 12
airport_inside/airport_inside_0002.JPEG 3
airport_inside/airport_inside_0003.JPEG 23
airport_inside/airport_inside_0004.JPEG 3
airport_inside/airport_inside_0005.JPEG 3
airport_inside/airport_inside_0007.JPEG 12
airport_inside/airport_inside_0009.JPEG 5
airport_inside/airport_inside_0010.JPEG 4
airport_inside/airport_inside_0011.JPEG 12
airport_inside/airport_inside_0013.JPEG 25

Thank you! :D

A few possible things:
(1) OpenCV on windows is flaky: http://stackoverflow.com/questions/7011125/can-not-read-image-with-opencv2-3-imread. But based on your path, I guess you are using a *nix system so it may not apply to you.

(2) Mixing debug and release libraries: http://stackoverflow.com/questions/2584273/issue-with-reading-an-image-using-cvimread-function-in-opencv.

It seems to be an opencv downstream problem (maybe an incorrectly compiled opencv?). Double-checking the opencv install may help.

Hello!
I apologize for my late reply.
I...just solved the problem!
And, it was super simple...my bad!

So, in "train.txt file, I had files listed as:
bedroom/aa041717.JPEG 0
bedroom/AA041511.JPEG 1

But, all my image files had file extensions as .jpg not .JPEG.
As I changed all image file extensions to .JPEG...it magically worked!
So, for new users of Caffe like me, make sure to use SAME file extension for images and images listed in text file!

I appreciate all the comments and helping me!

I've met this problem. but it is solved now!!
the covert tools located in caffe_root/tools/convert_imageset.cpp, so does the binary file located in build.
the code read lable.txt in such a function as below:

while (std::getline(infile, line)) {
    pos = line.find_last_of(' '); // that's the mode where and what to split a line. only a **space** is accepted!
    label = atoi(line.substr(pos + 1).c_str()); //the second one to last convert to a intergter
    lines.push_back(std::make_pair(line.substr(0, pos), label)); // the total path as well as string
  }

we should make sure the string in every line of lable.txt should only have one and only sapce

Was this page helpful?
0 / 5 - 0 ratings