Caffe: problem with memory_data_layer in python

Created on 12 Aug 2014 · 6Comments · Source: BVLC/caffe

Hi everyone,

I am currently trying to use memory_data_layer to feed a network with a lot of in memory image
because it is quicker than crop image and create a leveldb database.
Therefore i am using this code in python:

caffe_root = 'XXX'
solverPrototxt='/XXX_solver.prototxt'
import numpy as np
import skimage
import time
import os
#make sure that caffe is on the python path:
import sys
sys.path.insert(0, caffe_root + 'python')
import caffe
#choose which GPU we want to use - by default=0
os.environ["CUDA_VISIBLE_DEVICES"] ="1" 
#load one image data
fileProcessesd="/XXX.bmp"
image32=caffe.io.load_image(fileProcessesd)
#crop 640 image from the image loaded and associated it to label 0 
#nb batch=64 for test and train
labels=np.zeros((640),dtype='float32')
imageData=np.zeros((640,3,101,101),dtype='float32')
for i in range(64) :
    for j in range(10) : 
        sys.stdout.flush()
        sys.stdout.write("\ri={0:4d}:j={1:4d}".format(i,j))
        imageData[j+i*10,:,:,:]=np.transpose(image32[i:101+i,j:101+j,:],axes=[2,0,1])
#load solver in GPU mode
solver=caffe.SGDSolver(solverPrototxt)
solver.net.set_mode_gpu()
#set input and solve
solver.net.set_input_arrays(imageData, labels)
solver.solve()

and at the end i have this error that i can solve

I0812 14:55:50.548095 32664 solver.cpp:82] Solving XXX-train
I0812 14:55:50.548144 32664 solver.cpp:135] Iteration 0, Testing net (#0)
F0812 14:55:50.548180 32664 memory_data_layer.cpp:41] Check failed: data_ MemoryDataLayer needs to be initalized by calling Reset
*** Check failure stack trace: ***
Aborted (core dumped)

Any idea? Thx.
Henri

Source

hnoel

👍1

Most helpful comment

@hnoel: It appears that you are using a test net with a MemoryDataLayer, which is not currently supported. (Note that solver.net is the train net.) You can remove this test net from your solver prototxt, or you can use a test net that doesn't use a MemoryDataLayer, or you can write code (PR?) to expose test nets in the Python interface, allowing you to test with a MemoryDataLayer.

@shelhamer, yes, I'll try to add an example for this soon.

longjon on 14 Aug 2014

👍3

All 6 comments

To go further, the problem in fact is that I can locate 'reset' method and therefore use it in python.
I have readen in #119 that it may be a problem to change batch size on the fly but
it is not my idea but should be the reason of the error, am I wrong?
Do anyone have encountered this problem? Thx.
Henri

hnoel on 13 Aug 2014

@longjon another vote for a Python solving / in-memory data layer example.
Perhaps it could be a nice addition to the LeNet example?

On Tuesday, August 12, 2014, hnoel [email protected] wrote:

To go further, the problem in fact is that I can locate 'reset' method and
therefore use it in python.
I have readen in #119 https://github.com/BVLC/caffe/issues/119 that it
may be a problem to change batch size on the fly but
it is not my idea but should be the reason of the error, am I wrong?
Do anyone have encountered this problem? Thx.
Henri

—
Reply to this email directly or view it on GitHub
https://github.com/BVLC/caffe/issues/912#issuecomment-52015234.

Evan Shelhamer

shelhamer on 13 Aug 2014

Thanks @shelhamer to link my problem to other one encountered.
As for me, one strong problem is the management of resources
when you own big database and want test different pre-processing strategies.
However I am not sure that it is the good kind of data layer for that use...
Python is a quick way to load and perform pre-process of data
without fill all the hard drive with extra images then
next problem is how to feed the network for training/testing which lead to my problem ..
This strategy is maybe not the quickest (need to perform pre-process each times)
but it reduces the need of hard drive memory to save database (no leveldb database required),
am I right ? Therefore an example should not be a waste fo time !
similar to #701 issue in fact

hnoel on 13 Aug 2014

@shelhamer, yes, I'll try to add an example for this soon.

longjon on 14 Aug 2014

👍3

1196 exposes the test nets and lets these nets take input by `MemoryDataLayer`.

shelhamer on 2 Oct 2014

Hi! This error also occured in my code, which I using the MemoryDataLayer as input layer in TRAIN and TEST phase, but I found that the program will stuck in the TEST phase,and gave me such error: memory_data_layer.cpp:113] Check failed: data_ MemoryDataLayer needs to be initalized by calling Reset. I am not sure this issue whether it was belong to develop probblem , but when I search help matraial in Internet, I found this issue discussion, so I decide post my question here, I am sorry if this action was not proper! Below are the C++ code, deploy.prototxt and solver.prototxt. Thanks very much!
my C++ code:
// Copyright 2013 Yangqing Jia
//
// This is a simple script that allows one to quickly train a network whose
// parameters are specified by text format protocol buffers.
// Usage:
// train_net net_proto_file solver_proto_file [resume_point_file]

include

include "caffe/caffe.hpp"

include "caffe/util/io.hpp"

include "caffe/blob.hpp"

include "/usr/local/include/opencv2/core/core.hpp"

include "/usr/local/include/opencv2/highgui/highgui.hpp"

using namespace caffe;

int main(int argc, char** argv) {
::google::InitGoogleLogging(argv[0]);
if (argc < 2) {
LOG(ERROR) << "Usage: train_net solver_proto_file [resume_point_file]";
return 0;
}

SolverParameter solver_param;
ReadProtoFromTextFile(argv[1], &solver_param);

LOG(INFO) << "Starting Optimization";
SGDSolver solver(solver_param);
if (argc == 3) {
LOG(INFO) << "Resuming from " << argv[2];
solver.Solve(argv[2]);
} else {
solver.Solve();
}
LOG(INFO) << "Optimization Done.";

return 0;
}

delploy.prototxt:
name: "TestMemoryDataLayer"
layers
{
name: "data"
type: MEMORY_DATA
top: "data"
top: "label"
memory_data_param
{
batch_size: 1
channels: 3
height: 224
width: 224
}
transform_param
{
crop_size: 224
mirror: true
#mean_file: "/to/caffe-dev/data/ilsvrc12/imagenet_mean.binaryproto"
}
include: { phase: TRAIN }
}

layers
{
name: "data"
type: MEMORY_DATA
top: "data"
top: "label"
memory_data_param
{
batch_size: 1
channels: 3
height: 224
width: 224
}
transform_param
{
crop_size: 224
mirror: true
#mean_file: "/to/caffe-dev/data/ilsvrc12/imagenet_mean.binaryproto"
}
include: { phase: TEST }
}
layers {
name: "conv1"
type: CONVOLUTION
bottom: "data"
top: "conv1"
convolution_param {
num_output: 96
kernel_size: 11
stride: 4
}
}
layers {
name: "relu1"
type: RELU
bottom: "conv1"
top: "conv1"
}
layers {
name: "pool1"
type: POOLING
bottom: "conv1"
top: "pool1"
pooling_param {
pool: MAX
kernel_size: 3
stride: 2
}
}
layers {
name: "norm1"
type: LRN
bottom: "pool1"
top: "norm1"
lrn_param {
local_size: 5
alpha: 0.0001
beta: 0.75
}
}
layers {
name: "conv2"
type: CONVOLUTION
bottom: "norm1"
top: "conv2"
convolution_param {
num_output: 256
pad: 2
kernel_size: 5
group: 2
}
}
layers {
name: "relu2"
type: RELU
bottom: "conv2"
top: "conv2"
}
layers {
name: "pool2"
type: POOLING
bottom: "conv2"
top: "pool2"
pooling_param {
pool: MAX
kernel_size: 3
stride: 2
}
}
layers {
name: "norm2"
type: LRN
bottom: "pool2"
top: "norm2"
lrn_param {
local_size: 5
alpha: 0.0001
beta: 0.75
}
}
layers {
name: "conv3"
type: CONVOLUTION
bottom: "norm2"
top: "conv3"
convolution_param {
num_output: 384
pad: 1
kernel_size: 3
}
}
layers {
name: "relu3"
type: RELU
bottom: "conv3"
top: "conv3"
}
layers {
name: "conv4"
type: CONVOLUTION
bottom: "conv3"
top: "conv4"
convolution_param {
num_output: 384
pad: 1
kernel_size: 3
group: 2
}
}
layers {
name: "relu4"
type: RELU
bottom: "conv4"
top: "conv4"
}
layers {
name: "conv5"
type: CONVOLUTION
bottom: "conv4"
top: "conv5"
convolution_param {
num_output: 256
pad: 1
kernel_size: 3
group: 2
}
}
layers {
name: "relu5"
type: RELU
bottom: "conv5"
top: "conv5"
}
layers {
name: "pool5"
type: POOLING
bottom: "conv5"
top: "pool5"
pooling_param {
pool: MAX
kernel_size: 3
stride: 2
}
}
layers {
name: "fc6"
type: INNER_PRODUCT
bottom: "pool5"
top: "fc6"
inner_product_param {
num_output: 4096
}
}
layers {
name: "relu6"
type: RELU
bottom: "fc6"
top: "fc6"
}
layers {
name: "drop6"
type: DROPOUT
bottom: "fc6"
top: "fc6"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "fc7"
type: INNER_PRODUCT
bottom: "fc6"
top: "fc7"
inner_product_param {
num_output: 4096
}
}
layers {
name: "relu7"
type: RELU
bottom: "fc7"
top: "fc7"
}
layers {
name: "drop7"
type: DROPOUT
bottom: "fc7"
top: "fc7"
dropout_param {
dropout_ratio: 0.5
}
}
layers {
name: "fc8"
type: INNER_PRODUCT
bottom: "fc7"
top: "fc8"
inner_product_param {
num_output: 1000
}
}
layers {
name: "accuracy"
type: ACCURACY
bottom: "fc8" #"prob" #"fc8-new"
bottom: "label"
top: "accuracy"
#top_k: 5
include: { phase: TEST }
}

layers {
name: "loss"
type: SOFTMAX_LOSS
bottom: "fc8" #predicted label, used in forward-pass
bottom: "label"
top: "loss"
}

solver.prototct:
net: "deploy.prototxt"
test_iter: 1000
test_interval: 1000
base_lr: 0.01
lr_policy: "step"
gamma: 0.1
stepsize: 100000
display: 20
max_iter: 450000
momentum: 0.9
weight_decay: 0.0005
snapshot: 10000
snapshot_prefix: "caffenet_train"
solver_mode: GPU