Darkflow: AssertionError: Over-read bin/yolo.weights

[UPDATE]
It seems that the PJ Reddie files are not fully compatible, so I have to download the pre built weights from the provided link in the README: https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU

The problem was that I had to do this server-side, so after digging a bit, the simplest solution was to use the Firefox addon gcurl that mimic the direct download command:

cd darkflow/bin && \
    curl --header 'Host: doc-0k-88-docs.googleusercontent.com' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3' --header 'Referer: https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU' --header 'Connection: keep-alive' --header 'Upgrade-Insecure-Requests: 1' 'https://doc-0k-88-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/etft348j4r0b37n9p1jkokvbfagiimk2/1491782400000/16010642207042931662/*/0B1tW_VtY7onieS0zMzdkSG11OW8?e=download' -o 'tiny-yolo-v1.1.weights' -L && \
    curl --header 'Host: doc-08-88-docs.googleusercontent.com' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3' --header 'Referer: https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU' --header 'Connection: keep-alive' --header 'Upgrade-Insecure-Requests: 1' 'https://doc-08-88-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/s29gvf1n83pqr37pk2l7t2cvcjnssd8s/1491782400000/16010642207042931662/*/0B1tW_VtY7oniTjM3YUxlRHpDVW8?e=download' -o 'tiny-yolo-voc.weights' -L && \
    curl --header 'Host: doc-10-88-docs.googleusercontent.com' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3' --header 'Referer: https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU' --header 'Connection: keep-alive' --header 'Upgrade-Insecure-Requests: 1' 'https://doc-10-88-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/fmtsb8ruqk7a4tg5p98khj6qp4fq3p6i/1491782400000/16010642207042931662/*/0B1tW_VtY7onicFlqclhnRGlodGM?e=download' -o 'yolo-full.weights' -L && \
    curl --header 'Host: doc-0k-88-docs.googleusercontent.com' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3' --header 'Referer: https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU' --header 'Connection: keep-alive' --header 'Upgrade-Insecure-Requests: 1' 'https://doc-0k-88-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/knshmif3mk3n4ogtq9p77nf2f5v3dr0d/1491782400000/16010642207042931662/*/0B1tW_VtY7onibmdQWE1zVERxcjQ?e=download' -o 'yolo-tiny.weights' -L && \
    curl --header 'Host: doc-0c-88-docs.googleusercontent.com' --header 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: it-IT,it;q=0.8,en-US;q=0.5,en;q=0.3' --header 'Referer: https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU' --header 'Connection: keep-alive' --header 'Upgrade-Insecure-Requests: 1' 'https://doc-0c-88-docs.googleusercontent.com/docs/securesc/ha0ro937gcuc7l7deffksulhg5h7mbp1/ggut18kr6ubhoc8btg5hf6f32stoh50n/1491782400000/16010642207042931662/*/0B1tW_VtY7oniZGlkLTh5YVl1WWs?e=download' -o 'yolo.weights' -L

Btw I'm not sure if those links will work forever, after the download my bin/ folder looks like:

# cd darkflow/bin/
# ls -lh
total 1.6G
-rw-r--r-- 1 root root 104M Apr 10 00:20 tiny-yolo-v1.1.weights
-rw-r--r-- 1 root root  61M Apr 10 00:20 tiny-yolo-voc.weights
-rw-r--r-- 1 root root 1.1G Apr 10 00:20 yolo-full.weights
-rw-r--r-- 1 root root 173M Apr 10 00:21 yolo-tiny.weights
-rw-r--r-- 1 root root 258M Apr 10 00:21 yolo.weights

Files on official YOLO are always renewed as he experiments new configs, you have to check if there is corresponding .cfg released. weights file I uploaded are older versions and is compatible with the .cfg in this repo.

@thtrieu thanks, I solved in the meanwhile using your files!

this error is caused by the mismatch between the cfg and binary weight file. Here is the related code which causes it.
At the beginning of flow, it will first try to initialize the TFNet from the given cfg and bin specified by the user in the command line.

# in darkflow/utils/loader.py

class weights_walker(object):
  ...
  self.size = os.path.getsize(path)

this walker is created and called function walk
This walk will read weights from the bin file for each layer with the exact amount of bytes according to the cfg. Before actually reading it, it does a check on if the bytes to read is less than the remaining bytes in the file. Here is the code for that

    def walk(self, size):

        if self.eof: return None
        end_point = self.offset + 4 * size

        assert end_point <= self.size, \
        'Over-read {}'.format(self.path)

here the end_point is where the file descriptor cursor would be located when it finishes reading the required amount from the binary file for the current layer's weights. The self.size is the total size of the binary file. You can see how the comparison of these two variables will indicate if there is a conflict between the cfg and bin file.

Based on the discussions above, I have downloaded the latest cfg and binary file from the yolov2 website. However, it still doesn't work and I am now sure why. Hope I can write down a successful story later on.

@chadrick-kwag Hi, did you solve it?

@AlexeyAB unfortunately no. I had to just switch to tiny-yolo-voc which worked without a glitch. My project is simple so tiny-yolo suits me as well. Sorry to tell you the bad news :(

@chadrick-kwag Ok thanks. Do you use tiny-yolo-voc from https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU or from https://pjreddie.com/darknet/yolo/ ?

@AlexeyAB I used the one from pjreddie . FYI I am using the one that I downloaded about 6 months ago. I hope it still works.

Anyone has the direct download link to the cfg file that corresponds to the the weight file (yolo.weights) from the above Google Drive link?

@shailensobhee I tried all possible combinations of yolov2-voc cfg and weight files between darkflow and darknet repo. There seems to be no single match.

@thtrieu @AlexeyAB Any idea ?

you can fix this issue by changing the offset in the loader.py in your darkflow directory.
recommended decreasing the offset to 16 then test again.
GL

Drive

Did you get the link? If not, here's what I've been using: https://drive.google.com/drive/folders/0B1tW_VtY7onidEwyQ2FtQVplWEU