Rust-rocksdb: Comparator error when re-opening db

Created on 18 May 2020  ·  7Comments  ·  Source: rust-rocksdb/rust-rocksdb

I found that I get the below error when re-opening a DB that was created a certain way.
Invalid argument: leveldb.BytewiseComparator: does not match existing comparator

I believe it is because when opening a DB without specifying CF's, the default CF is created without using the custom Options. And I think the problem might be here https://github.com/rust-rocksdb/rust-rocksdb/blob/master/src/db.rs#L214 in the code.

A possible fix would be to move the check https://github.com/rust-rocksdb/rust-rocksdb/blob/master/src/db.rs#L219 up before the empty check and just add default to the cf string list.

The steps to reproduce:

  1. open a DB with a comparator set in the DB Options, but without specifying any column families
  2. add more CF's with the Options that have a custom comparator.
  3. close DB
  4. re-open the db with Options that have the custom comparator.

Most helpful comment

@zach-schoenberger Take a look here.

All 7 comments

Thanks for the report!

I'm going to take a look at this issue. Unfortunately, I suppose that your suggestion isn't going to work because all column families (not only the default one) use default options:
https://github.com/rust-rocksdb/rust-rocksdb/blob/master/src/db.rs#L123

It seems that a comparator should be provided for each column family separately. It seems we need to provide a slightly different API for such cases.

Thanks @DarkEld3r. I had the same thought. Honestly I was confused why the comparator was working at all for the exact reason you gave. I figured it had a chance of fixing it because of whatever magic was going on when it traversed the else side of the conditional.

Honestly I was confused why the comparator was working at all for the exact reason you gave.

I suppose you have created column families with create_cf call and used the same options that you have used for DB? These options are passed to rocksdb, so I guess it is the reason why everything is working as intended.

@zach-schoenberger What method do you use to open DB on the step 4 ?

@zach-schoenberger Take a look here.

@aleksuss Thanks, I have totally missed the open_cf_descriptors function. I can confirm that the following works for me:

let path = "...";

fn comparator(left: &[u8], right: &[u8]) -> Ordering {
    right.cmp(left)
}

let mut options = Options::default();
options.create_if_missing(true);
options.set_comparator("comparator", comparator);

{
    let mut db = DB::open(&options, &path).expect("Unable to create a DB");
    db.create_cf("cf1", &options).expect("Unable ");
}

{
    let mut cf_options = Options::default();
    cf_options.set_comparator("comparator", comparator);
    let descriptors = vec![ColumnFamilyDescriptor::new("cf1", cf_options)];

    let _ = DB::open_cf_descriptors(&options, &path, descriptors);
}

Interesting. I definitely would not have thought that at first. I'll update to use the above. Definitely wasn't intuitive to me. I was using DB::open_cf() and DB::open(). I did get my original scenario to appear to work by just including the "default" cf name with an initial DB::open_cf instead of DB::open, but i like the above much more since it appears to honor the options.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

rrichardson picture rrichardson  ·  7Comments

alex88 picture alex88  ·  7Comments

iSynaptic picture iSynaptic  ·  11Comments

cetra3 picture cetra3  ·  9Comments

yiyanwannian picture yiyanwannian  ·  6Comments