Orientdb: Big data support for hash index

Created on 22 Oct 2013  ·  3Comments  ·  Source: orientechnologies/orientdb

Current hash index implementation requires only single i/o for read and at most 3 i/os for write case but we still suffer from random I/Os overhead. Average random I/O takes 20 ms it is hell slow. Current write cache optimization amortize this overhead but we still will suffer from it in case of huge insertions. To avoid this overhead good to have optimizations which were applied for LSM tries. In nutshed LSM tree is are sorted dictionary, one instance of which in memory and second on the disk these instance are merged in background using very big data chunks, so we will have not 3 I/O s for write but about 3/16 IOs for single write which is much faster if we also take into account that additional write cache optimizations will be applied we will have very very fast index implementation. The additional optimization is usage of bloom filter, but not counting one which is total server resource wasting.

But it is resource consuming too, 4 months for single person and about 2.5 months for two persons. But the outcome should be really valuable.

This optimization should be implemented after https://github.com/orientechnologies/orientdb/issues/1756 issue.

enhancement

Most helpful comment

@saeedtabrizi also WiredTiger do not use transactions and it makes implementations of such things much simpler we now concentrate on Fractal Indexes which have good potential for integration in transaction based systems

All 3 comments

@laa based on this report , i think the LSM tree implementation is a most valuable step to grow orientdb .

@saeedtabrizi that is this report a bit cheating, it does not take in account cases when LSM Tree has many levels, write amplification so enormous that all write stops there.

@saeedtabrizi also WiredTiger do not use transactions and it makes implementations of such things much simpler we now concentrate on Fractal Indexes which have good potential for integration in transaction based systems

Was this page helpful?
0 / 5 - 0 ratings