Xgboost: High memory consumption in python xgboost

Created on 2 Apr 2020 · 3Comments · Source: dmlc/xgboost

I'm working on python AutoML package and one of my users reported very high memory usage while using xgboost.

I've made an investigation to show memory consumption by xgboost. You can find the notebook here. From the code, you can see that the model allocate over 7GB of RAM memory. When I save the model to hard disk (5 kB !) and then load it back I can save a huge amount of RAM.

For me, it look's like xgboost is storing the copy of data in its structure? Am I right?

Is there any way to slim down memory usage by xgboost? Do you think that saving model to the hard drive and then loading it back is way to handle this issue?

Source

pplonski

Most helpful comment

@pplonski, we implemented reducing memory consumption on CPU also in this PR https://github.com/dmlc/xgboost/pull/5334, but for 'hist' method only. It's included in master only for now, but I hope it will be a part of the future release.

memory, Kb | Airline | Higgs1m |
-- | -- | -- |
Before | 28311860 | 1907812 |
https://github.com/dmlc/xgboost/pull/5334 | 16218404 | 1155156 |
reduced: | 1.75 | 1.65 |

Agree with @trivialfis, there are many things to do in the area.

SmirnovEgorRu on 2 Apr 2020

👍2

All 3 comments

@pplonski We are trying to eliminate the copy for histogram algorithm. It's a working in progress. For GPU it is mostly done: https://github.com/dmlc/xgboost/pull/5420 https://github.com/dmlc/xgboost/pull/5465

CPU still has some more work to do.

trivialfis on 2 Apr 2020

👍1

memory, Kb | Airline | Higgs1m |
-- | -- | -- |
Before | 28311860 | 1907812 |
https://github.com/dmlc/xgboost/pull/5334 | 16218404 | 1155156 |
reduced: | 1.75 | 1.65 |

Agree with @trivialfis, there are many things to do in the area.

SmirnovEgorRu on 2 Apr 2020

👍2

Hi, I have recently faced a similar high memory problem with xgboost. I am using 'gpu_hist' for training.

I notice large system memory spikes when train() method is executed, which leads to my jupyter kernel crashing.

Is it correct to say that Xgboost is making a copy of my data in system RAM (even when I'm using 'gpu_hist')?
I was under the assumption that xgboost loads the entire training data to the GPU. Is that also incorrect?