Tsfresh: Allow NaN or None values to be passed in, and silently ignored

Created on 2 Dec 2016 · 3Comments · Source: blue-yonder/tsfresh

In my DataFrames, I oftentimes find it very reasonable to have quite a few NaN or None values.

In order to run tsfresh, I just set all those values to 0, which is... not ideal. Even imputing missing values would not work particularly well for several of my use cases.

Yet, it seems (from a super naive outsider's perspective), like this is filtering that tsfresh could do relatively easily itself.

When it grabs each time_series, it can simply remove or categorically ignore NaN/None, and compute features on the values that do exist. This makes my life easier when, say, one customer signs up a month after another customer, and thus has missing values for that month.

Again, super naive outsider perspective here, I know this might be impossible. But if it is possible, I'd love to add in that bit of filtering!

Source

ClimbsRocks

Most helpful comment

This was a design decision. tsfresh will not tinker with the input time series data by for example imputing values or dropping NAs.

In many cases doesn't it make sense to treat nans as nans, rather than requiring imputation? For example, features like mean, max and min are more informative if the nans are simply ignored when computing the value.

tompollard on 10 Sep 2019

👍2

All 3 comments

You can solve this by calling df.dropna(axis=1, inplace=True) on the data frame that contains the time series in tsfresh format

Afterwards, you can just pass df to tsfresh

MaxBenChrist on 2 Dec 2016

👍1

This was a design decision. tsfresh will not tinker with the input time series data by for example imputing values or dropping NAs.

Reason behind this: In data science projects, NAs should be handled with special care, often they contain a lot of information. We don't want our packages to silently remove those informations by dropping it. This is way we are not imputing the input data.

MaxBenChrist on 2 Dec 2016

👍2