Xxhash: Segfault in XXH3_64bits_reset_withSeed with x86 Visual Studio 2015

Could you update the xxHash module ?
It seems to be an "old" version.
Since then, an issue related to AVX2 alignment has been fixed.
Not sure if it's the one that happens here, but just in case ...

Use xxHash dev branch:

• https://github.com/ifduyue/appveyor-x86/tree/with-xxHash-dev
• log: https://ci.appveyor.com/project/duyue/appveyor-x86/builds/33647744

Use xxHash master branch:

• https://github.com/ifduyue/appveyor-x86/tree/with-xxHash-master
• log: https://ci.appveyor.com/project/duyue/appveyor-x86/builds/33647768

Use xxHash v0.7.3:

• https://github.com/ifduyue/appveyor-x86/tree/with-xxHash-v0.7.3
• log: https://ci.appveyor.com/project/duyue/appveyor-x86/builds/33647777

All these three have the same Command exited with code -1073741819

Are you sure it doesn't have to do with calling XXH3_64bits_digest on an uninitialized state?

@Cyan4973 We should probably switch to calloc for creating states.

Can confirm a crash with -fsanitize=address:

#define XXH_STATIC_LINKING_ONLY
#include <stdio.h>
#include <xxhash.h>

int main(void)
{
    XXH3_state_t *state = XXH3_createState();
    printf("%llx\n", (unsigned long long)XXH3_64bits_digest(state));
    XXH3_freeState(state);
}

AddressSanitizer:DEADLYSIGNAL
=================================================================
==20253==ERROR: AddressSanitizer: SEGV on unknown address 0xfafafbfafafafaf (pc 0x0063cc51ed88 bp 0x007fd51ed3d0 sp 0x007fd51ed0e0 T0)
==20253==The signal is caused by a READ memory access.
    #0 0x63cc51ed84 in XXH3_scrambleAcc_neon /data/data/com.termux/files/home/xxhash-clean/xxh3.h:1371:35
    #1 0x63cc524644 in XXH3_consumeStripes /data/data/com.termux/files/home/xxhash-clean/xxh3.h:2009:9
    #2 0x63cc51f808 in XXH3_digest_long /data/data/com.termux/files/home/xxhash-clean/xxh3.h:2112:9
    #3 0x63cc51f1c0 in XXH3_64bits_digest /data/data/com.termux/files/home/xxhash-clean/xxh3.h:2141:9
    #4 0x63cc527adc in main /data/data/com.termux/files/home/xxcrash.c:8:22
    #5 0x6fdaeb4844 in __libc_init (/apex/com.android.runtime/lib64/bionic/libc.so+0x7d844)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /data/data/com.termux/files/home/xxhash-clean/xxh3.h:1371:35 in XXH3_scrambleAcc_neon
==20253==ABORTING

Normally it doesn't show up because most Linux implementations use mmap for malloc which zeros by default. -fsanitize=address will simulate a "dirty malloc" which shows the crash.

Undefined behavior on your side, but we could improve this a bit with calloc because it accepts a zeroed state.

@easyaspi314 you're right, it's my fault, it's a use before init error. Reset right after creating a state and everything is fine.

Sorry to bother again, thanks in advance!

Actually I'm developing python-xxhash and trying to support xxh3.
When it's v0.7.3, always error at one of test_xxh3_128_reset, test_xxh3_128_reset_more, test_xxh3_64_reset and test_xxh3_64_seed_reset, only 32-bit python3.5/3.6/3.7/3.8 on appveyor windows image.
When it's dev branch, all tests pass.

• v0.7.3 log: https://ci.appveyor.com/project/duyue/python-xxhash/builds/33648972
• dev branch log: https://ci.appveyor.com/project/duyue/python-xxhash/builds/33648936
• python-xxhash xxh3 branch: https://github.com/ifduyue/python-xxhash/tree/xxh3

@Cyan4973 @easyaspi314 Anything special between v0.7.3 and dev?

Well v0.7.3's state is not movable due to a pointer issue. Not sure why it only crashes on 32-bit, though. (#365, fix #366).

For the reference, it goes like this

XXH3_state_t *s1 = XXH3_createState(), *s2 = XXH3_createState;
XXH3_64bits_reset_withSeed(s1, seed);
XXH3_copyState(s2, s1);
XXH3_freeState(s1);
// s2 has a wild pointer to the internal state of s1 in v0.7.3, fixed in dev

Or simplifying the wrappers, the bug is a simple use-after-free:

typedef struct XXH3_state_s {
    unsigned char customSecret[192];
    const unsigned char *secret;
} XXH3_state_t;

XXH3_state_t *s1 = malloc(sizeof(XXH3_state_t)), *s2 = malloc(sizeof(XXH3_state_t));
s1->secret = &s1->customSecret[0];
memcpy(s2, s1, sizeof(XXH3_state_t));
assert(s2->secret == &s1->customSecret[0]);
free(s1);

For a workaround for v0.7.3, you can do this:

XXH3_copyState(s2, s1);
#if XXH_VERSION_NUMBER < 704
// v0.7.3 and earlier have a bug where states reset with a seed
// will have a wild pointer to the original state when copied,
// causing a use-after-free if the original is freed.
if (s2->secret == &s1->customSecret[0])
    s2->secret = &s2->customSecret[0];
#endif 

@easyaspi314 Thanks for your detailed reply. Very helpful.