Linux: Workqueue: kmmcd mmc_rescan crash with Raspbian kernel 4.4.38-v7

Created on 8 Mar 2017  ·  29Comments  ·  Source: raspberrypi/linux

My Raspberry Pi 3 runs unstable and crashes every few days. Before the crash /var/log/messages shows the following error repeatedly with a separation of a few minutes:

Jan 25 18:36:22 raspberrypi kernel: [1103404.218839] kworker/u8:2 D 805b770c 0 20119 2 0x00000000
Jan 25 18:36:22 raspberrypi kernel: [1103404.218886] Workqueue: kmmcd mmc_rescan
Jan 25 18:36:22 raspberrypi kernel: [1103404.218923] [<805b770c>] (__schedule) from [<805b7c84>] (schedule+0x50/0xa8)
Jan 25 18:36:22 raspberrypi kernel: [1103404.218945] [<805b7c84>] (schedule) from [<8046d568>] (__mmc_claim_host+0xb8/0x1cc)
Jan 25 18:36:22 raspberrypi kernel: [1103404.218974] [<8046d568>] (__mmc_claim_host) from [<8046d6ac>] (mmc_get_card+0x30/0x34)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219006] [<8046d6ac>] (mmc_get_card) from [<80475780>] (mmc_sd_detect+0x2c/0x80)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219030] [<80475780>] (mmc_sd_detect) from [<8046fcbc>] (mmc_rescan+0xc8/0x324)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219057] [<8046fcbc>] (mmc_rescan) from [<8003c82c>] (process_one_work+0x154/0x458)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219080] [<8003c82c>] (process_one_work) from [<8003cb84>] (worker_thread+0x54/0x500)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219100] [<8003cb84>] (worker_thread) from [<80042860>] (kthread+0xec/0x104)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219122] [<80042860>] (kthread) from [<8000fbc8>] (ret_from_fork+0x14/0x2c)
Jan 25 18:36:23 raspberrypi rsyslogd-2007: action 'action 17' suspended, next retry is Wed Jan 25 18:37:53 2017 [try http://www.rsyslog.com/e/2007 ]

I am running Raspbian 8.0 (Jessie) with kernel

Linux raspberrypi 4.4.38-v7+ #938 SMP Thu Dec 15 15:22:21 GMT 2016 armv7l GNU/Linux

I am also discussing this at

http://raspberrypi.stackexchange.com/q/61050/58211.

I have also reported this at https://github.com/raspberrypi/firmware/issues/731 but I think it fits better here. Unfortunately I cannot move the issue, so its kind of a duplicate, now. Maybe it should be deleted at firmware.

Bug

All 29 comments

  1. Are you over-clocking the RPI?
  2. Have you tried a different SD card? I know these cheap cards can wear out or break after being used for awhile. And they are relatively cheap so it might be worth trying a replacement.

I doubt this is a firmware issue. It could actually be a combo of a bad card plus a bug in the driver. It might well be that the driver thinks you've removed the card even though you haven't. And since pulling out the SD card while the RPI is running is a no-no, I could see why that might not have been tested well.

I know some effort is being made to replace this driver with an upstream version. Also 4.9 is now considered the stable version of the kernel on the RPI.

  1. No, I didn't change the clock speed.
  2. Its is a high end 40 EUR Samsung MicroSDXC 128GB EVO Plus UHS-I Grade 1 Class 10 which two 8 GB partitions for /root and /home.

I have completely write tested the card with badblocks -w with the 5 different patterns before usage and found not a single bad block. A recent badblocks read test didn't show any errors either. I assume the card is perfect.

I just ordered another Pi 3 and a SanDisk Ultra 64GB SDXC (80 MB/Sek) Class 10 to rule out hardware issues.

You missed of the header of that kernel, but a stall at that point looks like a symptom that something else hasn't released the MMC device, possibly because something failed during a previous operation. The sdhost driver will log error messages, so in the absence of those I'm leaning towards the ARM<->VPU mailbox stalling while trying to configure the clock for the SD interface.

Errors like this are often the sign of an inadequate power supply. What does "vcgencmd get_throttled" report on your system after some typical activity?

Also 4.9 is now considered the stable version of the kernel on the RPI.

No - it's still the next version until we make a Raspbian release using it.

That is puzzling then...

Personally, I would have tried a very, very cheap card first to rule out hardware issue. Like the kind you get for $5 in the US. I've actually collected a whole bunch of those over the years.

What I meant by cheap cards, is the all these SD cards are cheap. I don't care how much you pay for them, they are essentially designed for the camera/camcorder/phone market so they don't typically get the abuse of something that's may well be on all the time.

@pelwell I actually see mailbox stalls from time to time and I'm not 100% sure it's the power supply since I can sometimes see it even if the RPI is connected to a bench variable power supply that can do 5 amps.

I've found that especially in 64 bit, the RPI is unstable at 1.2GHz. Setting the max at 1.0GHz seems to help. But I also haven't ruled out a firmware issue.

@pelwell What kernel header to you need? I am using the default Raspbian 8 Jessie kernel
Linux raspberrypi 4.4.34-v7+ #930 SMP Wed Nov 23 15:20:41 GMT 2016 armv7l GNU/Linux.

My power supply seems stable and sufficient as concluded from throttled=0x20000.
I obtained this value during a stress test that raised my CPU temperature to 80°C.

By the way if you google for kmmcd mmc_rescan you will find other people reporting this issue, too.

Meanwhile I noticed that this problem is triggered by the chromium-browser (Version 51.0.2704.91 Built on Ubuntu 14.04, running on Raspbian 8.0) when several sites are open.

Please update to the latest kernel which may help, 4.9 is now the release version. Please report back the results.

Unfortunately I still see the same problem after the upgrade.
Meanwhile I am using:
Linux raspberrypi 4.9.24-v7+ #993 SMP Wed Apr 26 18:01:23 BST 2017 armv7l GNU/Linux
and Chromium Version 56.0.2924.84 Built on Ubuntu 14.04, running on Raspbian 8.0

I never saw this before, and now saw it a bunch of times running 4.9.80-v7+ during a very fast wget download. I guess this is what made the Raspberry Pi 2 B temporarily unresponsive, but it didn't crash and the download didn't fail.

[39933.969207] INFO: task kworker/2:2:5185 blocked for more than 120 seconds.
[39933.969232]       Tainted: G         C      4.9.80-v7+ #1098
[39933.969236] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39933.969247] kworker/2:2     D    0  5185      2 0x00000000
[39933.969295] Workqueue: events_freezable mmc_rescan
[39933.969347] [<8072252c>] (__schedule) from [<80722a54>] (schedule+0x50/0xa8)
[39933.969363] [<80722a54>] (schedule) from [<805c30d8>] (__mmc_claim_host+0xb8/0x1cc)
[39933.969377] [<805c30d8>] (__mmc_claim_host) from [<805c321c>] (mmc_get_card+0x30/0x34)
[39933.969394] [<805c321c>] (mmc_get_card) from [<805cbc70>] (mmc_sd_detect+0x2c/0x80)
[39933.969410] [<805cbc70>] (mmc_sd_detect) from [<805c5ee0>] (mmc_rescan+0x1c8/0x38c)
[39933.969429] [<805c5ee0>] (mmc_rescan) from [<801367f8>] (process_one_work+0x158/0x454)
[39933.969450] [<801367f8>] (process_one_work) from [<80136b58>] (worker_thread+0x64/0x5b8)
[39933.969464] [<80136b58>] (worker_thread) from [<8013ca74>] (kthread+0x10c/0x124)
[39933.969483] [<8013ca74>] (kthread) from [<80108154>] (ret_from_fork+0x14/0x20)

Does this still happen on the latest 4.14 kernel?

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.

unfortunately it is:

Jul 23 04:03:22 raspberrypi kernel: [114645.769828] INFO: task kworker/0:0:18491 blocked for more than 120 seconds.
Jul 23 04:03:22 raspberrypi kernel: [114645.769846]       Tainted: G         C      4.14.52+ #1123
Jul 23 04:03:22 raspberrypi kernel: [114645.769850] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 23 04:03:22 raspberrypi kernel: [114645.769857] kworker/0:0     D    0 18491      2 0x00000000
Jul 23 04:03:22 raspberrypi kernel: [114645.769918] Workqueue: events_freezable mmc_rescan
Jul 23 04:03:22 raspberrypi kernel: [114645.769962] [<c06471c8>] (__schedule) from [<c0647698>] (schedule+0x50/0xa8)
Jul 23 04:03:22 raspberrypi kernel: [114645.769998] [<c0647698>] (schedule) from [<c04c9aa8>] (__mmc_claim_host+0x98/0x210)
Jul 23 04:03:22 raspberrypi kernel: [114645.770017] [<c04c9aa8>] (__mmc_claim_host) from [<c04c9c50>] (mmc_get_card+0x30/0x34)
Jul 23 04:03:22 raspberrypi kernel: [114645.770039] [<c04c9c50>] (mmc_get_card) from [<c04d2be8>] (mmc_sd_detect+0x20/0x70)
Jul 23 04:03:22 raspberrypi kernel: [114645.770053] [<c04d2be8>] (mmc_sd_detect) from [<c04cc7f8>] (mmc_rescan+0x288/0x490)
Jul 23 04:03:22 raspberrypi kernel: [114645.770072] [<c04cc7f8>] (mmc_rescan) from [<c0039350>] (process_one_work+0x11c/0x398)
Jul 23 04:03:22 raspberrypi kernel: [114645.770087] [<c0039350>] (process_one_work) from [<c0039608>] (worker_thread+0x3c/0x544)
Jul 23 04:03:22 raspberrypi kernel: [114645.770106] [<c0039608>] (worker_thread) from [<c003f504>] (kthread+0x120/0x15c)
Jul 23 04:03:22 raspberrypi kernel: [114645.770130] [<c003f504>] (kthread) from [<c000fe6c>] (ret_from_fork+0x14/0x28)

and my kernel is:

Linux raspberrypi 4.14.52+ #1123 Wed Jun 27 17:05:32 BST 2018 armv6l GNU/Linux

OK, we need to fathom out what is going on that might cause this. Since this appears to be very uncommon, is there anything about your setup that could be regarded as a little unusual that may promote it? What are you doing when it happens?

I also have the same problem, chromium-browser that "crashes" after a while (the display is frozen, but the logs from the pi suggests it was still "on")

What happens is that because of this after a while the pi starts killing tasks, and the pi is in somewhat of a zombie state. This is a pi used for display in the cafeteria, so I usually just reboot it instead of trying to connect to it through SSH to see what's going on. Next time it crashes I'll try to see wether the OS is still responsive or if the pi is really in some sort of zombie state.

The pi crashed around Jul 24 17:20:

Jul 24 17:12:45 raspberrypi kernel: [32438.191330] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 17:12:46 raspberrypi kernel: [32438.191334] kworker/3:1     D    0  7414      2 0x00000000
Jul 24 17:12:47 raspberrypi kernel: [32438.191352] Workqueue: events_freezable mmc_rescan
Jul 24 17:12:49 raspberrypi kernel: [32438.191372] [<8079bfd8>] (__schedule) from [<8079c650>] (schedule+0x50/0xa8)
Jul 24 17:12:50 raspberrypi kernel: [32438.191380] [<8079c650>] (schedule) from [<80617df0>] (__mmc_claim_host+0xb8/0x1cc)
Jul 24 17:12:50 raspberrypi kernel: [32438.191387] [<80617df0>] (__mmc_claim_host) from [<80617f34>] (mmc_get_card+0x30/0x34)
Jul 24 17:12:50 raspberrypi kernel: [32438.191393] [<80617f34>] (mmc_get_card) from [<80620b30>] (mmc_sd_detect+0x20/0x74)
Jul 24 17:12:50 raspberrypi kernel: [32438.191400] [<80620b30>] (mmc_sd_detect) from [<8061a7fc>] (mmc_rescan+0x1c8/0x394)
Jul 24 17:12:50 raspberrypi kernel: [32438.191411] [<8061a7fc>] (mmc_rescan) from [<801375c4>] (process_one_work+0x158/0x454)
Jul 24 17:12:50 raspberrypi kernel: [32438.191420] [<801375c4>] (process_one_work) from [<80137924>] (worker_thread+0x64/0x5b8)
Jul 24 17:12:50 raspberrypi kernel: [32438.191430] [<80137924>] (worker_thread) from [<8013d99c>] (kthread+0x13c/0x16c)
Jul 24 17:12:50 raspberrypi kernel: [32438.191439] [<8013d99c>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
Jul 24 17:20:31 raspberrypi dhcpcd[328]: wlan0: carrier lost
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1261 '/devices/platform/regulatory.0' is taking a long time
Jul 24 17:20:31 raspberrypi systemd[1]: systemd-journald.service: Watchdog timeout (limit 3min)!
Jul 24 17:20:31 raspberrypi systemd[1]: systemd-journald.service: Killing process 89 (systemd-journal) with signal SIGABRT.
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1261 '/devices/platform/regulatory.0' killed
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7685] terminated by signal 9 (Killed)
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7685] failed while handling '/devices/platform/regulatory.0'
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1262 '/devices/platform/regulatory.0' is taking a long time
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1262 '/devices/platform/regulatory.0' killed
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7712] terminated by signal 9 (Killed)
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7712] failed while handling '/devices/platform/regulatory.0'
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1263 '/devices/platform/regulatory.0' is taking a long time
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1263 '/devices/platform/regulatory.0' killed
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7735] terminated by signal 9 (Killed)
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7735] failed while handling '/devices/platform/regulatory.0'
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1264 '/devices/platform/regulatory.0' is taking a long time
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1264 '/devices/platform/regulatory.0' killed
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7743] terminated by signal 9 (Killed)
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: worker [7743] failed while handling '/devices/platform/regulatory.0'
Jul 24 17:20:31 raspberrypi systemd-udevd[133]: seq 1265 '/devices/platform/regulatory.0' is taking a long time
Jul 24 17:20:31 raspberrypi systemd-timesyncd[259]: Timed out waiting for reply from 5.148.175.134:123 (2.debian.pool.ntp.org).
Jul 24 17:20:31 raspberrypi systemd-udevd[7752]: Process '/sbin/crda' failed with exit code 249.
Jul 24 17:20:31 raspberrypi systemd-udevd[7752]: Process '/sbin/crda' failed with exit code 249.
Jul 24 17:20:31 raspberrypi systemd-udevd[7752]: Process '/sbin/crda' failed with exit code 249.
Jul 24 17:20:31 raspberrypi systemd-udevd[7752]: Process '/sbin/crda' failed with exit code 249.
Jul 24 17:20:31 raspberrypi systemd-udevd[7752]: Process '/sbin/crda' failed with exit code 249.
Jul 24 17:20:31 raspberrypi systemd-udevd[7752]: Process '/sbin/crda' failed with exit code 249.
Jul 24 17:20:31 raspberrypi systemd-udevd[7752]: Process '/sbin/crda' failed with exit code 249.
Jul 24 17:20:31 raspberrypi systemd[1]: Starting Flush Journal to Persistent Storage...
Jul 24 17:20:31 raspberrypi systemd-timesyncd[259]: Synchronized to time server 91.234.160.19:123 (2.debian.pool.ntp.org).
Jul 24 17:20:31 raspberrypi systemd[1]: Started Flush Journal to Persistent Storage.
Jul 24 17:20:31 raspberrypi dhcpcd[328]: wlan0: deleting address fe80::79c1:3936:1c89:9c35
Jul 24 17:20:31 raspberrypi avahi-daemon[294]: Withdrawing address record for fe80::79c1:3936:1c89:9c35 on wlan0.
Jul 24 17:20:31 raspberrypi avahi-daemon[294]: Leaving mDNS multicast group on interface wlan0.IPv6 with address fe80::79c1:3936:1c89:9c35.
Jul 24 17:20:31 raspberrypi avahi-daemon[294]: Interface wlan0.IPv6 no longer relevant for mDNS.
Jul 24 17:20:31 raspberrypi dhcpcd[328]: wlan0: deleting default route via 10.10.11.1
Jul 24 17:20:31 raspberrypi dhcpcd[328]: wlan0: deleting route to 10.10.11.0/24
Jul 24 17:20:31 raspberrypi avahi-daemon[294]: Withdrawing address record for 10.10.11.235 on wlan0.
Jul 24 17:20:31 raspberrypi avahi-daemon[294]: Leaving mDNS multicast group on interface wlan0.IPv4 with address 10.10.11.235.
Jul 24 17:20:31 raspberrypi avahi-daemon[294]: Interface wlan0.IPv4 no longer relevant for mDNS.
Jul 24 17:20:31 raspberrypi CRON[7785]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 24 17:20:32 raspberrypi dhcpcd[328]: wlan0: carrier acquired
Jul 24 17:20:32 raspberrypi dhcpcd[328]: wlan0: IAID eb:57:93:6a
Jul 24 17:20:32 raspberrypi dhcpcd[328]: wlan0: adding address fe80::79c1:3936:1c89:9c35
Jul 24 17:20:32 raspberrypi dhcpcd[328]: wlan0: rebinding lease of 10.10.11.235
Jul 24 17:20:32 raspberrypi dhcpcd[328]: wlan0: probing address 10.10.11.235/24
Jul 24 17:20:33 raspberrypi dhcpcd[328]: wlan0: soliciting an IPv6 router
Jul 24 18:17:01 raspberrypi CRON[8271]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 24 18:45:20 raspberrypi dhcpcd[328]: wlan0: failed to renew DHCP, rebinding
Jul 24 19:17:01 raspberrypi CRON[8753]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 24 20:17:01 raspberrypi CRON[9223]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 24 20:30:20 raspberrypi dhcpcd[328]: wlan0: failed to renew DHCP, rebinding
Jul 24 21:17:01 raspberrypi CRON[9720]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 24 22:15:20 raspberrypi dhcpcd[328]: wlan0: failed to renew DHCP, rebinding
Jul 24 22:17:01 raspberrypi CRON[10197]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 24 23:17:01 raspberrypi CRON[10674]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 25 00:00:20 raspberrypi dhcpcd[328]: wlan0: failed to renew DHCP, rebinding
Jul 25 00:17:02 raspberrypi CRON[11171]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 25 01:17:01 raspberrypi CRON[11636]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 25 01:45:20 raspberrypi dhcpcd[328]: wlan0: failed to renew DHCP, rebinding
Jul 25 02:17:34 raspberrypi CRON[12055]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jul 25 03:11:41 raspberrypi systemd[1]: Starting Daily apt download activities...
Jul 25 03:20:12 raspberrypi CRON[12219]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

We can see it "works" even tho the display is frozen.

I have found another interesting thing:

Jul 23 07:10:20 raspberrypi kernel: [255731.044362] Out of memory: Kill process 17475 (chromium-browse) score 976 or sacrifice child
Jul 23 07:10:20 raspberrypi kernel: [255731.044413] Killed process 17475 (chromium-browse) total-vm:1035536kB, anon-rss:365216kB, file-rss:0kB, shmem-rss:275948kB
Jul 23 07:10:20 raspberrypi kernel: [255731.162914] oom_reaper: reaped process 17475 (chromium-browse), now anon-rss:0kB, file-rss:0kB, shmem-rss:275844kB

The dates don't match, but maybe there's something to dig here more generally.

What is weird is that I have another pi setup exactly the same way and this one does not crash. I'll try to check the version differences, I think the hw is a tiny bit different too.

Same here, but it only happens since I upgraded from jessie to stretch:

This happens regularly after some time during relatively heavy writing (tvheadend recording) to a nfs mounted folder

[49641.471064] INFO: task kworker/0:2:15266 blocked for more than 120 seconds.
[49641.471083] Tainted: G C 4.14.62-v7+ #1134
[49641.471089] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[49641.471098] kworker/0:2 D 0 15266 2 0x00000000
[49641.471135] Workqueue: events_freezable mmc_rescan
[49641.471174] [<8079d258>] (__schedule) from [<8079d8c0>] (schedule+0x50/0xa8)
[49641.471195] [<8079d8c0>] (schedule) from [<80618b98>] (__mmc_claim_host+0xb8/0x1cc)
[49641.471215] [<80618b98>] (__mmc_claim_host) from [<80618cdc>] (mmc_get_card+0x30/0x34)
[49641.471233] [<80618cdc>] (mmc_get_card) from [<806218d8>] (mmc_sd_detect+0x20/0x74)
[49641.471251] [<806218d8>] (mmc_sd_detect) from [<8061b5a4>] (mmc_rescan+0x1c8/0x394)
[49641.471274] [<8061b5a4>] (mmc_rescan) from [<801375b8>] (process_one_work+0x158/0x454)
[49641.471297] [<801375b8>] (process_one_work) from [<80137918>] (worker_thread+0x64/0x5b8)
[49641.471318] [<80137918>] (worker_thread) from [<8013d99c>] (kthread+0x13c/0x16c)
[49641.471339] [<8013d99c>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)

'me too', and I can reproduce it on two different 3B+, with different SD cards, and different power supplies. The stock kernel in raspbian-stretch-lite 2018-06-27 does it, 4.14.70-v7+ #1144 does it, 4.14.73-v7+ #1148 does it.

This is a plain install, with ssh enabled, and my workload is pretty repetitive: Try to wear out the SD card. The loop goes:
Write random 512-byte data to random locations in a 2MB file, one million times, flushing after each write.
sync, and drop the caches.
compare the contents of the file to 'truth'.
use dd to read-test the entire card..

Kit with instructions: https://cm.kasperkp.dk/githubbug-pi3bp-1.tgz

[33002.940127] gogo (5419): drop_caches: 3
[34299.150323] gogo (5624): drop_caches: 3
[35595.924224] gogo (5796): drop_caches: 3
[35756.120924] INFO: task kworker/0:0:5960 blocked for more than 120 seconds.
[35756.120941] Tainted: G C 4.14.70-v7+ #1144
[35756.120946] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[35756.120953] kworker/0:0 D 0 5960 2 0x00000000
[35756.120986] Workqueue: events_freezable mmc_rescan
[35756.121018] [<8079dad0>] (__schedule) from [<8079e138>] (schedule+0x50/0xa8)
[35756.121035] [<8079e138>] (schedule) from [<806193f8>] (__mmc_claim_host+0xb8/0x1cc)
[35756.121054] [<806193f8>] (__mmc_claim_host) from [<8061953c>] (mmc_get_card+0x30/0x34)
[35756.121069] [<8061953c>] (mmc_get_card) from [<80622138>] (mmc_sd_detect+0x20/0x74)
[35756.121084] [<80622138>] (mmc_sd_detect) from [<8061be04>] (mmc_rescan+0x1c8/0x394)
[35756.121104] [<8061be04>] (mmc_rescan) from [<801376e8>] (process_one_work+0x158/0x454)
[35756.121124] [<801376e8>] (process_one_work) from [<80137a48>] (worker_thread+0x64/0x5b8)
[35756.121142] [<80137a48>] (worker_thread) from [<8013dacc>] (kthread+0x13c/0x16c)
[35756.121159] [<8013dacc>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[36892.693513] gogo (6006): drop_caches: 3
[38184.638909] gogo (6179): drop_caches: 3

Occasionally the temporary hang is long enough that I lose ssh.

I had the idea to test this on various models, to see if multicore or wifi was the trigger: No, it is Pi3/Pi3B+.

Pi 0W (4pcs): 48 hours runtime, no failures
I tested 4 cards (Sandisk A1, SanDisk industrial, NetList SLC, Panasonic Industrial), one in each 0W, with wifi enabled and talking, for a total of 192 card-hours.

Pi 2 original (1pc): 48 hours runtime, no failures
If this is a locking bug, maybe it would fail on the Pi2. This is the first Pi2, not using the Pi3 SoC.

Pi 3 original (1pc): 8 hours runtime, 6 failures, throttled=0x0
This is powered from a really nice and clean 5V 5A supply, with short fat wires, and a large heatsink.

Pi 3B+ (3pcs): Highly variable. The best has 6 days of runtime and only a single failure, the worst fails once per hour on average.

Various things tried:
330uF extra on 3.3V. 5V supplied to header. Disabling wifi. Iperf3 on wifi while copying. No difference.

I see there are no updates in this issue for a while now...

Also got this issue since I've upgraded to stretch
Hardware: Rpi3
OS Raspbian 9.6 (stretch)
Kernel: Linux raspberrypi 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux

I'm running multiple docker containers, one uses an NFS share might that be causing it? (transmission)

I use this to mount it:

mount | grep nfs
192.168.0.100:/volume1/Downloads on /nas type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.33,local_lock=none,addr=192.168.0.100)

Here are the log entries from syslog:

Jan 23 06:23:32 raspberrypi kernel: [161707.549270] INFO: task kworker/3:2:31771 blocked for more than 120 seconds.
Jan 23 06:23:32 raspberrypi kernel: [161707.549279]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:23:32 raspberrypi kernel: [161707.549281] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:23:32 raspberrypi kernel: [161707.549285] kworker/3:2     D    0 31771      2 0x00000000
Jan 23 06:23:32 raspberrypi kernel: [161707.549304] Workqueue: events_freezable mmc_rescan
Jan 23 06:23:32 raspberrypi kernel: [161707.549325] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:23:32 raspberrypi kernel: [161707.549334] [<8079f5d8>] (schedule) from [<8061a2d0>] (__mmc_claim_host+0xb8/0x1cc)
Jan 23 06:23:32 raspberrypi kernel: [161707.549344] [<8061a2d0>] (__mmc_claim_host) from [<8061a414>] (mmc_get_card+0x30/0x34)
Jan 23 06:23:32 raspberrypi kernel: [161707.549352] [<8061a414>] (mmc_get_card) from [<80623010>] (mmc_sd_detect+0x20/0x74)
Jan 23 06:23:32 raspberrypi kernel: [161707.549360] [<80623010>] (mmc_sd_detect) from [<8061ccdc>] (mmc_rescan+0x1c8/0x394)
Jan 23 06:23:32 raspberrypi kernel: [161707.549370] [<8061ccdc>] (mmc_rescan) from [<801379b4>] (process_one_work+0x158/0x454)
Jan 23 06:23:32 raspberrypi kernel: [161707.549379] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
Jan 23 06:23:32 raspberrypi kernel: [161707.549387] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
Jan 23 06:23:32 raspberrypi kernel: [161707.549398] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 23 06:25:35 raspberrypi kernel: [161830.428723] INFO: task kworker/3:2:31771 blocked for more than 120 seconds.
Jan 23 06:25:36 raspberrypi kernel: [161830.428731]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:25:36 raspberrypi kernel: [161830.428734] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:25:36 raspberrypi kernel: [161830.428738] kworker/3:2     D    0 31771      2 0x00000000
Jan 23 06:25:36 raspberrypi kernel: [161830.428756] Workqueue: events_freezable mmc_rescan
Jan 23 06:25:36 raspberrypi kernel: [161830.428777] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:25:36 raspberrypi kernel: [161830.428786] [<8079f5d8>] (schedule) from [<8061a2d0>] (__mmc_claim_host+0xb8/0x1cc)
Jan 23 06:25:36 raspberrypi kernel: [161830.428796] [<8061a2d0>] (__mmc_claim_host) from [<8061a414>] (mmc_get_card+0x30/0x34)
Jan 23 06:25:36 raspberrypi kernel: [161830.428804] [<8061a414>] (mmc_get_card) from [<80623010>] (mmc_sd_detect+0x20/0x74)
Jan 23 06:25:36 raspberrypi kernel: [161830.428817] [<80623010>] (mmc_sd_detect) from [<8061ccdc>] (mmc_rescan+0x1c8/0x394)
Jan 23 06:25:36 raspberrypi kernel: [161830.428827] [<8061ccdc>] (mmc_rescan) from [<801379b4>] (process_one_work+0x158/0x454)
Jan 23 06:25:36 raspberrypi kernel: [161830.428836] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
Jan 23 06:25:36 raspberrypi kernel: [161830.428844] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
Jan 23 06:25:36 raspberrypi kernel: [161830.428854] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 23 06:26:47 raspberrypi CRON[32174]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Jan 23 06:29:41 raspberrypi kernel: [162076.187865] INFO: task kworker/3:2:31771 blocked for more than 120 seconds.
Jan 23 06:29:42 raspberrypi kernel: [162076.187881]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:29:42 raspberrypi kernel: [162076.187886] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:29:42 raspberrypi kernel: [162076.187893] kworker/3:2     D    0 31771      2 0x00000000
Jan 23 06:29:42 raspberrypi kernel: [162076.187922] Workqueue: events_freezable mmc_rescan
Jan 23 06:29:42 raspberrypi kernel: [162076.187957] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:29:42 raspberrypi kernel: [162076.187975] [<8079f5d8>] (schedule) from [<8061a2d0>] (__mmc_claim_host+0xb8/0x1cc)
Jan 23 06:29:42 raspberrypi kernel: [162076.187994] [<8061a2d0>] (__mmc_claim_host) from [<8061a414>] (mmc_get_card+0x30/0x34)
Jan 23 06:29:42 raspberrypi kernel: [162076.188010] [<8061a414>] (mmc_get_card) from [<80623010>] (mmc_sd_detect+0x20/0x74)
Jan 23 06:29:42 raspberrypi kernel: [162076.188026] [<80623010>] (mmc_sd_detect) from [<8061ccdc>] (mmc_rescan+0x1c8/0x394)
Jan 23 06:29:42 raspberrypi kernel: [162076.188044] [<8061ccdc>] (mmc_rescan) from [<801379b4>] (process_one_work+0x158/0x454)
Jan 23 06:29:42 raspberrypi kernel: [162076.188061] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
Jan 23 06:29:42 raspberrypi kernel: [162076.188078] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
Jan 23 06:29:42 raspberrypi kernel: [162076.188097] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 23 06:31:44 raspberrypi kernel: [162199.067155] INFO: task kworker/3:2:31771 blocked for more than 120 seconds.
Jan 23 06:31:45 raspberrypi kernel: [162199.067164]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:31:45 raspberrypi kernel: [162199.067166] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:31:45 raspberrypi kernel: [162199.067170] kworker/3:2     D    0 31771      2 0x00000000
Jan 23 06:31:45 raspberrypi kernel: [162199.067192] Workqueue: events_freezable mmc_rescan
Jan 23 06:31:45 raspberrypi kernel: [162199.067254] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:31:45 raspberrypi kernel: [162199.067263] [<8079f5d8>] (schedule) from [<8061a2d0>] (__mmc_claim_host+0xb8/0x1cc)
Jan 23 06:31:45 raspberrypi kernel: [162199.067273] [<8061a2d0>] (__mmc_claim_host) from [<8061a414>] (mmc_get_card+0x30/0x34)
Jan 23 06:31:45 raspberrypi kernel: [162199.067281] [<8061a414>] (mmc_get_card) from [<80623010>] (mmc_sd_detect+0x20/0x74)
Jan 23 06:31:45 raspberrypi kernel: [162199.067289] [<80623010>] (mmc_sd_detect) from [<8061ccdc>] (mmc_rescan+0x1c8/0x394)
Jan 23 06:31:45 raspberrypi kernel: [162199.067299] [<8061ccdc>] (mmc_rescan) from [<801379b4>] (process_one_work+0x158/0x454)
Jan 23 06:31:45 raspberrypi kernel: [162199.067309] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
Jan 23 06:31:45 raspberrypi kernel: [162199.067318] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
Jan 23 06:31:45 raspberrypi kernel: [162199.067376] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 23 06:33:47 raspberrypi kernel: [162321.946637] INFO: task kworker/3:2:31771 blocked for more than 120 seconds.
Jan 23 06:33:51 raspberrypi kernel: [162321.946646]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:33:55 raspberrypi kernel: [162321.946649] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:33:59 raspberrypi kernel: [162321.946654] kworker/3:2     D    0 31771      2 0x00000000
Jan 23 06:34:03 raspberrypi kernel: [162321.946672] Workqueue: events_freezable mmc_rescan
Jan 23 06:34:07 raspberrypi kernel: [162321.946693] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:34:14 raspberrypi kernel: [162321.946712] [<8079f5d8>] (schedule) from [<8061a2d0>] (__mmc_claim_host+0xb8/0x1cc)
Jan 23 06:34:18 raspberrypi kernel: [162321.946722] [<8061a2d0>] (__mmc_claim_host) from [<8061a414>] (mmc_get_card+0x30/0x34)
Jan 23 06:34:22 raspberrypi kernel: [162321.946730] [<8061a414>] (mmc_get_card) from [<80623010>] (mmc_sd_detect+0x20/0x74)
Jan 23 06:34:29 raspberrypi kernel: [162321.946739] [<80623010>] (mmc_sd_detect) from [<8061ccdc>] (mmc_rescan+0x1c8/0x394)
Jan 23 06:34:35 raspberrypi kernel: [162321.946748] [<8061ccdc>] (mmc_rescan) from [<801379b4>] (process_one_work+0x158/0x454)
Jan 23 06:34:44 raspberrypi kernel: [162321.946757] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
Jan 23 06:34:48 raspberrypi kernel: [162321.946765] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
Jan 23 06:34:54 raspberrypi kernel: [162321.946775] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 23 06:35:50 raspberrypi kernel: [162444.826176] INFO: task transmission-da:3748 blocked for more than 120 seconds.
Jan 23 06:35:56 raspberrypi kernel: [162444.826201]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:36:01 raspberrypi kernel: [162444.826203] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:36:06 raspberrypi kernel: [162444.826208] transmission-da D    0  3748   3705 0x00000080
Jan 23 06:36:15 raspberrypi kernel: [162444.826236] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:36:21 raspberrypi kernel: [162444.826246] [<8079f5d8>] (schedule) from [<807a31bc>] (schedule_timeout+0x1d0/0x3f4)
Jan 23 06:36:25 raspberrypi kernel: [162444.826257] [<807a31bc>] (schedule_timeout) from [<807a025c>] (wait_for_common+0xc0/0x184)
Jan 23 06:36:32 raspberrypi kernel: [162444.826266] [<807a025c>] (wait_for_common) from [<807a0340>] (wait_for_completion+0x20/0x24)
Jan 23 06:36:44 raspberrypi kernel: [162444.826276] [<807a0340>] (wait_for_completion) from [<80133dc4>] (call_usermodehelper_exec+0x18c/0x1a0)
Jan 23 06:36:53 raspberrypi kernel: [162444.826288] [<80133dc4>] (call_usermodehelper_exec) from [<80454624>] (call_sbin_request_key+0x268/0x2cc)
Jan 23 06:37:00 raspberrypi kernel: [162444.826298] [<80454624>] (call_sbin_request_key) from [<80454aa0>] (request_key_and_link+0x418/0x4fc)
Jan 23 06:37:05 raspberrypi kernel: [162444.826306] [<80454aa0>] (request_key_and_link) from [<80454bdc>] (request_key+0x58/0x8c)
Jan 23 06:37:12 raspberrypi kernel: [162444.826314] [<80454bdc>] (request_key) from [<803d52a0>] (nfs_idmap_get_key+0xc0/0x1d8)
Jan 23 06:37:19 raspberrypi kernel: [162444.826333] [<803d52a0>] (nfs_idmap_get_key) from [<803d53ec>] (nfs_idmap_lookup_id+0x34/0x68)
Jan 23 06:37:26 raspberrypi kernel: [162444.826340] [<803d53ec>] (nfs_idmap_lookup_id) from [<803d5924>] (nfs_map_name_to_uid+0x124/0x130)
Jan 23 06:37:30 raspberrypi kernel: [162444.826350] [<803d5924>] (nfs_map_name_to_uid) from [<803cd5d4>] (decode_getfattr_attrs+0xce4/0xe08)
Jan 23 06:37:34 raspberrypi kernel: [162444.826359] [<803cd5d4>] (decode_getfattr_attrs) from [<803cd798>] (decode_getfattr_generic.constprop.60+0xa0/0xd8)
Jan 23 06:37:44 raspberrypi kernel: [162444.826368] [<803cd798>] (decode_getfattr_generic.constprop.60) from [<803cdeb4>] (nfs4_xdr_dec_getattr+0x90/0x98)
Jan 23 06:37:52 raspberrypi kernel: [162444.826378] [<803cdeb4>] (nfs4_xdr_dec_getattr) from [<80766fd4>] (rpcauth_unwrap_resp+0x6c/0x74)
Jan 23 06:38:05 raspberrypi kernel: [162444.826388] [<80766fd4>] (rpcauth_unwrap_resp) from [<8075985c>] (call_decode+0x14c/0x3b8)
Jan 23 06:38:08 raspberrypi kernel: [162444.826396] [<8075985c>] (call_decode) from [<80764e84>] (__rpc_execute+0x84/0x3ec)
Jan 23 06:38:11 raspberrypi kernel: [162444.826404] [<80764e84>] (__rpc_execute) from [<80765950>] (rpc_execute+0x88/0x104)
Jan 23 06:38:20 raspberrypi kernel: [162444.826412] [<80765950>] (rpc_execute) from [<807589ec>] (rpc_run_task+0x150/0x168)
Jan 23 06:38:26 raspberrypi kernel: [162444.826422] [<807589ec>] (rpc_run_task) from [<803b6ae8>] (nfs4_call_sync_sequence+0x6c/0x8c)
Jan 23 06:38:31 raspberrypi kernel: [162444.826431] [<803b6ae8>] (nfs4_call_sync_sequence) from [<803c0f54>] (nfs4_proc_getattr+0x128/0x1d4)
Jan 23 06:38:37 raspberrypi kernel: [162444.826440] [<803c0f54>] (nfs4_proc_getattr) from [<8039c4a0>] (__nfs_revalidate_inode+0xc0/0x28c)
Jan 23 06:38:44 raspberrypi kernel: [162444.826449] [<8039c4a0>] (__nfs_revalidate_inode) from [<803962e8>] (nfs_do_access+0x254/0x3c4)
Jan 23 06:38:48 raspberrypi kernel: [162444.826456] [<803962e8>] (nfs_do_access) from [<80396664>] (nfs_permission+0x1c8/0x1f0)
Jan 23 06:38:55 raspberrypi kernel: [162444.826465] [<80396664>] (nfs_permission) from [<80297604>] (__inode_permission+0xb8/0xf4)
Jan 23 06:39:01 raspberrypi kernel: [162444.826475] [<80297604>] (__inode_permission) from [<80297664>] (inode_permission+0x24/0x54)
Jan 23 06:39:07 raspberrypi kernel: [162444.826484] [<80297664>] (inode_permission) from [<80299204>] (link_path_walk+0x64/0x4a4)
Jan 23 06:39:09 raspberrypi kernel: [162444.826492] [<80299204>] (link_path_walk) from [<80299be4>] (path_openat+0x88/0xff8)
Jan 23 06:39:15 raspberrypi kernel: [162444.826502] [<80299be4>] (path_openat) from [<8029be04>] (do_filp_open+0x70/0xd4)
Jan 23 06:39:23 raspberrypi kernel: [162444.826512] [<8029be04>] (do_filp_open) from [<80288c38>] (do_sys_open+0x120/0x1d0)
Jan 23 06:39:26 raspberrypi kernel: [162444.826521] [<80288c38>] (do_sys_open) from [<80288d14>] (SyS_open+0x2c/0x30)
Jan 23 06:39:30 raspberrypi kernel: [162444.826530] [<80288d14>] (SyS_open) from [<80108198>] (__sys_trace_return+0x0/0x10)
Jan 23 06:39:38 raspberrypi kernel: [162444.826538] INFO: task kworker/3:2:31771 blocked for more than 120 seconds.
Jan 23 06:39:46 raspberrypi kernel: [162444.826542]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:39:50 raspberrypi kernel: [162444.826544] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:39:52 raspberrypi kernel: [162444.826547] kworker/3:2     D    0 31771      2 0x00000000
Jan 23 06:39:57 raspberrypi kernel: [162444.826567] Workqueue: events_freezable mmc_rescan
Jan 23 06:40:03 raspberrypi kernel: [162444.826578] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:40:08 raspberrypi kernel: [162444.826587] [<8079f5d8>] (schedule) from [<8061a2d0>] (__mmc_claim_host+0xb8/0x1cc)
Jan 23 06:40:13 raspberrypi kernel: [162444.826595] [<8061a2d0>] (__mmc_claim_host) from [<8061a414>] (mmc_get_card+0x30/0x34)
Jan 23 06:40:17 raspberrypi kernel: [162444.826604] [<8061a414>] (mmc_get_card) from [<80623010>] (mmc_sd_detect+0x20/0x74)
Jan 23 06:40:22 raspberrypi kernel: [162444.826612] [<80623010>] (mmc_sd_detect) from [<8061ccdc>] (mmc_rescan+0x1c8/0x394)
Jan 23 06:40:36 raspberrypi kernel: [162444.826622] [<8061ccdc>] (mmc_rescan) from [<801379b4>] (process_one_work+0x158/0x454)
Jan 23 06:40:41 raspberrypi kernel: [162444.826629] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
Jan 23 06:40:49 raspberrypi kernel: [162444.826637] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
Jan 23 06:40:56 raspberrypi kernel: [162444.826645] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 23 06:41:02 raspberrypi kernel: [162567.705489] INFO: task transmission-da:3748 blocked for more than 120 seconds.
Jan 23 06:41:15 raspberrypi kernel: [162567.705498]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:41:22 raspberrypi kernel: [162567.705501] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:41:31 raspberrypi kernel: [162567.705505] transmission-da D    0  3748   3705 0x00000080
Jan 23 06:41:40 raspberrypi kernel: [162567.705532] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:41:50 raspberrypi kernel: [162567.705543] [<8079f5d8>] (schedule) from [<807a31bc>] (schedule_timeout+0x1d0/0x3f4)
Jan 23 06:42:00 raspberrypi kernel: [162567.705553] [<807a31bc>] (schedule_timeout) from [<807a025c>] (wait_for_common+0xc0/0x184)
Jan 23 06:42:08 raspberrypi kernel: [162567.705575] [<807a025c>] (wait_for_common) from [<807a0340>] (wait_for_completion+0x20/0x24)
Jan 23 06:42:15 raspberrypi kernel: [162567.705586] [<807a0340>] (wait_for_completion) from [<80133dc4>] (call_usermodehelper_exec+0x18c/0x1a0)
Jan 23 06:42:23 raspberrypi kernel: [162567.705599] [<80133dc4>] (call_usermodehelper_exec) from [<80454624>] (call_sbin_request_key+0x268/0x2cc)
Jan 23 06:42:28 raspberrypi kernel: [162567.705609] [<80454624>] (call_sbin_request_key) from [<80454aa0>] (request_key_and_link+0x418/0x4fc)
Jan 23 06:42:36 raspberrypi kernel: [162567.705616] [<80454aa0>] (request_key_and_link) from [<80454bdc>] (request_key+0x58/0x8c)
Jan 23 06:42:42 raspberrypi kernel: [162567.705625] [<80454bdc>] (request_key) from [<803d52a0>] (nfs_idmap_get_key+0xc0/0x1d8)
Jan 23 06:42:47 raspberrypi kernel: [162567.705634] [<803d52a0>] (nfs_idmap_get_key) from [<803d53ec>] (nfs_idmap_lookup_id+0x34/0x68)
Jan 23 06:42:51 raspberrypi kernel: [162567.705641] [<803d53ec>] (nfs_idmap_lookup_id) from [<803d5924>] (nfs_map_name_to_uid+0x124/0x130)
Jan 23 06:43:01 raspberrypi kernel: [162567.705650] [<803d5924>] (nfs_map_name_to_uid) from [<803cd5d4>] (decode_getfattr_attrs+0xce4/0xe08)
Jan 23 06:43:08 raspberrypi kernel: [162567.705659] [<803cd5d4>] (decode_getfattr_attrs) from [<803cd798>] (decode_getfattr_generic.constprop.60+0xa0/0xd8)
Jan 23 06:43:14 raspberrypi kernel: [162567.705668] [<803cd798>] (decode_getfattr_generic.constprop.60) from [<803cdeb4>] (nfs4_xdr_dec_getattr+0x90/0x98)
Jan 23 06:43:20 raspberrypi kernel: [162567.705679] [<803cdeb4>] (nfs4_xdr_dec_getattr) from [<80766fd4>] (rpcauth_unwrap_resp+0x6c/0x74)
Jan 23 06:43:26 raspberrypi kernel: [162567.705706] [<80766fd4>] (rpcauth_unwrap_resp) from [<8075985c>] (call_decode+0x14c/0x3b8)
Jan 23 06:43:30 raspberrypi kernel: [162567.705715] [<8075985c>] (call_decode) from [<80764e84>] (__rpc_execute+0x84/0x3ec)
Jan 23 06:43:43 raspberrypi kernel: [162567.705723] [<80764e84>] (__rpc_execute) from [<80765950>] (rpc_execute+0x88/0x104)
Jan 23 06:43:50 raspberrypi kernel: [162567.705731] [<80765950>] (rpc_execute) from [<807589ec>] (rpc_run_task+0x150/0x168)
Jan 23 06:43:59 raspberrypi kernel: [162567.705741] [<807589ec>] (rpc_run_task) from [<803b6ae8>] (nfs4_call_sync_sequence+0x6c/0x8c)
Jan 23 06:44:04 raspberrypi kernel: [162567.705749] [<803b6ae8>] (nfs4_call_sync_sequence) from [<803c0f54>] (nfs4_proc_getattr+0x128/0x1d4)
Jan 23 06:44:08 raspberrypi kernel: [162567.705758] [<803c0f54>] (nfs4_proc_getattr) from [<8039c4a0>] (__nfs_revalidate_inode+0xc0/0x28c)
Jan 23 06:44:17 raspberrypi kernel: [162567.705767] [<8039c4a0>] (__nfs_revalidate_inode) from [<803962e8>] (nfs_do_access+0x254/0x3c4)
Jan 23 06:44:27 raspberrypi kernel: [162567.705774] [<803962e8>] (nfs_do_access) from [<80396664>] (nfs_permission+0x1c8/0x1f0)
Jan 23 06:44:37 raspberrypi kernel: [162567.705783] [<80396664>] (nfs_permission) from [<80297604>] (__inode_permission+0xb8/0xf4)
Jan 23 06:44:42 raspberrypi kernel: [162567.705793] [<80297604>] (__inode_permission) from [<80297664>] (inode_permission+0x24/0x54)
Jan 23 06:44:51 raspberrypi kernel: [162567.705802] [<80297664>] (inode_permission) from [<80299204>] (link_path_walk+0x64/0x4a4)
Jan 23 06:45:01 raspberrypi kernel: [162567.705816] [<80299204>] (link_path_walk) from [<80299be4>] (path_openat+0x88/0xff8)
Jan 23 06:45:09 raspberrypi kernel: [162567.705825] [<80299be4>] (path_openat) from [<8029be04>] (do_filp_open+0x70/0xd4)
Jan 23 06:45:16 raspberrypi kernel: [162567.705835] [<8029be04>] (do_filp_open) from [<80288c38>] (do_sys_open+0x120/0x1d0)
Jan 23 06:45:25 raspberrypi kernel: [162567.705844] [<80288c38>] (do_sys_open) from [<80288d14>] (SyS_open+0x2c/0x30)
Jan 23 06:45:38 raspberrypi kernel: [162567.705854] [<80288d14>] (SyS_open) from [<80108198>] (__sys_trace_return+0x0/0x10)
Jan 23 06:45:49 raspberrypi kernel: [162567.705863] INFO: task kworker/3:2:31771 blocked for more than 120 seconds.
Jan 23 06:45:54 raspberrypi kernel: [162567.705866]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:46:01 raspberrypi kernel: [162567.705868] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:46:09 raspberrypi kernel: [162567.705871] kworker/3:2     D    0 31771      2 0x00000000
Jan 23 06:46:13 raspberrypi kernel: [162567.705883] Workqueue: events_freezable mmc_rescan
Jan 23 06:46:19 raspberrypi kernel: [162567.705893] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:46:27 raspberrypi kernel: [162567.705929] [<8079f5d8>] (schedule) from [<8061a2d0>] (__mmc_claim_host+0xb8/0x1cc)
Jan 23 06:46:33 raspberrypi kernel: [162567.705946] [<8061a2d0>] (__mmc_claim_host) from [<8061a414>] (mmc_get_card+0x30/0x34)
Jan 23 06:46:42 raspberrypi kernel: [162567.705954] [<8061a414>] (mmc_get_card) from [<80623010>] (mmc_sd_detect+0x20/0x74)
Jan 23 06:46:47 raspberrypi kernel: [162567.705964] [<80623010>] (mmc_sd_detect) from [<8061ccdc>] (mmc_rescan+0x1c8/0x394)
Jan 23 06:46:56 raspberrypi kernel: [162567.705973] [<8061ccdc>] (mmc_rescan) from [<801379b4>] (process_one_work+0x158/0x454)
Jan 23 06:47:03 raspberrypi kernel: [162567.705980] [<801379b4>] (process_one_work) from [<80137d14>] (worker_thread+0x64/0x5b8)
Jan 23 06:47:16 raspberrypi kernel: [162567.705988] [<80137d14>] (worker_thread) from [<8013dd98>] (kthread+0x13c/0x16c)
Jan 23 06:47:20 raspberrypi kernel: [162567.705997] [<8013dd98>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
Jan 23 06:47:29 raspberrypi kernel: [162690.585011] INFO: task transmission-da:3748 blocked for more than 120 seconds.
Jan 23 06:47:38 raspberrypi kernel: [162690.585022]       Tainted: G         C      4.14.79-v7+ #1159
Jan 23 06:47:42 raspberrypi kernel: [162690.585025] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 23 06:47:48 raspberrypi kernel: [162690.585029] transmission-da D    0  3748   3705 0x00000080
Jan 23 06:47:54 raspberrypi kernel: [162690.585056] [<8079ef70>] (__schedule) from [<8079f5d8>] (schedule+0x50/0xa8)
Jan 23 06:48:02 raspberrypi kernel: [162690.585066] [<8079f5d8>] (schedule) from [<807a31bc>] (schedule_timeout+0x1d0/0x3f4)
Jan 23 06:48:07 raspberrypi kernel: [162690.585077] [<807a31bc>] (schedule_timeout) from [<807a025c>] (wait_for_common+0xc0/0x184)
Jan 23 06:48:18 raspberrypi kernel: [162690.585087] [<807a025c>] (wait_for_common) from [<807a0340>] (wait_for_completion+0x20/0x24)
Jan 23 06:48:21 raspberrypi kernel: [162690.585098] [<807a0340>] (wait_for_completion) from [<80133dc4>] (call_usermodehelper_exec+0x18c/0x1a0)
Jan 23 06:48:31 raspberrypi kernel: [162690.585111] [<80133dc4>] (call_usermodehelper_exec) from [<80454624>] (call_sbin_request_key+0x268/0x2cc)
Jan 23 06:48:36 raspberrypi kernel: [162690.585120] [<80454624>] (call_sbin_request_key) from [<80454aa0>] (request_key_and_link+0x418/0x4fc)
Jan 23 06:48:38 raspberrypi kernel: [162690.585128] [<80454aa0>] (request_key_and_link) from [<80454bdc>] (request_key+0x58/0x8c)
Jan 23 06:48:47 raspberrypi kernel: [162690.585137] [<80454bdc>] (request_key) from [<803d52a0>] (nfs_idmap_get_key+0xc0/0x1d8)
Jan 23 06:48:52 raspberrypi kernel: [162690.585145] [<803d52a0>] (nfs_idmap_get_key) from [<803d53ec>] (nfs_idmap_lookup_id+0x34/0x68)
Jan 23 06:48:59 raspberrypi kernel: [162690.585153] [<803d53ec>] (nfs_idmap_lookup_id) from [<803d5924>] (nfs_map_name_to_uid+0x124/0x130)
Jan 23 06:49:10 raspberrypi kernel: [162690.585162] [<803d5924>] (nfs_map_name_to_uid) from [<803cd5d4>] (decode_getfattr_attrs+0xce4/0xe08)
Jan 23 06:49:25 raspberrypi kernel: [162690.585172] [<803cd5d4>] (decode_getfattr_attrs) from [<803cd798>] (decode_getfattr_generic.constprop.60+0xa0/0xd8)
Jan 23 06:49:32 raspberrypi kernel: [162690.585181] [<803cd798>] (decode_getfattr_generic.constprop.60) from [<803cdeb4>] (nfs4_xdr_dec_getattr+0x90/0x98)
Jan 23 06:49:40 raspberrypi kernel: [162690.585191] [<803cdeb4>] (nfs4_xdr_dec_getattr) from [<80766fd4>] (rpcauth_unwrap_resp+0x6c/0x74)
Jan 23 06:49:44 raspberrypi kernel: [162690.585202] [<80766fd4>] (rpcauth_unwrap_resp) from [<8075985c>] (call_decode+0x14c/0x3b8)
Jan 23 06:49:51 raspberrypi kernel: [162690.585211] [<8075985c>] (call_decode) from [<80764e84>] (__rpc_execute+0x84/0x3ec)
Jan 23 06:50:02 raspberrypi kernel: [162690.585219] [<80764e84>] (__rpc_execute) from [<80765950>] (rpc_execute+0x88/0x104)
Jan 23 06:50:12 raspberrypi kernel: [162690.585227] [<80765950>] (rpc_execute) from [<807589ec>] (rpc_run_task+0x150/0x168)
Jan 23 06:50:20 raspberrypi kernel: [162690.585237] [<807589ec>] (rpc_run_task) from [<803b6ae8>] (nfs4_call_sync_sequence+0x6c/0x8c)
Jan 23 06:50:31 raspberrypi kernel: [162690.585245] [<803b6ae8>] (nfs4_call_sync_sequence) from [<803c0f54>] (nfs4_proc_getattr+0x128/0x1d4)
Jan 23 06:50:46 raspberrypi kernel: [162690.585255] [<803c0f54>] (nfs4_proc_getattr) from [<8039c4a0>] (__nfs_revalidate_inode+0xc0/0x28c)
Jan 23 06:51:00 raspberrypi kernel: [162690.585263] [<8039c4a0>] (__nfs_revalidate_inode) from [<803962e8>] (nfs_do_access+0x254/0x3c4)
Jan 23 06:51:04 raspberrypi kernel: [162690.585270] [<803962e8>] (nfs_do_access) from [<80396664>] (nfs_permission+0x1c8/0x1f0)
Jan 23 06:51:09 raspberrypi kernel: [162690.585280] [<80396664>] (nfs_permission) from [<80297604>] (__inode_permission+0xb8/0xf4)
Jan 23 06:51:16 raspberrypi kernel: [162690.585289] [<80297604>] (__inode_permission) from [<80297664>] (inode_permission+0x24/0x54)
Jan 23 06:51:26 raspberrypi kernel: [162690.585298] [<80297664>] (inode_permission) from [<80299204>] (link_path_walk+0x64/0x4a4)
Jan 23 06:51:33 raspberrypi kernel: [162690.585307] [<80299204>] (link_path_walk) from [<80299be4>] (path_openat+0x88/0xff8)
Jan 23 06:51:40 raspberrypi kernel: [162690.585316] [<80299be4>] (path_openat) from [<8029be04>] (do_filp_open+0x70/0xd4)
Jan 23 06:51:46 raspberrypi kernel: [162690.585326] [<8029be04>] (do_filp_open) from [<80288c38>] (do_sys_open+0x120/0x1d0)
Jan 23 06:51:53 raspberrypi kernel: [162690.585336] [<80288c38>] (do_sys_open) from [<80288d14>] (SyS_open+0x2c/0x30)
Jan 23 06:52:02 raspberrypi kernel: [162690.585345] [<80288d14>] (SyS_open) from [<80108198>] (__sys_trace_return+0x0/0x10)

Same for me. System got back operating after about 15 minutes of being unresponsive and spamming the kern.log file.

4.14.98-v7+ #1200 SMP Tue Feb 12 20:27:48 GMT 2019 armv7l GNU/Linux

Since a couple of weeks I have the same error. But I have a raspberry pi 2. I've got a Samsung 32gb SD card in it. I'm running influxdb on the card, next to Hass, grafana, traefik and Telegraph, all in Docker containers. I monitor the stats of the raspberry pi, but see no problem (enough memory, no cpu spike). Maybe it's because too Mach io on SD card? After a reboot it's up and running fine, even for a week.

I have the same problem since a few month ago, every node of my Raspberry Pi Kubernetes Cluster has every day (nearly every 24 hours) that exception. As nodes i use Raspberry Pi 3 B+ and Raspberry Pi 3.

Kernel: Linux k8s-node3 4.19.42-v7+ #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l GNU/Linux

[Sat Jun 1 17:40:26 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:40:26 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:40:26 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:40:26 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:40:26 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:40:26 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:40:26 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:40:26 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:40:26 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:40:26 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:40:26 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:40:26 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:40:26 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:40:26 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:40:26 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:40:26 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:40:26 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:40:26 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:42:29 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:42:29 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:42:29 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message [Sat Jun 1 17:42:29 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:42:29 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:42:29 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:42:29 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:42:29 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:42:29 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:42:29 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:42:29 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:42:29 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:42:29 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:42:29 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:42:29 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:42:29 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:42:29 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:42:29 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:44:32 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:44:32 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:44:32 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:44:32 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:44:32 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:44:32 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:44:32 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:44:32 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:44:32 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:44:32 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:44:32 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:44:32 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:44:32 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:44:32 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:44:32 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:44:32 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:44:32 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:44:32 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:46:35 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:46:35 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:46:35 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:46:35 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:46:35 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:46:35 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:46:35 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:46:35 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:46:35 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:46:35 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:46:35 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:46:35 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:46:35 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:46:35 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:46:35 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:46:35 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:46:35 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:46:35 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:48:38 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:48:38 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:48:38 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:48:38 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:48:38 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:48:38 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:48:38 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:48:38 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:48:38 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:48:38 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:48:38 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:48:38 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>](worker_thread+0x5c/0x5a4) [Sat Jun 1 17:48:38 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:48:38 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:48:38 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:48:38 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:48:38 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:48:38 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:50:40 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:50:40 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:50:40 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:50:40 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:50:40 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:50:40 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:50:40 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:50:40 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:50:40 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:50:40 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:50:40 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:50:40 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:50:40 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:50:40 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:50:40 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:50:40 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:50:40 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:50:40 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:52:43 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:52:43 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:52:43 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:52:43 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:52:43 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:52:43 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:52:43 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:52:43 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:52:43 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:52:43 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:52:43 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:52:43 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:52:43 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:52:43 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:52:43 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:52:43 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:52:43 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:52:43 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:54:46 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:54:46 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:54:46 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:54:46 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:54:46 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:54:46 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:54:46 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:54:46 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:54:46 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:54:46 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:54:46 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:54:46 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:54:46 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:54:46 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:54:46 2019] Exception stack(0x99aedfb0 to 0x99aedff8) [Sat Jun 1 17:54:46 2019] dfa0: 00000000 00000000 00000000 00000000 [Sat Jun 1 17:54:46 2019] dfc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [Sat Jun 1 17:54:46 2019] dfe0: 00000000 00000000 00000000 00000000 00000013 00000000 [Sat Jun 1 17:56:49 2019] INFO: task kworker/2:1:7143 blocked for more than 120 seconds. [Sat Jun 1 17:56:49 2019] Tainted: G C 4.19.42-v7+ #1219 [Sat Jun 1 17:56:49 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Sat Jun 1 17:56:49 2019] kworker/2:1 D 0 7143 2 0x00000000 [Sat Jun 1 17:56:49 2019] Workqueue: events_freezable mmc_rescan [Sat Jun 1 17:56:49 2019] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8) [Sat Jun 1 17:56:49 2019] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228) [Sat Jun 1 17:56:49 2019] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c) [Sat Jun 1 17:56:49 2019] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c) [Sat Jun 1 17:56:49 2019] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c) [Sat Jun 1 17:56:49 2019] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458) [Sat Jun 1 17:56:49 2019] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4) [Sat Jun 1 17:56:49 2019] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168) [Sat Jun 1 17:56:49 2019] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) [Sat Jun 1 17:56:49 2019] Exception stack(0x99aedfb0 to 0x99aedff8)

Can anyone who is affected by this issue, test this upstream patch also available in rpi-5.1.y ?

I have ported the patch locally back to the 4.19.y branch and build the kernel to test the patch. System is rebooted and if the error happens again, it will be in the next 24-28 hours...

@lategoodbye It seems that the patch is not working, in the morning i have the same errors as before.

[Wed Jun 5 03:46:39 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 03:46:39 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 03:46:39 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 03:46:39 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 03:46:39 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 03:46:39 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 03:46:39 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 03:46:39 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 03:46:39 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 03:46:39 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 03:46:39 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 03:46:39 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 03:46:39 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 03:46:39 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 03:46:39 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 03:46:39 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 03:46:39 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 03:46:39 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 03:48:42 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 03:48:42 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 03:48:42 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 03:48:42 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 03:48:42 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 03:48:42 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 03:48:42 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 03:48:42 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 03:48:42 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 03:48:42 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 03:48:42 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 03:48:42 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 03:48:42 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 03:48:42 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 03:48:42 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 03:48:42 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 03:48:42 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 03:48:42 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 03:50:45 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 03:50:45 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 03:50:45 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 03:50:45 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 03:50:45 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 03:50:45 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 03:50:45 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 03:50:45 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 03:50:45 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 03:50:45 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 03:50:45 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 03:50:45 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 03:50:45 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 03:50:45 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 03:50:45 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 03:50:45 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 03:50:45 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 03:50:45 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 03:52:48 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 03:52:48 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 03:52:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 03:52:48 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 03:52:48 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 03:52:48 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 03:52:48 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 03:52:48 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 03:52:48 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 03:52:48 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 03:52:48 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 03:52:48 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 03:52:48 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 03:52:48 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 03:52:48 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 03:52:48 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 03:52:48 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 03:52:48 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 03:54:51 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 03:54:51 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 03:54:51 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 03:54:51 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 03:54:51 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 03:54:51 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 03:54:51 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 03:54:51 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 03:54:51 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 03:54:51 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 03:54:51 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 03:54:51 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 03:54:51 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 03:54:51 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 03:54:51 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 03:54:51 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 03:54:51 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 03:54:51 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 03:56:54 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 03:56:54 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 03:56:54 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 03:56:54 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 03:56:54 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 03:56:54 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 03:56:54 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 03:56:54 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 03:56:54 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 03:56:54 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 03:56:54 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 03:56:54 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 03:56:54 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 03:56:54 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 03:56:54 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 03:56:54 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 03:56:54 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 03:56:54 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 03:58:56 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 03:58:56 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 03:58:56 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 03:58:56 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 03:58:56 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 03:58:56 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 03:58:56 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 03:58:56 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 03:58:56 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 03:58:56 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 03:58:56 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 03:58:56 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 03:58:56 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 03:58:56 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 03:58:56 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 03:58:56 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 03:58:56 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 03:58:56 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 04:00:59 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 04:00:59 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 04:00:59 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 04:00:59 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 04:00:59 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 04:00:59 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 04:00:59 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 04:00:59 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 04:00:59 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 04:00:59 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 04:00:59 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 04:00:59 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 04:00:59 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 04:00:59 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 04:00:59 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 04:00:59 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 04:00:59 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 04:00:59 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 04:03:02 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 04:03:02 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 04:03:02 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 04:03:02 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 04:03:02 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 04:03:02 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 04:03:02 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 04:03:02 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 04:03:02 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 04:03:02 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 04:03:02 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 04:03:02 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 04:03:02 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 04:03:02 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 04:03:02 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)
[Wed Jun 5 04:03:02 2019] 3fa0: 00000000 00000000 00000000 00000000
[Wed Jun 5 04:03:02 2019] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[Wed Jun 5 04:03:02 2019] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[Wed Jun 5 04:05:05 2019] INFO: task kworker/2:3:17085 blocked for more than 120 seconds.
[Wed Jun 5 04:05:05 2019] Tainted: G C 4.19.46-v7+ #1
[Wed Jun 5 04:05:05 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Wed Jun 5 04:05:05 2019] kworker/2:3 D 0 17085 2 0x00000000
[Wed Jun 5 04:05:05 2019] Workqueue: events_freezable mmc_rescan
[Wed Jun 5 04:05:05 2019] [<8082447c>] (__schedule) from [<80824b58>] (schedule+0x50/0xa8)
[Wed Jun 5 04:05:05 2019] [<80824b58>] (schedule) from [<8067c57c>] (__mmc_claim_host+0xf0/0x228)
[Wed Jun 5 04:05:05 2019] [<8067c57c>] (__mmc_claim_host) from [<8067c6ec>] (mmc_get_card+0x38/0x3c)
[Wed Jun 5 04:05:05 2019] [<8067c6ec>] (mmc_get_card) from [<806854ac>] (mmc_sd_detect+0x24/0x7c)
[Wed Jun 5 04:05:05 2019] [<806854ac>] (mmc_sd_detect) from [<8067ec54>] (mmc_rescan+0x1e4/0x404)
[Wed Jun 5 04:05:05 2019] [<8067ec54>] (mmc_rescan) from [<8013b5f8>] (process_one_work+0x23c/0x51c)
[Wed Jun 5 04:05:05 2019] [<8013b5f8>] (process_one_work) from [<8013c640>] (worker_thread+0x58/0x5dc)
[Wed Jun 5 04:05:05 2019] [<8013c640>] (worker_thread) from [<801420c8>] (kthread+0x140/0x170)
[Wed Jun 5 04:05:05 2019] [<801420c8>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
[Wed Jun 5 04:05:05 2019] Exception stack(0x9bbb3fb0 to 0x9bbb3ff8)

@sbiermann The patch isn't intended for this issue. So it was only a try.

It would be helpful for me, if you could reproduce this issue with the mmc/host/bcm2835 (upstream) instead of mmc/host/bcm2835-sdhost (downstream). Both use the same DT compatible so you will need to change your kernel config accordingly (enable MMC_BCM2835 and disable MMC_BCM2835_SDHOST).

Thanks

@lategoodbye Sorry, i'm not so familar with building a kernel, normally i use only default kernel images or in a very few cases rpi-update given kernel. This https://www.raspberrypi.org/documentation/linux/kernel/building.md instructions i have followed to build a 4.19.y kernel with your changes of the drivers/dma/bcm2835-dma.c.

Now i have changed my .config file so i looks like:

CONFIG_MMC_BCM2835_MMC=y
CONFIG_MMC_BCM2835_DMA=y
CONFIG_MMC_BCM2835_PIO_DMA_BARRIER=2
# CONFIG_MMC_BCM2835_SDHOST is not set

Is that correct?

Yes, but you also need to have (sorry about the confusing names)

CONFIG_MMC_BCM2835=y

That doesn't work and brokes the kernel, the Pi never boots up. To recover i have copied the old kernel back to the system and now it starts up again.

I have solved the issue for me, but it is not a general solutions. Anyway i describe here my workarround maybe it helps some other to solve the problem.
As is wrote earlier, my setup is a raspberry pi kubernetes cluster with 3 worker and 1 master. The master and 2 of the worker are Raspberry Pi 3 B+ and one worker is a older Pi 3 B. The problem happens on every worker after 24-28 hours when the worker runs my home automation software called FHEM. When the problem happens, kubernetes rescheduled FHEM to run on another worker, so every worker has the problem. FHEM collects many data from external sensors like temperature in rooms or values from the heating or wind data. This many data is written via GlusterFS to a GlusterFS cluster which is based on two Raspberry 1 with USB Sticks for the data. Kubernetes and Docker wrote their status messages and other stuff to the local SD card. My solution is to restart FHEM every night at 00:05:00 am. After that the problem never happens again, it is running stable since 5. june. The restart command shuts down the kubernetes pod and kubernetes restarts the pod after a few seconds. This means that in the background the docker container in which the FHEM runs will be shutdown and restarted.

Was this page helpful?
0 / 5 - 0 ratings