Linux: USB3 disk writes trigger OOM reaper on Pi 4
Describe the bug
Writing to a USB3 SSD disk results in kernel out of memory reaper killing random processes
To reproduce
Boot Pi 4 from SATAII SSD HDD on USB3 with quirks mode enabled
Run sudo dd if=/dev/zero of=~/test.tmp bs=500K count=8024
Run dmesg
Expected behaviour
No processes killed due to OOM reaper
Actual behaviour
Bunch of core system processes are killed by the OOM reaper.
dmesg output
[222125.725243] Mem-Info:
[222125.725255] active_anon:8353 inactive_anon:8505 isolated_anon:0
active_file:18133 inactive_file:179255 isolated_file:84
unevictable:20 dirty:5438 writeback:393 unstable:0
slab_reclaimable:11578 slab_unreclaimable:12597
mapped:8368 shmem:8196 pagetables:698 bounce:0
free:605316 free_pcp:475 free_cma:47644
[222125.725262] Node 0 active_anon:33412kB inactive_anon:34020kB active_file:72532kB inactive_file:717020kB unevictable:80kB isolated(anon):0kB isolated(file):336kB mapped:33472kB dirty:21752kB writeback:1572kB shmem:32784kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[222125.725272] DMA free:210020kB min:16384kB low:20480kB high:24576kB active_anon:0kB inactive_anon:0kB active_file:716kB inactive_file:244kB unevictable:0kB writepending:0kB present:434176kB managed:388116kB mlocked:0kB kernel_stack:1744kB pagetables:0kB bounce:0kB free_pcp:72kB local_pcp:0kB free_cma:190576kB
[222125.725276] lowmem_reserve[]: 0 0 3008 3008
[222125.725300] HighMem free:2211244kB min:512kB low:33016kB high:65520kB active_anon:33868kB inactive_anon:34788kB active_file:72316kB inactive_file:716976kB unevictable:80kB writepending:22908kB present:3080192kB managed:3080192kB mlocked:80kB kernel_stack:0kB pagetables:2792kB bounce:0kB free_pcp:1816kB local_pcp:252kB free_cma:0kB
[222125.725304] lowmem_reserve[]: 0 0 0 0
[222125.725323] DMA: 594*4kB (UHC) 583*8kB (UC) 315*16kB (UEHC) 148*32kB (UHC) 52*64kB (UEHC) 25*128kB (UEHC) 10*256kB (C) 6*512kB (C) 3*1024kB (C) 3*2048kB (C) 42*4096kB (C) = 210224kB
[222125.725408] HighMem: 1113*4kB (UM) 4974*8kB (UM) 5790*16kB (UM) 2307*32kB (UM) 240*64kB (UM) 10*128kB (UM) 1*256kB (M) 1*512kB (M) 85*1024kB (M) 48*2048kB (UM) 439*4096kB (M) = 2211604kB
[222125.725491] 205652 total pagecache pages
[222125.725503] 135 pages in swap cache
[222125.725507] Swap cache stats: add 609, delete 474, find 58/90
[222125.725511] Free swap = 6954484kB
[222125.725515] Total swap = 6957052kB
[222125.725518] 878592 pages RAM
[222125.725522] 770048 pages HighMem/MovableOnly
[222125.725526] 11515 pages reserved
[222125.725529] 65536 pages cma reserved
[222125.725533] Tasks state (memory values in pages):
[222125.725537] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[222125.725556] [ 238] 0 238 4606 947 49152 1 -1000 systemd-udevd
[222125.725564] [ 385] 100 385 5592 1327 57344 3 0 systemd-timesyn
[222125.725569] [ 423] 0 423 1986 564 45056 1 0 cron
[222125.725575] [ 425] 0 425 3276 1443 53248 0 0 systemd-logind
[222125.725580] [ 431] 65534 431 1080 544 32768 4 0 thd
[222125.725586] [ 440] 104 440 1715 968 36864 4 -900 dbus-daemon
[222125.725591] [ 447] 0 447 2675 1012 45056 2 0 wpa_supplicant
[222125.725597] [ 449] 0 449 923 175 28672 4 0 alsactl
[222125.725603] [ 450] 0 450 6914 17 40960 3 0 rngd
[222125.725608] [ 454] 0 454 6378 1004 53248 0 0 rsyslogd
[222125.725614] [ 461] 108 461 1475 750 40960 0 0 avahi-daemon
[222125.725619] [ 480] 108 480 1443 60 36864 3 0 avahi-daemon
[222125.725625] [ 495] 0 495 740 502 28672 10 0 dhcpcd
[222125.725632] [ 516] 0 516 2094 669 36864 2 0 vncserver-x11-s
[222125.725637] [ 555] 0 555 10035 1802 73728 0 0 lightdm
[222125.725642] [ 564] 0 564 2699 1400 45056 0 -1000 sshd
[222125.725647] [ 565] 0 565 11426 2116 81920 3 0 polkitd
[222125.725652] [ 576] 0 576 2740 813 45056 0 0 wpa_supplicant
[222125.725658] [ 598] 0 598 1431 631 40960 10 0 login
[222125.725663] [ 736] 1000 736 3699 1863 53248 0 0 systemd
[222125.725668] [ 740] 1000 740 4231 424 61440 28 0 (sd-pam)
[222125.725674] [ 768] 1000 768 1668 870 40960 0 0 dbus-daemon
[222125.725679] [ 817] 1000 817 13157 1476 69632 0 0 gvfsd
[222125.725684] [ 822] 1000 822 14192 1669 81920 0 0 gvfsd-fuse
[222125.725690] [ 937] 1000 937 14005 1720 86016 0 0 gvfs-afc-volume
[222125.725695] [ 946] 1000 946 10025 1264 69632 0 0 gvfs-goa-volume
[222125.725700] [ 954] 1000 954 10031 1120 61440 0 0 gvfs-mtp-volume
[222125.725706] [ 966] 1000 966 10423 1219 69632 0 0 gvfs-gphoto2-vo
[222125.725711] [ 1045] 1000 1045 10895 1604 77824 0 0 gnome-keyring-d
[222125.725717] [ 1085] 1000 1085 7775 1225 65536 0 0 gvfsd-metadata
[222125.725722] [ 1115] 1000 1115 2601 1018 45056 1 0 zsh
[222125.725727] [ 1155] 0 1155 535 31 28672 1 0 hciattach
[222125.725733] [ 1159] 0 1159 2452 1142 40960 0 0 bluetoothd
[222125.725739] [ 1163] 0 1163 6676 1000 57344 3 0 bluealsa
[222125.725748] [ 4614] 0 4614 2701 1417 45056 0 0 sshd
[222125.725754] [ 4617] 1000 4617 2701 931 45056 0 0 sshd
[222125.725760] [ 4619] 1000 4619 2632 1125 49152 0 0 zsh
[222125.725766] [ 4732] 0 4732 2701 1416 45056 0 0 sshd
[222125.725772] [ 4736] 1000 4736 2701 905 40960 0 0 sshd
[222125.725778] [ 4738] 1000 4738 2632 1079 45056 0 0 zsh
[222125.725783] [ 4793] 1000 4793 2154 918 40960 0 0 bash
[222125.725790] [ 4954] 0 4954 8141 2001 65536 0 0 lightdm
[222125.725796] [ 5114] 1000 5114 27034 2664 126976 0 0 gvfs-udisks2-vo
[222125.725804] [ 5162] 1000 5162 13806 2009 90112 0 0 gvfsd-trash
[222125.725810] [ 5366] 1000 5366 16127 1987 90112 0 0 gvfsd-network
[222125.725815] [ 5402] 1000 5402 13564 1625 77824 0 0 gvfsd-dnssd
[222125.725879] [ 6277] 0 6277 2514 815 53248 0 0 sudo
[222125.725885] [ 6282] 0 6282 2355 674 40960 0 0 slabtop
[222125.725890] [ 6295] 1000 6295 1751 459 40960 0 0 dd
[222125.725896] Out of memory: Kill process 5114 (gvfs-udisks2-vo) score 1 or sacrifice child
[222125.725917] Killed process 5114 (gvfs-udisks2-vo) total-vm:108136kB, anon-rss:1296kB, file-rss:9360kB, shmem-rss:0kB
**[222125.729422] oom_reaper: reaped process 5114 (gvfs-udisks2-vo), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB**
[222126.015293] systemd[1]: Starting Journal Service...
[222126.021366] systemd[1]: fail2ban.service: Main process exited, code=killed, status=9/KILL
[222126.021730] systemd[1]: fail2ban.service: Failed with result 'signal'.
[222126.243642] systemd-journald[6309]: File /run/log/journal/d5d73d639cf747f6a9c246e71e2e9554/system.journal corrupted or uncleanly shut down, renaming and replacing.
slabtop output
Active / Total Objects (% used) : 635208 / 659018 (96.4%)
Active / Total Slabs (% used) : 15395 / 15395 (100.0%)
Active / Total Caches (% used) : 89 / 111 (80.2%)
Active / Total Size (% used) : 88160.57K / 93933.56K (93.9%)
Minimum / Average / Maximum Object : 0.02K / 0.14K / 8.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
231680 230343 99% 0.06K 3620 64 14480K buffer_head
59456 54537 91% 0.06K 929 64 3716K kmalloc-64
52736 52736 100% 0.03K 412 128 1648K anon_vma_chain
44730 36236 81% 0.13K 1491 30 5964K dentry
29610 29610 100% 0.09K 705 42 2820K kernfs_node_cache
27729 27029 97% 0.10K 711 39 2844K vm_area_struct
26937 26937 100% 0.05K 369 73 1476K anon_vma
21735 21012 96% 0.19K 1035 21 4140K filp
15876 15876 100% 0.11K 441 36 1764K ext4_groupinfo_4k
17900 14718 82% 0.71K 1132 22 18112K ext4_inode_cache
12614 12204 96% 0.38K 608 21 4864K inode_cache
12974 10711 82% 0.30K 499 26 3992K radix_tree_node
9010 9010 100% 0.02K 53 170 212K pde_opener
7424 7424 100% 0.03K 58 128 232K ext4_extent_status
6112 6050 98% 0.12K 191 32 764K cred_jar
5984 5712 95% 0.12K 187 32 748K kmalloc-128
5508 5508 100% 0.04K 54 102 216K eventpoll_pwq
4928 4865 98% 0.06K 77 64 308K pid
4864 4864 100% 0.03K 38 128 152K ftrace_event_field
4620 4536 98% 0.19K 220 21 880K skbuff_head_cache
4944 4515 91% 0.50K 310 16 2480K kmalloc-512
4094 4094 100% 0.09K 89 46 356K seq_file
4091 3924 95% 0.41K 219 19 1752K proc_inode_cache
2880 2880 100% 0.06K 45 64 180K kmem_cache_node
2720 2720 100% 0.05K 32 85 128K inotify_inode_mark
2709 2709 100% 0.19K 129 21 516K kmem_cache
2448 2448 100% 0.25K 153 16 612K kmalloc-256
2808 2407 85% 0.42K 157 18 1256K shmem_inode_cache
2496 2334 93% 1.00K 156 16 2496K kmalloc-1024
1895 1895 100% 0.75K 91 21 1456K UNIX
1701 1652 97% 0.44K 95 18 760K sock_inode_cache
1587 1587 100% 0.69K 69 23 1104K signal_cache
1536 1536 100% 1.31K 64 24 2048K sighand_cache
1326 1326 100% 0.04K 13 102 52K jbd2_inode
1281 1281 100% 0.19K 61 21 244K kmalloc-192
1275 1275 100% 0.05K 15 85 60K trace_event_file
1152 1152 100% 0.06K 18 64 72K jbd2_journal_head
1104 1104 100% 0.50K 69 16 552K mm_struct
1129 1097 97% 3.75K 142 8 4544K task_struct
1024 1024 100% 0.02K 4 256 16K fsnotify_mark_connector
960 960 100% 0.25K 60 16 240K files_cache
928 897 96% 2.00K 58 16 1856K kmalloc-2048
882 882 100% 0.19K 42 21 168K biovec-16
842 842 100% 4.00K 107 8 3424K kmalloc-4096
736 723 98% 0.25K 46 16 184K task_group
704 704 100% 0.06K 11 64 44K blkdev_ioc
703 703 100% 0.81K 37 19 592K UDP`
System
System Information
------------------
Raspberry Pi 4 Model B Rev 1.1
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
Raspberry Pi reference 2019-07-10
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage5
Linux mbox 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l GNU/Linux
Hardware : BCM2835
Revision : c03111
Serial : 100000003f3b5427
Throttled flag : throttled=0x0
Camera : supported=0 detected=0
Videocore information
---------------------
Aug 15 2019 12:03:51
Copyright (c) 2012 Broadcom
version 9f8431fb7839c7f00f52b81f5822ddab2b31d0db (clean) (release) (start)
alloc failures: 0
compactions: 0
legacy block fails: 0
Filesystem information
----------------------
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 114854492 20116484 88860636 19% /
devtmpfs 1602056 0 1602056 0% /dev
tmpfs 1734152 0 1734152 0% /dev/shm
tmpfs 1734152 18084 1716068 2% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 1734152 0 1734152 0% /sys/fs/cgroup
/dev/mmcblk0p1 258096 40982 217114 16% /boot
tmpfs 346828 0 346828 0% /run/user/1000
/dev/mmcblk0p2 7347920 6085992 884776 88% /media/pi/rootfs
/dev/dm-0 1922695992 1787748528 37257148 98% /media/pi/2tb_luks_ind
tmpfs 346828 0 346828 0% /run/user/109
Filename Type Size Used Priority
/var/swap file 6957052 17152 -2
Package version information
---------------------------
raspberrypi-ui-mods:
Installed: 1.20190717
raspberrypi-sys-mods:
Installed: 20190730
openbox:
Installed: 3.6.1-8+rpt2
lxpanel:
Installed: 0.10.0-2+rpt6
pcmanfm:
Installed: 1.3.1-1+rpt8
rpd-plym-splash:
Installed: 0.19
Networking Information
----------------------
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet x.x.x.x netmask x.x.x.x broadcast x.x.x.x
ether m.m.m.m txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet x.x.x.x netmask x.x.x.x broadcast x.x.x.x
ether m.m.m.m txqueuelen 1000 (Ethernet)
RX packets 139915 bytes 11835558 (11.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 144225 bytes 39066373 (37.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet x.x.x.x netmask x.x.x.x
loop txqueuelen 1000 (Local Loopback)
RX packets 408 bytes 31629 (30.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 408 bytes 31629 (30.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
USB Information
---------------
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=dwc_otg/1p, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
|__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
config.txt
----------
arm_freq=1500
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=500
core_freq_min=250
disable_commandline_tags=2
disable_l2cache=1
display_hdmi_rotate=-1
display_lcd_rotate=-1
enable_gic=1
force_eeprom_read=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=500
gpu_freq_min=500
init_uart_clock=0x2dc6c00
lcd_framerate=60
mask_gpu_interrupt0=1024
mask_gpu_interrupt1=0x10000
max_framebuffers=2
pause_burst_frames=1
program_serial_random=1
hdmi_force_cec_address:0=65535
hdmi_force_cec_address:1=65535
hdmi_pixel_freq_limit:0=0x11e1a300
hdmi_pixel_freq_limit:1=0x11e1a300
device_tree=-
hdmi_cvt:0=
hdmi_cvt:1=
hdmi_edid_filename:0=
hdmi_edid_filename:1=
hdmi_timings:0=
hdmi_timings:1=
cmdline.txt
-----------
coherent_pool=1M 8250.nr_uarts=0 cma=64M cma=256M video=HDMI-A-1:1920x1080@60 smsc95xx.macaddr=DC:A6:32:07:A4:17 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 fsck.mode=force usb-storage.quirks=152d:0578:u,1058:1140:u dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=PARTUUID=698e33bd-01 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait ipv6.disable=1
raspi-gpio settings
-------------------
BANK0 (GPIO 0 to 27):
GPIO 0: level=1 fsel=0 func=INPUT pull=UP
GPIO 1: level=1 fsel=0 func=INPUT pull=UP
GPIO 2: level=1 fsel=0 func=INPUT pull=UP
GPIO 3: level=1 fsel=0 func=INPUT pull=UP
GPIO 4: level=1 fsel=0 func=INPUT pull=UP
GPIO 5: level=1 fsel=0 func=INPUT pull=UP
GPIO 6: level=1 fsel=0 func=INPUT pull=UP
GPIO 7: level=1 fsel=0 func=INPUT pull=UP
GPIO 8: level=1 fsel=0 func=INPUT pull=UP
GPIO 9: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 10: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 11: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 12: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 13: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 14: level=1 fsel=0 func=INPUT pull=NONE
GPIO 15: level=1 fsel=0 func=INPUT pull=UP
GPIO 16: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 17: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 18: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 19: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 20: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 21: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 22: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 23: level=0 fsel=1 func=OUTPUT pull=NONE
GPIO 24: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 25: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 26: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 27: level=0 fsel=0 func=INPUT pull=DOWN
BANK1 (GPIO 28 to 45):
GPIO 28: level=1 fsel=2 alt=5 func=RGMII_MDIO pull=UP
GPIO 29: level=0 fsel=2 alt=5 func=RGMII_MDC pull=DOWN
GPIO 30: level=0 fsel=7 alt=3 func=CTS0 pull=UP
GPIO 31: level=0 fsel=7 alt=3 func=RTS0 pull=NONE
GPIO 32: level=1 fsel=7 alt=3 func=TXD0 pull=NONE
GPIO 33: level=1 fsel=7 alt=3 func=RXD0 pull=UP
GPIO 34: level=1 fsel=7 alt=3 func=SD1_CLK pull=NONE
GPIO 35: level=1 fsel=7 alt=3 func=SD1_CMD pull=UP
GPIO 36: level=1 fsel=7 alt=3 func=SD1_DAT0 pull=UP
GPIO 37: level=1 fsel=7 alt=3 func=SD1_DAT1 pull=UP
GPIO 38: level=1 fsel=7 alt=3 func=SD1_DAT2 pull=UP
GPIO 39: level=1 fsel=7 alt=3 func=SD1_DAT3 pull=UP
GPIO 40: level=0 fsel=4 alt=0 func=PWM1_0 pull=NONE
GPIO 41: level=0 fsel=4 alt=0 func=PWM1_1 pull=NONE
GPIO 42: level=0 fsel=1 func=OUTPUT pull=UP
GPIO 43: level=1 fsel=0 func=INPUT pull=UP
GPIO 44: level=1 fsel=0 func=INPUT pull=UP
GPIO 45: level=1 fsel=0 func=INPUT pull=UP
BANK2 (GPIO 46 to 53):
GPIO 46: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 47: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 48: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 49: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 50: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 51: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 52: level=0 fsel=0 func=INPUT pull=DOWN
GPIO 53: level=0 fsel=0 func=INPUT pull=DOWN
vcdbg log messages
------------------
002670.230: arasan: arasan_emmc_open
002672.420: arasan: arasan_emmc_set_clock C0: 0x00000000 C1: 0x000ef447 emmc: 200000000 actual: 200000 div: 0x000001f4 target: 200000 min: 100000 max: 400000 delay: 10
002789.189: arasan: arasan_emmc_set_clock C0: 0x00000000 C1: 0x000ef447 emmc: 200000000 actual: 200000 div: 0x000001f4 target: 200000 min: 100000 max: 400000 delay: 10
002796.343: arasan: arasan_emmc_set_clock C0: 0x00000f00 C1: 0x000e01c7 emmc: 200000000 actual: 130039 div: 0x00000301 target: 130000 min: 100000 max: 400000 delay: 15
002838.933: arasan: arasan_emmc_set_clock C0: 0x00000f06 C1: 0x000e0307 emmc: 200000000 actual: 33333333 div: 0x00000003 target: 40000000 min: 0 max: 40000000 delay: 1
003139.347: brfs: File read: /mfs/sd/config.txt
003140.085: brfs: File read: 1747 bytes
003185.745: bootloader_eeprom_find_files unexpected length: 0xffffffffn
003196.209: brfs: File read: /mfs/sd/config.txt
003397.128: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
003399.601: *** Restart logging
003399.639: brfs: File read: 1747 bytes
003436.372: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
003436.407: HDMI0: hdmi_pixel_encoding: 300000000
003436.421: HDMI1: hdmi_pixel_encoding: 300000000
003437.706: brfs: File read: /mfs/sd/cmdline.txt
003437.789: Read command line from file 'cmdline.txt':
003437.831: 'fsck.mode=force usb-storage.quirks=152d:0578:u,1058:1140:u dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=698e33bd-01 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait ipv6.disable=1'
003438.123: brfs: File read: 487 bytes
004247.053: brfs: File read: /mfs/sd/kernel7l.img
004247.118: Loading 'kernel7l.img' to 0x8000 size 0x55d880
004251.536: No kernel trailer - assuming DT-capable
004251.630: brfs: File read: 5625984 bytes
004263.090: brfs: File read: /mfs/sd/bcm2711-rpi-4-b.dtb
004263.153: Loading 'bcm2711-rpi-4-b.dtb' to 0x565880 size 0x9e6f
004572.988: brfs: File read: 40559 bytes
004576.672: brfs: File read: /mfs/sd/config.txt
004577.407: dtparam: audio=on
004614.287: brfs: File read: 1747 bytes
004661.234: brfs: File read: /mfs/sd/overlays/vc4-fkms-v3d.dtbo
004718.727: Loaded overlay 'vc4-fkms-v3d'
006203.480: Device tree loaded to 0x1a7e5c00 (size 0xa3e4)
006206.128: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined
008902.583: vchiq_core: vchiq_init_state: slot_zero = 0xca580000, is_master = 1
008906.030: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
008912.346: TV service:host side not connected, dropping notification 0x00000002, 0x00000001, 0x00000010
020783.801: brfs: File read: 1255 bytes
021485.401: TV service:host side not connected, dropping notification 0x00000008, 0x00000004, 0x00000000
1866353.078: TV service:host side not connected, dropping notification 0x00000001, 0x00000002, 0x00000000
1866407.776: TV service:host side not connected, dropping notification 0x00000002, 0x00000004, 0x00000000
dmesg log
---------
Forum troubleshooting thread - link
rowanalex123
All 59 comments
- What does
lsusbreport? - How are you powering the SSD?
- There is a new test firmware for the VL805 that seems to be very promising. You can download it here: https://drive.google.com/file/d/1nUFfKGI118bJ6DmiDm9euE6kWbOxbG7q/view?usp=sharing
pelwell
on 6 Sep 2019
The SSD is powered using the official 3A Pi 4 power supply. vl805_update_0137ab does not fix the issue. The WD USB disk drive is powered by it's own power supply. I'm testing on the SSD only.
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 152d:0578 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge
Bus 002 Device 002: ID 1058:1140 Western Digital Technologies, Inc. My Book Essential (WDBACW)
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
After the firmware upgrade, I ran:
sudo ./vl805
VL805 FW version: 000137ab
sudo dd if=/dev/zero of=/home/pi/test.tmp bs=4M count=3824
dmesg output
sudo dmesg
[ 669.808484] device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: [email protected]
[ 669.996916] NET: Registered protocol family 38
[ 670.013086] cryptd: max_cpu_qlen set to 1000
[ 674.981739] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
[ 2690.825013] lxpanel invoked oom-killer: gfp_mask=0x7080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=0
[ 2690.825022] lxpanel cpuset=/ mems_allowed=0
[ 2690.825040] CPU: 0 PID: 870 Comm: lxpanel Tainted: G C 4.19.66-v7l+ #1253
[ 2690.825043] Hardware name: BCM2835
[ 2690.825071] [<c0212c7c>] (unwind_backtrace) from [<c020d49c>] (show_stack+0x20/0x24)
[ 2690.825082] [<c020d49c>] (show_stack) from [<c097f080>] (dump_stack+0xd4/0x118)
[ 2690.825095] [<c097f080>] (dump_stack) from [<c033f5b4>] (dump_header+0x80/0x250)
[ 2690.825104] [<c033f5b4>] (dump_header) from [<c033e92c>] (oom_kill_process+0x358/0x3a8)
[ 2690.825113] [<c033e92c>] (oom_kill_process) from [<c033f25c>] (out_of_memory+0x134/0x36c)
[ 2690.825122] [<c033f25c>] (out_of_memory) from [<c0345620>] (__alloc_pages_nodemask+0xfc0/0x1180)
[ 2690.825133] [<c0345620>] (__alloc_pages_nodemask) from [<c021f5d0>] (copy_process.part.5+0x1f4/0x1ad4)
[ 2690.825141] [<c021f5d0>] (copy_process.part.5) from [<c0221078>] (_do_fork+0xd8/0x438)
[ 2690.825146] [<c0221078>] (_do_fork) from [<c02214fc>] (sys_clone+0x34/0x3c)
[ 2690.825153] [<c02214fc>] (sys_clone) from [<c0201000>] (ret_fast_syscall+0x0/0x28)
[ 2690.825157] Exception stack(0xc64d3fa8 to 0xc64d3ff0)
[ 2690.825162] 3fa0: b57311d8 00000001 01200011 00000000 00000000 00000000
[ 2690.825168] 3fc0: b57311d8 00000001 b3d15a40 00000078 00000001 b6473000 b5731630 00000013
[ 2690.825171] 3fe0: b5731170 beec16c0 b63c8854 b63c8788
[ 2690.825175] Mem-Info:
[ 2690.825185] active_anon:24288 inactive_anon:31049 isolated_anon:1
active_file:8763 inactive_file:360689 isolated_file:0
unevictable:16 dirty:6541 writeback:628 unstable:0
slab_reclaimable:13552 slab_unreclaimable:11228
mapped:12179 shmem:19585 pagetables:1503 bounce:0
free:393281 free_pcp:64 free_cma:46791
[ 2690.825194] Node 0 active_anon:97152kB inactive_anon:124196kB active_file:35052kB inactive_file:1442756kB unevictable:64kB isolated(anon):4kB isolated(file):0kB mapped:48716kB dirty:26164kB writeback:2512kB shmem:78340kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 2690.825203] DMA free:203832kB min:16384kB low:20480kB high:24576kB active_anon:0kB inactive_anon:0kB active_file:3252kB inactive_file:344kB unevictable:0kB writepending:0kB present:434176kB managed:388116kB mlocked:0kB kernel_stack:3424kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:187164kB
[ 2690.825207] lowmem_reserve[]: 0 0 3008 3008
[ 2690.825232] HighMem free:1369292kB min:512kB low:33016kB high:65520kB active_anon:97300kB inactive_anon:124196kB active_file:31800kB inactive_file:1442292kB unevictable:64kB writepending:28704kB present:3080192kB managed:3080192kB mlocked:64kB kernel_stack:0kB pagetables:6012kB bounce:0kB free_pcp:384kB local_pcp:0kB free_cma:0kB
[ 2690.825235] lowmem_reserve[]: 0 0 0 0
[ 2690.825254] DMA: 317*4kB (UEC) 258*8kB (UE) 220*16kB (UEC) 102*32kB (UEH) 64*64kB (UE) 22*128kB (UEHC) 2*256kB (C) 2*512kB (C) 3*1024kB (C) 3*2048kB (C) 43*4096kB (C) = 203908kB
[ 2690.825336] HighMem: 4872*4kB (UM) 995*8kB (UM) 68*16kB (UM) 38*32kB (UM) 28*64kB (UM) 245*128kB (UM) 1453*256kB (UM) 580*512kB (UM) 149*1024kB (UM) 79*2048kB (UM) 79*4096kB (M) = 1369784kB
[ 2690.825417] 393633 total pagecache pages
[ 2690.825426] 4700 pages in swap cache
[ 2690.825430] Swap cache stats: add 84212, delete 79512, find 21231/29672
[ 2690.825434] Free swap = 3944188kB
[ 2690.825437] Total swap = 4194300kB
[ 2690.825441] 878592 pages RAM
[ 2690.825445] 770048 pages HighMem/MovableOnly
[ 2690.825448] 11515 pages reserved
[ 2690.825451] 65536 pages cma reserved
[ 2690.825455] Tasks state (memory values in pages):
[ 2690.825459] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 2690.825475] [ 127] 0 127 5347 864 204800 202 0 systemd-journal
[ 2690.825481] [ 239] 0 239 4606 457 45056 199 -1000 systemd-udevd
[ 2690.825490] [ 386] 100 386 5592 547 57344 147 0 systemd-timesyn
[ 2690.825496] [ 425] 0 425 1986 425 36864 39 0 cron
[ 2690.825503] [ 426] 108 426 1443 464 36864 51 0 avahi-daemon
[ 2690.825510] [ 428] 65534 428 1080 323 36864 37 0 thd
[ 2690.825516] [ 430] 0 430 2392 279 40960 83 0 cron
[ 2690.825522] [ 431] 0 431 19555 736 110592 656 0 udisksd
[ 2690.825528] [ 433] 0 433 923 295 32768 46 0 alsactl
[ 2690.825534] [ 436] 0 436 3275 553 49152 129 0 systemd-logind
[ 2690.825539] [ 448] 0 448 6378 428 53248 183 0 rsyslogd
[ 2690.825545] [ 452] 104 452 1704 438 40960 168 -900 dbus-daemon
[ 2690.825550] [ 458] 0 458 2675 443 45056 88 0 wpa_supplicant
[ 2690.825556] [ 462] 0 462 6914 0 40960 25 0 rngd
[ 2690.825562] [ 466] 108 466 1443 0 32768 63 0 avahi-daemon
[ 2690.825568] [ 471] 1000 471 485 53 24576 18 0 sh
[ 2690.825574] [ 475] 1000 475 3915 293 53248 1308 0 python
[ 2690.825580] [ 483] 0 483 740 299 28672 102 0 dhcpcd
[ 2690.825585] [ 536] 0 536 7994 530 90112 317 0 nmbd
[ 2690.825591] [ 543] 0 543 2094 211 40960 55 0 vncserver-x11-s
[ 2690.825597] [ 544] 0 544 11067 1069 94208 1368 0 unattended-upgr
[ 2690.825603] [ 546] 0 546 12812 2388 90112 249 0 fail2ban-server
[ 2690.825608] [ 552] 0 552 11425 630 86016 350 0 polkitd
[ 2690.825613] [ 554] 0 554 8436 587 94208 2305 0 vncserver-x11-c
[ 2690.825619] [ 557] 0 557 237176 1711 180224 2853 0 containerd
[ 2690.825624] [ 560] 0 560 243656 2320 245760 5556 -500 dockerd
[ 2690.825629] [ 572] 0 572 10026 821 73728 0 0 lightdm
[ 2690.825635] [ 579] 0 579 2671 563 40960 116 -1000 sshd
[ 2690.825640] [ 594] 0 594 2740 200 45056 125 0 wpa_supplicant
[ 2690.825646] [ 610] 0 610 27543 10211 233472 0 0 Xorg
[ 2690.825651] [ 611] 0 611 1431 385 36864 111 0 login
[ 2690.825657] [ 634] 0 634 2869 488 49152 122 0 vncagent
[ 2690.825662] [ 739] 0 739 8160 859 69632 11 0 lightdm
[ 2690.825668] [ 750] 1000 750 3668 508 57344 263 0 systemd
[ 2690.825674] [ 754] 1000 754 4231 235 53248 424 0 (sd-pam)
[ 2690.825680] [ 773] 1000 773 14083 1002 102400 318 0 lxsession
[ 2690.825686] [ 782] 1000 782 1636 398 32768 110 0 dbus-daemon
[ 2690.825693] [ 821] 1000 821 1120 155 32768 69 0 ssh-agent
[ 2690.825699] [ 838] 1000 838 10852 476 69632 169 0 gvfsd
[ 2690.825705] [ 848] 1000 848 14192 548 81920 193 0 gvfsd-fuse
[ 2690.825710] [ 864] 1000 864 16285 1113 98304 587 0 openbox
[ 2690.825716] [ 868] 1000 868 11783 657 94208 206 0 lxpolkit
[ 2690.825722] [ 870] 1000 870 37434 2161 192512 967 0 lxpanel
[ 2690.825728] [ 871] 1000 871 20542 1608 122880 521 0 pcmanfm
[ 2690.825733] [ 886] 1000 886 1120 0 32768 78 0 ssh-agent
[ 2690.825739] [ 900] 1000 900 485 53 32768 17 0 cmstart.sh
[ 2690.825746] [ 905] 1000 905 485 53 28672 26 0 kodi
[ 2690.825752] [ 908] 1000 908 1218 111 36864 68 0 xcompmgr
[ 2690.825758] [ 909] 1000 909 157839 13554 733184 1421 0 kodi.bin
[ 2690.825763] [ 926] 0 926 199847 125 45056 114 -999 containerd-shim
[ 2690.825768] [ 944] 9001 944 352 29 28672 8 0 tini
[ 2690.825774] [ 952] 1000 952 7175 525 61440 141 0 menu-cached
[ 2690.825780] [ 954] 1000 954 24506 846 98304 217 0 gvfs-udisks2-vo
[ 2690.825786] [ 964] 1000 964 14005 475 81920 203 0 gvfs-afc-volume
[ 2690.825791] [ 985] 1000 985 10025 253 69632 126 0 gvfs-goa-volume
[ 2690.825797] [ 1001] 1000 1001 10031 279 73728 136 0 gvfs-mtp-volume
[ 2690.825804] [ 1013] 1000 1013 7952 784 86016 529 0 vncserverui
[ 2690.825810] [ 1020] 1000 1020 10423 291 65536 164 0 gvfs-gphoto2-vo
[ 2690.825817] [ 1034] 1000 1034 13804 570 86016 244 0 gvfsd-trash
[ 2690.825822] [ 1039] 1000 1039 7865 815 86016 378 0 vncserverui
[ 2690.825828] [ 1078] 1000 1078 10895 298 77824 174 0 gnome-keyring-d
[ 2690.825834] [ 1105] 1000 1105 2601 372 45056 273 0 zsh
[ 2690.825839] [ 1109] 1000 1109 7744 221 65536 114 0 gvfsd-metadata
[ 2690.825845] [ 1116] 0 1116 12087 617 122880 424 0 smbd
[ 2690.825850] [ 1130] 0 1130 11356 400 102400 414 0 smbd-notifyd
[ 2690.825856] [ 1131] 0 1131 11354 355 102400 414 0 cleanupd
[ 2690.825862] [ 1136] 0 1136 12087 374 106496 427 0 lpqd
[ 2690.825868] [ 1151] 0 1151 535 1 28672 31 0 hciattach
[ 2690.825874] [ 1159] 0 1159 2452 316 40960 92 0 bluetoothd
[ 2690.825879] [ 1168] 0 1168 6676 299 57344 112 0 bluealsa
[ 2690.825886] [ 1210] 9001 1210 130992 14854 569344 30120 0 java
[ 2690.825892] [ 2753] 0 2753 2701 679 49152 5 0 sshd
[ 2690.825898] [ 2759] 1000 2759 2701 486 45056 4 0 sshd
[ 2690.825903] [ 2761] 1000 2761 2632 625 49152 0 0 zsh
[ 2690.825909] [ 2795] 0 2795 2701 703 49152 0 0 sshd
[ 2690.825915] [ 2798] 1000 2798 2701 461 45056 0 0 sshd
[ 2690.825920] [ 2800] 1000 2800 3354 762 49152 0 0 zsh
[ 2690.825926] [ 2918] 1000 2918 1882 574 45056 0 0 watch
[ 2690.825935] [ 17741] 0 17741 2514 712 40960 0 0 sudo
[ 2690.825941] [ 17746] 0 17746 2650 1312 45056 0 0 dd
[ 2690.825948] Out of memory: Kill process 1210 (java) score 23 or sacrifice child
[ 2690.826172] Killed process 1210 (java) total-vm:523968kB, anon-rss:56164kB, file-rss:3252kB, shmem-rss:0kB
[ 2690.867610] oom_reaper: reaped process 1210 (java), now anon-rss:0kB, file-rss:16kB, shmem-rss:0kB
[ 2896.534660] lxpanel invoked oom-killer: gfp_mask=0x7080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=0
[ 2896.534668] lxpanel cpuset=/ mems_allowed=0
[ 2896.534684] CPU: 3 PID: 870 Comm: lxpanel Tainted: G C 4.19.66-v7l+ #1253
[ 2896.534687] Hardware name: BCM2835
[ 2896.534708] [<c0212c7c>] (unwind_backtrace) from [<c020d49c>] (show_stack+0x20/0x24)
[ 2896.534717] [<c020d49c>] (show_stack) from [<c097f080>] (dump_stack+0xd4/0x118)
[ 2896.534727] [<c097f080>] (dump_stack) from [<c033f5b4>] (dump_header+0x80/0x250)
[ 2896.534736] [<c033f5b4>] (dump_header) from [<c033e92c>] (oom_kill_process+0x358/0x3a8)
[ 2896.534744] [<c033e92c>] (oom_kill_process) from [<c033f25c>] (out_of_memory+0x134/0x36c)
[ 2896.534752] [<c033f25c>] (out_of_memory) from [<c0345620>] (__alloc_pages_nodemask+0xfc0/0x1180)
[ 2896.534762] [<c0345620>] (__alloc_pages_nodemask) from [<c021f5d0>] (copy_process.part.5+0x1f4/0x1ad4)
[ 2896.534769] [<c021f5d0>] (copy_process.part.5) from [<c0221078>] (_do_fork+0xd8/0x438)
[ 2896.534775] [<c0221078>] (_do_fork) from [<c02214fc>] (sys_clone+0x34/0x3c)
[ 2896.534782] [<c02214fc>] (sys_clone) from [<c0201000>] (ret_fast_syscall+0x0/0x28)
[ 2896.534786] Exception stack(0xc64d3fa8 to 0xc64d3ff0)
[ 2896.534790] 3fa0: b57311d8 00000001 01200011 00000000 00000000 00000000
[ 2896.534795] 3fc0: b57311d8 00000001 b3d15a40 00000078 00000001 b6473000 b5731630 00000013
[ 2896.534799] 3fe0: b5731170 beec16c0 b63c8854 b63c8788
[ 2896.534803] Mem-Info:
[ 2896.534814] active_anon:37559 inactive_anon:44097 isolated_anon:0
active_file:8303 inactive_file:311069 isolated_file:0
unevictable:16 dirty:4427 writeback:683 unstable:0
slab_reclaimable:13049 slab_unreclaimable:11342
mapped:12602 shmem:19501 pagetables:1545 bounce:0
free:417099 free_pcp:140 free_cma:47312
[ 2896.534823] Node 0 active_anon:150236kB inactive_anon:176388kB active_file:33212kB inactive_file:1244276kB unevictable:64kB isolated(anon):0kB isolated(file):0kB mapped:50408kB dirty:17708kB writeback:2732kB shmem:78004kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 2896.534832] DMA free:207016kB min:16384kB low:20480kB high:24576kB active_anon:0kB inactive_anon:0kB active_file:1112kB inactive_file:336kB unevictable:0kB writepending:0kB present:434176kB managed:388116kB mlocked:0kB kernel_stack:3752kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:189248kB
[ 2896.534835] lowmem_reserve[]: 0 0 3008 3008
[ 2896.534860] HighMem free:1461380kB min:512kB low:33016kB high:65520kB active_anon:150236kB inactive_anon:176536kB active_file:32100kB inactive_file:1243552kB unevictable:64kB writepending:20616kB present:3080192kB managed:3080192kB mlocked:64kB kernel_stack:0kB pagetables:6180kB bounce:0kB free_pcp:676kB local_pcp:0kB free_cma:0kB
[ 2896.534863] lowmem_reserve[]: 0 0 0 0
[ 2896.534882] DMA: 522*4kB (UEHC) 568*8kB (UEHC) 337*16kB (UEHC) 117*32kB (UEHC) 48*64kB (UEHC) 6*128kB (C) 2*256kB (C) 4*512kB (C) 3*1024kB (C) 1*2048kB (C) 44*4096kB (C) = 207512kB
[ 2896.534964] HighMem: 153*4kB (UM) 98*8kB (U) 58*16kB (UM) 34*32kB (UM) 569*64kB (UM) 12*128kB (U) 266*256kB (UM) 788*512kB (UM) 317*1024kB (UM) 117*2048kB (UM) 94*4096kB (M) = 1462164kB
[ 2896.535044] 339613 total pagecache pages
[ 2896.535051] 845 pages in swap cache
[ 2896.535055] Swap cache stats: add 99831, delete 98988, find 29008/42747
[ 2896.535059] Free swap = 4054268kB
[ 2896.535062] Total swap = 4194300kB
[ 2896.535066] 878592 pages RAM
[ 2896.535069] 770048 pages HighMem/MovableOnly
[ 2896.535073] 11515 pages reserved
[ 2896.535076] 65536 pages cma reserved
[ 2896.535080] Tasks state (memory values in pages):
[ 2896.535083] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 2896.535099] [ 127] 0 127 5348 685 208896 205 0 systemd-journal
[ 2896.535105] [ 239] 0 239 4606 256 45056 213 -1000 systemd-udevd
[ 2896.535112] [ 386] 100 386 5592 267 57344 156 0 systemd-timesyn
[ 2896.535118] [ 425] 0 425 1986 300 36864 31 0 cron
[ 2896.535123] [ 426] 108 426 1443 347 36864 51 0 avahi-daemon
[ 2896.535129] [ 428] 65534 428 1080 228 36864 39 0 thd
[ 2896.535134] [ 430] 0 430 2392 107 40960 82 0 cron
[ 2896.535140] [ 431] 0 431 19555 506 110592 664 0 udisksd
[ 2896.535145] [ 433] 0 433 923 240 32768 47 0 alsactl
[ 2896.535150] [ 436] 0 436 3275 298 49152 163 0 systemd-logind
[ 2896.535156] [ 448] 0 448 6378 327 53248 191 0 rsyslogd
[ 2896.535161] [ 452] 104 452 1704 275 40960 186 -900 dbus-daemon
[ 2896.535166] [ 458] 0 458 2675 262 45056 88 0 wpa_supplicant
[ 2896.535171] [ 462] 0 462 6914 0 40960 31 0 rngd
[ 2896.535177] [ 466] 108 466 1443 1 32768 62 0 avahi-daemon
[ 2896.535182] [ 471] 1000 471 485 0 24576 17 0 sh
[ 2896.535187] [ 475] 1000 475 3915 664 53248 1088 0 python
[ 2896.535192] [ 483] 0 483 740 201 28672 102 0 dhcpcd
[ 2896.535197] [ 536] 0 536 7994 385 90112 320 0 nmbd
[ 2896.535203] [ 543] 0 543 2094 116 40960 54 0 vncserver-x11-s
[ 2896.535208] [ 544] 0 544 11067 783 94208 1375 0 unattended-upgr
[ 2896.535213] [ 546] 0 546 12812 1530 90112 1106 0 fail2ban-server
[ 2896.535219] [ 552] 0 552 11425 481 86016 353 0 polkitd
[ 2896.535224] [ 554] 0 554 8436 444 94208 2328 0 vncserver-x11-c
[ 2896.535229] [ 557] 0 557 237176 1237 180224 3289 0 containerd
[ 2896.535234] [ 560] 0 560 245705 80 249856 7683 -500 dockerd
[ 2896.535240] [ 572] 0 572 10026 675 73728 0 0 lightdm
[ 2896.535245] [ 579] 0 579 2671 259 40960 116 -1000 sshd
[ 2896.535250] [ 594] 0 594 2740 219 45056 126 0 wpa_supplicant
[ 2896.535256] [ 610] 0 610 27543 10081 233472 0 0 Xorg
[ 2896.535261] [ 611] 0 611 1431 191 36864 114 0 login
[ 2896.535266] [ 634] 0 634 2869 390 49152 121 0 vncagent
[ 2896.535271] [ 739] 0 739 8160 639 69632 10 0 lightdm
[ 2896.535277] [ 750] 1000 750 3668 236 57344 271 0 systemd
[ 2896.535282] [ 754] 1000 754 4231 135 53248 430 0 (sd-pam)
[ 2896.535287] [ 773] 1000 773 14083 878 102400 318 0 lxsession
[ 2896.535292] [ 782] 1000 782 1636 265 32768 112 0 dbus-daemon
[ 2896.535297] [ 821] 1000 821 1120 141 32768 69 0 ssh-agent
[ 2896.535303] [ 838] 1000 838 10852 349 69632 168 0 gvfsd
[ 2896.535308] [ 848] 1000 848 14192 452 81920 192 0 gvfsd-fuse
[ 2896.535313] [ 864] 1000 864 16285 960 98304 587 0 openbox
[ 2896.535318] [ 868] 1000 868 11783 536 94208 206 0 lxpolkit
[ 2896.535324] [ 870] 1000 870 37467 2068 192512 949 0 lxpanel
[ 2896.535329] [ 871] 1000 871 20542 1418 122880 552 0 pcmanfm
[ 2896.535351] [ 886] 1000 886 1120 1 32768 77 0 ssh-agent
[ 2896.535356] [ 900] 1000 900 485 0 32768 16 0 cmstart.sh
[ 2896.535361] [ 905] 1000 905 485 0 28672 25 0 kodi
[ 2896.535367] [ 908] 1000 908 1218 35 36864 67 0 xcompmgr
[ 2896.535372] [ 909] 1000 909 157839 12995 733184 1421 0 kodi.bin
[ 2896.535377] [ 952] 1000 952 7175 431 61440 141 0 menu-cached
[ 2896.535382] [ 954] 1000 954 24506 586 98304 413 0 gvfs-udisks2-vo
[ 2896.535387] [ 964] 1000 964 14005 344 81920 202 0 gvfs-afc-volume
[ 2896.535392] [ 985] 1000 985 10025 124 69632 125 0 gvfs-goa-volume
[ 2896.535397] [ 1001] 1000 1001 10031 154 73728 135 0 gvfs-mtp-volume
[ 2896.535403] [ 1013] 1000 1013 7952 616 86016 559 0 vncserverui
[ 2896.535408] [ 1020] 1000 1020 10423 157 65536 163 0 gvfs-gphoto2-vo
[ 2896.535414] [ 1034] 1000 1034 13804 498 86016 244 0 gvfsd-trash
[ 2896.535419] [ 1039] 1000 1039 7865 672 86016 377 0 vncserverui
[ 2896.535424] [ 1078] 1000 1078 10895 159 77824 173 0 gnome-keyring-d
[ 2896.535429] [ 1105] 1000 1105 2601 262 45056 282 0 zsh
[ 2896.535434] [ 1109] 1000 1109 7744 99 65536 113 0 gvfsd-metadata
[ 2896.535439] [ 1116] 0 1116 12087 310 122880 491 0 smbd
[ 2896.535445] [ 1130] 0 1130 11356 248 102400 472 0 smbd-notifyd
[ 2896.535450] [ 1131] 0 1131 11354 216 102400 471 0 cleanupd
[ 2896.535455] [ 1136] 0 1136 12087 224 106496 485 0 lpqd
[ 2896.535462] [ 1151] 0 1151 535 2 28672 30 0 hciattach
[ 2896.535467] [ 1159] 0 1159 2452 208 40960 99 0 bluetoothd
[ 2896.535472] [ 1168] 0 1168 6676 188 57344 116 0 bluealsa
[ 2896.535477] [ 2753] 0 2753 2701 270 49152 133 0 sshd
[ 2896.535482] [ 2759] 1000 2759 2701 347 45056 117 0 sshd
[ 2896.535487] [ 2761] 1000 2761 2632 827 49152 0 0 zsh
[ 2896.535492] [ 2795] 0 2795 2701 409 49152 0 0 sshd
[ 2896.535497] [ 2798] 1000 2798 2701 465 45056 0 0 sshd
[ 2896.535502] [ 2800] 1000 2800 3354 617 49152 0 0 zsh
[ 2896.535512] [ 17741] 0 17741 2514 362 40960 0 0 sudo
[ 2896.535517] [ 17746] 0 17746 2650 1216 45056 0 0 dd
[ 2896.535523] [ 18353] 0 18353 199671 166 45056 92 -999 containerd-shim
[ 2896.535528] [ 18373] 9001 18373 352 45 28672 8 0 tini
[ 2896.535533] [ 18554] 9001 18554 130520 43107 569344 1766 0 java
[ 2896.535540] [ 19220] 0 19220 2701 821 45056 0 0 sshd
[ 2896.535544] [ 19228] 1000 19228 2701 663 45056 0 0 sshd
[ 2896.535549] [ 19230] 1000 19230 2632 827 45056 0 0 zsh
[ 2896.535555] [ 19282] 0 19282 2514 724 40960 0 0 sudo
[ 2896.535560] [ 19330] 0 19330 4649 2330 65536 0 0 iotop
[ 2896.535565] Out of memory: Kill process 18554 (java) score 23 or sacrifice child
[ 2896.535844] Killed process 18554 (java) total-vm:522080kB, anon-rss:169252kB, file-rss:3176kB, shmem-rss:0kB
[ 2896.572585] oom_reaper: reaped process 18554 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 2896.572963] lxpanel invoked oom-killer: gfp_mask=0x7080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=0
[ 2896.572968] lxpanel cpuset=/ mems_allowed=0
[ 2896.572985] CPU: 3 PID: 870 Comm: lxpanel Tainted: G C 4.19.66-v7l+ #1253
[ 2896.572988] Hardware name: BCM2835
[ 2896.573008] [<c0212c7c>] (unwind_backtrace) from [<c020d49c>] (show_stack+0x20/0x24)
[ 2896.573018] [<c020d49c>] (show_stack) from [<c097f080>] (dump_stack+0xd4/0x118)
[ 2896.573029] [<c097f080>] (dump_stack) from [<c033f5b4>] (dump_header+0x80/0x250)
[ 2896.573038] [<c033f5b4>] (dump_header) from [<c033e92c>] (oom_kill_process+0x358/0x3a8)
[ 2896.573046] [<c033e92c>] (oom_kill_process) from [<c033f25c>] (out_of_memory+0x134/0x36c)
[ 2896.573054] [<c033f25c>] (out_of_memory) from [<c0345620>] (__alloc_pages_nodemask+0xfc0/0x1180)
[ 2896.573063] [<c0345620>] (__alloc_pages_nodemask) from [<c021f5d0>] (copy_process.part.5+0x1f4/0x1ad4)
[ 2896.573069] [<c021f5d0>] (copy_process.part.5) from [<c0221078>] (_do_fork+0xd8/0x438)
[ 2896.573075] [<c0221078>] (_do_fork) from [<c02214fc>] (sys_clone+0x34/0x3c)
[ 2896.573082] [<c02214fc>] (sys_clone) from [<c0201000>] (ret_fast_syscall+0x0/0x28)
[ 2896.573086] Exception stack(0xc64d3fa8 to 0xc64d3ff0)
[ 2896.573091] 3fa0: b57311d8 00000001 01200011 00000000 00000000 00000000
[ 2896.573096] 3fc0: b57311d8 00000001 b3d15a40 00000078 00000001 b6473000 b5731630 00000013
[ 2896.573100] 3fe0: b5731170 beec16c0 b63c8854 b63c8788
[ 2896.573104] Mem-Info:
[ 2896.573114] active_anon:16234 inactive_anon:22806 isolated_anon:0
active_file:8340 inactive_file:299336 isolated_file:0
unevictable:16 dirty:5833 writeback:276 unstable:0
slab_reclaimable:12901 slab_unreclaimable:11342
mapped:11529 shmem:19501 pagetables:1545 bounce:0
free:471456 free_pcp:316 free_cma:47312
[ 2896.573122] Node 0 active_anon:64936kB inactive_anon:91224kB active_file:33360kB inactive_file:1197344kB unevictable:64kB isolated(anon):0kB isolated(file):0kB mapped:46116kB dirty:23332kB writeback:1104kB shmem:78004kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[ 2896.573132] DMA free:207520kB min:16384kB low:20480kB high:24576kB active_anon:0kB inactive_anon:0kB active_file:1112kB inactive_file:336kB unevictable:0kB writepending:0kB present:434176kB managed:388116kB mlocked:0kB kernel_stack:3752kB pagetables:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:189248kB
[ 2896.573135] lowmem_reserve[]: 0 0 3008 3008
[ 2896.573160] HighMem free:1678304kB min:512kB low:33016kB high:65520kB active_anon:65060kB inactive_anon:92368kB active_file:32100kB inactive_file:1197576kB unevictable:64kB writepending:25128kB present:3080192kB managed:3080192kB mlocked:64kB kernel_stack:0kB pagetables:6180kB bounce:0kB free_pcp:1260kB local_pcp:0kB free_cma:0kB
[ 2896.573163] lowmem_reserve[]: 0 0 0 0
[ 2896.573182] DMA: 545*4kB (UEHC) 599*8kB (UEHC) 362*16kB (UEHC) 118*32kB (UEHC) 48*64kB (UEHC) 6*128kB (C) 2*256kB (C) 4*512kB (C) 3*1024kB (C) 1*2048kB (C) 44*4096kB (C) = 208284kB
[ 2896.573265] HighMem: 7376*4kB (UM) 4935*8kB (UM) 2448*16kB (UM) 1170*32kB (UM) 1500*64kB (UM) 112*128kB (UM) 274*256kB (UM) 788*512kB (UM) 317*1024kB (UM) 117*2048kB (UM) 94*4096kB (M) = 1678776kB
[ 2896.573346] 327831 total pagecache pages
[ 2896.573353] 720 pages in swap cache
[ 2896.573357] Swap cache stats: add 99859, delete 99141, find 29013/42779
[ 2896.573361] Free swap = 4061692kB
[ 2896.573365] Total swap = 4194300kB
[ 2896.573368] 878592 pages RAM
[ 2896.573372] 770048 pages HighMem/MovableOnly
[ 2896.573375] 11515 pages reserved
[ 2896.573379] 65536 pages cma reserved
[ 2896.573383] Tasks state (memory values in pages):
[ 2896.573386] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 2896.573403] [ 127] 0 127 5348 683 208896 205 0 systemd-journal
[ 2896.573409] [ 239] 0 239 4606 256 45056 213 -1000 systemd-udevd
[ 2896.573417] [ 386] 100 386 5592 267 57344 156 0 systemd-timesyn
[ 2896.573422] [ 425] 0 425 1986 300 36864 31 0 cron
[ 2896.573428] [ 426] 108 426 1443 347 36864 51 0 avahi-daemon
[ 2896.573433] [ 428] 65534 428 1080 228 36864 39 0 thd
[ 2896.573439] [ 430] 0 430 2392 107 40960 82 0 cron
[ 2896.573444] [ 431] 0 431 19555 506 110592 664 0 udisksd
[ 2896.573450] [ 433] 0 433 923 240 32768 47 0 alsactl
[ 2896.573455] [ 436] 0 436 3275 298 49152 163 0 systemd-logind
[ 2896.573460] [ 448] 0 448 6378 326 53248 191 0 rsyslogd
[ 2896.573466] [ 452] 104 452 1704 275 40960 186 -900 dbus-daemon
[ 2896.573471] [ 458] 0 458 2675 262 45056 88 0 wpa_supplicant
[ 2896.573477] [ 462] 0 462 6914 0 40960 31 0 rngd
[ 2896.573483] [ 466] 108 466 1443 1 32768 62 0 avahi-daemon
[ 2896.573489] [ 471] 1000 471 485 0 24576 17 0 sh
[ 2896.573495] [ 475] 1000 475 3915 664 53248 1088 0 python
[ 2896.573501] [ 483] 0 483 740 201 28672 102 0 dhcpcd
[ 2896.573506] [ 536] 0 536 7994 385 90112 320 0 nmbd
[ 2896.573512] [ 543] 0 543 2094 116 40960 54 0 vncserver-x11-s
[ 2896.573518] [ 544] 0 544 11067 783 94208 1375 0 unattended-upgr
[ 2896.573524] [ 546] 0 546 12812 1530 90112 1106 0 fail2ban-server
[ 2896.573529] [ 552] 0 552 11425 481 86016 353 0 polkitd
[ 2896.573535] [ 554] 0 554 8436 444 94208 2328 0 vncserver-x11-c
[ 2896.573541] [ 557] 0 557 237176 1237 180224 3289 0 containerd
[ 2896.573547] [ 560] 0 560 245705 80 249856 7683 -500 dockerd
[ 2896.573552] [ 572] 0 572 10026 675 73728 0 0 lightdm
[ 2896.573558] [ 579] 0 579 2671 259 40960 116 -1000 sshd
[ 2896.573565] [ 594] 0 594 2740 219 45056 126 0 wpa_supplicant
[ 2896.573571] [ 610] 0 610 27543 10081 233472 0 0 Xorg
[ 2896.573577] [ 611] 0 611 1431 191 36864 114 0 login
[ 2896.573582] [ 634] 0 634 2869 390 49152 121 0 vncagent
[ 2896.573588] [ 739] 0 739 8160 639 69632 10 0 lightdm
[ 2896.573594] [ 750] 1000 750 3668 236 57344 271 0 systemd
[ 2896.573600] [ 754] 1000 754 4231 135 53248 430 0 (sd-pam)
[ 2896.573606] [ 773] 1000 773 14083 878 102400 318 0 lxsession
[ 2896.573611] [ 782] 1000 782 1636 265 32768 112 0 dbus-daemon
[ 2896.573617] [ 821] 1000 821 1120 141 32768 69 0 ssh-agent
[ 2896.573623] [ 838] 1000 838 10852 349 69632 168 0 gvfsd
[ 2896.573629] [ 848] 1000 848 14192 452 81920 192 0 gvfsd-fuse
[ 2896.573635] [ 864] 1000 864 16285 960 98304 587 0 openbox
[ 2896.573641] [ 868] 1000 868 11783 536 94208 206 0 lxpolkit
[ 2896.573647] [ 870] 1000 870 37467 2068 192512 949 0 lxpanel
[ 2896.573654] [ 871] 1000 871 20542 1418 122880 552 0 pcmanfm
[ 2896.573661] [ 886] 1000 886 1120 1 32768 77 0 ssh-agent
[ 2896.573668] [ 900] 1000 900 485 0 32768 16 0 cmstart.sh
[ 2896.573673] [ 905] 1000 905 485 0 28672 25 0 kodi
[ 2896.573679] [ 908] 1000 908 1218 35 36864 67 0 xcompmgr
[ 2896.573684] [ 909] 1000 909 157839 12995 733184 1421 0 kodi.bin
[ 2896.573690] [ 952] 1000 952 7175 431 61440 141 0 menu-cached
[ 2896.573696] [ 954] 1000 954 24506 586 98304 413 0 gvfs-udisks2-vo
[ 2896.573702] [ 964] 1000 964 14005 344 81920 202 0 gvfs-afc-volume
[ 2896.573708] [ 985] 1000 985 10025 124 69632 125 0 gvfs-goa-volume
[ 2896.573714] [ 1001] 1000 1001 10031 154 73728 135 0 gvfs-mtp-volume
[ 2896.573720] [ 1013] 1000 1013 7952 616 86016 559 0 vncserverui
[ 2896.573726] [ 1020] 1000 1020 10423 157 65536 163 0 gvfs-gphoto2-vo
[ 2896.573731] [ 1034] 1000 1034 13804 498 86016 244 0 gvfsd-trash
[ 2896.573737] [ 1039] 1000 1039 7865 672 86016 377 0 vncserverui
[ 2896.573743] [ 1078] 1000 1078 10895 159 77824 173 0 gnome-keyring-d
[ 2896.573750] [ 1105] 1000 1105 2601 262 45056 282 0 zsh
[ 2896.573755] [ 1109] 1000 1109 7744 99 65536 113 0 gvfsd-metadata
[ 2896.573761] [ 1116] 0 1116 12087 310 122880 491 0 smbd
[ 2896.573767] [ 1130] 0 1130 11356 248 102400 472 0 smbd-notifyd
[ 2896.573774] [ 1131] 0 1131 11354 216 102400 471 0 cleanupd
[ 2896.573781] [ 1136] 0 1136 12087 224 106496 485 0 lpqd
[ 2896.573787] [ 1151] 0 1151 535 2 28672 30 0 hciattach
[ 2896.573793] [ 1159] 0 1159 2452 208 40960 99 0 bluetoothd
[ 2896.573798] [ 1168] 0 1168 6676 188 57344 116 0 bluealsa
[ 2896.573804] [ 2753] 0 2753 2701 270 49152 133 0 sshd
[ 2896.573809] [ 2759] 1000 2759 2701 347 45056 117 0 sshd
[ 2896.573816] [ 2761] 1000 2761 2632 827 49152 0 0 zsh
[ 2896.573822] [ 2795] 0 2795 2701 409 49152 0 0 sshd
[ 2896.573828] [ 2798] 1000 2798 2701 465 45056 0 0 sshd
[ 2896.573834] [ 2800] 1000 2800 3354 617 49152 0 0 zsh
[ 2896.573846] [ 17741] 0 17741 2514 362 40960 0 0 sudo
[ 2896.573851] [ 17746] 0 17746 2650 1216 45056 0 0 dd
[ 2896.573858] [ 18353] 0 18353 199671 166 45056 92 -999 containerd-shim
[ 2896.573864] [ 18373] 9001 18373 352 45 28672 8 0 tini
[ 2896.573872] [ 18644] 9001 18554 130520 0 569344 0 0 C1 CompilerThre
[ 2896.573878] [ 19220] 0 19220 2701 821 45056 0 0 sshd
[ 2896.573884] [ 19228] 1000 19228 2701 663 45056 0 0 sshd
[ 2896.573889] [ 19230] 1000 19230 2632 827 45056 0 0 zsh
[ 2896.573894] [ 19282] 0 19282 2514 724 40960 0 0 sudo
[ 2896.573901] [ 19330] 0 19330 4649 2330 65536 0 0 iotop
[ 2896.573907] Out of memory: Kill process 909 (kodi.bin) score 7 or sacrifice child
[ 2896.574001] Killed process 909 (kodi.bin) total-vm:631356kB, anon-rss:30324kB, file-rss:21648kB, shmem-rss:8kB
[ 2896.589324] oom_reaper: reaped process 909 (kodi.bin), now anon-rss:16kB, file-rss:0kB, shmem-rss:8kB
[ 2986.581674] lxpanel invoked oom-killer: gfp_mask=0x7080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=1, oom_score_adj=0
[ 2986.581681] lxpanel cpuset=/ mems_allowed=0
[ 2986.581698] CPU: 2 PID: 870 Comm: lxpanel Tainted: G C 4.19.66-v7l+ #1253
[ 2986.581702] Hardware name: BCM2835
[ 2986.581723] [<c0212c7c>] (unwind_backtrace) from [<c020d49c>] (show_stack+0x20/0x24)
[ 2986.581733] [<c020d49c>] (show_stack) from [<c097f080>] (dump_stack+0xd4/0x118)
[ 2986.581746] [<c097f080>] (dump_stack) from [<c033f5b4>] (dump_header+0x80/0x250)
[ 2986.581755] [<c033f5b4>] (dump_header) from [<c033e92c>] (oom_kill_process+0x358/0x3a8)
[ 2986.581763] [<c033e92c>] (oom_kill_process) from [<c033f25c>] (out_of_memory+0x134/0x36c)
[ 2986.581771] [<c033f25c>] (out_of_memory) from [<c0345620>] (__alloc_pages_nodemask+0xfc0/0x1180)
[ 2986.581780] [<c0345620>] (__alloc_pages_nodemask) from [<c021f5d0>] (copy_process.part.5+0x1f4/0x1ad4)
[ 2986.581787] [<c021f5d0>] (copy_process.part.5) from [<c0221078>] (_do_fork+0xd8/0x438)
[ 2986.581793] [<c0221078>] (_do_fork) from [<c02214fc>] (sys_clone+0x34/0x3c)
[ 2986.581800] [<c02214fc>] (sys_clone) from [<c0201000>] (ret_fast_syscall+0x0/0x28)
.
.
.
**[ 2986.631754] Out of memory: Kill process 610 (Xorg) score 5 or sacrifice child
[ 2986.631783] Killed process 610 (Xorg) total-vm:110164kB, anon-rss:14388kB, file-rss:25568kB, shmem-rss:576kB**
Expecting the 4B to power an external SSD, even with a known-good power supply, is ambitious.
That particular USB<->SATA adaptor has been seen to cause problems.
pelwell
on 6 Sep 2019
On the USB-SATA adapter part - I disagree. I use 2 Western Digital 8TB usb disks with ZFS, and it has same issue.
The WD disk has its own power supply. And I am using the genuine 5v3a standard power.
wushilin
on 4 Dec 2019
@wushilin
I also had the same issue while running ZFS (and rsync), and it was gone after switching to the 64-bit kernel.
randyoo
on 10 Dec 2019
Using the 64bit kernel did not solve the problem. At least for me.
mandev
on 13 Dec 2019
Tried multiple SATA USB controllers including the Asmedia ASM1153. Issue still persists. Dont think its a power supply issue as well. I'm using the official power supply with no peripherals (keyboard, mouse etc) attached. Should be more than enough power to run the SATA SSD drive.
balajisraghavan
on 22 Dec 2019
Please update to the latest release kernel and retry, please report results.
JamesH65
on 14 Feb 2020
What's changed @JamesH65 ?
balajisraghavan
on 14 Feb 2020
I had this happen to my Pi last night too. I was doing a DD to a USB 3 hard drive and got the OOM errors in the syslog file.
This is what my Pi shows when I do a free -m (after a reboot). Is it strange that the swap is being slightly used even with so much memory spare?
total used free shared buff/cache available
Mem: 3776 450 266 159 3059 3068
Swap: 99 19 80
bkev
on 20 Aug 2020
Having same issue. Firstly caused by Transmission, then Deluge when downloading more torrents. Then just by coping files over network to PI. Tried to make huge swap without any solution. Now I am trying to disable oom killer by https://serverfault.com/questions/141988/avoid-linux-out-of-memory-application-teardown. It really driving me as I cannot use pi as a backup network storage. I have latest kernel and Raspberry OS.
EDIT: Nope I cannot disable OOM killer seems...
Lipown
on 24 Aug 2020
I'm no expert, but it seems that it's just the data transfer that causes the issue, not the application then. I was just using dd to make a copy of the image to the USB 3 hard drive and I had the same issue, with OOM kicking in and killing all kinds of things in an attempt to get it to work.
I have a Pi v4 with 4Gb of RAM and I've never seen the amount used go over around 20% used.
I've not rebooted my pi since posting the memory above and it's now
total used free shared buff/cache available
Mem: 3776 483 165 235 3126 2970
Swap: 99 39 60
bkev
on 25 Aug 2020
This issue is a definite deal breaker for anyone trying to use the Pi 4 as a desktop replacement or a media hub. Please look into this.
rowanalex123
on 25 Aug 2020
@rowanalex123 So, are you still having the same issue, with the same things in the logs, even now?
I know that your level of detail in this post was what helped me figure out I was having the same, or a very similar issue.
I don't think it does it all the time for me, but last week whilst doing dd like you did, it slowed my Pi to a crawl and OOM starting closing all sorts of things down even though the memory seemed to be OK.
bkev
on 25 Aug 2020
I've not tested it myself off late. Going by the above two reports though, think its safe to say the issue is not resolved with the latest kernel?
rowanalex123
on 25 Aug 2020
I regularly run apt update and full-upgrade so I'm running the latest firmware, bootloader and kernel.
bkev
on 25 Aug 2020
I also tried huge 80GB swap but it's not used at all. I would like to have this resolved. When I had the storage drive as NTFS and transfer speeds were <20MBs that time over network I had not this issue but now since I have storage on ext4 I have speeds ~100MBs it's crashing by oom killer.
Lipown
on 25 Aug 2020
Can people seeing the problem please post the output of the following commands:
$ grep Revision /proc/cpuinfo
$ uname -a
$ vmstat
And:
$ lsusb
I also tried huge 80GB swap but it's not used at all. I would like to have this resolved. When I had the storage drive as NTFS and transfer speeds were <20MBs that time over network I had not this issue but now since I have storage on ext4 I have speeds ~100MBs it's crashing by oom killer.
Actually this rings a bell with me. I have my external drive formatted as NTFS but I kept on getting all sorts of errors etc related to NTFS-3G, but I can't remember any issues with OOM. I reformatted the drive to ext4 and got the OOM error.
bkev
on 25 Aug 2020
Can people seeing the problem please post the output of the following commands:
$ grep Revision /proc/cpuinfo $ uname -a $ vmstat
Hi, thanks for the info. Here are the results of the commands on my Pi:
Revision : c03111
Linux raspberrypiv4 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 41472 224508 130100 3006784 0 0 5 19 5 22 2 2 95 1 0
Bus 002 Device 002: ID 1058:2627 Western Digital Technologies, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 1e3d:4083 Chipsbank Microelectronics Co., Ltd
Bus 001 Device 004: ID 0781:5583 SanDisk Corp. Ultra Fit
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
bkev
on 25 Aug 2020
That's a rev 1.1 4B with 4GB RAM running a recent kernel. The rev 1.1 boards where fitted with BCM2711B0 - the first production silicon - which had a few restrictions that were improved in the later C0 revision:
- The EMMC2 bus can only directly address the first 1GB.
- The PCIe interface can only directly address the first 3GB.
The latter restriction may be having an effect on the disk throughput. You can eliminate it as a possibility by (as an experiment!) rebooting with total_mem=3072 in config.txt. See if that makes any difference.
pelwell
on 25 Aug 2020
How can I tell which silicon revision my BCM2711 is? Is it etched into the top of the package? All mine seem to be B0 - I'm guessing it's the B0 in the middle of this:
2711ZPKFSB06BOT
Do the chips with C0 silicon say C0 instead?
andrum99
on 25 Aug 2020
It's the penultimate 2 characters that matter: 2711ZPKFSB06COT
On a running system the easiest way to tell is to look at the declaration of the bus the SD card controller is on:
# B0
pi@raspberrypi:~$ od -An -tx1 /proc/device-tree/emmc2bus/dma-ranges
00 00 00 00 c0 00 00 00 00 00 00 00 00 00 00 00
40 00 00 00
# C0
pi@raspberrypi:~$ od -An -tx1 /proc/device-tree/emmc2bus/dma-ranges
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc 00 00 00
As far as I know, all production 8GB units are fitted with C0s.
pelwell
on 25 Aug 2020
That's a rev 1.1 4B with 4GB RAM running a recent kernel. The rev 1.1 boards where fitted with BCM2711B0 - the first production silicon - which had a few restrictions that were improved in the later C0 revision:
- The EMMC2 bus can only directly address the first 1GB.
- The PCIe interface can only directly address the first 3GB.
The latter restriction may be having an effect on the disk throughput. You can eliminate it as a possibility by (as an experiment!) rebooting with
total_mem=3072in config.txt. See if that makes any difference.
Thanks for the info.
So, my board not only has the USB-C power issue, it also can't access all of the memory?
Will the total_mem=3072 limit the board to 3Gb instead of 4Gb of memory?
bkev
on 25 Aug 2020
It's a test - I'm trying to ascertain whether or not SWIOTLB is a factor or not. Yes it will limit the system to 3GB _during the test_.
pelwell
on 25 Aug 2020
Hi @pelwell thanks for the info.
My Pi will be doing a dd tomorrow night as part of a cron job. I'll let it do it without me changing anything to see if it occurs again. If it does, I'll make the change and then let it do the dd again to see if it happens again and I will report back. This will probably be the end of next week though.
Thanks again.
bkev
on 25 Aug 2020
Here is my output of the commands:
Revision : c03111
Linux raspberrypi 5.4.59-v7l+ #1336 SMP Wed Aug 19 16:09:04 BST 2020 armv7l GNU/Linux
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 738352 25536 1797696 0 0 11 8 12 76 2 2 95 1 0
Bus 002 Device 004: ID 152d:0578 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge
Bus 002 Device 003: ID 0bda:0411 Realtek Semiconductor Corp.
Bus 002 Device 002: ID 0bda:0411 Realtek Semiconductor Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 0451:16a8 Texas Instruments, Inc.
Bus 001 Device 008: ID 045e:0797 Microsoft Corp. Optical Mouse 200
Bus 001 Device 007: ID 04d9:1603 Holtek Semiconductor, Inc. Keyboard
Bus 001 Device 006: ID 0763:400b M-Audio
Bus 001 Device 005: ID 0bda:5411 Realtek Semiconductor Corp.
Bus 001 Device 003: ID 0bda:5411 Realtek Semiconductor Corp.
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
I will run the test in few days cause I am not at home and worring that test might crash me the ssh access.
Lipown
on 26 Aug 2020
Since I am impatient I tried
total_mem=3072 neither 2048 value solves the issue unfortunatelly :(
Lipown
on 26 Aug 2020
Something came up to my mind reacting on others person post on another forum. Me neither the person having those problems when coping small files. Maybe files larger than ram doing that? I am a noob so I am just guessing....
Lipown
on 26 Aug 2020
I didn't have this solved but bought another x64 based mini pc running
freebsd as Nas, it did work well.
I could have saved the 300 bucks if pi4 is okay. Worst thing is I bought
two of them...
On Wed, Aug 26, 2020, 9:20 PM Lipown notifications@github.com wrote:
Something came up to my mind reacting on others person post on another
forum. Me neither the person having those problems when coping small files.
Maybe files larger than ram doing that? I am a noob so I am just
guessing....—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raspberrypi/linux/issues/3210#issuecomment-680874930,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABVR2NBCN2VYETEB6Y5KEWDSCUDZ7ANCNFSM4IUF6ESA
.
wushilin
on 26 Aug 2020
Could it be that with large file transfers that if there is an issue with the throughput to the usb hub, it starts to back up into the ram causing the oom? And that might be why it only affects large files?
Just a complete stab in the dark.
bkev
on 26 Aug 2020
Has anyone looked on this issue please? I am going to buy another 4 4GB but I need that the issue is no more on the latest revision.
Lipown
on 2 Sep 2020
I had the issue occur again for me.
The load on the Pi was at about 7 and it was quite unresponsive with errors in the log with I think are maybe related to this
xhci_hcd 0000:01:00.0: WARN Cannot submit Set TR Deq Ptr
xhci_hcd 0000:01:00.0: A Set TR Deq Ptr command is pending.
usb 2-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
I have added quirks to my cmdline.txt now to see if it makes any difference:
https://www.raspberrypi.org/forums/viewtopic.php?t=245931#:~:text=The%20kernel%20can%20be%20told,needs%20to%20go%20into%20cmdline.
bkev
on 4 Sep 2020
The quirks above haven't helped me. I've started to get the xhci_hcd errors again less than 24 hours later.
bkev
on 5 Sep 2020
Another log of oom killer just by using Chromium with google spreadsheet triggered this:
The pi running few days straight. What is wrong :/
Lipown
on 6 Sep 2020
As well as chromium running a spreadsheet you also have Hassio, containerd, mysqld, and others - this isn't a vanilla system. And it was dockerd that triggered the OOM killer.
pelwell
on 6 Sep 2020
As well as chromium running a spreadsheet you also have Hassio, containerd, mysqld, and others - this isn't a vanilla system. And it was dockerd that triggered the OOM killer.
Indeed, that's why I shut down the services after this and found I/O errors were caused by long USB Cables from my drive. But shutting down docker (and hassio), mysqld and all unnecessities I did 200 GB copy test and it ended up oom killing. The pi has 4GB of memory. I remember copying that with far less memory and did copied eight years ago similar amount of data from network to external drive connectted to Samsung tablet without issues. I believe all of that pi can make if another mini pc can...Nevetherless that 4 is currently most powerful...
Lipown
on 7 Sep 2020
The issue still exists for me. Can copy or download a 6gb file without the oom reaper kicking in. This is with the latest updates installed on the 4gb Pi 4
rowanalex123
on 7 Sep 2020
The issue still exists for me. Can copy or download a 6gb file without the oom reaper kicking in. This is with the latest updates installed on the 4gb Pi 4
Can or can not?
I did again a test with log to Home Assistant. When copy from PI over network its perfectly fine with CPU usage about 25% percent. I can work on pi during that.
When I copying to PI processor go to 40%. Everything slows down. Just starting Chromium triggers the OOM killer, tha ram is about 40% used, then it crash totaly with login screen in few minutes,
Lipown
on 7 Sep 2020
Can or can not?
Sorry typo. Should have been can not. Writing a 6GB file seems to trigger the OOM reaper.
rowanalex123
on 8 Sep 2020
Joining to the issue list.
Revision : c03112
Linux BreGu 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 512 365132 13768 2964392 0 0 18 2 46 46 6 0 94 0 0
Interesting that I can NOT reproduce the issue by using the dd command to my external drive. I can write out 6 or 8GB file there without any issues.
However if I start up rtorrent the issue is kicking in almost immediately. At the time when the issue is kicking in the free command shows the following (I started another terminal with watch -n2 free -h:
total used free shared buff/cache available
Mem: 3.3Gi 132Mi 356Mi 33Mi 2.8Gi 3.0Gi
Swap: 1.8Gi 0.0Ki 1.8Gi
So it's should NOT be a memory issue... but it seems it is somehow. It's really frustrating.
jozsefDevs
on 8 Oct 2020
Anything done on this?
Lipown
on 22 Oct 2020
I'm still getting the same. Every few days and it's so frustrating. For me, my Pi really slows down and the load goes up when I start seeing this in the log...
usb 2-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
And it never recovers apart from having to do a reboot.
bkev
on 23 Oct 2020
Okay, so for me this issue is resolved. I just set arm_64bit=1 into /boot/config.txt and my Pi runs since 8 days without any OOM reaping. Again, this is the Pi4 / 4GB version.
jozsefDevs
on 26 Oct 2020
For what it's worth, I had a similar experience. Regular OOM reaper issues before switching to 64 bit kernel, and never happened since.
randyoo
on 26 Oct 2020
Okay, so for me this issue is resolved. I just set
arm_64bit=1into /boot/config.txt and my Pi runs since 8 days without any OOM reaping. Again, this is the Pi4 / 4GB version.
I did a test of copying of 150 GB into and outside the pi external USB drive without oom killer so I consider it success. How you came to this solution please?
Lipown
on 26 Oct 2020
@Lipown: There is unfortunately no reasonable explanation from my side. I just browsed through all related topics from the forums and did some trial-and-error tests.
jozsefDevs
on 28 Oct 2020
Using arm_64bit=1 breaks vlc hardware h264 decoding. It's not a viable option for me until that's fixed. But, good to know we can address this issue in at least some cases.
balajisraghavan
on 28 Oct 2020
Phil, I have seen similar symptoms on another platform. With a 32-bit kernel a high rate of I/O lead to fragmentation of kernel lowmem (memory with a direct mapping to kernel address space, I think on ARM 32-bit this is a contiguous 512MB). That fragmentation lead to longer CPU and elapsed times as the kernel worked to obtain memory for buffers, which is accounted towards the process doing the large amount of I/O. That in turn lead to selection of the process by the OOM killer. Moving to a 64-bit kernel (with its massively greater lowmem) 'solved' our problem. I haven't investigated your particular issue, but thought sharing this experience might be helpful. All the best, Glen.
vk5tu
on 2 Nov 2020
Phil, I have seen similar symptoms on another platform. With a 32-bit kernel a high rate of I/O lead to fragmentation of kernel lowmem (memory with a direct mapping to kernel address space, I think on ARM 32-bit this is a contiguous 512MB). That fragmentation lead to longer CPU and elapsed times as the kernel worked to obtain memory for buffers, which is accounted towards the process doing the large amount of I/O. That in turn lead to selection of the process by the OOM killer. Moving to a 64-bit kernel (with its massively greater lowmem) 'solved' our problem. I haven't investigated your particular issue, but thought sharing this experience might be helpful. All the best, Glen.
That's interesting. I think my issues are initially caused by these errors:
xhci_hcd 0000:01:00.0: WARN Cannot submit Set TR Deq Ptr
xhci_hcd 0000:01:00.0: A Set TR Deq Ptr command is pending.
usb 2-1: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Which then impacts everything else as the system starts to cache more and more as it isn't able to write to the USB device for some reason. I still haven't figured out why this is. It will be fine for days and then all of a sudden get those errors and I don't know any way of recovering it other that rebooting the Pi.
It's a shame that there isn't any kind of reset to the ports as I think this would fix it, but all the automated stuff the OS tries doesn't work!
I've swapped to 64 bit now too, just to see if this helps.
bkev
on 2 Nov 2020
I can confirm I have similar issue with Raspberry Pi 4 8GB.
If I run deluged and download speed reaches smth higher than 5MB/s it will suddenly cause OOM killer to kill deluged or some other process. USB3 SATA SSD is connected
Enabled swap 1GB to SSD with no luck.
WIll give it a try with arm_64bit=1, do you know any issues with this, besides the one mentioned above regarding VLC hardware decoding?
UP:
Indeed adding arm_64bit=1 in /boot/config.txt (and rebooting to apply changes) fixes the issue with oom killer for particular use case. Thank you @mandev!
vovan-
on 9 Nov 2020
I have what appears to be a similar issue, so FWIW here are some slightly different data points.
Scenario: prepping a new USB hard drive via command line luksFormat, luksOpen, and dd'ing /dev/zero until the drive cries 'No mas!' Encrypting a 4TB drive will take several days and hence it's perfect task for a Pi 4 4G I have on the network. I can connect via VNC to set things up and then occasionally check the progress on a display off to the side.
This afternoon -- a few hours shy of the anticipated conclusion -- I glance over at the display to see the Raspbian OS login box. This can't be good, I think. But the VNC server is working fine and lsblk -f still shows the encrypted partition as previously mapped. What's going on? Checking the last few lines of /var/log/kern.log reveals the dreaded oom_reaper:
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178931] [ 5997] 0 5997 2487 804 36864 0 0 sudo
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178938] [ 6002] 0 6002 1882 630 40960 0 0 dd
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178946] [ 2829] 0 2829 2487 810 45056 0 0 sudo
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178953] [ 2830] 0 2830 2016 736 40960 0 0 htop
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178963] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=Xorg,pid=545,uid=0
Nov 14 15:59:49 BlackBoxPi kernel: [471926.179006] Out of memory: Killed process 545 (Xorg) total-vm:147712kB, anon-rss:4612kB, file-rss:13852kB, shmem-rss:17212kB, UID:0 pgtables:296kB oom_score_adj:0
Nov 14 15:59:49 BlackBoxPi kernel: [471926.189788] oom_reaper: reaped process 545 (Xorg), now anon-rss:0kB, file-rss:8100kB, shmem-rss:17272kB
Nov 14 15:59:51 BlackBoxPi kernel: [471929.001990] broken atomic modeset userspace detected, disabling atomic
and there's a similar report at the end of dmesg.
Hardware: known good supply, electronic load tested to 5.2 V @ 3.0 A. Only peripheral attached is a powered USB3 hub, which the new USB3 Toshiba drive is plugged into.
Software: Up to date OS and apps. Only application open is LXTerminal with tabs for dd and htop.
Per previous messages, here's the data dump:
Bus 002 Device 009: ID 0480:0820 Toshiba America Inc Canvio Advance Disk
Bus 002 Device 002: ID 05e3:0616 Genesys Logic, Inc. hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
od -An -tx1 /proc/device-tree/emmc2bus/dma-ranges:
00 00 00 00 c0 00 00 00 00 00 00 00 00 00 00 00
40 00 00 00
Revision : c03111
Linux BlackBoxPi 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 1 11008 3197780 167708 250456 0 0 18 21 0 7 0 14 76 10 0
total used free shared buff/cache available
Mem: 3919788 150556 3508824 82076 260408 3555096
Swap: 102396 11008 91388
If I get particularly curious/motivated I'll test things in 64bit mode; before reading this thread I simply punted by creating a new partition for the last part of the drive and repeating the previous procedure.
rtsperry
on 15 Nov 2020
The 64 bit mode really works. I am finally running a NAS setup on PI
without issues.
On Sun, 15 Nov 2020 at 06:51, rtsperry notifications@github.com wrote:
I have what appears to be a similar issue, so FWIW here are some slightly
different data points.Scenario: prepping a new USB hard drive via command line luksFormat,
luksOpen, and dd'ing /dev/zero until the drive cries 'No mas!' Encrypting a
4TB drive will take several days and hence it's perfect task for a Pi 4 4G
I have on the network. I can connect via VNC to set things up and then
occasionally check the progress on a display off to the side.This afternoon -- a few hours shy of the anticipated conclusion -- I
glance over at the display to see the Raspbian OS login box. This can't be
good, I think. But the VNC server is working fine and lsblk -f still
shows the encrypted partition as previously mapped. What's going on?
Checking the last few lines of /var/log/kern.log reveals the dreaded
oom_reaper:Nov 14 15:59:49 BlackBoxPi kernel: [471926.178931] [ 5997] 0 5997 2487 804 36864 0 0 sudo
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178938] [ 6002] 0 6002 1882 630 40960 0 0 dd
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178946] [ 2829] 0 2829 2487 810 45056 0 0 sudo
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178953] [ 2830] 0 2830 2016 736 40960 0 0 htop
Nov 14 15:59:49 BlackBoxPi kernel: [471926.178963] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=Xorg,pid=545,uid=0
Nov 14 15:59:49 BlackBoxPi kernel: [471926.179006] Out of memory: Killed process 545 (Xorg) total-vm:147712kB, anon-rss:4612kB, file-rss:13852kB, shmem-rss:17212kB, UID:0 pgtables:296kB oom_score_adj:0
Nov 14 15:59:49 BlackBoxPi kernel: [471926.189788] oom_reaper: reaped process 545 (Xorg), now anon-rss:0kB, file-rss:8100kB, shmem-rss:17272kB
Nov 14 15:59:51 BlackBoxPi kernel: [471929.001990] broken atomic modeset userspace detected, disabling atomicand there's a similar report at the end of dmesg:
[471926.178946] [ 2829] 0 2829 2487 810 45056 0 0 sudo
[471926.178953] [ 2830] 0 2830 2016 736 40960 0 0 htop
[471926.178963] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=Xorg,pid=545,uid=0
[471926.179006] Out of memory: Killed process 545 (Xorg) total-vm:147712kB, anon-rss:4612kB, file-rss:13852kB, shmem-rss:17212kB, UID:0 pgtables:296kB oom_score_adj:0
[471926.189788] oom_reaper: reaped process 545 (Xorg), now anon-rss:0kB, file-rss:8100kB, shmem-rss:17272kB
[471929.001990] broken atomic modeset userspace detected, disabling atomicHardware: known good supply, electronic load tested to 5.2 V @ 3.0 A. Only
peripheral attached is a powered USB3 hub, which the new USB3 Toshiba drive
is plugged into.
Software: Up to date OS and apps. Only application open is LXTerminal with
tabs for dd and htop.Per previous messages, here's the data dump:
Bus 002 Device 009: ID 0480:0820 Toshiba America Inc Canvio Advance Disk
Bus 002 Device 002: ID 05e3:0616 Genesys Logic, Inc. hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hubod -An -tx1 /proc/device-tree/emmc2bus/dma-ranges:
00 00 00 00 c0 00 00 00 00 00 00 00 00 00 00 00
40 00 00 00Revision : c03111
Linux BlackBoxPi 5.4.72-v7l+ #1356 SMP Thu Oct 22 13:57:51 BST 2020 armv7l GNU/Linux
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 1 11008 3197780 167708 250456 0 0 18 21 0 7 0 14 76 10 0total used free shared buff/cache availableMem: 3919788 150556 3508824 82076 260408 3555096
Swap: 102396 11008 91388If I get particularly curious/motivated I'll test things in 64bit mode;
before reading this thread I simply punted by creating a new partition for
the last part of the drive and repeating the previous procedure.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raspberrypi/linux/issues/3210#issuecomment-727519865,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANTOVFYKE3ZAR7LY7BFG4K3SP5T5ZANCNFSM4IUF6ESA
.
Lipown
on 15 Nov 2020
A follow-up (anecdotal) report on encrypting my 4T Toshiba hard drive with 32-bit software: I downloaded the latest 32-bit Ubuntu server image and performed the same procedure as described above on the same hardware and encrypted the 4T drive without incident. I next tried using a Pi 4 8G model configured to boot into 32-bit CLI mode. Although the Alt-F1 dd process concluded successfully, the htop instance on Alt-F2 fell victim to the oom_reaper. Finally, I tested things with my new Pi 400 and it didn't even make it half-way before aborting things with oom-killer messages. Reviewing the Pi 400's journal output I noticed it was a full half hour from the RNG reporting no problems to the OOM errors, with no messages between indicating problems.
My conclusion is the unremarkable observation that this isn't a hardware issue as originally postulated. Using the c03130 revision of the BCM2711 in my Pi 400 didn't fix things. I'd be happy to post the relevant journal messages if someone thinks it will help but at this point it doesn't seem a lack of journal logs is the issue. The problem is known, even if the timing of the OOM explosions is somewhat unpredictable.
While I look forward to retesting things when the 64-bit version of Raspian OS makes it out of beta and into gamma I'm also aware that the developers have limited time and a long list of things to work on. Given a shift to 64-bit mode means I'm not going to have access to all of the Pi software ecosystem anyway my next step will be testing an Arch image for my NAS. It's my preferred OS in x86 land and I'm already familiar with it.
rtsperry
on 26 Nov 2020
While I look forward to retesting things when the 64-bit version of Raspian OS makes it out of beta
You don't need to move to 64-bit RpiOS to try this. Just add arm_64bit=1 to config.txt and reboot.
popcornmix
on 27 Nov 2020
Good point, popcornmix. Better is often the enemy of good enough. And it's true that adding arm_64bit=1 to config.txt eliminated oom messages from my logs.
Alas, after a few hours dd froze up completely. htop shows one of the cores at 100%. And my log is littered with thousands of new complaints along the lines of:
Nov 28 06:25:01 BlackBoxPi kernel: sdhci-iproc fe340000.emmc2: swiotlb buffer is full (sz: 8192 bytes), total 32768 (slots), used 27103 (slots)
Nov 28 06:25:01 BlackBoxPi kernel: sdhci-iproc fe340000.emmc2: overflow 0x00000001b258e000+8192 of DMA mask ffffffff bus limit ffffffff
A quick web search shows swiotlb problems in aarch64 are neither new nor unique to Pis. Given my firmware and kernel are current as of Oct 22 yet other users haven't been bitten by swiotlb, I don't know what to say except that my issues with large USB transfers continue. I don't want to clutter a thread with off topic complaints, so I guess I'm back to a strategy of awaiting further developments.
rtsperry
on 28 Nov 2020
I have the same issue with an external HDD (externally powered) and I kind of get the feeling it is not (or not only) related to USB but network (or both). I didn't have time for dedicated tests yet, but so far I only observed it under combined network and USB load.
Output of the commands requested above:
[root@pi4 ~]# grep Revision /proc/cpuinfo
Revision : c03111
[root@pi4 ~]# uname -a
Linux pi4 5.4.75-1-ARCH #1 SMP PREEMPT Mon Nov 9 16:58:39 UTC 2020 armv7l GNU/Linux
[root@pi4 ~]# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 1721236 27264 969116 0 0 28 12 11 14 6 1 93 0 0
[root@pi4 ~]# lsusb
Bus 002 Device 003: ID 1058:0a10 Western Digital Technologies, Inc. My Book Duo 0A10
Bus 002 Device 002: ID 1058:0a21 Western Digital Technologies, Inc. ASM107x
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 1058:0a20 Western Digital Technologies, Inc. USB Hub 0A20
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
[root@pi4 ~]# od -An -tx1 /proc/device-tree/emmc2bus/dma-ranges
00 00 00 00 c0 00 00 00 00 00 00 00 00 00 00 00
40 00 00 00
I am on Archlinuxarm and also noticed that it got a lot worse when I upgraded kernel/firmware before Christmas: the pi basically stopped working (not reachable on network anymore, not even for ping) and I had to pull the power every couple hours to make it reboot. I am running it headless and the logs don't really say anything, they just stop and continue after reboot. I suspect a kernel panic, maybe similar to #4048
Sample log snippet:
[...]
Jan 01 13:49:29 pi4 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=c3bfc687d8cfd370c282dae4db3344e7729bd2d0f910e2ea8938b452101517b2,mems_allowed=0,global_oom,task_memcg=/system.slice/kodi.service,task=kodi-gbm,pid=31172,uid=977
Jan 01 13:49:29 pi4 kernel: Out of memory: Killed process 31172 (kodi-gbm) total-vm:759316kB, anon-rss:91256kB, file-rss:82164kB, shmem-rss:30020kB, UID:977 pgtables:900kB oom_score_adj:0
Jan 01 13:49:30 pi4 kernel: oom_reaper: reaped process 31172 (kodi-gbm), now anon-rss:0kB, file-rss:0kB, shmem-rss:30020kB
Jan 01 13:49:32 pi4 kernel: kworker/0:3 invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Jan 01 13:49:32 pi4 kernel: CPU: 0 PID: 7073 Comm: kworker/0:3 Tainted: G C 5.4.75-1-ARCH #1
Jan 01 13:49:32 pi4 kernel: Hardware name: BCM2711
Jan 01 13:49:32 pi4 kernel: Workqueue: events get_values_poll [raspberrypi_hwmon]
Jan 01 13:49:32 pi4 kernel: [<c0210cdc>] (unwind_backtrace) from [<c020c594>] (show_stack+0x10/0x14)
Jan 01 13:49:32 pi4 kernel: [<c020c594>] (show_stack) from [<c0d069ec>] (dump_stack+0x90/0xb0)
Jan 01 13:49:32 pi4 kernel: [<c0d069ec>] (dump_stack) from [<c0387a30>] (dump_header+0x54/0x1f0)
Jan 01 13:49:32 pi4 kernel: [<c0387a30>] (dump_header) from [<c0386d90>] (oom_kill_process+0x1a4/0x1b0)
Jan 01 13:49:32 pi4 kernel: [<c0386d90>] (oom_kill_process) from [<c0387710>] (out_of_memory+0x10c/0x350)
Jan 01 13:49:32 pi4 kernel: [<c0387710>] (out_of_memory) from [<c03d358c>] (__alloc_pages_nodemask+0xc54/0x12bc)
Jan 01 13:49:32 pi4 kernel: [<c03d358c>] (__alloc_pages_nodemask) from [<c03e7988>] (alloc_slab_page+0x28/0x514)
Jan 01 13:49:32 pi4 kernel: [<c03e7988>] (alloc_slab_page) from [<c03e8180>] (allocate_slab+0x30c/0x340)
Jan 01 13:49:32 pi4 kernel: [<c03e8180>] (allocate_slab) from [<c03e9860>] (___slab_alloc.constprop.0+0x258/0x70c)
Jan 01 13:49:32 pi4 kernel: [<c03e9860>] (___slab_alloc.constprop.0) from [<c03ea07c>] (__kmalloc+0x368/0x38c)
Jan 01 13:49:32 pi4 kernel: [<c03ea07c>] (__kmalloc) from [<c0b66328>] (rpi_firmware_property+0x24/0x108)
Jan 01 13:49:32 pi4 kernel: [<c0b66328>] (rpi_firmware_property) from [<bf09c170>] (get_values_poll+0x40/0x148 [raspberrypi_hwmon])
Jan 01 13:49:32 pi4 kernel: [<bf09c170>] (get_values_poll [raspberrypi_hwmon]) from [<c0248c00>] (process_one_work+0x1e8/0x584)
Jan 01 13:49:32 pi4 kernel: [<c0248c00>] (process_one_work) from [<c0248ff8>] (worker_thread+0x5c/0x568)
Jan 01 13:49:32 pi4 kernel: [<c0248ff8>] (worker_thread) from [<c024eb10>] (kthread+0x170/0x188)
Jan 01 13:49:32 pi4 kernel: [<c024eb10>] (kthread) from [<c02010d8>] (ret_from_fork+0x14/0x3c)
Jan 01 13:49:32 pi4 kernel: Exception stack(0xdfa11fb0 to 0xdfa11ff8)
Jan 01 13:49:32 pi4 kernel: 1fa0: 00000000 00000000 00000000 00000000
Jan 01 13:49:32 pi4 kernel: 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 01 13:49:32 pi4 kernel: 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
Jan 01 13:49:32 pi4 kernel: Mem-Info:
Jan 01 13:49:32 pi4 kernel: active_anon:63859 inactive_anon:55 isolated_anon:0
active_file:20167 inactive_file:94065 isolated_file:88
unevictable:8817 dirty:188 writeback:351 unstable:0
slab_reclaimable:9383 slab_unreclaimable:12897
mapped:40850 shmem:9059 pagetables:1243 bounce:0
free:639597 free_pcp:0 free_cma:59008
Jan 01 13:49:32 pi4 kernel: Node 0 active_anon:255436kB inactive_anon:220kB active_file:80668kB inactive_file:376260kB unevictable:35268kB isolated(anon):0kB isolated(file):352kB mapped:163400kB dirty:900kB writeback:1404kB shmem:36236kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
Jan 01 13:49:32 pi4 kernel: DMA free:236908kB min:2488kB low:3108kB high:3728kB active_anon:48kB inactive_anon:16kB active_file:2948kB inactive_file:2748kB unevictable:0kB writepending:0kB present:524288kB managed:396528kB mlocked:0kB kernel_stack:3272kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:236032kB
Jan 01 13:49:32 pi4 kernel: lowmem_reserve[]: 0 0 3008 3008
Jan 01 13:49:32 pi4 kernel: HighMem free:2321480kB min:512kB low:5448kB high:10384kB active_anon:255388kB inactive_anon:204kB active_file:77780kB inactive_file:373548kB unevictable:35268kB writepending:2376kB present:3080192kB managed:3080192kB mlocked:0kB kernel_stack:0kB pagetables:4972kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jan 01 13:49:32 pi4 kernel: lowmem_reserve[]: 0 0 0 0
Jan 01 13:49:32 pi4 kernel: DMA: 33*4kB (UEC) 45*8kB (UEC) 54*16kB (UEC) 21*32kB (UC) 22*64kB (C) 20*128kB (C) 18*256kB (C) 18*512kB (C) 16*1024kB (C) 18*2048kB (C) 40*4096kB (C) = 236908kB
Jan 01 13:49:32 pi4 kernel: HighMem: 4608*4kB (UM) 7243*8kB (UM) 8635*16kB (UM) 9086*32kB (UM) 6696*64kB (UM) 4565*128kB (UM) 1968*256kB (UM) 501*512kB (UM) 38*1024kB (M) 2*2048kB (UM) 0*4096kB = 2321480kB
Jan 01 13:49:32 pi4 kernel: 122205 total pagecache pages
Jan 01 13:49:32 pi4 kernel: 0 pages in swap cache
Jan 01 13:49:32 pi4 kernel: Swap cache stats: add 0, delete 0, find 0/0
Jan 01 13:49:32 pi4 kernel: Free swap = 0kB
Jan 01 13:49:32 pi4 kernel: Total swap = 0kB
Jan 01 13:49:32 pi4 kernel: 901120 pages RAM
Jan 01 13:49:32 pi4 kernel: 770048 pages HighMem/MovableOnly
Jan 01 13:49:32 pi4 kernel: 31940 pages reserved
Jan 01 13:49:32 pi4 kernel: 65536 pages cma reserved
Jan 01 13:49:32 pi4 kernel: Tasks state (memory values in pages):
Jan 01 13:49:32 pi4 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jan 01 13:49:32 pi4 kernel: [ 224] 0 224 11499 1956 86016 0 -250 systemd-journal
Jan 01 13:49:32 pi4 kernel: [ 240] 0 240 6501 691 57344 0 -1000 systemd-udevd
Jan 01 13:49:32 pi4 kernel: [ 244] 0 244 3112 31 32768 0 0 lvmetad
Jan 01 13:49:32 pi4 kernel: [ 249] 981 249 3532 783 57344 0 0 systemd-network
Jan 01 13:49:32 pi4 kernel: [ 313] 0 313 20336 149 77824 0 0 rngd
Jan 01 13:49:32 pi4 kernel: [ 315] 979 315 5825 470 61440 0 0 systemd-timesyn
Jan 01 13:49:32 pi4 kernel: [ 326] 975 326 2833 532 49152 0 0 avahi-daemon
Jan 01 13:49:32 pi4 kernel: [ 327] 81 327 2792 465 45056 0 -900 dbus-daemon
Jan 01 13:49:32 pi4 kernel: [ 330] 45 330 46119 6796 253952 0 0 mpd
Jan 01 13:49:32 pi4 kernel: [ 331] 102 331 59384 2383 147456 0 0 polkitd
Jan 01 13:49:32 pi4 kernel: [ 335] 0 335 1958 221 36864 0 -1000 sshd
Jan 01 13:49:32 pi4 kernel: [ 338] 0 338 3436 857 49152 0 0 systemd-logind
Jan 01 13:49:32 pi4 kernel: [ 340] 0 340 3300 284 45056 0 0 fcron
Jan 01 13:49:32 pi4 kernel: [ 341] 65534 341 1566 318 36864 0 0 ympd
Jan 01 13:49:32 pi4 kernel: [ 342] 975 342 2745 111 45056 0 0 avahi-daemon
Jan 01 13:49:32 pi4 kernel: [ 360] 0 360 254918 11151 290816 0 0 dockerd
Jan 01 13:49:32 pi4 kernel: [ 361] 974 361 6231 3745 73728 0 0 unbound
Jan 01 13:49:32 pi4 kernel: [ 369] 45 369 6017 591 57344 0 0 mpdas
Jan 01 13:49:32 pi4 kernel: [ 371] 977 371 2173 399 40960 0 0 kodi-standalone
Jan 01 13:49:32 pi4 kernel: [ 385] 0 385 237531 5626 196608 0 0 containerd
Jan 01 13:49:32 pi4 kernel: [ 551] 0 551 229700 4931 184320 0 0 docker
Jan 01 13:49:32 pi4 kernel: [ 645] 0 645 215509 239 90112 0 0 docker-proxy
Jan 01 13:49:32 pi4 kernel: [ 669] 0 669 215573 239 90112 0 0 docker-proxy
Jan 01 13:49:32 pi4 kernel: [ 751] 0 751 200700 993 53248 0 1 containerd-shim
Jan 01 13:49:32 pi4 kernel: [ 848] 0 848 21239 735 147456 0 0 smbd
Jan 01 13:49:32 pi4 kernel: [ 1024] 0 1024 20715 665 139264 0 0 smbd-notifyd
Jan 01 13:49:32 pi4 kernel: [ 1025] 0 1025 20713 630 135168 0 0 cleanupd
Jan 01 13:49:32 pi4 kernel: [ 1102] 0 1102 3651 917 57344 0 0 systemd
Jan 01 13:49:32 pi4 kernel: [ 1103] 0 1103 4912 609 61440 0 0 (sd-pam)
Jan 01 13:49:32 pi4 kernel: [ 1132] 0 1132 1119 161 32768 0 0 ssh-agent
-- Boot 97e4355932c9466eae691775cc7e6f39 --
Jan 01 15:58:53 pi4 systemd-timesyncd[313]: Initial synchronization to time server 213.251.52.217:123 (0.ch.pool.ntp.org).
Jan 01 15:58:53 pi4 kernel: EXT4-fs (sda1): recovery complete
Jan 01 15:58:53 pi4 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: data=ordered
[...]
Kernel/firmware used:
linux-raspberrypi4 5.4.83-1
linux-firmware 20201218.646f159-1
The crashes stopped once I downgraded again (oom_reaper still triggered frequently, but system keeps running):
linux-raspberrypi4 5.4.75-1
linux-firmware 20201023.dae4b4c-1
SuperBFG7
on 5 Jan 2021
Related issues
dkerr64
·
7Comments
steros76
·
3Comments
pvouzis
·
9Comments
awlx
·
4Comments
incyi
·
9Comments
Most helpful comment
For what it's worth, I had a similar experience. Regular OOM reaper issues before switching to 64 bit kernel, and never happened since.