Linux: bcm2835_v4l2 on RPi1 kernel oops

Can you identify the exact update which caused this. See:
https://github.com/Hexxeh/rpi-firmware/commits/master

If you click on each commit the end of the url contains a git hash. Run
sudo rpi-update <hash>
to revert back to that version. Report the first version with this error.

At a guess https://github.com/Hexxeh/rpi-firmware/commit/6d158adcc0cfa03afa17665715706e6e5f0750d2 is going to be the first version with the issue - that was the first one to include querying the GPU for max sensor resolution which is done in bm2835_mmal_init.
9dc0b88f1d3b5e9400e4adaadd13a4ef5e7348fa is the one previous to that.

I can't see anything wrong with the handling there so can't currently give any advice. I had tested that fairly thoroughly on Pi2 and CM3, and there's no significant difference in that area for Pi1. Seeing as you're on 4.4.8 I assume you've used rpi-update and updated the firmware as well as kernel. There was a firmware change associated around that request as well.

I can confirm modprobing bcm2835-v4l2 crashes kernel here.

Specs:

Hardware        : BCM2708
Revision        : 000e

Running Arch Linux ARM:

linux-raspberrypi 4.4.8-2
raspberrypi-firmware-tools 20160503-1
raspberrypi-firmware-bootloader-x 20160503-1
raspberrypi-firmware-bootloader 20160503-1
linux-firmware 20160315.deb1d83-1

Not seeing a kernel crash here. What does:

vcgencmd version
uname -a

report?

/opt/vc/bin/vcgencmd version
May  3 2016 17:47:44 
Copyright (c) 2012 Broadcom
version b870a5bbf6c662327916ee582c22309c9f997b3d (clean) (release)

uname -a
Linux alarmpi 4.4.8-2-ARCH #1 Tue Apr 26 19:13:10 MDT 2016 armv6l GNU/Linux

Tried with fresh 4.4.9 kernel:

[  111.269707] media: Linux media interface: v0.10
[  111.370081] Linux video capture interface: v2.00
[  111.490692] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[  111.499124] pgd = d30d0000
[  111.501988] [00000000] *pgd=11050831, *pte=00000000, *ppte=00000000
[  111.508592] Internal error: Oops: 17 [#1] ARM
[  111.513040] Modules linked in: bcm2835_v4l2(+) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core v4l2_common videodev media cfg80211 i2c_bcm2708 spi_bcm2835 bcm2835_gpiomem bcm2835_wdt uio_p
drv_genirq uio sch_fq_codel bcm2835_rng rng_core ip_tables x_tables ipv6
[  111.538374] CPU: 0 PID: 655 Comm: modprobe Not tainted 4.4.9-1-ARCH #1
[  111.545014] Hardware name: BCM2708
[  111.548486] task: d6b0dda0 ti: d3330000 task.ti: d3330000
[  111.554070] PC is at bm2835_mmal_init+0x78/0x738 [bcm2835_v4l2]
[  111.560098] LR is at 0x0
[  111.562689] pc : [<bf1c3078>]    lr : [<00000000>]    psr: a0000113
               sp : d3331d90  ip : 00000000  fp : 2cded15c
[  111.574345] r10: 00000001  r9 : bf1bf394  r8 : 00000000
[  111.579664] r7 : bf1c3000  r6 : 00000000  r5 : 00000000  r4 : d5479800
[  111.586299] r3 : bf1be644  r2 : 00000000  r1 : 00000000  r0 : d5479800
[  111.592939] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  111.600195] Control: 00c5387d  Table: 130d0008  DAC: 00000055
[  111.608475] Process modprobe (pid: 655, stack limit = 0xd3330188)
[  111.617228] Stack: (0xd3331d90 to 0xd3332000)
[  111.624239] 1d80:                                     00000002 c04bb040 ff0a0005 00000000
[  111.637730] 1da0: 00000020 00000001 d3334000 ffffffff 00000a20 00000798 00000000 00000000
[  111.651265] 1dc0: d55e4640 00000000 d3212ec0 00000001 2cded15c c04bdc84 d541e018 d3331e08
[  111.664982] 1de0: d6a19480 bf1bf620 00000001 c0a94f30 c0a94f30 d55e4640 bf1c3000 00000000
[  111.678910] 1e00: d3212ec0 00000001 2cded15c c00094b8 f348c558 00000019 00000001 d3330000
[  111.693103] 1e20: d3330000 d7001e60 d3212ec8 024000c0 c00cf9b8 0000000c d3331e4c c0794be4
[  111.707440] 1e40: 024000c0 d7001e60 d3331e54 c0794c30 2cded15c c0114618 2cded15c c00d6fd8
[  111.721988] 1e60: bf1bf620 00000001 d55e45e0 bf1bf620 bf1bf668 00000001 2cded15c c00cf9f4
[  111.736774] 1e80: d3331f44 00000001 d3331f44 00000001 d3212ec8 c007c2c0 bf1bf62c 00007fff
[  111.751777] 1ea0: bf1bf620 c0079ae4 c0079560 bf1bf718 00000000 d90d9304 bf1bf7d4 bf1bf62c
[  111.767050] 1ec0: 00000000 c079e4d0 00400000 d32123c0 0192b280 c0105160 00400000 00000000
[  111.782634] 1ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  111.798354] 1f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0000e354
[  111.814304] 1f20: 00000000 b4c1335c d90d9354 0002bce8 d3330000 00000000 0192b280 c007c8ac
[  111.830396] 1f40: 00001a04 d90cb000 0000e354 d90d8c9c d90d8ad8 d90d5f24 00007814 00008944
[  111.846489] 1f60: 00000000 00000000 00000000 000031dc 00000029 0000002a 0000001f 00000023
[  111.862574] 1f80: 00000018 00000000 00005a04 0192b060 b4c05008 00040000 00000080 c000f0a4
[  111.878688] 1fa0: d3330000 c000ef00 0192b060 b4c05008 b4c05008 0000e354 0002bce8 c7037e00
[  111.894805] 1fc0: 0192b060 b4c05008 00040000 00000080 0002bce8 00000000 00000000 0192b280
[  111.910935] 1fe0: 0003e164 bed28930 00020bb4 b6e34670 60000010 b4c05008 8080827f 7f808283
[  111.927210] [<bf1c3078>] (bm2835_mmal_init [bcm2835_v4l2]) from [<c00094b8>] (do_one_initcall+0x80/0x1dc)
[  111.944797] [<c00094b8>] (do_one_initcall) from [<c00cf9f4>] (do_init_module+0x5c/0x1c0)
[  111.960780] [<c00cf9f4>] (do_init_module) from [<c007c2c0>] (load_module+0x1ad8/0x1fa0)
[  111.976609] [<c007c2c0>] (load_module) from [<c007c8ac>] (SyS_init_module+0x124/0x140)
[  111.992332] [<c007c8ac>] (SyS_init_module) from [<c000ef00>] (ret_fast_syscall+0x0/0x34)
[  112.008146] Code: 0a000188 e59d200c e59f3634 e082e186 (e792c186)
[  112.018482] ---[ end trace 8fb7666a31e04815 ]---
[  113.949638] cfg80211: Verifying active interfaces after reg change
[root@campi kad]# /opt/vc/bin/vcgencmd version
May  6 2016 13:57:24
Copyright (c) 2012 Broadcom
version 0cc642d53eab041e67c8c373d989fef5847448f8 (clean) (release)
[root@campi kad]# uname -a
Linux campi 4.4.9-1-ARCH #1 Fri May 6 12:52:21 MDT 2016 armv6l GNU/Linux
[root@campi kad]#

kernel build corresponds to commit 68bead249ca6ad97fc39a0780471f4b65402afc1
reference: https://github.com/archlinuxarm/PKGBUILDs/blob/master/core/linux-raspberrypi/PKGBUILD

@kad were you running raspbian or arch?

Place where I can reproduce this crash is running Arch

I don't believe this affects raspbian. Arch builds a custom kernel, so the problem would be better reported there.

Issue reported in both places...
Arch uses kernel from this repository, and according to PKGBUILD, I don't see how userspace components or additional patches for aufs4 and BFQ would be able to interfere with V4L, thus reported upstream, meaning to this repo.

Arch builds a custom kernel based on the sources here, but with different patches and config options so we can't offer support for issues that affect Arch and not raspbian.

Reproduce the issue with raspbian, and we can help more.

Can you point to any raspbian (test) image that has 4.4.8+ kernel ? places where I have raspbian, all running 4.1.19.

Just use latest raspbian and run rpi-update.
Or wait about a week for an updated raspbian image.

Indeed, with binary kernel from https://github.com/Hexxeh/rpi-firmware I can't reproduce same crash.
There is no code difference between Arch kernel and rpi-firmware in that space, but kernel configs are different.

CC: @kmihelich

At same time, briefly looking at bcm2835-camera.c I can see couple of places where pointers followed without verification that they are not NULL :(

This to me looks very much like a bug report for a kernel, which is what this repository is for. Not for a distribution of Debian rebuilt for ARMv6.

A bug in your code is obviously present, given the logs above. If a bug in your kernel has surfaced due to a different configuration, you should investigate why your code is failing. If you aren't checking your pointers adequately and this somehow isn't a problem in your particular configuration and build environment, that does _not_ mean it is not a problem with your code. Attempting to sweep this under the rug by saying, "you aren't using our configuration, therefore it's not our problem," is a rather unprofessional attitude to take to kernel development.

Bugs in, or the lack of, error handling are some of the most common sources of issues. Although we will investigate and fix any bugs we find, that may not help your users who are still faced with an error message, so check your house is in order.

We're talking about a null pointer dereference error message _from the kernel_. This message is not coming from userspace code.

Which, if it is in error path, is provoked by an error (hence the name). Since Raspbian doesn't exhibit the crash, the error is likely to be caused by either a different kernel configuration or different user space activity, which you may want to look into.

In other words, if the crash is caused by an error, fixing the crash won't fix the error.

And when the kernel stops dereferencing a null pointer, userspace can be looked into at that time to determine if the issue is still present.

Some people appreciate the heads-up...

Where can I download the precise bcm2835-v4l2 module that corresponds to this trace? I know the kernel commit, but since I want to disassemble it I want to avoid toolchain and build option differences.

http://mirror.archlinuxarm.org/armv6h/core/linux-raspberrypi-4.4.9-1-armv6h.pkg.tar.xz

Keep in mind that newer versions of GCC do a better job of exposing incorrect code that causes these types of issues. I've seen similar issues from in-house kernel code that was only ever built with old versions such as 4.9 and previous; never qualified against current stable versions.

That specific build was against GCC 5.3.0, and you can get a copy of the cross-compile toolchain here to test with:
https://archlinuxarm.org/builder/xtools/5.3.0-5/

Any future builds will be built with GCC 6.1.1, which has a corresponding cross toolchain in the immediate subdirectory.

I've downloaded the archive and extracted the module, but it doesn't match the trace - the .ko doesn't even contain the symbol bm2835_mmal_init.

The modules are gzip'd.

nm bcm2835-v4l2.ko | grep bm2835_mmal_init
00000000 t bm2835_mmal_init
00002ce0 T bm2835_mmal_init_controls

Interesting - the objdump in my toolchain is reporting the aliased name instead - init_module.

If it is indeed a matching component then it is a very strange crash. It looks as though the address of a local variable (resolutions), which has been stored on the stack (both the data and a pointer to it), is now zero (the pointer, not the data).

Line 1877 appears to be the faulting line. At that point, r2 is resolutions (loaded from sp+12 - 00000000), while ip holds the address of resolutions[camera][0] (also zero, because camera (r6) is zero).

FYI, here is config diff between kernel from Raspbian project and Archlinux:
https://gist.github.com/kad/bbc59804a9ce640124719979e531ad2a
Raspbian binary kernel able to boot and load module without problems (same device, same bootloaders, same userspace in both test cases, only difference is /lib/modules/* and /boot/kernel.img)

@pelwell in line 1532 there is potential issue where stack after resolutions[] would be overwritten, due to fact of writing to index that is bigger than MAX_BCM2835_CAMERAS (size of resoltions[]). it shouldn't be the case, as cam_info.num_cameras should be 0 or 1... but still.

So raspbian kernel works okay with arch image?
Does it still work if you rebuild it using arch tools, but raspbian .config? I wonder if gcc version or build flags may have an effect.

Binary kernel from https://github.com/Hexxeh/rpi-firmware works on top of Arch image.
Haven't tried to rebuild it, as so far don't have cross-compilation environment setup for those purposes.
So far was just happy user than actual developer for arch packages :)

We can also try the pre-compiled module with an otherwise straight Raspbian installation.

The kernel used in Alarm has seen numerous updates.
The issue seems to remain. Any news about this?

My suggestion is for Arch maintainers to build a kernel using their compiler and our config (bcm2709_defconfig).
If that works it would suggest a config issue which can be tracked down.
If that doesn't work it suggests a problem with the tools (perhaps gcc version or build flags).

@pelwell in line 1532 there is potential issue where stack after resolutions[] would be overwritten, due to fact of writing to index that is bigger than MAX_BCM2835_CAMERAS (size of resoltions[]). it shouldn't be the case, as cam_info.num_cameras should be 0 or 1... but still.

https://github.com/raspberrypi/linux/blob/rpi-4.4.y/drivers/media/platform/bcm2835/bcm2835-camera.c#L1532

    for (i = 0;
         i < (cam_info.num_cameras > num_resolutions ?
            cam_info.num_cameras :
            num_resolutions);

Yup, minor bug - the two results are reversed.

         i < (cam_info.num_cameras > num_resolutions ?
            num_resolutions :
            cam_info.num_cameras);
);

So if the GPU did return "big number" you could run off the end of the array, but otherwise you'd always end up with num_resolutions values being written into resolutions.
@popcornmix Could you review and create a quick patch for me on that one - I'm not in a position to create one at the moment.

Won't result in a dereference of NULL though, just stack trash. It would also imply you've got a very screwed firmware/kernel to get such rubbish back from the GPU.

Okay pushed the fix. I agree fix is correct, but I can't see the error causing this issue, but it could be worth a test with Arch.

This line (https://github.com/raspberrypi/linux/blob/rpi-4.4.y/drivers/media/platform/bcm2835/bcm2835-camera.c#L1522) also bothered me, but again I couldn't see how it could trash the stack.

I don't see L1522 having an issue.
It's saying that in the event of failing to create the component, guess that there is 1 camera present. It might be better to set resolution[0][] to 2592x1944 too - there's no way that imx219 will work against a firmware not supporting camera_info correctly, so guess at OV5647's 5MP.
Alternatively say no camera (as it's going to be ancient firmware and highly likely to fail), and return 0.

The camera_info component has been in the firmware build for probably the entire lifespan of Pi. Failure on the component create is likely to result in all future calls failing.

Still no NULL dereference or stack trash.

I have just seen what look like 2 incorrect error handling paths that could leak resources (but not crash).
https://github.com/raspberrypi/linux/blob/rpi-4.4.y/drivers/media/platform/bcm2835/bcm2835-camera.c#L1874 ought to be goto free_dev: so that other nodes are cleaned up.
Line 1957 should be for ( ; camera >= 0; camera--) otherwise device index 0 won't be cleaned up.
Both issues would only hit Compute Module users, as otherwise num_cameras would be 1.

Always happy for code reviews, but there aren't volunteers clamouring for the job.

Ah the camera >= 0 change was what I was suggesting here:
https://github.com/raspberrypi/linux/pull/1393#discussion_r58740685

Ah the camera >= 0 change was what I was suggesting here: #1393 (comment)

Apologies - I got the second bit from that comment, missed the first. :-(

Hi
I got a similar problem with raspi camera module, at boot I got a kernel oops.
Maybe it's related to your issue?
If I can help somehow?

It seems kernel 4.4.7 works. Were can we find the old pkg file? I don't have it on my pacman cache (new install). I can be temporary a fix.

@ouafnico Arch is not the supported distribution - please ask them where to find packages.

If you read the details here, it is only Arch that has an issue. Even the Arch distro with Raspbian kernel works.

There are two potential commits that could have caused this: a980ac51b05f0e0dc275b6be44f00f144af9ab7b (16th Apr) or 5fe9c7b56989bed7360dc560f18cac7f36c9a7d4 (8th April). Inspection of both of them shows no obvious issues.
This linux repo bumped from to 4.4.8 on 20th April, and to 4.4.7on 12th April, so that would indicate a980ac5. How that manages a null pointer deref is not obvious.

Do calls to vchiq potentially can be affected by presence/absence of CONFIG_OABI_COMPAT ?
That's the most suspicious part that I found so far in diff between Arch and Raspbian configs.

It was working on 4.4.7 but if I'm not wrong CONFIG_OABI_COMPAT was on config too for this version.

True, this config option was for a long time. But new code that nowadays query resolution of the camera might expose that, if my suspicion is correct.

is the 4.4.11-4 working ?

No, sorry. As soon as I modprobe the module the RPi is no longer reachable.

No, unfortunately still crashing :(

[root@campi kad]# uname -a
Linux campi 4.4.11-4-ARCH #1 Tue May 31 19:50:35 MDT 2016 armv6l GNU/Linux
[root@campi kad]# modprobe bcm2835-v4l2
Segmentation fault
[root@campi kad]# dmesg 
[   69.272269] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[   69.280589] pgd = d30a4000
[   69.283483] [00000000] *pgd=156ed831, *pte=00000000, *ppte=00000000
[   69.290006] Internal error: Oops: 17 [#1] ARM
[   69.294448] Modules linked in: bcm2835_v4l2(+) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core v4l2_common videodev media cfg80211 i2c_bcm2708 spi_bcm2835 bcm2835_gpiomem bcm2835_wdt uio_pdrv_genirq uio sch_fq_codel bcm2835_rng rng_core ip_tables x_tables ipv6
[   69.319752] CPU: 0 PID: 632 Comm: modprobe Not tainted 4.4.11-4-ARCH #1
[   69.326476] Hardware name: BCM2708
[   69.329944] task: d3003580 ti: d5480000 task.ti: d5480000
[   69.335524] PC is at bm2835_mmal_init+0x78/0x768 [bcm2835_v4l2]
[   69.341547] LR is at 0x0
[   69.344136] pc : [<bf1c8078>]    lr : [<00000000>]    psr: a0000113
               sp : d5481d98  ip : 00000000  fp : bf1c4688
[   69.355791] r10: 00000000  r9 : bf1c439c  r8 : 00000000
[   69.361105] r7 : bf1c8000  r6 : 00000000  r5 : 00000000  r4 : d6b1c800
[   69.367738] r3 : bf1c364c  r2 : 00000000  r1 : 00000000  r0 : d6b1c800
[   69.374371] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   69.381624] Control: 00c5387d  Table: 130a4008  DAC: 00000055
[   69.389935] Process modprobe (pid: 632, stack limit = 0xd5480188)
[   69.398686] Stack: (0xd5481d98 to 0xd5482000)
...
[   69.707458] [<bf1c8078>] (bm2835_mmal_init [bcm2835_v4l2]) from [<c00094e8>] (do_one_initcall+0x80/0x1d8)
[   69.725075] [<c00094e8>] (do_one_initcall) from [<c00d05a8>] (do_init_module+0x5c/0x1a8)
[   69.741113] [<c00d05a8>] (do_init_module) from [<c007c960>] (load_module+0x1ac0/0x1fec)
[   69.756997] [<c007c960>] (load_module) from [<c007cfac>] (SyS_init_module+0x120/0x144)
[   69.772754] [<c007cfac>] (SyS_init_module) from [<c000ee00>] (ret_fast_syscall+0x0/0x34)
[   69.788592] Code: 0a000174 e59d200c e59f3664 e082e186 (e792c186)
[   69.798911] ---[ end trace 97f95d643ae9c613 ]---

To clarify that: raspistill is making absolutely fine images. So hardware and connection is not the problem.

4.4.12-1-ARCH, camera and bcm2835_v4l2 works again.

Can confirm: works again also for me.

Bug can be closed.

Do we know what was changed to fix it?

Arch kernel moved from 37b930adffa15bfaa9b3ffb83e021afff31bb3d5 to e64101f36454e54b877fde44ed866320636e590b

@popcornmix , so I suspect this "bcm2835-camera: Fix max/min error when looping over cameras/resolutions". Maybe something in addition...

Can you actually use the video device once the module is loaded?
I've tried several ways to create streams using VLC but the client does not seem to get any data and fail to fill its buffer. Tried both HTTP and RTSP.

Motion does seem to be able to use the device but it also maxes out the CPU so the framerate is terrible.

Bypassing the v4l2 device and piping raspivid through VLC does work so I don't think I set it up wrong. That gives a delay of several seconds though.

@TwoD MJPGStreamer with 4.4.12 works for me.

@kad Well, MJPGStreamer does work, but it does not use the v4l2 device so it doesn't need the bcm2835_v4l2 kernel module.

in my case, mjpgstreamer is using v4l:

mjpg_streamer -i input_uvc.so -d /dev/video0  -r 1280x720 -f 15 -o output_http.so -p 8090 -w /usr/share/mjpeg-streamer/www/

@kad Where did you get that version from? The one I have (https://github.com/jacksonliam/mjpg-streamer) does not recognize the -d argument.
Its upstream seems to contain another version of mjpg_streamer (without "-experimental"). It does mention something about it being possible to compile with V4L2 support, but no -d argument. The documentation feels practically useless. :/

@kad Never mind, I got it working with that version, was using the wrong input module. Thanks for the hint. Not sure why it won't work with VLC yet, but this does prove the device should be working.

(For the record, the command above is missing quotes. It should be mjpg_streamer -i "input_uvc.so -d /dev/video0 -r 1280x720 -f 15" -o "output_http.so -p 8090 -w /path/to/mjpg_streamer/www")

Version I use is the old, original version back from 2009: https://aur.archlinux.org/packages/mjpg-streamer It doesn't contain any other input drivers than uvc. Newer forks have other drivers and a bit different arguments. But just to check module working, you can use v4l2-ctl to query camera parameters at least

This appears to be rearing its head again, in a slightly different manner:
https://archlinuxarm.org/forum/viewtopic.php?f=15&t=11354&p=54830#p54823

Users have traced it specifically down to the bcm2835_v4l2 & bm2835_mmal_init at what seems to be https://github.com/raspberrypi/linux/blob/204050d0eafb565b68abf512710036c10ef1bd23/drivers/media/platform/bcm2835/bcm2835-camera.c#L1901

http://lists.infradead.org/pipermail/linux-rpi-kernel/2017-March/005741.html has fingered a potentially dodgy memcpy on probing the camera parameters.
I'm investigating, but at first glance it looks like the person who re-implemented the MMAL within the kernel didn't quite make it match.
https://github.com/raspberrypi/userland/blob/master/interface/mmal/vc/mmal_vc_api.c#L1187
!=
https://github.com/raspberrypi/linux/blob/rpi-4.9.y/drivers/media/platform/bcm2835/mmal-vchiq.c#L1307

Any mismatch in parameter size is an error further up, but I'm not immediately seeing anything obvious.
https://github.com/raspberrypi/userland/blob/master/interface/mmal/mmal_parameters_camera.h#L526 appears to match https://github.com/raspberrypi/linux/blob/rpi-4.9.y/drivers/media/platform/bcm2835/mmal-parameters.h#L682, and both firmware and userland use the first of those. A few debug prints of sizeof(struct) are called for.

BTW, I also noticed that the function in the kernel has some strange code to add 8 bytes to the size it sends to the firmware.

static int port_parameter_get(struct vchiq_mmal_instance *instance,
struct vchiq_mmal_port *port,
u32 parameter_id, void *value, u32 *value_size)
{
int ret;
struct mmal_msg m;
struct mmal_msg *rmsg;
VCHI_HELD_MSG_T rmsg_handle;

    m.h.type = MMAL_MSG_TYPE_PORT_PARAMETER_GET;

    m.u.port_parameter_get.component_handle = port->component->handle;
    m.u.port_parameter_get.port_handle = port->handle;
    m.u.port_parameter_get.id = parameter_id;
    m.u.port_parameter_get.size = (2 * sizeof(u32)) + *value_size;

@6by9 I believe this can be closed? Just want to double check.

Yes, can be closed.