Grafana: Docker: ARM images doesn't work since v6.4.x

Thanks for reporting this @theWaldschrat we'll investigate it further

@theWaldschrat what device are you using? Is that 32- or 64-bit architecture (armv6m armv7, armv8 etc)?

Maybe we need to include https://pkgs.alpinelinux.org/package/edge/main/armhf/libc6-compat in Grafana docker image?

apk add --no-cache --repository=http://dl-cdn.alpinelinux.org/alpine/edge/main libc6-compat

@theWaldschrat can you confirm that above solves the prioblem? Harder for us to verify without access to an actual ARM device.Docker based arm image maybe a possibility, but appreciate if you can help us out here. Thanks

The device is a Raspberry Pi 4B. Technically it's an ARM64v8, but Raspbian by default runs a 32 bit kernel and userland, so ARM32v7.

uname -a
Host OS:
Linux raspberrypi 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l GNU/Linux
Grafana 6.3.6 image:
Linux 97f0bb9a456d 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l armv7l armv7l GNU/Linux
Grafana 6.4.X (latest) image:
Linux 84a01cb75816 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l Linux

I didn't build a lot of Docker images yet, so I'm not able to try above command in a new image, at least not quickly. But what I did do: Run a grafana:latest container with entry point bash and user root:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana
Here's the result:

• Manually running /run.sh gives the same error message, so that's confirmed.
• Run above command: No error.
• Again run /run.sh: New error messages:

Error relocating /usr/share/grafana/bin/grafana-server: __memset_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __memcpy_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __vfprintf_chk: symbol not found
Error relocating /usr/share/grafana/bin/grafana-server: __fprintf_chk: symbol not found

• Running ldd /usr/share/grafana/bin/grafana-server doesn't complain about missing libraries anymore, but gives the same results as above.

I'm not an expert, but I guess the lib-c are still not compatible.

@theWaldschrat thank you very helpful.

Just to verify some additional things can you specifically try these to verify that you get the same problem:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm32v7-linux:6.4.1

docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm32v7-linux:6.4.0-beta1

Just to be sure, can you also try to run and start grafana-server:
docker run -it -p 3001:3000 --entrypoint="bash" --user=root grafana/grafana-arm64v8-linux:6.4.1

First two all do the same as previously described.
Running /run.sh or directly /usr/share/grafana/bin/grafana-server doesn't make a difference.

The third one doesn't even start with an arch mismatch:
standard_init_linux.go:211: exec user process caused "exec format error"

I have the same issue and had to downgrade to version 6.3.6, so it seems that all Alpine-based 6.4.x images are broken for ARMv7.

Thanks. After you entered bash can you try and install the musl-dev packade using apk add?

musl-dev installs fine for me, but doesn't have an impact on the problem, with or without libc6-compat.

Installing the glibc apks from https://github.com/armhf-docker-library/alpine-pkg-glibc/releases allows grafana-server to start. If I understand the issue correctly, it's better to statically link the binaries with musl, though.

It's the idea of Alpine to statically link to musl instead of dynamic glibc linking. It's faster, smaller, more stable and potentially more secure. At least that's what they say.
But as far as I can see, Grafana is build outside the target image linking against glibc, so it's probably the best idea to either install glibc as above or use a different base image that already includes glibc.

Considering this change effectively broke the docker image for ARM devices, I was expecting something better than a tag of "needs investigation".

Sigh! The curse of "agile development", I guess.

I can reproduce the error in OS X, even though it looks a bit different from yours:

$ docker run --platform arm grafana/grafana
/lib/ld-linux-armhf.so.3: No such file or directory

I will see if I can fix it.

I might have a lead on the root cause for this issue, hoping I can fix it by tomorrow.

Working to solve this by building musl binaries in addition to glibc ones.

Just ran into this as well. My system is aarch64 (RockPro64) and I'm getting the same error:

/run.sh: line 80: /usr/share/grafana/bin/grafana-server: No such file or directory

Based on work in #19798 we've pushed a tag named dev-musl to grafana docker hub repositories. We've only been able to test run the arm and arm64 docker images using emulation so we ask kindly anyone to help out testing the arm and arm64 docker images to verify they work as expected. No manifest have been pushed to grafana/grafana so if you want to try arm or arm64 you have to manually specify correct repository, see below.

linux/amd64:
docker run <args> grafana/grafana:dev-musl

linux/arm64:
docker run <args> grafana/grafana-arm64v8-linux:dev-musl

linux/arm:
docker run <args> grafana/grafana-arm32v7-linux:dev-musl

Please note that these images are based on current developer branch (master/Grafana v6.5.0-pre) of Grafana so if you want to test with an existing Grafana installation please remember to backup your existing data.

Test scope:

• Verify that container can successfully run and using docker logs <image name> shouldn't output anything unexpected like errors.
• If possible, try to add/connect to a datasource and render a dashboard/panel
• Optional: Verify that you can download/install plugins
• Provide feedback in this issue or PR #19798 by making a comment telling which device/architecture you've used and the outcome.

Thanks in advance

$ uname -a
Linux black-pearl 4.14.70-hypriotos-v7+ #1 SMP Sat Sep 22 05:54:18 UTC 2018 armv7l GNU/Linux

LGTM running on a Raspberry 3B

• [x] Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors.
• [x] If possible, try to add/connect to a datasource and render a dashboard/panel
• [x] Optional: Verify that you can download/install plugins

SBC: Cubietruck (aka CubieBoard 3)

$ uname -a
Linux fernia 4.19.62-sunxi #5.92 SMP Wed Jul 31 22:07:23 CEST 2019 armv7l armv7l armv7l GNU/Linux

LGTM

• [x] Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors.
• [x] If possible, try to add/connect to a datasource and render a dashboard/panel
• [x] Optional: Verify that you can download/install plugins

Thanks a lot for the quick responses and help. Very appreciated.

Agreed, thanks for helping test!

On Tue, Oct 22, 2019, 19:05 Marcus Efraimsson notifications@github.com
wrote:

Thanks a lot for the quick responses and help. Very appreciated.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/grafana/grafana/issues/19585?email_source=notifications&email_token=AACEVV4YMCESH5G7XWTY3QLQP4XHPA5CNFSM4I42J4CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6PL5A#issuecomment-545060340,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACEVV3OBIAWAV3ZNAP4XEDQP4XHPANCNFSM4I42J4CA
.

uname -a
Linux raspberrypi4 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

raspberry pi 4b

docker -v
Docker version 19.03.4, build 9013bf5

• [x] Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
• [X] if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb with graph panel
• [ ] Optional: Verify that you can download/install plugins

LGTM: raspberry pi 4

$ uname -a
Linux worker-3 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

• [x] Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
• [x] if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb, prometheus, loki)
• [x] Verify that you can download/install plugins (raintank-worldping-app, grafana-kubernetes-app, devopsprodigy-kubegraf-app, grafana-piechart-panel)

LGTM

rockchip rock64

$ uname -a
Linux rock64 4.4.132-1072-rockchip-ayufan-ga1d27dba5a2e #1 SMP Sat Jul 21 20:18:03 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

• [x] Verify that container can successfully run and using docker logs shouldn't output anything unexpected like errors
• [x] if possible, try to add/connect to a datasource and render a dashboard/panel (influxdb, prometheus, loki)
• [ ] Verify that you can download/install plugins (raintank-worldping-app, grafana-kubernetes-app, devopsprodigy-kubegraf-app, grafana-piechart-panel)

Docker skill level
Noob
Grafana skill level
Noob
-uname -a
Linux SwingerPictureServer 4.19.75-v7+ #1270 SMP ........... armv71 GNU/Linux
HW
Raspberry pi 3B
compose.sh file:
docker run \
--name Grafana_test \
-p 3001:3001 \
-e "GF_SERVER_ROOT_URL=http://:3001" \
-e "GF_SECURITY_ADMIN_PASSOWRD=" \
--mount type=bind,source="/home/pi/DockerConf/Grafana/test/config",target="/etc/grafana":ro \
grafana/grafana-arm32v7-linux:dev-musl

LOG file:
warn: msg="phantomJS is deprecated and will be removed in the future release...

Thanks all. We've merged this to master, but decided to include this fix in Grafana v6.5.0, to be released in a couple of weeks. Until then you may use nightly builds if you want to run Grafana v6.5-pre ARM-compatible Grafana docker images by using the grafana/grafana:master tag.

Please add a note to docker hub so it is easier to find this issue. If you are currently pulling grafana/grafana you still get a non-functional image on armhf.

Just commenting to note that grafana/grafana-arm32v7-linux:latest now works fine for me (this image), so I've unpinned versions 👍

@mhansen you can use directly the base image (grafana/grafana:latest), it's multiarch :)

I'm currently using grafana/grafana:6.5.1@sha256:befcd84da2c1f3310b23d93ba9eec4a80df4c86c04bd39455623ac632fbcefdd in an ARM cluster.

@theWaldschrat @pedroetb @mhansen @herm @SySfRaMe @krystian-wojtas @pgolm @gcgarner @JochenLutz @iwittkau @JasonSwindle @protik77 @ata4 we could use some help testing new builds (Docker images and tar archives) on various ARM architectures, might you lend a hand? We would appreciate it!

The Docker images in question

• grafana/grafana-arm64v8-linux:master-df1d43167af035c6819923ecce135056f37c79c2-new-pipeline
• grafana/grafana-arm64v8-linux:master-df1d43167af035c6819923ecce135056f37c79c2-ubuntu-new-pipeline
• grafana/grafana-arm32v7-linux:master-df1d43167af035c6819923ecce135056f37c79c2-new-pipeline
• grafana/grafana-arm32v7-linux:master-df1d43167af035c6819923ecce135056f37c79c2-ubuntu-new-pipeline

The tar archives in question

MUSL archives are for Alpine Linux, GLIBC ones are for regular Linux distros:

• ARM64/MUSL
• ARM64/GLIBC
• ARMv6/GLIBC
• ARMv7/MUSL
• ARMv7/GLIBC

RPMs

• ARM64
• ARMv7

Docker image
grafana/grafana-arm64v8-linux:master-df1d43167af035c6819923ecce135056f37c79c2-new-pipeline works fine on Raspberry Pi 4B with Kernel 4.19.97-v8+ and Docker CE 19.03.5.

Thanks @volschin!

Had a problem with the container today after running about 24h (no templating init). This is nothing that happend in the last months. So perhaps there is a stability problem.

Had a problem with the container today after running about 24h (no templating init). This is nothing that happend in the last months. So perhaps there is a stability problem.

What kind of problem did you see exactly @volschin?

@aknuds1 sorry, I did not get around to fully testing the new docker arm images yet. Is there a way to automate testing?

I don't have any automated method, sorry @iwittkau.

I'm no longer seeing grafana/grafana:latest as being multiarch, only amd64/linux.

$ docker run --rm mplatform/mquery grafana/grafana
Image: grafana/grafana
 * Manifest List: No
 * Supports: amd64/linux

I've switched to grafana/grafana:master

I'm no longer seeing grafana/grafana:latest as being multiarch, only amd64/linux.

$ docker run --rm mplatform/mquery grafana/grafana
Image: grafana/grafana
 * Manifest List: No
 * Supports: amd64/linux

I've switched to grafana/grafana:master

@mhansen Interesting, thanks for the heads up. I will have to check on this.

For what it's worth, I am using grafana/grafana-arm32v7-linux:latest for now. Though it installed 6.7.1.