Linux: Dropped RX packets - RPi 1, 2, and 3

Any relevant messages in syslog? Also, any ideas when this problems started to occur?

A quick glance and some minor debugging at the driver level seems to indicate that it is not the driver itself that is dropping the packet - the code that increments the rx_dropped counter is not called. So presumably somewhere else in the stack is deciding to drop the packet. I'm not sure how to find out where in the stack though! Will investigate further.

It's quite possible this is harmless. The dropped packet count is not only used for errors, but also flagging up packets that are dropped because the linux stack doesn't handle them (some IPv6 stuff if you don't have IPv6 enabled for example). This might be one of those messages. I will try and find out which it is.

It's not IPv6 related as first thing I usually do after installation is
disabling IPv6 on all interfaces via sysctl.conf kernel parameter.

Is it harmless? You never really know, dropped packets shouldn't be there
for sure.

19.05.2017. 17.11, "James Hughes" notifications@github.com је написао/ла:

It's quite possible this is harmless. The dropped packet count is not only
used for errors, but also flagging up packets that are dropped because the
linux stack doesn't handle them (some IPv6 stuff if you don't have IPv6
enabled for example). This might be one of those messages. I will try and
find out which it is.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/raspberrypi/linux/issues/1954#issuecomment-302729554,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMNhf-Eu7da9v0nABpFCHCfdfwgiJGI5ks5r7bEhgaJpZM4M33E8
.

OK, not IPv6, but that was only given as an example of some packets that are simply discarded by the network stack. Dropped packets are not, necessarily, an error. If the stack gets a correctly formatted packet it doesn't implement, then it is dropped. It's not an error as such. So it is perfectly reasonable to be getting dropped packets and not worry about them. However, in this case I would like to know whether this is the case here.

Here is some (probably quite old) text on why a packet may be dropped.

Also worth noting that even if IPv6 is disable on the Pi, it could still receive IPv6 packets from other devices on the network, which will cause dropped packets

Softnet backlog full -- (Measured from /proc/net/softnet_stat)

Bad / Unintended VLAN tags

Unknown / Unregistered protocols

IPv6 frames when the server is not configured for IPv6

If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

After much faffing, I have build a utility I found called dropwatch, rebuilt the kernel to turn on a particular form of net logging and now have a list of addresses where packets are being dropped. However, cross references those addresses to the kernel isn't giving valid results, so i suspect all the drops are in modules.

What is worth noting is that the ifconfig dropped packets counter is a small subset of the actual number of packets dropped. I can see which address in my list is causing the ifconfigs drops, just need to figure out what code corresponds to that location.

OK, thanks to a timely intervention from @pelwell I do have some idea of the code that is dropping the packets. The ifconfig dropped packet counter appears to be incremented in the __netif_receive_skb_core function, approx line 4214. Still some effort required in backtracking from there to determine the reason for the dropped packets.

I also see these these drops on my pi3

pi@raspberrypi:~ $ uname -a Linux raspberrypi 4.9.13-v7+ #974 SMP Wed Mar 1 20:09:48 GMT 2017 armv7l GNU/Linux

pi@raspberrypi:~ $ ifconfig eth0 Link encap:Ethernet HWaddr b8:27:eb:05:2e:7f inet addr:192.168.10.11 Bcast:192.168.10.255 Mask:255.255.255.0 inet6 addr: fe80::xxxxxxx/64 Scope:Link inet6 addr: 2003:xxxxxx/64 Scope:Global UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14465964 errors:0 dropped:276326 overruns:0 frame:0

If you need more infos feel free to ask !

@JamesH65 so we can conclude issue is located in the kernel core? On some cloud providers I also experience dropped packets in RX line.

I don't think any conclusion can be made since I don't yet know the cause of the dropped packets - it could be entirely benign.

I think everyone sees the dropped packets on the Pi, I certainly have been seeing them for years, but ignored then.

Well, it started to happen only after kernel/firmware update year ago or so.

On this version I have zero dropped packets:

4.1.19+ #858 Tue Mar 15 15:52:03 GMT 2016 armv6l GNU/Linux

Mar 15 2016 14:48:20 
Copyright (c) 2012 Broadcom
version 1bf9a9a77026af9128a339c82d72e331d3532ee4 (clean) (release)

Heh, it might be that this old version has a bug of not incrementing counter :laughing:

Or newer version added more places where the counter could be incremented...

stamster, can you identify the exact update which caused this. See:
https://github.com/Hexxeh/rpi-firmware/commits/master

If you click on each commit the end of the url contains a git hash. Run
sudo rpi-update <hash>
to revert back to that version. Report the first version with the packets dropped error.

(I suspect it will be one of the major bumps - e.g. the first 4.4 kernel)

I've been super busy recently - I'll try to experiment with down-grades this weekend.

I was doing some testing today with the latest Raspbian release, 5GB of data transferred with no dropped packets. This was on ethernet. If doing testing check the latest release first.

Well, I got that kernel 5 days ago and still there were dropped packets.
Now I've fetched:

*** Updating kernel modules
 *** depmod 4.9.37-v7+
 *** depmod 4.9.37+

Let's see...

Let me add my stats: RX drop was about 4% with unpatched rasbian-jessie.
After updating Jessie to 4.9.37-v7+ the drop count "dropped" to 0.005%.
I think this is an improvement.

I still have drops, even with latest kernel.

4.9.37+ #1017 Thu Jul 13 11:14:43 BST 2017 armv6l GNU/Linux
RX packets:62278 errors:0 dropped:4203 overruns:0 frame:0

Hmm, just been doing some testing on this, not seeing dropped packets on the ethernet at all. It is possible that this is environmental?

Linux raspberrypi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux

RX packets:828974 errors:0 dropped:0 overruns:0 frame:0
TX packets:503029 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:1142250236 (1.0 GiB)  TX bytes:43037541 (41.0 MiB)

A similar issue has just come up on the linux networking list, unrelated to Pi. Suggestion is RX queueing problem. I'm not sure how to go further here, since I cannot see any dropped packets on the ethernet connection at all. This is a continuation of the previous stats, after leaving it over the weekend, not though with anything hammering the connection.

RX packets:1685234 errors:0 dropped:0 overruns:0 frame:0
TX packets:515057 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000 
RX bytes:1242494098 (1.1 GiB)  TX bytes:44171593 (42.1 MiB)

@stamster Could you give me some idea of how your network is set up? Since I (nd others) am seeing no errors on the onboard Ethernet at all, it would be interesting to know if you have some odd setup that might be causing an issue.

ping @stamster Are you still seeing this? If so are there any strange things on your network that might have some effect.

Also, I'm currently using Stretch with a 4.13 kernel and seeing no drops. It would be interesting, if you have the ability to build your own kernel and install it, to see if you still see errors with the latest kernel code.

I tested with over 10GB of data sent back and forth with no dropped packets at all. This is on Stretch with a 4.13 kernel (admittedly not yet a release kernel, but will be released eventually). I'm inclined, unless I hear otherwise to close this issue.

Just a thought, but seeing as the ethernet is on USB this may be the same root cause as #2134.

That affected 4.9 (actually from 4.8), but was fixed in 4.9.43, and 4.13.
Worth retesting with the latest 4.9 kernel.

@JamesH65 sorry I missed notifications somehow.

Well, nothing is really odd with my network(s). No smart switches, no VLAN's.
On primary location there's a bunch of servers, so they are 24/7 and don't have any drops, even though they have large traffic (internally).

But on another location where there is no broadcast of any kind as it is home network w/o servers etc, RPi's all have same issue.

Bottomline old kernel on Rpi B+ still have zero drops on the same network where all new kernels and RPi flavors having the drops.

@stamster Any chance of building/running the latest kernel from our github. Using that I never see issues. Which kernel version are you running at the moment on the failing systems? Newer than 4.9.43?

ATM:

Linux raspberrypi 4.9.43+ #1026 Wed Aug 16 22:26:02 BST 2017 armv6l GNU/Linux

eth0      Link encap:Ethernet  HWaddr b8:27:eb:xx:xx:xx
          inet addr:192.168.100.3  Bcast:192.168.100.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:788619 errors:0 dropped:41135 overruns:0 frame:0
          TX packets:135760 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:80246523 (76.5 MiB)  TX bytes:11768548 (11.2 MiB)

up 28 days

Running now:

sudo rpi-update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** Performing self-update
 *** Relaunching after update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
...........

*** Updating firmware
 *** Updating kernel modules
 *** depmod 4.9.50+
 *** depmod 4.9.50-v7+

..............


*** If no errors appeared, your firmware was successfully updated to a31db890da1e5135fe666909d562bdc6113c14d6
 *** A reboot is needed to activate the new firmware

Running now: 4.9.50+ #1035 Wed Sep 13 22:53:09 BST 2017

One minute uptime, and:

RX packets:586 errors:0 dropped:1 overruns:0 frame:0
:/

UPDATE:
6 mins uptime,
RX packets:1952 errors:0 dropped:6 overruns:0 frame:0

So getting 1 drop per minute :) Well, it looks better now, still getting drops but let's see if they'll be many or not.

While on another Rpi, just to the left side of the problematic one and on the same network/room/setup:

Uptime: 28 days
RX packets:481057 errors:0 dropped:0 overruns:0 frame:0

But with older kernel:

Linux pibplus 4.1.19+ #858 Tue Mar 15 15:52:03 GMT 2016 armv6l GNU/Linux

Any other RPI I have tried with newer kernel than ~ above mentioned will have drops as well (Rpi 1, 2, and 3).

So weird. As stated above, I have run for hours with the latest kernel, somewhat newer than yours (4.13), no drops at all. I do wonder what the difference is.

Is it currently good enough to wait until 4.12/13 is released and see how that performs?

Kernel 4.9.50+
up 2 days, 16:16,
RX packets:78308 errors:0 dropped:3855 overruns:0 frame:0

Is it currently good enough to wait until 4.12/13 is released and see how that performs?

Sure, like I have something to lose :)

Well, you could try 4.13 from our repo, seems pretty stable to me. I would
be interested to know if that exhibits the issue. Meanwhile, I'll try and
find time to set up a system with that kernel to see if I can replicate.

On 18 September 2017 at 10:17, Jonathan Aaron Steel <
[email protected]> wrote:

Kernel 4.9.50+
up 2 days, 16:16,
RX packets:78308 errors:0 dropped:3855 overruns:0 frame:0

Is it currently good enough to wait until 4.12/13 is released and see how
that performs?
Sure, like I have something to lose :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/raspberrypi/linux/issues/1954#issuecomment-330166744,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADqrHb-yX9myPNlSSmVnREX6kCTf-d33ks5sjjUsgaJpZM4M33E8
.

--
James Hughes
Principal Software Engineer,
Raspberry Pi (Trading) Ltd

I'll spend a few minutes trying to track down what conditions a dropped frame gets logged.
There are various legit reasons for dropping frames, and it's possible those only occur due to a device on one of your networks. The later kernels may have actually had a fix to correctly log these events.

So with some googling and code digging, the most useful info is https://www.novell.com/support/kb/doc.php?id=7007165

Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:

• Softnet backlog full -- (Measured from /proc/net/softnet_stat)
• Bad / Unintended VLAN tags
• Unknown / Unregistered protocols
• IPv6 frames when the server is not configured for IPv6

If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.

So my reading is that those are all harmless, it is just that packets are being delivered that there are no handlers for further up the stack. Looking at the actual ethernet driver, the only place it flags a drop, it also sets the rx_error flag, but you ifconfig output isn't showing that.

You say:

But on another location where there is no broadcast of any kind as it is home network w/o servers etc, RPi's all have same issue.

No servers, but what about IOT devices such as Sonos, smart thermostats, smart lights, or similar? They all tend to stuff out random data intended only for devices on the local network for auto-discovery.
How have you determined no broadcasts? ARP is broadcast, and without that you haven't got a workable network.

It'll be a bit of a trawl, but I suspect you'll really need to take a tcpdump of the received data until you get a drop, and then look back. 3800 dropped in 78000 over two days should actually be fairly obvious.

My gut feel is that this isn't an issue in the kernel, but probably more acurate reporting. Either that or IPv6 is disabled on the new Pi's - your ifconfig output isn't listing an IPv6 address, but I don't know which level IPv6 needs to be disabled at for the packets to be flagged as dropped.

I'm having the same issue on 4.9.50-v7+ (and earlier) on my RPi 3. I have IPv6 enabled and it is fully functional. The dropped packets are not affecting performance or throughput at all as far as I can tell. I see 2 or 3 drops per minute.

I notice, though, that whenever I run tcpdump on eth0, I never see any drops. Based on this observation, I suspected that if I put the interface into promiscuous mode (using sudo ifconfig eth0 promisc and verifying with netstat -i), the drops would stop, but that it not the case - the drops continue whether the interface is in promiscuous mode or not (if I'm not running tcpdump).

Also, tcpdump is supposed to put the capture interface into promiscuous mode, and you can tell it not to use promiscuous mode by using the -p argument. I notice that when running tcpdump netstat -i does not show the P flag regardless of whether I started tcpdump with -p. I see no drops when running tcpdump whether or not I use -p.

Might be useful to test the latest kernel on this one- we have 4.14 in the repo. There have been a lot of networking changes.

Has any one in the thread had any experience with the latest kernels with regard to this issue? I'm inclined to close this issue in a month or two unless there is evidence that this is still a problem.

RX packets:641083 errors:0 dropped:44651 overruns:0 frame:0
TX packets:83239 errors:0 dropped:0 overruns:0 carrier:0

Linux raspberrypi 4.9.79+ #1091 Tue Feb 6 12:58:13 GMT 2018 armv6l GNU/Linux

Hmm, any chance you could try the 4.14 kernel? Can be installed with rpi-update, but probably not really suitable for production equipment (although we are fairly close to release)

uname -a; uptime; ifconfig

Linux raspberrypi 4.14.24+ #1097 Mon Mar 5 16:16:18 GMT 2018 armv6l GNU/Linux
 12:23:42 up 15:06,  1 user,  load average: 0.12, 0.07, 0.05
eth0      Link encap:Ethernet  HWaddr b8:27:eb:99:20:2e  
          inet addr:192.168.100.3  Bcast:192.168.100.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13497 errors:0 dropped:905 overruns:0 frame:0
          TX packets:4459 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1440583 (1.3 MiB)  TX bytes:397404 (388.0 KiB)

I'll reinstall RPI from scratch and try to reproduce the problem with fresh install, as this is simply not going away. The problem persists no matter which Kernel I use.

Once again, only single RPi flavour working great (e.g. w/o drops) on my network is still this version:

Linux pibplus 4.1.19+ #858 Tue Mar 15 15:52:03 GMT 2016 armv6l GNU/Linux
 12:26:55 up 169 days,  9:57,  2 users,  load average: 0.00, 0.01, 0.05
eth0      Link encap:Ethernet  HWaddr b8:27:eb:a5:f5:ce  
          inet addr:192.168.100.30  Bcast:192.168.100.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8886984 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6771841 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:667223990 (636.3 MiB)  TX bytes:2667069239 (2.4 GiB)

Which aligns with all other servers and devices - network works very well.

Weird. On the ones showing drops is there a decrease in overall performance? As stated above, the number of things that can cause a dropped packet has increased, so it might just be counting stuff it simply wasn't counting before but that still happened.

Have you tried drop_watch? https://jvns.ca/blog/2017/09/05/finding-out-where-packets-are-being-dropped/

IIRC, Dropwatch does compile and run successfully on a Pi so definitely worth trying.

Hi guys,

we have deployed a nice solution to our customer using RPI3 and some are on wifi... ALOT of dropped packets :(

For example 1 min uptime :(

:~$ ifconfig; uptime
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

wlan0     Link encap:Ethernet  HWaddr b8:27:eb:c9:2c:10
          inet addr:10.15.9.204  Bcast:10.15.9.255  Mask:255.255.255.0
          inet6 addr: fe80::ba27:ebff:fec9:2c10/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:437 errors:0 dropped:166 overruns:0 frame:0
          TX packets:387 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:57522 (56.1 KiB)  TX bytes:86780 (84.7 KiB)

 17:05:09 up 1 min,  0 users,  load average: 0.34, 0.16, 0.06

Linux 4.4.34-v7+ #930 SMP Wed Nov 23 15:20:41 GMT 2016 armv7l GNU/Linux

Any ideas ?
Anyone came to any conclusion ?

Guys,

Just ran rpi-update on second rpi3

and result after 10 minutes.... 0 dropped ?

uname -a; uptime; ifconfig wlan0;
**Linux 4.14.24-v7+** #1097 SMP Mon Mar 5 16:42:05 GMT 2018 armv7l GNU/Linux
 17:12:16 up 10 min,  0 users,  load average: 0.02, 0.05, 0.03
wlan0     Link encap:Ethernet  HWaddr b8:27:eb:e1:13:b5
          inet addr:192.168.3.201  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::ba27:ebff:fee1:13b5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1599 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1681 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:253559 (247.6 KiB)  TX bytes:329068 (321.3 KiB)

Will leave it over night and see if there is still such difference.

BR,

AIUI, most of the posts on this issue have been related to the onboard ethernet rather than Wifi. I suspect Wifi might be a environmental issue since we've not had reports on that before.

Although there have been recent changes to the wireless networking drivers that may help.

Edit: I note you have been using 4.4 - that is REALLY old and there have been multiple improvements to the wireless networking since that version.

We are using jessie lite. What do you propose? rpi-update?

With kind regards,

You can upgrade to 4.9 simply using

sudo apt update
sudo apt upgrade

rpi-update is not recommended unless you want really bleeding edge stuff, although currently it pulls in 4.14, which has actually had a lot of testing, and is about to become the default version. When updating, always test first of a less important system just to make sure everything works as you want.

On Jessie that won't get a very recent kernel.

Ah, forgot was using Jessie.

sudo apt dist-upgrade instead?

Go straight to 4.14 with sudo rpi-update - I'm not aware of any issues, and as James says it will be the new standard (and best supported) kernel very soon.

Is there a way to setup a local rpi-update repository as i need to update units inside a very restricted network?
So we are in consesus :) rpi update ? I tried both and both work for me, both have now 0-tomax5 dropped count which is amazing coming down from 2/3 of all received packages...

You mean only for kernel? What about core packages (apt)?
IMHO your best shot for restricted networks is using HTTP proxy. So let them open you one channel only and use that as a HTTP proxy during updates. That's how I do it with Debian/GNU systems for years in the enterprise.

Thank you! We got a limited time window with open network ;)

Are people still seeing overly high dropped packet counts with the latest Raspbian? Or even with rpi-update kernel (4.14.50) if you fancy trying it out?

+1 getting drops on 4.14.50+ #1122 Tue Jun 19 12:21:21 BST 2018

RX packets:116923 errors:0 dropped:7986 overruns:0 frame:0

up 5 days, 13:07

Hmm. No idea what this is. Out of 4M packets sent in a currently running test, I see 7 dropped packets. I can only think its environmental. Bad cable somewhere? Bad router? Interference?

In the same room I have another RPi mentioned earlier, which is unaffected. Same switch used. The only difference I see is the kernel / firmware version (latest vs. older kernel) - e.g. software layer brings this issue.

As a matter of fact, SSH connections are stable 100%, but it's TCP layer to polish things behind the scenes I guess. So no issues I can think of - only that RX dropped counter is increasing constantly.

Can you put both rpi’s one on top of eachother... to check if there is some
interference of other wifi’s on one or other... or mybe none at some point

Dne pon., 2. jul. 2018 ob 23:51 je Jonathan Aaron Steel <
[email protected]> napisal/-a:

In the same room I have another RPi mentioned earlier, which is
unaffected. Same switch used. The only difference I see is the kernel /
firmware version (latest vs. older kernel).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/raspberrypi/linux/issues/1954#issuecomment-401949004,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AKj8_52d7S1Qt4-xqqkO0FdtBl-Sdk93ks5uCpXfgaJpZM4M33E8
.

>

*Alen Lončarič *

Avera d.o.o. ● Pristava pri Mestinju 23 ● 3253 Pristava pri Mestinju
● Slovenia
*e: *alen.[email protected] ● *g: *+386 40 818 861 ● *www: *

www.avera.si

Would it be possible to measure throughput on the two Pi's? It would be interesting to know if the one showing dropped packets is actually slower.

@MrLonko Those are RPi's 1st generation (revisions B and B+). So no WiFi there, only good old ethernet jack.

@JamesH65 sure I can run iperf or something.

I have recently discovered that Raspbian Stretch has a bug which very, very infrequently causes ethernet packets to be dropped. By running 'sudo rpi-update' on my Pi 3b, which upgraded the kernel from 4.7 to 4.14, I have completely eliminated the 2-5 per day kiwirecorder.py restarts from which I was previously suffering. Running 4.14, my WSPR decoding script 'kiwiwspr.sh' can simultaneously record 17 uncompressed audio streams (3.5 Mbps = 750 receive packets per second ) from 3 Kiwis on one Pi without a single restart in 24 hours.

I'm running the latest kernel (4.19.42-v7+) on a 3B and I see 7-15% packets dropped on ping. Transfer speeds are affected by this, too. In my particular case it's on AP mode and wlan0 bridged with eth0. There are no losses when pinging the ethernet side.
RX packets 667550 bytes 89343147 (85.2 MiB)
RX errors 0 dropped 4182 overruns 0 frame 0