Swapper : page allocation failure

[saxmark]

Hi..

My bubba has been acting strange in a while.. Dropps USB devices and i have to start up samba manually a couple of times / day..

I get loads of these in dmesg.

swapper: page allocation failure. order:1, mode:0x20
Call Trace:
[c0403a70] [c0008334] 0xc0008334 (unreliable)
[c0403aa0] [c0063568] 0xc0063568
[c0403b30] [c00844c0] 0xc00844c0
[c0403b90] [c00848dc] 0xc00848dc
[c0403bb0] [c02743c8] 0xc02743c8
[c0403bd0] [c027454c] 0xc027454c
[c0403bf0] [c02ada94] 0xc02ada94
[c0403c00] [c02c4298] 0xc02c4298
[c0403c20] [c02c2e34] 0xc02c2e34
[c0403c40] [c02c4120] 0xc02c4120
[c0403c90] [c02c25c8] 0xc02c25c8
[c0403cc0] [c02c2be0] 0xc02c2be0
[c0403cf0] [c02a4478] 0xc02a4478
[c0403d10] [c02a3e58] 0xc02a3e58
[c0403d40] [c0281244] 0xc0281244
[c0403d70] [c0201d2c] 0xc0201d2c
[c0403dc0] [c02022ec] 0xc02022ec
[c0403e10] [c0281cb8] 0xc0281cb8
[c0403e50] [c0028628] 0xc0028628
[c0403e90] [c0006230] 0xc0006230
[c0403ea0] [c0028414] 0xc0028414
[c0403eb0] [c00062d0] 0xc00062d0
[c0403ed0] [c0011c3c] 0xc0011c3c
--- Exception: 501 at 0xc0009304
LR = 0xc0009304
[c0403f90] [c0009280] 0xc0009280 (unreliable)
[c0403fb0] [c0003ea8] 0xc0003ea8
[c0403fc0] [c03a9794] 0xc03a9794
[c0403ff0] [00003438] 0x003438
Mem-Info:
DMA per-cpu:
CPU 0: hi: 90, btch: 15 usd: 37
active_anon:1263 inactive_anon:10938 isolated_anon:0
active_file:11543 inactive_file:34947 isolated_file:0
unevictable:0 dirty:6296 writeback:0 unstable:0
free:698 slab_reclaimable:1844 slab_unreclaimable:1722
mapped:4894 shmem:1511 pagetables:380 bounce:0
DMA free:2792kB min:2036kB low:2544kB high:3052kB active_anon:5052kB inactive_anon:43752kB active_file:46172kB inactive_file:139788kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:260096kB mlocked:0kB dirty:25184kB writeback:0kB mapped:19576kB shmem:6044kB slab_reclaimable:7376kB slab_unreclaimable:6888kB kernel_stack:672kB pagetables:1520kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 688*4kB 5*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2792kB
48232 total pagecache pages
224 pages in swap cache
Swap cache stats: add 575, delete 351, find 996/1027
Free swap = 1155552kB
Total swap = 1156672kB
65536 pages RAM
1583 pages reserved
59746 pages shared
21225 pages non-shared

I have tried to do a bit googeling but it doesn´t make me any wiser..
My main concern is if i have a problem or just shall redirect syslog -> null ?

System is a bubba2 with a pretty clean install on it..

[RandomUsername]

If you run top, what's the usage stats for your swap space (it will near the top along with CPU and memory)?

[g__]

I have a similar problem when copying large files from an external drive. Is there a fix? (Kernel stack traces worry me; speaking of which, I'm running 2.6.32.13 #1 from the excito apt repo.)

And the kernel messages:

Code: Select all

Oct  5 23:31:32 swedishchef kernel: esent:260096kB mlocked:0kB dirty:24880kB writeback:56kB mapped:26404kB shmem:444kB slab_reclaimable:13220kB slab_unreclaimable:10268kB ker
nel_stack:1040kB pagetables:1756kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct  5 23:31:32 swedishchef kernel: lowmem_reserve[]: 0 0 0
Oct  5 23:31:32 swedishchef kernel: DMA: 602*4kB 18*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2552kB
Oct  5 23:31:32 swedishchef kernel: 45486 total pagecache pages
Oct  5 23:31:32 swedishchef kernel: 0 pages in swap cache
Oct  5 23:31:32 swedishchef kernel: Swap cache stats: add 0, delete 0, find 0/0
Oct  5 23:31:32 swedishchef kernel: Free swap  = 1156672kB
Oct  5 23:31:32 swedishchef kernel: Total swap = 1156672kB
Oct  5 23:31:32 swedishchef kernel: 65536 pages RAM
Oct  5 23:31:32 swedishchef kernel: 1583 pages reserved
Oct  5 23:31:32 swedishchef kernel: 63622 pages shared
Oct  5 23:31:32 swedishchef kernel: 17576 pages non-shared
Oct  5 23:31:32 swedishchef kernel: skbuff alloc of size 3872 failed
Oct  5 23:31:32 swedishchef kernel: flush-253:0: page allocation failure. order:1, mode:0x20
Oct  5 23:31:32 swedishchef kernel: Call Trace:
Oct  5 23:31:32 swedishchef kernel: [ceea98a0] [c0008334] 0xc0008334 (unreliable)
Oct  5 23:31:32 swedishchef kernel: [ceea98d0] [c0063568] 0xc0063568
Oct  5 23:31:32 swedishchef kernel: [ceea9960] [c00844c0] 0xc00844c0
Oct  5 23:31:32 swedishchef kernel: [ceea99c0] [c0084830] 0xc0084830
Oct  5 23:31:32 swedishchef kernel: [ceea99e0] [c0279474] 0xc0279474
Oct  5 23:31:32 swedishchef kernel: [ceea9a00] [d107b034] 0xd107b034
Oct  5 23:31:32 swedishchef kernel: [ceea9a20] [d125185c] 0xd125185c
Oct  5 23:31:32 swedishchef kernel: [ceea9a80] [d12509ec] 0xd12509ec
Oct  5 23:31:32 swedishchef kernel: [ceea9aa0] [c0027ab0] 0xc0027ab0
Oct  5 23:31:32 swedishchef kernel: [ceea9ac0] [c0028628] 0xc0028628
Oct  5 23:31:32 swedishchef kernel: [ceea9b00] [c0006230] 0xc0006230
Oct  5 23:31:32 swedishchef kernel: [ceea9b10] [c0028414] 0xc0028414
Oct  5 23:31:32 swedishchef kernel: [ceea9b20] [c00062d0] 0xc00062d0
Oct  5 23:31:32 swedishchef kernel: [ceea9b40] [c0011c3c] 0xc0011c3c
Oct  5 23:31:32 swedishchef kernel: --- Exception: 501 at 0xc02578f0
Oct  5 23:31:32 swedishchef kernel:     LR = 0xc0257834
Oct  5 23:31:32 swedishchef kernel: [ceea9c20] [c016c6f4] 0xc016c6f4
Oct  5 23:31:32 swedishchef kernel: [ceea9cc0] [c016c928] 0xc016c928
Oct  5 23:31:32 swedishchef kernel: [ceea9d00] [c00ad5dc] 0xc00ad5dc
Oct  5 23:31:32 swedishchef kernel: [ceea9d20] [c00b0bbc] 0xc00b0bbc
Oct  5 23:31:32 swedishchef kernel: [ceea9d70] [c00e4fd4] 0xc00e4fd4
Oct  5 23:31:32 swedishchef kernel: [ceea9d90] [c006384c] 0xc006384c
Oct  5 23:31:32 swedishchef kernel: [ceea9da0] [c0064268] 0xc0064268
Oct  5 23:31:32 swedishchef kernel: [ceea9e50] [c00a70c4] 0xc00a70c4
Oct  5 23:31:32 swedishchef kernel: [ceea9e80] [c00a7e34] 0xc00a7e34
Oct  5 23:31:32 swedishchef kernel: [ceea9ed0] [c00a80d0] 0xc00a80d0
Oct  5 23:31:32 swedishchef kernel: [ceea9f40] [c00a83d4] 0xc00a83d4
Oct  5 23:31:32 swedishchef kernel: [ceea9f80] [c00a84d4] 0xc00a84d4
Oct  5 23:31:32 swedishchef kernel: [ceea9fa0] [c0070510] 0xc0070510
Oct  5 23:31:32 swedishchef kernel: [ceea9fc0] [c0039274] 0xc0039274
Oct  5 23:31:32 swedishchef kernel: [ceea9ff0] [c00113c8] 0xc00113c8
Oct  5 23:31:32 swedishchef kernel: Mem-Info:
Oct  5 23:31:32 swedishchef kernel: DMA per-cpu:
Oct  5 23:31:32 swedishchef kernel: CPU    0: hi:   90, btch:  15 usd:  34
Oct  5 23:31:32 swedishchef kernel: active_anon:1926 inactive_anon:8571 isolated_anon:0
Oct  5 23:31:32 swedishchef kernel:  active_file:10122 inactive_file:35247 isolated_file:0
Oct  5 23:31:32 swedishchef kernel:  unevictable:0 dirty:6220 writeback:14 unstable:0
Oct  5 23:31:32 swedishchef kernel:  free:638 slab_reclaimable:3305 slab_unreclaimable:2567
Oct  5 23:31:32 swedishchef kernel:  mapped:6601 shmem:111 pagetables:439 bounce:0
Oct  5 23:31:32 swedishchef kernel: DMA free:2552kB min:2036kB low:2544kB high:3052kB active_anon:7704kB inactive_anon:34284kB active_file:40488kB inactive_file:140988kB unev
ictable:0kB isolated(anon):0kB isolated(file):0kB present:260096kB mlocked:0kB dirty:24880kB writeback:56kB mapped:26404kB shmem:444kB slab_reclaimable:13220kB slab_unreclaim
able:10268kB kernel_stack:1040kB pagetables:1756kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct  5 23:31:32 swedishchef kernel: lowmem_reserve[]: 0 0 0
 ... (repeats) ... 

Ps. I know it's cheesy to name my bubba after this guy: http://www.youtube.com/watch?v=B7UmUX68KtE ... Please forgive me!

[tor]

Hi g__,

Could you try this as root:

Code: Select all

echo 4096 > /proc/sys/vm/min_free_kbytes

And see if this could help your problem.

Note that above setting does not survive a reboot.

/Tor

[g__]

Yup, that seems to help. I'll watch it and see if it comes back. Why do you think it helped? (Was it set too low for this kind of system?)

Cheers,

Geoff

[tor]

Hi g__,

The problem isn't that you run out of memory even if it seems like it. What is happening is that you run out of preallocated memory for e.g. DMA transfers. The adjusted setting tells the OS to allocate more memory for these operations.

If this solves your problem, please let me know, we should consider raising this on all units. I know that others have had similar problems.

if you want to persist this setting you have to edit the file /etc/sysctl.conf and add a line like this:

Code: Select all

vm.min_free_kbytes = 4096

/Tor

[g__]

Done, thank you!

[aphex]

4096 wasn't enough for me, still got page allocation failure etc (dropped USB devices, smb crashes when streaming big files, compiler crashes etc).

Bumping it to 8192 made it better.

[Ubi]

If other people have this same issue, it's maybe a good idea to make this setting default for a new bubba update release...?

[DanielM]

If other people have this same issue, it's maybe a good idea to make this setting default for a new bubba update release...?

Agreed. This setting stopped my B3 from occasionally locking up completely.

/Daniel

[loki_val]

My b3 would lock up when I was overloading it with compilation and fiddling with the web interface at the same time. Setting this to 8192 seemed to do the trick for me. Though it was still slow, it worked.

[jonj1]

I got a similar message with a bubba2. Will try the 8192 setting and see how it goes...

Code: Select all

Feb 21 20:41:30 bubba kernel: mount: page allocation failure. order:5, mode:0xd0
Feb 21 20:41:30 bubba kernel: Call Trace:
Feb 21 20:41:30 bubba kernel: [cdbcfc40] [c0008334] 0xc0008334 (unreliable)
Feb 21 20:41:30 bubba kernel: [cdbcfc70] [c0063568] 0xc0063568
Feb 21 20:41:30 bubba kernel: [cdbcfd00] [c00844c0] 0xc00844c0
Feb 21 20:41:30 bubba kernel: [cdbcfd60] [c0084830] 0xc0084830
Feb 21 20:41:30 bubba kernel: [cdbcfd80] [d1476584] 0xd1476584
Feb 21 20:41:30 bubba kernel: [cdbcfe20] [c008ab8c] 0xc008ab8c
Feb 21 20:41:30 bubba kernel: [cdbcfe70] [d1472ddc] 0xd1472ddc
Feb 21 20:41:30 bubba kernel: [cdbcfe80] [c008986c] 0xc008986c
Feb 21 20:41:30 bubba kernel: [cdbcfea0] [c008996c] 0xc008996c
Feb 21 20:41:30 bubba kernel: [cdbcfec0] [c00a29e0] 0xc00a29e0
Feb 21 20:41:30 bubba kernel: [cdbcff10] [c00a3084] 0xc00a3084
Feb 21 20:41:30 bubba kernel: [cdbcff40] [c00115a4] 0xc00115a4
Feb 21 20:41:30 bubba kernel: --- Exception: c01 at 0xfe64e64
Feb 21 20:41:30 bubba kernel:    LR = 0x10003408
Feb 21 20:41:30 bubba kernel: Mem-Info:
Feb 21 20:41:30 bubba kernel: DMA per-cpu:
Feb 21 20:41:30 bubba kernel: CPU    0: hi:   90, btch:  15 usd:   0
Feb 21 20:41:30 bubba kernel: active_anon:706 inactive_anon:2455 isolated_anon:1
Feb 21 20:41:30 bubba kernel: active_file:13096 inactive_file:40614 isolated_file:50
Feb 21 20:41:30 bubba kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Feb 21 20:41:30 bubba kernel: free:1917 slab_reclaimable:2823 slab_unreclaimable:1578
Feb 21 20:41:30 bubba kernel: mapped:1801 shmem:119 pagetables:171 bounce:0
Feb 21 20:41:30 bubba kernel: DMA free:7668kB min:2036kB low:2544kB high:3052kB active_anon:2824kB inactive_anon:9820kB active_file:52384kB inactive_file:162456kB unevictable:0kB isolated(anon):4kB isolated(file):200kB present:260096kB mlocked:0kB dirty:0kB writeback:0kB mapped:7204kB shmem:476kB slab_reclaimable:11292kB slab_unreclaimable:6312kB kernel_stack:640kB pagetables:684kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 21 20:41:30 bubba kernel: lowmem_reserve[]: 0 0 0
Feb 21 20:41:30 bubba kernel: DMA: 715*4kB 343*8kB 77*16kB 20*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7668kB
Feb 21 20:41:30 bubba kernel: 53861 total pagecache pages
Feb 21 20:41:30 bubba kernel: 17 pages in swap cache
Feb 21 20:41:30 bubba kernel: Swap cache stats: add 18, delete 1, find 0/0
Feb 21 20:41:30 bubba kernel: Free swap  = 262064kB
Feb 21 20:41:30 bubba kernel: Total swap = 262136kB
Feb 21 20:41:30 bubba kernel: 65536 pages RAM
Feb 21 20:41:30 bubba kernel: 1583 pages reserved
Feb 21 20:41:30 bubba kernel: 19732 pages shared
Feb 21 20:41:30 bubba kernel: 45647 pages non-shared
Feb 21 20:41:30 bubba kernel: EXT4-fs (dm-4): mounted filesystem with ordered data mode

[jonj1]

I got a similar message with a bubba2. Will try the 8192 setting and see how it goes...

8192 fixed it...