| Viacheslav | |
| Sep 9 2020, 5:17 PM |
| F839310: kernel_panic.png | |
| Sep 9 2020, 5:17 PM |
To reproduce this bug.
set policy route MSS-CLAMP rule 10 protocol tcp set policy route MSS-CLAMP rule 10 set tcp-mss pmtu set policy route MSS-CLAMP rule 10 tcp flags SYN set interfaces ethernet eth0 policy route MSS-CLAMP commit
Initiate any TCP session.
[email protected]:~$ sudo curl https://yahoo.com
Original post https://forum.vyos.io/t/kernel-panic-with-set-mss-pmtu/5850
More logs
[email protected]# sudo curl https://yahoo.com [ 2045.620295] BUG: kernel NULL pointer dereference, address: 0000000000000008 [ 2045.628755] #PF: supervisor read access in kernel mode [ 2045.630777] #PF: error_code(0x0000) - not-present page [ 2045.632483] PGD 0 P4D 0 [ 2045.633374] Oops: 0000 [#1] SMP PTI [ 2045.634465] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.5-amd64-vyos #1 [ 2045.635948] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1 04/01/2014 [ 2045.637941] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS] [ 2045.639334] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 89 f7 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 89 4c 24 08 48 83 e7 fe <48> 8b 47 08 48 8b 40 20 e8 13 d7 d5 f7 8b 74 24 10 4c 8b 4c 24 08 [ 2045.644694] RSP: 0018:ffffb762000038c8 EFLAGS: 00010246 [ 2045.645850] RAX: 00000000000005dc RBX: ffff8fd3c3756c62 RCX: 0000000000000002 [ 2045.647383] RDX: 0000000000000000 RSI: 00000000198f064a RDI: 0000000000000000 [ 2045.649438] RBP: ffffb76200003978 R08: ffffb762000038f0 R09: 0000000000000014 [ 2045.650995] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028 [ 2045.652575] R13: ffff8fd3d11aa500 R14: 0000000000000028 R15: ffffb76200003ba8 [ 2045.654101] FS: 0000000000000000(0000) GS:ffff8fd3d8c00000(0000) knlGS:0000000000000000 [ 2045.655683] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2045.657013] CR2: 0000000000000008 CR3: 000000000602e006 CR4: 0000000000160ef0 [ 2045.658538] Call Trace: [ 2045.659148] <IRQ> [ 2045.659622] tcpmss_tg4+0x2c/0xa0 [xt_TCPMSS] [ 2045.660541] nft_target_eval_xt+0x30/0x50 [nft_compat] [ 2045.661592] nft_do_chain+0x149/0x4c0 [nf_tables] [ 2045.662561] ? pollwake+0x6f/0x90 [ 2045.663377] ? wake_up_q+0xa0/0xa0 [ 2045.664257] ? sock_def_readable+0x32/0x60 [ 2045.665276] ? __udp_enqueue_schedule_skb+0x133/0x260 [ 2045.666320] ? udp_queue_rcv_one_skb+0x2be/0x460 [ 2045.667207] ? udp_unicast_rcv_skb.isra.66+0x6f/0x80 [ 2045.668455] ? __udp4_lib_rcv+0x553/0xb70 [ 2045.669392] nft_do_chain_ipv4+0x61/0x80 [nf_tables] [ 2045.670528] nf_hook_slow+0x3f/0xc0 [ 2045.671364] nf_hook_slow_list+0x89/0x130 [ 2045.673401] ip_sublist_rcv+0x1fb/0x210 [ 2045.674261] ? ip_rcv_finish_core.isra.22+0x400/0x400 [ 2045.675423] ip_list_rcv+0x132/0x156 [ 2045.676398] __netif_receive_skb_list_core+0x296/0x2c0 [ 2045.677722] netif_receive_skb_list_internal+0x1a1/0x2c0 [ 2045.679238] ? check_preempt_curr+0x75/0x90 [ 2045.680142] gro_normal_list.part.162+0x14/0x30 [ 2045.680957] napi_complete_done+0x62/0x170 [ 2045.681702] virtqueue_napi_complete+0x25/0x60 [virtio_net] [ 2045.682697] virtnet_poll+0x2e0/0x330 [virtio_net] [ 2045.683513] net_rx_action+0xf6/0x2e0 [ 2045.684220] __do_softirq+0xd2/0x227 [ 2045.685054] asm_call_on_stack+0x12/0x20 [ 2045.685985] </IRQ> [ 2045.686596] do_softirq_own_stack+0x34/0x40 [ 2045.687725] irq_exit_rcu+0x98/0xa0 [ 2045.688530] common_interrupt+0x73/0x140 [ 2045.689440] asm_common_interrupt+0x1e/0x40 [ 2045.690348] RIP: 0010:native_safe_halt+0xe/0x10 [ 2045.691283] Code: 48 8b 04 25 c0 7b 01 00 3e 80 48 02 20 48 8b 00 a8 08 75 c4 eb 80 cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d 96 b5 55 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 86 b5 55 00 f4 c3 cc cc e8 bb 6a a2 [ 2045.694894] RSP: 0018:ffffffffb8c03eb8 EFLAGS: 00000246 [ 2045.696154] RAX: ffffffffb82acce0 RBX: 0000000000000000 RCX: ffff8fd3d8c232c0 [ 2045.697897] RDX: 00000000000674ca RSI: 0000000000000087 RDI: 0000000000000000 [ 2045.699533] RBP: ffffffffb8c90dc0 R08: 0000032a500d3a2d R09: 0000000000000000 [ 2045.701165] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 2045.703893] R13: 0000000000000000 R14: ffffffffffffffff R15: ffffffffb8c134c0 [ 2045.705614] ? __sched_text_end+0x6/0x6 [ 2045.706625] default_idle+0x5/0x10 [ 2045.707539] do_idle+0x212/0x2d0 [ 2045.708374] cpu_startup_entry+0x14/0x20 [ 2045.709342] start_kernel+0x515/0x534 [ 2045.710369] secondary_startup_64+0xa4/0xb0 [ 2045.711445] Modules linked in: ip_set xt_TCPMSS xt_comment fuse nft_chain_nat xt_CT xt_tcpudp nft_compat nfnetlink_cthelper nft_counter nf_tables nfnetlink nf_nat_pptp nf_conntrack_pptp nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat_tftp nf_nat_ftp nf_nat nf_conntrack_tftp nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper iTCO_wdt pcspkr evdev iTCO_vendor_support button virtio_balloon virtio_console mpls_iptunnel mpls_router ip_tunnel mpls_gso br_netfilter bridge stp llc virtio_rng rng_core ip_tables x_tables autofs4 usb_storage ohci_hcd uhci_hcd ehci_hcd sd_mod t10_pi squashfs zstd_decompress loop overlay ext4 crc32c_generic crc16 mbcache jbd2 nls_ascii hid_generic usbhid hid sr_mod cdrom ahci libahci virtio_net net_failover failover virtio_blk libata xhci_pci crc32c_intel i2c_i801 i2c_smbus lpc_ich scsi_mod xhci_hcd virtio_pci virtio_ring virtio [ 2045.729452] CR2: 0000000000000008 [ 2045.730318] ---[ end trace 54d1da27fbab7803 ]--- [ 2045.731486] RIP: 0010:tcpmss_mangle_packet+0x3a0/0x440 [xt_TCPMSS] [ 2045.733004] Code: 48 8b 7c 24 20 89 44 24 10 e8 9c e9 89 f7 44 8b 54 24 04 4c 8b 4c 24 08 49 8b 7d 58 44 89 54 24 04 4c 89 4c 24 08 48 83 e7 fe <48> 8b 47 08 48 8b 40 20 e8 13 d7 d5 f7 8b 74 24 10 4c 8b 4c 24 08 [ 2045.738357] RSP: 0018:ffffb762000038c8 EFLAGS: 00010246 [ 2045.739661] RAX: 00000000000005dc RBX: ffff8fd3c3756c62 RCX: 0000000000000002 [ 2045.741324] RDX: 0000000000000000 RSI: 00000000198f064a RDI: 0000000000000000 [ 2045.743053] RBP: ffffb76200003978 R08: ffffb762000038f0 R09: 0000000000000014 [ 2045.744727] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000028 [ 2045.746401] R13: ffff8fd3d11aa500 R14: 0000000000000028 R15: ffffb76200003ba8 [ 2045.748186] FS: 0000000000000000(0000) GS:ffff8fd3d8c00000(0000) knlGS:0000000000000000 [ 2045.750434] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2045.751923] CR2: 0000000000000008 CR3: 000000000602e006 CR4: 0000000000160ef0 [ 2045.753728] Kernel panic - not syncing: Fatal exception in interrupt [ 2045.755228] Kernel Offset: 0x36c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 2045.757667] Rebooting in 60 seconds..
LTS version
[email protected]# commit [ interfaces ethernet eth0 policy route MSS-CLAMP ] iptables: Invalid argument. Run `dmesg' for more information. [[interfaces ethernet eth0]] failed Commit failed [edit] [email protected]# [email protected]# sudo dmesg | tail -n 5 [ 9.311242] Bridge firewalling registered [ 9.328112] fuse init (API version 7.27) [ 10.245771] Process accounting resumed [ 49.562499] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks [ 154.168320] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks [edit] [email protected]#
So it's some wrong logic after rewriting firewall?
@c-po , the same behavior even with kernel 5.8.8
[email protected]:~$ uname -a Linux R1 5.8.8-amd64-vyos #1 SMP Thu Sep 10 08:58:42 UTC 2020 x86_64 GNU/Linux
Any reason why we need to use MSS Clamp in mangle chain?
Still present in the latest rolling
VyOS 1.3-rolling-202009140541 Linux r1-roll 4.19.145-amd64-vyos #1 SMP Mon Sep 14 05:08:01 UTC 2020 x86_64 GNU/Linux
It seems a wrong logic.
We want that option to have an effect on "local" and "forward" directions, so we use table mangle and "PREROUTING" and VYATTA_FW_IN_HOOK hook
Generated rules
*mangle :VYATTA_FW_IN_HOOK -A PREROUTING -j VYATTA_FW_IN_HOOK -A VYATTA_FW_IN_HOOK -i eth0 -j MSS-CLAMP -A MSS-CLAMP -p tcp -m tcp --tcp-flags SYN SYN -m comment --comment MSS-CLAMP-10 -j TCPMSS --clamp-mss-to-pmtu -A MSS-CLAMP -m comment --comment "MSS-CLAMP-10000 default-action accept" -j RETURN
For "prerouting" VYATTA_FW_IN_HOOK we can't use the option -j TCPMSS --clamp-mss-to-pmtu, we can only handle -j TCPMSS --set-mss xxxx
And -j TCPMSS --clamp-mss-to-pmtu should be used only in FORWARD, OUTPUT and POSTROUTING hooks
ref. https://github.com/torvalds/linux/blob/master/net/netfilter/xt_TCPMSS.c#L263-L284
What we see, when we are trying to use that rule without any hook in mangle => prerouting
[email protected]:/home/vyos# sudo iptables -t mangle -A PREROUTING -i eth0 -p tcp -m tcp --tcp-flags SYN SYN -j TCPMSS --clamp-mss-to-pmtu iptables v1.8.2 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain PREROUTING [email protected]# sudo dmesg | tail -n 1 [48323.007403] xt_TCPMSS: path-MTU clamping only supported in FORWARD, OUTPUT and POSTROUTING hooks
It makes sense to use this option only for FORWARD or OUTPUT traffic. So I think we should get rid of that option, and define it in some else CLI with different logic.
set policy route MSS-CLAMP rule 10 set tcp-mss pmtu