How to properly configure multiple static IPv4 WAN addresses and IPv6 prefix to internal DHCP and static hosts
Event Timeline
So, I ended up handling my IPv4 addresses using 1:1 NAT. It works, and I don't love it, but I think it's the best it's going to get with Comcast's clunky static IP infrastructure. But I'm having no luck with IPv6, and could really use some help with someone who understand's static IPv6 and VyOS a little better. I have a static IPv6 prefix, and I need to statically assign some of those to public-facing servers behind my firewall/router, but it's like pulling teeth from a rhinoceros.
For better illustration, the static IPv6 prefix I've been given is 2603:xxxx:xxxx:8700::/56 and the gateway address is 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 (you'll notice that gateway address is within that /56; in fact, it's within the first /64).
I can ping 2607:f8b0:4002:c07::66 (Google) from my VyOS firewall/router if I do the following:
# set interfaces ethernet eth1 2603:xxxx:xxxx:8700::1/60 # set protocols static route6 ::/0 next-hop 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 # commit
Then:
$ ip -6 route get 2607:f8b0:4002:c07::66 2607:f8b0:4002:c07::66 from :: via 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 dev eth1 src 2603:xxxx:xxxx:8700::1 metric 0 cache $ ping6 -c 1 2607:f8b0:4002:c07::66 PING 2607:f8b0:4002:c07::66(2607:f8b0:4002:c07::66) 56 data bytes 64 bytes from 2607:f8b0:4002:c07::66: icmp_seq=1 ttl=46 time=30.6 ms
I can also ping 2603:xxxx:xxxx:8700::1 from a remote server I have access to with known working IPv6. This works with many different IPv6 addresses on eth1:
- 2603:xxxx:xxxx:8700::2/60
- 2603:xxxx:xxxx:8700:1::1/60
- 2603:xxxx:xxxx:8700:2::2/60
All of those work ... anything in the :8700/64 works fine. Note that :8700:anything/60 should provide addresses :8700 through :870f. However, if I try to set the eth1 address to 2603:xxxx:xxxx:8701::1/60 (which is within the same /60 as 2603:xxxx:xxxx:8700::1/60), nothing works. ip -6 route and ping6 both say unreachable to all IPv6 addresses, and I can't ping the address from the outside server. I tried using /56 instead of /60 with no change (in fact, that's what I originally tried, but when it didn't work I tried /60 and got the same results). I tried setting the static default route to the link-local address (fe80::7454:7dff:feb1:d391) of the gateway instead of the global address, but no change.
And here's where it gets weirder. If I add BOTH addresses to eth1:
# set interfaces ethernet eth1 2603:xxxx:xxxx:8700::1/60 # set interfaces ethernet eth1 2603:xxxx:xxxx:8701::1/60 # commit
Now ip -6 route shows a route out through 2603:xxxx:xxxx:8701::1 via 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391 and ping6 no longer says unreachable, but I just never get a response to any pings. A little tracing shows that's because nobody can route back. I can ping 2603:xxxx:xxxx:8700::1 but not 2603:xxxx:xxxx:8701::1 from the remote server, and since my pings to Google are coming from 2603:xxxx:xxxx:8701::1, the recipient tries to respond on 2603:xxxx:xxxx:8701::1. If I delete both IPv6 addresses, commit, and then add them in the reverse order:
# set interfaces ethernet eth1 2603:xxxx:xxxx:8701::1/60 # set interfaces ethernet eth1 2603:xxxx:xxxx:8700::1/60 # commit
ip -6 route now shows a route out through 2603:xxxx:xxxx:8700::1 via 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391, and ping6, since it's now sending out through 2603:xxxx:xxxx:8700::1, gets responses. Still can't ping 2603:xxxx:xxxx:8701::1 from the remote server, though.
Going back to the working config for a minute (2603:xxxx:xxxx:8700::1/60 plus the static route), this only helps me reach the outside world from the VyOS firewall/router. I still need to assign static IP addresses to my servers (in fact, for one server, I need to assign multiple static IPv6 addresses to a single interface).
The interface facing my servers is bond0 vif 900. I set its address to 2603:xxxx:xxxx:8700:92::1/64, and then I assign an address 2603:xxxx:xxxx:8700:92::173/64 to a server's interface with a default route of 2603:xxxx:xxxx:8700:92::1. From the server, I can ping 2603:xxxx:xxxx:8700:92::1, but I can't ping 2603:xxxx:xxxx:8700::1 (eth1 of the router), and I can't ping Google (2607:f8b0:4002:c07::66). The server says it has a route to 2607:f8b0:4002:c07::66 via 2603:xxxx:xxxx:8700:92::1, and using monitor interface I can see that the ICPM packets are making it to eth1, but nothing is ever coming back (even from VyOS itself when pinging eth1).
And I haven't even gotten started on stateless configuration on the private LAN/WLAN yet. Couldn't get it to work, either, but I decided at this point to stick to getting something simpler (static) working an then tackle that.
Really at a loss here. I thought static IPv6 would be much simpler than this, but I'm stuck.
Maybe this is relevant? https://phabricator.vyos.net/T421
Alas, no. Maybe in the sense that, worst-case scenario, I could probable use NPTv6 (NAT for IPv6), but that's not the idea here. That post is about Prefix Delegation / address auto-configuration. I'll need autoconf eventually, but all I'm trying to accomplish right now is static IPv6 configuration.
Perhaps you could make a drawing of what you try to get working? With proper interface naming etc. eth0 - wan, eth1 - dmz, eth2 - lan or whatever you are using. It makes it easier to understand what you try to do. And for the interfaces why do you want to use the /60?
I am willing to give some advice but it's an issue to understand your infrastructure based on a very fuzzy set of details.
The basic rule of thumb that I can think of is that you cannot assign ip addresses with the same or overlapping prefix on two interfaces and route between them.
I do not know if the VyOS kernel supports IPV6 NAT feature but this should be a very last resort for specific scenarios.
If you need some examples on how IPv6 prefixes are being used you can try to peek at some IPv6 brokers such as Hurricane Electric.
They give you a very specific IPv6 address and prefix for the WAN side with a specific default route,
Then they give you a different prefix to assign the internal network which is behind the main gateway.
Is your setup different then what HE offers?
So the attempts with /56 and /60 were part of my hundreds of different combinations/attempts to get this to work. I have one /56 assigned to me (2603:xxxx:xxxx:8700::/56) with one gateway assigned to me (2603:xxxx:xxxx:8700:7454:7dff:feb1:d391). Skipping the WAN for just a second because I believe(d) it to need different configuration, I expected to be able to break that /56 up into /64s and use them like so:
- bond0 vif 100 (private LAN/WLAN) gets 2603:xxxx:xxxx:8752::1/64 and uses stateless autoconfig for all the hosts on that network
- bond0 vif 200 (guest LAN/WLAN) gets 2603:xxxx:xxxx:8754::1/64 and uses stateless autoconfig for all the hosts on that network
- bond0 vif 900 (DMZ) gets 2603:xxxx:xxxx:8792::1/64 and uses pure static addressing for all the hosts on that network
I've been using either the whole 2603:xxxx:xxxx:8700::1/56 or a smaller 2603:xxxx:xxxx:8700::1/60 on the WAN interface (eth1) because I thought (perhaps erroneously) that its prefix needed to be broader to encompass all of the prefixes behind it (on bond0 VIFs 100, 200, and 900). I might have been wrong about this, because I had slightly better luck with (ignoring 100 and 200 for the time being):
- eth1 gets 2603:xxxx:xxxx:8700::1/64 with a static route ::/0 set to 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391.
- bond0 vif 900 gets 2603:xxxx:xxxx:8792::1/64
- One of the hosts on bond0 vif 900 gets 2603:xxxx:xxxx:8792::170 with default gateway 2603:xxxx:xxxx:8792::1
With that setup, the public server (2603:xxxx:xxxx:8792::170) can ping its gateway bond0 vif 900 at 2603:xxxx:xxxx:8792::1 AND it can now finally ping the WAN address (2603:xxxx:xxxx:8700::1). This is an improvement. It has not been able to ping this before. However, it still can't ping the external gateway (2603:xxxx:xxxx:8700:7454:7dff:feb1:d391) or Google (2607:f8b0:4002:c07::66). Using monitor interfaces, I can see pings to those two addresses leaving eth1 to enter the broader internet, but no response packets ever return to that interface. I also cannot ping 2603:xxxx:xxxx:8792::1 or 2603:xxxx:xxxx:8792::170 from the outside. I've confirmed no firewall is in the way.
@aopdal, perhaps this diagram of my desired configuration will help:
OFF-SITE COMCAST EQUIPMENT
Global IP: Unknown
LL IP: fe80::201:5cff:fe6d:d246
|
|
|
ON-SITE COMCAST BUSINESS IP GATEWAY/MODEM
WAN Global IP: 2001:xxxx:xxxx:e:512d:8177:a1ea:1ce8
WAN LL IP: fe80::7654:7dff:feb1:d390
Default Gateway: fe80::201:5cff:fe6d:d246
Assigned Static Prefix: 2603:xxxx:xxxx:8700::/56
LAN Global IP (my gateway): 2603:xxxx:xxxx:8700:7454:7dff:feb1:d391
LAN LL IP: fe80::7454:7dff:feb1:d391
(All of these values are assigned/permanent and I cannot change)
|
|
|
MY VYOS MACHINE (PLUGGED INTO LAN OF BUSINESS IP GATEWAY)
|
|
eth1
2603:xxxx:xxxx:8700::1/64
|
bond0
No IPv6 address
|
|----------------------------+----------------------------|
| | |
vif 100 vif 200 vif 900
2603:xxxx:xxxx:8752::1/64 2603:xxxx:xxxx:8754::1/64 2603:xxxx:xxxx:8792::1/64
SLAAC SLAAC Pure static addressing@beamerblvd have you added routes for your vif 100,200 and 900 in your "COMCAST BUSINESS IP GATEWAY"?
@beamerblvd have you added routes for your vif 100,200 and 900 in your "COMCAST BUSINESS IP GATEWAY"?
There is no option for doing this in the Business IP Gateway. There is an Advanced > Static Routing screen, but the user manual for the equipment says that it's so that devices on private IPv4 networks behind the LAN/WLAN built in to the Business IP Gateway can talk to private IPv4 networks behind routers connected to the Business IP Gateway and vice-versa. If I try to enter anything IPv6 on this screen, I get the error:
Please enter an IPv4 address in the format #.#.#.#
Without routing you probably can't get it to work. Are your addresses managed from Comcast using prefix delegation?
Are your addresses managed from Comcast using prefix delegation?
Not that I'm told. The email I got, and the Business Internet account management screen online, just said "Static IPv6 Addresses: 2603:xxxx:xxxx:8700::/56."
Without routing you probably can't get it to work.
I sure hope that's not true. What possible use is a static /56 of IP addresses if you can't route anything other than the IP address(es) assigned directly to the WAN interface of your router?
With prefix delegation you have a static prefix on your inside, but the "wan" interface on the router is using DHCP.
There are no way to route IP if you cant configure routes in your router...
If you can't configure IPv6 routes in the router from Comcast - don't use it ;-)
With prefix delegation you have a static prefix on your inside, but the "wan" interface on the router is using DHCP.
Even if that were the case, VyOS doesn't support Prefix Delegation yet (in any version), so I can't use it.
If you can't configure IPv6 routes in the router from Comcast - don't use it ;-)
Not an option. You can only use a customer-owned DOCSIS modem / router / gateway with fully dynamic addressing. Comcast has a proprietary mechanism for committing static IP addresses that requires a secure private key hard-coded into the DOCSIS modem, so you _have_ to rent and use their Business IP Gateway with bridge mode disabled.