Need help with complicated routing & ipsec setup

Networking
1

Hello everyone,

I am currently trying to figure out how to implement Routing with multiple IPSec Gateways.
I am struggling with the implementation, as I already tried a few things with limited success.

First of all, here is an image on how things should look like

IPFire_SOLL
IPFire_SOLL1748×1444 164 KB

Now, here are my questions:
First of all, is a constellation like the picture even possible? Can I add multiple distinctive IPSec routers behind a single IPFire, while IPFire itself is also handling IPSec tunnels? We need to do this, because some of our customers want to use their preconfigured hardware to establish IPSec tunnels.
I am thinking about a CUSTOMPREROUTING and CUSTOMPOSTROUTING rule in firewall.local to redirect traffic from Customer2 and Customer3 to the corresponding IPSec router.
I tried this already in a test case, but it didn’t work.
I tested with:
Firewall rule RED -> Red (firewall) allow all (just to test)
Network RED -> redirect all to ipsec router (just to test)

Image (new users can only post one image…): https://i.imgur.com/cIOW0ae.png

That didn’t work, the router couldn’t establish an ipsec link with a timeout error.
Then I tried adding custom IPTables rules:
iptables -t nat -A CUSTOMPREROUTING -i red0 -s Customer -j DNAT --to IPSec Router
iptables -t nat -A CUSTOMPOSTROUTING -o red0 -d Customer -j SNAT --to IPSec Router
That didn’t work either, the rules didn’t even get a hit when I looked in the iptables tab in the webUI.

So my question is “how can I redirect IPSec traffic from a specific IP addres / customer while IPFire is also handling IPSec itself?”

My second question is, how does IPSec with a virtual network work in IPfire?
We have one customer, cutomer1 in the first image, who uses a “virtual transfer network” for the IPSec tunnel. IPSec config looks something like this:
Image: https://i.imgur.com/H2zIJsY.png

Note that Local subnet is NOT our 10.0.0.0/24 office network.
“Local subnet” MUST be the “virtual network”, or else the link doesn’t establish. The target IPSec device is a Cisco device.
The link does come up, but IPFire can’t route packets through it, traceroute tells me either “Target network not reachable” or “send not allowed”.
I also tried several things with custom IPTables rules to SNAT the packets to the right IPs, but to no avail.

So here is my second question:
How can I route traffic through an IPSec tunnel if the customer demands a different local subnet in the IPSec tunnel than we physically have in our network?

Thanks for reading, and thanks for potentially helping me out!

Kind regards

2

I think the short answer is yes. But you will have some issues here…

Not really. You won’t be able to have the remote site initiate the connection to your routers in your orange network. You could get away with that by assigning each router a different public IP address, but that would entail that you have a small subnet with public IP address space available.

You should use the IPFire box to handle all IPsec VPNs in my opinion.

I have no idea why you needed to setup firewall rules on the console. The web UI can do that for you.

You probably didn’t setup any static routes.

This is not how networks or routing works.

Unless your customer is AT&T, you are using public address space here - for starters.

What is a “virtual” network? Aren’t they all quite virtual?

You have a very complicated setup here and since you are using this in some business, I can only ask you learn how basic routing works or have an expert have a look at it.

3

Thanks for your reply @ms

Sadly that’s not possible. The customers demand that we use their hardware. We can’t even configure the hardware, the VPN Routers come completely preconfigured. We give them an IP and a local network, they send us their hardware, we plug it in, and it works.
We’d also like to use IPFire for their IPSec, but they won’t allow that.

Yes that is a “plan B”, but our current internet plan only provides one public IP. We are located in germany, so no AT&T for us…
We are already in the talks to get multiple public IPs to our office, but we don’t know the details yet, so I am trying to see if we can get the VPN running using only one public IP, by separating the incoming traffic based on where it comes from.
I was trying to look if it is possible, for example say
“I receive a packet from IP [CustomerXY IPSec public Gateway], so I redirect this packet to [CustomerXY Router in DMZ]”

Basically what I was trying to explain is that the customer IPSec tunnel expects a different IP than we have configured for our IPFire router. So we’d need to NAT between GREEN and the target customer network.
We use private IPs in the subnet 10.0.0.0/24 for our office.
However, the IPSec Router of the customer expects IPfire to behave as if the packets are coming from an IP within 172.0.0.0/29, lets say 172.0.0.1. Our current IPSec router, a Lancom device, allows us to configure that within the IPSec settings. In IPFire, in the IPSec settings, we need to set “local subnet” to 172.0.0.0/29, but then it seems as if a route from 10.0.0.0/24 (GREEN) ↔ 172.0.0.0/29 is missing.
Now, this “virtual” 172.0.0.0/29 isn’t even the target customer network. that would be even another network, for example 160.0.0.0/24.
The traffic would look like this:
FROM (Green) NAT TO 172.0.0.1 ← IPSec Tunnel → (customer network) ← target machine 160.0.0.x

I am currently reading into NETMAP, if that’s the correct solution for our problem.

Here is a (german) thread about a similar problem, they also couldn’t get it to work: IPFire Community

4

Fiddly, and this will probably create some new problems for you.

Are you guaranteed that they all have static IP addresses on their side?

Why would you not let the individual routers handle this?

That is public IP address space belonging to AT&T.

IPFire currently does not support mapping a whole subnet to another one. Creating firewall rules will become complex and very error-prone. Hence I would not recommend using this.

The NETMAP module is there, but you will have problems with the auto-generated firewall rules for the VPNs and will need some more rules to work around the firewall engine.

As mentioned before, you are not really solving your problem. You are just adding more complexity to it, which I would absolutely not recommend.

5

Yes, we are garuanteed that the customer IPSec gateways all have static IPs.

IPFire is the Router for that IPSec Tunnel. We don’t have a router that sits in the 172 subnet.
Would it work if we add another interface to the IPFire machine with network set to the 172 subnet?
That way the IPFire machine has a network for the one the customer is expecting on the tunnel, and we can NAT between Green and the new network…?

Okay, so this explains why I was searching for options that don’t exist.

Thanks for your input. But that’s the way it currently works in production on our Lancom router. In there we can define rules to map our 10.0.0.0/24 to a specific network, just for the IPSec tunnel
And we can also tell the router to behave as a device in that “virtual network”
image

I know I am throwing screenshots around from a different piece of hardware, but we have to let go of it and it’s IPFire that we settled on to replace it. Well, seems like I need to find a different way to accomplish this IPSec setup

6

Event 160.x.y.z is public network, not private…

7

As I said, IPFire does not support re-mapping the address space of an entire network.

As also said, you can create this with IPFire. But it is a bad idea. You will get yourself into more trouble than you are in already.

Those problems will be the same than if you would replace IPFire with something else. You have a bad network design there and you should build it again from scratch to ensure that you can filter traffic properly.

8

It’s not like we are content with how the network is structured.
Those are the requirements of our customers. We sadly have no say in either the topology nor address spaces.
We require IPSec tunnels to those customers, and we either take whatever configuration and hardware they give us or we simply won’t get a tunnel.

Do you have a suggestion on how things should be restructured?
We have to use those customer IPSec devices in those weird address ranges.
Because of internal affairs I also got the task to see if the network setup pictured above (Multiple IPSec hardware devices over one public IP) is possible.

We know that we are “blind” to certain parts of the internet because our customers chose public IPs for their internernal “private” IPs, but we have to live with that.

Either way, I made some progress with the “Customer1” IPSec tunnel within IPFire. I got things to work by using a second IPFire instance. that topology now looks something like this:

Internet ↔ IPFireMain (10.0.0.1) ↔ IPFireCustomer1 (172.0.0.1) < (IPSec over Internet) > Customer network
Link is up, I can connect to the customer network when I am directly connected to IPFireCustomer1.

Now I need to figure out how to route things, because those two IPFires don’t live in the same subnet. Maybe I’ll source another network card for those two machines to talk with one another.
And it seems like we need to utilize multiple external, public IPs in order for the other Customer IPSec routers to work.

A new idea I have is to utilize our hosting provider and maybe ask them if we could send them the Customer IPSec hardware. We’d need a device to create an IPSec tunnel from our office to the devices in the rack, but that way we centralized, multiple IPs to use for the customers.
That could indeed work, if our provider allows unknown routing hardware in their server racks…

9

I understand your problem. You have now just accepted all the problems to deal with and it is up to you to do that.

I think that you will need to plan a flat network design and allocate a different subnet to each customer. That is something you will need to do anyways because sooner or later you will have collisions, no matter how much you NAT address space.

Regarding the public IP address space: You are creating blind spots for yourself on the internet. They are very large subnets that you won’t reach any more. Just wait until the next cloud provider is using IP address space from there.

Your customers should either use actual public address space allocated to them, or they should use IP address space from RFC 1918.

I think you should. As stated above, you can NAT the shit out of it.

If I was running a DC I would oppose that. And if I were you I would not want that either - depending on how painful a trip to the DC is. As long as you have the bandwidth, you can of course be a hop in the middle, creating an extra SPOF.

10

I know that this question might be a little… “over the top” as question, @markusb: is the customer threatening you? Do you have any kind of danger/liability/fine above your head if you don’t comply to succeed this setup?
Sometime be a part of an abuse configure some connivance because “you know you’re doing something wrong and you do it anyway”. You can tell me "it’s only an internal network issue, and i can agree, but if in the future a certification blows up because of this incorrect network setup maybe you can be considered culprit because you did it anyway.

In any case… this not a complicated setup. This is an out of the RFC setup.

11

This is not complicated. This is dangerous.

12

No, it’s not that they are “threatening” us.
It works as follows: Our customers are really big, multinational corporations. We sell them our software. We need an IPSec tunnel to do support for our software on their networks.
They give us their IPSec solution. If their config is really bad we’d have to live with it because it’s either we take their solution and work with it, or we simply cannot do support for our own software in their networks.
They are so big that they literally don’t care that their IPSec solutions are bad, or dangerous

We did some further prodding with two IPFires, one dedicated to Customer1, but we can’t get the second IPFire to forward traffic from RED to its IPSec tunnel.
We will do some further testing in an isolated test network in-house, this solution seems to be the most promising one for customer1.
As for customer2 and customer3 and their dedicated IPSec hardware behind a DMZ, that needs some further research and prototyping. Maybe I’ll ask their support if they can configure the hardware to initiate the IPSec connection (we cannot configure the customer IPSec hardware. We don’t even have logins for them).

13

I didn’t mean that it is dangerous for them. This is dangerous for you.

And yes, I am aware of how this works with certain corporations.

14

Hi Markus

I have read through from the top to bottom and you started off by asking how to get your existing stuff to work correctly.
Yet your last post, as quoted, actually explains what the scenario is.

Short recap, you need to provide support for your software to clients that provide you with an IPSEC certificate.

The client would supply a road-warrior certificate, why does it look like you are trying to use it as N2N one instead?
The more important question is why did your sales staff sell the client on the idea that you can provide support via IPSec when you clearly have not tested or had this working in a testbed scenario?
Be that as it may, you may want to rethink your remote support options, as you cannot expect the community to resolve this business model for you.

Many roads lead to hours of frustration when you don’t look at a problem holistically, I have extensive experience in that department.
Your options to provide said support are as follows:

  1. Teamviewer, or similar alternatives, to the server with your software on.
  2. Provide a dedicated server or desktop machine to client that you have remote access to, similar option to 1.
  3. Provide a reconfigured router device (i.e. EdgeRouter Lite, or Mikrotik) you have VPN access to, that the client plugs into their network.
  4. Provide the client with an IPFire preconfigured as you need to for VPN access, similar option to 2.
  5. Try and integrate said supplied IPSEC certificate into a solution you already have working.

Obviously all approaches have pros and cons. Choose the one you will have most success with in the least amount of time. Either way you will have to go back to the client.

Good luck, live long and prosper

15

Thats not completely correct. Please excuse me if our intentions didn’t come through correctly, english is not my native language.
Take a look at my first screenshots, that’s how things should be looking in terms of network layout.
We have two physical IPSec routers for Customer2 & 3, and we have one IPSec Route which we have to configure within IPFire itself for Customer1, with your said certificate.
We already got the IPSec route to Customer1 working within IPFire, but we need to use a second IPFire instance because Customer1 requires our network to be in a specific network range.

We are trying to N2N because everyone in the building should have access to the customer, multiple people at once, and the type of IPSec that Customer1 provides us is a N2N type, not a Road Warrior type.

It has been working in production for many years, but we are currently undergoing several changes in terms of infrastructure and other factors, which promt us to redo our networking. We will have to let go of our current router which hanles the IPSec route to Customer1, and we want to replace that device with IPFire.

We do that with other customers. But the machines of these three customers who want IPSec routes are isolated from the internet. Creating an IPSec route is the only option for us to access those machines.
We wouldn’t want to use IPSec if absolutely nescessary. We already use a mix of your options 1 & 2 with the vast majority of our other customers.

Option 3 and 4 is out of the question. We get their hardware. They won’t take our hardware.

We are currently testing with two IPFires as stated before, will update if we find anything usefull

16

I see… in that case here is an alternative.

How about setting up a VM server (qemu, xen, kvm, vmware… list is long) in the DMZ with individual ipfire sessions per customer? At least you would only have to deal with the configurations and routes for one network at a time, and not multiple clients at the same time. Makes logistics, resource allocation, security (most important) and what not considerably easier… less grey hairs and sleepless nights.

Each IPsec session has it’s own port and you add NAT routes for each on your main ipFire which is acting as your networks bouncer and facial control system, for example.

red0 is your main FW WAN port
Customer 1 -> VM ipFire1 IPsec running on UDP/500
Customer 2 -> VM ipFire2 IPsec running on UDP/505
Their internal IP range on each VM ipFire can be the same or whatever you like, it becomes rather irrelevant.

If you cannot figure out the WebGUI firewall section, then you can do…

/sbin/iptables -t nat -A CUSTOMPREROUTING ! -i red0 -p udp --destination-port 500 -j DNAT --to VM-ipfire1-IP:500
/sbin/iptables -A CUSTOMFORWARD -p tcp -d VM-ipfire1-IP --destination-port 500 -j ACCEPT

and

/sbin/iptables -t nat -A CUSTOMPREROUTING ! -i red0 -p udp --destination-port 505 -j DNAT --to VM-ipfire2-IP:505
/sbin/iptables -A CUSTOMFORWARD -p tcp -d VM-ipfire2-IP --destination-port 505 -j ACCEPT

by editing /etc/sysconfig/firewall.local

You would also need a corresponding stop rule

/sbin/iptables -t nat -F CUSTOMPREROUTING
/sbin/iptables -F CUSTOMFORWARD

Not 100% sure if I did that right, best check against known NAT rules that work…. but you get the idea. And the corresponding rules on the actual VM firewalls to accept traffic from the main firewall etc.

Each client would need to use a different UDP or TCP port to run their IPsec session on, or else this is a useless exercise and you are back to the drawing board.
Now your support staff can gain access to the VM firewall via which ever way you like, just consider your internal network security can be compromised depending on how you proceed forward.

Good luck, cheers.

PS using public IPs inside your network is a no go, leaving the ethical part if owned by someone else aside, it’s a nightmare for routing and configuration. There is a reason that we have 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16 as private range and they are non-propagating. You would think that nearly 18million IPs to choose from would be ample. You may still end up with problems regarding that part… but it’s for your client to resolve, their network, their baby.

17

You won’t be able to DNAT IPSec to another port than 500.

And I have outlines above how this could work with only one public IP address. The problems lie elsewhere.

And this can all be doing using the UI. Therefore you should not add any custom rules. The UI usually adds more rules, too.

And your approach will only work for incoming connections…

I would disagree. It gets complicated when you only have 8 IP addresses, but for larger installations you do not want to NAT everything at the firewall and send every single packet through it.

18

Bummer…You are right, forgot IPsec is a difficult beast unlike OpenVPN which is more forgiving.
UDP/500 and UDP/4500 is it.

I take it adding in a few forwarding rules in the FW GUI for example
Src IP1 port udp/500 <-> Dst IP1 port udp/500
and a
Src IP2 port udp/500 <-> Dst IP2 port udp/500
Src IPs being the different client originating IPs, and Dst IPs being the different VM FW machines,
would not fully work either then?

Re: Public IPs
Having multiple live IPs on a WAN side, linked however you like yes, but not for use internal on a LAN, which is not directly accessible from the internet.
How are router, firewall and the like supposed to know if the IP they need to chat to is on the internet or LAN? It would be a waste of resources and time to have your IT staff constantly check the IP range used (and not owned), has not all of a sudden gone live, and the services everyone is trying to connect to is not actually on Tom’s laptop in the accounts department, but rather on a server sitting in a DC in Iceland.
Unless we are all talking about two different things here… :grin:

@Markus, stick with Michael… he has the right ideas, I’m clearly on the slow side and still catching up with what he said hahaha.

19

No it isn’t. It just works completely different to other VPNs, because it is part of IP. OpenVPN is sitting on top of it and therefore you can apply the usual rules of forwarding layer 3 protocols like UDP and TCP.

IPsec uses more than that. So I am afraid to say, that your solution is still inadequate. I do not even know where to begin…

I am stressing it, for one last time, that Markus solution simply won’t work well. He has his limitations, that is fine, they exist in the real world, but he isn’t able to move anything. So he will have to deal with the downsides of his setup. It looks like data breaches or a simply unstable setup seem to be acceptable and he seems to be happy with it.

20

Okay, so we settled for a solution for now.

Customer1 we got to work completely within a separate IPFire, which has the 172.xxx network range as its green interface. But we are still working out the routing, because somehow we can’t route from the Office IPFire to the Customer1 IPFire.
As a temporary solution to this we will put a few virtual machines behind the green network of the Customer1 IPFire.

As for the hardware IPSec Routers of customers 2&3, we called their IT departsment, the hardware can indeed be put behind a single public IP address because their routers are IPSec tunnel initiators, if I recall correctly.
So those two are out of the equation and can be put in our network, our Office IPFire just needs some static routes for the customer2&3 networks.

As I had them on the phone I asked them if they could provide different IP ranges for the IPSec tunnel. Short answer: No.
Long answer: They don’t care. “Something something this is how it has been running for years, something something never had any problems, too much work to reconfigure their entire network”.

We will implement this in the coming weeks.