Bulk Addition of Networks

daaaa · 5 December 2021 12:20

I’ve been using IPFire for a while, and find it a supurb system: both powerful and convenient.

But there’s one thing that’s really started to irritate me: is there a way in IPFire to do a bulk alias like in pfSense?

Having to manually type in every Zoom server network range was particularly tiresome in IPFire.

I now have a need to do it for AWS, and as such, am about to switch to pfSense, as setting up a new firewall would be faster than typing in all those networks one by one then clicking them all one by one to add them to a group.

Any ideas? Am I just missing something? Or is this not a capability in IPFire? That would seem odd to me…

anon42188109 · 5 December 2021 12:39

Does Firewall Groups > Services help?

pmueller · 5 December 2021 13:16

Hi,

first, welcome to the IPFire community.

Indeed, a bulk import functionality would be nice, but I am not aware of this being implemented via the web interface at the moment. There is one - um - hacky solution though:

IPFire stores custom defined networks, host groups, etc. in /var/ipfire/fwhosts/. If you know what you’re doing, you can change these files manually, script your changes and sync it to other IPFire installations.

For example, my /var/ipfire/fwhosts/customnetworks currently only contains Zoom server network ranges and looks like this:

34,ZOOM-34,52.81.215.0,255.255.255.0
2,ZOOM-2,3.21.137.128,255.255.255.128
5,ZOOM-5,3.25.41.128,255.255.255.128
67,ZOOM-67,152.67.152.0,255.255.248.0
84,ZOOM-84,168.138.74.0,255.255.255.128
19,ZOOM-19,3.235.71.128,255.255.255.128
78,ZOOM-78,162.255.36.0,255.255.252.0
38,ZOOM-38,64.211.144.0,255.255.255.0
93,ZOOM-93,193.122.32.0,255.255.240.0
42,ZOOM-42,99.79.20.0,255.255.255.128
88,ZOOM-88,168.138.244.0,255.255.255.0
74,ZOOM-74,160.1.56.128,255.255.255.128
71,ZOOM-71,152.67.240.0,255.255.248.0
97,ZOOM-97,193.123.0.0,255.255.224.0
10,ZOOM-10,3.101.32.128,255.255.255.128
81,ZOOM-81,168.138.48.0,255.255.255.0
63,ZOOM-63,147.124.96.0,255.255.224.0
31,ZOOM-31,50.239.204.0,255.255.255.0
105,ZOOM-105,204.80.104.0,255.255.248.0
92,ZOOM-92,193.122.16.0,255.255.240.0
43,ZOOM-43,101.36.167.0,255.255.255.0
21,ZOOM-21,3.235.73.0,255.255.255.128
51,ZOOM-51,120.29.148.0,255.255.255.0
104,ZOOM-104,202.177.207.128,255.255.255.224
47,ZOOM-47,115.110.154.192,255.255.255.192
54,ZOOM-54,129.151.48.0,255.255.240.0
8,ZOOM-8,3.80.20.128,255.255.255.128
24,ZOOM-24,4.35.64.128,255.255.255.128
3,ZOOM-3,3.22.11.0,255.255.255.0
7,ZOOM-7,3.25.49.0,255.255.255.0
58,ZOOM-58,130.61.164.0,255.255.252.0
112,ZOOM-112,213.244.140.0,255.255.255.0
28,ZOOM-28,18.157.88.0,255.255.255.0
62,ZOOM-62,144.195.0.0,255.255.0.0
9,ZOOM-9,3.96.19.0,255.255.255.0
15,ZOOM-15,3.208.72.0,255.255.255.128
53,ZOOM-53,129.151.40.0,255.255.252.0
101,ZOOM-101,193.123.168.0,255.255.248.0
41,ZOOM-41,69.174.108.0,255.255.252.0
23,ZOOM-23,4.34.125.128,255.255.255.128
6,ZOOM-6,3.25.42.0,255.255.255.128
57,ZOOM-57,129.159.208.0,255.255.240.0
44,ZOOM-44,103.122.166.0,255.255.254.0
27,ZOOM-27,13.52.146.0,255.255.255.128
72,ZOOM-72,158.101.64.0,255.255.255.0
32,ZOOM-32,52.61.100.128,255.255.255.128
113,ZOOM-113,221.122.88.64,255.255.255.224
48,ZOOM-48,115.114.56.192,255.255.255.192
82,ZOOM-82,168.138.56.0,255.255.248.0
110,ZOOM-110,213.19.144.0,255.255.255.0
37,ZOOM-37,64.125.62.0,255.255.255.0
4,ZOOM-4,3.23.93.0,255.255.255.0
64,ZOOM-64,149.137.0.0,255.255.128.0
87,ZOOM-87,168.138.116.0,255.255.252.0
52,ZOOM-52,129.151.0.0,255.255.224.0
106,ZOOM-106,204.141.28.0,255.255.252.0
22,ZOOM-22,3.235.96.0,255.255.254.0
77,ZOOM-77,162.12.232.0,255.255.252.0
16,ZOOM-16,3.211.241.0,255.255.255.128
68,ZOOM-68,152.67.168.0,255.255.252.0
91,ZOOM-91,192.204.12.0,255.255.252.0
94,ZOOM-94,193.122.208.0,255.255.240.0
73,ZOOM-73,158.101.184.0,255.255.252.0
98,ZOOM-98,193.123.40.0,255.255.248.0
61,ZOOM-61,140.238.232.0,255.255.252.0
33,ZOOM-33,52.81.151.128,255.255.255.128
83,ZOOM-83,168.138.72.0,255.255.255.0
25,ZOOM-25,8.5.128.0,255.255.254.0
107,ZOOM-107,207.226.132.0,255.255.255.0
55,ZOOM-55,129.159.0.0,255.255.240.0
99,ZOOM-99,193.123.176.0,255.255.240.0
13,ZOOM-13,3.120.121.0,255.255.255.128
60,ZOOM-60,140.238.128.0,255.255.255.0
69,ZOOM-69,152.67.180.0,255.255.255.0
17,ZOOM-17,3.235.69.0,255.255.255.128
76,ZOOM-76,161.199.136.0,255.255.252.0
100,ZOOM-100,193.123.128.0,255.255.224.0
86,ZOOM-86,168.138.96.0,255.255.252.0
90,ZOOM-90,173.231.80.0,255.255.240.0
116,ZOOM-116,221.123.139.192,255.255.255.224
36,ZOOM-36,52.215.168.0,255.255.255.128
109,ZOOM-109,209.9.215.0,255.255.255.0
49,ZOOM-49,115.114.115.0,255.255.255.192
26,ZOOM-26,13.52.6.128,255.255.255.128
12,ZOOM-12,3.104.34.128,255.255.255.128
103,ZOOM-103,198.251.128.0,255.255.128.0
56,ZOOM-56,129.159.160.0,255.255.224.0
85,ZOOM-85,168.138.80.0,255.255.248.0
35,ZOOM-35,52.202.62.192,255.255.255.192
75,ZOOM-75,161.189.199.0,255.255.255.128
40,ZOOM-40,69.174.57.0,255.255.255.0
111,ZOOM-111,213.19.153.0,255.255.255.0
29,ZOOM-29,18.205.93.128,255.255.255.128
59,ZOOM-59,134.224.0.0,255.255.0.0
1,ZOOM-1,3.7.35.0,255.255.255.128
95,ZOOM-95,193.122.224.0,255.255.240.0
102,ZOOM-102,193.123.192.0,255.255.224.0
46,ZOOM-46,111.33.181.0,255.255.255.128
65,ZOOM-65,152.67.20.0,255.255.255.0
115,ZOOM-115,221.122.89.128,255.255.255.128
114,ZOOM-114,221.122.88.128,255.255.255.128
50,ZOOM-50,115.114.131.0,255.255.255.192
20,ZOOM-20,3.235.72.128,255.255.255.128
45,ZOOM-45,111.33.115.0,255.255.255.128
108,ZOOM-108,209.9.211.0,255.255.255.0
70,ZOOM-70,152.67.184.0,255.255.252.0
30,ZOOM-30,50.239.202.0,255.255.254.0
80,ZOOM-80,168.138.16.0,255.255.252.0
96,ZOOM-96,193.122.240.0,255.255.240.0
11,ZOOM-11,3.101.52.0,255.255.255.128
14,ZOOM-14,3.127.194.128,255.255.255.128
89,ZOOM-89,170.114.0.0,255.255.0.0
39,ZOOM-39,65.39.152.0,255.255.255.0
18,ZOOM-18,3.235.82.0,255.255.254.0
79,ZOOM-79,165.254.88.0,255.255.254.0
66,ZOOM-66,152.67.118.0,255.255.255.0

Truth to be told, I wish there was a more elegant way for this. But if you are familiar with SSH and some IPFire internals, it should not be an impossible task to do - and manually filing every network via the WUI is tiresome indeed.

Thanks, and best regards,
Peter Müller

daaaa · 6 December 2021 11:20

Hi Paul,

Thanks for the suggestion, which is helpful, but doesn’t quite deal with my challenge. I already use the Services options (including Service Groups) to match allowed port + server combinations.

Its the only way I’ve found to avoid browser extension based VPNs from bypassing IPFire:

Allow nothing through the firewall.
Add servers for known Apps (e.g., the Zoom network list)
Add ports for known Apps (e.g., Zoom ports)
Add firewall rules to only allow outgoing data through the firewall for combinations of ports that should be going to particular servers
Force all other traffic to use IPFire’s proxy

Peter, your suggestion is a good one and it may just save my IPFire’s day - thanks so much! I see that I already have fairly complicated and full custom* files in /var/ipfire/fwhosts/customnetworks, but the structure looks simple enough. A little bit of pre-processing should allow me to just copy/paste collections of networks/hosts into the relevant files. I’ll give it a go one weekend.

daaaa · 12 December 2021 00:44

Things didn’t go as easily as I hoped.

I downloaded WinSCP, put jq on my IPFire and used this curl command to extract the AMAZON service networks to a text file.

I noted the IPFire format for files in /var/ipfire/fwhosts is basically CSV, so I thought I could copy them to my Window PC, do a few simple additions to the files in Excel and then put them back onto IPFire.

Unfortunately, the network list is about 4000 lines, and includes a whole bunch of /32 entries randomly scattered throughout. These need separating out to be put in IPFire’s customhosts file rather than customnetworks file. There’s probably a smart way to do it but it eludes me at the moment.

More of an issue, and for which there is no doubt a smart way to do things too, is that while IPFire’s web interface uses CIDR format, the customnetworks file actually has separate columns for the network and full netmask (e.g., /24 is in the next column as 255.255.255.0). The curl command exports as CIDR format, and I’ve yet to find an easy way to convert all 4000 entries to full netmasks in a separate column.

tphz · 12 December 2021 04:01

You can probably use filtering in a spreadsheet then copy to a new sheet.

You save the sheet with the /24 addresses as a .csv file.
You open it, for example, in Notepad++
You press CTRL+H
Replace /24 with ,255.255.255.0
EOL conversion to Unix(LF)
Save file.

Maybe it can be better automated, but I don’t have a better idea at the moment.

daaaa · 12 December 2021 10:08

tphz,

I saw your suggestion in the email notification, and I think it’ll work. I can do some search/replace tricks to achieve the effect I think - thanks for the suggestion! I’ll report back.

Cheers!

daaaa · 15 December 2021 11:01

All good!

With a bit of text processing combining Excel and PSPad I created the needed pair of files, merged them with customhosts and customnetworks and successfully modified the customgroups file to match.

There were two ‘gotchas’, which would be worth others knowing about:

The files created on Windows needed trailing commas on all cells except those with comments. This differs from the original files downloaded from IPFire, which had no trailing commas on any cells. I’d suggest it’s probably an end-of-line marker formatting difference, except that lines with comments didn’t need them in either case. Wierd.
WinSCP copies two of the files back with owner.group root.root, and one with nobody.nobody. That seems odd and random. The web interface can’t write to the files with owner.group root.root, so there’s a need to SSH in and do a chown to fix that problem.

I tried to include a copy of the final files for anyone else, to save them the hassle of creating them themselves, but the character count exceeded the allowed limit by about 10-fold. I’ll try and upload them instead - note that the initial numbers will of course need to be changed to match the sequence in your own files.
AWS host and networks.zip (39.8 KB)

Cheers!