IPSec with public Alias-IPs: port 500/udp open on all IPs

Hello community,

we use IPFire (latest patchlevel) with 1 genuine public IP-address and 3 additional public Alias IP-addresses.
Only 2 of the total 3 Alias-IP-addresses are in use for IPSec-N2N connections. Unexpectedly, nmap reports port 500/udp to be open on all 4 IP-addresses. How can I prevent this behaviour?

Any help is welcome - thanks!