Httpd not showed in netstat output

Hi all,

A couple of days back I installed ipfire but now I noticed something odd when I login via ssh.

When I do netstat -natlup and compare it with lsof -i output the httpd process is not showed.

Why is httpd hidden in netstat output but not in lsof output?

I can see httpd with both commands.

This is with IPFire 2.25 (x86_64) - Core Update 142

netstat -natlup;lsof -i
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      14320/unbound       
tcp        0      0 127.0.0.1:8953          0.0.0.0:*               LISTEN      14320/unbound       
tcp        0      0 0.0.0.0:222             0.0.0.0:*               LISTEN      12932/sshd          
tcp        0    196 *******:222       *******:54820    ESTABLISHED 21850/0             
udp        0      0 0.0.0.0:514             0.0.0.0:*                           11938/syslogd       
udp        0      0 0.0.0.0:53              0.0.0.0:*                           14320/unbound       
udp        0      0 0.0.0.0:67              0.0.0.0:*                           29952/dhcpd         
udp        0      0 0.0.0.0:68              0.0.0.0:*                           14200/dhcpcd        
udp        0      0 *******:123       0.0.0.0:*                           12840/ntpd          
udp        0      0 *******:123        0.0.0.0:*                           12840/ntpd          
udp        0      0 *******:123       0.0.0.0:*                           12840/ntpd          
udp        0      0 127.0.0.1:123           0.0.0.0:*                           12840/ntpd          
udp        0      0 0.0.0.0:123             0.0.0.0:*                           12840/ntpd          
COMMAND   PID   USER   FD   TYPE DEVICE SIZE NODE NAME
httpd    1353 nobody    4u  IPv6  23020       TCP *:81 (LISTEN)
httpd    1353 nobody    6u  IPv6  23024       TCP *:snpp (LISTEN)
httpd    1353 nobody    8u  IPv6  23028       TCP *:1013 (LISTEN)
httpd    1353 nobody   17u  IPv6 363457       TCP *******.localdomain:snpp->*******:45046 (ESTABLISHED)
syslogd 11938   root    5u  IPv4  18685       UDP *:syslog 
ntpd    12840   root   16u  IPv6  22060       UDP *:ntp 
ntpd    12840   root   17u  IPv4  22063       UDP *:ntp 
ntpd    12840   root   18u  IPv4  22068       UDP localhost.localdomain:ntp 
ntpd    12840   root   19u  IPv4  23403       UDP *******.localdomain:ntp 
ntpd    12840   root   20u  IPv4  23405       UDP *******.localdomain:ntp 
ntpd    12840   root   21u  IPv4  23476       UDP *******.localdomain:ntp 
sshd    12932   root    3u  IPv4  22997       TCP *:rsh-spx (LISTEN)
httpd   12946   root    4u  IPv6  23020       TCP *:81 (LISTEN)
httpd   12946   root    6u  IPv6  23024       TCP *:snpp (LISTEN)
httpd   12946   root    8u  IPv6  23028       TCP *:1013 (LISTEN)
dhcpcd  14200   root   10u  IPv4  23412       UDP *:bootpc 
unbound 14320 nobody    4u  IPv4  24310       UDP *:domain 
unbound 14320 nobody    5u  IPv4  24311       TCP *:domain (LISTEN)
unbound 14320 nobody    6u  IPv4  24312       TCP localhost.localdomain:8953 (LISTEN)
sshd    21850   root    4u  IPv4 354459       TCP ******.localdomain:rsh-spx->**********:54820 (ESTABLISHED)
dhcpd   29952   root    9u  IPv4 223134       UDP *:bootps

Added: ******* for some privacy.

Trying to find out by posting this here if my install is hacked or not.

The image checksum checked out good from the ipfire website.

Hi all,
have here also no problem with this -->

$ netstat -tulpn | grep httpd
tcp        0      0 0.0.0.0:2345            0.0.0.0:*               LISTEN      9190/httpd          
tcp        0      0 0.0.0.0:81              0.0.0.0:*               LISTEN      9190/httpd          
tcp        0      0 0.0.0.0:1010            0.0.0.0:*               LISTEN      9190/httpd          
tcp        0      0 0.0.0.0:8181            0.0.0.0:*               LISTEN      9190/httpd          
tcp        0      0 0.0.0.0:1013            0.0.0.0:*               LISTEN      9190/httpd          
tcp        0      0 0.0.0.0:444             0.0.0.0:*               LISTEN      9190/httpd          
tcp        0      0 0.0.0.0:55555           0.0.0.0:*               LISTEN      9190/httpd 

Now I am getting worried.

Why do you guys see it and why am I not seeing it :confused:

What else is hidden on my install then?

Coming from PFSENSE/OPNSENSE first install of IPFIRE, this is just great, not.

What delivers this one

ps aux | grep httpd | grep -v grep

on your system ?

Don´t worry be happy :slightly_smiling_face:

Best,

Erik

Its started, and I can see it with ps output, but not with netstat.

[root@nbox ~]# ps aux | grep httpd | grep -v grep
root     13465  0.0  0.1  10504  6852 ?        Ss   13:14   0:00 /usr/sbin/httpd -k start
nobody   13466  0.0  0.0   9660  3768 ?        S    13:14   0:00 /usr/sbin/httpd -k start
nobody   13470  1.4  0.2 1141176 8144 ?        Sl   13:14   0:00 /usr/sbin/httpd -k start

I nuked the install and went with something other. I don’t like it if there is stuff covered from my view with netstat, it makes me paranoid when it comes to network devices.

Clean install of latest ipfire. Sha checksum checked out good.

Pakfire packages I installed where: htop, traceroute, speedtest-cli and iptraf.

Perhaps some other time in the future I have more time to experiment wit ipfire, for now I have no more time.

Hi folks,
same here.

Have installed a fresh flash image of Core Update 143 on Banana Pi.

Missing listen ports of https in netstat:

# netstat -tnpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      1576/unbound        
tcp        0      0 127.0.0.1:8953          0.0.0.0:*               LISTEN      1576/unbound        
tcp        0      0 0.0.0.0:222             0.0.0.0:*               LISTEN      2755/sshd [listener

lsof shows the missing listen ports:

# lsof -i -P | grep LISTEN
unbound 1576 nobody    4u  IPv4  10027       TCP *:53 (LISTEN)
unbound 1576 nobody    5u  IPv4  10028       TCP localhost.localdomain:8953 (LISTEN)
httpd   2507   root    4u  IPv6  14513       TCP *:81 (LISTEN)
httpd   2507   root    6u  IPv6  14517       TCP *:444 (LISTEN)
httpd   2507   root    8u  IPv6  14521       TCP *:1013 (LISTEN)
httpd   2510 nobody    4u  IPv6  14513       TCP *:81 (LISTEN)
httpd   2510 nobody    6u  IPv6  14517       TCP *:444 (LISTEN)
httpd   2510 nobody    8u  IPv6  14521       TCP *:1013 (LISTEN)
sshd    2755   root    3u  IPv4  14134       TCP *:222 (LISTEN)

httpd is bound to IPv6 - but … IPv6 is’nt active :thinking:

# cat /proc/sys/net/ipv6/conf/all/disable_ipv6
1

I also have an Banana Pi with Update Core 142 (updated, not fresh setup) and there are all listen post also showing in netstat:

# netstat -tnpl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:81              0.0.0.0:*               LISTEN      2402/httpd          
tcp        0      0 0.0.0.0:1013            0.0.0.0:*               LISTEN      2402/httpd          
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      1494/unbound        
tcp        0      0 192.168.1.1:3128        0.0.0.0:*               LISTEN      2074/(squid-1)      
tcp        0      0 127.0.0.1:8953          0.0.0.0:*               LISTEN      1494/unbound        
tcp        0      0 0.0.0.0:444             0.0.0.0:*               LISTEN      2402/httpd          
tcp        0      0 0.0.0.0:222             0.0.0.0:*               LISTEN      2388/sshd           
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      2402/httpd

Mysterious! What’s going on here?

Addition:

After updating the above mentioned second Banana Pi to Core Update 143 the same effect occurs as described…

what does grep ipv6 /etc/sysctl.conf give …

Hello pavlos, look here:

# grep ipv6 /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

Today update to Core Update 144, situation unchanged… :face_with_head_bandage:

you should use ss that is replacing netstat:

[root@router init.d]# ss -lnp | grep httpd
RTNETLINK answers: Invalid argument
u_str  LISTEN  0       100                   /var/run/cgisock.3905 21605                                                  * 0                                    users:(("httpd",pid=3906,fd=4))
tcp    LISTEN  0       511                                       *:445                                                    *:*                                    users:(("httpd",pid=3908,fd=6),("httpd",pid=3905,fd=6))
tcp    LISTEN  0       511                                       *:82                                                     *:*                                    users:(("httpd",pid=3908,fd=4),("httpd",pid=3905,fd=4))
tcp    LISTEN  0       511                                       *:1014                                                   *:*                                    users:(("httpd",pid=3908,fd=8),("httpd",pid=3905,fd=8))

Thank you, poli4! I didn’t know ‘ss’ yet.

But that only seems to help.

‘ss -lnp’ shows all listen ports - both IPv4 and IPv6.

With ‘ss -4lnp’ the httpd is gone again.

This is also the reason why netstat does not show this anymore: netstat on ipfire dates back to 2001, is completely outdated and does not even know IPv6!

$ netstat -V
net-tools 1.60
netstat 1.42 (2001-04-15)
...

So the question remains: Why is httpd bound to the inactive IPv6 and not to IPv4? And why does it work anyway?

I do not (yet?) understand the logic behind it.

Is this a bug or a feature?

Btw.:

In the meantime I have tested the scenario on x86_64 (Core Update 144) on an APU3C4. The same problem exists there as well.

I too noticed this. Whilst I cannot answer as to HOW httpd is listening on IPv6 if it’s disabled, a workaround is to set it’s listen address to 0.0.0.0:81 and 0.0.0.0:444 instead of simply 81 and 444.

I have tested this on current IPFire and it indeed shows up in netstat again.

Ref: https://httpd.apache.org/docs/2.4/bind.html