This post is older than a year. Consider some information might not be accurate anymore.
This blog post demonstrates a small example how helpful netstat can be. netstat is capable to display all used ports of the user processes.
Following command can be a life-saver
netstat -tulpn
Options explained:
- -t = tcp
- -u = udp
- -l = listening
- -p = program
- -n = numeric
Problem
First of all, I have a Java EE application, that claims to run the management port 12099
tan@server:~> ps -Af | grep demo
tan 11182 10919 0 Jan19 ? 00:07:18 /opt/java/jdk1.8.0_51/bin/java -D[Standalone] -server -XX:+UseCompressedOops -verbose:gc -Xloggc:/var/log/jboss/demo_prd_1/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.modules.policy-permissions=true -Dorg.jboss.boot.log.file=/var/log/jboss/demo_prd_1/log/server.log -Dlogging.configuration=file:/opt/demo_prd_1/configuration/logging.properties -jar /opt/jboss/jboss-eap-6.4.2/jboss-modules.jar -mp /opt/jboss/jboss-eap-6.4.2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/opt/jboss/jboss-eap-6.4.2 -Djboss.server.base.dir=/opt/apps/demo_prd_1 -Djboss.server.base.dir=/opt/apps/demo_prd_1 -b=192.168.100.74 -Djboss.management.native.port=12099 -bmanagement=192.168.100.74 -Djboss.server.log.dir=/var/log/jboss/demo_prd_1/log -P /opt/apps/demo_prd_1/configuration/demo_prd_1.properties --server-config=standalone.xml
Even the log is telling me, that this port is being used
2016-01-19 05:29:16,069 INFO [MSC service thread 1-8] [] [] [] [] [] (org.jboss.as) JBAS015899: JBoss EAP 6.4.2.GA (AS 7.5.2.Final-redhat-2) starting
2016-01-19 05:29:16,074 DEBUG [MSC service thread 1-8] [] [] [] [] [] (org.jboss.as.config) Configured system properties:
[Standalone] =
..
jboss.management.native.port = 12099
jboss.modules.dir = /opt/jboss/jboss-eap-6.4.2/modules
Now I try to connect to the CLI to change the log level at runtime, if you haven’t setup JAVA_HOME properly, you can setup it accordingly with these commands:
tan@server:~> export JAVA_HOME=/opt/java/jdk1.8.0_51
tan@server:~> export PATH=$PATH:$JAVA_HOME/bin
tan@server:~> /opt/jboss/jboss-eap-6.4.2/bin/jboss-cli.sh --controller=192.168.100.74:12099 --connect
org.jboss.as.cli.CliInitializationException: Failed to connect to the controller
..
Caused by: org.jboss.as.cli.CommandLineException: The controller is not available at 192.168.100.74:12099
... 8 more
Caused by: java.io.IOException: java.net.ConnectException: JBAS012174: Could not connect to remote://192.168.100.74:12099. The connection failed
... 11 more
Caused by: java.net.ConnectException: JBAS012174: Could not connect to remote://192.168.100.74:12099. The connection failed
... 13 more
Caused by: java.net.ConnectException: Connection refused
... 23 more
WTF happened? The port should be ok.
Check with netstat
Now we can check with netstat which ports are used by process 11182.
tan@server:~> netstat -tulpn | grep 11182
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 10.152.48.74:12090 0.0.0.0:* LISTEN 11182/java
tcp 0 0 10.152.48.74:13099 0.0.0.0:* LISTEN 11182/java
tcp 0 0 10.152.48.74:8943 0.0.0.0:* LISTEN 11182/java
tcp 0 0 10.152.48.74:12047 0.0.0.0:* LISTEN 11182/java
tcp 0 0 10.152.48.74:12080 0.0.0.0:* LISTEN 11182/java
Well we haven’t found 12099, but 13099! Now let’s check the login.
tan@server:~> /opt/jboss/jboss-eap-6.4.2/bin/jboss-cli.sh --controller=192.168.100.74:13099 --connect
[standalone@192.168.100.74:13099 /] /subsystem=logging/logger=net.cinhtau.net.dmr:write-attribute(name="level", value="INFO")
{"outcome" => "success"}
[standalone@192.168.100.74:13099 /] exit
Check the console.log, that tells the truth
tan@server:~/logs> cat console.log
..
05:29:16,787 INFO [org.jboss.as.remoting] (MSC service thread 1-45) JBAS017100: Listening on 192.168.100.74:12047
..
05:29:16,908 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-27) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/192.168.100.74:12080
05:29:16,912 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-27) JBWEB003000: Coyote HTTP/1.1 starting on: http-/192.168.100.74:12080
..
05:29:16,960 INFO [org.jboss.as.remoting] (MSC service thread 1-14) JBAS017100: Listening on 192.168.100.74:13099
