Issue: Application not able to connect to the Oracle database. Connections are not going through and even when you try to telnet to the IP address, you get an error message:
telnet 172.32.362.171 1521
Connecting To 172.32.362.171…“Could not open connection to the host, on port 1521: Connect failed”
- Checked for Firewall setting and checked fire wall logs to confirm that access is being allowed through the firewall. Firewall does show connection attempts going through it.
- Checked for database listener and listener logs for any errors or connection attempts refused, however there were no errors or information on connection attempts.
- Checked for listener and tnsnames entries for any incorrect information.
- Traced network packets at the database host to confirm
- Whether or not the database server is seeing the traffic from the application server
- Whether database server is sending back the response to the application server.
- After doing network trace it was confirmed that “traffic was only going in towards database, however database host server was not responding or acknowledging any packet requests”
- On database host, done netstat –a, and from the results we could clearly see that packets are reaching from 172.32.334.100, however their status is SYN_RCVD, which means a remote host has asked for starting a connection however confirmation message is not being send to 172.32.334.100. This may be because of firewall from 172.32.362.171 to 172.32.334.100 or may be because of difference in routing. For successful connection netstat status must be “ESTABLISHED” instead of “SYN_RCVD”.
test1clust.1521 172.32.334.100.3934 0 0 49640 0 SYN_RCVD
test1clust.1521 172.32.334.100.3938 0 0 49640 0 SYN_RCVD
- Above information proved that there are issues around database host server itself. Possible scenarios:
- Firewall issues (which we have already checked and confirmed that there are no issues around this)
- Host-based firewall is not allowing these connections to go up to DB listener (hence the DB listener is not seeing any connection attempts and hence not responding back).
- Database host doesn’t have static route to correctly route back response traffic back to the required subnet.
Upon checking the route, it was found that there is no route for required subnet.
The responses from database host server should leave via the same network interface (on the host) through which application server access-attempts are hitting database host. Therefore the gateway configured for the destination subnet (172.32.334.100) should be the same as the gateway configured for this network interface.
This was then determined to be an incorrect route on the DB host which is causing the problem. We added a route on the DB server and that resolved the issue.