TROUBLESHOOTING STEPS - Failure of server APACHE bridge

"Failure of server APACHE bridge" Explained

This generic error is seen in the end user browser. 

It occurs when there is some sort of 'disconnect' between the Oracle HTTP Server (OHS) mod_wl_ohs module and Oracle WebLogic Server. (The error can also be seen if you are front ending WebLogic Server with native Apache using mod_wl. This support document will focus on troubleshooting the error when using Oracle HTTP Server's mod_wl_ohs module.)  

The following Middleware 11g products - Portal / Forms / Reports / Discoverer - by default configured 'out of the box' to use mod_wl_ohs. 

Below is an example diagram showing how OHS front ends Forms Services via mod_wl_ohs.



Therefore, if you are using Portal / Forms / Reports or Discoverer you may encounter the 'Failure of server APACHE bridge' error as OHS attempts to communicate with the managed servers WLS_PORTAL, WLS_FORMS, WLS_REPORTS and/or WLS_DISCO.

The use of mod_wl_ohs is not, however, exclusive to Portal / Forms / Reports and Discoverer. It is possible to configure Oracle HTTP Server to be a 'front end' for Oracle WebLogic Server based applications e.g.

·         WebLogic JEE applications

·         SOA

·         WebCenter

·         Webgate for Oracle Access Manager

When Oracle HTTP Server is manually configured as a front end to such applications, the 'Failure of server APACHE bridge' may be seen.
 

Common causes are:

·         The target WebLogic Managed Servers are simply not started

·         The target WebLogic Managed Servers are hanging or not responsive

·         There is a problem with the mod_wl_ohs syntax which an administrator has added to the HTTP Server configuration file

·         There is an underlying network error or misconfiguration

The module mod_wl_ohs is based on the WebLogic Server Apache HTTP Server Plug-In - mod_wl . This Apache Plug-In is discussed in this document

Oracle Fusion Middleware Using Web Server 1.1 Plug-Ins with Oracle WebLogic Server 11g Release 1
Chapter 3 Installing and Configuring the Apache HTTP Server Plug-In

If, however, you are using Fusion Middleware 11g Web Tier refer to:

Oracle Fusion Middleware Administrator's Guide for Oracle HTTP Server 11g Release 1
Chapter 3 Understanding Oracle HTTP Server Modules
3.10 mod_wl_ohs

Oracle Fusion Middleware Administrator's Guide for Oracle HTTP Server 11g Release 1 (11.1.1)
Section 4.4.4 Configuring the mod_wl_ohs Module

Oracle Fusion Middleware Using Web Server 1.1 Plug-Ins with Oracle WebLogic Server 11g Release 1 (10.3.5)
Chapter 4 Configuring the Plug-In for Oracle HTTP Server

Common Causes / Solutions

Have the WebLogic Managed Server(s) or Cluster been started?

Unlike OracleAS10g Rel 2 and Rel 3 , java components of the technology stack are not managed by OPMN. 

(In OracleAS 10.1.2 and 10.1.3, the java component - OC4J - could be managed by OPMN and/or AS / EM Console). 

In Fusion Middleware 11g the java components - running in WebLogic - are managed by WebLogic Server administration tools (e.g. WLST, WebLogic Admin Console) and/or Fusion Middleware Control.  

For example, if you only issue the command

opmnctl startall

System / non java components (e.g HTTP Server, EM Agent.) should be up and running, but the WebLogic based java components will still be down.

Therefore you must ensure that the WebLogic Managed Servers or Cluster (depending on the choices made at AS Instance configuration time) are started.

For example:

·         Using Oracle Portal? WLS_PORTAL must be started

·         Using Oracle Forms? WLS_FORMS must be started

·         Using Oracle Reports? WLS_REPORTS must be started

·         Using Oracle Discoverer? WLS_DISCO must be started

The intuitive way to check and control the WebLogic Servers is to use Fusion Middleware Control. The screenshot below shows a snippet of the Fusion Middleware Control Forms Home Page. 



The WebLogic Managed Server - WLS_FORMS - has a status of UP. If the status was showing as DOWN the WLS_FORMS would need to be started to avoid the 'Failure of server APACHE bridge' error.

Here are the documentation references to find out more about starting and managing the products which use a combination of System (Web Tier) and Java Components (Weblogic) components.

General

Oracle Fusion Middleware Administrator's Guide 11g Release 1 (11.1.1)
Chapter 4 Starting and Stopping Oracle Fusion Middleware

Portal

Oracle Fusion Middleware Administrator's Guide for Oracle Portal 11g Release 1 
Chapter 8 Monitoring and Administering Oracle Portal

Forms

Oracle Fusion Middleware Forms Services Deployment Guide 11g Release 1
Chapter 4 Configuring and Managing Forms Services

Reports

Oracle Fusion Middleware Publishing Reports to the Web with Oracle Reports Services 11g Release 1
Chapter 7 Administering Oracle Reports Services Using Oracle Enterprise Manager

Discoverer

Oracle Fusion Middleware Configuration Guide for Oracle Business Intelligence Discoverer
11g Release 1
Chapter 4 Managing and Configuring Discoverer

Oracle Access Manager

Oracle Fusion Middleware Administrator's Guide for Oracle Access Manager
11g Release 1 
Chapter 2 Getting Started with OAM Administration and Navigation

Tip: On MS Windows the WebLogic Managed Servers do not, by default, get automatically started after a machine reboot. This is because WebLogic Managed Servers are not configured as MS Windows Services. If you wish to set up MS Windows Services for the individual WebLogic Managed Servers review the advice given in

How to Install MS Windows Services For Oracle Middleware 11g Rel 1 WebLogic Admin and Managed Servers? (Document 1060058.1)

Are the WebLogic Managed Servers Responsive?

If the managed servers are up and running, the error can still occur if there is an underlying issue which is causing the server to hang or respond slowly. For example, a hang may be caused by an application or related to an interaction with the Database.

There is no easy way to tell whether this is the case. Suggestions are:

·         Check the managed server logs for any errors / warnings or unusual output which may have occurred leading up to the appearance of the "Failure of Apache Bridge".

·         Can you successfully stop and restart the managed server?

Also a managed server functions as a lightweight HTTP Listener. Therefore, can you reach the managed server via its own port.

For example, if a managed server is listening on port 7004, what does the following URL return

http://machinehostingWLS.oracle.com:7004

If the WebLogic Managed Server is running and response you should at least see



Some products like Oracle Forms will allow you to test a more specific URL against the managed server listen port - such as

http://machinehostingWLS.oracle.com:7004/forms/lservlet

If WLS_FORM is responsive you will see: 

Note: The advice in this support document does not extend to troubleshooting issues / errors related to the WebLogic Managed Servers of individual products

In the event that: 

·         the managed server are up and running

·         there are no errors in the logs or signs of hanging

check the network configuration of the WebLogic Managed Server. For more detail refer to the "Network Diagnosis" section under "Diagnostic Tips".

Is MOD_WL_OHS front ending a cluster?

Review the following documentation

Oracle Fusion Middleware Using Web Server 1.1 Plug-Ins with Oracle WebLogic Server 11g Release 1 (11.1.1)
6.4 Understanding Connection Errors and Clustering Failover

Be prepared to investigate the performance and behaviour of your application / system. If necessary tune mod_wl_ohs by explicitly setting appropriate values for parameters which influence failover and timing.  

·         WLSocketTimeoutSecs

·         WLIOTimeoutSecs (new name for HungServerRecoverSecs)

·         Idempotent

·         DynamicServerList

·         WLCookieName

·         WLDNSRefreshInterval

·         MaxSkipTime

·         ConnectTimeoutSecs

·         ConnectRetrySecs

·         KeepAliveSecs

·         KeepAliveEnabled 

·         AcceptBacklog parameter

For more information about this parameters, refer to 7.1 General Parameters for Web Server Plug-Ins

Example which shows how to insert / set values for these parameters

SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9013
WLSocketTimeoutSecs 30
WLIOTimeoutSecs 500

1. Useful reference for OBIEE users:

Note 1476458.1 OBIEE 11.1.1.x - Avoid Error "Failure of server APACHE bridge" by Increasing Timeouts in Oracle HTTP Server Plug-in

2. The parameter Idempotent by default is set to ON. If, for some reason, it has been changed to OFF, plug-in clustering fail over will not work - making  "Failure of server APACHE bridge" more likely.

 

Has MOD_WL_OHS been correctly configured?

If you are using Portal / Forms / Reports and/or Discoverer the mod_wl_ohs configuration should be correct. 

Note: It is not recommended to change the mod_weblogic configuration which has been put in place the Configuration Assistant.

If you are manually configuring mod_wl_ohs for a JEE application or service - whether this be a custom application, SOA or WebCenter, then be sure to carefully follow the relevant documentation e.g.

Oracle Fusion Middleware Using Web Server Plug-Ins with Oracle WebLogic Server 11g Release 1
Chapter 3 Installing and Configuring the Apache HTTP Server Plug-In

Additional Useful Reference:

Note 1316142.1 How To Configure mod_wl_ohs with Oracle HTTP Server and Oracle WebLogic Server

The file containing the specific mod_wl_ohs configuration varies depending on the product. Portal / Forms / Reports / Discoverer mod_wl_ohs examples (as put in place by the Configuration Assistant) are shown below:

Product -> Portal

File Location -> $ORACLE_INSTANCE/config/OHS//moduleconf/portal.conf

WLS Cluster ->  

SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001



SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001



SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001



SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001

SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001



SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001



SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001

No Cluster -> 

SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9001



SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9001



SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9001



SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9001



SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9001



SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9001



SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9001

 

Product -> Forms

File Location -> $ORACLE_INSTANCE/config/OHS//moduleconf/forms.conf

WLS Cluster -> 
SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9001
DynamicServerList ON

No Cluster -> 
SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9012

 

Product -> Reports

File Location -> $ORACLE_INSTANCE/config/OHS//moduleconf/reports_ohs.conf

WLS Cluster -> 

SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com
WebLogicPort 9013
DynamicServerList ON

No Cluster -> 

SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9013

 

Product -> Discoverer

File Location -> $ORACLE_INSTANCE/config/OHS//moduleconf/module_disco.conf

WLS Cluster -> 

SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9002
DynamicServerList ON

No Cluster -> 

SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 9014

 

Product -> General

File Location -> $ORACLE_INSTANCE/config/OHS//httpd.conf OR in an HTTP Server "include" .conf file. By default, a Fusion Middleware 11g Web Tier configuration creates a $ORACLE_INSTANCE/config/OHS//mod_wl_ohs.conf file.

WLS Cluster -> 

SetHandler weblogic-handler
WebLogicCluster demohost.us.oracle.com:9013

No Cluster -> 

SetHandler weblogic-handler
WebLogicHost demohost.us.oracle.com
WebLogicPort 7001

Tips:

1. The Port numbers shown in the table above are just examples. When reviewing the mod_wl_ohs configuration lines double check that the port values match up with the port values assigned to the WebLogic Managed Server or WebLogic Managed Servers in a WebLogic Cluster.

A WebLogic Cluster consists of one or more WebLogic Managed Servers that reside in the same WebLogic Domain. If there is more than one WebLogic Managed Server in a WebLogic Cluster then typically these Managed Servers reside on different host machines. All the WebLogic Managed Servers in a WebLogic Cluster have the same port value.

For example:

·         No Cluster

o    WLS_FORMS - demohost1.us.oracle.com:7004

·         Cluster

o    WLS_FORMS_CLUSTER:

§  WLS_FORMS - demohost1.us.oracle.com:7004

§  WLS_FORMS - demohost2.us.oracle.com:7004

2. Be aware that the value of the directive DynamicList affects the behaviour of the weblogic plug-in regards clusters. If DynamicList is set to OFF, it can lead to the "Failure of SERVER Apache Bridge" error, for example:

Reference:

Note 1380762.1 Forms / Reports 11g Intermittent "Failure of Server APACHE Bridge" - OHS Log Shows "apr_socket_connect call failed"

If you do not know how to check the port values or you think they may have changed, review the advice given in:

Oracle Fusion Middleware Administrator's Guide 11g Release 1
Chapter 5 Managing Ports

The chapter does advise an administrator to review and amend the mod_wl_ohs configuration in the relevant Oracle HTTP Server .conf file should a WebLogic Managed Server port value be changed post running the Configuration Assistant.

2. Incorrect syntax or typos in the mod_wl_ohs configuration can result in an "APACHE Bridge Failure". 

Example: If you are configuring mod_wl_ohs to use SSL parameters (e.g. WLSSLWallet ) an incorrectly specified attribute value or typo causes the wallet not to be found. The browser throws "APACHE Bridge Failure" and the HTTP Server logs report "Internal error: WL SSL Init failed for path".

Is the Network correctly configured?

There is a known issue if you are working with Fusion Middleware 11g on MS Windows and wish to use IPv6 addresses:

apr_socket_connect call failed Logged in OHS 11.1.1.2 / 11.1.1.3 Running on Windows (Document 1083385.1)

The issue is fixed if apply the latest Fusion Middleware 11g patchset, refer to:

Steps to Maintain Oracle Fusion Middleware 11g (11.1.1) (Document 1073776.1)

If this issue does not apply to your circumstances read on, work through the network checks described in the Diagnostic Tips.

Are you configuring SSL?

Be aware that "Failure of Server APACHE Bridge" can be seen if the parameter "WlSSLWallet" has not been configured correctly. Refer to the problem described in

"Internal error: WL SSL Init failed" and APACHE Bridge Failures After Configuring SSL for OHS/mod_wl_ohs and WLS (Document 1531143.1)

Review Logs

There are no specific mod_wl_ohs logs. Instead, check the Oracle HTTP Server error log. For more information about viewing the HTTP Server's logs read

Oracle Fusion Middleware Administrator's Guide for Oracle HTTP Server 11g Release 1 
Section 7.4, "Viewing Oracle HTTP Server Logs"

It may also be necessary to check the WebLogic Managed Server logs for errors i.e ////logs e.g

MIDDLEWARE_HOME/user_projects/domains/Forms_Domain/servers/WLS_FORMS/logs

Diagnostic Tips

Viewing mod_wl_ohs Routing and Debug Info

Follow the advice given in

How to View FMW 11g Web Tier mod_wl_ohs Routing and Other Debug Information? (Document 967889.1)

and review the mod_wl_ohs routing and debug information

Network Diagnosis

The mod_wl_ohs configuration lines include a reference to the hostname. If the hostname cannot be resolved you will see  the error 'Failure of server APACHE bridge'.

If the Oracle HTTP Server and WebLogic Server Managed Servers are residing on different machines, it is important to check that there is network connectivity between the two machines - going in both directions

machinehostingOHS.oracle.com <--->  machinehostingWLS.oracle.com

Before you investigate the network, take a look at the HTTP Server error_log. Errors found in this file,like the ones below, indicate a network misconfiguration or problem.

[2010-05-28T09:37:55.0919+02:00] [OHS] [WARNING:32] [OHS-9999] [core.c] [host_id: FMW11g] [host_addr: x.x.x.x] [pid: 3532] [tid: 1668] [user: SYSTEM] [VirtualHost: main] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed.

[2010-05-28T09:37:55.0919+02:00] [OHS] [WARNING:32] [OHS-9999] [core.c] [host_id: FMW11g] [host_addr: x.x.x.x] [pid: 3532] [tid: 1668] [user: SYSTEM] [VirtualHost: main] (OS 64)The specified network name is no longer available. : winnt_accept: Asynchronous AcceptEx failed.

[2010-05-28T09:40:26.2871+02:00] [OHS] [ERROR:32] [OHS-9999] [mod_weblogic.c] [host_id: FMW11g] [host_addr: x.x.x.x] [pid: 3532] [tid: 1640] [user: SYSTEM] [VirtualHost: main] ap_proxy: trying GET /forms/frmservlet/admin at backend host '::1/9001; got exception 'NO_RESOURCES: [os error=0, line 1787 of ../nsapi/URL.cpp]: apr_socket_connect call failed '

[2010-05-28T09:40:29.2979+02:00] [OHS] [ERROR:32] [OHS-9999] [mod_weblogic.c] [host_id: FMW11gr: x.x.x.x] [pid: 3532] [tid: 1640] [user: SYSTEM] [VirtualHost: main] ap_proxy: trying GET /forms/frmservlet/admin at backend host '::1/9001; got exception 'NO_RESOURCES: [os error=0, line 1787 of ../nsapi/URL.cpp]: apr_socket_connect call failed '

To check whether a network misconfiguration is causing the error, work through the following steps:

Note: In a high availability architecture you may multiple machines hosting Oracle HTTP Server and multiple machines hosting WebLogic Server Managed Servers. So the checks described below may be many.

1. What hostname / ip address is the WebLogic Managed Server configured to listen against?

To quote from:

Oracle Fusion Middleware Oracle WebLogic Server Administration Console Online Help 11g Release 1
Servers: Configuration: General

The IP address or DNS name this server uses to listen for incoming connections.

Servers can be reached through the following URL:
protocol://listen-address:listen-port

Any network channel that you configure for this server can override this listen address.

If a server's listen address is undefined, clients can reach the server through an IP address of the computer that hosts the server, a DNS name that resolves to the host, or the localhost string. The localhost string can be used only for requests from clients that running on the same computer as the server.

If you want to limit the valid addresses for a server instance, specify one of the following:

·         If you provide an IP address, clients can specify either the IP address or a DNS name that maps to the IP address. Clients that specify an IP address and attempt to connect through an SSL port must disable hostname verification.

·         If you provide a DNS name, clients can specify either the DNS name or the corresponding IP address.

Do not leave the listen address undefined on a computer that uses multiple IP address (a multihomed computer). On such a computer, the server will bind to all available IP addresses.

Notes:

To resolve a DNS name to an IP address, WebLogic Server must be able to contact an appropriate DNS server or obtain the IP address mapping locally. Therefore, if you specify a DNS name for the listen address, you must either leave a port open long enough for the WebLogic Server instance to connect to a DNS server and cache its mapping or you must specify the IP address mapping in a local file. If you specify an IP address for ListenAddress and then a client request specifies a DNS name, WebLogic Server will attempt to resolve the DNS name, but if it cannot access DNS name mapping, the request will fail.

Note also that you are using the demo certificates in a multi-server domain, Managed Server instances will fail to boot if you specify the fully-qualified DNS name. For information about this limitation and suggested workarounds, see "Limitation on CertGen Usage" in Securing Oracle WebLogic Server.

MBean Attribute:
ServerMBean.ListenAddress

Changes take effect after you redeploy the module or restart the server.

Check the WebLogic Managed Server network configuration in the WebLogic Admin Server console.

Reference:
Oracle Fusion Middleware Oracle WebLogic Server Administration Console Online Help 11g Release 1
Configure listen addresses

If an explicit hostname / ip address has been configured for the WebLogic Managed Server, the details need to match up with the hostname / ip address mapping as configured in mod_wl_ohs.

2. Does the hostname in the mod_wl_ohs configuration resolve to an IP address? 

From the OHS machine:

ping machinehostingWLS.oracle.com

From the WLS machine(s):

ping machinehostingOHS.oracle.com

Furthermore, is the IP address returned by ping the one you expect to be associated with the hostname?

3. Is the hostname / ip address mapping held in a DNS Server and/or in the machine's local hosts file? 

Check the following:

a) Review the output from 'nslookup' - run this command e.g

From the OHS machine:

nslookup machinehostingWLS.oracle.com

From the WLS machine(s):

nslookup machinehostingWLS.oracle.com

b) Collect the IP address configuration - run these commands on each machine hosting Oracle HTTP Server and WebLogic Managed Servers.

MS Windows:
ipconfig -a

Unix:
ifconfig -a

c) Review the local hosts files on the machines hosting Oracle HTTP Server and WebLogic Managed Servers. If hostname / ip address mappings are held in these files ensure there are no mistakes

·         the IP addresses referenced by the hostname in the hosts file should match up with the information returned by the ipconfig/ifconfig commands

·         the hostname should NOT be used as an alias for the loopback address 127.0.0.1 e.g

127.0.0.1 localhost machinehostingWLS
11.22.33.44 machinehostingWLS.oracle.com machinehostingWLS

·         ideally the hostname should NOT be used as an alias for multiple IP addresses as this can cause confusion e.g

11.22.33.44 machinehostingWLS.oracle.com machinehostingWLS
66.77.88.99 anothermachine.oracle.com machinehostingWLS

4. As a troubleshooting test backup and edit the HTTP Server conf file containing the mod_wl_ohs configuration lines. Change the hostname reference to be the explicit IP address.

In the case of Forms for example, edit 

$ORACLE_INSTANCE\config\OHS\ohs1\moduleconf\forms.conf and change the hostname to the IP address of the machine hosting the WebLogic Managed Servers.

Change

SetHandler weblogic-handler
WebLogicHost machinehostingWLS
WebLogicPort 9001

to:

SetHandler weblogic-handler
WebLogicHost 11.22.33.44
WebLogicPort 9001

Save the file, and stop/ restart HTTP Server for the change to take effect.

5 Can you ping the port value using a tool like 'telnet'

telnet demohost.us.oracle.com 9001

If the telnet fails (and the WebLogic Managed Server is up and listening to the specified port value) there is likely to be a network misconfiguration.

6. FIREWALLS? 

Are there any firewalls between the machine hosting the Oracle HTTP Server(s) and the WebLogic Managed Servers. If so, ensure that the Firewall has rules to allow traffic through on the ports values configured in mod_wl_ohs.

MoS Community Discussions

Still have questions?  Use Oracle Application Server Community in My Oracle Support to search for similar discussions or start a new discussion on this subject.