connection reset error, followed by broken pipe (pg74.213.jdbc3.jar)

Hi,

I'm having trouble with a tomcat-postgres installation. Server 1 runs
postgresql and tomcat, server 2 has an identical tomcat installation
and application code running (except rebranding). Server 2 has errors
about every week with 'connection reset' in logs/appname.date.log,
futher connection attempts have a 'broken pipe' error. The solution
to get the app running again is to restart tomcat. Both servers have
low load ~50 users per week.

tomcat 5.0.28
postgresql 7.4.8
java jdk 1.5.0_02
jdbc driver pg74.213.jdbc3.jar

Stack trace errors are:

2006-04-13 15:54:03 StandardWrapperValve[AccessPortal]:
Servlet.service() for servlet AccessPortal threw exception
javax.servlet.ServletException: org.postgresql.util.PSQLException: An
I/O error has occured while flushing the output - Exception:
java.net.SocketException: Connection reset
Stack Trace:

java.net.SocketException: Connection reset
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at org.postgresql.core.PGStream.flush(PGStream.java:411)
at
org.postgresql.core.QueryExecutor.sendQueryV3(QueryExecutor.java:337)
at
org.postgresql.core.QueryExecutor.executeV3(QueryExecutor.java:121)
at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:100)
at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:43)
at
org.postgresql.jdbc1.AbstractJdbc1Statement.execute(AbstractJdbc1State
ment.java:517)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2State
ment.java:50)
at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:233)
at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:221)
at
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingSta
tement.java:205)
at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:126)
at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
at java.lang.Thread.run(Thread.java:595)
End of Stack Trace

at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:190)
at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
at java.lang.Thread.run(Thread.java:595)

2006-04-13 16:06:51 StandardWrapperValve[AccessPortal]:
Servlet.service() for servlet AccessPortal threw exception
javax.servlet.ServletException: org.postgresql.util.PSQLException: An
I/O error has occured while flushing the output - Exception:
java.net.SocketException: Broken pipe
Stack Trace:

java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at org.postgresql.core.PGStream.flush(PGStream.java:411)
at
org.postgresql.core.QueryExecutor.sendQueryV3(QueryExecutor.java:337)
at
org.postgresql.core.QueryExecutor.executeV3(QueryExecutor.java:121)
at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:100)
at org.postgresql.core.QueryExecutor.execute(QueryExecutor.java:43)
at
org.postgresql.jdbc1.AbstractJdbc1Statement.execute(AbstractJdbc1State
ment.java:517)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2State
ment.java:50)
at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:233)
at
org.postgresql.jdbc1.AbstractJdbc1Statement.executeQuery(AbstractJdbc1
Statement.java:221)
at
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingSta
tement.java:205)
at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:126)
at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
at java.lang.Thread.run(Thread.java:595)
End of Stack Trace

at
com.ourdomain.ourapp.AccessPortal.getUserData(AccessPortal.java:190)
at
com.ourdomain.ourapp.AccessPortal.processRequest(AccessPortal.java:71)
at com.ourdomain.ourapp.AccessPortal.doGet(AccessPortal.java:88)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:237)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:157)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:214)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardC
ontextValve.java:198)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:152)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat
orBase.java:540)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:137)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:118)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:102)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:109)
at
org.apache.catalina.core.StandardValveContext.invokeNext(StandardValve
Context.java:104)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java
:520)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
at
org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160
)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proces
sConnection(Http11Protocol.java:705)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:
577)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:683)
at java.lang.Thread.run(Thread.java:595)

I haven't been able to find any matches for this problem on the
internet, so I'm posting here hoping someone is aware of the problem.

There is a release pg74.216.jdbc3.jar, I can't find a changelog for
it and builds 213-216, anyone know? It may be that these include a
bug fix. I'm checking pg74.216.jdbc3.jar on a development server, so
will probably try this next.

Thanks,
Ed

Hi, Ed,

Ed Blackmore wrote:

> I'm having trouble with a tomcat-postgres installation. Server 1 runs
> postgresql and tomcat, server 2 has an identical tomcat installation
> and application code running (except rebranding). Server 2 has errors
> about every week with 'connection reset' in logs/appname.date.log,
> futher connection attempts have a 'broken pipe' error.

So it seems that the network connection breaks from time to time.
Independently from making your application more robust, you might want
to put some research into that.

> The solution to get the app running again is to restart tomcat.

There was a thread here some days ago with the title "No automatic
reconnect after network error", you might have a look there.

If you're using the tomcat connection pooling, my proposed solution
there might help you:

| At least JBoss, you can use the following approach: In the Datasource
| definition, put the following:
|
| <datasources>
| <no-tx-datasource>
| <jndi-name>boo</jndi-name>
| <connection-url>jdbc:postgresql://localhost:5432/bar</connection-url>
| [ ... other attributes ... ]
| <new-connection-sql>SELECT version()</new-connection-sql>
| <check-valid-connection-sql>SELECT
| version()</check-valid-connection-sql>
| </no-tx-datasource>
| </datasources>
|
| This validates the connection before it is handed out on the
| application, and throws away invalid connections. Be aware that the SQL
| is case sensitive, we had a problem with earlier versions of OpenMDX/CRX
| that had upper case letters in the tags so they were ignored by jboss.

HTH,
Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf. | Software Development GIS

Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org