org.apache.cxf.interceptor.Fault: Could not send Message.

Eight Stars

org.apache.cxf.interceptor.Fault: Could not send Message.

Last night, I started seeing this error while POSTING data to a RESTful service via TESB open studio 6.3.1 deployed in Karaf Runtime container. I am accepting requests via tRestRequest in the job and deployed the .kar file on the runtime environment.

 

I have never seen this before and I have no clue how to address it. is it a server side/ client side error?

 

One thing I did is reduce the data being posted, and this error went away. 
I don't remember the exact size of the data posted.

 

I have a SAM feature running saving data in SQLSERVER DB.

 

My version is TESB Open - 6.3.1. Any comments/ feedback appreciated.

 

ID: 74
Response-Code: 500
Encoding: ISO-8859-1
Content-Type: 
Headers: {flowid=[urn:uuid:6de2f3d7-ef8c-4f6c-88b3-70569b612ee2]}
Payload: <ns1:XMLFault xmlns:ns1="http://cxf.apache.org/bindings/xformat"><ns1:faultstring xmlns:ns1="http://cxf.apache.org/bindings/xformat">java.lang.RuntimeException: org.eclipse.jetty.io.EofException: Early EOF</ns1:faultstring></ns1:XMLFault>
--------------------------------------
01:38:54,605 | WARN  | tp1706866333-134 | ache.cxf.common.logging.LogUtils  449 | 90 - org.apache.cxf.cxf-core - 3.1.7 | Interceptor for {http://service_0_73.mine/}RestServiceProviderImpl4TalendJob has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:112)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.phase.PhaseInterceptorChain.wrapExceptionAsFault(PhaseInterceptorChain.java:366)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:324)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:254)[116:org.apache.cxf.cxf-rt-transports-http:3.1.7]
	at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:234)[117:org.apache.cxf.cxf-rt-transports-http-jetty:3.1.7]
	at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)[117:org.apache.cxf.cxf-rt-transports-http-jetty:3.1.7]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.Server.handle(Server.java:499)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)[190:org.eclipse.jetty.io:9.2.19.v20160908]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[201:org.eclipse.jetty.util:9.2.19.v20160908]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[201:org.eclipse.jetty.util:9.2.19.v20160908]
	at java.lang.Thread.run(Thread.java:745)[:1.8.0_77]
Caused by: org.eclipse.jetty.io.EofException
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:192)[190:org.eclipse.jetty.io:9.2.19.v20160908]
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:408)[190:org.eclipse.jetty.io:9.2.19.v20160908]
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)[190:org.eclipse.jetty.io:9.2.19.v20160908]
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:129)[190:org.eclipse.jetty.io:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:694)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:246)[201:org.eclipse.jetty.util:9.2.19.v20160908]
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:208)[201:org.eclipse.jetty.util:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:471)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:763)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:801)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:140)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:242)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)[:1.8.0_77]
	at org.apache.cxf.io.AbstractWrappedOutputStream.flush(AbstractWrappedOutputStream.java:84)[90:org.apache.cxf.cxf-core:3.1.7]
	at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)[:1.8.0_77]
	at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:55)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:216)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.io.CacheAndWriteOutputStream.postClose(CacheAndWriteOutputStream.java:56)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.io.CachedOutputStream.close(CachedOutputStream.java:216)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:741)[116:org.apache.cxf.cxf-rt-transports-http:3.1.7]
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)[90:org.apache.cxf.cxf-core:3.1.7]
	... 20 more
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)[:1.8.0_77]
	at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)[:1.8.0_77]
	at sun.nio.ch.IOUtil.write(IOUtil.java:148)[:1.8.0_77]
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)[:1.8.0_77]
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:172)[190:org.eclipse.jetty.io:9.2.19.v20160908]
	... 42 more
01:38:54,910 | INFO  | faultExecutor-15 | m.agent.collector.EventCollector  313 | 233 - sam-agent - 6.3.1 | Put events(2) to Monitoring Server.
01:39:06,388 | WARN  | tp1706866333-133 | ache.cxf.common.logging.LogUtils  449 | 90 - org.apache.cxf.cxf-core - 3.1.7 | Interceptor for http://service_0_73.mine/}RestServiceProviderImpl4TalendJob has thrown exception, unwinding now
java.lang.RuntimeException: org.eclipse.jetty.io.EofException: Early EOF
	at org.talend.esb.sam.agent.wiretap.WireTapIn.handleMessage(WireTapIn.java:98)[233:sam-agent:6.3.1]
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:254)[116:org.apache.cxf.cxf-rt-transports-http:3.1.7]
	at org.apache.cxf.transport.http_jetty.JettyHTTPDestination.doService(JettyHTTPDestination.java:234)[117:org.apache.cxf.cxf-rt-transports-http-jetty:3.1.7]
	at org.apache.cxf.transport.http_jetty.JettyHTTPHandler.handle(JettyHTTPHandler.java:70)[117:org.apache.cxf.cxf-rt-transports-http-jetty:3.1.7]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1129)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1065)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.Server.handle(Server.java:499)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)[190:org.eclipse.jetty.io:9.2.19.v20160908]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)[201:org.eclipse.jetty.util:9.2.19.v20160908]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)[201:org.eclipse.jetty.util:9.2.19.v20160908]
	at java.lang.Thread.run(Thread.java:745)[:1.8.0_77]
Caused by: org.eclipse.jetty.io.EofException: Early EOF
	at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:519)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:125)[198:org.eclipse.jetty.server:9.2.19.v20160908]
	at java.io.FilterInputStream.read(FilterInputStream.java:133)[:1.8.0_77]
	at org.apache.cxf.metrics.interceptors.CountingInputStream.read(CountingInputStream.java:47)[101:org.apache.cxf.cxf-rt-features-metrics:3.1.7]
	at java.io.FilterInputStream.read(FilterInputStream.java:107)[:1.8.0_77]
	at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:209)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.apache.cxf.helpers.IOUtils.copy(IOUtils.java:151)[90:org.apache.cxf.cxf-core:3.1.7]
	at org.talend.esb.sam.agent.wiretap.WireTapIn.handleMessage(WireTapIn.java:94)[233:sam-agent:6.3.1]
	... 17 more
01:39:06,389 | WARN  | tp1706866333-133 | entproducer.MessageToEventMapper  345 | 233 - sam-agent - 6.3.1 | Could not find CachedOutputStream in message. Continuing without message content
01:39:06,403 | WARN  | tp1706866333-133 | ache.cxf.common.logging.LogUtils  449 | 90 - org.apache.cxf.cxf-core - 3.1.7 | WS-Addressing - failed to retrieve Message Addressing Properties from context
01:39:06,403 | INFO  | tp1706866333-133 | eptor.AbstractLoggingInterceptor  253 | 90 - org.apache.cxf.cxf-core - 3.1.7 | Outbound Message
---------------------------

 


Accepted Solutions
Eight Stars

Re: org.apache.cxf.interceptor.Fault: Could not send Message.

Hi @Tobi,
When I was running my code on local network, it never gave an issue (regardless of the payload size). Howecer, when deployed on production, with all the unknown security settings, we were able to see that the client connection was cutoff after some 10MB of payload or similar. After we got that figured, we just had to open this endpoint for no limits and it's been going great.

Sorry this may not be what you wanna hear, but work a shot to check the network/ firewall settings.

It worked for me.

All Replies
Community Manager

Re: org.apache.cxf.interceptor.Fault: Could not send Message.

org.apache.cxf.interceptor.Fault: Could not send Message.
This error is usually due to the server being unavailable or the endpoint is not right. I think your case is the server is down for some reason. Does it occurs randomly?

----------------------------------------------------------
Talend | Data Agility for Modern Business
Eight Stars

Re: org.apache.cxf.interceptor.Fault: Could not send Message.

It just happened once when a huge payload was sent. Other than that, I have not seen it go crazy. I am not sure if that big payload is the root cause. Anything else I can scout for? Thanks
Eight Stars

Re: org.apache.cxf.interceptor.Fault: Could not send Message.

Any suggestions? Any settings to tweak and check with the Container? It started again last night.

Six Stars

Re: org.apache.cxf.interceptor.Fault: Could not send Message.

Hi Plumberg,

any news from your side? I am randomly experiencing this error as well on an OData interface in Dynamics 365. Do you think increasing the connection timeout in the tRestClient component could help? From my perspective it is an error, which is tough to handle, as it appears within the flow. I don't see a way to recover from this error and retry to post the record. How did you handle it?

Happy for any input!

Regards.

Tobi

Eight Stars

Re: org.apache.cxf.interceptor.Fault: Could not send Message.

Hi @Tobi,
When I was running my code on local network, it never gave an issue (regardless of the payload size). Howecer, when deployed on production, with all the unknown security settings, we were able to see that the client connection was cutoff after some 10MB of payload or similar. After we got that figured, we just had to open this endpoint for no limits and it's been going great.

Sorry this may not be what you wanna hear, but work a shot to check the network/ firewall settings.

It worked for me.