Revision history [back]
 | 1 | initial version |
Persistence error in Cygnus
Hello,
We are experiencing an error in cygnus when trying to persist data to cosmos. We tried to create directories directly with an http request ourselves and that worked fine. I am not sure what kind of information you might need to help, but this is the error message we get in the logs:
14 Dec 2015 17:11:41,405 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.backends.http.HttpBackend.doRequest:195) - Http request: PUT http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1 14 Dec 2015 17:11:41,405 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection:182) - Connection request: [route: {}->http://cosmos.lab.fiware.org:14000][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 500] 14 Dec 2015 17:11:41,406 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection:214) - Connection leased: [id: 7][route: {}->http://cosmos.lab.fiware.org:14000][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 500] 14 Dec 2015 17:11:41,406 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection:175) - Connecting to cosmos.lab.fiware.org:14000 14 Dec 2015 17:11:41,439 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestAddCookies.process:122) - CookieSpec selected: best-match 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestAuthCache.process:75) - Auth cache not set in the context 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestTargetAuthentication.process:78) - Target auth state: UNCHALLENGED 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestProxyAuthentication.process:87) - Proxy auth state: UNCHALLENGED 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.client.DefaultRequestDirector.tryExecute:680) - Attempt 1 to execute request 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:268) - Sending request: PUT /webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "PUT /webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1[\r][\n]" 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "X-Auth-Token: *********************** [\r][\n]" 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "Content-Length: 0[\r][\n]" 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "Host: cosmos.lab.fiware.org:14000[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "Connection: Keep-Alive[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "User-Agent: Apache-HttpClient/4.3.1 (java 1.5)[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:272) - >> PUT /webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> X-Auth-Token: PuVzUcFpNz0SaCUzI5k6QoWjfhB3WL 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> Content-Length: 0 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> Host: cosmos.lab.fiware.org:14000 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> Connection: Keep-Alive 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> User-Agent: Apache-HttpClient/4.3.1 (java 1.5) 14 Dec 2015 17:11:41,928 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:176) - Connection 0.0.0.0:53805<->130.206.80.46:14000 closed 14 Dec 2015 17:11:41,928 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.client.DefaultRequestDirector.tryExecute:686) - Closing the connection. 14 Dec 2015 17:11:41,928 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:176) - Connection 0.0.0.0:53805<->130.206.80.46:14000 closed 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.shutdown:160) - Connection 0.0.0.0:53805<->130.206.80.46:14000 shut down 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:176) - Connection 0.0.0.0:53805<->130.206.80.46:14000 closed 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.PoolingClientConnectionManager.releaseConnection:269) - Connection released: [id: 7][route: {}->http://cosmos.lab.fiware.org:14000][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 500] 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.backends.http.HttpBackend.doRequest:129) - There was a problem when performing the request (details=Persistence error (Connection reset)). Most probably the used Http endpoint is not active, trying another one (host=cosmos.lab.fiware.org) 14 Dec 2015 17:11:41,930 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:301) - [com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImplREST.createDir(HDFSBackendImplREST.java:98), com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation(OrionHDFSSink.java:926), com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistBatch(OrionHDFSSink.java:426), com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:292), org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68), org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147), java.lang.Thread.run(Unknown Source)] 14 Dec 2015 17:11:41,930 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:305) - Persistence error (The /user/developer/parking/occupancy/206_parkingsection directory could not be created in HDFS. Server response: 503 Service unavailable)
Many Thanks for your kind help in advance
| | 2 | No.2 Revision |
Persistence error in Cygnus
Hello,
We are experiencing an error in cygnus when trying to persist data to cosmos. We tried to create directories directly with an http request ourselves and that worked fine. I am not sure what kind of information you might need to help, but this is the error message we get in the logs:
14 Dec 2015 17:11:41,405 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.backends.http.HttpBackend.doRequest:195) - Http request: PUT http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1 14 Dec 2015 17:11:41,405 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection:182) - Connection request: [route: {}->http://cosmos.lab.fiware.org:14000][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 500] 14 Dec 2015 17:11:41,406 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection:214) - Connection leased: [id: 7][route: {}->http://cosmos.lab.fiware.org:14000][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 500] 14 Dec 2015 17:11:41,406 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection:175) - Connecting to cosmos.lab.fiware.org:14000 14 Dec 2015 17:11:41,439 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestAddCookies.process:122) - CookieSpec selected: best-match 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestAuthCache.process:75) - Auth cache not set in the context 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestTargetAuthentication.process:78) - Target auth state: UNCHALLENGED 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.client.protocol.RequestProxyAuthentication.process:87) - Proxy auth state: UNCHALLENGED 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.client.DefaultRequestDirector.tryExecute:680) - Attempt 1 to execute request 14 Dec 2015 17:11:41,440 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:268) - Sending request: PUT /webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "PUT /webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1[\r][\n]" 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "X-Auth-Token: *********************** [\r][\n]" 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "Content-Length: 0[\r][\n]" 14 Dec 2015 17:11:41,441 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "Host: cosmos.lab.fiware.org:14000[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "Connection: Keep-Alive[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "User-Agent: Apache-HttpClient/4.3.1 (java 1.5)[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.Wire.wire:72) - >> "[\r][\n]" 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:272) - >> PUT /webhdfs/v1/user/developer/parking/occupancy/206parkingsection?op=mkdirs&user.name=developer HTTP/1.1 14 Dec 2015 17:11:41,442 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> X-Auth-Token: PuVzUcFpNz0SaCUzI5k6QoWjfhB3WL 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> Content-Length: 0 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> Host: cosmos.lab.fiware.org:14000 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> Connection: Keep-Alive 14 Dec 2015 17:11:41,443 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader:275) - >> User-Agent: Apache-HttpClient/4.3.1 (java 1.5) 14 Dec 2015 17:11:41,928 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:176) - Connection 0.0.0.0:53805<->130.206.80.46:14000 closed 14 Dec 2015 17:11:41,928 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.client.DefaultRequestDirector.tryExecute:686) - Closing the connection. 14 Dec 2015 17:11:41,928 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:176) - Connection 0.0.0.0:53805<->130.206.80.46:14000 closed 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.shutdown:160) - Connection 0.0.0.0:53805<->130.206.80.46:14000 shut down 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.DefaultClientConnection.close:176) - Connection 0.0.0.0:53805<->130.206.80.46:14000 closed 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.http.impl.conn.PoolingClientConnectionManager.releaseConnection:269) - Connection released: [id: 7][route: {}->http://cosmos.lab.fiware.org:14000][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 500] 14 Dec 2015 17:11:41,929 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.backends.http.HttpBackend.doRequest:129) - There was a problem when performing the request (details=Persistence error (Connection reset)). Most probably the used Http endpoint is not active, trying another one (host=cosmos.lab.fiware.org) 14 Dec 2015 17:11:41,930 DEBUG [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:301) - [com.telefonica.iot.cygnus.backends.hdfs.HDFSBackendImplREST.createDir(HDFSBackendImplREST.java:98), com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistAggregation(OrionHDFSSink.java:926), com.telefonica.iot.cygnus.sinks.OrionHDFSSink.persistBatch(OrionHDFSSink.java:426), com.telefonica.iot.cygnus.sinks.OrionSink.process(OrionSink.java:292), org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68), org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147), java.lang.Thread.run(Unknown Source)] 14 Dec 2015 17:11:41,930 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:305) - Persistence error (The /user/developer/parking/occupancy/206_parkingsection directory could not be created in HDFS. Server response: 503 Service unavailable)
Many Thanks for your kind help in advance
Cygnus verison:
Installed Packages Name : cygnus Arch : x86_64 Version : 0.11.0 Release : 1.g2a9c87f Size : 133 M Repo : installed From repo : testbed-fi-ware Summary : Package for cygnus component License : AGPLv3 Description : This connector is a (conceptual) derivative work of ngsi2cosmos, and implements : a Flume-based connector for context data coming from Orion Context Broker.
running on CentOS release 6.6 (Final)
