We use proprietary and third party´s cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
Ask Your Question
1

Cygnus to CKAN data store

asked 2016-01-27 07:17:14 +0100

tivanova gravatar image

I'm trying to configure Cygnus to store all entities coming from ORION to CKAN. So I have: Cygnus 0.11 installed, ORION 0.26, and CKAN 2.4.1 installed in my environment. I made a subscription in Orion for Cygnus. But receive the following errors in the Cygnus-log:

25 Jan 2016 18:35:24,472 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:224) - Runtime error (Don't know how to treat the response code. Possibly the package already exists in a deleted state (respCode=403, pkgName=some_tests22))

I have tried with different pkgName, but the same result.

Any idea?

edit retag flag offensive close merge delete

Comments

I will need to know your Cygnus configuration. Thanks!

frb gravatar imagefrb ( 2016-01-28 02:49:37 +0100 )edit

2 answers

Sort by » oldest newest most voted
0

answered 2016-01-28 07:10:42 +0100

tivanova gravatar image

updated 2016-01-28 07:11:32 +0100

I have changed the API aouthentification and have some more results ... but again with similar error.

28 Jan 2016 12:53:33,450 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (com.telefonica.iot.cygnus.sinks.OrionSink.process:224) - Runtime error (Don't know how to treat the response code. Possibly the datastore already exists (respCode=409, resourceId=8dbaf6d9-7d56-4631-b089-910891a711cb))

It starts to create resourses in CKAN, but crashes after the first insert. I made tests with row and column persistence, but with the same result.

Here is my config file:

===================================================================

cygnusagent.sources = http-source cygnusagent.sinks = ckan-sink cygnusagent.channels = ckan-channel

cygnusagent.sources.http-source.channels = ckan-channel cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource cygnusagent.sources.http-source.port = 5050 cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler cygnusagent.sources.http-source.handler.notificationtarget = /ckanlight cygnusagent.sources.http-source.handler.defaultservice = MyOrg cygnusagent.sources.http-source.handler.defaultservicepath = MyPath cygnusagent.sources.http-source.handler.eventsttl = 10 cygnusagent.sources.http-source.interceptors = ts gi cygnusagent.sources.http-source.interceptors.ts.type = timestamp cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder cygnusagent.sources.http-source.interceptors.gi.groupingrulesconffile = /usr/cygnus/conf/groupingrules.conf

cygnusagent.sinks.ckan-sink.channel = ckan-channel cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink cygnusagent.sinks.ckan-sink.apikey = 68608bf6-e1e0-4a1e-b117-c18d433ecdd7 cygnusagent.sinks.ckan-sink.ckanhost = 10.10.1.71 cygnusagent.sinks.ckan-sink.ckanport = 80 cygnusagent.sinks.ckan-sink.orionurl = http://10.10.1.27:1026 cygnusagent.sinks.ckan-sink.attr_persistence = column cygnusagent.sinks.ckan-sink.ssl = false

cygnusagent.channels.ckan-channel.type = memory cygnusagent.channels.ckan-channel.capacity = 10000 cygnusagent.channels.ckan-channel.transactionCapacity = 10000

edit flag offensive delete link more
0

answered 2016-01-28 09:02:13 +0100

frb gravatar image

updated 2016-02-01 00:26:08 +0100

I've seen you are using attr_persistence=column

Did you provisioned in advance the organization, package, resource and datastore structure? Please, have a look on this piece of documentation.

edit flag offensive delete link more

Comments

I created organization and dataset previously... nothing more. I thought that cygnus makes the rest of objects. Btw I make tests with row-persistense, but can not see the difference. Every time I make a suscribtion - I see a new resource created in the CKAN without any data inside.

tivanova gravatar imagetivanova ( 2016-01-28 09:36:25 +0100 )edit

I've edited the link, I think it was not working. Anyway, when working in column mode, all the CKAN data structure (organization, dataset, resource, datastore and viewer) must be provisioned in advance. Only when working in row mode, everything is automatically created by Cygnus.

frb gravatar imagefrb ( 2016-02-01 00:28:07 +0100 )edit

Regarding the row mode, could you post a full log? Please, use a service like pastrebin or similar.

frb gravatar imagefrb ( 2016-02-01 00:29:17 +0100 )edit

OK. Here is the flume.log: http://pastebin.com/raw/YqTkArUs Each time I start the cygnus - it aggregates some data to previously created tables, creates new table for the next entityId and crashes. In column mode: how should I create the CKANstructure in order to have all entities in the same table?

tivanova gravatar imagetivanova ( 2016-02-01 04:28:01 +0100 )edit

Hi all, I'm using the attr_persistence=column attribute, and I would like to know how to create the CKAN data structure for the vehicles example. Thank you in advance.

pasquy73 gravatar imagepasquy73 ( 2016-03-17 10:54:32 +0100 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2016-01-27 07:17:14 +0100

Seen: 2,990 times

Last updated: Feb 01 '16