We use proprietary and third party´s cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
Ask Your Question
0

Make Fiware-Cygnus storing to Apache-Phoenix database

asked 2017-11-06 17:36:33 +0100

whitini gravatar image

I just asked this question on Stackoverflow too.

I'm new to Cygnus and to Phoenix db. I want Cygnus to store data in an Apache Phoenix db, so I have to configure the Phoenix Sink to work with Cygnus.

I know that Cygnus is an extension of Apache Flume, so any Flume agent can work with it by default (theoretically). I configured my Flume agent agent_phoenix.conf file for Cygnus like this:

cygnus-ngsi.channels = memoryChannel
cygnus-ngsi.sinks = phoenix-sink

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = memoryChannel
cygnus-ngsi.sources.http-source.port = 5051
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf

cygnus-ngsi.channels.memoryChannel.type=memory
cygnus-ngsi.channels.memoryChannel.transactionCapacity=100
cygnus-ngsi.channels.memoryChannel.byteCapacityBufferPercentage=20

cygnus-ngsi.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink
cygnus-ngsi.sinks.phoenix-sink.channel=memoryChannel
cygnus-ngsi.sinks.phoenix-sink.batchSize=100
cygnus-ngsi.sinks.phoenix-sink.table=PROVA_PHOENIX
c.sinks.phoenix-sink.ddl=CREATETABLEIFNOTEXISTSPROVA_PHOENIX(uidVARCHARNOTNULL,f_hostVARCHARCONSTRAINTpkPRIMARYKEY(uid),colonnaProvaVARCHAR)
cygnus-ngsi.sinks.phoenix-sink.zookeeperQuorum=localhost
cygnus-ngsi.sinks.phoenix-sink.serializer=REGEX
cygnus-ngsi.sinks.phoenix-sink.serializer.rowkeyType=uuid
cygnus-ngsi.sinks.phoenix-sink.serializer.columns=colonnaProva

and launched it using this command on CentOS command line:

/usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi -Dflume.root.logger=INFO,console

It starts doing some initializations but then I get this message:

76666 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.serializer.BaseEventSerializer  - error No suitable driver found for jdbc:phoenix:localhost; occurred during initializing connection 
176667 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.sink.PhoenixSink  - Error No suitable driver found for jdbc:phoenix:localhost; in initializing the serializer.
176667 [lifecycleSupervisor-1-8] ERROR org.apache.flume.lifecycle.LifecycleSupervisor  - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@29918f3 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at com.google.common.base.Throwables.propagate(Throwables.java:156)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:124)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at ...
(more)
edit retag flag offensive close merge delete

1 answer

Sort by » oldest newest most voted
0

answered 2017-12-04 10:30:15 +0100

Hello, I am checking your question and I notice that the error is related to the Phoenix driver. Currently, Cygnus has a native support to this Sinks: Cartodb, ckan, dynamo, hdfs, hive, mongodb, mysql, postgresql, but doesn't have the native driver for Phoenix, that is the reason why you receive the message that Cygnus doesn't found the driver for Phoenix. It means that you can't connect if you don't have the corresponding driver, some insights about how to add the Phoenix driver in Apache flume are posted in https://phoenix.apache.org/flume.html

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-11-06 17:36:33 +0100

Seen: 1,141 times

Last updated: Nov 06 '17