We use proprietary and third party´s cookies to improve your experience and our services, identifying your Internet Browsing preferences on our website; develop analytic activities and display advertising based on your preferences. If you keep browsing, you accept its use. You can get more information on our Cookie Policy
Cookies Policy
Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Make Fiware-Cygnus storing to Apache-Phoenix database

I just asked this question on Stackoverflow too.

I'm new to Cygnus and to Phoenix db. I want Cygnus to store data in an Apache Phoenix db, so I have to configure the Phoenix Sink to work with Cygnus.

I know that Cygnus is an extension of Apache Flume, so any Flume agent can work with it by default (theoretically). I configured my Flume agent agent_phoenix.conf file for Cygnus like this:

cygnus-ngsi.channels = memoryChannel
cygnus-ngsi.sinks = phoenix-sink

cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = memoryChannel
cygnus-ngsi.sources.http-source.port = 5051
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notification_target = /notify
cygnus-ngsi.sources.http-source.handler.default_service = default
cygnus-ngsi.sources.http-source.handler.default_service_path = /
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.grouping_rules_conf_file = /opt/apache-flume/conf/grouping_rules.conf

cygnus-ngsi.channels.memoryChannel.type=memory
cygnus-ngsi.channels.memoryChannel.transactionCapacity=100
cygnus-ngsi.channels.memoryChannel.byteCapacityBufferPercentage=20

cygnus-ngsi.sinks.phoenix-sink.type=org.apache.phoenix.flume.sink.PhoenixSink
cygnus-ngsi.sinks.phoenix-sink.channel=memoryChannel
cygnus-ngsi.sinks.phoenix-sink.batchSize=100
cygnus-ngsi.sinks.phoenix-sink.table=PROVA_PHOENIX
c.sinks.phoenix-sink.ddl=CREATETABLEIFNOTEXISTSPROVA_PHOENIX(uidVARCHARNOTNULL,f_hostVARCHARCONSTRAINTpkPRIMARYKEY(uid),colonnaProvaVARCHAR)
cygnus-ngsi.sinks.phoenix-sink.zookeeperQuorum=localhost
cygnus-ngsi.sinks.phoenix-sink.serializer=REGEX
cygnus-ngsi.sinks.phoenix-sink.serializer.rowkeyType=uuid
cygnus-ngsi.sinks.phoenix-sink.serializer.columns=colonnaProva

and launched it using this command on CentOS command line:

/usr/cygnus/bin/cygnus-flume-ng agent --conf /usr/cygnus/conf/ -f /usr/cygnus/conf/agent_phoenix.conf -n cygnus-ngsi -Dflume.root.logger=INFO,console

It starts doing some initializations but then I get this message:

76666 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.serializer.BaseEventSerializer  - error No suitable driver found for jdbc:phoenix:localhost; occurred during initializing connection 
176667 [lifecycleSupervisor-1-8] ERROR org.apache.phoenix.flume.sink.PhoenixSink  - Error No suitable driver found for jdbc:phoenix:localhost; in initializing the serializer.
176667 [lifecycleSupervisor-1-8] ERROR org.apache.flume.lifecycle.LifecycleSupervisor  - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@29918f3 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at com.google.common.base.Throwables.propagate(Throwables.java:156)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:124)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost;
at java.sql.DriverManager.getConnection(DriverManager.java:596)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.phoenix.flume.serializer.BaseEventSerializer.initialize(BaseEventSerializer.java:140)
at org.apache.phoenix.flume.sink.PhoenixSink.start(PhoenixSink.java:119)
... 10 more

HBase and Phoenix are installed correctly on this machine, infact HBase can start and Phoenix has been tested with SQLline file. Do you have any solutions to this problem?