Hi All,

I am trying to setup and understand Cygnus. But I am facing issue during installation.

I followed below given steps.

1. Install Cygnus using Docker (docker run -d -p 5050:5050 -p 8081:8081 fiware/cygnus-common)
2. Executed version command (curl http://172.17.0.2:8081/v1/version) which gave following response {"success":"true","version":"1.8.0_SNAPSHOT.39b2aa4789c61fa92fe6edc905410f1ddeb33490"}
3. Login into Cygnus container using command docker exec -it /bin/bash
4. Created new file named “agent_mysql.conf” in “/opt/apache-flume/conf/” folder.

Configuration details given below

cygnus-ngsi.sources = http-source
cygnus-ngsi.sinks = mysql-sink
cygnus-ngsi.channels = mysql-channel

cygnus-ngsi.sources.http-source.channels = mysql-channel
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.port = 5050
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler.notificationtarget = /notify
cygnus-ngsi.sources.http-source.handler.defaultservice = defserv
cygnus-ngsi.sources.http-source.handler.defaultservicepath = defservpath
cygnus-ngsi.sources.http-source.handler.eventsttl = 2
cygnus-ngsi.sources.http-source.interceptors = ts gi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.NGSIGroupingInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.gi.groupingrulesconffile = /Applications/apache-flume-1.4.0-bin/conf/grouping_rules.conf

cygnus-ngsi.channels.mysql-channel.type = memory

cygnus-ngsi.channels.mysql-channel.capacity = 1000

cygnus-ngsi.channels.mysql-channel.transactionCapacity = 100


cygnus-ngsi.sinks.mysql-sink.channel = mysql-channel
cygnus-ngsi.sinks.mysql-sink.type = com.telefonica.iot.cygnus.sinks.NGSIMySQLSink
cygnus-ngsi.sinks.mysql-sink.mysqlhost = localhost
cygnus-ngsi.sinks.mysql-sink.mysqlport = 3306
cygnus-ngsi.sinks.mysql-sink.mysqlusername = root
cygnus-ngsi.sinks.mysql-sink.mysqlpassword = <mypassword>
cygnus-ngsi.sinks.mysql-sink.attr_persistence = row

1. Changed "cygnus-entrypoint.sh" file in / (root) folder and added following command by removing existing one.
   ${FLUMEHOME}/bin/cygnus-flume-ng agent --conf ${CYGNUSCONFPATH} -f ${CYGNUSCONFPATH}/agentmysql.conf -n cygnus-ngsi -p ${CYGNUSAPIPORT} -Dflume.root.logger=${CYGNUSLOGLEVEL},${CYGNUSLOGAPPENDER} -Dfile.encoding=UTF-8
2. Exited Docker container and came back to Ubuntu.
3. Stop and restart Docker container.
4. And I am getting following errors in logs

Please check and let me know what am I doing wrong? Appreciate your help.

LOGS

n$AgentConfiguration[1016] : Processing:mysql-sink time=2018-04-30T14:24:00.807Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=validateConfiguration | msg=org.apache.flume.conf.FlumeConfiguration[140] : Post-validation flume configuration contains configuration for agents: [cygnus-ngsi] time=2018-04-30T14:24:00.808Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[150] : Creating channels time=2018-04-30T14:24:00.816Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=create | msg=org.apache.flume.channel.DefaultChannelFactory[40] : Creating instance of channel mysql-channel type memory time=2018-04-30T14:24:00.825Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=loadChannels | msg=org.apache.flume.node.AbstractConfigurationProvider[205] : Created channel mysql-channel time=2018-04-30T14:24:00.832Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=create | msg=org.apache.flume.source.DefaultSourceFactory[39] : Creating instance of source http-source, type org.apache.flume.source.http.HTTPSource time=2018-04-30T14:24:00.836Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=configure | msg=org.apache.flume.source.http.HTTPSource[113] : Error while configuring HTTPSource. Exception follows. java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.NGSIRestHandler at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:102) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) time=2018-04-30T14:24:00.840Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=loadSources | msg=org.apache.flume.node.AbstractConfigurationProvider[366] : Source http-source has been removed due to an error during configuration java.lang.RuntimeException: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.NGSIRestHandler at com.google.common.base.Throwables.propagate(Throwables.java:156) at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:114) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.handlers.NGSIRestHandler at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.flume.source.http.HTTPSource.configure(HTTPSource.java:102) ... 11 more time=2018-04-30T14:24:00.841Z | lvl=INFO | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=create | msg=org.apache.flume.sink.DefaultSinkFactory[40] : Creating instance of sink: mysql-sink, type: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink time=2018-04-30T14:24:00.842Z | lvl=ERROR | corr=N/A | trans=N/A | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=run | msg=org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable[142] : Failed to load configuration data. Exception follows. org.apache.flume.FlumeException: Unable to load sink type: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink, class: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:69) at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: com.telefonica.iot.cygnus.sinks.NGSIMySQLSink at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67) ... 11 more

Additional Queries

Also, it would be better if you could provide step by step guide for using Cygnus considering any example.
Actually, we have historical data and would like to show different reports using different criteria/filters (Date wise, User based, Area based etc.).
I am planning to use Cygnus with MySQL agent to store historical data. Then create new REST APIs to fetch data from MySQL based on filters and return the JSON for reports to GUI application (May be WireCloud or custom application based on AngularJS).

Is this approach correct? Please suggest.

Can we filter out data using Cygnus while fetching? I mean does Cygnus provide fetch APIs for stored historical data? Or will Analytics related GE fit with above requirement?

Kindly guide.

Do let me know in case you need further information.


Regards,
Krishan