How can I know if Cygnus is connected to Cosmos correctly?

answered 2016-03-29 10:01:39 +0200

frb
191 ●1 ●2 ●4

You will have to use WebHDFS in order to browse your HDFS user space. Detailed referece for WebHDFS can be found at https://hadoop.apache.org/docs/curren....

For instance, by using my HDFS user "frb" I can do:

$ curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/frb?op=liststatus&user.name=frb" -H "X-Auth-Token: <MY_TOKEN>" | python -m json.tool { "FileStatuses": { "FileStatus": [ { "accessTime": 0, "blockSize": 0, "group": "frb", "length": 0, "modificationTime": 1455191589436, "owner": "frb", "pathSuffix": "-p", "permission": "755", "replication": 0, "type": "DIRECTORY" }, { "accessTime": 0, "blockSize": 0, "group": "frb", "length": 0, "modificationTime": 1456152801883, "owner": "frb", "pathSuffix": "FRB", "permission": "740", "replication": 0, "type": "DIRECTORY" },...

Cygnus stores the data as:

hdfs:///user/<myuser>/<fiware-service>/<fiware-service-path>/<entityId>_<entityType>/<entityId>_<entityType>.txt

You can check the Cygnus logs as well in order to know if the data has been successfully persisted. Try finding logs like:

time=2016-03-29T09:55:50.779CEST | lvl=INFO | trans=1459238120-153-0000000000 | svc=default | subsvc=/ | function=persistAggregation | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.OrionHDFSSink[954] : [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (myservice/myservicepath/entityId_entityType/entityId_entityType.txt), Data ({"recvTime":"2016-03-29T07:55:24.441Z","fiwareServicePath":"myservicepath","entityId":"xxx","entityType":"xxx", "xxx":"111", "xxx_md":[]})