fiware - OAuth2 in Cygnus

Question

Since it was announced the access to Cosmos' WebHDFS in FIWARE Lab is protected with OAuth2, Cygnus (version 0.8.1) is not able to persist Orion's context data in Cosmos HDFS:

15/07/09 08:45:21 INFO handlers.OrionRestHandler: Starting transaction (1436424291-649-0000000000)
15/07/09 08:45:21 INFO handlers.OrionRestHandler: Received data ({  "subscriptionId" : "51c0ac9ed714fb3b37d7d5a8",  "originator" : "localhost",  "contextResponses" : [    {      "contextElement" : {        "attributes" : [          {            "name" : "temperature",            "type" : "centigrade",            "value" : "26.5"          }        ],        "type" : "Room",        "isPattern" : "false",        "id" : "Room1"      },      "statusCode" : {        "code" : "200",        "reasonPhrase" : "OK"      }    }  ]})
15/07/09 08:45:21 INFO handlers.OrionRestHandler: Event put in the channel (id=109760451, ttl=10)
15/07/09 08:45:21 INFO sinks.OrionSink: Event got from the channel (id=109760451, headers={timestamp=1436424321536, content-type=application/json, transactionId=1436424291-649-0000000000, fiware-service=def_serv, fiware-servicepath=def_serv_path, ttl=10, destination=room1_room}, bodyLength=460)
15/07/09 08:45:22 INFO sinks.OrionHDFSSink: [hdfs-sink] Persisting data at OrionHDFSSink. HDFS file (def_serv/def_serv_path/room1_room/room1_room.txt), Data ({"recvTime":"2015-07-09T06:45:21.536Z","temperature":"26.5", "temperature_md":[]})
15/07/09 08:45:22 ERROR sinks.OrionSink: Persistence error (The /user/frb/def_serv/def_serv_path/room1_room directory could not be created in HDFS. HttpFS response: 503 Service unavailable)
15/07/09 08:45:22 INFO sinks.OrionSink: An event was put again in the channel (id=109760451, ttl=9)
15/07/09 08:45:22 INFO sinks.OrionSink: Finishing transaction (1436424291-649-0000000000)

I figure out an OAuth2 token has to be configured somewhere, but how?

Answer 1

IMPORTANT: Before implementing any of the commands below, backup your Cygnus configurations since they will be still valid with minor changes.

Sadly, available Cygnus versions have no support for OAuth2. Until now. You will have to upgrade to the latest 0.8.2, which allows to configure a token through this OrionHDFSSink parameter:

<your_agent_nam>.sinks.<your_hdfs_sink_name>.oauth2_token = <token>

Upgrading to Cygnus 0.8.2 is as easy as installing it from the FIWARE repo:

$ (sudo) yum clean all      # just to clean the yum cache
$ (sudo) yum list cygnus    # this will show you 0.82 is available
$ (sudo) sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus # this is needed if you have installed a version < 0.8.0
$ (sudo) yum install cygnus # this installs 0.8.2

Remember the FIWARE repo is setup by editing /etc/yum.repos.d/fiware.repo (most probably you already have it):

[Fiware]
name=FIWARE repository
baseurl=http://repositories.testbed.fi-ware.eu/repo/rpm/x86_64/
gpgcheck=0
enabled=1

In the example below, we are moving from a 0.7.0 version to latest 0.8.2:

$ sudo yum clean all
Loaded plugins: fastestmirror, security
Cleaning repos: ...
Cleaning up Everything
Cleaning up list of fastest mirrors
$ sudo yum list cygnus
Loaded plugins: fastestmirror, security
Determining fastest mirrors
 * base: sunsite.rediris.es
 * epel: ftp.cica.es
 * extras: sunsite.rediris.es
 * updates: sunsite.rediris.es
...    
Installed Packages
cygnus.x86_64               0.7.0-g4cddd26            @Fiware
Available Packages
cygnus.x86_64               0.8.2-0.g7353160          Fiware 
$ sudo rpm -e -vv --allmatches --nodeps --noscripts --notriggers cygnus
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
...
D: closed   db index       /var/lib/rpm/Name
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm
$ sudo yum install cygnus
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: sunsite.rediris.es
 * epel: ftp.cica.es
 * extras: sunsite.rediris.es
 * updates: sunsite.rediris.es
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package cygnus.x86_64 0:0.7.0-g4cddd26 will be updated
---> Package cygnus.x86_64 0:0.8.2-0.g7353160 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================
 Package                            Arch                               Version                                       Repository                          Size
==============================================================================================================================================================
Updating:
 cygnus                             x86_64                             0.8.2-0.g7353160                              Fiware                              94 M

Transaction Summary
==============================================================================================================================================================
Upgrade       1 Package(s)

Total download size: 94 M
Is this ok [y/N]:
Downloading Packages:
cygnus-0.8.2-0.g7353160.x86_64.rpm                                                                                                     |  94 MB     00:03     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : cygnus-0.8.2-0.g7353160.x86_64                                                                                                             1/2 
[INFO] Creating cygnus user
[INFO] Creating log directory
Done
  Cleanup    : cygnus-0.7.0-g4cddd26.x86_64                                                                                                               2/2 
[INFO] Uninstall the cygnus
Stopping Cygnus chatrooms...                      [  OK  ]
[INFO] Deleting links
rm: no se puede borrar ?/etc/cygnus/flume.conf?: No existe el fichero o el directorio
[INFO] Removing application log files
[INFO] Deleting the cygnus folder
[INFO] Deleting the cygnus user
Done

Updated:
  cygnus.x86_64 0:0.8.2-0.g7353160                                                                                                                            

Complete!

In order to check you have the right version, you can do several things. For instance, using the rpm command:

$ rpm -qa | grep cygnus
cygnus-0.8.2-0.g7353160.x86_64

Or using the yum list:

$ (sudo) yum list cygnus
Loaded plugins: fastestmirror, security
Loading mirror speeds from cached hostfile
 * base: sunsite.rediris.es
 * epel: ftp.cica.es
 * extras: sunsite.rediris.es
 * updates: sunsite.rediris.es
Installed Packages
cygnus.x86_64              0.8.2-0.g7353160                @Fiware

Or simply check the oauth2_token parameter is there ;)

$ cat /usr/cygnus/conf/agent.conf.template |grep oauth2_token
cygnusagent.sinks.hdfs-sink.oauth2_token = xxxxxxxx

IMPORTANT: Backup your Cygnus configurations since they will be still valid with minor changes:

• Please observe if you were using a Cygnus version < 0.8.0 the Java packages have changed. Now, they are no more es.tid.fiware.fiwareconnectors.cygnus.etc but com.telefonica.iot.cygus.etc.
• From Cygnus 0.8.1 the configuration file matching_table.conf (CSV-like content) now is named as grouping_rules.conf (JSON-like content with the same semantic than the CSV-like version).