Blog do projektu Open Source JavaHotel

sobota, 7 grudnia 2013

IBM InfoSphere Streams, NetezzaLoad, ODBCAppend and input control operator

In the previous post it was demonstrated how to connect to Netezza using connection data in the connections.xml file. But this method in not always convenient. For instance: assume we use different credentials in the test environment and production environment. Having credentials hardcoded in the connections.xml enforces recompiling our application every time it is deployed.
Happily there is a solution to this problem. It is possible also to send credentials during submit time and even to change them dynamically during a run time. It can be accomplished by applying "control" input stream to ODBCAppend and NetezzaLoad. Description (click on Operator Control Input Port). Unfortunately, there is no sample for this option and it is not obvious at the first glance how to use it.
Sample code
A sample code for connection using control input port is available here.


  • The producer of connection data is Custom operator submitting credentials to both operators (ODBCAppend and NetezzaLoad). Although the credentials are hardcoded here it is not a problem to get them from somewhere, for instance a configuration file.
  • ODBCAppend and NetezzaLoad have a connectionPassword with invalid password hardcoded. It is on purpose to be sure that during a connection a password got from control port is used.
  • ODBCAppend and NetezzaLoad have parameter connectionPolicy set to Deferred (default is Immediate). Otherwise both operators would try to initialize connection at the beginning and fail. By means of Deferred value the connection is not started until first tuple arrives.
  • It is also necessary to postpone tuples arrival to ODBCAppend and NetezzaLoad until connection credentials are sent. Switch control serves as a gate between flow of tuples and both data operators. Switch operator having parameter start set to true serves as a gate latch. 
  • Custom operator sends credentials to ODBCAppend and NetezzaLoad and at the same time sends signal to Switch operator. The gate is broken and data can flow through not interrupted.
How it is visualized in the diagram above
  1. Beacon operator starts sending tuples. Tuples are sent (and blocked at the very beginning) to the FlowOfControl (Switch) operator.
  2. ControlPort (Custom) operator sends connection data to the ToODBC and NetezzaLoad operator. 
  3. After sending the connection data sends a signal to FlowOfControl (Switch) operator to open a gate.
  4. At the arrival of first tuple the connection is started using (valid) credentials from control input port (not from connections.xml file or invalid connectionPassword parameter).
  5. Then the constant flow of tuples from producer (Beacon) to consumers (ODBAppend and NetezzaLoad) is running. Tuples are passing through FlowOfControl gate without any friction.

Brak komentarzy:

Prześlij komentarz