Blog do projektu Open Source JavaHotel

środa, 19 sierpnia 2015

Pro*C to DB2 Embedded SQL migration, structures

Introduction
Embedded SQL (DB2) and Pro*C (Oracle) allow to use C structure as hosts variable. Instead of enumerating all columns in INSERT or SELECT statement the developer can use name of the structure variable. Implementation of structure in DB2 and Oracle is similar but a little different.
Oracle code
Pro*C Oracle sample code

create table personal_data (id integer, name varchar(100));
int addPersonalData(personal_data data) {

  int failed;
  
  EXEC SQL WHENEVER SQLERROR DO sql_error("Insert personal_data",&failed);  
 
  EXEC SQL INSERT INTO personal_data (id,name) 
      VALUES (:data); 
  if (failed) return 0;
  
  EXEC SQL INSERT INTO personal_data (id,name) 
      VALUES (:data.id,NULL); 
      
  if (failed) return 0;
  return 1;       
}  

int calculatePersonalStat(int *all, int *nullno) {

   int failed;
   personal_data data;
   struct
        {
   short ind_id;
   short ind_name;
    } data_info_ind;
  
   EXEC SQL WHENEVER SQLERROR DO sql_error("calculate personal stat",&failed);  

   EXEC SQL DECLARE persdata CURSOR FOR 
          SELECT ID,NAME 
              FROM personal_data;
       
   if (failed) return 0;       
       
   EXEC SQL OPEN persdata;       
   
   if (failed) return 0;
   
   EXEC SQL WHENEVER NOT FOUND DO break;   
   
   *all = *nullno = 0;
   
    for (;;) 
    { 

        EXEC SQL FETCH  persdata INTO :data INDICATOR :data_info_ind;        
        if (failed) return 0;
 (*all)++;
 if (data_info_ind.ind_name != 0) (*nullno)++; 
    } 
    
    EXEC SQL CLOSE persdata;
   
    return 1;
            
}     
In this code we see usage of structure to FETCH and INSERT data into the table and also structure as INDICATOR host variable.
DB2 code
Embedded SQL sample code, equivalent of Oracle code above

EXEC SQL BEGIN DECLARE SECTION;

typedef struct { 
  int id;
  char name[100];
} personal_data_struct_db2 ; 

EXEC SQL END DECLARE SECTION;

int addPersonalData(personal_data data) {

    struct sqlca sqlca;
  
   EXEC SQL BEGIN DECLARE SECTION;
      personal_data_struct_db2 rec;
   EXEC SQL END DECLARE SECTION;
     int error = 0;
     
   memcpy(&rec,&data, sizeof(rec));
 
  EXEC SQL INSERT INTO personal_data (id,name) 
      VALUES (:rec); 

      
  EVALRESULT("Insert personal rec");
  if (error) return 0;
  
  EXEC SQL INSERT INTO personal_data (id,name) 
      VALUES (10,NULL); 

  EVALRESULT("Insert personal rec with null");
  if (error) return 0;
      
  return 1;       
}  

int calculatePersonalStat(int *all, int *nullno) {

   struct sqlca sqlca;
  
   EXEC SQL BEGIN DECLARE SECTION;
      personal_data_struct_db2 data;
      short data_info_ind[2];
   EXEC SQL END DECLARE SECTION;
     int error = 0;
    
  
//   EXEC SQL WHENEVER SQLERROR DO sql_error("calculate personal stat",&failed);  

   EXEC SQL DECLARE persdata CURSOR FOR 
          SELECT ID,NAME 
              FROM personal_data;
   EVALRESULT("Declare cursor");
   if (error) return 0;
       
   EXEC SQL OPEN persdata;       
   EVALRESULT("Open cursor");
   if (error) return 0;
  
   
   
   *all = *nullno = 0;
   
    for (;;) 
    { 
        EXEC SQL FETCH  persdata INTO :data INDICATOR :data_info_ind;        
        EVALRESULT("Fetch next");
        if (error) return 0;
 if (SQLCODE != 0) break;
 (*all)++;
 if (data_info_ind[1] != 0) (*nullno)++; 
    } 
    
    EXEC SQL CLOSE persdata;
   
    return 1;
            
}         
The DB2 code is similar but there are differences:
  • Structure declaration must be enclosed in EXEC SQL BEGIN DECLARE and EXEC SQL END DECLARE SECTION. Unfortunately it requires duplicating of the structure definiton.
  • Also structure variable should be declared in similar way. Pay attention too memcpy invocation.
  • Also implementation of INDICATOR host variable is different. Istead of structure a table should be used
Conclusion
Despite these differences migration is simple and straightforward. The code should be modifed but no changes in application logic are necessary.

niedziela, 9 sierpnia 2015

IBM InfoSphere Streams and power of parallelism

Introduction
Basic method to improve performance in IBM InfoSphere Streams is to apply parallelism. Unlike any other programming framework the parallelism in Streams can be achieved using very simple method and the result is amazing.
Application
To prove it I created a very simple application calculating FFT (Fast Fourier Transformation) on a series of numbers. Choice of FFT is arbitrary, it is only an example of CPU thirsty method without paying any attention to validity of input and output values.
Source code of the application is available here. It produces series of random numbers (SeriesSource.spl), aggregate them in 512 size buffer and apply FFT (FFTPerform.spl) and consume  the result (SinkFFT.spl). The performance is gauged by measuring number of FFT calculations output flowing to SinkFFT.
All tests were conducted on two VMWare machines (4 virtual CPUs and 4 GB memory).
First step
First version does not contain any parallel calculation.

The performance was 23-25 tuples/sec (number of FFT output failing into SinkFFT).
Second step
It is very easy to detect that bottleneck is FFT operator. The simplest way to execute FFT in parallel is apply UDP (User Define Parallelism) to FFT.

  
                    @parallel(width = 5)
                    (stream<list<float64> l> Out0) as FFT_2 =
    FFT(Custom_1_out0)
   {
    param
     algorithm : DCT ;
     inputTimeSeries : l ;
     resolution : 1024u ;
    output
     Out0 : l = magnitude() ;
   }

So we have 5 FFT operators running in parallel and performance is 86-86 tuples/ per second.
Third step
Next step is to distribute calculation between two nodes. So we have to create two instances of FFTPerform operator and put them into separate PE (Processing Element).
namespace application ;

use com.testperf.series::SeriesSource ;
use com.testperf.fft::FFTPerform ;
use com.testperf.sink::SinkFFT ;

composite Main
{
 graph
  (stream<float64 num> SeriesSource_1_out0) as SeriesSource_1 = SeriesSource()
  {
  }

  (stream<list<float64> l> FFTPerform_3_out0) as FFTPerform_3 =
   FFTPerform(SeriesSource_1_out0)
  {
   config
    placement : partitionColocation("CC") ;
  }

  () as SinkFFT_5 = SinkFFT(FFTPerform_3_out0, FFTPerform_6_out0)
  {
  }

  (stream<list<float64> l> FFTPerform_6_out0) as FFTPerform_6 =
   FFTPerform(SeriesSource_1_out0)
  {
   config
    placement : partitionColocation("BB") ;
  }

 config
  placement : partitionColocation("AB") ;
}

We do not have do anything with operator logic, just make a copy of SinkFFT operator in main composite and by using partitionCollocation option put them to separate PE. When application is deployed IBM InfoSphere Streams Scheduler will push them into separate hosts.

Now we have 10 FFT operators running in parallel and distributed evenly between two hosts. The performance is 120-125 tuples per second.
Conclusion

  • The default model for Streams application is maximum parallelism, every operator in a separate PE (process). But is does not make any sense, having random tens or hundreds PE (processes) does not improve performance. So a good starting point is fuse all application into single PE (no parallelism at all) and later introduce parallelism according to design.
  • Firstly we have to decide how to measure performance. In the test application above it was very simple but in case of complex application it requires some effort.
  • Next step is identify bottleneck. In the test application it was obvious but in real application it could require a lot of tests and search.
  • Applying UDP annotation is the a very good method to introduce parallelism. But selecting the optimal parallel level number requires some effort, greater number does not mean better performance. The rule of thumb is that should not exceed the number of cores in the host machine.
  • Parallelism is not limited only to multiplying number of threads conducting a particular task but also spraying execution between different hosts. Identifying which operator or job should be distributed into separate host to achieve maximum performance requires some effort. 

niedziela, 19 lipca 2015

Byliśmy na koncertach

28 czerwca byliśmy na koncercie muzyki organowej w kościele Ewangelicko-Reformowanym przy ul Solidarności, podobało nam się bardzo. Repertuar składał się z utworów kompozytorów XIX i XX wiecznych, zaś utwory dawniejszych mistrzów we współczesnym opracowaniu na organy. Bardzo dobrze brzmiało "Priere" Cesara Francka. Ukłonem w stronę publiczności było organowe wykonanie znanego Poloneza op 40 no 1 Chopina w transkrypcji organowej, gdyby ktoś usłyszał ten utwór pierwszy raz mógłby pomyśleć, że to właśnie organy było instrumentem na który to dzieło zostało pierwotnie napisane. Końcowym akcentem były organowe wariacje na temat znanej "Sarabandy/La Folia" Haendla (HWV 437). Jest ciekawostką, że ta sama "Sarabanda" w wersji orkiestrowej posłużyła jako ilustracja muzyczna w znanym filmie akcji Raid 2, i to jeszcze w jednej z najbardziej ociekających krwią scen (nie klikaj, jeśli nie lubisz takich filmów). Trudno sobie wyobrazić większy kontrast z pięknym i surowym wnętrzem kalwińskiego kościoła, gdzie koncert miał miejsce. Zapewne i samemu wielkiemu Haendlowi nie śniło się, że ten skromny fragment suity na klawesyn po latach znajdzie takie różnorodne zastosowanie.

5 czerwca byliśmy na koncercie orkiestry Sinfonia Viva, podobało nam się także. Tutaj prezentowany był repertuar kompozytorów XX wiecznych.  Oba koncerty miały jeden łączący element, znaną Etiudę d-moll op. 2 nr 3 Karola Szymanowskiego, jedno wykonanie w wersji organowej, a drugie w wersji na orkiestrę smyczkową. Była to jedna z najwcześniejszych kompozycji Szymanowskiego, która bardzo szybko weszła do światowego repertuaru. Tylko ten sukces miał także drugie dno, gdyż trochę przesłonił późniejszą twórczość kompozytora. Jak sam Szymanowski z goryczą zauważył : "Fatalnie jest już w tak młodym wieku skomponować swoją IX Symfonię!”. Ale główną atrakcją było brawurowe wykonanie znanego cyklu pieśni "Iluminacje" Beniamina Brittena w pierwotnym, sopranowym wykonaniu. Koncert dopełniła mniej znana kompozycja Richarda Straussa "Metamorfozy".


sobota, 11 lipca 2015

New version of MVP Jython framework

Introduction
New version has been deployed. Introduces new functionality and enhancements.
Source code and and Google App Engine demo version.
Hide and show columns in list programmatically
Columns can be hidden and revealed again from Jython code. More detailed description is available here.
Demo version: Different lists->Change header name. After clicking "Column 2 visible" check box "Number 2 column" disappears or reappears.

Remember feature
Entering data is very dull task but "remember feature" provides some relief. Just restores value used last time. So far it could be achieved programmatically but from now on it is done automatically by adding "remember" attribute to a field description. Value can be remembered for each field individually or several fields can share the same value. More detailed description is available here.
Demo version : Alerts -> Dialog with remember.

Fields "Remember us" shares the same remembered value.
Suggest feature
"Remember" recalls only one last value but "Suggest" brings back a list of values to choose from. To implement "Suggest feature" GWT SuggestBox widget is used. Suggest list can be static or dynamic. Static gives the same list of predefined values all the time. Dynamic is enhanced automatically, every time user enters new value it is added to the list. In case of overflow the oldest value is removed. List can be stored individually for each field or the same list can be shared between different fields. More detailed description is available here.
Demo version :  Alerts->Dialog with suggest.

"Suggest custom" is using a static suggest list and "Suggest" dynamic
Get data
Data from any list can be downloaded or send to the recipient. Just click: "Tool" -> "Get data".


Several formats are supported: cvs, xml, html, JSon and PDF.

piątek, 22 maja 2015

Pro*C to DB2 Embedded SQL migration, BLOB, large object

Pro*C BLOB
Uploading and downloading files from and to BLOB column is a painstaking process in Oracle Pro*C. More details: download, read BLOB and upload, write BLOB.
Pro*C code example:
create table blobtable (id number(12), filename varchar2(100), fileb blob);
Code sample for reading and writing BLOB column is here.
One has to recognize whether file can be swallowed in a single mouthful (uff !) and issue EXEC SQL LOB WRITE ONE. If not then a complicated logic should be developed to recognize the first chunk (EXEC SQL LOB WRITE FIRST), intermediate chunk(s) (EXEC SQL LOB WRITE NEXT) and the closing chunk (EXEC SQL LOB WRITE LAST). The same for reading a BLOB column. DBMB_LOB utility is only a partial solution because it can read/write files from and to Oracle server.
Embedded SQL (DB2) BLOB
In DB2 (Embedded SQL) it is much simpler. Just use BLOB_FILE host variable for BLOB reading and writing. The whole code looks very simple now. Works also while executed at the client or server side.
int writeBlob(char *filename) { 
   struct sqlca sqlca;
   int error = 0;
  EXEC SQL BEGIN DECLARE SECTION;
    int id;
    char pfilename[500];
    SQL TYPE IS BLOB_FILE FILEB;
  EXEC SQL END DECLARE SECTION;
      
   strcpy(pfilename,filename);

   EXEC SQL select id into :id FROM blobtable WHERE id = 1  ;
   EVALRESULT("Select blob record");
   if (error) return 0;
   
   if (sqlca.sqlcode == 100) {
     // not found
     EXEC SQL INSERT INTO blobtable VALUES(1,:pfilename,NULL);
     EVALRESULT("Insert record to blobtable");
     if (error) return 0;
   }     
   strcpy (FILEB.name, pfilename);
   FILEB.name_length = strlen(pfilename);
   FILEB.file_options = SQL_FILE_READ;
   EXEC SQL UPDATE blobtable SET filename=:pfilename,fileb=:FILEB WHERE ID=1;
   EVALRESULT("Insert blob value");
   if (error) return 0;
   
   return 1;
 
 
}

int readBlob(char *filename) { 
   struct sqlca sqlca;
   int error = 0;
  EXEC SQL BEGIN DECLARE SECTION;
    SQL TYPE IS BLOB_FILE FILEB;
  EXEC SQL END DECLARE SECTION;
      
   
   strcpy (FILEB.name, filename);
   FILEB.name_length = strlen(filename);
   FILEB.file_options = SQL_FILE_OVERWRITE;

   EXEC SQL select fileb into :FILEB FROM blobtable WHERE id = 1  ;
   EVALRESULT("Read blob record");
   if (error) return 0;
   
   printf("sql code=%u\n",sqlca.sqlcode);
   if (sqlca.sqlcode == 100) {
     // not found
     printf("No record in blobtable table\n");
     return 0;
   }     
   
   return 1;
 
  
}

niedziela, 17 maja 2015

New version of MVP Jython framework

Introduction
New version of MVP Jython framework was deployed. New features had been added and several bugs fixed.
Sample application (Google App Engine)
Source code
Change title(tooltip)
More detailed description 
Change column header
More detailed description
Sample application Different lists -> Change column header
After clicking "Change column header" button column header will be modified accordingly.
Changes saving
User can modify some parameters related to list. For instance: number of rows.

So far the changes were active only for a current window. After refreshing standard values were restored. From now on the changes will be persisted in browser cookies and reused.
Modify the list layout
Additional option is implemented : "Change column settings"


After selecting the option it is possible to modify column headers and visibility.

Changes are persistent, new layout is stored in browser cookies. It is possible also to restore standard layout by clicking "Restore default" button.
Important: changing column order is planned in the future.
Setting/unsetting all column
Sample application Different lists -> List check
In "list edit" mode (more details) one can make "bool" column editable for check/uncheck. Option has been added to check/uncheck all list after clicking the column header. Whole list is modified, not only a single row.
Important: Side effect is involved because clicking the header causes also list sorting by boolean values. Will be improved in the future.
Small fixes
  • Footer numerical value is formatted like number in the column (decimal point, spaces between thousands). 
  • Jython 2.7 (final release) is used. Unfortunately, nasty problem was detected while converting date time value to Java Timestamp (source code modifier). Also this version does not work in Google App Engine (java.io package) so the old Jython 2.7-b3 is deployed for this version.
Future
  • List: download as csv, PDF (for printing) and send as an attachment in mail.

piątek, 15 maja 2015

New version of JSPWiki In The Cloud

I deployed new version of Google App Engine JSPWiki port. Caching mechanism was added (by means of Google App Engine MemCache). Performance improves a little bit but it is not a significant change.
Sources are available on GitHub.
I also moved demo version to http://myjspwiki.appspot.com/ because previous location is infested by spam.