Król Roger, Teatr Narodowy

11 grudnia byliśmy na przedstawieniu Króla Rogera w Teatrze Narodowym. Było to jednak przedstawienie do oglądania, nie do słuchania.
Król Roger II to postać historyczna, słynący z mądrości i pobożności średniowieczny król Sycylii. W operze Szymanowskiego w państwie króla pojawia się tajemniczy Pasterz, prorok nowego boga, o którym śpiewa "Mój Bóg jest piękny jako ja". Przybycie Pasterza na dwór króla wyzwala żywioły, które niczym wir mielą i przemieniają wszystko, dwór, królestwo i Roksanę, ukochaną żonę króla, zostawiając Rogera jako samotnego pielgrzyma.
Sama opera nie jest łatwa w odbiorze, to nie jest historia, którą trzeba opowiedzieć od początku do końca. To ciąg trzech scen, w pierwszej widzimy królestwo Rogera, harmonijne połączenie władzy świeckiej i kościelnej, w drugiej jest konfrontacja Rogera i Pasterza, w trzeciej Roger jako odarty z atrybutów królewskich pielgrzym wędruje w przemienionym mocą Pasterza świecie szukając Roksany.
Operę można odczytać na wiele sposobów. Realizacja w Teatrze Wielkim oczywiście ucieka jak najdalej od średniowiecznego kontekstu. Dwór królewski to zarząd korporacji z Rogerem jako CEO. Pośrodku jest ogromny stół konferencyjny, Roksana to zblazowana dama w wyższych sfer. Mamy szklane klatki, jest łazienka z marmurową umywalką, wózek inwalidzki, migające stroboskopowe światła, lustra, krwawe sceny w estetyce filmów gore, maski z rogami muflona, zwielokrotnione postacie, wizualizacje video, ściana mieniących się cyferek jak z Matrixa. Końcowa scena odrealnionego świata jest odwołaniem do finału "2001 Odyseja Kosmiczna", pojawia się nawet małe dziecko niczym embrion Bowmana w słynnym filmie Kubricka. Aby się przemienić i zrozumieć, musimy narodzić się na nowo.
Niestety, w tym wszystkim zagubiła się sama muzyka, największa wartość opery. Tak jak w końcowej scenie 2 aktu, gdzie z podwieszonego głową do dołu Rogera ścieka krew na leżącą na stole Roksanę, złożony został tutaj w ofierze sam Szymanowski i wszystko zostało postawione na odwrót, ogon zamerdał psem, forma przerosła treść. Wprowadzająca wizualizacja wideo, gdzie pełzający wąż symbolizuje zgniliznę toczącą pozornie perfekcyjne królestwo/korporację, w rzeczywistości zapowiada klęskę samego spektaklu.
Muzyka jest tutaj wyłącznie sound-trackiem do efektownego widowiska, jakby nie miała odwagi unieść się z kanału orkiestrowego. Początkowy chór "Hagios! Hagios!" jest odtwarzany z taśmy, gdy na końcu przedstawienia do oklasków wybiega chór, na widowni czuć konsternację skąd te osoby się tutaj wzięły. Nawet zapowiedź, że wykonawca tytułowej roli, Łukasz Goliński, nie jest w pełni dysponowany, dla spektaklu nie miała wielkiego znaczenia, gdyż forma solistów nie była tutaj istotna, w zasadzie całą warstwę muzyczną łącznie z orkiestrą można było odtworzyć z taśmy.
Miała być opera, a było wyłącznie widowisko światło i dźwięk z akcentem na pierwszy człon.

TPC-DS benchmark

Inspiration
TCP-DS benchmark is an industry standard tool to evaluate the performance of the relational database engine. The tool can be used not only to leverage the general efficiency. The database administrator or DevOp engineer can utilize the tools to estimate the robustness of a particular database installation by comparing it against a reference instance. Also, it can be useful to verify the database health after upgrade or maintenance. Another use case is to gauge the performance gain after tunning or scaling.
The main disadvantage of the standard package is that it is not easy to use, requires a number of manual adjustment and fixes. It makes the tool less reliable and the process less repeatable. To compare the result against the reference results we have to be sure that the benchmark for both environments, the reference, and the current, are prepared and executed using the same queries and according to the same pattern.
Solution
I created a simple solution to make the task very simple. The project is available in GitHub.
The following databases are supported so far: Oracle, IBM DB2, MySQL/MariaDB, PostgreSQL, Hive, SparkSQL, Netezza, and IBM BigSQL.
Also, a wiki is available having hints on how to set up a test for a particular database. Having the tool, the TCP-DS benchmark is ready to use. Just download, configure for a particular environment and run.
So far, Qualify and Power Tests are implemented.
Queries
The TPC-DS queries are not ready to use out of the box for all SQL engines. The specification allows some minor fixes to make them executable for a particular database. To avoid keeping a different version of queries, I decided to modify them on the fly. I also apply only changes possible to implement through the simple string or regular expression text replacement.
Test Validation
I was unable to match the answer data set provided against any output. It requires further investigation. So I decided to use Oracle output as a reference result set. Unfortunately, even Oracle result does match all of them and is not fully reliable. The life is never an easy road free of stone.
Next steps
Throughput and Data Maintenance tests.

BigSQL monitoring

https://www.ibm.com/support/knowledgecenter/en/SSCRJT_5.0.3/com.ibm.swg.im.bigsql.doc/doc/admin_monitor-bigsql-query.html

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.admin.wlm.doc/doc/c0055265.html

GitHub project

Inspiration
BigSQL is nothing more than DB2 running on the top of Hadoop. It also inherits a lot of goodies from DB2 including a sophisticated rigging of metrics. But the metrics are only numbers and by itself do not provide anything meaningful unless one is deeply versed in DB2 internals.
Metrics are cumulative and are growing constantly. Instead of looking at pure values much more interesting is observing how the values are changing over time and trying to discover some patterns or trends. For instance, if the metric suddenly starts to surge it can indicate that the BigSQL is under a heavy workload.
Also, it could be interesting to use the metrics for prediction, for instance, that there is a risk of delays or prolonged response if an adequate pattern is found.
Solution
I developed a simple solution, it can be downloaded here.
The purpose of the tool is to retain historical data and to get easy access to the difference between consecutive metric values for analysis.
The solution contains the following elements:

• Database schema to keep historical data. The metrics are pivoted, instead of a single row of metrics it breaks down the row into a series of records: metric id/value
• The supporting view providing a difference between consecutive metric values instead of the pure value.
• Several stored procedures to harvest and extract collected metrics.
• Two methods of metrics collecting are available: as Linux crontab job or as DB2 scheduled task.

Simple analysis

I also developed a simple tool for heavy workload prediction. Although in DB2 there are several hundred different metrics, only subset of them seems to be related to workload specific for BigSQL. The idea is to nail down a period of normal workload and to monitor the current period. If the average of several metrics being observed exceeds significantly the average of the normal period, the alarm is raised that heavy workload is underway.

But during my testing, the solution does not seem to be usable. More tuning is necessary or a different approach should be built which has teeth.

Polymer 3, upgrade to 3.01

I was blocked by a strange problem after upgrading from 3.0.0-pre.19 to 3.0.1. Suddenly  I found an extraordinarily long delay before displaying the game board for the first time.

Usually, it took several seconds to move from the first screen to the third. After the upgrade, the time was prolonged to almost 50 seconds making it almost useless. It is not easy to find a bottleneck in an asynchronous framework. So firstly I discovered it is the map (12 * 8 tiles) which is the cause of the delay. Then I gritted my teeth and after several sleepless nights including meticulously comparing old and new JS files, I pinned down the culprit.
The problem was caused by iron-resizable-behavior/iron-resizable-behavior.js file. Next to the end, there is a slightly new code.

_requestResizeNotifications: function () {
    if (!this.isAttached) {
      return;
    }

    if (document.readyState === 'loading') {
      var _requestResizeNotifications = this._requestResizeNotifications.bind(this);

      document.addEventListener('readystatechange', function readystatechanged() {
        document.removeEventListener('readystatechange', readystatechanged);

        _requestResizeNotifications();
      });
    } else {
      this._findParent();

      if (!this._parentResizable) {
        // If this resizable is an orphan, tell other orphans to try to find
        // their parent again, in case it's this resizable.
        ORPHANS.forEach(function (orphan) {
          if (orphan !== this) {
            orphan._findParent();
          }
        }, this);
        window.addEventListener('resize', this._boundNotifyResize);
        this.notifyResize();
      } else {
        // If this resizable has a parent, tell other child resizables of
        // that parent to try finding their parent again, in case it's this
        // resizable.
//        this._parentResizable._interestedResizables.forEach(function (resizable) {
//          if (resizable !== this) {
//            resizable._findParent();
//          }
//        }, this);
      }
    }
  },
  _findParent: function () {
    this.assignParentResizable(null);
    this.fire('iron-request-resize-notifications', null, {
      node: this,
      bubbles: true,
      cancelable: true
    });

    if (!this._parentResizable) {
      ORPHANS.add(this);
    } else {
      ORPHANS.delete(this);
    }
  }
};

The temporary workaround is to comment out the code.

        // If this resizable has a parent, tell other child resizables of
        // that parent to try finding their parent again, in case it's this
        // resizable.
//        this._parentResizable._interestedResizables.forEach(function (resizable) {
//          if (resizable !== this) {
//            resizable._findParent();
//          }
//        }, this);
      }

Unfortunately, I'm unable to provide any explanation for that and the solution is nothing more than kicking the can down the road. Obviously, the code inside the else clause is doing something CPU thirsty but that is all I can make out of it. But it works for me.

BigSQL, joins and partition elimination.

Partition elimination in joins
One of the advantages of BigSQL over Hive is enabling partition elimination also during join execution. It is described in this article.
It can be explained using a simple example.

db2 "CREATE HADOOP TABLE p_x(n VARCHAR(64)) PARTITIONED BY (x int) STORED AS PARQUETFILE"
db2 "CREATE HADOOP TABLE p_x(n VARCHAR(64)) PARTITIONED BY (x int) STORED AS ORC"
db2 "insert into p_x values('a',0)"
db2 "insert into p_x values('b',1)"
db2 "insert into p_x values('c',2)"
db2 "insert into p_x values('d',3)"
db2 "insert into p_x values('e',4)"
db2 "insert into p_x values('f',5)"
db2 "insert into p_x values('g',5)"
db2 "insert into p_x values('h',6)"
db2 "insert into p_x values('i',6)"
db2 "insert into p_x values('j',7)"
db2 "insert into p_x values('k',8)"
db2 "insert into p_x values('l',9)"
db2 "CREATE HADOOP TABLE p_y(n VARCHAR(64),x int)"
db2 "insert into p_y values('e',4)"
db2 "insert into p_y values('f',5)"
db2 "insert into p_y values('g',5)"

Then run a straightforward join query.

db2 "select * from p_x,p_y where p_x.x = p_y.x"

The range of values in the p_y table is 4 to 5 so in order to resolve this join, it is enough to scan only corresponding partitions in p_x table.
It can be detected by looking into BigSQL log. But firstly DEBUG level for BigSQL Scheduler should be turned on.

BigSQL->Configs->Advanced bigsql-log4j The following two parameters should be modified:

• log4j.logger.com.ibm.biginsights.bigsql.scheduler.GlobalLog=DEBUG 
• log4j.logger.com.ibm=ALL

Then BigSQL is to be restarted to get new settings taking effect.

The query partition elimination is reported in /var/ibm/bigsql/logs/bigsql-sched.log

DEBUG com.ibm.biginsights.bigsql.scheduler.server.expr.ExprUtils [pool-1-thread-5] : found column with partition-key. db2ColumnIndex: 1
DEBUG com.ibm.biginsights.bigsql.scheduler.server.StorageHandlerScanState [pool-1-thread-5] : [createScanState] Partition elimination expr:  (  ( x <= 5 )  AND  ( x >= 4 )  ) 
DEBUG com.ibm.biginsights.bigsql.scheduler.server.StorageHandlerScanState [pool-1-thread-5] : found default/dummy partition. skipping. HdfsPartition{fileDescriptors=[]}
[eliminatePartition]partition elimination: checking partition: x=0
[eliminatePartition]partitionEliminated? x=0 true
[eliminatePartition]partition elimination: checking partition: x=1
[eliminatePartition]partitionEliminated? x=1 true
[eliminatePartition]partition elimination: checking partition: x=2
[eliminatePartition]partitionEliminated? x=2 true
[eliminatePartition]partition elimination: checking partition: x=3
[eliminatePartition]partitionEliminated? x=3 true
[eliminatePartition]partition elimination: checking partition: x=4
[eliminatePartition]partitionEliminated? x=4 false
The locations are: [TScanRangeLocation(host_idx:0, volume_id:-1), TScanRangeLocation(host_idx:1, volume_id:-1)]
[eliminatePartition]partition elimination: checking partition: x=5
[eliminatePartition]partitionEliminated? x=5 false
[createScanState] The locations are: [TScanRangeLocation(host_idx:2, volume_id:-1), TScanRangeLocation(host_idx:1, volume_id:-1)]
[createScanState] The locations are: [TScanRangeLocation(host_idx:2, volume_id:-1), TScanRangeLocation(host_idx:0, volume_id:-1)]
[eliminatePartition]partition elimination: checking partition: x=6
[eliminatePartition]partitionEliminated? x=6 true
[eliminatePartition]partition elimination: checking partition: x=7
[eliminatePartition]partitionEliminated? x=7 true
[eliminatePartition]partition elimination: checking partition: x=8
[eliminatePartition]partitionEliminated? x=8 true
[eliminatePartition]partition elimination: checking partition: x=9
[eliminatePartition]partitionEliminated? x=9 true
[createScanState]Finished partition elimination. partition-elimination-stats (eliminated/total): 8 / 10 Took 124 milliseconds
2018-10-30 00:51:47,775 INFO com.ibm.biginsights.bigsql.scheduler.server.cache.TableLock [pool-1-thread-5] : [removeReadLock(String)]Removed Read Lock on table sb.p_x: 0
2018-10-30 00:51:47,775 DEBUG com.ibm.biginsights.bigsql.scheduler.Dev.Assignment [pool-1-thread-5] : [assignSplits] Workers node-numbers: [1, 2, 3]
2018-10-30 00:51:47,775 DEBUG com.ibm.biginsights.bigsql.scheduler.Dev.Assignment [pool-1-thread-5] : [assignSplits] Workers ip to node-numbers: {172.16.186.139=[3], 172.16.186.104=[2], 172.16.186.9=[1]}

As one can see, all partitions except x=4 and x=5 are ignored during execution of the join query.
Bigger example
Of course, partition elimination is nice but what we are really interested in is the performance boost. How to measure it?
I created a simple project. The general idea is to prepare two huge tables, one non-partitioned and the second partitioned and run identical join query against the table.  The source code and detailed description can be found here.
The final result?
For non-partitioned table:

[sbartkowski@oc0522068411 bigjoin]$ time ./run.sh
Run runnonquery.sql script
PASSED

real 0m16.521s
user 0m0.036s
sys 0m0.100s

For partitioned table and partition elimination in place:

[sbartkowski@oc0522068411 bigjoin]$ time ./run.sh
Run runnonquery.sql script
PASSED

real 0m8.860s
user 0m0.037s
sys 0m0.095s

So the performance improved twice which should come as no surprise.
The same query submitted in Hive against the same tables executes as below. Obviously Hive has some way to go.

select D.ID,max(T.tm) from monit.testdim AS D , monit.testpart AS T WHERE T.part=D.part GROUP BY D.ID;

--------------------------------------------------------------------------------
+---------+-----------------------------+--+
|  d.id   |             _c1             |
+---------+-----------------------------+--+
| PART 5  | 2014-09-18 00:19:11.893212  |
| PART 2  | 2014-09-18 00:03:35.599702  |
| PART 4  | 2014-09-18 00:26:17.660366  |
| PART 3  | 2014-09-18 00:24:47.644779  |
+---------+-----------------------------+--+
4 rows selected (41,863 seconds)

The BigSQL is really big.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features

• "Writing" technology implemented, the player can cancel city action of his opponent
• "HangingGarden" wonder implemented

"Writing" technology
If you have "Writing" technology and was able to recruit the Spy then you can tell "every dog has its day".

The execution of the player city action is postponed unless his opponent decides to cancel it or be merciful (this time). You have to take a decision quickly otherwise the opponent may know that you have something up your sleeve.

The opponent wants to build a Libary and now you have to take a decision: turn thumb down and cancel it or turn thumb up.
If you decide to slap him in the face, do not be astonished that your Spy is uncovered and gone.

The HangingGarden wonder

Now at the beginning of the turn, you can purchase your figure for free provided is available.

Next step
Implement a journal, display messages related to the game. For instance, if "Writing" technology was used the opponent should be informed what had happened.

Pandas DataFrame and Scale Spark DataFrame

In pandas DataFrame (similar but different then Spark's DataFrame), data is provided by series.

import numpy
from pandas import DataFrame, Series
d = {'one' : [1., 2., 3., 4.],  'two' : [4., 3., 2., 1.]}
df = DataFrame(d)

one two 
0 1.0 4.0 
1 2.0 3.0 
2 3.0 2.0 
3 4.0 1.0

In Spark's DataFrame data is provided by features, rows in feature matrix.

val sqlC = new org.apache.spark.sql.SQLContext(sc)
import sqlC.implicits._
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val df = Seq((1.0,4.0),(2.0,3.0),(3.0,2.0),(4.0,1.1)).toDF("one","two")

+---+---+
|one|two|
+---+---+
|1.0|4.0|
|2.0|3.0|
|3.0|2.0|
|4.0|1.1|
+---+---+

Of course, there is a plenty of methods to create DataFrame from a file or any external source. But sometimes it is convenient to create Spark's DataFrame manually using panda's convention.
So I created a simple Scala method for creating DataFrame using series, not features.
Zeppelin notebook

import org.apache.spark.sql._
import org.apache.spark.sql.types._

def createDF(spark: SparkSession, names: Seq[String], series: Seq[Any]*): DataFrame = {
    require(names.length == series.length)
    //    val datas : Seq[Seq[Any]] = List.fill(names.length)(Nil)
    //    val rows : Seq[Row] = List.fill(names.length)(Row())
    val numof: Int = series(0).length
    var rows: Seq[Row] = Nil
    for (i <- 0 until numof) {
      var da: Seq[Any] = Nil
      for (j <- 0 until series.length)
        da = da :+ series(j)(i)
      val r: Row = Row.fromSeq(da)
      rows = rows :+ r
    }
    val rdd = spark.sparkContext.makeRDD(rows)
    // schema
    val schema: Seq[StructField] =
      for (i <- 0 until names.length)
        yield StructField(names(i),
          series(i)(0) match {
            case t: Int => IntegerType
            case t: Double => DoubleType
            case _ => StringType
          },
          false
        )
    spark.createDataFrame(rdd, StructType(schema))
  }

Usage example

val names2 = Seq("one", "tow")
    val seriesone = Seq(1.0,2.0,3.0,4.0)
    val seriestwo = Seq(4.0,3.0,2.0,1.0)
    val da =  createDF(spark, names2,seriesone,seriestwo)
    da.show

Example taken from Udacity course.

val names1 = Seq("countries","gold","silver","bronze")
    val countries = Seq("Russian Fed.", "Norway", "Canada", "United States",
    "Netherlands", "Germany", "Switzerland", "Belarus",
    "Austria", "France", "Poland", "China", "Korea",
    "Sweden", "Czech Republic", "Slovenia", "Japan",
    "Finland", "Great Britain", "Ukraine", "Slovakia",
    "Italy", "Latvia", "Australia", "Croatia", "Kazakhstan")

    val gold = Seq(13, 11, 10, 9, 8, 8, 6, 5, 4, 4, 4, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0)
    val silver = Seq(11, 5, 10, 7, 7, 6, 3, 0, 8, 4, 1, 4, 3, 7, 4, 2, 4, 3, 1, 0, 0, 2, 2, 2, 1, 0)
    val bronze = Seq(9, 10, 5, 12, 9, 5, 2, 1, 5, 7, 1, 2, 2, 6, 2, 4, 3, 1, 2, 1, 0, 6, 2, 1, 0, 1)

    val da1 =  createDF(spark, names1, countries, gold,silver,bronze)
    da1.show(3)

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

Several new features are implemented: conquering the alien city and victories.  Also, the performance is improved by caching.

Conquering the city and taking the double loot

The alien city can be attacked and pulled down. After winning (or losing) the winner can take the double loot.

According to game rules, the double loot is at the winner's disposal. Some loots are ranked as one, for instance, stealing the trade and some are ranked as two, for instance, the new technology. The number of loot cannot exceed the two.
Military Victory
If an alien capital city is conquered, the victor will get the medal of the Military Victory.

Culture Victory
After reaching the top of the culture track, the Culture Victory is announced.

Economic Victory
When a player puts aside at least 16 golden coins is acclaimed as Economic Winner by the astonished world.

Technology Victory
When a player piles up the stack of technology high enough, the world recognizes him as Steve Jobs and Elon Musk in one person.

Performance improvement
I started to sample the Civilization Engine using VisualVM profiler and immediately discovered the first bottleneck. The game board is every time recreated by executing all commands starting from the initial layout and it was the main resource consumer. The solution was pretty simple, just cache the current board and the latency was reduced even for my free Heroku quota, 1 CPU and 0.5 GB of memory.
Next steps
Unfortunately, it is high time to implement "hanging" feature like "Writing" :
"Cancel a city action being performed by another player (may not cancel resource ability)". I do not have a clear idea of how to accomplish it. One player has to be triggered that another player executes a command eligible for the action and should have a choice: let it or cancel it.

HortonWorks, Atlas and Hive integration not working

Problem
I installed HDP 2.6.5 and tried to enable Hive for Atlas. But no Hive action was reflected in Atlas. I doublechecked the configuration and found it correct. There were no error messages neither in Hive logs nor in Atlas logs. Also, it looked that Atlas Hive Hook was activated for every action but it did not land in Atlas dashboard.

018-08-20 18:39:56,696 INFO  [HiveServer2-Background-Pool: Thread-899]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) -
2018-08-20 18:39:56,697 INFO  [HiveServer2-Background-Pool: Thread-899]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -

Solution
The solution was very simple and was related to Kafka. Atlas is using Kafka as the intermediate medium to push data into Atlas realm. I installed only single Kafka broker but configuration requires at least 3 Kafka brokers to create a quorum. After adding two lacking Kafka brokers the flow was unlocked and all current and pending request found their way to Atlas.

Polymer 2 to Polymer 3

Introduction
I decided to upgrade from Polymer 2 to Polymer 3.  According to

"we've made a smooth upgrade path our top priority for Polymer 3.0. Polymer's API remains almost unchanged, and we're providing an upgrade tool (polymer-modulizer) that will automatically handle most of the work in converting your 2.x-based elements and apps to 3.0.". 

Encouraged by this advertisement, I run "modulizer --out ." and ... It depends on what one means by "most of the work" and "smooth".

Module names instead of path names
Instead of import pathnames like:

<href="../bower_components/polymer/polymer-element.html" rel="import"></link>

module names are used:

import {PolymerElement, html} from '@polymer/polymer/polymer-element.js'

I understand the rationale behind that but Chrome browser I'm using obviously does not. It still insists that @polymer is a directory name and demands the path name to resolve it. Adding the path names manually is challenging because not only custom elements should be touched but also all internal Polymer elements. So it looks that some kind of "build" process is necessary after every upgrade. After some trials, errors and research I ended up using:

polymer build --module-resolution node

which does the job. Then I replace current "node_modules" directory with "build/default/node_modules". In the case of the Polymer upgrade,  "yarn install" should be launched beforehand. Previously, no build process was required. It was enough to run "bower install".

ReferenceError: IntlMessageFormat is not defined

Uncaught (in promise) ReferenceError: IntlMessageFormat is not defined
    at HTMLElement. (app-localize-behavior.js:285)
    at runMethodEffect (property-effects.js:905)
    at Function._evaluateBinding (property-effects.js:3079)
    at Object.runBindingEffect [as fn] (property-effects.js:648)
    at runEffectsForProperty (property-effects.js:169)
    at runEffects (property-effects.js:131)
    at HTMLElement._propagatePropertyChanges (property-effects.js:1933)
    at HTMLElement._propertiesChanged (property-effects.js:1891)
    at HTMLElement._flushProperties (properties-changed.js:370)
    at HTMLElement._flushProperties (property-effects.js:1731)

Unfortunately, "polymer build" does not catch "node_modules/intl-messageformat" package and it is absent after build. So we have to reinstall the package again.

npm install intl-messageformat 

ReferenceError: IntlMessageFormat is not defined
Although "intl-messageformat" is downloaded, it should be also imported somewhere to make "IntMessageFormat" class visible.
I ended up patching manually "node_modules/@polymer/app-localize-behavior/app-localize-behavior.js" file.

import "../polymer/polymer-legacy.js";
import "../iron-ajax/iron-ajax.js";
import "../../intl-messageformat/dist/intl-messageformat.js";

Uncaught TypeError: Cannot set property 'IntlMessageFormat' of undefined

Uncaught TypeError: Cannot set property 'IntlMessageFormat' of undefined
    at main.js:7
    at main.js:7

It was a stab in the back because Chrome does not tell where this disaster happens. After some time I found a hint here.
The solution is to patch manually "node_modules/intl-messageformat/dist/intl-messageformat.js".
At the end of the file replace:

    var src$main$$default = $$core$$default;
    this['IntlMessageFormat'] = src$main$$default;
}).call(window);

with:

    var src$main$$default = $$core$$default;
    this['IntlMessageFormat'] = src$main$$default;
}).call(this);

Uncaught ReferenceError: KeyframeEffect is not defined
This exception was thrown from paper-dropdown-menu element. Instead of spending time trying to resolve it, I decided to remove this dependency and replaced it with something different. Replace namespace with imports
In Polymer 2 I was using Polymer namespace for custom classes.

olymer.CivData = function(superClass) {

        return class extends Polymer.CivLocalize(superClass) {

                constructor() {
                        super();
                }

..........
class CivTLevel extends Polymer.CivData(Polymer.Element) {

    static get is() {
        return 'civ-tlevel';
        }

Polymer 3 enforces transforming all class to ES6 modules. So sequence above should be replaced by:

import { CivLocalize} from "../js/civ-localize.js";

export const CivData = function (superClass) {
  return class extends CivLocalize(superClass) {
    constructor() {
      super();
    }

...........
import { html } from "../node_modules/@polymer/polymer/lib/utils/html-tag.js";
import { PolymerElement } from "../node_modules/@polymer/polymer/polymer-element.js";
import { CivData} from "../js/civ-data.js";

class CivTLevel extends CivData(PolymerElement) {
  static get template() {

"modulizer" utility has nothing to do with that, it should be done manually.
Conclusion
"Smooth upgrade" ended up in several sleepless nights. But in the end, I made it. On the whole, I really like this ES6 module, it provides a good encapsulation and isolation which is very important in every programming language including JavaScript.

Byliśmy na operze

20 maja byliśmy na operze Sergieja Prokofiewa "Ognisty Anioł" w Teatrze Wielkim. Muzyka podobała nam się bardzo, chociaż sama inscenizacja niekoniecznie w każdym calu.
Praca nad operą zajęła Prokofiewowi wiele lat pomiędzy rokiem 1919 i 1929, ale dzieło zostało pierwszy raz wystawione dopiero po śmierci kompozytora, w 1955 roku. Inspiracją dla Prokofiewa była powieść rosyjskiego symbolisty, Waleriego Brusowa pod tym samym tytułem.
Przestawienie w Teatrze Wielkim jest autorską realizacją Mariusza Terlińskiego. Realizator odszedł od dosłownego odczytania dzieła, nie mamy tutaj rycerza, zamku, klasztoru, okultyzmu, diabłów, aniołów, seansów magii. To co mogło szokować publiczność prawie 100 lat temu, raczej nie poruszy współczesnych odbiorców. Zamiast tego widzimy podróżującego akwizytora z walizką, tanie hotele z migającymi neonami w estetyce lat 60-ych, gdzieś nawet przemykają nam bitelsi, mistrz wiedzy tajemnej to diler narkotyków, seans okultystyczny zamienia się w orgię z użyciem pobudzających substancji, zaś zamiast do klasztoru przenosimy się do toksycznego internatu dla dziewcząt. Ale zostawione jest to co wspólne: obsesja, tajemnica, poszukiwanie, wędrówka, miłość i nienawiść. Reżyser jednak nie poprzestał na narysowaniu starej historii współczesną kreską. Końcowa scena w klasztorze z Inkwizytorem jest retrospektywą. Inkwizytor, Ognisty Anioł oraz hrabia Henryk zlewają się tutaj w jedną postać, sadystycznego wychowawcy. Domyślamy się, że krzywda wyrządzona Renacie jest traumatycznym wydarzeniem, który wpłynął na jej późniejsze życie.
Można oczywiście zadać pytanie, czy takie opowiedzenie historii ułatwia odczytanie i zrozumienie dzieła współczesnym widzom, czy nowa symbolika zastępuje starą. Raczej jest to wątpliwe, scena jest po prostu wizualnym tłem dla śpiewaków i muzyków. Dzieje się tam zbyt dużo, aby udało się to połączyć w całość, natłok pomysłów realizatorskich zamienia się w feerię mrugających świateł, neonów i  kwadratów, na scenie pojawiają się i mnożą postacie, czasami widzimy kilka Renat i zastanawiamy się, która jest prawdziwa.
Na szczęście główną wartości jest muzyka Prokofiewa i w tym wykonaniu doskonale się broni. Wykonawcy głównych ról, Renaty i Ruprechta, śpiewają znakomicie, niezwykle trudna jest zwłaszcza główna rola, która jest osią przedstawienia od pierwszej do ostatniej nuty i wymagająca nie tylko umiejętności wokalnych a także aktorskich. Można się ograniczyć do słuchania samej muzyki, patrzeć na scenę i nie zmuszać się do śledzenia zamysłu reżysera.
Bardzo ładnym gestem jest także program teatralny z opracowaniem w języku rosyjskim, wyraz szacunku dla kompozytora i jego współczesnych rodaków.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

Several civilizations feature are implemented: America, China, Egypt, Germany, Rome and Russia.

America
Free random Great Person at the beginning (Great Persons capabilities are not implemented yet).

When capital is built, the Great Persons is added to the great person registry and can be placed at the city outskirts.
China
After any battle, China can save on the killed units and return it into the standing force.

Egypt
Egypt can build an unblocked building during City Management phase.

Germany
When Germany researches a technology which upgrades or unlocks a unit, it can also pick up one of the resources from the market.

Russia
Russia can sacrifice the figure, army or scout, ending its move in the foreign city and learn for free a technology already researched by this civilization.
When the figure is close enough to the foreign city, an additional button is enabled.

The player has to select the figure and foreign city and the technology dialogue pops up.

Next steps

• Complete the implementation of the two remaining civilizations.
• Attack the city

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features implemented

• Great Persons, only resources augment is implemented, not Great Person card abilities
• Event Card, cards only, events are not implemented

Great Person

While advancing on culture track, the player is receiving Great Person.

The Great Person can be placed immediately on the map or can be held in reserve and placed on the map later during Start Of Turn phase.

After placing the Great Person on the map, its resource capabilities augment the capacity of the adjacent city.

You are welcome, Great General!

Culture Cards
Culture Cards are collected (like Great Persons) during advancing in culture. Culture Cards Events are not implemented yet.

If a number of Culture Cards exceeds the size of the hand, the players should reject one of the cards.
Next steps
Capabilities of civilizations.

DB2 and user-defined aggregate functions

Introduction
It is hard to believe but in DB2 there was not a feature to create user-defined aggregate function. It caused problems while migrating from Oracle to DB2 and to overcome them required sometimes a lot of effort.
Finally, this long-awaited option has been shipped in DB2 11.1. It is good news. But there also a bad news, it is very scanty documented. The only information I was able to find was an article in official DB2 Knowledge Centre.
So I decided to fill the gap and to create my own port of exemplary Oracle aggregate function.
The source code of Oracle UDF and DB2 equivalent is available here (db2aggr branch). Unfortunately, the custom aggregates are less flexible than standard and there is no way to apply additional argument, for instance, the delimiter character. This applies to DB2 and Oracle. In the example, hardcoded coma is used.
Aggregate function
The aggregate function is similar to standard LISTAGG function. But it also removes duplicates and sorts the result.
Assuming the table:

CREATE TABLE "TESTID" 
   "ID" VARCHAR2(20 BYTE)
  ); 
  INSERT INTO TESTID VALUES('G');  
  INSERT INTO TESTID VALUES('H');  
  INSERT INTO TESTID VALUES('A');
  INSERT INTO TESTID VALUES('B');
  INSERT INTO TESTID VALUES('C');
  INSERT INTO TESTID VALUES('C');
  
  INSERT INTO TESTID SELECT * FROM TESTID;

The expected result of running:

  SELECT LISTAGGDUPL(ID) FROM TESTID;

is:

A,B,C,G,H

Oracle
Oracle PL/SQL code of the aggregate function is available here. It simply collects values in the nested table and the final removing duplicates and sorting are done by running:

FOR Result IN ( SELECT DISTINCT O.COLUMN_VALUE FROM TABLE(self.gCollect) O ORDER BY 1 )

DB2
User-defined aggregate function in DB2 are Java or C++, there is no way to develop it in pure DB2 SQL/PL. So the implementation consists of two parts: Java code and SQL/PL code for defining the signature and deployment.

• Java project is accessible here. That is very simple and straightforward, no comments are required.
• SQL/PL code is available here.
• Test code, the same for Oracle and DB2.

Additional remarks

• In order to run the function in DB2 it is recommended to increase the Java heap space. Example: db2 update dbm cfg using JAVA_HEAP_SZ 76800
• I was unable to trigger "MERGE PROCEDURE" phase. So I'm not sure of the Java code implementing it is valid.
• During the test, I discovered that INITIALIZE and FINALIZE code is run twice, I do not understand why. 

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features implemented 

• Culture track
• Spend culture tokens to advance a culture level
• Devote city to arts
• MetalWorking
• Code Of Laws
• Currency
• Irrigation
• Navigation

Devote city to arts

The player can choose a city to devote it to arts to collect culture tokens. Culture tokens can be also sent from scout or scouts occupying an appropriate square.

Of course, the noble sacrifice is not in vain, it is piled in the resource panel.

Advance culture level, culture track

If the player stocks enough culture tokens can spend them to advance culture.

The cost depends on the culture level, in the beginning, it is 3 culture tokens. After spending culture, the player is climbing up one step on the culture ladder.

The player can advance only once using the dialog window but it can be repeated several times in the same City Management phase.

Important: the current implementation supports only marking the progress on the culture track. The culture cards, great persons, and culture victory are still pending.

MetalWorking technology

During the battle, the iron token can be used to increase the attack strength. Also, iron harvested from hut and villages can be consumed. The current implementation spends firstly iron token. If the iron token is not gathered, the hut or village is used. In the future implementation, the player will have a choice which iron to spend. 

Code Of Law technology

After winning the battle, the economy is increased and a coin is added to the technology card.

Currency technology

The player can spend incense to gain three culture tokens. Explicit incense token can be used or incense collected from hut or village. The player has an option to point it.

Irrigation technology

Irrigation allows building the third city. Research this technology as quickly as you can, it is the must in this game.

Navigation technology

After researching this technology, the player can cross the water but cannot stop in it. It is also the must in water covered area.

Next steps

• Great persons
• More technologies

KVM, HortonWorks and IBM Spectrum Scale

IBM Spectrum Scale (former GPFS) is a clustered file system giving single access point to distributed parallel nodes. HDP is Hadoop implementation created and supported by HortonWorks. Both work together smoothly, GPFS can replace HDFS in a transparent way.
If you are entitled to use IBM Spectrum Scale, it is tempting to install IBM Spectrum Scale locally and explore its capabilities without setting up huge infrastructure.
Unfortunately, although there is plenty of ample information available, it is not easy to make the first step and just install it.
So I decided to fill the gap and install HDP HortonWorks on the top of IBM Spectrum Scale (GPFS) using KVM virtualization. I'm not paying too much attention to stuff like tunning, mirroring, replication, configuration etc. Just install using a default setting as much as possible to give the first taste of this fantastic feature.
More details can be found here.
Before you start:

• Make sure you are entitled to use IBM Spectrum Scale that way, review license agreement. IBM Spectrum Scale is not publicly available.
• Do not start unless you at least 32GB memory, otherwise, 4 KVM with at least 6 GB memory will kill your machine.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features

• "Pottery" technology is enabled and you can gain coins by spending resources on them.
• Economy progress is scored.

Pottery and economy

"Pottery" technology is the first to be enabled so far.

When technology is researched and you have at least two resources hoarded you have the opportunity to transform wheat or incense into gleaming gold.

• "Use Technology" 
• Choose technology you want to use, here Pottery only
• Point the city where the action will be performed
• Select two resources to spend

After spending resources you will be two resources lighter but one gold coin heavier.

Also "Pottery" card weights one coin now.

Although you can collect coins, the economic victory is not implemented yet.
Next step

• Score culture progress.

Find missing number or numbers

Exercise
There is an exercise in Cracking the Code Interview.

Missing Two: You are given an array with all the numbers from 1 to N appearing exactly once,
except for one number that is missing. How can you find the missing number in O(N) time and
0( 1) space? What if there were two numbers missing?

The solution is quite simple, just sum up all numbers from 0 to N using well-known formula, then sum up all numbers in the table and the difference is the number missing.
In case of two numbers missing, this method gives back the sum of the numbers. So it is necessary to conduct additional calculation, for instance, to sum square of the numbers. This method we receive the sum of the squares of the missing number and after resolving quadratic formula, we come up with the numbers.
But this solution has a weak point. If the number is big enough, there is a risk of overflow and the whole calculation is wrong. The threshold is around the square root of the maximum value of the integer involved. For 32 bit integer, it is around 46341.
Alternative solution for one missing
The alternative solution for one number missing is to calculate the number of ones and zeros in a binary representation of all numbers and the same for the table with missing one. Comparing which bits are missing, we can reconstruct the number lost.
The C++ solution is available here. Only number of ones should be calculated.
It does not break the time and space complexity requirement. Although we have to enumerate through bits for every number, the loop is still constant, 32 for 32 integer. 32*N is still O(N). Also, we need additional space for storing the number of bits but it is also constant and keeps O(1).
Solution for two missing
A similar solution can be applied to two numbers missing but only the sum can be reconstructed. If the difference between ones for the whole sum and the table is 0, it means that both missing numbers have zeros at the position. If the difference is 2, we can assume ones. But if the difference is 1, it means a combination of one and zero. Unfortunately, we cannot assign them to the numbers. But in case of the sum, it is not important because addition is commutative so we can assign them as we like.

uint ReconstructMissingSum(ByteCounter &all) const {
   uint missing1 = 0;
   uint missing2 = 0;
   for (int i=BN-1; i>=0; i--) {
 missing1 <<= 1;
 missing2 <<= 1;
 if (all.one_sum[i] != one_sum[i]) missing1 += 1;
 // Sum have two ones at the position. Move the second 1 to the second number.
 // We are calculating the sum, the order does not matter
 if (all.one_sum[i] - one_sum[i] == 2) missing2 += 1;
 }
   // it is not the reconstruction of numbers but the sum only.
   return missing1 + missing2;
}

But, as above, we need an additional equation to extract the exact values of the numbers. The same method can be applied for squares of the numbers. Unfortunately, this solution falls under the curse of overflow. Maybe there is a method to improve it but I was unable to find it so far.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features

• Wonders of The World.

You can buy wonders and spend money on them. Unfortunately, the functionality of the wonders is not implemented yet, will be added later.

Wonders are displayed in the separate panel. 

No graphic is attached, only the first letter of the Wonder name is exposed. It would be difficult to prepare several dozens of different and meaningful images. Only in the background a symbol of Ancient, Medieval or Modern is added.
When the player piles up the stack of money big enough, is allowed to buy a wonder. Discount for discovering a proper technic is granted also.

One player put aside enough to buy Pyramids or Great Lighthouse. After selecting a wonder, a square where the wonder is to be positioned should be selected. If square already had a building or another wonder atop, the previous content will be replaced.

The wonder is built and the world is staring at it with mouth open.
Next step

• Economy, coins gathering
• Culture progress

Circus tower

Introduction
There is an exercise in "Cracking the coding interview", Circus Tower.

Circus Tower: A circus is designing a tower routine consisting of people standing atop one anoth­er's shoulders. For practical and aesthetic reasons, each person must be both shorter and lighter than the person below him or her. Given the heights and weights of each person in the circus, write a method to compute the largest possible number of people in such a tower.

EXAMPLE

lnput(ht,wt): (65, 100) (70, 150) (56, 90) (75, 190) (60, 95) (68, 110)

Output: The longest tower is length 6 and includes from top to bottom:

(56, 90) (60,95) (65,100) (68,110) (70,150) (75,190)

The solution is based on finding the longest subsequence.
Alternative solution
But I developed a very similar solution in term of the directed graph and the solution is to find the longest possible path in the graph. Usually, we want to find the shortest path but this time it is different.  Athletes in the circus are graph nodes and if an athlete can stand atop the weightier athlete then there is an edge from the second to the first.
The C++ code as Eclipse CDC project is available here.
The class Athlete describes a single person and class AthleteGraph describes the connection. The graph is represented as a two-dimensional adjacency matrix. If there is a connection from i to j, then graph[i][j] is true.
The algorithm is recursive.
Assume that we are at the node i, and there are several adjacent nodes to i :  i1,i2 ... i(n). We calculated recursively the longest paths for i1, i2 .. i(n). The longest path starting from i is i itself concatenated with the longest among paths for i1,i2 ... i(n).
Complexity
The most complex is the graph creation because we have to compare a person with every other person to detect edges. So the complexity is O(n squared). Having the graph built, the complexity is O(m), where m is the number of edges, every edge is visited only once.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features

• Buildings
• Improvements

Buildings

The player can buy building after researching appropriate technology.

Of course, the city should provide enough production.

Buying building is very simple. Click "Buy Building" button and city you want to enrich.

Last, but not least, point the square you want to industrialize and skyscraper is ready.

Replacement
But building new tower requires sometimes pulling down the old one. In order to avoid mistakes, a different button is displayed if a new building is supposed to be placed over the previous.
If new "star" building is ready to be set up, not only building on the same square could be removed but also another "star" construction.

Next step 
Wonders of the world.

Byliśmy na operze

23 stycznia byliśmy na operze Peleas i Melizanda w Teatrze Wielkim. Muzyka podobała się bardzo, ale samo przedstawienie mniej.
Peleas i Melizanda to jedyna opera Debussyego do libretta według dramatu Maeterlincka pod tym samym tytułem. To baśń o tajemnicy, prawdziwym uczuciu i śmierci, osadzona w bliżej nieokreślonym średniowiecznym świecie. Na początku pojawia się młoda dziewczyna która podaje tylko swoje imię, Melizanda. Jej przybycie na dwór króla Golauda wprowadza świeże powietrze do zastygłego świata królewskiej rodziny ale też uruchamia spiralę miłości i śmierci. Gdy na końcu umiera, nie wiemy o niej nic więcej poza imieniem, ale zostawia po sobie małą córkę i właśnie to imię. Jak przepowiada na końcu król Arkel, "Teraz musi żyć w jej imieniu, teraz kolej na to biedne maleństwo". Cykl tworzenia i destrukcji zatoczy kolejne koło.
Sztuka nie tworzy jakiegoś określonego ciągu wydarzeń prowadzących do finału, dialog są wieloznaczne, jest tutaj dużo pytań bez odpowiedzi i odpowiedzi do niezadanych pytań. W tle czujemy atmosferę lęku, niepewności, niewypowiedzianej tajemnicy, bohaterowie sprawiają wrażenie, że nie decyduję o swoim losie, kieruje nimi jakieś zewnętrzna siła, fatum. Sama muzyka nie jest operą do jakiej jesteśmy przyzwyczajeni, nie ma tutaj podziału na arie, recytatywy, partie solowe czy duety. Muzyka idzie za bohaterami, przechodzi płynnie z jednej sceny w drugą.
Ale dlaczego realizacja podobała się mniej? Była próbą osadzenia nieokreśloności na bardziej twardym gruncie. Sama akcja to sen Melizandy, gdzie bohaterka jest jednocześnie demiurgiem i podmiotem. Zasypia na początkui i budzi się na końcu. Pałac Golauda to burżuazyjna posiadłość i dysfunkcyjna rodzina, zaś Melisanda jest ofiarą zazdrości i namiętności rozpalonych samców. Niektóre sceny przypominają estetykę z filmów dla dorosłych. Czasami widzimy dwie Melisandy i nie jest jasne która jest prawdziwa, zaś która to alter ego. Nie jest zrozumiałe, dlaczego Melisanda czasami siada na barana na Golaudzie.
Na szczęscie od strony muzycznej i wykonawczej przedstawienie jest znakomite, muzyka się broni. Najlepiej kupić bilety, usiąść na widowni, zamknąć oczy i po prostu słuchać. Jeśli ktoś tu poniósł porażkę, to na pewno nie Debussy.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features

• Greeting window more verbose
• Research, building up technology tree

Greeting window

Civilizations tiles more descriptive. But that's description only, civilization capabilities, government and technologies are not implemented yet.
Technology tree
The player can spend trade and enhance technology tree.

The current technology tree can be reviewed any time by clicking technology icon. The player can also peek opponent's achievements on this field.

Unfortunately, the thumb is down. The technology abilities are not implemented.
Next steps
Producing unlocked buildings.