Król Roger, Teatr Narodowy

11 grudnia byliśmy na przedstawieniu Króla Rogera w Teatrze Narodowym. Było to jednak przedstawienie do oglądania, nie do słuchania.
Król Roger II to postać historyczna, słynący z mądrości i pobożności średniowieczny król Sycylii. W operze Szymanowskiego w państwie króla pojawia się tajemniczy Pasterz, prorok nowego boga, o którym śpiewa "Mój Bóg jest piękny jako ja". Przybycie Pasterza na dwór króla wyzwala żywioły, które niczym wir mielą i przemieniają wszystko, dwór, królestwo i Roksanę, ukochaną żonę króla, zostawiając Rogera jako samotnego pielgrzyma.
Sama opera nie jest łatwa w odbiorze, to nie jest historia, którą trzeba opowiedzieć od początku do końca. To ciąg trzech scen, w pierwszej widzimy królestwo Rogera, harmonijne połączenie władzy świeckiej i kościelnej, w drugiej jest konfrontacja Rogera i Pasterza, w trzeciej Roger jako odarty z atrybutów królewskich pielgrzym wędruje w przemienionym mocą Pasterza świecie szukając Roksany.
Operę można odczytać na wiele sposobów. Realizacja w Teatrze Wielkim oczywiście ucieka jak najdalej od średniowiecznego kontekstu. Dwór królewski to zarząd korporacji z Rogerem jako CEO. Pośrodku jest ogromny stół konferencyjny, Roksana to zblazowana dama w wyższych sfer. Mamy szklane klatki, jest łazienka z marmurową umywalką, wózek inwalidzki, migające stroboskopowe światła, lustra, krwawe sceny w estetyce filmów gore, maski z rogami muflona, zwielokrotnione postacie, wizualizacje video, ściana mieniących się cyferek jak z Matrixa. Końcowa scena odrealnionego świata jest odwołaniem do finału "2001 Odyseja Kosmiczna", pojawia się nawet małe dziecko niczym embrion Bowmana w słynnym filmie Kubricka. Aby się przemienić i zrozumieć, musimy narodzić się na nowo.
Niestety, w tym wszystkim zagubiła się sama muzyka, największa wartość opery. Tak jak w końcowej scenie 2 aktu, gdzie z podwieszonego głową do dołu Rogera ścieka krew na leżącą na stole Roksanę, złożony został tutaj w ofierze sam Szymanowski i wszystko zostało postawione na odwrót, ogon zamerdał psem, forma przerosła treść. Wprowadzająca wizualizacja wideo, gdzie pełzający wąż symbolizuje zgniliznę toczącą pozornie perfekcyjne królestwo/korporację, w rzeczywistości zapowiada klęskę samego spektaklu.
Muzyka jest tutaj wyłącznie sound-trackiem do efektownego widowiska, jakby nie miała odwagi unieść się z kanału orkiestrowego. Początkowy chór "Hagios! Hagios!" jest odtwarzany z taśmy, gdy na końcu przedstawienia do oklasków wybiega chór, na widowni czuć konsternację skąd te osoby się tutaj wzięły. Nawet zapowiedź, że wykonawca tytułowej roli, Łukasz Goliński, nie jest w pełni dysponowany, dla spektaklu nie miała wielkiego znaczenia, gdyż forma solistów nie była tutaj istotna, w zasadzie całą warstwę muzyczną łącznie z orkiestrą można było odtworzyć z taśmy.
Miała być opera, a było wyłącznie widowisko światło i dźwięk z akcentem na pierwszy człon.

TPC-DS benchmark

Inspiration
TCP-DS benchmark is an industry standard tool to evaluate the performance of the relational database engine. The tool can be used not only to leverage the general efficiency. The database administrator or DevOp engineer can utilize the tools to estimate the robustness of a particular database installation by comparing it against a reference instance. Also, it can be useful to verify the database health after upgrade or maintenance. Another use case is to gauge the performance gain after tunning or scaling.
The main disadvantage of the standard package is that it is not easy to use, requires a number of manual adjustment and fixes. It makes the tool less reliable and the process less repeatable. To compare the result against the reference results we have to be sure that the benchmark for both environments, the reference, and the current, are prepared and executed using the same queries and according to the same pattern.
Solution
I created a simple solution to make the task very simple. The project is available in GitHub.
The following databases are supported so far: Oracle, IBM DB2, MySQL/MariaDB, PostgreSQL, Hive, SparkSQL, Netezza, and IBM BigSQL.
Also, a wiki is available having hints on how to set up a test for a particular database. Having the tool, the TCP-DS benchmark is ready to use. Just download, configure for a particular environment and run.
So far, Qualify and Power Tests are implemented.
Queries
The TPC-DS queries are not ready to use out of the box for all SQL engines. The specification allows some minor fixes to make them executable for a particular database. To avoid keeping a different version of queries, I decided to modify them on the fly. I also apply only changes possible to implement through the simple string or regular expression text replacement.
Test Validation
I was unable to match the answer data set provided against any output. It requires further investigation. So I decided to use Oracle output as a reference result set. Unfortunately, even Oracle result does match all of them and is not fully reliable. The life is never an easy road free of stone.
Next steps
Throughput and Data Maintenance tests.

BigSQL monitoring

https://www.ibm.com/support/knowledgecenter/en/SSCRJT_5.0.3/com.ibm.swg.im.bigsql.doc/doc/admin_monitor-bigsql-query.html

https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.admin.wlm.doc/doc/c0055265.html

GitHub project

Inspiration
BigSQL is nothing more than DB2 running on the top of Hadoop. It also inherits a lot of goodies from DB2 including a sophisticated rigging of metrics. But the metrics are only numbers and by itself do not provide anything meaningful unless one is deeply versed in DB2 internals.
Metrics are cumulative and are growing constantly. Instead of looking at pure values much more interesting is observing how the values are changing over time and trying to discover some patterns or trends. For instance, if the metric suddenly starts to surge it can indicate that the BigSQL is under a heavy workload.
Also, it could be interesting to use the metrics for prediction, for instance, that there is a risk of delays or prolonged response if an adequate pattern is found.
Solution
I developed a simple solution, it can be downloaded here.
The purpose of the tool is to retain historical data and to get easy access to the difference between consecutive metric values for analysis.
The solution contains the following elements:

• Database schema to keep historical data. The metrics are pivoted, instead of a single row of metrics it breaks down the row into a series of records: metric id/value
• The supporting view providing a difference between consecutive metric values instead of the pure value.
• Several stored procedures to harvest and extract collected metrics.
• Two methods of metrics collecting are available: as Linux crontab job or as DB2 scheduled task.

Simple analysis

I also developed a simple tool for heavy workload prediction. Although in DB2 there are several hundred different metrics, only subset of them seems to be related to workload specific for BigSQL. The idea is to nail down a period of normal workload and to monitor the current period. If the average of several metrics being observed exceeds significantly the average of the normal period, the alarm is raised that heavy workload is underway.

But during my testing, the solution does not seem to be usable. More tuning is necessary or a different approach should be built which has teeth.

Polymer 3, upgrade to 3.01

I was blocked by a strange problem after upgrading from 3.0.0-pre.19 to 3.0.1. Suddenly  I found an extraordinarily long delay before displaying the game board for the first time.

Usually, it took several seconds to move from the first screen to the third. After the upgrade, the time was prolonged to almost 50 seconds making it almost useless. It is not easy to find a bottleneck in an asynchronous framework. So firstly I discovered it is the map (12 * 8 tiles) which is the cause of the delay. Then I gritted my teeth and after several sleepless nights including meticulously comparing old and new JS files, I pinned down the culprit.
The problem was caused by iron-resizable-behavior/iron-resizable-behavior.js file. Next to the end, there is a slightly new code.

_requestResizeNotifications: function () {
    if (!this.isAttached) {
      return;
    }

    if (document.readyState === 'loading') {
      var _requestResizeNotifications = this._requestResizeNotifications.bind(this);

      document.addEventListener('readystatechange', function readystatechanged() {
        document.removeEventListener('readystatechange', readystatechanged);

        _requestResizeNotifications();
      });
    } else {
      this._findParent();

      if (!this._parentResizable) {
        // If this resizable is an orphan, tell other orphans to try to find
        // their parent again, in case it's this resizable.
        ORPHANS.forEach(function (orphan) {
          if (orphan !== this) {
            orphan._findParent();
          }
        }, this);
        window.addEventListener('resize', this._boundNotifyResize);
        this.notifyResize();
      } else {
        // If this resizable has a parent, tell other child resizables of
        // that parent to try finding their parent again, in case it's this
        // resizable.
//        this._parentResizable._interestedResizables.forEach(function (resizable) {
//          if (resizable !== this) {
//            resizable._findParent();
//          }
//        }, this);
      }
    }
  },
  _findParent: function () {
    this.assignParentResizable(null);
    this.fire('iron-request-resize-notifications', null, {
      node: this,
      bubbles: true,
      cancelable: true
    });

    if (!this._parentResizable) {
      ORPHANS.add(this);
    } else {
      ORPHANS.delete(this);
    }
  }
};

The temporary workaround is to comment out the code.

        // If this resizable has a parent, tell other child resizables of
        // that parent to try finding their parent again, in case it's this
        // resizable.
//        this._parentResizable._interestedResizables.forEach(function (resizable) {
//          if (resizable !== this) {
//            resizable._findParent();
//          }
//        }, this);
      }

Unfortunately, I'm unable to provide any explanation for that and the solution is nothing more than kicking the can down the road. Obviously, the code inside the else clause is doing something CPU thirsty but that is all I can make out of it. But it works for me.

BigSQL, joins and partition elimination.

Partition elimination in joins
One of the advantages of BigSQL over Hive is enabling partition elimination also during join execution. It is described in this article.
It can be explained using a simple example.

db2 "CREATE HADOOP TABLE p_x(n VARCHAR(64)) PARTITIONED BY (x int) STORED AS PARQUETFILE"
db2 "CREATE HADOOP TABLE p_x(n VARCHAR(64)) PARTITIONED BY (x int) STORED AS ORC"
db2 "insert into p_x values('a',0)"
db2 "insert into p_x values('b',1)"
db2 "insert into p_x values('c',2)"
db2 "insert into p_x values('d',3)"
db2 "insert into p_x values('e',4)"
db2 "insert into p_x values('f',5)"
db2 "insert into p_x values('g',5)"
db2 "insert into p_x values('h',6)"
db2 "insert into p_x values('i',6)"
db2 "insert into p_x values('j',7)"
db2 "insert into p_x values('k',8)"
db2 "insert into p_x values('l',9)"
db2 "CREATE HADOOP TABLE p_y(n VARCHAR(64),x int)"
db2 "insert into p_y values('e',4)"
db2 "insert into p_y values('f',5)"
db2 "insert into p_y values('g',5)"

Then run a straightforward join query.

db2 "select * from p_x,p_y where p_x.x = p_y.x"

The range of values in the p_y table is 4 to 5 so in order to resolve this join, it is enough to scan only corresponding partitions in p_x table.
It can be detected by looking into BigSQL log. But firstly DEBUG level for BigSQL Scheduler should be turned on.

BigSQL->Configs->Advanced bigsql-log4j The following two parameters should be modified:

• log4j.logger.com.ibm.biginsights.bigsql.scheduler.GlobalLog=DEBUG 
• log4j.logger.com.ibm=ALL

Then BigSQL is to be restarted to get new settings taking effect.

The query partition elimination is reported in /var/ibm/bigsql/logs/bigsql-sched.log

DEBUG com.ibm.biginsights.bigsql.scheduler.server.expr.ExprUtils [pool-1-thread-5] : found column with partition-key. db2ColumnIndex: 1
DEBUG com.ibm.biginsights.bigsql.scheduler.server.StorageHandlerScanState [pool-1-thread-5] : [createScanState] Partition elimination expr:  (  ( x <= 5 )  AND  ( x >= 4 )  ) 
DEBUG com.ibm.biginsights.bigsql.scheduler.server.StorageHandlerScanState [pool-1-thread-5] : found default/dummy partition. skipping. HdfsPartition{fileDescriptors=[]}
[eliminatePartition]partition elimination: checking partition: x=0
[eliminatePartition]partitionEliminated? x=0 true
[eliminatePartition]partition elimination: checking partition: x=1
[eliminatePartition]partitionEliminated? x=1 true
[eliminatePartition]partition elimination: checking partition: x=2
[eliminatePartition]partitionEliminated? x=2 true
[eliminatePartition]partition elimination: checking partition: x=3
[eliminatePartition]partitionEliminated? x=3 true
[eliminatePartition]partition elimination: checking partition: x=4
[eliminatePartition]partitionEliminated? x=4 false
The locations are: [TScanRangeLocation(host_idx:0, volume_id:-1), TScanRangeLocation(host_idx:1, volume_id:-1)]
[eliminatePartition]partition elimination: checking partition: x=5
[eliminatePartition]partitionEliminated? x=5 false
[createScanState] The locations are: [TScanRangeLocation(host_idx:2, volume_id:-1), TScanRangeLocation(host_idx:1, volume_id:-1)]
[createScanState] The locations are: [TScanRangeLocation(host_idx:2, volume_id:-1), TScanRangeLocation(host_idx:0, volume_id:-1)]
[eliminatePartition]partition elimination: checking partition: x=6
[eliminatePartition]partitionEliminated? x=6 true
[eliminatePartition]partition elimination: checking partition: x=7
[eliminatePartition]partitionEliminated? x=7 true
[eliminatePartition]partition elimination: checking partition: x=8
[eliminatePartition]partitionEliminated? x=8 true
[eliminatePartition]partition elimination: checking partition: x=9
[eliminatePartition]partitionEliminated? x=9 true
[createScanState]Finished partition elimination. partition-elimination-stats (eliminated/total): 8 / 10 Took 124 milliseconds
2018-10-30 00:51:47,775 INFO com.ibm.biginsights.bigsql.scheduler.server.cache.TableLock [pool-1-thread-5] : [removeReadLock(String)]Removed Read Lock on table sb.p_x: 0
2018-10-30 00:51:47,775 DEBUG com.ibm.biginsights.bigsql.scheduler.Dev.Assignment [pool-1-thread-5] : [assignSplits] Workers node-numbers: [1, 2, 3]
2018-10-30 00:51:47,775 DEBUG com.ibm.biginsights.bigsql.scheduler.Dev.Assignment [pool-1-thread-5] : [assignSplits] Workers ip to node-numbers: {172.16.186.139=[3], 172.16.186.104=[2], 172.16.186.9=[1]}

As one can see, all partitions except x=4 and x=5 are ignored during execution of the join query.
Bigger example
Of course, partition elimination is nice but what we are really interested in is the performance boost. How to measure it?
I created a simple project. The general idea is to prepare two huge tables, one non-partitioned and the second partitioned and run identical join query against the table.  The source code and detailed description can be found here.
The final result?
For non-partitioned table:

[sbartkowski@oc0522068411 bigjoin]$ time ./run.sh
Run runnonquery.sql script
PASSED

real 0m16.521s
user 0m0.036s
sys 0m0.100s

For partitioned table and partition elimination in place:

[sbartkowski@oc0522068411 bigjoin]$ time ./run.sh
Run runnonquery.sql script
PASSED

real 0m8.860s
user 0m0.037s
sys 0m0.095s

So the performance improved twice which should come as no surprise.
The same query submitted in Hive against the same tables executes as below. Obviously Hive has some way to go.

select D.ID,max(T.tm) from monit.testdim AS D , monit.testpart AS T WHERE T.part=D.part GROUP BY D.ID;

--------------------------------------------------------------------------------
+---------+-----------------------------+--+
|  d.id   |             _c1             |
+---------+-----------------------------+--+
| PART 5  | 2014-09-18 00:19:11.893212  |
| PART 2  | 2014-09-18 00:03:35.599702  |
| PART 4  | 2014-09-18 00:26:17.660366  |
| PART 3  | 2014-09-18 00:24:47.644779  |
+---------+-----------------------------+--+
4 rows selected (41,863 seconds)

The BigSQL is really big.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features

• "Writing" technology implemented, the player can cancel city action of his opponent
• "HangingGarden" wonder implemented

"Writing" technology
If you have "Writing" technology and was able to recruit the Spy then you can tell "every dog has its day".

The execution of the player city action is postponed unless his opponent decides to cancel it or be merciful (this time). You have to take a decision quickly otherwise the opponent may know that you have something up your sleeve.

The opponent wants to build a Libary and now you have to take a decision: turn thumb down and cancel it or turn thumb up.
If you decide to slap him in the face, do not be astonished that your Spy is uncovered and gone.

The HangingGarden wonder

Now at the beginning of the turn, you can purchase your figure for free provided is available.

Next step
Implement a journal, display messages related to the game. For instance, if "Writing" technology was used the opponent should be informed what had happened.

Pandas DataFrame and Scale Spark DataFrame

In pandas DataFrame (similar but different then Spark's DataFrame), data is provided by series.

import numpy
from pandas import DataFrame, Series
d = {'one' : [1., 2., 3., 4.],  'two' : [4., 3., 2., 1.]}
df = DataFrame(d)

one two 
0 1.0 4.0 
1 2.0 3.0 
2 3.0 2.0 
3 4.0 1.0

In Spark's DataFrame data is provided by features, rows in feature matrix.

val sqlC = new org.apache.spark.sql.SQLContext(sc)
import sqlC.implicits._
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val df = Seq((1.0,4.0),(2.0,3.0),(3.0,2.0),(4.0,1.1)).toDF("one","two")

+---+---+
|one|two|
+---+---+
|1.0|4.0|
|2.0|3.0|
|3.0|2.0|
|4.0|1.1|
+---+---+

Of course, there is a plenty of methods to create DataFrame from a file or any external source. But sometimes it is convenient to create Spark's DataFrame manually using panda's convention.
So I created a simple Scala method for creating DataFrame using series, not features.
Zeppelin notebook

import org.apache.spark.sql._
import org.apache.spark.sql.types._

def createDF(spark: SparkSession, names: Seq[String], series: Seq[Any]*): DataFrame = {
    require(names.length == series.length)
    //    val datas : Seq[Seq[Any]] = List.fill(names.length)(Nil)
    //    val rows : Seq[Row] = List.fill(names.length)(Row())
    val numof: Int = series(0).length
    var rows: Seq[Row] = Nil
    for (i <- 0 until numof) {
      var da: Seq[Any] = Nil
      for (j <- 0 until series.length)
        da = da :+ series(j)(i)
      val r: Row = Row.fromSeq(da)
      rows = rows :+ r
    }
    val rdd = spark.sparkContext.makeRDD(rows)
    // schema
    val schema: Seq[StructField] =
      for (i <- 0 until names.length)
        yield StructField(names(i),
          series(i)(0) match {
            case t: Int => IntegerType
            case t: Double => DoubleType
            case _ => StringType
          },
          false
        )
    spark.createDataFrame(rdd, StructType(schema))
  }

Usage example

val names2 = Seq("one", "tow")
    val seriesone = Seq(1.0,2.0,3.0,4.0)
    val seriestwo = Seq(4.0,3.0,2.0,1.0)
    val da =  createDF(spark, names2,seriesone,seriestwo)
    da.show

Example taken from Udacity course.

val names1 = Seq("countries","gold","silver","bronze")
    val countries = Seq("Russian Fed.", "Norway", "Canada", "United States",
    "Netherlands", "Germany", "Switzerland", "Belarus",
    "Austria", "France", "Poland", "China", "Korea",
    "Sweden", "Czech Republic", "Slovenia", "Japan",
    "Finland", "Great Britain", "Ukraine", "Slovakia",
    "Italy", "Latvia", "Australia", "Croatia", "Kazakhstan")

    val gold = Seq(13, 11, 10, 9, 8, 8, 6, 5, 4, 4, 4, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0)
    val silver = Seq(11, 5, 10, 7, 7, 6, 3, 0, 8, 4, 1, 4, 3, 7, 4, 2, 4, 3, 1, 0, 0, 2, 2, 2, 1, 0)
    val bronze = Seq(9, 10, 5, 12, 9, 5, 2, 1, 5, 7, 1, 2, 2, 6, 2, 4, 3, 1, 2, 1, 0, 6, 2, 1, 0, 1)

    val da1 =  createDF(spark, names1, countries, gold,silver,bronze)
    da1.show(3)