Blog do projektu Open Source JavaHotel

czwartek, 22 listopada 2018

Polymer 3, upgrade to 3.01

I was blocked by a strange problem after upgrading from 3.0.0-pre.19 to 3.0.1. Suddenly  I found an extraordinarily long delay before displaying the game board for the first time.



Usually, it took several seconds to move from the first screen to the third. After the upgrade, the time was prolonged to almost 50 seconds making it almost useless. It is not easy to find a bottleneck in an asynchronous framework. So firstly I discovered it is the map (12 * 8 tiles) which is the cause of the delay. Then I gritted my teeth and after several sleepless nights including meticulously comparing old and new JS files, I pinned down the culprit.
The problem was caused by iron-resizable-behavior/iron-resizable-behavior.js file. Next to the end, there is a slightly new code.

_requestResizeNotifications: function () {
    if (!this.isAttached) {
      return;
    }

    if (document.readyState === 'loading') {
      var _requestResizeNotifications = this._requestResizeNotifications.bind(this);

      document.addEventListener('readystatechange', function readystatechanged() {
        document.removeEventListener('readystatechange', readystatechanged);

        _requestResizeNotifications();
      });
    } else {
      this._findParent();

      if (!this._parentResizable) {
        // If this resizable is an orphan, tell other orphans to try to find
        // their parent again, in case it's this resizable.
        ORPHANS.forEach(function (orphan) {
          if (orphan !== this) {
            orphan._findParent();
          }
        }, this);
        window.addEventListener('resize', this._boundNotifyResize);
        this.notifyResize();
      } else {
        // If this resizable has a parent, tell other child resizables of
        // that parent to try finding their parent again, in case it's this
        // resizable.
//        this._parentResizable._interestedResizables.forEach(function (resizable) {
//          if (resizable !== this) {
//            resizable._findParent();
//          }
//        }, this);
      }
    }
  },
  _findParent: function () {
    this.assignParentResizable(null);
    this.fire('iron-request-resize-notifications', null, {
      node: this,
      bubbles: true,
      cancelable: true
    });

    if (!this._parentResizable) {
      ORPHANS.add(this);
    } else {
      ORPHANS.delete(this);
    }
  }
};
The temporary workaround is to comment out the code.
        // If this resizable has a parent, tell other child resizables of
        // that parent to try finding their parent again, in case it's this
        // resizable.
//        this._parentResizable._interestedResizables.forEach(function (resizable) {
//          if (resizable !== this) {
//            resizable._findParent();
//          }
//        }, this);
      }
Unfortunately, I'm unable to provide any explanation for that and the solution is nothing more than kicking the can down the road. Obviously, the code inside the else clause is doing something CPU thirsty but that is all I can make out of it. But it works for me.

wtorek, 30 października 2018

BigSQL, joins and partition elimination.

Partition elimination in joins
One of the advantages of BigSQL over Hive is enabling partition elimination also during join execution. It is described in this article.
It can be explained using a simple example.
db2 "CREATE HADOOP TABLE p_x(n VARCHAR(64)) PARTITIONED BY (x int) STORED AS PARQUETFILE"
db2 "CREATE HADOOP TABLE p_x(n VARCHAR(64)) PARTITIONED BY (x int) STORED AS ORC"
db2 "insert into p_x values('a',0)"
db2 "insert into p_x values('b',1)"
db2 "insert into p_x values('c',2)"
db2 "insert into p_x values('d',3)"
db2 "insert into p_x values('e',4)"
db2 "insert into p_x values('f',5)"
db2 "insert into p_x values('g',5)"
db2 "insert into p_x values('h',6)"
db2 "insert into p_x values('i',6)"
db2 "insert into p_x values('j',7)"
db2 "insert into p_x values('k',8)"
db2 "insert into p_x values('l',9)"
db2 "CREATE HADOOP TABLE p_y(n VARCHAR(64),x int)"
db2 "insert into p_y values('e',4)"
db2 "insert into p_y values('f',5)"
db2 "insert into p_y values('g',5)"
Then run a straightforward join query.
db2 "select * from p_x,p_y where p_x.x = p_y.x"
The range of values in the p_y table is 4 to 5 so in order to resolve this join, it is enough to scan only corresponding partitions in p_x table.
It can be detected by looking into BigSQL log. But firstly DEBUG level for BigSQL Scheduler should be turned on.

BigSQL->Configs->Advanced bigsql-log4j The following two parameters should be modified:
  • log4j.logger.com.ibm.biginsights.bigsql.scheduler.GlobalLog=DEBUG 
  • log4j.logger.com.ibm=ALL
Then BigSQL is to be restarted to get new settings taking effect.
The query partition elimination is reported in /var/ibm/bigsql/logs/bigsql-sched.log
DEBUG com.ibm.biginsights.bigsql.scheduler.server.expr.ExprUtils [pool-1-thread-5] : found column with partition-key. db2ColumnIndex: 1
DEBUG com.ibm.biginsights.bigsql.scheduler.server.StorageHandlerScanState [pool-1-thread-5] : [createScanState] Partition elimination expr:  (  ( x <= 5 )  AND  ( x >= 4 )  ) 
DEBUG com.ibm.biginsights.bigsql.scheduler.server.StorageHandlerScanState [pool-1-thread-5] : found default/dummy partition. skipping. HdfsPartition{fileDescriptors=[]}
[eliminatePartition]partition elimination: checking partition: x=0
[eliminatePartition]partitionEliminated? x=0 true
[eliminatePartition]partition elimination: checking partition: x=1
[eliminatePartition]partitionEliminated? x=1 true
[eliminatePartition]partition elimination: checking partition: x=2
[eliminatePartition]partitionEliminated? x=2 true
[eliminatePartition]partition elimination: checking partition: x=3
[eliminatePartition]partitionEliminated? x=3 true
[eliminatePartition]partition elimination: checking partition: x=4
[eliminatePartition]partitionEliminated? x=4 false
The locations are: [TScanRangeLocation(host_idx:0, volume_id:-1), TScanRangeLocation(host_idx:1, volume_id:-1)]
[eliminatePartition]partition elimination: checking partition: x=5
[eliminatePartition]partitionEliminated? x=5 false
[createScanState] The locations are: [TScanRangeLocation(host_idx:2, volume_id:-1), TScanRangeLocation(host_idx:1, volume_id:-1)]
[createScanState] The locations are: [TScanRangeLocation(host_idx:2, volume_id:-1), TScanRangeLocation(host_idx:0, volume_id:-1)]
[eliminatePartition]partition elimination: checking partition: x=6
[eliminatePartition]partitionEliminated? x=6 true
[eliminatePartition]partition elimination: checking partition: x=7
[eliminatePartition]partitionEliminated? x=7 true
[eliminatePartition]partition elimination: checking partition: x=8
[eliminatePartition]partitionEliminated? x=8 true
[eliminatePartition]partition elimination: checking partition: x=9
[eliminatePartition]partitionEliminated? x=9 true
[createScanState]Finished partition elimination. partition-elimination-stats (eliminated/total): 8 / 10 Took 124 milliseconds
2018-10-30 00:51:47,775 INFO com.ibm.biginsights.bigsql.scheduler.server.cache.TableLock [pool-1-thread-5] : [removeReadLock(String)]Removed Read Lock on table sb.p_x: 0
2018-10-30 00:51:47,775 DEBUG com.ibm.biginsights.bigsql.scheduler.Dev.Assignment [pool-1-thread-5] : [assignSplits] Workers node-numbers: [1, 2, 3]
2018-10-30 00:51:47,775 DEBUG com.ibm.biginsights.bigsql.scheduler.Dev.Assignment [pool-1-thread-5] : [assignSplits] Workers ip to node-numbers: {172.16.186.139=[3], 172.16.186.104=[2], 172.16.186.9=[1]}
As one can see, all partitions except x=4 and x=5 are ignored during execution of the join query.
Bigger example
Of course, partition elimination is nice but what we are really interested in is the performance boost. How to measure it?
I created a simple project. The general idea is to prepare two huge tables, one non-partitioned and the second partitioned and run identical join query against the table.  The source code and detailed description can be found here.
The final result?
For non-partitioned table:
[sbartkowski@oc0522068411 bigjoin]$ time ./run.sh
Run runnonquery.sql script
PASSED

real 0m16.521s
user 0m0.036s
sys 0m0.100s
For partitioned table and partition elimination in place:
[sbartkowski@oc0522068411 bigjoin]$ time ./run.sh
Run runnonquery.sql script
PASSED

real 0m8.860s
user 0m0.037s
sys 0m0.095s
So the performance improved twice which should come as no surprise.
The same query submitted in Hive against the same tables executes as below. Obviously Hive has some way to go.
select D.ID,max(T.tm) from monit.testdim AS D , monit.testpart AS T WHERE T.part=D.part GROUP BY D.ID;

--------------------------------------------------------------------------------
+---------+-----------------------------+--+
|  d.id   |             _c1             |
+---------+-----------------------------+--+
| PART 5  | 2014-09-18 00:19:11.893212  |
| PART 2  | 2014-09-18 00:03:35.599702  |
| PART 4  | 2014-09-18 00:26:17.660366  |
| PART 3  | 2014-09-18 00:24:47.644779  |
+---------+-----------------------------+--+
4 rows selected (41,863 seconds)
The BigSQL is really big.

niedziela, 30 września 2018

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:
New features
  • "Writing" technology implemented, the player can cancel city action of his opponent
  • "HangingGarden" wonder implemented
"Writing" technology
If you have "Writing" technology and was able to recruit the Spy then you can tell "every dog has its day".
The execution of the player city action is postponed unless his opponent decides to cancel it or be merciful (this time). You have to take a decision quickly otherwise the opponent may know that you have something up your sleeve.

The opponent wants to build a Libary and now you have to take a decision: turn thumb down and cancel it or turn thumb up.
If you decide to slap him in the face, do not be astonished that your Spy is uncovered and gone.

The HangingGarden wonder
Now at the beginning of the turn, you can purchase your figure for free provided is available.

Next step
Implement a journal, display messages related to the game. For instance, if "Writing" technology was used the opponent should be informed what had happened.

piątek, 31 sierpnia 2018

Pandas DataFrame and Scale Spark DataFrame

In pandas DataFrame (similar but different then Spark's DataFrame), data is provided by series.

import numpy
from pandas import DataFrame, Series
d = {'one' : [1., 2., 3., 4.],  'two' : [4., 3., 2., 1.]}
df = DataFrame(d)
one two 
0 1.0 4.0 
1 2.0 3.0 
2 3.0 2.0 
3 4.0 1.0
In Spark's DataFrame data is provided by features, rows in feature matrix.
val sqlC = new org.apache.spark.sql.SQLContext(sc)
import sqlC.implicits._
import org.apache.spark.sql._
import org.apache.spark.sql.types._
val df = Seq((1.0,4.0),(2.0,3.0),(3.0,2.0),(4.0,1.1)).toDF("one","two")
+---+---+
|one|two|
+---+---+
|1.0|4.0|
|2.0|3.0|
|3.0|2.0|
|4.0|1.1|
+---+---+
Of course, there is a plenty of methods to create DataFrame from a file or any external source. But sometimes it is convenient to create Spark's DataFrame manually using panda's convention.
So I created a simple Scala method for creating DataFrame using series, not features.
Zeppelin notebook
import org.apache.spark.sql._
import org.apache.spark.sql.types._

def createDF(spark: SparkSession, names: Seq[String], series: Seq[Any]*): DataFrame = {
    require(names.length == series.length)
    //    val datas : Seq[Seq[Any]] = List.fill(names.length)(Nil)
    //    val rows : Seq[Row] = List.fill(names.length)(Row())
    val numof: Int = series(0).length
    var rows: Seq[Row] = Nil
    for (i <- 0 until numof) {
      var da: Seq[Any] = Nil
      for (j <- 0 until series.length)
        da = da :+ series(j)(i)
      val r: Row = Row.fromSeq(da)
      rows = rows :+ r
    }
    val rdd = spark.sparkContext.makeRDD(rows)
    // schema
    val schema: Seq[StructField] =
      for (i <- 0 until names.length)
        yield StructField(names(i),
          series(i)(0) match {
            case t: Int => IntegerType
            case t: Double => DoubleType
            case _ => StringType
          },
          false
        )
    spark.createDataFrame(rdd, StructType(schema))
  }
Usage example
val names2 = Seq("one", "tow")
    val seriesone = Seq(1.0,2.0,3.0,4.0)
    val seriestwo = Seq(4.0,3.0,2.0,1.0)
    val da =  createDF(spark, names2,seriesone,seriestwo)
    da.show
Example taken from Udacity course.
val names1 = Seq("countries","gold","silver","bronze")
    val countries = Seq("Russian Fed.", "Norway", "Canada", "United States",
    "Netherlands", "Germany", "Switzerland", "Belarus",
    "Austria", "France", "Poland", "China", "Korea",
    "Sweden", "Czech Republic", "Slovenia", "Japan",
    "Finland", "Great Britain", "Ukraine", "Slovakia",
    "Italy", "Latvia", "Australia", "Croatia", "Kazakhstan")

    val gold = Seq(13, 11, 10, 9, 8, 8, 6, 5, 4, 4, 4, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0)
    val silver = Seq(11, 5, 10, 7, 7, 6, 3, 0, 8, 4, 1, 4, 3, 7, 4, 2, 4, 3, 1, 0, 0, 2, 2, 2, 1, 0)
    val bronze = Seq(9, 10, 5, 12, 9, 5, 2, 1, 5, 7, 1, 2, 2, 6, 2, 4, 3, 1, 2, 1, 0, 6, 2, 1, 0, 1)

    val da1 =  createDF(spark, names1, countries, gold,silver,bronze)
    da1.show(3)

wtorek, 21 sierpnia 2018

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:
Several new features are implemented: conquering the alien city and victories.  Also, the performance is improved by caching.
Conquering the city and taking the double loot
The alien city can be attacked and pulled down. After winning (or losing) the winner can take the double loot.

According to game rules, the double loot is at the winner's disposal. Some loots are ranked as one, for instance, stealing the trade and some are ranked as two, for instance, the new technology. The number of loot cannot exceed the two.
Military Victory
If an alien capital city is conquered, the victor will get the medal of the Military Victory.

Culture Victory
After reaching the top of the culture track, the Culture Victory is announced.

Economic Victory
When a player puts aside at least 16 golden coins is acclaimed as Economic Winner by the astonished world.

Technology Victory
When a player piles up the stack of technology high enough, the world recognizes him as Steve Jobs and Elon Musk in one person.



Performance improvement
I started to sample the Civilization Engine using VisualVM profiler and immediately discovered the first bottleneck. The game board is every time recreated by executing all commands starting from the initial layout and it was the main resource consumer. The solution was pretty simple, just cache the current board and the latency was reduced even for my free Heroku quota, 1 CPU and 0.5 GB of memory.
Next steps
Unfortunately, it is high time to implement "hanging" feature like "Writing" :
"Cancel a city action being performed by another player (may not cancel resource ability)". I do not have a clear idea of how to accomplish it. One player has to be triggered that another player executes a command eligible for the action and should have a choice: let it or cancel it.

poniedziałek, 20 sierpnia 2018

HortonWorks, Atlas and Hive integration not working

Problem
I installed HDP 2.6.5 and tried to enable Hive for Atlas. But no Hive action was reflected in Atlas. I doublechecked the configuration and found it correct. There were no error messages neither in Hive logs nor in Atlas logs. Also, it looked that Atlas Hive Hook was activated for every action but it did not land in Atlas dashboard.
018-08-20 18:39:56,696 INFO  [HiveServer2-Background-Pool: Thread-899]: log.PerfLogger (PerfLogger.java:PerfLogBegin(149)) -
2018-08-20 18:39:56,697 INFO  [HiveServer2-Background-Pool: Thread-899]: log.PerfLogger (PerfLogger.java:PerfLogEnd(177)) -
Solution
The solution was very simple and was related to Kafka. Atlas is using Kafka as the intermediate medium to push data into Atlas realm. I installed only single Kafka broker but configuration requires at least 3 Kafka brokers to create a quorum. After adding two lacking Kafka brokers the flow was unlocked and all current and pending request found their way to Atlas.

środa, 1 sierpnia 2018

Polymer 2 to Polymer 3

Introduction
I decided to upgrade from Polymer 2 to Polymer 3.  According to
"we've made a smooth upgrade path our top priority for Polymer 3.0. Polymer's API remains almost unchanged, and we're providing an upgrade tool (polymer-modulizer) that will automatically handle most of the work in converting your 2.x-based elements and apps to 3.0.". 
Encouraged by this advertisement, I run "modulizer --out ." and ... It depends on what one means by "most of the work" and "smooth".

Module names instead of path names
Instead of import pathnames like:
<href="../bower_components/polymer/polymer-element.html" rel="import"></link>
module names are used:
import {PolymerElement, html} from '@polymer/polymer/polymer-element.js'
I understand the rationale behind that but Chrome browser I'm using obviously does not. It still insists that @polymer is a directory name and demands the path name to resolve it. Adding the path names manually is challenging because not only custom elements should be touched but also all internal Polymer elements. So it looks that some kind of "build" process is necessary after every upgrade. After some trials, errors and research I ended up using:
polymer build --module-resolution node
which does the job. Then I replace current "node_modules" directory with "build/default/node_modules". In the case of the Polymer upgrade,  "yarn install" should be launched beforehand. Previously, no build process was required. It was enough to run "bower install".

ReferenceError: IntlMessageFormat is not defined
Uncaught (in promise) ReferenceError: IntlMessageFormat is not defined
    at HTMLElement. (app-localize-behavior.js:285)
    at runMethodEffect (property-effects.js:905)
    at Function._evaluateBinding (property-effects.js:3079)
    at Object.runBindingEffect [as fn] (property-effects.js:648)
    at runEffectsForProperty (property-effects.js:169)
    at runEffects (property-effects.js:131)
    at HTMLElement._propagatePropertyChanges (property-effects.js:1933)
    at HTMLElement._propertiesChanged (property-effects.js:1891)
    at HTMLElement._flushProperties (properties-changed.js:370)
    at HTMLElement._flushProperties (property-effects.js:1731)
Unfortunately, "polymer build" does not catch "node_modules/intl-messageformat" package and it is absent after build. So we have to reinstall the package again.
npm install intl-messageformat 
ReferenceError: IntlMessageFormat is not defined
Although "intl-messageformat" is downloaded, it should be also imported somewhere to make "IntMessageFormat" class visible.
I ended up patching manually "node_modules/@polymer/app-localize-behavior/app-localize-behavior.js" file.
import "../polymer/polymer-legacy.js";
import "../iron-ajax/iron-ajax.js";
import "../../intl-messageformat/dist/intl-messageformat.js";
Uncaught TypeError: Cannot set property 'IntlMessageFormat' of undefined
Uncaught TypeError: Cannot set property 'IntlMessageFormat' of undefined
    at main.js:7
    at main.js:7
It was a stab in the back because Chrome does not tell where this disaster happens. After some time I found a hint here.
The solution is to patch manually "node_modules/intl-messageformat/dist/intl-messageformat.js".
At the end of the file replace:
    var src$main$$default = $$core$$default;
    this['IntlMessageFormat'] = src$main$$default;
}).call(window);

with:
    var src$main$$default = $$core$$default;
    this['IntlMessageFormat'] = src$main$$default;
}).call(this);
Uncaught ReferenceError: KeyframeEffect is not defined
This exception was thrown from paper-dropdown-menu element. Instead of spending time trying to resolve it, I decided to remove this dependency and replaced it with something different. Replace namespace with imports
In Polymer 2 I was using Polymer namespace for custom classes.
olymer.CivData = function(superClass) {

        return class extends Polymer.CivLocalize(superClass) {

                constructor() {
                        super();
                }

..........
class CivTLevel extends Polymer.CivData(Polymer.Element) {

    static get is() {
        return 'civ-tlevel';
        }
Polymer 3 enforces transforming all class to ES6 modules. So sequence above should be replaced by:
import { CivLocalize} from "../js/civ-localize.js";

export const CivData = function (superClass) {
  return class extends CivLocalize(superClass) {
    constructor() {
      super();
    }

...........
import { html } from "../node_modules/@polymer/polymer/lib/utils/html-tag.js";
import { PolymerElement } from "../node_modules/@polymer/polymer/polymer-element.js";
import { CivData} from "../js/civ-data.js";

class CivTLevel extends CivData(PolymerElement) {
  static get template() {

"modulizer" utility has nothing to do with that, it should be done manually.
Conclusion
"Smooth upgrade" ended up in several sleepless nights. But in the end, I made it. On the whole, I really like this ES6 module, it provides a good encapsulation and isolation which is very important in every programming language including JavaScript.