My HiBench

Problem
HiBench is regarded as a leading benchmark in the BigData world according to this webpage. But while trying to run it in my test HDP 2.6.5 or 3.1environment, 2.6.5 or 3.1,  I found it very frustrating, particularly when the cluster is secured by Kerberos.
Because the project seems to be abandoned two years ago, I started implementing manual patches to the code, it is described here. But at some point, I came to a dead end. Kafka 0.8 client does not support Kerberos, Kerberos support was added in Kafka 0.10 client, but Kafka 0.10 is not backward compatible and moving there required redeveloping substantial part of Kafka related code in Java and Scala.
Solution
So finally I decided to make my own fork of HiBench project and split completely from the trunk. The result is a new GitHub project here.

Main changes include:

• Kerberos support
• Kafka 2.0 implemented
• HDP 3.1/Apache Hadoop 3.1 support
• Hive 3.1 support, the standard HiBench was developed for Hive 0.17 and does not talk to 3.1

Features lost comparing to standard HiBench

• Only HDP is supported and tested, standalone Spark, Apache Hadoop, Kafka etc are not supported. 
• Only support for HDP 2.6.5 and HDP 3.1 is implemented. Support for an older version of HDP, Spark, Scala is abandoned.
• Support for Apache Flink and Gearpunmo is removed.

Tests

The project was tested in two HDP 3.1 and HDP 2.6.5 environments, tiny in local KVM cluster and tiny/small/large in a larger multi-host cluster.

Future plans

• Review Kafka streaming benchmark, while analyzing the code I found some mysteries out there
• The same for Dfsio benchmark test
• Test in wire encryption secured cluster

TPC-DS benchmark and HDP

I conducted TPC-DS benchmark on HDP 2.6.5 cluster and compared three SQL engines: Hive, SparkSQL and IBM BigSQL.
The results are published here.
The data size scaling factor is 100, meaning 100 GB of data. It is not a qualification database in terms of TPC-DS specification and the result should not be used for any formal comparison. I'm going only to run the test in different HDP environments and collect the results.
The queries were executed against ORC and Parquet files.
The ad-hoc impressions:

• Hive falls behind as expected because of M/R paradigm.
• Parquet format: IBM BigSQL and SparkSQL go head to head. Some queries execute faster by BigSQL, some faster by SparkSQL.
• SparkSQL behaves worse while running against ORC, even comparing to Hive. 

The benchmark test was executed using TPC-DS framework available here. The Throughput Test requires running four queries in parallel. It was achieved by launching ptest.sh script four times with the parameter describing the stream number.

• nohup ./ptest.sh 0 &
• nohup ./ptest.sh 1 &
• nohup ./ptest.sh 2 &
• nohup ./ptest.sh 3 &

In a tiny environment, the cluster was adjusted according to the demands of a particular SQL engine. Hive and SparSQL were using four dedicated yarn queues to achieve better parallelism level. BigSQL was configured to remove memory restraint, otherwise, during Throughput Test, BigSQL would fail because of lack of available memory. After the test, the specific configuration should be reverted.

The tables were created using Hive external table feature to deploy text file and then target table was loaded by the sequence:

• LOAD DATA INPATH hdfs_path OVERWRITE INTO TABLE external_table
• CREATE TABLE .... STORED AS PARQUET/OCR as SELECT * FROM external_table

Next steps

• The TPC-DS tables are not partitioned. Try to partition them and take advantage of "partition elimination" feature in BigSQL. This feature optimizes join queries by ignoring partitions not covered by join predicates. Should boost BigSQL performance.
• Migrate to HDP 3.1 and compare the results in the same hardware framework.

Docker DB2

Docker and IBM DB2
There is an official IBM DB2 Docker container. But it is quite easy to create own Docker image using freely available IBM DB2-Express C to keep up with the latest version and have more flexibility.
The Dockerfile and description are available at github project. The Dockerfile allows also to consume commercial DB2 AESE version and apply the current FixPack.
Preparing an image takes some time but when ready, rule them all. Run, start, stop,  and destroy DB2 instance with a tap of your finger.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features
The journal was added. There is a window panel where the activities of the player and the opponent are outputted.

The journal panel can be brought to live by clicking the "Journal" button. It can be hidden afterwards by clicking the "Hide" button. The panel can be dragged across the screen but cannot be scaled. The actions are added to the panel dynamically during the progress of the game.
There were several problems to be resolved. Some messages are "public", meaning the message is visible for both players. For instance, if the player buys the building, the information is visible for both of them. There are also private messages visible only for the player and not replicated to the opponent. Some messages are modified according to the recipient. For instance: if the player is buying the unit, the unit type and the unit strength is reported to him. The same information is sent to the opponent but the unit strength is removed as private.
Next steps
Some culture cards are to be implemented.

MIT Kerberos, Ubuntu and Docker

I was using this version of Dockerized Kerberos but finally, I decided the prepare my own based on Ubuntu. Ubuntu has a smaller footprint then Centos, 230 MB against 650 MB. The Dockerfile and usage description is available here, as GitHub project.
Enjoy.

Król Roger, Teatr Narodowy

11 grudnia byliśmy na przedstawieniu Króla Rogera w Teatrze Narodowym. Było to jednak przedstawienie do oglądania, nie do słuchania.
Król Roger II to postać historyczna, słynący z mądrości i pobożności średniowieczny król Sycylii. W operze Szymanowskiego w państwie króla pojawia się tajemniczy Pasterz, prorok nowego boga, o którym śpiewa "Mój Bóg jest piękny jako ja". Przybycie Pasterza na dwór króla wyzwala żywioły, które niczym wir mielą i przemieniają wszystko, dwór, królestwo i Roksanę, ukochaną żonę króla, zostawiając Rogera jako samotnego pielgrzyma.
Sama opera nie jest łatwa w odbiorze, to nie jest historia, którą trzeba opowiedzieć od początku do końca. To ciąg trzech scen, w pierwszej widzimy królestwo Rogera, harmonijne połączenie władzy świeckiej i kościelnej, w drugiej jest konfrontacja Rogera i Pasterza, w trzeciej Roger jako odarty z atrybutów królewskich pielgrzym wędruje w przemienionym mocą Pasterza świecie szukając Roksany.
Operę można odczytać na wiele sposobów. Realizacja w Teatrze Wielkim oczywiście ucieka jak najdalej od średniowiecznego kontekstu. Dwór królewski to zarząd korporacji z Rogerem jako CEO. Pośrodku jest ogromny stół konferencyjny, Roksana to zblazowana dama w wyższych sfer. Mamy szklane klatki, jest łazienka z marmurową umywalką, wózek inwalidzki, migające stroboskopowe światła, lustra, krwawe sceny w estetyce filmów gore, maski z rogami muflona, zwielokrotnione postacie, wizualizacje video, ściana mieniących się cyferek jak z Matrixa. Końcowa scena odrealnionego świata jest odwołaniem do finału "2001 Odyseja Kosmiczna", pojawia się nawet małe dziecko niczym embrion Bowmana w słynnym filmie Kubricka. Aby się przemienić i zrozumieć, musimy narodzić się na nowo.
Niestety, w tym wszystkim zagubiła się sama muzyka, największa wartość opery. Tak jak w końcowej scenie 2 aktu, gdzie z podwieszonego głową do dołu Rogera ścieka krew na leżącą na stole Roksanę, złożony został tutaj w ofierze sam Szymanowski i wszystko zostało postawione na odwrót, ogon zamerdał psem, forma przerosła treść. Wprowadzająca wizualizacja wideo, gdzie pełzający wąż symbolizuje zgniliznę toczącą pozornie perfekcyjne królestwo/korporację, w rzeczywistości zapowiada klęskę samego spektaklu.
Muzyka jest tutaj wyłącznie sound-trackiem do efektownego widowiska, jakby nie miała odwagi unieść się z kanału orkiestrowego. Początkowy chór "Hagios! Hagios!" jest odtwarzany z taśmy, gdy na końcu przedstawienia do oklasków wybiega chór, na widowni czuć konsternację skąd te osoby się tutaj wzięły. Nawet zapowiedź, że wykonawca tytułowej roli, Łukasz Goliński, nie jest w pełni dysponowany, dla spektaklu nie miała wielkiego znaczenia, gdyż forma solistów nie była tutaj istotna, w zasadzie całą warstwę muzyczną łącznie z orkiestrą można było odtworzyć z taśmy.
Miała być opera, a było wyłącznie widowisko światło i dźwięk z akcentem na pierwszy człon.

TPC-DS benchmark

Inspiration
TCP-DS benchmark is an industry standard tool to evaluate the performance of the relational database engine. The tool can be used not only to leverage the general efficiency. The database administrator or DevOp engineer can utilize the tools to estimate the robustness of a particular database installation by comparing it against a reference instance. Also, it can be useful to verify the database health after upgrade or maintenance. Another use case is to gauge the performance gain after tunning or scaling.
The main disadvantage of the standard package is that it is not easy to use, requires a number of manual adjustment and fixes. It makes the tool less reliable and the process less repeatable. To compare the result against the reference results we have to be sure that the benchmark for both environments, the reference, and the current, are prepared and executed using the same queries and according to the same pattern.
Solution
I created a simple solution to make the task very simple. The project is available in GitHub.
The following databases are supported so far: Oracle, IBM DB2, MySQL/MariaDB, PostgreSQL, Hive, SparkSQL, Netezza, and IBM BigSQL.
Also, a wiki is available having hints on how to set up a test for a particular database. Having the tool, the TCP-DS benchmark is ready to use. Just download, configure for a particular environment and run.
So far, Qualify and Power Tests are implemented.
Queries
The TPC-DS queries are not ready to use out of the box for all SQL engines. The specification allows some minor fixes to make them executable for a particular database. To avoid keeping a different version of queries, I decided to modify them on the fly. I also apply only changes possible to implement through the simple string or regular expression text replacement.
Test Validation
I was unable to match the answer data set provided against any output. It requires further investigation. So I decided to use Oracle output as a reference result set. Unfortunately, even Oracle result does match all of them and is not fully reliable. The life is never an easy road free of stone.
Next steps
Throughput and Data Maintenance tests.