My Wiki

Introduction
I started creating my GitHub Wiki containing some practical guidelines and steps related to HDP and  BigSQL. I do not want to duplicate or retell HortonWorks or IBM official documents and tutorial. Just my success stories and some practical advice to remember and avoid, like Sisyphus, to roll your boulder up that Kerberos hill again and again.
The list and the content are not closed and static, I'm constantly updating and enhancing it.
https://github.com/stanislawbartkowski/wikis
Below is the current content.
LDAP and Kerberos authentication for CentOS
https://github.com/stanislawbartkowski/wikis/wiki/Centos---Kerberos---LDAP
The guidelines for how to set up Kerberos and LDAP authentication for CentOS. It is a good practice to have host systems secured before installing HDP. Although described in many places, I had a hard time to put it together and end up in the loving arms of Kerberos. Particularly, secure LDAP connection and two-ways LDAP security required a lot of patience with gritted teeth.
Dockerized Kerberos
https://github.com/stanislawbartkowski/docker-kerberos
Docker is the buzzword these days. There are several Kerberos on Docker implementations circulating around but I decided to create my own I'm confident with.
HDP and GPFS
https://github.com/stanislawbartkowski/javahotel/tree/hdpgpf
GPFS is the alternative for HDFS. HDP running on the top GPFS has some advantages and disadvantages. Before going live, it is a good idea to practice using a local cluster. Here I'm presenting practical steps on how to set up HDP using GPFS as a data storage. Important: you need a valid IBM GPFS license to do that.
DB2 11 and aggregate UDF
https://github.com/stanislawbartkowski/javahotel/tree/db2aggr
Seems strange but until DB2 11 it was not possible to implement custom aggregate UDF in DB2. This feature was finally added in DB2 11 but I found it not well documented. So it took me some time to create even a simple aggregate UDF but in the end, I made it.
IBM BigSQL, monitoring and maintenance queries
https://github.com/stanislawbartkowski/wikis/wiki/BigSQL,-useful-commands
For some time, I was supporting the IBM client on issues related to IBM BigSQL. During my engagement, I created a notebook with a number of useful SQL queries related to different aspects of maintenance, performance, security etc for copying and pasting.
Dockerized IBM DB2
https://github.com/stanislawbartkowski/docker-db2
Yet another DB2 on Docker. Download the free release of DB2 or, if you are lucky enough, your licensed edition and be happy to get up DB2 by the tap of your finger.
Enable CentOS for Active Directory
https://github.com/stanislawbartkowski/wikis/wiki/CentOS---Active-Directory
Enable CentOS or RedHat for Active Directory authentication is easy comparing to MIT Kerberos/OpenLDAP but still, there are some hooks and nooks to know.
Monitoring tool for IBM BigSQL
https://github.com/stanislawbartkowski/bigsqlmoni
IBM BigSQL/DB2 contains a huge variety of monitoring queries but in most case what is important is not the value but delta. So I created a simple tool to store and provide deltas for some performance and monitoring indicators. But still looking for a way to make practical use of it.
Enable HDP for Active Directory/Kerberos/LDAP
https://github.com/stanislawbartkowski/wikis/wiki/HDP-2.6.5-3.1-and-Active-Directory
Practical steps on implementing Kerberos security in HDP. A simple test to make sure that security is in place.
Enable HDP services for Active Directory/Kerberos/LDAP
https://github.com/stanislawbartkowski/hdpactivedirectory
Wiki: https://github.com/stanislawbartkowski/hdpactivedirectory/wiki
Basic Kerberization enables Kerberos authentication for Hadoop services and makes HDFS secure. Next step is to enable a particular service for Kerberos authentication and LDAP authorization. It is highly recommended to activate Ranger security. The GitHub Wiki attached contains guidelines and practical steps to enable Hadoop services for AD/Kerberos. Every chapter contains also a basic test to make sure that security is enforced and has teeth.
HiBench for HDP
https://github.com/stanislawbartkowski/MyHiBench
HiBench is widely recognized as a leading benchmark tool for Hadoop services. Unfortunately, the development seems to be closed two years ago and I found it difficult to run it in HDP 3.1. Also, the tool seems not to be enabled for Kerberos security. After spending some time trying to find a workaround, I decided to create my own fork of HiBench.
This fork is dedicated to HDP only, the original HiBench can be used also against a standalone instalment of some Hadoop services. It required also redeveloping some Java and Scala code related particularly to Kafka and Spark Streaming.
Several Java/Scala tools to test Kerberos security
HDFS Java client https://github.com/stanislawbartkowski/KafkaSample
Kafka Java client https://github.com/stanislawbartkowski/KafkaSample
Scala Spark Streaming against Kafka https://github.com/stanislawbartkowski/SampleSparkStreaming
Several simple Java/Scala tools to test Kerberos connectivity. The tools come with source code to review. The tools are using as an additional test, next to batch or command line, for testing Kerberos security.

HDP, Ranger and plugins

Problem
I spent several sleepless nights trying to resolve a strange problem related to HDP (HortonWorks Data Platform), Ranger service and plugins.
After installing Ranger and enabling any plugin, an appropriate service entry should be created and visible in Ranger Admin UI. More details are here. In the beginning, a default policy is created which can be customized later according to needs.
But in my environment, the service entry was not created thus blocking any attempt to implement authorization policy. What is more, even disabling/enabling plugin, stopping/restarting the cluster does not make any change, I was unable to conjure the service entry. At some point, I even removed the Ranger, recreated the Ranger database and reinstalled the service again from scratch, but it did not help.
Solution
Finally, after carefully browsing through the log files, I found the solution. The culprit is the local directory /etc/ranger. There is a subdirectory reflecting the service entry in Ranger Admin UI.

ls /etc/ranger/MyCluster_hadoop/
cred.jceks
policycache

This directory contains a copy of ranger/service policy and is used as a recovery point in case of the database failure. It seems that after enabling the plugin if the service discovers this cache, the ranger/service policy is recreated but in this scenario, the Ranger Admin UI service entry is not restored. This cache is not removed after disabling the plugin and even after removing the whole Ranger service.
Unfortunately, it is not documented and badly implemented.
The solution is to switch off the plugin, manually remove the /etc/ranger/{service name} directory and switch on the plugin again. The service entry and default policy are recreated.
Keep in mind that the directory /etc/ranger/{service name} is located on the host where the appropriate service is installed, not the Ranger service host.

My HiBench

Problem
HiBench is regarded as a leading benchmark in the BigData world according to this webpage. But while trying to run it in my test HDP 2.6.5 or 3.1environment, 2.6.5 or 3.1,  I found it very frustrating, particularly when the cluster is secured by Kerberos.
Because the project seems to be abandoned two years ago, I started implementing manual patches to the code, it is described here. But at some point, I came to a dead end. Kafka 0.8 client does not support Kerberos, Kerberos support was added in Kafka 0.10 client, but Kafka 0.10 is not backward compatible and moving there required redeveloping substantial part of Kafka related code in Java and Scala.
Solution
So finally I decided to make my own fork of HiBench project and split completely from the trunk. The result is a new GitHub project here.

Main changes include:

• Kerberos support
• Kafka 2.0 implemented
• HDP 3.1/Apache Hadoop 3.1 support
• Hive 3.1 support, the standard HiBench was developed for Hive 0.17 and does not talk to 3.1

Features lost comparing to standard HiBench

• Only HDP is supported and tested, standalone Spark, Apache Hadoop, Kafka etc are not supported. 
• Only support for HDP 2.6.5 and HDP 3.1 is implemented. Support for an older version of HDP, Spark, Scala is abandoned.
• Support for Apache Flink and Gearpunmo is removed.

Tests

The project was tested in two HDP 3.1 and HDP 2.6.5 environments, tiny in local KVM cluster and tiny/small/large in a larger multi-host cluster.

Future plans

• Review Kafka streaming benchmark, while analyzing the code I found some mysteries out there
• The same for Dfsio benchmark test
• Test in wire encryption secured cluster

TPC-DS benchmark and HDP

I conducted TPC-DS benchmark on HDP 2.6.5 cluster and compared three SQL engines: Hive, SparkSQL and IBM BigSQL.
The results are published here.
The data size scaling factor is 100, meaning 100 GB of data. It is not a qualification database in terms of TPC-DS specification and the result should not be used for any formal comparison. I'm going only to run the test in different HDP environments and collect the results.
The queries were executed against ORC and Parquet files.
The ad-hoc impressions:

• Hive falls behind as expected because of M/R paradigm.
• Parquet format: IBM BigSQL and SparkSQL go head to head. Some queries execute faster by BigSQL, some faster by SparkSQL.
• SparkSQL behaves worse while running against ORC, even comparing to Hive. 

The benchmark test was executed using TPC-DS framework available here. The Throughput Test requires running four queries in parallel. It was achieved by launching ptest.sh script four times with the parameter describing the stream number.

• nohup ./ptest.sh 0 &
• nohup ./ptest.sh 1 &
• nohup ./ptest.sh 2 &
• nohup ./ptest.sh 3 &

In a tiny environment, the cluster was adjusted according to the demands of a particular SQL engine. Hive and SparSQL were using four dedicated yarn queues to achieve better parallelism level. BigSQL was configured to remove memory restraint, otherwise, during Throughput Test, BigSQL would fail because of lack of available memory. After the test, the specific configuration should be reverted.

The tables were created using Hive external table feature to deploy text file and then target table was loaded by the sequence:

• LOAD DATA INPATH hdfs_path OVERWRITE INTO TABLE external_table
• CREATE TABLE .... STORED AS PARQUET/OCR as SELECT * FROM external_table

Next steps

• The TPC-DS tables are not partitioned. Try to partition them and take advantage of "partition elimination" feature in BigSQL. This feature optimizes join queries by ignoring partitions not covered by join predicates. Should boost BigSQL performance.
• Migrate to HDP 3.1 and compare the results in the same hardware framework.

Docker DB2

Docker and IBM DB2
There is an official IBM DB2 Docker container. But it is quite easy to create own Docker image using freely available IBM DB2-Express C to keep up with the latest version and have more flexibility.
The Dockerfile and description are available at github project. The Dockerfile allows also to consume commercial DB2 AESE version and apply the current FixPack.
Preparing an image takes some time but when ready, rule them all. Run, start, stop,  and destroy DB2 instance with a tap of your finger.

Civilization The Board Game, next version

Introduction
I deployed a new version of my computer implementation of  Civilization The Board Game. The implementation consists of three parts:

• Civilization Engine 
• Civilization Web Interface
• Demo. Demo version is deployed to Heroku, free quota. Please wait several minutes unless the dyno is brought back to life.

New features
The journal was added. There is a window panel where the activities of the player and the opponent are outputted.

The journal panel can be brought to live by clicking the "Journal" button. It can be hidden afterwards by clicking the "Hide" button. The panel can be dragged across the screen but cannot be scaled. The actions are added to the panel dynamically during the progress of the game.
There were several problems to be resolved. Some messages are "public", meaning the message is visible for both players. For instance, if the player buys the building, the information is visible for both of them. There are also private messages visible only for the player and not replicated to the opponent. Some messages are modified according to the recipient. For instance: if the player is buying the unit, the unit type and the unit strength is reported to him. The same information is sent to the opponent but the unit strength is removed as private.
Next steps
Some culture cards are to be implemented.

MIT Kerberos, Ubuntu and Docker

I was using this version of Dockerized Kerberos but finally, I decided the prepare my own based on Ubuntu. Ubuntu has a smaller footprint then Centos, 230 MB against 650 MB. The Dockerfile and usage description is available here, as GitHub project.
Enjoy.