Simple RestService library

Motivation
REST API is the mean of choice to communicate between different loosely coupled applications. There are plenty of REST API implementations but I was looking for a solution as simple as possible with minimal external dependencies or prerequisites. Finally, I ended up with the compact library utilizing existing in Java JDK HttpServer.
Links
Intellij IDEA project, source code and javadoc.
Sample project utilizing the RestService project.
Another project having RestService dependency.
Highlights

• Very lightweight, no external dependency, just Java JDK.
• Can be dockerized, sample Dockerfile.
• Adds-on making a life of developer easier.
• Validating and extracting URL query parameters including type control.
• Upload data
• CORS relaxation
• Sending data with valid HTTP response code.
• "Dynamic" and "static" REST API call. "Dynamic" means that specification of particular REST API endpoint can be defined after the request reached the server but before handling the request thus allowing providing different custom logic according to the URL path.

Usage

The service class should extend RestHelper.RestServiceHelper abstract class and implements two methods:

• getParams : delivers REST API call specification (look below) including URL query parameter definition. The method is called after the REST API request accepted by HTTP Server but before validating and running the call. 
• servicehandle: custom logic to serve the particular REST API endpoint. The method should conclude handling the request by proper "produceresponse" call. The "servicehandle" can take a URL query parameters and utilize several helper methods.

REST API specification

The REST API endpoint specification is defined through RestParams class. The specification consists of:

• HTTP request method: GET, POST, PUT etc
• List of allowed  URL query parameters. Three parameters type are supported: BOOLEAN, INT and STRING (meaning any other).
• CORS should be relaxed for this particular endpoint.
• Response type content (TEXT, JSON or not specified), Content-Type.
• List of methods allowed in the response header, Access-Control-Allow-Methods.

Main

The main class should extend RestStart abstract class.

HDP 3.1 and Spark job

I spent several sleepless nights trying to solve the problem while running Spark/HBase application. The application was dying giving the nasty error stack.
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown at io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:587) ... 22 more Caused by: java.lang.NoSuchMethodError: org.apache.spark.network.util.AbstractFileRegion.transferred()J at org.apache.spark.network.util.AbstractFileRegion.transfered(AbstractFileRegion.java:28) at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:228) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:282) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:879)
Usually problems like that point at versioning problem. But how could it happen if the application while running is dependent only on the client libraries provided by the HDP cluster?
Finally, after browsing through source code and comparing different versions of libraries, I crawled out of the swamp.
The culprit was an incompatible library in HBase client directory, netty-all-4.0.52.Final.jar. This library is calling deprecated transfered method which, in turn, calls a non-existing transferred method in Spark class. New version netty-all-4.1.17.Final.jar calls a correct transferred method.

The solution was dazzling simple. Just reverse the order of classpath in submit-spark command and give precedence to correct libraries in spark client jars.
spark-submit ... --conf "spark.driver.extraClassPath=$CONF:$LIB" ...

• wrong: LIB=/usr/hdp/current/hbase-client/lib/*:/usr/hdp/current/spark2-client/jars/*
• correct: LIB=/usr/hdp/current/spark2-client/jars/*:/usr/hdp/current/hbase-client/lib/*

Kerberos and Phoenix

Phoenix is a SQL solution on the top of HBase. There are two methods connecting to Phoenix, using full JDBC driver and thin JDBC driver. The latter requires additional server component, Phoenix Query Server.
I created the article about how to connect to Phoenix using both methods in kerberized environment. I also created a simple IntelliJ IDEA project demonstrating a connection to Phoenix from  Java program. Both JDBC drivers are supported.
I also added information on how to connect to Phoenix from Zeppelin notebook.
The article and sample project are available here.

AMS (Ambari Metrics System) is powered by standalone HBase and Phoenix server. Using a command line, one can get access to AMS Phoenix and query the metrics without going to UI. I also added information on how to accomplish it.
I was successful only by launching the command tool from the node where AMS is installed. I will work how to access it remotely.

MyTPC-DS,

An enhancement to my implementation of TPC-DS test. BigSQL test can be conducted by jsqsh utility, the dependency on DB2 client software is removed. Jsqsh is the part of BigSQL package and no additional dependency software is required now. How to use jsqsh for MyTPC-DS is described here.

I also added some additional results of the TPC-DS test conducted on HDP 3.1. The result is presented in the shape of Bar Plot to make them easier to understand. The graphic is created with the help of matplotlib. The Python source code is uploaded here. The input tables are taken directly from GitHub Wiki page. The pick up the correct table, a simple HTML tag is added to the page before every table.

The tests were executed using 100 GB data set and cannot be the basis of any far-reaching conclusion. But one apparent difference between HDP 2.6 and 3.1 is the significant performance improvement of Hive, from Hive 2.1 to Hive 3.1. Now the performance of Hive is almost as effective as the performance of BigSQL and ahead of SparkSQL. Also, the query coverage of Hive is much more comprehensive, from 50% (Hive 2.1) to almost 100% (Hive 3.1).

Useful links

• MyTPC-DS execution framework, link
• TPC-DS results using Bar Plot graphics, link
• TPC-DS results analytics tables, link 
• Python source code to compile the test result and prepare GitHub Wiki table, link
• Python source code to prepare graphics using matplotlib package, link
• Run BigSQL TPC-DS test using jsqsh tool, link

Hortonworks, Yarn Timeline server v.2

Problem
I spent several sleepless nights trying to resolve an issue with MapReduce jobs in HDP 3.1. While running MapReduce Service Check job, in the log file an error was reported although the job passed in green. The same message came up in others MapReduce jobs.
ERROR [pool-10-thread-1] org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl: Response from the timeline server is not successful, HTTP error code: 500, Server response:
{"exception":"WebApplicationException","message":"org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 252 actions: IOException: 252 times, servers with issues: null","javaClassName":"javax.ws.rs.WebApplicationException"}
[Job ATS Event Dispatcher] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Failed to process Event JOB_FINISHED for the job : job_1572998114693_0003 org.apache.hadoop.yarn.exceptions.YarnException: Failed while publishing entity
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:548)
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processEventForNewTimelineService(JobHistoryEventHandler.java:1405)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleTimelineEvent(JobHistoryEventHandler.java:742)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.access$1200(JobHistoryEventHandler.java:93)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.hand
In the yarn log file (/var/log/hadoop-yarn/yarn) more detailed message could be found. Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/meta-region-server
at org.apache.hadoop.hbase.client.ConnectionImplementation.get(ConnectionImplementation.java:2002)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateMeta(ConnectionImplementation.java:762)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:729)
at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:707)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:911)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:732)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:325)
... 17 more

The problem is related to Yarn Timeline server v.2. The service is using HBase as backup storage and it looked that this embedded HBase server was not started or not working properly. HBase is keeping its runtime configuration data like active HBase Region Serves in Zookeeper znodes.
Also, the znode /atsv2-hbase-secure did not exist.
zkCli.sh
ls /

[hive, cluster, brokers, hbase-unsecure, kafka-acl, kafka-acl-changes, admin, isr_change_notification, atsv2-hbase-secure, log_dir_event_notification, kafka-acl-extended, rmstore, hbase-secure, kafka-acl-extended-changes, consumers, latest_producer_id_block, registry, controller, zookeeper, delegation_token, hiveserver2, controller_epoch, hiveserver2-leader, atsv2-hbase-unsecure, ambari-metrics-cluster, config, ams-hbase-unsecure, ams-hbase-secure]

Embedded HBase service is running as Yarn service named ats-hbase. I started to examine the local container the ats-hbase is running in. It is the directory in the shape of:
/data/hadoop/yarn/local/usercache/yarn-ats/appcache/application_1573000895404_0001/container_e82_1573000895404_0001_01_000003. And it turned out, that the local hbase-site.xml configuration file defines the znode as atsv2-hbase-unsecure
   <property>
      <name>zookeeper.znode.parent</name>
      <value>/atsv2-hbase-unsecure</value>
   </property>

So the embedded HBase was storing its configuration in improper Zookeper znode. But how it could happen when the Yarn configuration parameter defines the znode as atsv2-hbase-secure and where the other services expect to find the embedded HBase runtime data. Also, the HDFS /user/yarn-ats/3.1.0.0-78/hbase-site.xml file, it is the source file used by Application Master to create a local container directory, contains a valid znode value.
Solution
After closer examination of the container log files, I discover the following entry:
INFO provider.ProviderUtils: Added file for localization: conf/hadoop-metrics2-hbase.properties -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hadoop-metrics2-hbase.properties
INFO provider.ProviderUtils: Added file for localization: conf/hbase-policy.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hbase-policy.xml
INFO provider.ProviderUtils: Added file for localization: conf/core-site.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/core-site.xml INFO provider.ProviderUtils: Added file for localization: conf/hbase-site.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hbase-site.xml
INFO provider.ProviderUtils: Added file for localization: conf/log4j.properties -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/log4j.properties
So it turned out, that Application/Serrvice Master is using not /user/yarn-ats/3.1.0.0-78/ HDFS path but another hidden /user/yarn-ats/.yarn/ directory to create the local container.
Finally, the solution was quite simple:

• kill the ats-hbase Yarn application
• stop the Yarn service
• remove  /user/yarn-ats/.yarn HDFS directory
• start the Yarn service

HDP, Kafka, LEADER_NOT_AVAILABLE

Problem
After HDP cluster kerberization, the Kafka does not work even though Kafka Healthcheck passes green. Any execution of kafka-console-producer.sh ends up with the error message: /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list kafka-host:6667 --producer-property security.protocol=SASL_PLAINTEXT --topic xxx

WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 2 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
kafka-topics.sh reports that the leader is not assigned to the partition.
/usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh -zookeeper zookeeper-host:2181 --describe xxx --unavailable-partitions

Topic: ambari_kafka_service_check Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: identity Partition: 0 Leader: none Replicas: 1002 Isr:
Topic: xxx Partition: 0 Leader: none Replicas: 1002 Isr:
Topic: xxxx Partition: 0 Leader: none Replicas: 1002 Isr:

There is nothing special in Kafka /var/log/kafka/server.log file. Only /var/log/kafka/controller.log suggests that something went wrong: cat controller.log

INFO [ControllerEventThread controllerId=1002] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
ERROR [ControllerEventThread controllerId=1002] Error processing event Startup (kafka.controller.ControllerEventManager$ControllerEventThread)
java.lang.NullPointerException
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:857)
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2571)
at kafka.utils.Json$.parseBytes(Json.scala:62)
at kafka.zk.ControllerZNode$.decode(ZkData.scala:56)
at kafka.zk.KafkaZkClient.getControllerId(KafkaZkClient.scala:902)
at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1199)

Solution
Uncle Google brings back many entries related to LEADER_NOT_AVAILABLE error but none of them led to the solution. Finally, I found this entry.
So the healing is very simple.

• Stop Kafka
• Run zkCli.sh Zookeeper command line
• Remove /controller znode, rmr /controller
• Start Kafka again

The evil spell is defeated.

RedHat, Steam and Crusaders King 2

I'm playing King Crusaders II on my RedHat desktop for some time. Although Steam officially supports only Ubuntu distribution, the games, particularly the older ones developed for Ubuntu 16.04, work fine also on RedHat/CentOS. Unfortunately, after the latest background update of the King Crusaders, the game stubbornly refused to run. After closer examination, I discovered that probably the updated game was compiled using newer version of GNU GCC compiler and the required level of libstdc++.so.6 library is not available on my platform.
./ck2: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./ck2)
./ck2: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./ck2)
So it was bad news. But there is good news, after the epic battle I straightened it out. It was as simple as installing a newer version of GNU GCC to get more modern libraries.
The solution is described here.