Recursive HDFS

That's strange but I was unable to find any solution to download the HDFS tree. The project I developed is covering the gap.

https://github.com/stanislawbartkowski/webhdfsdirectory

The only dependency is Python3 and the requests package. The HDFS tree is downloaded using HDFS Rest/API or Web/HDFS. It does not support Kerberos authentication and wasn't tested on a secured/encrypted connection.

The solution contains a simple hdfs package doing the main stuff and Python and bash wrappers. It also allows applying a RegularExpession directory selector to and "dry-run" option to test it without downloading anything. It was tested in a real production environment.

My own Hadoop/HDP benchmark

 Introduction

I spent some time trying to come to terms with HiBench but finally, I gave up. The project seemed to be abandoned, there was a lot of problems to run it in a secure (kerberized) environment and adjust it to new versions of Hive and Spark. Also, it is the project with a long history and there are a lot of layers not consistent with each other

So I ended up with creating my own version of HiBench benchmark. Only code migrated from HiBench is Spark/Scala and Jave source code upgraded to new versions of dependencies and more consistent parameter handling.

Features

• Dedicated to HDP 3.1.5. Standalone services are not supported.  
• Enabled for Kerberos
• Hive 3.0
• Spark 2.x
• Simple to run and expand, minimal configuration

Features not supported

• Streaming, pending
• Nutch indexing, not under development any longer
• Flink, Gearpump, not part of HDP stack

Configuration 

    All details are here.

SSL for masses

Motivation

 I expanded my tool for enabling wired encryption in the HDP cluster.

https://github.com/stanislawbartkowski/hdpwiredencryption

Previously, only self-signed certificates were supported. I added automation for CA-signed certificates. Important: it works only if CA-signed certificate package follows the supported format.

There are two paths possible: self-signed certificates and CA-signed certificates.

Self-signed certificates

1. ./run.sh 0 Creates self-signed certificate and truststores for every node.
2. ./run.sh.1 Creates and distributes all-client truststore.
3. ./run.sh 2 Secure keystores and truststores. Apply owner and Linux permissions.

CA-signed certificates

1. ./run.sh 3 Creates self-signed certificates and CSR (Certificate Signing Request) for every node
2. Manual step. Send all CSR to CA centre for signing. The CA-signed certificates  should be stored in a designed format.
3. ./run.sh 4 CA-signed certificates are imported into corresponding keystore and replacing the self-signed certificates. Truststores are created.
4. ./run.sh 1 Creates and distributes all-client trustore.
5. ./run.sh 2 Secure keystores and trustores.

Bonus

https://github.com/stanislawbartkowski/hdpwiredencryption/wiki

There is a number of pages containing practical steps on how to enable SSL for HDP components. It is based on documentation but more practical based on experience. 

For instance:

HDFS Ranger Plugin for SSL

NiFi service for SSL

HBase, Phoenix and CsvBulkLoadTool

 I'm running MyBench in a new environment and it fails while loading data into Phoenix table using CsvBulkLoadTool utility.

WARN tool.LoadIncrementalHFiles: Attempt to bulk load region containing  into table BENCH.USERVISITS with files [family:0 path:hdfs://bidev/tmp/386640ec-d49e-4760-8257-05858a409321/BENCH.USERVISITS/0/b467b5560eee4d61a42d4c9e6a78eb7e] failed.  This is recoverable and they will be retried.

INFO tool.LoadIncrementalHFiles: Split occurred while grouping HFiles, retry attempt 100 with 1 files remaining to group or split

ERROR tool.LoadIncrementalHFiles: -------------------------------------------------

Bulk load aborted with some files not yet loaded:

After closer examination, I discovered that the error takes place while moving/renaming input file into HBase staging directory /apps/hbase/data/staging. In this cluster, the HBase data is encrypted and moving data between encrypted and normal zone is not possible.

java.io.IOException: Failed to move HFile: hdfs://bidev/apps/hbase/data/staging/ambari-qa__BENCH.USERVISITS__dbb5qdfppq1diggr0dmdbcb1ji74ol4b9jn9ee2dgp1ttn9n5i6llfih7101fi1d/0/3a7f2d612c034253ad375ae002cc6ade to hdfs://bidev/tmp/fc43e454-00b3-4db0-8bdd-8b475885ab49/BENCH.USERVISITS/0/3a7f2d612c034253ad375ae002cc6ade

at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$SecureBulkLoadListener.failedBulkLoad(SecureBulkLoadManager.java:423)

at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6035)

at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$3.run(SecureBulkLoadManager.java:284)

The source code can be found here.

if (!FSUtils.isSameHdfs(conf, srcFs, fs)) {
	LOG.debug("Bulk-load file " + srcPath + " is on different filesystem than " +
	"the destination filesystem. Copying file over to destination staging dir.");
	FileUtil.copy(srcFs, p, fs, stageP, false, conf);
	} else if (copyFile) {
	LOG.debug("Bulk-load file " + srcPath + " is copied to destination staging dir.");
	FileUtil.copy(srcFs, p, fs, stageP, false, conf);
	} else {
	LOG.debug("Moving " + p + " to " + stageP);
	FileStatus origFileStatus = fs.getFileStatus(p);
	origPermissions.put(srcPath, origFileStatus.getPermission());
	if(!fs.rename(p, stageP)) {
	throw new IOException("Failed to move HFile: " + p + " to " + stageP);
	}
	}

When data is moved between different file system, the copying is enforced but unfortunately, data movement between encrypted and decrypted zone is not covered here.

Another option is to make use of "copyFile" parameter which enforces copying. After analyzing the control flow I discovered that there exists "hbase-site.xml" parameter  always.copy.files which seems to be the solution to the problem. But after applying this parameter, nothing has changed. 

Further examination with a little help of remote debugging unearthed a sad truth. CsvBulkLoadTool is passing the control to LoadIncrementalHFiles.java and "doBulkLoad" function.

public Map<LoadQueueItem, ByteBuffer> doBulkLoad(Path hfofDir, final Admin admin, Table table,  RegionLocator regionLocator) throws TableNotFoundException, IOException {

      return doBulkLoad(hfofDir, admin, table, regionLocator, false, false);

  }

Unfortunately, the "copyFiles" parameter is hardcoded as "false" although there is a sound and ready to use "isAlwaysCopyFiles()" function utilizing "hbase-site.xml" config file.

The only solution is manual fix and recreating the package from source files. But it does not go easy because one has to leverage different and outdated versions of HBase and Phoenix to create "Phoenix client" package matching HDP 3.1.

So two days spent without a solution.

HDP 3.1, HBase Phoenix and IBM Java

 I spent several sleepless nights trying to sort a nasty problem. While connecting to HBase Phoenix, the Phoenix client was stalled before displaying the prompt.
/usr/hdp/current/phoenix-client/bin/sqlline.py
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix: none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/phoenix/phoenix-5.0.0.3.1.0.0-78-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
20/08/07 11:28:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

What's more, there was no single error message, neither from the client nor in the HBase log file, Master or Region Servers. Simply nothing, no clue. Besides that, everything was healthy and running soundly.

The first step toward the solution was to enable tracing in Phoenix client. This setting is not managed by Ambari and requires manual modification.

vi /usr/hdp/3.1.0.0-78/phoenix/bin/log4j.properties
....
#psql.root.logger=WARN,console
psql.root.logger=TRACE,console

After TRACING was enabled, the sqlline client was more talkative and provided the first clue. INFO client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=4724 ms ago, cancelled=false, msg=Call to data2-worker.mycloud.com/10.135.118.222:16020 failed on local exception: javax.security.sasl.SaslException: Failure to initialize security context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0
major string: Invalid credentials
minor string: SubjectCredFinder: no JAAS Subject], details=row 'SYSTEM:CATALOG' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=data2-worker.mycloud.com,16020,1596794212924, seqNum=-1
The HDP cluster was Kerberized and the problem seemed to be related to Kerberos authentication. Kerberos is the source of many problems but everything except this worked smoothly here.

Finally, I found the hint under this link. The culprit was IBM Java. The cluster is running under control of  OpenJDK Java but the sqlline client is using default host Java. update-alternatives --config java

There are 3 programs which provide 'java'.

Selection Command
-----------------------------------------------
*+ 1 /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/bin/java
2 /usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64/jre/bin/java
3 /usr/java/jdk1.8.0_202-amd64/jre/bin/java

So the solution was extremely simple, just switch to OpenJDK Java and Phoenix client works like a dream.

HDP 3.1.5, OpenJDK, Infra Solr and AD/Kerberos

Problem 
I spent several sleepless nights caused by very nasty problem coming up after HDP 3.1.5 Kerberization. The Infra Solr components could not start just blocking the whole cluster. The message in the Ambari Console was saying.

Skip /infra-solr/configs and /infra-solr/collections
Set world:anyone to 'cr' on  /infra-solr/configs and /infra-solr/collections
KeeperErrorCode = NoAuth for /infra-solr/configs
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /infra-solr/configs
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
 at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
 at org.apache.zookeeper.ZooKeeper.setACL(ZooKeeper.java:1399)

It looked that infra-solr user could not update the ZooKeeper /infra-solr znode because of not sufficient privileges. But the ACL privileges looked correct.
[zk: localhost:2181(CONNECTED) 0] getAcl /infra-solr
'sasl,'infra-solr
: cdrwa
'world,'anyone
: r
[zk: localhost:2181(CONNECTED) 1]

After closer examination, I discovered strange stuff in ZooKeeper log. 2020-03-23 01:33:12,260 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@120] - Successfully authenticated client: authenticationID=$6O1000-3NO0GILCOJUA@FYRE.NET; authorizationID=infra-solr/a1.fyre.ibm.com@FYRE.NET.
2020-03-23 01:33:12,261 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: $6O1000-3NO0GILCOJUA
2020-03-23 01:33:12,261 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: $6O1000-3NO0GILCOJUA
2020-03-23 01:33:24,011 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x17104ad52230007, likely client has closed socket
So it seemed that Zookeeper to apply authorization rights was using not AD principal name (infra-solr) but sAMAccountName attribute of infra-solr AD principal ($6O1000-3NO0GILCOJUA). Ambari Kerberos Wizard is filling this attribute with random data only to keep it unique.
Solution 
The problem is described here, it is the bug coming with 1.8.0_242 version of OpenJDK.
The only workaround is to downgrade the OpenJDK to 232 level or switch to Oracle JDK.
yum downgrade java*

java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

And last but not least.
Block the Java upgrade unless the bug is fixed.
vi /etc/yum.conf

exclude=java*

HDP 3.1 and Spark job

I spent several sleepless nights trying to solve the problem while running Spark/HBase application. The application was dying giving the nasty error stack.
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown at io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:587) ... 22 more Caused by: java.lang.NoSuchMethodError: org.apache.spark.network.util.AbstractFileRegion.transferred()J at org.apache.spark.network.util.AbstractFileRegion.transfered(AbstractFileRegion.java:28) at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:228) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:282) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:879)
Usually problems like that point at versioning problem. But how could it happen if the application while running is dependent only on the client libraries provided by the HDP cluster?
Finally, after browsing through source code and comparing different versions of libraries, I crawled out of the swamp.
The culprit was an incompatible library in HBase client directory, netty-all-4.0.52.Final.jar. This library is calling deprecated transfered method which, in turn, calls a non-existing transferred method in Spark class. New version netty-all-4.1.17.Final.jar calls a correct transferred method.

The solution was dazzling simple. Just reverse the order of classpath in submit-spark command and give precedence to correct libraries in spark client jars.
spark-submit ... --conf "spark.driver.extraClassPath=$CONF:$LIB" ...

• wrong: LIB=/usr/hdp/current/hbase-client/lib/*:/usr/hdp/current/spark2-client/jars/*
• correct: LIB=/usr/hdp/current/spark2-client/jars/*:/usr/hdp/current/hbase-client/lib/*

Kerberos and Phoenix

Phoenix is a SQL solution on the top of HBase. There are two methods connecting to Phoenix, using full JDBC driver and thin JDBC driver. The latter requires additional server component, Phoenix Query Server.
I created the article about how to connect to Phoenix using both methods in kerberized environment. I also created a simple IntelliJ IDEA project demonstrating a connection to Phoenix from  Java program. Both JDBC drivers are supported.
I also added information on how to connect to Phoenix from Zeppelin notebook.
The article and sample project are available here.

AMS (Ambari Metrics System) is powered by standalone HBase and Phoenix server. Using a command line, one can get access to AMS Phoenix and query the metrics without going to UI. I also added information on how to accomplish it.
I was successful only by launching the command tool from the node where AMS is installed. I will work how to access it remotely.

MyTPC-DS,

An enhancement to my implementation of TPC-DS test. BigSQL test can be conducted by jsqsh utility, the dependency on DB2 client software is removed. Jsqsh is the part of BigSQL package and no additional dependency software is required now. How to use jsqsh for MyTPC-DS is described here.

I also added some additional results of the TPC-DS test conducted on HDP 3.1. The result is presented in the shape of Bar Plot to make them easier to understand. The graphic is created with the help of matplotlib. The Python source code is uploaded here. The input tables are taken directly from GitHub Wiki page. The pick up the correct table, a simple HTML tag is added to the page before every table.

The tests were executed using 100 GB data set and cannot be the basis of any far-reaching conclusion. But one apparent difference between HDP 2.6 and 3.1 is the significant performance improvement of Hive, from Hive 2.1 to Hive 3.1. Now the performance of Hive is almost as effective as the performance of BigSQL and ahead of SparkSQL. Also, the query coverage of Hive is much more comprehensive, from 50% (Hive 2.1) to almost 100% (Hive 3.1).

Useful links

• MyTPC-DS execution framework, link
• TPC-DS results using Bar Plot graphics, link
• TPC-DS results analytics tables, link 
• Python source code to compile the test result and prepare GitHub Wiki table, link
• Python source code to prepare graphics using matplotlib package, link
• Run BigSQL TPC-DS test using jsqsh tool, link

Hortonworks, Yarn Timeline server v.2

Problem
I spent several sleepless nights trying to resolve an issue with MapReduce jobs in HDP 3.1. While running MapReduce Service Check job, in the log file an error was reported although the job passed in green. The same message came up in others MapReduce jobs.
ERROR [pool-10-thread-1] org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl: Response from the timeline server is not successful, HTTP error code: 500, Server response:
{"exception":"WebApplicationException","message":"org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 252 actions: IOException: 252 times, servers with issues: null","javaClassName":"javax.ws.rs.WebApplicationException"}
[Job ATS Event Dispatcher] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Failed to process Event JOB_FINISHED for the job : job_1572998114693_0003 org.apache.hadoop.yarn.exceptions.YarnException: Failed while publishing entity
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:548)
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processEventForNewTimelineService(JobHistoryEventHandler.java:1405)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleTimelineEvent(JobHistoryEventHandler.java:742)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.access$1200(JobHistoryEventHandler.java:93)
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.hand
In the yarn log file (/var/log/hadoop-yarn/yarn) more detailed message could be found. Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/meta-region-server
at org.apache.hadoop.hbase.client.ConnectionImplementation.get(ConnectionImplementation.java:2002)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateMeta(ConnectionImplementation.java:762)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:729)
at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:707)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:911)
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:732)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:325)
... 17 more

The problem is related to Yarn Timeline server v.2. The service is using HBase as backup storage and it looked that this embedded HBase server was not started or not working properly. HBase is keeping its runtime configuration data like active HBase Region Serves in Zookeeper znodes.
Also, the znode /atsv2-hbase-secure did not exist.
zkCli.sh
ls /

[hive, cluster, brokers, hbase-unsecure, kafka-acl, kafka-acl-changes, admin, isr_change_notification, atsv2-hbase-secure, log_dir_event_notification, kafka-acl-extended, rmstore, hbase-secure, kafka-acl-extended-changes, consumers, latest_producer_id_block, registry, controller, zookeeper, delegation_token, hiveserver2, controller_epoch, hiveserver2-leader, atsv2-hbase-unsecure, ambari-metrics-cluster, config, ams-hbase-unsecure, ams-hbase-secure]

Embedded HBase service is running as Yarn service named ats-hbase. I started to examine the local container the ats-hbase is running in. It is the directory in the shape of:
/data/hadoop/yarn/local/usercache/yarn-ats/appcache/application_1573000895404_0001/container_e82_1573000895404_0001_01_000003. And it turned out, that the local hbase-site.xml configuration file defines the znode as atsv2-hbase-unsecure
   <property>
      <name>zookeeper.znode.parent</name>
      <value>/atsv2-hbase-unsecure</value>
   </property>

So the embedded HBase was storing its configuration in improper Zookeper znode. But how it could happen when the Yarn configuration parameter defines the znode as atsv2-hbase-secure and where the other services expect to find the embedded HBase runtime data. Also, the HDFS /user/yarn-ats/3.1.0.0-78/hbase-site.xml file, it is the source file used by Application Master to create a local container directory, contains a valid znode value.
Solution
After closer examination of the container log files, I discover the following entry:
INFO provider.ProviderUtils: Added file for localization: conf/hadoop-metrics2-hbase.properties -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hadoop-metrics2-hbase.properties
INFO provider.ProviderUtils: Added file for localization: conf/hbase-policy.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hbase-policy.xml
INFO provider.ProviderUtils: Added file for localization: conf/core-site.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/core-site.xml INFO provider.ProviderUtils: Added file for localization: conf/hbase-site.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hbase-site.xml
INFO provider.ProviderUtils: Added file for localization: conf/log4j.properties -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/log4j.properties
So it turned out, that Application/Serrvice Master is using not /user/yarn-ats/3.1.0.0-78/ HDFS path but another hidden /user/yarn-ats/.yarn/ directory to create the local container.
Finally, the solution was quite simple:

• kill the ats-hbase Yarn application
• stop the Yarn service
• remove  /user/yarn-ats/.yarn HDFS directory
• start the Yarn service

HDP, Kafka, LEADER_NOT_AVAILABLE

Problem
After HDP cluster kerberization, the Kafka does not work even though Kafka Healthcheck passes green. Any execution of kafka-console-producer.sh ends up with the error message: /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list kafka-host:6667 --producer-property security.protocol=SASL_PLAINTEXT --topic xxx

WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 2 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
kafka-topics.sh reports that the leader is not assigned to the partition.
/usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh -zookeeper zookeeper-host:2181 --describe xxx --unavailable-partitions

Topic: ambari_kafka_service_check Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001
Topic: identity Partition: 0 Leader: none Replicas: 1002 Isr:
Topic: xxx Partition: 0 Leader: none Replicas: 1002 Isr:
Topic: xxxx Partition: 0 Leader: none Replicas: 1002 Isr:

There is nothing special in Kafka /var/log/kafka/server.log file. Only /var/log/kafka/controller.log suggests that something went wrong: cat controller.log

INFO [ControllerEventThread controllerId=1002] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
ERROR [ControllerEventThread controllerId=1002] Error processing event Startup (kafka.controller.ControllerEventManager$ControllerEventThread)
java.lang.NullPointerException
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:857)
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2571)
at kafka.utils.Json$.parseBytes(Json.scala:62)
at kafka.zk.ControllerZNode$.decode(ZkData.scala:56)
at kafka.zk.KafkaZkClient.getControllerId(KafkaZkClient.scala:902)
at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1199)

Solution
Uncle Google brings back many entries related to LEADER_NOT_AVAILABLE error but none of them led to the solution. Finally, I found this entry.
So the healing is very simple.

• Stop Kafka
• Run zkCli.sh Zookeeper command line
• Remove /controller znode, rmr /controller
• Start Kafka again

The evil spell is defeated.

How to obtain an active NameNode remotely

Problem
While using WebHDFS REST API interface, the client is dealing directly with the NameNode. In HA (high availability) environment, only one NameNode is active, the second is standby. If the standby NameNode is addressed, the request is denied. So the client should be aware of which NameNode is active and construct a valid URL.  But how to discover remotely the active NameNode automatically and avoid redirecting the client manually in case of failover?
Sounds strange but the there is no Ambari REST API to detect an active NameNode.
One obvious solution is to use WebHDFS Knox Gateway which, assuming configured properly, is propagating the query to the valid NameNode.
Solution
There are two convenient methods to discover the active NameNode outside Knox Gateway. One is to use JMX query and the second is to use hdfs haadmin.
The solution is described here in more details. I also added a convenient bash script to extract the active NameNode using both methods: JMX Query and hdfs haadmin. The script can be easily customized. If hdfs haadmin method is used, the script can be executed inside the cluster only so the remote shell call should be implemented.

HDP 3.1, HBase REST API, security gap

Problem
I found a nasty problem with HDP 3.1 which cost me several sleepless nights. There is a security gap in HBase REST API. The HBase REST API service does not impersonate users and all HBase commands are executed as hbase user. The same behaviour is passed to Knox HBase. It means that any user having access to HBase REST API or Knox Gateway HBase is authorized to do any action bypassing any security settings in Ranger or HBase service directly.
Solution
The only solution I found was to compile the current version of HBase downloaded from GitHub and replace the legacy hbase-rest jar with the new one.

Clone GitHub repository and build the packages
git clone https://github.com/apache/hbase.git -b branch-2.0
cd hbase
mvn package -DskipTests

As root user
cd /usr/hdp/3.1.0.0-78/hbase/lib

Archive existing jar
mkdir arch
mv mv hbase-rest-2.0.2.3.1.0.0-78.jar arch/
unlink hbase-rest.jar

Replace with the new one
ln -s /home/hbase/hbase/hbase-rest/target/hbase-rest-2.0.6-SNAPSHOT.jar hbase-rest.jar

Restart HBase REST API server.

HDP 3.1, Wired Encryption

Introduction
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/configuring-wire-encryption/content/wire_encryption.html
A Wired Encryption is not only adding the next layer of security, but also the next layer of complexity. Although everything is described in the HortonWorks documentation, it is not easy to extract the practical steps on how to set up the encryption, it took me some time to accomplish it. So I decided to recap my experience and publish several useful scripts and practical procedure.
https://github.com/stanislawbartkowski/hdpwiredencryption
What is described

• Self-signed certificates
• Enable SSL for WebHDFS, MapReduce, Tez, and YARN

Future plans

• Certificates signed by CA
• Other HDP services
• Application connection

Problem to solve

After enabling the encryption, BigSQL LOAD HADOOP command refused working. It is related to the TimeLine service secure connection. Will try to sort it.