tag:blogger.com,1999:blog-23997591037418498972024-03-08T03:33:03.581-08:00javahotelBlog do projektu Open Source JavaHotelStanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.comBlogger335125tag:blogger.com,1999:blog-2399759103741849897.post-66873253261783662492021-09-26T16:33:00.001-07:002021-09-26T16:33:49.066-07:00Recursive HDFS download - continued<p>I extended my <a href="https://github.com/stanislawbartkowski/webhdfsdirectory">WebHDFS downloader</a>. In the case of HDFS directories having an enormous number of files, scanning the directory using WebHDFS REST API fails because of time-out. </p><p>The solution is to obtain the HDFS directory tree using the standard <i>hdfs dfs -ls -R</i> command, ship the result to the node where <i>hdfsdownloader</i> is executed and use the file as the input for the tool.</p><p>More detailed description: <a href="https://github.com/stanislawbartkowski/webhdfsdirectory">https://github.com/stanislawbartkowski/webhdfsdirectory</a></p><p>To extend the tool, I used the inheritance feature of Python. The tool now is running in two modes, get the list of HDFS files using <i>WebHDFS REST/API</i> or get the list from the input text file. So the only difference is the way of obtaining the list of files, the main tool flow is the same regardless of the method.</p><p>Source code: <a href="https://github.com/stanislawbartkowski/webhdfsdirectory/blob/main/src/proc/hdfs.py">https://github.com/stanislawbartkowski/webhdfsdirectory/blob/main/src/proc/hdfs.py</a></p><p>Class <i>ClassHDFS</i> runs the application flow. It calls <i>getdir</i> method to receive a list of names of files and directories in the HDFS path provided as a parameter and iterates over the list. For a regular file, the file is downloaded. For the directory entry, the class makes a recursive call and goes one level down in HDFS directory tree.</p><p><i>getdir</i> method is defined in an appropriate inherited class. In the case of <i>WebHDFS REST/API</i> mode, the method inherits from <i>TRAVERSEHDFS</i> class. In the case of input text file, the method is implemented in <i>TRAVERSEFILE</i> class.</p>
<pre>class DIRHDFS(CLASSHDFS, TRAVERSEHDFS):
def __init__(self, WEBHDFSHOST, WEBHDFSPORT, WEBHDFSUSER, DIRREG=None, dryrun=False):
TRAVERSEHDFS.__init__(self, WEBHDFSHOST, WEBHDFSPORT, WEBHDFSUSER)
CLASSHDFS.__init__(self, DIRREG, dryrun)
class FILEHDFS(CLASSHDFS, TRAVERSEFILE):
def __init__(self, txtfile,WEBHDFSHOST, WEBHDFSPORT, WEBHDFSUSER, DIRREG=None, dryrun=False):
self.WEBHDFSHOST = WEBHDFSHOST
self.WEBHDFSPORT = WEBHDFSPORT
self.WEBHDFSUSER = WEBHDFSUSER
TRAVERSEFILE.__init__(self, txtfile)
CLASSHDFS.__init__(self, DIRREG, dryrun)
</pre>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-27618511161320689552021-06-30T04:43:00.001-07:002021-06-30T04:43:45.953-07:00Recursive HDFS<p>That's strange but I was unable to find any solution to download the HDFS tree. The project I developed is covering the gap.</p><p><a href="https://github.com/stanislawbartkowski/webhdfsdirectory">https://github.com/stanislawbartkowski/webhdfsdirectory</a></p><p>The only dependency is <a href="https://www.python.org/downloads/">Python3</a> and the <a href="https://pypi.org/project/requests/">requests</a> package. The HDFS tree is downloaded using HDFS Rest/API or Web/HDFS. It does not support Kerberos authentication and wasn't tested on a secured/encrypted connection.</p><p>The solution contains a simple <a href="https://github.com/stanislawbartkowski/webhdfsdirectory/blob/main/src/proc/hdfs.py">hdfs</a> package doing the main stuff and Python and bash wrappers. It also allows applying a RegularExpession directory selector to and "dry-run" option to test it without downloading anything. It was tested in a real production environment.</p><p><br /></p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-67966844451542312732021-04-27T12:16:00.002-07:002021-04-27T15:57:53.839-07:00DB2 Warehouse and encryption<p> I spent some trying to move the database from one DB2 instance to another, just a trivial backup and restore scenario. Unfortunately, RESTORE command failed with a nasty message.</p><blockquote>db2 restore db bludb from /tmp taken at 20210425102241 </blockquote><pre>SQL1743N The RESTORE command failed because the backup image is of an <br /><br />encrypted database but the target database is not encrypted.</pre>I was trying to restore the encrypted database into a plain world. To be sure:<div><blockquote>db2ckbkp -H /tmp/BLUDB.0.db2inst1.DBPART000.20210425102241.001 </blockquote><pre>Encrypt Info Flags -- 0x1
Source DB was encrypted</pre></div>
<div>Yes, the backup is encrypted but how to recreate it?</div><div><br /></div><div>To decrypt the backup, we need the key used for encryption. The source keystore and the label of the secret key can be extracted from the backup.</div><div><br /></div><div><blockquote>db2 "restore db bludb from /mnt/backup taken at 20210425102241 encropts 'show master key details'"</blockquote></div><div>The result is tricky, in the diagnostic directory look for a specific file containing information about source keystore and label.</div><blockquote><div>cd /database/config/db2inst1/sqllib/db2dump/DIAG0000</div><div>cat BLUDB.0.db2inst1.DBPART000.20210425102241.masterKeyDetails </div></blockquote><p> KeyStore Type: PKCS12</p><pre> KeyStore Location: /mnt/blumeta0/db2/keystore/keystore.p12
KeyStore Host Name: thinkde
KeyStore IP Address: 127.0.1.1
KeyStore IP Address Type: IPV4
Encryption Algorithm: AES
Encryption Algorithm Mode: CBC
Encryption Key Length: 256
Master Key Label: DB2_SYSGEN_db2inst1_BLUDB_2021-03-19-13.57.02_A8CF4EED</pre>The next step is to go to the source keystore and extract the secret key. If the source database is lost and there is no copy of the keystore, the backup is also lost because there is no way to decipher it.<div><br /></div><div>Display the content of keystore.</div><div><br /></div><div><blockquote>gsk8capicmd_64 -cert - -list -db /mnt/blumeta0/db2/keystore/keystore.p12 -stashed</blockquote></div>
<pre>Certificates found
* default, - personal, ! trusted, # secret key
# DB2_SYSGEN_db2inst1_BLUDB_2021-03-19-13.57.02_A8CF4EED
</pre>
Yes, the wanted secret key still exists in the keystore. Now export it and transport it to the target database.
<blockquote>gsk8capicmd_64 -cert -export -db /tmp/keystore/keystore.p12 -stashed -label DB2_SYSGEN_db2inst1_BLUDB_2021-03-19-13.57.02_A8CF4EED -target secretkey.p12
<br /> Target database password : </blockquote>
The target database password is used to encrypt secretkey.p12 file. On the target host, create an empty keystore and reconfigure DB2.<blockquote><div>gsk8capicmd_64 -keydb -create -db /mnt/blumeta0/db2/keystore/keystore.p12 -pw "secret" -type pkcs12 -stash</div><div>db2 update dbm cfg using keystore_location /mnt/blumeta0/db2/keystore/keystore.p12 keystore_type pkcs12</div><div></div></blockquote><div>Now import a secret key into the newly created keystore.</div><div><blockquote>gsk8capicmd_64 -cert -import -db /mnt/backup/secretkey.p12 -pw "secret" -target /mnt/blumeta0/db2/keystore/keystore.p12 -stashed -label DB2_SYSGEN_db2inst1_BLUDB_2021-03-19-13.57.02_A8CF4EED </blockquote></div><div>And now restore the database again.</div><div><blockquote>db2 restore db bludb from /mnt/backup taken at 20210425102241 no encrypt</blockquote></div><div>Success.</div><div><br /></div>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-76482516004422475752021-03-31T12:08:00.004-07:002021-03-31T12:08:38.074-07:00Java, Parquet and DB2 Warehouse in Cloud<p>I expanded my tool for moving data from Parquet file into JDBC database (<a href="http://hoteljavaopensource.blogspot.com/2021/02/java-parquet-and-jdbc.html">blog</a>). During my test, loading 6 million rows from 200 MB Parquet file into DB2 Warehouse took around 30-40 minutes. Not so bad for small data volume but not feasible while dealing with terabytes of data.</p><p>In DB2 Warehouse on Cloud, there is a very effective method of data loading from local (local to the DB2 Warehouse) or Amazon S3 bucket. It takes around 30-40 seconds to load 1.2 GB text file containing 6 million lines. </p><p>This load option is available from DB2 Warehouse Console or through DB2 Warehouse on Cloud <a href="https://cloud.ibm.com/apidocs/db2-warehouse-on-cloud/db2-warehouse-on-cloud-v4#createetloadjob">REST/API</a>.</p><p>So the idea to boost up the loading performance is to use AWS S3 bucket mounted in the local file system, deploy the delimited text files extracted from the Parquet file there, and launch DB2 Warehouse REST/API to load data from AWS S3 bucket. </p><p>In the nutshell, it includes the following steps.</p><p></p><ul style="text-align: left;"><li>Mount AWS S3 bucket in the local file system. "Local" - meaning the host where the application is running. <a href="https://linuxroutes.com/mount-s3-bucket-linux-centos-rhel-ubuntu-using-s3fs/">RedHat/Centos</a></li><li>The application is reading the input Parquet file and producing several delimited text file in the AWS S3 mounted directory. For instance: /mnt/s3/sbtest/out1/export-0, /mnt/s3/sbtest/out1/export-1, /mnt/s3/sbtest/out1/export-2, /mnt/s3/sbtest/out1/export-3</li><li>When the delimited text file is ready, the application is launching DB2 Warehouse on Cloud REST/API and loads data from AWS S3 bucket using file names: /sbtest/out1/export-0, /sbtest/out1/export-1, /sbtest/out1/export-2, /sbtest/out1/export-3</li><li>After data is loaded, the application can produce the next chunk of data and continue.</li></ul><div>The loading time was reduced to 1 minute.</div><div><br /></div><div>The current version of the application is available <a href="https://github.com/stanislawbartkowski/ParquetJDBC">here</a>.</div><div><br /></div><div>Several remarks:</div><div><ul style="text-align: left;"><li>The load commands are running in parallel. In my test environment, the optimal level of parallelism was 4. Beyond this number, the DB2 Warehouse is reporting loading errors - the problem requires further investigation.</li><li>Only single LOAD command can be executed against a single table. So every loading task is loading data into a private table (CREATE TABLE TABLE-1 LIKE TABLE) and later inserts data into the main table (INSERT INTO TABLE SELECT * FROM TABLE-1). The INSERT command is running very fast, it takes around 10 sec to INSERT 100 MB of data.</li></ul></div><p></p><p><br /></p><p><br /></p><p><br /></p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-15783917399087116922021-02-28T12:45:00.001-08:002021-02-28T12:45:30.237-08:00Maven and GitHub<p> I spent half a day trying to upload my package to GitHub Maven following the guidelines <a href="https://docs.github.com/en/packages/guides/configuring-apache-maven-for-use-with-github-packages">GitHub Maven</a>. As usual, it seemed too good to be true to be successful immediately. Most of the time, I was haunted by a mysterious message which failed <i>mvn deploy</i> command.</p><blockquote><p>RROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project RestService: Failed to deploy artifacts: Could not transfer artifact com.restservice:RestService:jar:1.0-20210228.194713-1 from/to github (https://maven.pkg.github.com/stanislawbartkowski/RestService): Failed to transfer file: https://maven.pkg.github.com/stanislawbartkowski/RestService/com/restservice/RestService/1.0-SNAPSHOT/RestService-1.0-20210228.194713-1.jar. Return code is: 422, ReasonPhrase: Unprocessable Entity. -> [Help 1]</p></blockquote><p>After several hours I was at the point of giving up. The salvation came from <a href="https://stackoverflow.com/questions/64322121/publishing-github-packages-returns-422-error">stackoverflow</a>.</p><blockquote><p><span face="Arial, "Helvetica Neue", Helvetica, sans-serif" style="background-color: white; color: #242729; font-size: 15px;">Could you try lower casing your artifact ID</span></p></blockquote><p>It was the point, "Return code is: 422, ReasonPhrase: Unprocessable Entity" translates as "use lower case".</p><p>Wrong:</p><blockquote><p><artifactId>RestService</artifactId></p></blockquote><p>Correct:</p><blockquote><p> <artifactId>restservice</artifactId></p></blockquote><p><br /></p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com1tag:blogger.com,1999:blog-2399759103741849897.post-16891103223102485162021-02-28T02:34:00.003-08:002021-02-28T02:35:36.171-08:00Java, Parquet and JDBC<p> That is strange but it is almost impossible to access <a href="https://parquet.apache.org/">Parquet</a> files outside Hadoop/Spark context. I was trying to move data imprisoned in Parquet to JDBC accessed relational database using standalone Java application and failed.</p><p>So I ended up with Spark/Java application to address the issue. </p><p>Source and description: <a href="https://github.com/stanislawbartkowski/ParquetJDBC">https://github.com/stanislawbartkowski/ParquetJDBC</a></p><p>The application loads Parquet formatted data, it can be a single file or a directory, partition the data into several chunks, launches executors and loads data into JDBC databases in parallel. The number of partitions and executors are configurable. </p><p>The application was tested as a local and single-node Spark configuration. The next step is to configure and test the application in a distributed Hadoop environment.</p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-55964128467468681202021-01-31T05:13:00.001-08:002021-01-31T05:13:16.176-08:00DB2 audit<p> DB2 audit is a powerful tool allowing to supervise the usage of DB2 instance and databases. Some practical advice how to set up and use DB audit is described <a href="https://github.com/stanislawbartkowski/wikis/wiki/DB2-auditing">here</a>.</p><p>But DB2 audit by itself collects data. Next step is to make practical use of the tool. It is for no advantage to collect data without analyzing them.</p><p>So I developed a simple solution to discover and escalate any suspicious behaviour. The solution and description are available <a href="https://github.com/stanislawbartkowski/db2audit">here</a>.</p><p>The solution consists of several bash scripts and does not require any additional dependency.</p><p>Two tasks are implemented:</p><p></p><ul style="text-align: left;"><li>Collecting audit records and moving them to additional DB2 database ready for further analysis. This part can be executed as a crontab job</li><li> Running investigative SQL queries on the audit database to discover suspicious and not expected behaviour. This part can be executed on-demand or as a crontab job. Example</li><ul><li>Not authorized user connected to DB2 database.</li><li>Read-only user run an update SQL statement.</li><li>Failed command reported as "not authorized" suggesting a user trying to overuse its authority.</li></ul></ul>Some examples of investigating queries are implemented <a href="https://github.com/stanislawbartkowski/db2audit/tree/main/config.template/queries">already</a>. Any new query can be added.<p></p><p>The solution is running at the instance level but investigative queries can be customized at the database level. In the case of several databases in a single instance, every database can come with its own security rules.</p><p>Every violation can be escalated using a customizable script. The script example reporting violations in a special text file is available <a href="https://github.com/stanislawbartkowski/db2audit/blob/main/config.template/alert.sh">here</a>.</p><p><br /></p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-44305565279697993472020-12-28T14:56:00.005-08:002020-12-28T14:57:31.723-08:00Mail server in a single docker<p> I created a simple mail server, SMPT and IMAPS, in a single docker/podman container. The mail services engines are Postfix (SMP) and Dovecot (IMAPS). The solution is described <a href="https://github.com/stanislawbartkowski/docker-mail">here</a>. I also added guidelines on how to test and configure several mail clients: Evolution and <b>mutt</b>.</p><p>The storage is ephemeral, not recommended for any production environment but ideal for testing, easy to create and easy to dismantle. </p><p>I also added a sample <b><a href="https://github.com/stanislawbartkowski/docker-mail/blob/main/openshift/mail.yaml">yaml</a> </b>configuration file and remarks on how to deploy the container to OpenShift/Kubernetes cluster.</p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-49240688119340278792020-11-29T16:00:00.009-08:002020-11-29T16:00:57.756-08:00My own Hadoop/HDP benchmark<p><span style="font-size: large;"> Introduction</span></p><p>I spent some time trying to come to terms with <a href="https://github.com/Intel-bigdata/HiBench">HiBench</a> but finally, I gave up. The project seemed to be abandoned, there was a lot of problems to run it in a secure (kerberized) environment and adjust it to new versions of Hive and Spark. Also, it is the project with a long history and there are a lot of layers not consistent with each other</p><p>So I ended up with creating my <a href="https://github.com/stanislawbartkowski/MyBench">own version</a> of HiBench benchmark. Only code migrated from HiBench is Spark/Scala and Jave source code upgraded to new versions of dependencies and more consistent parameter handling.</p><p><span style="font-size: large;">Features</span></p><p></p><ul style="text-align: left;"><li>Dedicated to HDP 3.1.5. Standalone services are not supported. </li><li>Enabled for Kerberos</li><li>Hive 3.0</li><li>Spark 2.x</li><li>Simple to run and expand, minimal configuration</li></ul><div><span style="font-size: large;">Features not supported</span></div><div><ul style="text-align: left;"><li>Streaming, pending</li><li>Nutch indexing, not under development any longer</li><li>Flink, Gearpump, not part of HDP stack</li></ul><div><span style="font-size: large;">Configuration </span></div></div><div><br /></div><div> All details are <a href="https://github.com/stanislawbartkowski/MyBench">here</a>.<br /></div><div><br /></div><p></p><p><br /></p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-18334973639719480352020-10-31T12:47:00.010-07:002020-10-31T12:48:39.247-07:00SSL for masses<p><span style="font-size: large;">Motivation</span></p><p> I expanded my tool for enabling wired encryption in the HDP cluster.</p><p><a href="https://github.com/stanislawbartkowski/hdpwiredencryption">https://github.com/stanislawbartkowski/hdpwiredencryption</a><br /></p><p>Previously, only self-signed certificates were supported. I added automation for CA-signed certificates. Important: it works only if CA-signed certificate package follows the supported format.</p><p>There are two paths possible: self-signed certificates and CA-signed certificates.</p><p><span style="font-size: large;">Self-signed certificates</span></p><p></p><ol style="text-align: left;"><li><i>./run.sh 0 </i>Creates self-signed certificate and truststores for every node.</li><li><i>./run.sh.1</i> Creates and distributes all-client truststore.</li><li><i>./run.sh 2</i> Secure keystores and truststores. Apply owner and Linux permissions.</li></ol><div><span style="font-size: large;">CA-signed certificates</span></div><div><ol style="text-align: left;"><li><i>./run.sh 3</i> Creates self-signed certificates and CSR (Certificate Signing Request) for every node</li><li>Manual step. Send all CSR to CA centre for signing. The CA-signed certificates should be stored in a designed format.</li><li><i>./run.sh 4</i> CA-signed certificates are imported into corresponding keystore and replacing the self-signed certificates. Truststores are created.</li><li><i>./run.sh 1</i> Creates and distributes all-client trustore.</li><li><i>./run.sh 2</i> Secure keystores and trustores.</li></ol></div><p></p><p><span style="font-size: large;">Bonus</span></p><p><a href="https://github.com/stanislawbartkowski/hdpwiredencryption/wiki">https://github.com/stanislawbartkowski/hdpwiredencryption/wiki</a></p><p>There is a number of pages containing practical steps on how to enable SSL for HDP components. It is based on documentation but more practical based on experience. </p><p>For instance:</p><p><a href="https://github.com/stanislawbartkowski/hdpwiredencryption/wiki/SSL-Ranger-UI-HDFS-Ranger-Plugin">HDFS Ranger Plugin for SSL</a><br /></p><p><a href="https://github.com/stanislawbartkowski/hdpwiredencryption/wiki/Enable-NiFi-service-for-SSL">NiFi service for SSL</a><br /></p><p><br /></p><p><br /></p>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-80460321986536494322020-08-13T06:52:00.012-07:002020-08-13T06:55:12.932-07:00HBase, Phoenix and CsvBulkLoadTool<p> I'm running <a href="https://github.com/stanislawbartkowski/MyBench">MyBench</a> in a new environment and it fails while loading data into Phoenix table using CsvBulkLoadTool utility.</p><blockquote><p>WARN tool.LoadIncrementalHFiles: Attempt to bulk load region containing into table BENCH.USERVISITS with files [family:0 path:hdfs://bidev/tmp/386640ec-d49e-4760-8257-05858a409321/BENCH.USERVISITS/0/b467b5560eee4d61a42d4c9e6a78eb7e] failed. This is recoverable and they will be retried.</p><p>INFO tool.LoadIncrementalHFiles: Split occurred while grouping HFiles, retry attempt 100 with 1 files remaining to group or split</p><p>ERROR tool.LoadIncrementalHFiles: -------------------------------------------------</p><p>Bulk load aborted with some files not yet loaded:
</p></blockquote>
<p>After closer examination, I discovered that the error takes place while moving/renaming input file into HBase staging directory /apps/hbase/data/staging. In this cluster, the HBase data is encrypted and moving data between encrypted and normal zone is not possible.</p><blockquote><p>java.io.IOException: Failed to move HFile: hdfs://bidev/apps/hbase/data/staging/ambari-qa__BENCH.USERVISITS__dbb5qdfppq1diggr0dmdbcb1ji74ol4b9jn9ee2dgp1ttn9n5i6llfih7101fi1d/0/3a7f2d612c034253ad375ae002cc6ade to hdfs://bidev/tmp/fc43e454-00b3-4db0-8bdd-8b475885ab49/BENCH.USERVISITS/0/3a7f2d612c034253ad375ae002cc6ade</p><p><span style="white-space: pre;"> </span>at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$SecureBulkLoadListener.failedBulkLoad(SecureBulkLoadManager.java:423)</p><p><span style="white-space: pre;"> </span>at org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:6035)</p><p><span style="white-space: pre;"> </span>at org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$3.run(SecureBulkLoadManager.java:284)</p></blockquote>
The source code can be found <a href="https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java">here</a>.<blockquote><table class="highlight tab-size js-file-line-container" data-paste-markdown-skip="" data-tab-size="2" style="background-color: white; border-collapse: collapse; border-spacing: 0px; color: #24292e; font-size: 14px; tab-size: 2;"><tbody style="box-sizing: border-box;"><tr style="box-sizing: border-box;"><td class="blob-code blob-code-inner js-file-line" id="LC386" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">if</span> (<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">!</span><span class="pl-smi" style="box-sizing: border-box;">FSUtils</span><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span>isSameHdfs(conf, srcFs, fs)) {</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="387" id="L387" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC387" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">LOG</span><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span><span class="pl-token" data-hydro-click-hmac="7700fcbcb066fe6b17a9ed1f8de1abb2f33019fd33885d8b1ab6fc74cd740664" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":20089857,"ref":"master","language":"Java","originating_url":"https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java","user_id":11859470}}" style="box-sizing: border-box;">debug</span>(<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Bulk-load file <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> srcPath <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span> is on different filesystem than <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span></td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="388" id="L388" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC388" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>the destination filesystem. Copying file over to destination staging dir.<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="389" id="L389" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC389" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-smi" style="box-sizing: border-box;">FileUtil</span><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span>copy(srcFs, p, fs, stageP, <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">false</span>, conf);</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="390" id="L390" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC390" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> } <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">else</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">if</span> (<span class="pl-token" data-hydro-click-hmac="7700fcbcb066fe6b17a9ed1f8de1abb2f33019fd33885d8b1ab6fc74cd740664" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":20089857,"ref":"master","language":"Java","originating_url":"https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java","user_id":11859470}}" style="box-sizing: border-box;">copyFile</span>) {</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="391" id="L391" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC391" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">LOG</span><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span>debug(<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Bulk-load file <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> srcPath <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span> is copied to destination staging dir.<span class="pl-pds" style="box-sizing: border-box;">"</span></span>);</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="392" id="L392" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC392" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-smi" style="box-sizing: border-box;">FileUtil</span><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span><span class="pl-token" data-hydro-click-hmac="7700fcbcb066fe6b17a9ed1f8de1abb2f33019fd33885d8b1ab6fc74cd740664" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":20089857,"ref":"master","language":"Java","originating_url":"https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java","user_id":11859470}}" style="box-sizing: border-box;">copy</span>(srcFs, p, fs, stageP, <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">false</span>, conf);</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="393" id="L393" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC393" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> } <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">else</span> {</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="394" id="L394" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC394" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">LOG</span><span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span>debug(<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Moving <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> p <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span> to <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> stageP);</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="395" id="L395" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC395" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-smi" style="box-sizing: border-box;">FileStatus</span> origFileStatus <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">=</span> fs<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span>getFileStatus(p);</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="396" id="L396" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC396" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> origPermissions<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span>put(srcPath, origFileStatus<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span>getPermission());</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="397" id="L397" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC397" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">if</span>(<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">!</span>fs<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">.</span><span class="pl-token" data-hydro-click-hmac="7700fcbcb066fe6b17a9ed1f8de1abb2f33019fd33885d8b1ab6fc74cd740664" data-hydro-click="{"event_type":"code_navigation.click_on_symbol","payload":{"action":"click_on_symbol","repository_id":20089857,"ref":"master","language":"Java","originating_url":"https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java","user_id":11859470}}" style="box-sizing: border-box;">rename</span>(p, stageP)) {</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="398" id="L398" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC398" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">throw</span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">new</span> <span class="pl-smi" style="box-sizing: border-box;">IOException</span>(<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span>Failed to move HFile: <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> p <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">"</span> to <span class="pl-pds" style="box-sizing: border-box;">"</span></span> <span class="pl-k" style="box-sizing: border-box; color: #d73a49;">+</span> stageP);</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="399" id="L399" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC399" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> }</td></tr><tr style="box-sizing: border-box;"><td class="blob-num js-line-number" data-line-number="400" id="L400" style="box-sizing: border-box; color: rgba(27, 31, 35, 0.3); cursor: pointer; font-size: 12px; line-height: 20px; min-width: 50px; padding: 0px 10px; text-align: right; user-select: none; vertical-align: top; white-space: nowrap; width: 50px;"></td><td class="blob-code blob-code-inner js-file-line" id="LC400" style="box-sizing: border-box; font-size: 12px; line-height: 20px; overflow-wrap: normal; overflow: visible; padding: 0px 10px; position: relative; vertical-align: top; white-space: pre;"> }</td></tr></tbody></table></blockquote>
When data is moved between different file system, the copying is enforced but unfortunately, data movement between encrypted and decrypted zone is not covered here.<div><br /><div>Another option is to make use of "copyFile" parameter which enforces copying. After analyzing the control flow I discovered that there exists "hbase-site.xml" parameter <a href="https://issues.apache.org/jira/browse/HBASE-16672">always.copy.files</a> which seems to be the solution to the problem. But after applying this parameter, nothing has changed. </div></div><div>Further examination with a little help of remote debugging unearthed a sad truth. CsvBulkLoadTool is passing the control to <a href="https://github.com/apache/hbase/blob/branch-2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/LoadIncrementalHFiles.java">LoadIncrementalHFiles.java</a> and "doBulkLoad" function.</div><div><br /></div><myquo><div><div>public Map<LoadQueueItem, ByteBuffer> doBulkLoad(Path hfofDir, final Admin admin, Table table, RegionLocator regionLocator) throws TableNotFoundException, IOException {</div><div> return doBulkLoad(hfofDir, admin, table, regionLocator, false, false);</div><div> }</div></div><div><br /></div></myquo>Unfortunately, the "copyFiles" parameter is hardcoded as "false" although there is a sound and ready to use "isAlwaysCopyFiles()" function utilizing "hbase-site.xml" config file.<div>The only solution is manual fix and recreating the package from source files. But it does not go easy because one has to leverage different and outdated versions of HBase and Phoenix to create "Phoenix client" package matching HDP 3.1.</div><div>So two days spent without a solution.</div>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-47436549023923774112020-08-07T05:02:00.000-07:002020-08-07T05:02:01.762-07:00HDP 3.1, HBase Phoenix and IBM Java<p> I spent several sleepless nights trying to sort a nasty problem. While connecting to HBase Phoenix, the Phoenix client was stalled before displaying the prompt.
<br />
<myquo>
/usr/hdp/current/phoenix-client/bin/sqlline.py<br />
Setting property: [incremental, false]<br />
Setting property: [isolation, TRANSACTION_READ_COMMITTED]<br />
issuing: !connect jdbc:phoenix: none none org.apache.phoenix.jdbc.PhoenixDriver<br />
Connecting to jdbc:phoenix:<br />
SLF4J: Class path contains multiple SLF4J bindings.<br />
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/phoenix/phoenix-5.0.0.3.1.0.0-78-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]<br />
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.0.0-78/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]<br />
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.<br />
20/08/07 11:28:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br />
</myquo>
<br />
What's more, there was no single error message, neither from the client nor in the HBase log file, Master or Region Servers. Simply nothing, no clue. Besides that, everything was healthy and running soundly.</p><p>The first step toward the solution was to enable tracing in Phoenix client. This setting is not managed by Ambari and requires manual modification.</p>
<myquo>
vi /usr/hdp/3.1.0.0-78/phoenix/bin/log4j.properties<br />
....<br />
#psql.root.logger=WARN,console<br />
psql.root.logger=TRACE,console<br />
</myquo>
<br />
After TRACING was enabled, the sqlline client was more talkative and provided the first clue.
<myquo>
INFO client.RpcRetryingCallerImpl: Call exception, tries=6, retries=36, started=4724 ms ago, cancelled=false, msg=Call to data2-worker.mycloud.com/10.135.118.222:16020 failed on local exception: javax.security.sasl.SaslException: Failure to initialize security context [Caused by org.ietf.jgss.GSSException, major code: 13, minor code: 0<br />
major string: Invalid credentials<br />
minor string: SubjectCredFinder: no JAAS Subject], details=row 'SYSTEM:CATALOG' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=data2-worker.mycloud.com,16020,1596794212924, seqNum=-1<br />
</myquo>
The HDP cluster was Kerberized and the problem seemed to be related to Kerberos authentication. Kerberos is the source of many problems but everything except this worked smoothly here.<div>Finally, I found the hint under this <a href="http://mail-archives.apache.org/mod_mbox/hbase-dev/202004.mbox/%3CJIRA.13300342.1587582823000.26168.1587582840143@Atlassian.JIRA%3E">link</a>. The culprit was IBM Java. The cluster is running under control of OpenJDK Java but the sqlline client is using default host Java.
<myquo>
update-alternatives --config java<br />
<br />
There are 3 programs which provide 'java'.<br />
<br />
Selection Command<br />
-----------------------------------------------<br />
*+ 1 /usr/lib/jvm/java-1.8.0-ibm-1.8.0.5.10-1jpp.1.el7.x86_64/jre/bin/java<br />
2 /usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64/jre/bin/java<br />
3 /usr/java/jdk1.8.0_202-amd64/jre/bin/java<br />
<br />
</myquo></div>
So the solution was extremely simple, just switch to OpenJDK Java and Phoenix client works like a dream.Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-74705278632133872282020-07-31T12:13:00.001-07:002020-07-31T12:24:59.776-07:00DB2 ODBC FDW PostgreSQL extensionI reinvigorated my Postgresql DB2 <a href="https://www.postgresql.org/docs/12/postgres-fdw.html">wrapper</a>. Now it works also with PostgreSQL 11 and 12. The source code and installation instruction are available <a href="https://github.com/stanislawbartkowski/db2odbc_fdw">here</a>.<div><br /><div>It took me some time to accomplish it. The PostgreSQL failed because of memory violation related problem after returning from GetForeignPaths function. The crashed happened inside the server code. What's more, it worked after recompiling the server directly from source code so nailing down the problem was impossible. Finally, the solution was to migrate the <a href="https://github.com/stanislawbartkowski/db2odbc_fdw/blob/master/db2odbc_fdw.c">db2odbc_fdw.c</a> to pure C code and get rid of all warnings. Probably the reason for the failure was a discrepancy between data types in C function signature and launching sequence.</div></div>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-34915585576060418682020-06-30T13:13:00.000-07:002020-06-30T13:13:44.079-07:00My TPC/DS, new features<h3 style="text-align: left;">Introduction</h3><div>I uploaded a new version of <a href="https://github.com/stanislawbartkowski/mytpcds">mytpcds</a>, a wrapper around <a href="http://www.tpc.org/tpcds/default5.asp">TPC/DS</a> benchmark allowing easy and quick roll-out of TPC/DS test against leading RDMS including Hadoop SQL engines. Just deploy, configure using a template provided and run. In the new version, the implementation of Query Validation Test is added.</div><h3 style="text-align: left;">Query Validation Test</h3><div>Query Validation Test verifies the accuracy of SQL engine. RDBMS is to run a sequence of SQL statements, called Validation Queries, on Qualification database and compare the result data set against the expected data set. During Validation Test, the queries should come back with the same result. The Validation Queries are standard TPC/DS queries templates having where query parameters are substituted by predefined constants.</div><div>The substitution values for Validation Queries are defined in "TPC-DS Specification" manual, chapter "Appendix B: Business Questions". It is a mundane and error-prone task to prepare Validation Queries manually in 99 queries templates and I wanted also to avoid having two different versions of TPC/DS queries. So I decided to make the process automatic. Firstly I extracted all parameters and substitution values into separate <a href=" https://github.com/stanislawbartkowski/mytpcds/tree/master/qualification">configuration files</a>. The name convention is <n>.par. The <n> maps to appropriate TPC/DS query. For instance, 1.par contains substitution values for query1.tpl.</div><div></div><blockquote><div>YEAR=2000</div><div><div>STATE=TN</div><div>AGG_FIELD=SR_RETURN_AMT</div></div></blockquote><div><div></div></div><div>The <a href="https://github.com/stanislawbartkowski/mytpcds/blob/master/sh.templates/run.sh">run.sh</a> launcher contains a separate task: <b>./tpc.sh queryqualification</b>. This task replaces all parameters placeholder with the corresponding validation values and put them in <TPC/DS root dir>/work/{dbtype}queries directory ready to be picked up by other <b>run.sh</b> tasks.</div><div>TPC/DS package comes with expected data sets for Query Validation. It is included in <TPC/DS root>/answer_sets. Unfortunately, the format of answer sets is not consistent which makes impossible the automated verification. So I prepared my own version of <a href="https://github.com/stanislawbartkowski/mytpcds/tree/master/res">answer sets</a> using DB2 output. Unfortunately, it does not comply with output from other RDBMS including Hive, so it is still a pending task which output is invalid.</div><h4 style="text-align: left;">QueryRunner</h4><div>Query Validation Test requires comparing the current result sets against a reference result set. Unfortunately, the output from different RDBS using the command line client varies significantly which make automated comparison impossible. So I decided to prepare my own Java<a href="https://github.com/stanislawbartkowski/mytpcds/tree/master/RunQueries"> QueryRunner </a>using JDBC and have the full control of how the result is produced. The target jar is produced by <b>mvn package</b> command. The only prerequisite for every database is JDBC driver jar. So far, I tested the QueryRunner for: NPS/Netezza, DB2, IBM BigSql, SqlServer and Hadoop Hive.</div><h4 style="text-align: left;">QueryRunner, Hive and Kerberos</h4><div>Because life is never an easy road free of stones, the real challenge was to execute QueryRunner for Hadoop/Hive in Kerberized environment. The parameters regarding Kerberos cannot be included in URL connection string. Before executing <i>DriverManager.getConnection(url, user, password); </i> the client should authenticate in Hadoop cluster using Hadoop related libraries. Of course, I wanted to avoid keeping two versions of QueryRunner and have the development consistent. So I developed a separate package <a href="https://github.com/stanislawbartkowski/mytpcds/tree/master/HadoopAuth">HadoopAuth</a> and in Hadoop/Hive environment, the Hadoop Kerberos authentication is done using HadoopAuth package but through Java reflection feature. This way I was able to keep QueryRunner clean. How to configure QueryRunner for Hadoop/Hive is described <a href="https://github.com/stanislawbartkowski/mytpcds/wiki/Hive#runqueries-client-with-kerberos-authentication">here</a>.</div><h4 style="text-align: left;">Next steps</h4><div><ul style="text-align: left;"><li>Further analysis of reference QueryValidation answer tests.</li><li>Add Microsoft SqlServer to RDBMS supported by myTPC-DS.</li><li>Enable QueryRunner for all RDBMS supported.</li></ul></div><div><br /></div><div><br /></div><div><br /></div>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-50882962543425434462020-05-31T12:08:00.001-07:002020-05-31T12:21:00.202-07:00Simple RestService library continued<h3 style="text-align: left;">Introduction</h3><div>I enhanced my Simple <a href="http://hoteljavaopensource.blogspot.com/2020/02/simple-restservice-library.html">RestService</a> library to a little more complex but still simple.</div><div>The full source code is <a href="https://github.com/stanislawbartkowski/RestService/tree/master/src/main/java/com/rest/restservice">here</a>. </div><div>The sample application depending on it is <a href="https://github.com/stanislawbartkowski/MockRestService">here</a>.</div><div>The principle behind this project is to set up a RestAPI server based on Java only without any additional dependencies. I added two features: SSL and Kerberos authentication.</div><h4 style="text-align: left;">SSL </h4><div>Java source code: <a href="https://github.com/stanislawbartkowski/RestService/blob/master/src/main/java/com/rest/restservice/ssl/SecureHttp.java">here</a></div><div>Allows to set up HTTPS RestService. The certificate can be self-signed or CA signed. Client certificate authentication is not supported.</div><h4 style="text-align: left;">Kerberos</h4><div>Java Source code: <a href="https://github.com/stanislawbartkowski/RestService/tree/master/src/main/java/com/rest/restservice/kerberos">here</a></div><div>Allows Kerberos authentication. Tested with AD and MIT KDC. Only authentication is implemented, no DoAs action.</div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><div><br /></div><br />Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-30122334366468780162020-04-30T13:06:00.000-07:002020-05-31T12:13:57.450-07:00My private CA CenterCertificates, self-signed certificates, certificates signed by Certificate Authority, sounds complicated. But the devil is not so black as he's painted. It is easy to create a self-signed certificate but sometimes one needs to have a CA-signed certificate without paying fees. So be the authority for yourselves.
I found a <a href="https://jamielinux.com/docs/openssl-certificate-authority/index.html">very good article</a> on how to create private CA using open-source tools. But following the procedure manually is not a good method of spending your free time, so I created a solution which automates it all.<br />
The solution and description are available <a href="https://github.com/stanislawbartkowski/CACenter">here</a>. The solution comes with three components.<br />
<ul>
<li>Bash script <a href="https://github.com/stanislawbartkowski/CACenter/blob/master/ca.sh">ca.sh</a>. Script automates procedure described in the <a href="https://jamielinux.com/docs/openssl-certificate-authority/index.html">article</a>. Creates a new Certificate Authority containing root and intermediate certificates. Also produces a certificate signed by the CA using CSR (Certificate Signing Request) or by providing all necessary data including CN.</li>
<li>Java server Rest/API. Assuming CA is created, the Java server generates signed certificate through Rest/API.</li>
<li>Docker script. The CA centre is created during Docker image creation and the container exposes Rest/API for certificate signing.</li>
</ul>
Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-54269792145574937312020-03-23T07:09:00.000-07:002020-05-31T12:14:09.048-07:00HDP 3.1.5, OpenJDK, Infra Solr and AD/Kerberos<span style="font-size: large;">Problem </span><br />
I spent several sleepless nights caused by very nasty problem coming up after HDP 3.1.5 Kerberization. The Infra Solr components could not start just blocking the whole cluster. The message in the Ambari Console was saying.<br/>
<pre>Skip /infra-solr/configs and /infra-solr/collections
Set world:anyone to 'cr' on /infra-solr/configs and /infra-solr/collections
KeeperErrorCode = NoAuth for /infra-solr/configs
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /infra-solr/configs
at org.apache.zookeeper.KeeperException.create(KeeperException.java:113)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.setACL(ZooKeeper.java:1399)
</pre>
It looked that infra-solr user could not update the ZooKeeper /infra-solr znode because of not sufficient privileges.
But the ACL privileges looked correct.
<br/>
<myquo>
[zk: localhost:2181(CONNECTED) 0] getAcl /infra-solr<br/>
'sasl,'infra-solr<br/>
: cdrwa<br/>
'world,'anyone<br/>
: r<br/>
[zk: localhost:2181(CONNECTED) 1] <br/>
</myquo>
<br />
After closer examination, I discovered strange stuff in ZooKeeper log.
<myquo>
2020-03-23 01:33:12,260 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@120] - Successfully authenticated client: authenticationID=$6O1000-3NO0GILCOJUA@FYRE.NET; authorizationID=infra-solr/a1.fyre.ibm.com@FYRE.NET.<br/>
2020-03-23 01:33:12,261 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: $6O1000-3NO0GILCOJUA<br />
2020-03-23 01:33:12,261 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: $6O1000-3NO0GILCOJUA<br />
2020-03-23 01:33:24,011 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception<br />
EndOfStreamException: Unable to read additional data from client sessionid 0x17104ad52230007, likely client has closed socket<br />
</myquo>
So it seemed that Zookeeper to apply authorization rights was using not AD principal name (infra-solr) but sAMAccountName attribute of infra-solr AD principal ($6O1000-3NO0GILCOJUA). Ambari Kerberos Wizard is filling this attribute with random data only to keep it unique.<br />
<span style="font-size: large;">Solution </span><br />
The problem is described <a href="https://bugs.openjdk.java.net/browse/JDK-8239385">here</a>, it is the bug coming with 1.8.0_242 version of OpenJDK.<br/>
The only workaround is to downgrade the OpenJDK to 232 level or switch to Oracle JDK.<br/>
<myquo>
yum downgrade java*<br/>
<br/>
java -version<br/>
openjdk version "1.8.0_232"<br/>
OpenJDK Runtime Environment (build 1.8.0_232-b09)<br/>
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)<br/>
</myquo>
<br/>
And last but not least.<br />
Block the Java upgrade unless the bug is fixed.<br />
<myquo>
vi /etc/yum.conf <br />
<br />
exclude=java*
<br />
</myquo>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-3001198105281237842020-02-29T13:30:00.000-08:002020-02-29T13:30:11.072-08:00Simple RestService library<span style="font-size: large;">Motivation</span><br />
REST API is the mean of choice to communicate between different loosely coupled applications. There are plenty of REST API implementations but I was looking for a solution as simple as possible with minimal external dependencies or prerequisites. Finally, I ended up with the compact library utilizing existing in Java JDK <a href="https://docs.oracle.com/javase/8/docs/jre/api/net/httpserver/spec/com/sun/net/httpserver/HttpServer.html">HttpServer.</a><br />
<span style="font-size: large;">Links</span><br />
Intellij IDEA project, <a href="https://github.com/stanislawbartkowski/RestService">source code</a> and <a href="https://stanislawbartkowski.github.io/javadoc/RestService/">javadoc</a>.<br />
<a href="https://github.com/stanislawbartkowski/MockRestService">Sample project </a>utilizing the RestService project.<br />
<a href="https://github.com/stanislawbartkowski/CivilizationREST">Another project </a>having RestService dependency.<br />
<span style="font-size: large;">Highlights</span><br />
<ul>
<li>Very lightweight, no external dependency, just Java JDK.</li>
<li>Can be dockerized, <a href="https://github.com/stanislawbartkowski/CivilizationREST/blob/master/Dockerfile">sample Dockerfile</a>.</li>
<li>Adds-on making a life of developer easier.</li>
<ul>
<li>Validating and extracting URL query parameters including type control.</li>
<li>Upload data</li>
<li>CORS relaxation</li>
<li>Sending data with valid HTTP response code.</li>
</ul>
<li>"Dynamic" and "static" REST API call. "Dynamic" means that specification of particular REST API endpoint can be defined after the request reached the server but before handling the request thus allowing providing different custom logic according to the URL path.</li>
</ul>
<div>
<span style="font-size: large;">Usage</span></div>
<div>
The service class should extend <a href="https://github.com/stanislawbartkowski/RestService/blob/master/src/main/java/com/rest/restservice/RestHelper.java">RestHelper.RestServiceHelper</a> abstract class and implements two methods:<br />
<ul>
<li>getParams : delivers REST API call specification (look below) including URL query parameter definition. The method is called after the REST API request accepted by HTTP Server but before validating and running the call. </li>
<li>servicehandle: custom logic to serve the particular REST API endpoint. The method should conclude handling the request by proper "produceresponse" call. The "servicehandle" can take a URL query parameters and utilize several helper methods.</li>
</ul>
</div>
<div>
<span style="font-size: large;">REST API specification</span></div>
The REST API endpoint specification is defined through <a href="https://github.com/stanislawbartkowski/RestService/blob/master/src/main/java/com/rest/restservice/RestParams.java">RestParams</a> class. The specification consists of:<br />
<ul>
<li>HTTP request method: GET, POST, PUT etc</li>
<li>List of allowed URL query parameters. Three <a href="https://github.com/stanislawbartkowski/RestService/blob/master/src/main/java/com/rest/restservice/PARAMTYPE.java">parameters</a> type are supported: BOOLEAN, INT and STRING (meaning any other).</li>
<li>CORS should be relaxed for this particular endpoint.</li>
<li>Response type content (TEXT, JSON or not specified), Content-Type.</li>
<li>List of methods allowed in the response header, Access-Control-Allow-Methods.</li>
</ul>
<div>
<span style="font-size: large;">Main</span></div>
<div>
The main class should extend <a href="https://github.com/stanislawbartkowski/RestService/blob/master/src/main/java/com/rest/restservice/RestStart.java">RestStart</a> abstract class.</div>
<br />
<br />
<br />
<br />
<br />
<br />Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-86291705620611537762020-02-04T07:43:00.001-08:002020-02-04T07:43:46.686-08:00HDP 3.1 and Spark jobI spent several sleepless nights trying to solve the problem while running <a href="https://github.com/stanislawbartkowski/ConnectHBase">Spark/HBase application</a>. The application was dying giving the nasty error stack.<br />
<myquo>
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown
at io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:587)
... 22 more
Caused by: java.lang.NoSuchMethodError: org.apache.spark.network.util.AbstractFileRegion.transferred()J
at org.apache.spark.network.util.AbstractFileRegion.transfered(AbstractFileRegion.java:28)
at io.netty.channel.nio.AbstractNioByteChannel.doWrite(AbstractNioByteChannel.java:228)
at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:282)
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:879)
</myquo>
<br />
Usually problems like that point at versioning problem. But how could it happen if the application while running is dependent only on the client libraries provided by the HDP cluster?<br />
Finally, after browsing through source code and comparing different versions of libraries, I crawled out of the swamp.<br />
The culprit was an incompatible library in HBase client directory, <b><a href="https://github.com/taojiaenx/netty-source-study/blob/master/transport/src/main/java/io/netty/channel/nio/AbstractNioByteChannel.java">netty-all-4.0.52.Final.jar</a></b>. This library is calling deprecated <b>transfered</b> method which, in turn, calls a non-existing <a href="https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/util/AbstractFileRegion.java"><b>transferred</b> </a>method in Spark class. New version<a href="https://jar-download.com/artifacts/io.netty/netty-all/4.1.3.Final/source-code/io/netty/channel/nio/AbstractNioByteChannel.java"> netty-all-4.1.17.Final.jar </a>calls a correct <b>transferred</b> method.<br />
<br />
The solution was dazzling simple. Just reverse the order of classpath in submit-spark command and give precedence to correct libraries in spark client jars.<br />
<myquo>spark-submit ... --conf "spark.driver.extraClassPath=$CONF:$LIB" ... </myquo>
<ul>
<li>wrong: LIB=/usr/hdp/current/hbase-client/lib/*:/usr/hdp/current/spark2-client/jars/*</li>
<li>correct: LIB=/usr/hdp/current/spark2-client/jars/*:/usr/hdp/current/hbase-client/lib/*</li>
</ul>
<br />
Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-74830330834329014962020-01-31T03:13:00.000-08:002020-01-31T03:13:11.024-08:00Kerberos and Phoenix<a href="https://phoenix.apache.org/">Phoenix </a>is a SQL solution on the top of <a href="https://hbase.apache.org/">HBase</a>. There are two methods connecting to Phoenix, using full JDBC driver and thin JDBC driver. The latter requires additional server component, <a href="https://phoenix.apache.org/server.html">Phoenix Query Server</a>.<br />
I created the article about how to connect to Phoenix using both methods in kerberized environment. I also created a simple <span style="color: #27282c; font-family: , , "blinkmacsystemfont" , "segoe ui" , "roboto" , "oxygen" , "ubuntu" , "cantarell" , "droid sans" , "helvetica neue" , "arial" , sans-serif; font-size: 15px; font-weight: bold; letter-spacing: -0.1px; white-space: nowrap;">IntelliJ IDEA </span>project demonstrating a connection to Phoenix from Java program. Both JDBC drivers are supported.<br />
I also added information on how to connect to Phoenix from Zeppelin notebook.<br />
The article and sample project are available <a href="https://github.com/stanislawbartkowski/ConnectPhoenix">here</a>.<br />
<br />
AMS (Ambari Metrics System) is powered by standalone HBase and Phoenix server. Using a command line, one can get access to AMS Phoenix and query the metrics without going to UI. I also added information on how to accomplish it.<br />
I was successful only by launching the command tool from the node where AMS is installed. I will work how to access it remotely.Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-49620119280920452652019-12-31T09:34:00.003-08:002019-12-31T09:36:15.387-08:00MyTPC-DS, An enhancement to <a href="https://github.com/stanislawbartkowski/mytpcds">my implementation </a>of <a href="http://www.tpc.org/tpcds/">TPC-DS</a> test. BigSQL test can be conducted by jsqsh utility, the dependency on DB2 client software is removed. Jsqsh is the part of BigSQL package and no additional dependency software is required now. How to use jsqsh for MyTPC-DS is described <a href="https://github.com/stanislawbartkowski/mytpcds/wiki/IBM-BigSQL,-jsqsh">here</a>.<br />
<br />
I also added some <a href="https://github.com/stanislawbartkowski/mytpcds/wiki/TPC-DS-BigData-results">additional results</a> of the TPC-DS test conducted on <a href="https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/">HDP 3.1</a>. The result is presented in the shape of Bar Plot to make them easier to understand. The graphic is created with the help of <a href="https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.bar.html">matplotlib</a>. The Python source code is uploaded <a href="https://github.com/stanislawbartkowski/mytpcds/tree/master/tpcgraph/src">here</a>. The input tables are taken directly from <a href="https://github.com/stanislawbartkowski/mytpcds/wiki/TPC-BigData-Result-Tables">GitHub Wiki page</a>. The pick up the correct table, a simple HTML tag is added to the page before every table.<br />
<br />
The tests were executed using 100 GB data set and cannot be the basis of any far-reaching conclusion. But one apparent difference between HDP 2.6 and 3.1 is the significant performance improvement of Hive, from Hive 2.1 to Hive 3.1. Now the performance of Hive is almost as effective as the performance of BigSQL and ahead of SparkSQL. Also, the query coverage of Hive is much more comprehensive, from 50% (Hive 2.1) to almost 100% (Hive 3.1).<br />
<br />
<span style="font-size: large;">Useful links</span><br />
<ul>
<li>MyTPC-DS execution framework, <a href="https://github.com/stanislawbartkowski/mytpcds">link</a></li>
<li>TPC-DS results using Bar Plot graphics, <a href="https://github.com/stanislawbartkowski/mytpcds/wiki/TPC-DS-BigData-results">link</a></li>
<li>TPC-DS results analytics tables, <a href="https://github.com/stanislawbartkowski/mytpcds/wiki/TPC-BigData-Result-Tables">link</a> </li>
<li>Python source code to compile the test result and prepare GitHub Wiki table, <a href="https://github.com/stanislawbartkowski/mytpcds/tree/master/prepres">link</a></li>
<li>Python source code to prepare graphics using matplotlib package, <a href="https://github.com/stanislawbartkowski/mytpcds/tree/master/tpcgraph/src">link</a></li>
<li>Run BigSQL TPC-DS test using jsqsh tool, <a href="https://github.com/stanislawbartkowski/mytpcds/wiki/IBM-BigSQL,-jsqsh">link</a></li>
</ul>
<br />
<br />Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-78419375394460913722019-11-06T05:41:00.002-08:002019-11-06T05:41:42.994-08:00Hortonworks, Yarn Timeline server v.2<span style="font-size: large;">Problem</span><br />
I spent several sleepless nights trying to resolve an issue with MapReduce jobs in HDP 3.1. While running MapReduce Service Check job, in the log file an error was reported although the job passed in green. The same message came up in others MapReduce jobs.<br />
<myquo>
ERROR [pool-10-thread-1] org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl: Response from the timeline server is not successful, HTTP error code: 500, Server response:<br />
{"exception":"WebApplicationException","message":"org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 252 actions: IOException: 252 times, servers with issues: null","javaClassName":"javax.ws.rs.WebApplicationException"}<br />
[Job ATS Event Dispatcher] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Failed to process Event JOB_FINISHED for the job : job_1572998114693_0003
org.apache.hadoop.yarn.exceptions.YarnException: Failed while publishing entity<br />
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:548)<br />
at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149)<br />
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processEventForNewTimelineService(JobHistoryEventHandler.java:1405)<br />
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleTimelineEvent(JobHistoryEventHandler.java:742)<br />
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.access$1200(JobHistoryEventHandler.java:93)<br />
at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.hand<br />
</myquo>
In the yarn log file (<b>/var/log/hadoop-yarn/yarn</b>) more detailed message could be found.
<myquo>
Caused by: java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure/meta-region-server<br />
at org.apache.hadoop.hbase.client.ConnectionImplementation.get(ConnectionImplementation.java:2002)<br />
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateMeta(ConnectionImplementation.java:762)<br />
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:729)<br />
at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:707)<br />
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:911)<br />
at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:732)<br />
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:325)<br />
... 17 more<br />
</myquo>
<br />
The problem is related to Yarn Timeline server v.2. The service is using HBase as backup storage and it looked that this embedded HBase server was not started or not working properly. HBase is keeping its runtime configuration data like active HBase Region Serves in Zookeeper znodes.
<br />
Also, the znode <b>/atsv2-hbase-secure</b> did not exist.<br />
<myquo>
zkCli.sh<br />
ls /<br />
<br />
[hive, cluster, brokers, hbase-unsecure, kafka-acl, kafka-acl-changes, admin, isr_change_notification, atsv2-hbase-secure, log_dir_event_notification, kafka-acl-extended, rmstore, hbase-secure, kafka-acl-extended-changes, consumers, latest_producer_id_block, registry, controller, zookeeper, delegation_token, hiveserver2, controller_epoch, hiveserver2-leader, atsv2-hbase-unsecure, ambari-metrics-cluster, config, ams-hbase-unsecure, ams-hbase-secure]
<br />
</myquo><br />
Embedded HBase service is running as Yarn service named <b>ats-hbase</b>. I started to examine the local container the <b>ats-hbase</b> is running in. It is the directory in the shape of:<br />
/data/hadoop/yarn/local/usercache/yarn-ats/appcache/application_1573000895404_0001/container_e82_1573000895404_0001_01_000003. And it turned out, that the local hbase-site.xml configuration file defines the znode as atsv2-hbase-unsecure <br />
<myquo>
<property><br />
<name>zookeeper.znode.parent</name><br />
<value>/atsv2-hbase-unsecure</value><br />
</property> <br />
</myquo>
<br />
So the embedded HBase was storing its configuration in improper Zookeper znode. But how it could happen when the Yarn configuration parameter defines the znode as<b> atsv2-hbase-secure</b> and where the other services expect to find the embedded HBase runtime data. Also, the HDFS <b>/user/yarn-ats/3.1.0.0-78/hbase-site.xml </b>file, it is the source file used by Application Master to create a local container directory, contains a valid znode value.<br />
<span style="font-size: large;">Solution</span><br />
After closer examination of the container log files, I discover the following entry:<br />
<myquo>
INFO provider.ProviderUtils: Added file for localization: conf/hadoop-metrics2-hbase.properties -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hadoop-metrics2-hbase.properties<br />
INFO provider.ProviderUtils: Added file for localization: conf/hbase-policy.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hbase-policy.xml<br />
INFO provider.ProviderUtils: Added file for localization: conf/core-site.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/core-site.xml
INFO provider.ProviderUtils: Added file for localization: conf/hbase-site.xml -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/hbase-site.xml<br />
INFO provider.ProviderUtils: Added file for localization: conf/log4j.properties -> /user/yarn-ats/.yarn/services/ats-hbase/components/1.0.0/master/master-0/log4j.properties<br />
</myquo>
So it turned out, that Application/Serrvice Master is using not <b>/user/yarn-ats/3.1.0.0-78/ </b>HDFS path but another hidden<b> /user/yarn-ats/.yarn/ </b>directory to create the local container. <br />
Finally, the solution was quite simple:<br />
<ul>
<li>kill the ats-hbase Yarn application</li>
<li>stop the Yarn service</li>
<li>remove /user/yarn-ats/.yarn HDFS directory</li>
<li>start the Yarn service</li>
</ul>
Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-67375659587222666402019-10-30T15:54:00.001-07:002019-10-30T15:55:33.580-07:00HDP, Kafka, LEADER_NOT_AVAILABLE<span style="font-size: large;">Problem</span><br />
After HDP cluster kerberization, the Kafka does not work even though Kafka Healthcheck passes green. Any execution of <b>kafka-console-producer.sh </b>ends up with the error message:
<myquo>
/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list kafka-host:6667 --producer-property security.protocol=SASL_PLAINTEXT --topic xxx<br />
<br />
WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)<br />
WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 2 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)<br />
WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {xxxx=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)<br />
</myquo><b>
kafka-topics.sh</b> reports that the leader is not assigned to the partition.<br />
<myquo>
/usr/hdp/3.1.0.0-78/kafka/bin/kafka-topics.sh -zookeeper zookeeper-host:2181 --describe xxx --unavailable-partitions<br />
<br />
Topic: ambari_kafka_service_check Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001<br />
Topic: identity Partition: 0 Leader: none Replicas: 1002 Isr: <br />
Topic: xxx Partition: 0 Leader: none Replicas: 1002 Isr: <br />
Topic: xxxx Partition: 0 Leader: none Replicas: 1002 Isr: <br />
</myquo><br />
There is nothing special in Kafka /var/log/kafka/server.log file. Only /var/log/kafka/controller.log suggests that something went wrong:
<myquo>
cat controller.log<br />
<br />
INFO [ControllerEventThread controllerId=1002] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)<br />
ERROR [ControllerEventThread controllerId=1002] Error processing event Startup (kafka.controller.ControllerEventManager$ControllerEventThread)<br />
java.lang.NullPointerException<br />
at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:857)<br />
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2571)<br />
at kafka.utils.Json$.parseBytes(Json.scala:62)<br />
at kafka.zk.ControllerZNode$.decode(ZkData.scala:56)<br />
at kafka.zk.KafkaZkClient.getControllerId(KafkaZkClient.scala:902)<br />
at kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1199)<br />
</myquo><br />
<span style="font-size: large;">Solution</span><br />
Uncle Google brings back many entries related to LEADER_NOT_AVAILABLE error but none of them led to the solution. Finally, I found <a href="https://community.cloudera.com/t5/Support-Questions/No-controller-for-Kafka-cluster-after-kerberos/td-p/205686">this entry</a>.<br />
So the healing is very simple.<br />
<ul>
<li>Stop Kafka</li>
<li>Run zkCli.sh Zookeeper command line</li>
<li>Remove /controller znode, rmr /controller</li>
<li>Start Kafka again</li>
</ul>
The evil spell is defeated.
Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-82985793816322179302019-10-29T13:13:00.000-07:002019-10-29T13:16:15.001-07:00RedHat, Steam and Crusaders King 2I'm playing <a href="https://store.steampowered.com/app/203770/Crusader_Kings_II/">King Crusaders II </a>on my RedHat desktop for some time. Although Steam officially supports only Ubuntu distribution, the games, particularly the older ones developed for Ubuntu 16.04, work fine also on RedHat/CentOS. Unfortunately, after the latest background update of the King Crusaders, the game stubbornly refused to run. After closer examination, I discovered that probably the updated game was compiled using newer version of GNU GCC compiler and the required level of <b>libstdc++.so.6 </b>library is not available on my platform.<br />
<myquo>
./ck2: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./ck2)<br />
./ck2: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./ck2)
</myquo><br />
So it was bad news. But there is good news, after the epic battle I straightened it out. It was as simple as installing a newer version of GNU GCC to get more modern libraries.<br />
The solution is described <a href="https://github.com/stanislawbartkowski/wikis/wiki/RedHat-7.x-and-Steam">here</a>.Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com0tag:blogger.com,1999:blog-2399759103741849897.post-40037588976864897172019-10-23T15:56:00.002-07:002019-10-30T16:00:23.214-07:00BigSQL 6.0 and HDP 3.1.4<span style="font-size: large;">Problem</span><br />
There is a problem with BigSQL 6.0 installed on the top HDP 3.1.4 or after upgrade from HDP 3.1. Look at this product support <a href="https://www.ibm.com/software/reports/compatibility/clarity-reports/report/html/prereqTabViewDetails?osPlatform=spcrAllValues&dsrType=tsr&deliverableId=64B37110F49711E783525C61262C1E6A&productName=IBM+Big+SQL+6.0.0.0&prereqId=0C2A7FD0BDCC11E88857DCC2171712A1&prereqName=Hortonworks+HDP+3.1&capName=Data+Integration&isDUOnly=true&isMandatory=true&osPlatforms=Linux&duComponentIds=S000">web page</a>. Installation is successful but there are plenty of entries in BigSQL diagnostic blog<br />
<myquo>
java.lang.NoSuchMethodError: com/google/common/base/Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V (loaded from file:/usr/ibmpacks/bigsql/6.0.0.0/bigsql/lib/java/guava-14.0.1.jar by sun.misc.Launcher$AppClassLoader@28528006) called from class org.apache.hadoop.conf.Configuration (loaded from file:/usr/hdp/3.1.4.0-315/hadoop/hadoop-common-3.1.1.3.1.4.0-315.jar by sun.misc.Launcher$AppClassLoader@28528006).<br />
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1358)<br />
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1339)<br />
at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518)<br />
at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:536)
</myquo>
<br />
<span style="font-size: large;">Solution</span><br />
The cause of the problem is the old <b>guava</b> jar file in the <b>/usr/ibmpacks/bigsql/6.0.0.0/bigsql/lib/java</b>. Replace the old guava with any new guava greater than 20. Or simply make a link to existing guava at the proper level. The fix should be applied on all BigSQL nodes, Head and Worker nodes. The BigSQL should be restarted to make the change to take effect.
<br />
<myquo>
rm /usr/ibmpacks/bigsql/6.0.0.0/bigsql/lib/java/guava-14.0.1.jar<br />cd /usr/ibmpacks/bigsql/6.0.0.0/bigsql/lib/java/<br />
ln -s /usr/hdp/3.1.4.0-315/hadoop/lib/guava-28.0-jre.jar<br />
</myquo>Stanisławhttp://www.blogger.com/profile/09580030186766542992noreply@blogger.com1