I spent several sleepless nights caused by very nasty problem coming up after HDP 3.1.5 Kerberization. The Infra Solr components could not start just blocking the whole cluster. The message in the Ambari Console was saying.
Skip /infra-solr/configs and /infra-solr/collections Set world:anyone to 'cr' on /infra-solr/configs and /infra-solr/collections KeeperErrorCode = NoAuth for /infra-solr/configs org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /infra-solr/configs at org.apache.zookeeper.KeeperException.create(KeeperException.java:113) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.setACL(ZooKeeper.java:1399)It looked that infra-solr user could not update the ZooKeeper /infra-solr znode because of not sufficient privileges. But the ACL privileges looked correct.
'sasl,'infra-solr
: cdrwa
'world,'anyone
: r
[zk: localhost:2181(CONNECTED) 1]
After closer examination, I discovered strange stuff in ZooKeeper log.
2020-03-23 01:33:12,261 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:SaslServerCallbackHandler@136] - Setting authorizedID: $6O1000-3NO0GILCOJUA
2020-03-23 01:33:12,261 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@1030] - adding SASL authorization for authorizationID: $6O1000-3NO0GILCOJUA
2020-03-23 01:33:24,011 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x17104ad52230007, likely client has closed socket
Solution
The problem is described here, it is the bug coming with 1.8.0_242 version of OpenJDK.
The only workaround is to downgrade the OpenJDK to 232 level or switch to Oracle JDK.
java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
And last but not least.
Block the Java upgrade unless the bug is fixed.
exclude=java*
Brak komentarzy:
Prześlij komentarz