Driver Error UnknownHostException: kyuubi-server-pod: Name or service not known

[preetsindhal]

Hi
any suggestion on how to fix this error from driver pod, driver is trying to connect to kyuubi pod and failing.

22/09/28 10:00:01 INFO ZooKeeper: Initiating client connection, connectString=kyuubi-server-6f7bb64bb-x74xb:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shade.org.apache.curator.ConnectionState@193e65d5
22/09/28 10:00:01 INFO X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
22/09/28 10:00:01 INFO ClientCnxnSocket: jute.maxbuffer value is 1048575 Bytes
22/09/28 10:00:01 INFO ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=false
22/09/28 10:00:01 INFO EngineServiceDiscovery: Service[EngineServiceDiscovery] is initialized.
22/09/28 10:00:01 ERROR StaticHostProvider: Unable to resolve address: kyuubi-server-6f7bb64bb-x74xb:2181
java.net.UnknownHostException: kyuubi-server-6f7bb64bb-x74xb: Name or service not known
	at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown Source)
	at java.base/java.net.InetAddress.getAddressesFromNameService(Unknown Source)
	at java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown Source)
	at java.base/java.net.InetAddress.getAllByName0(Unknown Source)
	at java.base/java.net.InetAddress.getAllByName(Unknown Source)
	at java.base/java.net.InetAddress.getAllByName(Unknown Source)
	at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:88)
	at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:141)
	at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:368)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1193)
22/09/28 10:00:01 INFO SparkTBinaryFrontendService: Service[SparkTBinaryFrontend] is initialized.
22/09/28 10:00:01 INFO SparkSQLEngine: Service[SparkSQLEngine] is initialized.
22/09/28 10:00:01 WARN ClientCnxn: Session 0x0 for sever kyuubi-server-6f7bb64bb-x74xb:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.IllegalArgumentException: Unable to canonicalize address kyuubi-server-6f7bb64bb-x74xb:2181 because it's not resolvable
	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:78)
	at org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41)
	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1147)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)

I have tried this kyuubi.frontend.connection.url.use.hostname=false as suggested here #2747 but not working.

Kyuubi version in use org.apache.kyuubi.server.KyuubiServer: Version: 1.6.0-incubating

[pan3793]

have you deployed zookeeper? the embedded zk does not suitable for production, and it does not work on K8s

[preetsindhal]

thanks for the response.
currently there's no zookeeper in use, let me configure one and retry.

[preetsindhal]

it worked with external zookeeper
thanks

[etheleon]

Hi @preetsindhal how did you set up the zookeeper?

Could you share

[etheleon]

Deployed zookeeper with this https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/
and set kyuubi.ha.addresses: zk-cs.<namespace>:2181

 k get svc -n kyuubi
NAME                   TYPE           CLUSTER-IP       EXTERNAL-IP                                                                   PORT(S)             AGE
zk-cs                  ClusterIP      10.100.171.138   <none>                                                                        2181/TCP            4d3h
zk-hs                  ClusterIP      None             <none>                                                                        2888/TCP,3888/TCP   4d3h