kyuubi supports spark engine multi-mode at the same time

[942011334]

Can kyuubi support spark engine multi-mode at the same time？For example, when we use CONNECTION mode, we also use USER mode in a kyuubi server.

[942011334]

@yaooqinn

[yaooqinn]

yes, it can be configured in each connection url.

[yaooqinn]

For further reading,

https://mp.weixin.qq.com/s/gi8w2oCsXqAiXn9blv9BBw
https://kyuubi.apache.org/docs/r1.4.0-incubating/deployment/engine_share_level.html#hybrid

[pan3793]

Yes, you can overwrite the specific conf in connection URL. e.g.

beeline -n batch_user -u 'jdbc:hive2://kyuubi:10009/;?spark.driver.memory=8G#kyuubi.engine.share.level=CONNECTION'

[kmsantanu]

which version of the kyuubi server it is supportin?

[pan3793]

Any version after 1.x, we recommend to use the latest version.

[kmsantanu]

how to solve our problem in production
problem 1:

1. We have deploy kyubi on machine-1 and we want two different users can connect kyuubi engine with same spark session.
2. We have one user and this user has two task which is running paralle .How kyubi can run the parallel request like count and groupby operation from a same user.
3. we are using cluster mode spark deployment like .one master node and 4 slave node .
   Each slave has 80gb ram and 24core.Now one kyubi engine is taking all resources and I can not start my other spark application.what will optimise kyuubi configuration for production to run multiple user with limited resources

[kmsantanu]

what is the different between kyuubi.engine.share.level=CONNECTION and kyuubi.engine.share.level=SERVER/GROUP
will it support on kyuubi server Kyuubi 1.2.0 (git revision a6ab236).
can you suggest which stable kyuubi version we can upgrade in our production and currntly we are using Kyuubi 1.2.0

[kmsantanu]

is it required to configure external databse to manage metadata configuration like mysql

[pan3793]

The latest stable version is v1.4.1-incubating, you can download it from https://kyuubi.apache.org/releases.html

[pan3793]

No, currently Kyuubi does not require MySQL or something to store metadata, maybe need in future.

[pan3793]

what is the different between kyuubi.engine.share.level=CONNECTION and kyuubi.engine.share.level=SERVER/GROUP

You can learn it from https://kyuubi.apache.org/docs/latest/deployment/engine_share_level.html

[kmsantanu]

Recently Kyuubi Support launched query engine asynchronously during opening session .we did not understand can you please provide some examples

[kmsantanu]

Kyuubi has a concept names "engine share level", ,is it any dependency with derby database. In production we have seen derby databae is creating inside kyuubi. we are not using any hive query.we are using hive jdbc to run sql .this is the url "jdbc:hive2://system-west01:2181/proddb" we are using..
quetion 1: will derby db created automatically if we connect to bee-line or using url "jdbc:hive2://system-west01:2181/proddb"
quetion 2: is there a performnce issue because of creating derby database for using kyuubi new feature "engine share level",?

[kmsantanu]

second quetion: is it required to set both "engine share level" and user level dynamic partion like configuration example kent.spark.dynamicAllocation.maxExecutors=20
"kent.spark.dynamicAllocation.maxExecutors=20"
For example if we set kent.spark.dynamicAllocation.maxExecutors=20 and bob.spark.dynamicAllocation.maxExecutors=10 configuration with dynamic resource allocation then two kyuubi spark engine will start and serve the request.Then what is use of "engine share level" configuration.

can anyone gives the comments please

[yaooqinn]

ideally，we shall use the group name that both users belong to. but currently this is not supported yet. the current implementation will let the user come first define the engine settings

[kmsantanu]

@[yaooqinn] as your above answer ,I understood that engine share leve setting to group level is only option and as per kyubi documentation setting of fwe dynamic allocation properties user lavel that is kent.spark.dynamicAllocation.maxExecutors=20 not working.

[yaooqinn]

@[yaooqinn] as your above answer ,I understood that engine share leve setting to group level is only option and as per kyubi documentation setting of fwe dynamic allocation properties user lavel that is kent.spark.dynamicAllocation.maxExecutors=20 not working.

Sorry, I don't quite understand what you were trying to tell me. Can you make yourself more clear?

[yaooqinn]

BTW, keep asking questions irrelevant to the original topic under a resolved discussion may disturb the irrelevant people. Would you mind creating a separate one, and making the question clear?

[zinking]

@kmsantanu that's what engine level provides you. you set it to user and then you get to setup two users with different DRA capabilities.

[kmsantanu]

can some one clarify above please