Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark Driver Pause for HWC SQL mode #286

Open
akashkale13 opened this issue Feb 26, 2023 · 0 comments
Open

Spark Driver Pause for HWC SQL mode #286

akashkale13 opened this issue Feb 26, 2023 · 0 comments

Comments

@akashkale13
Copy link

akashkale13 commented Feb 26, 2023 •
edited
Loading

I am using HWC in SQL mode to read data from Hive Acid table in Spark 2.3. Hive LLAP is not supported in deployed Hive setup hence using SQL mode. I am fetching about 1 GB data with millions of rows. One observation - the Hive query is executed in YARN UI however the Driver / Executor doesn't move to execution of next Spark Action entering a mode of pause. What is cause of it? Is it IO limiting threads between driver and executor for this data fetched?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@akashkale13