-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SUPPORT] Hudi Quickstart on EMR 6.15 with Hudi 1.0.1 not working #12963
Comments
@alberttwong Have you checked if |
I verified that with OSS Spark 3.4.1, the script works to create a fresh new Hudi table successfully. |
|
Likely, this is related to the EMR environment. Will check again. |
cc @CTTY |
happens also with EMR 7.6. #12974 |
Hi @alberttwong When you run in EMR, by default spark application will run in YARN mode. When operating in YARN mode, you need to specify the distributed file path, such as an HDFS or S3 path. To resolve this issue, you can specify the base path as an S3 or HDFS path, or you can launch the Spark shell in local mode by specifying --master "local[*]" |
I can try it but if we need update the quickstart. |
Describe the problem you faced
Following the quickstart at https://hudi.apache.org/docs/quick-start-guide/ on EMR 6.15.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Environment Description
Hudi version : 1.0.1
Spark version : 3.4
Hive version :
Hadoop version :
Storage (HDFS/S3/GCS..) :
Running on Docker? (yes/no) : no
The text was updated successfully, but these errors were encountered: