Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Generate a report of tasks on the core output #1546

Open
amahussein opened this issue Feb 18, 2025 · 0 comments
Open

[FEA] Generate a report of tasks on the core output #1546

amahussein opened this issue Feb 18, 2025 · 0 comments
Labels
core_tools Scope the core module (scala) feature request New feature or request

Comments

@amahussein
Copy link
Collaborator

Is your feature request related to a problem? Please describe.

The core tools generate reports about jobs/stages/sqls and failed entries including Tasks.
However, there is no report of the details of tasks. This sounds odd that we generated failed tasks but nothing about details of the tasks themselves.
This might be helpful to:

  • debug/verify the aggregations on higher level.
  • troubleshooting status and durations of different tasks
  • the consumer of the output has the ability to build their own logic to calculate the wall-clock-durations/sql-durations per app.

Describe the solution you'd like

  • no need to add the report to the text format. Tasks are huge number and we do not want that overhead.
  • Generate a new report csv file that
-   stageId: Int,
    stageAttemptId: Int,
    taskType: String,
    endReason: String,
    taskId: Long,
    attempt: Int,
    launchTime: Long,
    finishTime: Long,
    duration: Long,
    successful: Boolean,
    executorId: String,
    host: String,
    taskLocality: String,
    speculative: Boolean,
    gettingResultTime: Long,
@amahussein amahussein added ? - Needs Triage core_tools Scope the core module (scala) feature request New feature or request labels Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala) feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants