POC of code level optimization recommendation for Hive #618
Conversation
pralabhkumar
changed the title
| import org.codehaus.jettison.json.JSONObject; | ||
|
|
||
|
|
||
| public class AzkabanJarvisCodeExtractor implements CodeExtractor { |
There was a problem hiding this comment.
Jarvis is internal to LinkedIn. This should not go to open source ideally. Definitely the URL should be in config.
Can we modify it as a JSON code fetcher where the hierarchy of which node to fetch can be configured.
Or move this code to internal repo.
There was a problem hiding this comment.
Moved URL to configuration
| @@ -0,0 +1,121 @@ | |||
| package com.linkedin.drelephant.analysis.code.impl; | |||
There was a problem hiding this comment.
Add file header here and elsewhere
| @@ -331,6 +343,36 @@ public AppResult getAnalysis() throws Exception { | |||
|
|
|||
| // Retrieve information from job configuration like scheduler information and store them into result. | |||
| InfoExtractor.loadInfo(result, data); | |||
| if (result.queueName.toLowerCase().equals("ump_normal") || result.queueName.toLowerCase().equals("ump_hp")) { | |||
There was a problem hiding this comment.
Giving out internal queue configuration. At a minimum make this configurable. Better model the conditional execution as an interface? Applies elsewhere too
|
@varunsaxena As mentioned in the description , this is still work in progress . Please don't review it now , as there will be considerable changes in the PR |
pralabhkumar
changed the title
|
@varunsaxena Can u please review it |
|
@mkumar1984 Please review the same |
This PR is to add the code level optimization recommendation(as Heuristics) to Dr Elephant. This PR is specifically for Hive . For more details , please see the following design doc .
Design Doc
Testing
unit test cases are written and End to End testing is done on EI .