SDAP-70 Link events with dataset collections #33
Conversation
|
Can one of the admins verify this patch? |
|
There is still some work to be done on the WebUI. However the data modeling should be good. I would like to demo this shortly so will keep this PR open such that I can receive and accommodate feedback. |
| @@ -235,7 +237,7 @@ public void startMetaIngest() { | |||
| LOG.info("Metadata has been ingested successfully."); | |||
| } | |||
|
|
|||
| public void startFullIngest() { | |||
| public void startFullIngest(MudrodEngine me) { | |||
| DiscoveryEngineAbstract wd = new WeblogDiscoveryEngine(props, es, spark); | |||
There was a problem hiding this comment.
EventIngester has a variable of ESDriver type since EventIngester extends MudrodAbstract class. I don't think you have to change function from startFullIngest() to startFullIngest(MudrodEngine me).
There was a problem hiding this comment.
@lewismc I tested your code on my local computer and it works. The event module can download EONET events and save them into Elasticsearch.
| @@ -247,6 +249,9 @@ public void startFullIngest() { | |||
| DiscoveryEngineAbstract recom = new RecommendEngine(props, es, spark); | |||
| recom.preprocess(); | |||
| recom.process(); | |||
|
|
|||
| EventIngester eonet = new EventIngester(props, es, spark); | |||
| eonet.ingestAllEonetEvents(me); | |||
There was a problem hiding this comment.
I suggest removing input parameters "me" since EventIngester already has a variable of ESDriver.
| private void startFullEonetIngest(MudrodEngine me) { | ||
| LOG.info("Starting full EONET ingest."); | ||
| EventIngester eonet = new EventIngester(props, es, spark); | ||
| eonet.ingestAllEonetEvents(me); |
There was a problem hiding this comment.
I suggest removing input parameters "me" since EventIngester already has a variable of MudrodEngine.
| @SuppressWarnings("unused") | ||
| GetResult result = null; | ||
| String index = mEngine.getConfig().getProperty(MudrodConstants.ES_INDEX_NAME); | ||
| String eventType = "eonet_event"; |
There was a problem hiding this comment.
Did you add eventType to the configuration file?
| } | ||
| } | ||
| esDriver.destroyBulkProcessor(); | ||
| return ""; |
There was a problem hiding this comment.
The function will return empty string whether the index operations success or fail and made it hard to write a unit test case for function ingestAllEonetEvents.
This issue is linked to https://issues.apache.org/jira/browse/SDAP-70