Add analysis functions for M&E results #13
Conversation
ipod/utils.py
Outdated
| if len(matching_members.orbit_id.unique()) == 1: | ||
| resulting_orbit_members_complete = members_me.where( | ||
| pc.equal(members_me.column("orbit_id"), matching_members.orbit_id[0]) | ||
| ) | ||
| me_results["old_orbit_id"].append(orb_id) | ||
| me_results["merged_orbit_id"].append(matching_members.orbit_id[0]) | ||
| me_results["expected_num_obs"].append(len(orbit_members)) | ||
| me_results["number_observations_matched"].append(len(matching_members)) | ||
| me_results["total_obs_resulting_linkage"].append( | ||
| len(resulting_orbit_members_complete) | ||
| ) | ||
| elif len(matching_members.orbit_id.unique()) > 1: | ||
| # case where an expected linkage was split between several orbits | ||
| for new_id in matching_members.orbit_id.unique(): | ||
| resulting_orbit_members_complete = members_me.where( | ||
| pc.equal(members_me.column("orbit_id"), new_id) | ||
| ) | ||
| me_results["old_orbit_id"].append(orb_id) | ||
| me_results["merged_orbit_id"].append(new_id) | ||
| me_results["expected_num_obs"].append(len(orbit_members)) | ||
| me_results["number_observations_matched"].append( | ||
| len( | ||
| matching_members.where( | ||
| pc.equal(matching_members.column("orbit_id"), new_id) | ||
| ) | ||
| ) | ||
| ) | ||
| me_results["total_obs_resulting_linkage"].append( | ||
| len(resulting_orbit_members_complete) | ||
| ) |
There was a problem hiding this comment.
I think we don't need the conditional and can just use the logic where we have more than one unique orbit id. We will also need to control for when 0 orbit members match the expected orbit id.
ntellis
force-pushed
the
nt/me_analysis
branch
from
ab65f1f to
458b6db
Compare
|
I had mistaken the format of members_expected - had not realized Joachim had provided attributions within those data. I completely rewrote what I had in accordance with that. Now the output is structured around the primary designations, and how well the resulting M&E output mirrors the expected members for those primary designations. I adopted a sort of "best fit", counting primarily the orbit coming out of M&E that had the least missing observations, as well as noting for each designation if there were multiple, unmerged orbits with observations with that designation. The output I think is now actually useful, and allows us to rank how well a particular run has done. Attached is a sample csv with the breakdown for one M&E run. Ex derived stats from the attached csv: Num designations with a complete attribution: 88 |
| data_expected = { | ||
| "orbit_id": ["orbit1", "orbit1", "orbit2", "orbit2"], | ||
| "obs_id": ["obs1", "obs2", "obs3", "obs4"], | ||
| "primary_designation": [ | ||
| "designation1", | ||
| "designation1", | ||
| "designation2", | ||
| "designation2", | ||
| ], | ||
| } |
There was a problem hiding this comment.
How are orbits with overlapping observations handled?
There was a problem hiding this comment.
I would like to see a higher level function which takes each of these analysis_dict results and collapses it to a row that counts the following:
Given a short arc of a known object, do we extend it to all the observations in NSC?
Given two non-overlapping sub-arcs, do we merge those?
Do we merge two overlapping sub-arcs?
It's not clear to me if we have the full information here to determine the above.
Then each summary of the analyzed results becomes a row in a table that looks like
[ p1 | p2 | ... | short_arc_extended | non_overlap_subarcs_merged | overlap_subarcs_merged]
Where p_x are the parameter values, and the other columns are percentages of completion (e.g. 4/5 short arc extensions were completed).
Then we can quickly see at a glance which parameter set(s) are the most optimized.
Adds a pair of functions for evaluating the results of M&E wrt expected values and M&E inputs.
Currently
identify_merged_dropped_orbitsdoes not compare to expected merged orbits - would like to see the tracksubs to get an idea of how that should be structured.analyze_me_outputcollects some pretty basic metrics at the moment right now - essentially it tabulates the number of observations that were expected in a particular linkage, how many of them landed in which orbit after M&E, and the total observations in the resulting orbits. The implication there is that any "extra" obs are bogus, if we are operating with the expected linkages as the truth state.Currently there's no notion of arc lengths, maybe that would be useful for comparing different runs?
The
members_expectedis currently aFittedOrbitMembers, but could just as easily be anAssociationsWants tests/incorporation into existing tests (since the purpose is to analyze ipod results, after all)