Batch Edit: Support for editing basic fields

[sharadsw]

Fixes #5413
Uses code from #4929

⚠️ Note: This PR affects database migrations. See migration testing instructions.

This PR pretty much has the same code as #4929 but updated to production. To minimize the scope for testing, batch edit for relationships is disabled in this PR and will be enabled in the follow up PR - batch edit for relationships. There are still some merge conflicts resulting from some newer code in prod which will be resolved in the next PR.

Checklist

• Self-review the PR after opening it to make sure the changes look good
  and self-explanatory (or properly documented)
• Add automated tests
• Add relevant issue to release milestone

Testing instructions

• Create a query
• Click on Batch Edit. This will create a new Batch Edit dataset with columns involving relationships disabled (readonly)
• Change values you wish to edit
• Click Validate
• Verify changed values are highlighted as updated cells
• Click Commit. This will apply your changes
• Re-run the original query and verify the values changed as expected
• Click rollback
• Verify values were rolled back to original

Regression tests

• Retest Support COT catalog number format configuration in WorkBench #5489

[sharadsw]

POTENTIAL IMPROVEMENTS?

• Data Mapper in Batch Edit?
  • Should the Data Mapper be accessible in Batch Edit datasets? Since Batch Edit datasets can only be created from a Query, it seems confusing that users can still access the Data Mapper from a Batch Edit dataset with most of the mapper functionalities being disabled. It seems the mapper is only needed for changing Batch Edit Preferences. Could move the preferences dialog to Editor page instead.

[realVinayak]

Nice observation. My fault for keeping a hidden feature secret: Another, use for data mapper is to control “ignore when blank”-type behavior per field basis. That’s actually supported in that PR ( even for batch edit data mapper, you can modify this per-field setting. Ask grant for more context )

the only things not read only are

1. Batch edit prefs
2. Field behavior (never ignore, …)
3. must match (also, could you leave a note somewhere to make trees must match by default?)

search for usages of “readonlyspec”.

For purposes of this PR, you can theoretically not add data mapper (you also don’t need batch edit prefs, they don’t do anything when there are no relationships)

[sharadsw]

@specify/ux-testing Thanks for testing! Summary of the changes:

• Batch Edit is now disabled instead of hidden for Trees. For CO queries involving Taxon, batch edit should be now enabled except when the query has a specific rank selected (it will still work when using any rank).
  • The missing rank error has duplicates because missing ranks are being calculated by checking ranks of all trees. Looks like ojsmnh110242024 has another tree with rank Genus with Species below it and so they get added to missing ranks. Will be fixed in Batch Edit: Batch edit for (multiple) trees #6127
• Fixed scoping error with Loan
• Created a separate issue for uploaded cell links: Batch Edit: Add link to updated cells in Results #6164
• Cannot reproduce the rollback error locally. Will continue looking into it while the PR is tested

I updated the testing instructions to retest #5489 since this PR refactors code written in that PR

[realVinayak]

Ah, for the rollback, I thought I wrote up a comment about it, looks like I forgot to publish it.

For elizabeth's recent rollback case, it failed because the row was a new row (that is also why it is colored blue). While the upload logic is generic, I forgot to account for this case (rollbacks in a batch-edit dataset for a new row). We currently shouldn't allow adding a new row in a batch-edit dataset. So, it is actually two bugs:

1. Frontend incorrectly allowing adding new rows (which are treated as workbench)
2. Backend crashing when a new row is found in the batch-edit during rollbacks.

So, it needs to be fixed on both sides for it:

1. Frontend
   

   specify7/specifyweb/frontend/js_src/lib/components/WorkBench/WbSpreadsheet.tsx

   Lines 131 to 136 in f5a319d

   	row_above: {
   	disabled: () => isReadOnly,
   	},
   	row_below: {
   	disabled: () => isReadOnly,
   	},

   
   Disable both of them when dataset.isupdate is true (so batch-edit cases). Elizabeth, did you use a different way to add new rows? This specifically disables "insert row above/below" when you right click.

2. Backend
   

   specify7/specifyweb/workbench/upload/upload.py

   Line 561 in f5a319d

   r, be = look_up_in_backer(row)

   
   Here, I'm pretty sure this place is returning None.
   It should be something like:

pack = look_up_in_backer(row)
if pack is None: continue # Skip new rows.
r, be = pack

Side note

I also noticed another bug (related to UX). On commit success, it should be readonly, but it is not. Like once you upload in workbench, the entire sheet becomes readonly. Same thing should happen here after commit. Currently it is not, because HOT is a bit pesky. If you add readonly property at cell level, it always takes that (and overrides "global" properties from the top).

Replace this:

specify7/specifyweb/frontend/js_src/lib/components/WorkBench/handsontable.ts

Line 46 in f5a319d

const nullRecordsResults = identifyNullRecords?.(physicalRow, physicalColumn, property) ?? {};

By

const nullRecordsResults = dataset.uploadresult?.success === true ? {} : (identifyNullRecords?.(physicalRow, physicalColumn, property) ?? {});

EDIT: A significantly more cleaner way (for the side note) will be

Replacing just this:

specify7/specifyweb/frontend/js_src/lib/components/WorkBench/handsontable.ts

Line 71 in f5a319d

if (dataset.isupdate !== true || mappings === undefined) return undefined;

By

if (dataset.isupdate !== true || mappings === undefined || dataset.uploadresult?.success === true) return undefined;

[realVinayak]

Nice so far!

Some more user issues that I remember:

1. Currently, there is no way to select specific rows and batch edit just them. This might be a requirement, maybe not. There is a work around to do so. The current implementation supports batch edit out of a record set. So, the user can:

• Make a query.
• Select the records, make record set out of it.
• Query out of the record set, and batch edit on those results.

Quite a lot of steps. Check if that's acceptable for now, I've no clue if there have been any new expectations regarding BE. I think a decent implementation for this will be to follow the implementation in #5300 (I actually left a similar comment on there).

2. Data sorting. Currently, if you have one table (so no relationships), sorting should always (of multiple columns) be respected (UX guys: test this). If you have relationships, and you also sort on some of the relationship's columns, it's essentially arbitrary (no guarantees regarding what user sees). From the code perspective, it is not arbitrary (see if you can figure those places out). I know that we don't support relationships, but this point actually applies regardless, since the same procedure is applied, just that we ignore relationships in upload plan. This might UX issue, a way to work around will be to sort in the workbench instead (in the spreadsheet).

3. Potential performance issues, as mentioned in a below comment in stored_queries/views.py. IIRC, I once tried batch editing andy's entire voucher collection (45k CO records with determination and cataloger) and it took 1.2 minutes to construct the dataset (I think that's too slow - I deferred profiling it for later, maybe need to that).

EDIT: All the points are really optional, but would definitely recommend testing the performance and seeing if usability needs to be improved.

[sharadsw]

@realVinayak Thank you! I have made notes of your points in different issues