[Feature Request] Disable Undo Point Option

[gentlegiantJGC]

The Problem

The undo point operation takes quite a while to run.
Many users have expressed wanting to be able to disable this.
The undo point was added in as a mechanism to allow users to undo changes but also as a mechanism of freeing RAM.

Chunk Storage

About

When a chunk is loaded, Amulet needs to hold a reference to it so that if it is requested again the same chunk object can be returned. From now on we will refer to this as the RAM database.

There needs to be some condition when chunks can be unloaded from the RAM database to free memory, otherwise memory usage will grow over time.
Chunks cannot be unloaded if they have changed otherwise those changes would be lost.

Currently this condition is a manual function that unloads all chunks.
It would be better if chunks were automatically unloaded when they are no longer referenced (and are not changed)

The history database was a mechanism to allow the chunk to be persisted to disk for later loading and unloaded from the RAM database to free up RAM.
When loading a chunk the history database is first checked and if it contains the chunk it is loaded. If not it is loaded from the raw level.

One hurdle to removing the undo point is the behaviour when a chunk is changed but not marked as changed. Currently these changes are discarded but without a backup to revert to those changes will remain.
Should this be the correct behaviour or should a solution be found? Perhaps caching the previous state in RAM.

Changing the undo point behaviour is possible but it is currently a rather core part of how Amulet operates so it can't currently just be disabled.

To remove the undo point we would need to disable the disk caching system and all changed chunks would have to exist purely in RAM. We would also need to make sure that the chunks in RAM were not cleared.

History Database on disk.

Having a history database on disk allows RAM to be freed so is the best solution for most users with limited RAM.

History Database in RAM.

This is the fastest way to have a history system but requires a lot of RAM

No History Database.

This is the fastest solution but requires a bit of RAM.
Changed chunks cannot be unloaded from the RAM database.
Will have to manage the case where an operation changes a chunk but does not mark it as changed.

Prerequisites

#256 Chunk management improvement
#257 Chunk ownership

• Add a system to create the undo point during the operation rather than at the end. This would allow unloading changed chunks.

If I have more thoughts on this I will modify this.

[liquidwater2]

Yes please let us do this!!! I’m trying to move a 47 billion block structure segment by segment and the create undo point takes triple as long as each paste operation. Although I’ve not fully wrapped my head around the ram usage portion of this post I support the concept.

[Podshot]

A possible approach we could consider using would be using a local (most likely non-relational) database backend and use that to store the chunk history. The benefits would be that all of the logic of when to off-load old data to disk would be handled by better optimized code for the data we're storing as well as taking advantage that our main threads wouldn't get locked up while the database might be doing it's logic (async functions could benefit here). However, the downsides is that as far as my research has gone into this, it seems most database backends (Redis and MongoDB are the ones I've looked into the most for this) require a separate server process (IE: it can be a local server but can't initiated by python (except for running a command-line command) and thus adds something we need to install/bundle that may conflict for a variety of reasons). I'll keep putting some thought into this as well

[MKBandzz]

but for those of us with lots of ram (i have 32gb) i dont want chunks to prematurely delete themselves from memory, i want to be able to use all the ram i paid for so if this feature were to be considered please also consider those with higher specs.

[gentlegiantJGC]

That consideration is taken into account in the History Database in RAM. section

[PREMIEREHELL]

I really think that all that sould be saved is the changes that the user makes, like a ready to apply buffer, that does not apply anything until user saves, upon that user gets asked if he wants an undo or to save a backup of the edited chunks so they can reopen amulet and revert back if it did not go as planned. I think this is a better option because undo does not help after you find a mistake in game. You will not needs to store chunk history only what needs to take place to cause the changes. This could be kept in a temp directory pickled tasks , each with its own file where undo would simply need to delete the newest file. A memory var with a thresh hold of like 30 or(settings option ) something before saving to disk, major operation will need disk usage. You could get the list of chunks to pull from this if the user wishes, if amulet reloads the chunk it checks this area and re applys visual changes.

[gentlegiantJGC]

I think it would make more sense to merge the disk and ram undo options into one and have a configurable amount of chunks/memory until pushing the chunks to deep storage on disk.

[Buzzb0mber]

I'm in full support of having the option to disable the undo point entirely. It easily more than doubles the amount of time it takes to commit a change to a map for which I've already made several backup copies, the function has it's uses but for large-scale projects it can be crippling to the point of being an active detriment.

[PREMIEREHELL]

The buffer world plugin , Is an example that does improve speed, But I do understand, The necessity for an undo point, Hopefully this can be embedded so it's not so awkward.

[PREMIEREHELL]

I'm also working on plugins to bypass The undo point, By simply directly editing the world files. Yet any mistakes are on you. And you need to make sure you have a backup in case they're on me.

[gentlegiantJGC]

I have been thinking this over in relation to a couple of other issues to find the best solution.
The changes I want to make are rather substantial so would be part of a 2.0 release. Amulet V0 will continue using Core V1 and the Qt rewrite (Amulet V1) will use the new version of the library.

The changes that I want to make mean that there won't be a distinct undo creation stage.

[gentlegiantJGC]

I have thought about this some more and I think I have a solution.
I am going to use chunks as an example but the same system will be used to store all data.

The current version of the core library stores the actual chunk objects in a dictionary.
The new version will store the pickled bytes representation of the chunk and unpickle it each time get_chunk is called. The overhead is negligible.

There will be a cache consisting of a Least Recently Used cache in RAM with a user configurable size. When that gets filled up the least recently used entries will get pushed to disk.
Users with lots of RAM can increase the size of the RAM cache for optimal speed.

When loading a chunk for the first time it will get serialised and saved to the cache. Loading from the cache is faster than loading it from the level again because it skips the translation stage.

When setting/deleting a chunk, if the history system is enabled and the original chunk state does not exist in the cache it will load the chunk from the level and store it in the cache.

Disabling the history system will reduce the amount of data generated because the cache does not have to store multiple revisions. It will also mean that the original state will not need to be loaded if it was not already loaded.

I still need to think more on how the history system will work if multiple threads are running distinct operations.
My current thought is that when an operation is finished, it will increment the undo value and any changes from then on become a new undo point. If thread A and B make changes that overlap, each will create an undo point when it is finished and the undo would revert changes made during that time period.

|-------Undo 1-------|----Undo 2----|
|AAAAAAAAAAAAAAAAAAAA|              |
|         BBBBBBBBBBB|BBBBBBBBBBBBBB|

[Rsge]

@liquidwater2 Triple? You're lucky. I am trying to replace all instances of two of a mod's stone variants with Vanilla stone in the whole Overworld (long story):
The "actual" operation takes ca. 10 mins, the Create Undo Point operation takes over six hours. I had to let my PC run over night, because I didn't know if it would continue work after being put to Sleep.
So yeah, I'd greatly second this for future use cases like mine.