SMSs are not imported in chronological order

[jayb-g]

This was issue 802 of Simple Messages

Please refer to it for further details and discussions.

Checklist

• I can reproduce the bug with the latest version.
• I made sure that there are no existing issues - open or closed - to which I could contribute my information to.
• I have read the FAQ and my problem isn't listed.
• I have taken the time to fill in all the required details. I understand that the bug report will be dismissed otherwise.
• This issue contains only one bug.
• I have read and understood the contribution guidelines.

Describe the bug
When running it the first time after installation(and after making it default SMS app), it automatically starts loading the messages but all the messages show the same timestamp of when it was imported(and not that of when the message was sent or received). Although when opened, it shows correct time.

So messages order in my stock SMS app and in Simple SMS are completely different.

I also tried to manually export messages from the default SMS app so that I can import it in simple SMS but that also fails since probably the format(keys) don't match. Can you refer to some app that exports SMS in a compatible format?
I tried using SMS Import Export

To Reproduce
Steps to reproduce the behavior:
Detailed steps and behaviour explained in the original issue.

Expected behavior
All messages should have been imported in correct chronological order. Doesn't happen all the time. Maybe an external tool/script would help convert the exported SMSs to a compatible format importable by Fossify Messages.

Screenshots
If applicable, add screenshots to help explain your problem.

Device info (please complete the following information):

• Model: [Pixel 7]
• OS: [Android 14]
• App version [1.0.0]

Additional context
Issue explained in detail in the original issue on simple messages. Also, issue 783 and issue 499 may be related to this issue.

Issue 499 from simple messages:
I have exported SMS messages from Signal and then switched to Simple SMS Messenger. However, I have random order in the thread list, due to sorting by exported time, not sent/received time.

What's interesting: stock (AOSP, outdated) messages app shows everything correctly, in correct order, with correct date/time.

Tried reexporting, clearing app cache and data (to trigger app's database initialization) and unfortunately nothing helped.

[Aga-C]

This was issue 802 of Simple Messages
Please refer to it for further details and discussions.

Simple SMS Messenger repository can be deleted any time, so please copy these details here.

[jayb-g]

@Aga-C done.

It happens again and again. I have tried wiping all the messages and deleting cache for simple SMS app and then reimport messages to default app then again installing the simple SMS app but result is same.

Upon wiping and reimporting to default app, it shows correct order and timestamps. So when simple SMS imports messages from default SMS app, it should load in correct order with all timestamps but that doesn't happen.

So what I want to try is export SMSs to json first, then make it compatible with the schema supported by simple SMS app and then try importing from

[jayb-g]

Anyone can point out to the schema used by simple SMS import/export?

So it would be easier to create something that can be used by the community (temporarily until issue is looked into and fixed by devs,)which exports/converts exported jsons to the schema supported and importable by simple SMS.

[Aga-C]

Anyone can point out to the schema used by simple SMS import/export?

Here are Kotlin files with classes used during export to JSON:

• https://github.com/FossifyOrg/Messages/blob/master/app/src/main/kotlin/org/fossify/messages/models/ExportedMessage.kt
• https://github.com/FossifyOrg/Messages/blob/master/app/src/main/kotlin/org/fossify/messages/models/MmsBackup.kt
• https://github.com/FossifyOrg/Messages/blob/master/app/src/main/kotlin/org/fossify/messages/models/SmsBackup.kt

They are serialized with Gson, so schema is the same as these types.

[jayb-g]

@Aga-C Thanks, I'll try to use this for a conversion script meanwhile its fixed in the app.

[ikanakova]

Note for those switching from SMT: Export and Import between Simply Messages and Fossify Messages works without this issue.

[chesio]

Note for those switching from SMT: Export and Import between Simply Messages and Fossify Messages works without this issue.

If you don't switch to another phone as well, then there's no need to run export and import. Just install Fossify Messages, set it as default SMS app and remove Simply SMS Messagenger. All messages will be available in Fossify Messages as they are stored in phone SMS storage anyway.

[sudwhiwdh]

Unfortunately, the last version of Simple SMS Messenger only produced binary files for me when exporting my messages for an operating system migration.
That's why I used https://f-droid.org/packages/com.github.tmo1.sms_ie/ and now have the data as a .zip file.
Due to this issue, I have just ended up back with the standard SMS messaging app from Android, because the error does not occur there when re-importing.
I would be very happy if I could use Fossify Messages in the future.
How can this issue be solved technically?

[jayb-g]

@Aga-C Thanks, I'll try to use this for a conversion script meanwhile its fixed in the app.

@sudwhiwdh this is the only temporary solution until its fixed by devs. A script that converts the exported SMSs(by SMS Import Export) to a json format compatible with Fossify Messages Import.

[sudwhiwdh]

Support for another import format would be a small additional concern for me.

My main concern for this issue is a different one. Namely, that when I import messages through Fossify Messages and have selected this app as the new default application for messages, the messages should be sorted by date and time received/sent and not by import time.

So far, I assume that the messages land in the system memory after an import, for example via SMS import / export.
The standard Messages app can correctly read this information, whereas Fossify Messages cannot yet do this.

And this is what my question refers to. What does it take to make this possible in Fossify Messages as well?

[jayb-g]

@sudwhiwdh Probably some bug in the way fossify messages (mis)reads the messages db.

Hopefully someone who can go through the code of SMS Import Export and fossify messages can compare and pinpoint the difference.

[jayb-g]

I opened the ndjson and saw that SMSs that have this problem of incorrect date(ie date of import) are all first exported, then wiped, then restored using sms-ie; all have entries sub_id: -1 and creator: com.github....

Whereas other messages that appear to show correct date time, all have sub_id: 1 and creator: com.android.messaging

These are the only differences in metadata.

I think it probably has to do something with sub_id: -1 not being addressed in the right way in fossify messages, although its fine in default stock SMS app.

Will try changing all sub_id to 1 and see if it works.

[jayb-g]

Worth mentioning, even though timestamps are shown to be of the time imported in the list view; when you perform a search and list is filtered it shows message times correctly! Maybe this will give devs a hint what is wrong

[ghost]

As you can check in my open issue (not sure if it is similar or different fundamentally than yours), @tom93 has fixed the timestamp issue for importing backups for SMS (in PR). @Aga-C correct me if I am wrong.

[Aga-C]

@Abcd1234-dot Fix from e006a91 could fix it (I haven't tested it), but I don't see it in any open PR.

[tom93]

WARNING: If you are going to build from source, don't try to use commit e006a91 in isolation -- it would cause havoc unless you also include #149.

[tom93]

To add some details, the issue we came across was that, after importing messages from a SMT/Fossify .json backup, the conversation dates shown in the main view were the date of the import rather than the date of the last message in the conversation (details). It appears to be the same as this issue.

The root cause is that when a message is inserted, Android sets the conversation date to the current time rather than the time of the message (source). I found a workaround, and I'm planning to open a PR but I want to do some more work first (make it faster and document it).

Note that the way I intend to implement the workaround will only apply to newly-imported messages. If you already imported a .json backup (using the current release), and you try to re-import the same .json backup using a future release with my planned workaround, then it won't update the conversation dates because existing messages are skipped during import. So if you want to fix the dates on your existing conversations, you can use my build from #88 (comment) and re-import the .json file (but note that I'm not a project member so you shouldn't trust me). I can also make a build that will update all conversations (without having to import anything), let me know if that would be useful.

Finally, note that Android has another bug that breaks my workaround on conversations containing only MMS (details).

[jayb-g]

The root cause is that when a message is inserted, Android sets the conversation date to the current time rather than the time of the message.

@tom93
Even though the default AOSP shows conversation timestamps correctly after import. So I highly doubt that's the case.

I can also make a build that will update all conversations (without having to import anything), let me know if that would be useful.

That would be super useful. Or even better if devs can just include it to fix existing conversations in the next release.

[tom93]

Even though the default AOSP shows conversation timestamps correctly after import. So I highly doubt that's the case.

I believe the AOSP Messages app calculates the conversation date & snippet itself instead of using the values reported by the Telephony provider. Here is the code in the Telephony provider that sets the date to the current time: https://android.googlesource.com/platform/packages/providers/TelephonyProvider/+/android14-release/src/com/android/providers/telephony/MmsSmsDatabaseHelper.java#134

Or even better if devs can just include it to fix existing conversations in the next release.

My thinking was that updating all conversations can be slower than updating just the conversations affected by the import, so ideally the import should only update the affected conversations. But we'll see what the devs think. One option is to update all conversations as a one-off using a migration, for the benefit of users who imported using an older version of the app.

[jayb-g]

I believe the AOSP Messages app calculates the conversation date & snippet itself instead of using the values reported by the Telephony provider.

@tom93 Agree, then why not fossify messages can do the same? AOSP messages is also open source.

One option is to update all conversations as a one-off using a migration, for the benefit of users who imported using an older version of the app.

Yes. That would be good

[tom93]

then why not fossify messages can do the same?

Calculating the conversation date & snippet is more work (and will probably be less efficient). It may be worth it though, considering the Android bug with MMS-only conversations (also, Android's snippet for MMS is bad and could be improved). We'll have to see what the devs think.

[jayb-g]

@tom93 at least now there's some hope that the issue might be resolved.

[tom93]

Just to be clear, the main issue (incorrect dates on import) can be fixed using the simple workaround that I'm proposing.
Calculating the conversation date & snippet on the app's side is only necessary to work around a second Android bug that only affects MMS-only conversations (and to optionally to improve the snippets for MMS).

[jayb-g]

the main issue (incorrect dates on import) can be fixed using the simple workaround that I'm proposing.

My issue was mainly related to SMSs import only. I'm not sure if its interconnected with the MMS related bug or not. Or if it has to be fixed separately. When I encountered the bug I probably didn't have any MMS in my data. But not 100% sure.

Can you please create a PR for the fix you are proposing for import-timestamp issue? If devs agree then maybe we all get relief.

[jayb-g]

Awaiting the fix to be merged/implemented in the main app

[tom93]

@jayb-g I haven't opened a PR yet (still haven't done the cleanup); also blocked on #149 which needs to be merged first.

[jayb-g]

@tom93

I can confirm that this works as expected. Awesome work.

So I guess this is the only option until it gets committed

[jayb-g]

@tom93 any chance the fix might be merged into the final release?

[danielmmmm]

Yeah, a fix for this would be great. I just switched to another phone and was puzzled that importing old SMS apparently didn't import all messages. It took a while for me to figure out that my messages are now sorted after the very first message in any conversation. Everything is still here, but in the wrong order. Additionally, the preview text also shows the first message of a conversation, instead of the last one.

Is there an option or filter to sort messages manually?

[jayb-g]

The code is in branch "build/timestamp-fixer" of my fork (commit), and here is the APK:
unofficial-messages-timestamp-fixer.apk.zip
(again, note that I'm not a project member so you shouldn't trust me; consider building from source yourself with that patch).

@danielmmmm you can use this for the time being

[danielmmmm]

Great! 🙂
But, I guess we have to export and re-import our messages after updating to v1.1.0? (I can't test this because I have to wait for the f-droid build)

[tom93]

@danielmmmm: You will have to do some manual steps.
Due to the way I wrote the patch, it will only fix the dates of conversations in which new messages were imported (if all the messages already exist, it won't do anything). So you have a few options:
A. Export the messages, manually delete 1 message from each conversation*, then import the backup. (Only delete messages after you upgrade to 1.1.0, due to #148.)
B. If you trust the export & import process, you can do as in A but delete all the messages (it can be done quickly).
(C. Use the fixer APK from earlier.)

^{*In fact the act of deleting a message should be enough to fix the conversation date. The import in option A is just to restore the deleted messages.}

[jayb-g]

B. If you trust the export & import process, you can do as in A but delete all the messages

You mean export all messages, then delete all the messages, and import them back, right?

[tom93]

Yes.

[jayb-g]

@tom93 is there any standalone script that can convert SMSs export in json created by some other app to the format compatible with fossify messages?

I have another non-android device that has a lot of old SMSs that I want to import to fossify messages.

If you can share importable schema, I can manage to create a script to convert it

[tom93]

I'm not aware of any converters. There is no formal schema for Fossify, just the source code mentioned earlier (#42 (comment)). I think that plus a sample Fossify export is enough to write a converter.

[mpwhiten]

@tom93 I tried using your patched APK but wasn't able to launch it as the system denies access to be set as default and I can't figure out how to override this. The app is greyed out in the default SMS app selection screen. This is on a Pixel 8 running GrapheneOS. Not sure what's causing it, but thought I'd make you aware just in case. Screenshots below.

Click to expand

[tom93]

Hi @mpwhiten, thanks for raising this. It looks like a security feature was added in Android 15 that prevents sideloaded apps from requesting those permissions. It should be possible to bypass this by pressing "⫶" then "Allow restricted settings" (see the screenshots at the end of this article).

[mpwhiten]

@tom93, thanks, that did the trick!

[jayb-g]

C. Use the fixer APK from earlier.)

I think the issue should stay open until the patch is included in a release.

[Aga-C]

I think the issue should stay open until the patch is included in a release.

It is released. Doesn't it work for you?

[tom93]

To clarify, the patch from #265 (fix dates on new imports) was released in 1.1.0.
The patch to fix dates from old imports ("fixer APK" #42 (comment)) was not intended to be merged.

[tsjost]

The patch to fix dates from old imports ("fixer APK" #42 (comment)) was not intended to be merged.

This fixer patch seems to be the only way to fix new external imports too, thanks!

If I've understood correctly, SMSes on Android are stored in some standard system location, so I'm confused why Fossify is the only app (of the 5 or so I've tried) that uses the import-time as the latest-message-time in the conversation list, rather than the time of the latest message.

I'm using tmo1/sms-ie to import 1.2 GB of messages, so doing the export and reimport from within Fossify unfortunately doesn't work as Fossify just throws an OutOfMemoryError during export (on latest 1.1.4(7)).

I guess these merged PRs are only about fixing the internal importer, so it would be nice to make external importers work as well!
If it's of any use, here's an sms-ie export of a message received by Fossify. Name messages.ndjson, wrap it in a zip in order to import in sms-ie, and fossify will display the import-time instead of the timestamp: (number replaced with a number from a not-in-production test-only sequence)

{"_id":"1","thread_id":"488","address":"+46701740635","date":"1738356928518","date_sent":"0","read":"0","status":"-1","type":"1","subject":"","body":"hello world fossify","locked":"0","sub_id":"1","error_code":"-1","creator":"org.fossify.messages","seen":"0","__display_name":"test"}

[tom93]

SMSes on Android are stored in some standard system location, so I'm confused why Fossify is the only app (of the 5 or so I've tried) that uses the import-time as the latest-message-time in the conversation list

Android's Telephony provider has an API for getting the date and snippet of conversations. Fossify uses the values returned by this API, while it seems that most other SMS apps don't use these values and instead compute their own values based on the latest message in the conversation.

There is a bug in Android's Telephony provider that causes the conversation date to be set to the time of import (see the message of commit 3ad36c0). So apps like Fossify that use the value reported by Android are affected.

(Also, the documentation is incorrect -- it says the DATE column is "The date at which the thread was created.", but the Android code clearly intends to set it to the date of the latest message.)

external imports

Hmm, that's an interesting point (handling messages imported by other apps). I see several possible ways that this could be fixed:

A. Ideally, Google should fix the bug in Android (the linked commit message and my previous paragraph about the incorrect documentation should be enough information; maybe they'll also fix this other bug).

B. We could ask other apps (such as sms-ie) to implement the same workaround as us for the bug in Android (they just need to call contentResolver.delete(Threads.CONTENT_URI, "1 = 0", null), see our full code and documentation).

C. Fossify could try to detect when messages were imported by another app, and run the workaround.

D. Fossify could stop using the Android API for the conversation date and snippet, and compute those values manually. This would also work around this other bug, but it's more work and it's a shame for every app to reimplement this logic to work around Android's bugs.

(Note that if A or B are done, it might also require some changes on Fossify's side, because Fossify has an internal cache of the conversation information and I'm not sure whether or not this internal cache would be automatically invalidated.) Edit: Not an issue, see naveensingh's reply

reimport from within Fossify unfortunately doesn't work as Fossify just throws an OutOfMemoryError during export (on latest 1.1.4(7))

Yeah, #139 reduced the memory usage but it still requires the entire backup to fit in memory. The approach described in the "Future work" section of that PR would fix the issue completely but it hasn't been implemented (yet).

[naveensingh]

Android code clearly intends to set it to the date of the latest message.)

It's the same with drafts, even empty drafts will "reset" a conversation's date.

A is the ideal solution but even if it's reported to Google (which I plan on doing), there's a good chance that it will be dismissed as wontfix (working as intended) after a few years. There is less incentive to improve/fix the official APIs now that Google Messages is pretty much the default messaging app for Android.

Note that if A or B are done, it might also require some changes on Fossify's side, because Fossify has an internal cache of the conversation information and I'm not sure whether or not this internal cache would be automatically invalidated.

Cached conversations are updated on startup but I think D is the way. We already handle drafts and scheduled messages internally, now I'm just waiting for one more nail.

[danielmmmm]

Here is another vote for option D.

Even if Android would fix the API, there are a lot of phones that won't see system updates any more. Alternatives like LineageOS are also not an option for many phones/users.

I managed to get my messages sorted correctly by deleting all and re-importing them. But, trying to import SMS and MMS together would crash the Messages app eventually. My backup file is only 15 MB big. Importing SMS and MMS separately worked, but the process took approximately 2 hours! I am not complaining, as this is not something one does every day. A warning message in the import dialogue would have been nice though. The import animation seems to be wonky too, and randomly stop working. When that happens, it is unclear for the user whether Messages is still working on the import or not.

[naveensingh]

But, trying to import SMS and MMS together would crash the Messages app eventually.

Assuming you are referring to the OOM error mentioned before (see Tom's message above), it's an in-app issue that will be fixed. If it's some other error, please create a bug report for it.

The import animation seems to be wonky too, and randomly stop working.

Not sure what you mean, can you show a recording or describe the problem in more detail?

(Ideally it's better to use a notification for such tasks (see this) but I figured an indeterminate progress bar was better than no indication at all. All such import/export functionalities across all apps should be reworked to use a background service with a notification).

[danielmmmm]

Yes, a background service with a notification seems like a good idea.

Afair, the progress bar was still visible, but solid grey and without an animation for the time I observed it (some seconds, maybe up to 1-2 minutes). Since the import took so long, I switched off the screen and checked every 10-15 minutes repeatedly. Sometimes the progress bar was animated, sometimes it wasn't.
Is the animation tied to actual progress? Like, would anything temporarily blocking or slowing down the import process therefore also stop the animation? Or can the import process use so much of a phone's resources (RAM/CPU) that the animation can't be shown (I doubt that's the case)?

[naveensingh]

Is the animation tied to actual progress?

Not yet but it should be (see indeterminate).

Like, would anything temporarily blocking or slowing down the import process therefore also stop the animation?

No, that shouldn't happen. Progress bars usually just work™, this seems like an upstream issue:

• [Progress indicator] Doesn't spin on API level 21 material-components/material-components-android#4179
• [Linear Progress Indicator] Indicator colour is gone when screen/text is zoom  material-components/material-components-android#3856
• [Progress Bar] Indeterminate ProgressBar gets Frozen material-components/material-components-android#2355).