Need some help. Newbie in python language.

[ghost]

In your readme.md I've found this suggestion:
usage: smstools [-h] [--type {xml,json,android,csv,ios5,ios7,ios6}]
infiles [infiles ...] outfile
Well, executing this command line gives me an error on "{".
I'm sorry but I need some help. I'm a newbie in python.
Basically, I have two files: an Android .xml file and an iPhone .db file. I want to merge them in a unique iphone .db file. What is the correct command line I have to execute?
Thank you in advance for your kind attention and patience.

[t413]

The command you're looking for is smstools android.xml out.db --type ios6 (xml, json, etc are the options you can use for the optional --type argument). By default .db outputs to an android db so you'd have to specify.

BUT outputting to iOS databases isn't something I've built yet.. I don't have my jailbroken iPhone to test with– I loaned it to my cousin while she travels. I could try based on what I know about the database structure but you'd have to test it (by restoring a backup or by overwriting the file via ssh if your phone is jailbroken) to see if your messages show up as expected.

[ghost]

First of all, thank you for replying to my request.
Yesterday I've spent several hours to find the right command to execute and, indeed, I've found a command line similar to the one you've posted in your reply.
Yesterday I've tried with:

python.exe C:\Python27\Scripts\smstools android.xml ios6.db out.db --type ios6

assuming that android.xml and ios6.db are my original databases and out.db is the result of the merging process.
Unfortunately it gives me an out.db of 0 size with the following error:

645 messages read in 0.0940001010895 seconds from android.xml
Traceback (most recent call last):
File "C:\Python27\Scripts\smstools", line 38, in
texts.extend(smstools.readTextsFromFile(file))
File "C:\Python27\lib\site-packages\smstools\core.py", line 38, in readTextsFromFile
parser = getParser(file)
File "C:\Python27\lib\site-packages\smstools\core.py", line 52, in getParser
tableNames = core.getDbTableNames( file.name )
File "C:\Python27\lib\site-packages\smstools\core.py", line 76, in getDbTableNames
names = cur.execute("SELECT name FROM sqlite_master WHERE type='table'; ")
sqlite3.DatabaseError: file is encrypted or is not a database

I'm sure that ios6.db is not encrypted. I copied it directly from my jailbroken iphone. All permissions are granted.

Just now, I've tried with the command line you've posted in your reply:

python.exe C:\Python27\Scripts\smstools android.xml out.db --type ios6

assuming that (if I've understood the meaning of this command) smstools would merge the android.xml directly in out.db, where out.db is my original ios6.db file.
The result is an out.db with size 0 with the following error:

645 messages read in 0.109000205994 seconds from android.xml
sorting all 645 texts by date
Traceback (most recent call last):
File "C:\Python27\Scripts\smstools", line 43, in
outtype().write(texts, args.outfile)
File "C:\Python27\lib\site-packages\smstools\ios6.py", line 26, in write
raise Exception("not implemented!")
Exception: not implemented!

As you've stated, the process of outputting to iOS is probably not built yet.
So, I'll wait for further development of your useful tool. It's really what I need.
Thank you again for your reply.
Regards

[t413]

Alright, so I've updated the tool to give lots better error output. Try the first command again and see what the result is! They may have changed the database format in more recent Android versions or something and this will help figure out what changed.

I tested importing iOS 7 messages and it seems to work perfectly with the code I wrote for iOS 6.

[gelbrait]

Hi t413!
I'm also experiencing problems in merging an android with an iOS 6 database.
I'm using smstools v.0.1.2.
The command line is:

python.exe C:\Python27\Scripts\smstools C:\smstools\iossms.db C:\smstools\androidsms.xml C:\smstools\out.db --type ios6

as suggested above, where 'C:\smstools' is the folder in which I've stored my databases.
The output is:

Error reading from C:\smstools\iossms.db

Nothing more than this.
Would you please suggest me how to proceed?
Thanks in advance.

[gelbrait]

Alright!
I was having an issue with FTS3 module. I've googled "no such module: fts3" and found that updating sqlite3.dll in python\dlls would have solved the problem. Done!
Now no more

Error reading from C:\smstools\iossms.db

but

iOS output not yet implemented :/ (email me to help test!)!

I've tried to find out how to send you an email through github but with no luck.
So, if you agree, we can collaborate in this thread.
Please let me know if this is acceptable to you.
Thanks in advance.

[t413]

Sure, I'd be happy to! My email is [email protected] (it's goole apps) if you'd like to use gchat/hangouts or email there. Is your phone jailbroken?

[gelbrait]

Thanks for your email. OK, let's go!
Yes, my iphone is jailbroken.
Next step?
If you'd like to go on in private I might send you an email.

[t413]

I need to get an empty database from your phone and a db with 1 message in it. Here's how:

• Backup your sms.db file, delete the original. (It should be at /data/data/com.android.providers.telephony/databases/mmssms.db and I'd copy it via SSH)
• reboot and launch Messages (This will recreate that file)
• Now copy the new sms.db file form your phone and send it my way.
• Send a message (to anyone, it doesn't matter)
• Copy the sms.db file again and send that over too.

I'll use them to create the initial databases to output to. Having the single message version will help figure out what tables are updated when a new message arrives.

[gelbrait]

Ready!
In iphone sms.db is stored in

/private/var/mobile/Library/SMS/sms.db

Now I have an empty database and the one with 1 message in it.
I'm going to send you an email with an attachment "SMS.zip".
I've also included for each db the correspondent "sms.db-shm" and "sms.db-wal" files.
I think that these files will be necessary to visualize tables inside sms.db file. But I may be wrong.

[t413]

Hey everyone! I've spent an inordinate amount of time building up iOS export!

I pushed it to a new branch called testing. If you can help by testing it out please do!

• Get the new version!
  • Go to the commit, 3dc25a9, click download as zip
  • Or use git: git pull --all; git checkout -b testing origin/testing
• open a bash terminal in the downloaded directory
• uninstall your version of smstools in pip with pip uninstall smstools
• run python setup.py install to install this downloaded version
• try out the new smstools version by converting into ios6 (using smstools in.db out.db --type ios6)

Part of this effort was to support group MMS/iMessage chats. It's not implemented in Android import or export yet but it should preserve your iOS group conversations now.

[gelbrait]

Hi Tim!
Thanks for your extraordinary effort.
I'm sorry but I can't manage to download the new version of your tool.
Into the commit page there's no "download as zip" option, as far as I can see.
I've installed Git client for Windows but, using Git shell, the command you've provided gives me a fatal error.
Please give me further suggestion to download your tool.
Thanks again.

[gelbrait]

Alright Tim!
I've found out how to install the testing version of your tool. It was a bit difficult given my limited knowledge about GitHub.
However I've tested it on a iOS 6 database.
The command line is the one you've suggested but I'm getting this error:

Traceback (most recent call last):
File "C:\python27\scripts\smstools", line 6, in <module>
import smstools
File "C:\Python27\lib\site-packages\smstools\__init__.py", line 4, in <module>

from core import *
File "C:\Python27\lib\site-packages\smstools\core.py", line 3, in <module>
`import core, android, xmlmms, tabular, ios5, ios6, ios7, jsoner, googlevoice``

ImportError: No module named ios7

I think that maybe it could be an error in compiling.
Let's stay in touch.
Thanks again for your efforts.

[t413]

You beat me to it! Nice job!

Damn, I forgot to fix this. Patching it now.

-Tim

Tim O'Brien
t413.com
919.451.2187

On Wed, Aug 20, 2014 at 10:09 AM, gelbrait [email protected] wrote:

Alright Tim!
I've found out how to install the testing version of your tool. It was a
bit difficult given my limited knowledge about GitHub.
However I've tested it on a iOS 6 database.
The command line is the one you've suggested but a I'm getting this error:

Traceback (most recent call last):
File "C:\python27\scripts\smstools", line 6, in
import smstools
File "C:\Python27\lib\site-packages\smstools__init__.py", line 4, in

from core import *
File "C:\Python27\lib\site-packages\smstools\core.py", line 3, in
import core, android, xmlmms, tabular, ios5, ios6, ios7, jsoner,
googlevoice`

ImportError: No module named ios7

I think that maybe it could be an error in compiling.
Let's stay in touch.
Thanks again for your efforts.

—
Reply to this email directly or view it on GitHub
#9 (comment).

[t413]

Alrighty. Here's the fix on the testing branch (you can download the zip from that link too or run git pull --all)

It hopefully (crosses fingers) should let you export to iOS 6 (it's the same database as ios7) and work on the phone.

[gelbrait]

Sorry Tim, but the output is:

Traceback (most recent call last):
File "C:\python27\scripts\smstools", line 12, in <module>
parser.add_argument('-V', '--version', action='version', version="%(prog)s "
+smstools.__version__)
AttributeError: 'module' object has no attribute '__version__'

It's not a matter of life or death!
So, do everything quietly and take all the time necessary.
I remain at your disposal for testing when you'll be ready.
Keep up the good work. Thanks again.

[t413]

No life or death, certainly. But good practice. Anyway, that was a bug specific to running from git. I just pushed another patch to fix it, again to the testing branch (same url as my last comment)

[gelbrait]

OK Tim!
Out.db successfully created.
Testing in a few minutes...
...

[gelbrait]

Great!
It's almost perfect.
Just two or three small bugs.

• All messages are unread
• Your tool has correctly converted 160 messages (in the command prompt) but only 137 are present in the message application. I cannot find text messages from senders not providing a real number (such as corporations, banks, telcos, insurance, etc...)
• A conversation containing 8 sms is represented four times (same sender and same messages). I don't know why. But more significant is that the sender of the messages has no number. So maybe your tool is also able to identify "unknown" senders but has to be a bit improved.

But I must say: definitely a great job!
Let me know if you consider to keep on working on your tool in order to make it perfect. I'm always at your disposal here on GitHub and, whenever you want, by email.
Thanks again, Tim.

[t413]

Of course the goal is perfection!

I'm at a job interview now, but I'll get right to these issues tomorrow morning.

Thanks for the help!

[gelbrait]

Hi Tim!
I tested your last version of SMS-Tools.
The output is

install 'blessings' module for great terminal output color
165 messages read in 0 seconds from in.db using parser smstools.ios6.IOS6
latest text: 2014-08-21 08:00:16 AM Western Europe Local Time to "SENDER"
: "MESSAGE BODY" .
.
sorting all 165 texts by date
Creating empty iOS 6 SQLITE db
something failed at: smstools.core.Text({'date': 1388830359000L, 'body': u'MESSAGE BODY', 'num': u'SENDER', 'incoming': True, 'chatroom': None})
Traceback (most recent call last):
File "c:\python27\scripts\smstools", line 61, in <module>
outtype().write(texts, args.outfile)
File "C:\Python27\lib\site-packages\smstools\ios6.py", line 40, in write
self.write_cursor(texts, cursor)
File "C:\Python27\lib\site-packages\smstools\ios6.py", line 84, in write_cursor
handle_id = handles_lookup[clean_number]
KeyError: False

I edited this output only to hide the true "sender" and the "message body".
Thanks for your efforts.

[t413]

Cool. So you are getting a KeyError too. Not exactly sure why, and I'm getting them on some databases too. I'm writing an array of tests to use real databases to nail down what's causing all of the issues you've identified.

I put in the message/sender/date output there so you could check the date of the latest message. If the databases weren't sorting together correctly then it's because the date is off (probably in Android.db parsing). Could you compare the date it gives you with what's on the phones?

[gelbrait]

The date is on. The database I used for testing wasn't updated.
Message, sender and date of the latest message are correctly identified by your tool.
By the way, in the last test I've tried to convert an iOS database directly into another iOS db.
...