Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handling (escaped) slashes in folder name #208

Open
mmauger opened this issue Sep 28, 2024 · 4 comments
Open

Handling (escaped) slashes in folder name #208

mmauger opened this issue Sep 28, 2024 · 4 comments

Comments

@mmauger
Copy link

mmauger commented Sep 28, 2024

I'm sync'ing my Protonmail email thru the Protonmail Bridge which provides encrypted connections back to my Proton Mail account. Connectivity is successfully established and emails are downloaded properly.

However I have several folders in Proton that contain the slash character (/) which Proton permits and transmits with the folder name with slashes escaped by a backslash (\) as I'd expect. But offlineimap is not handling the escaped slashes explicitly so I end up with nested folders with parent folders ending in a backslash. That is, the folder "FSF / GNU / EFF" gets sent as "FSF \/ GNU \/ EFF" and gets created as nested directories in the Maildir tree: (EFF folder within an GNU \ folder, within an FSF \ folder). I tried to combat this somewhat with a nametrans rule to transform the \/ sequence in the folder name with a Unicode diamond character (♢). This fixes the Maildir folder creation but fails later when it sends the Maildir folder name back to the IMAP server rather than the IMAP name.

So, I think there are two issues here, but will leave it to others to properly debug and identify the source:

  1. The escaped slashes in the IMAP folder name are not treated as characters in the name, but rather as folder separator characters
  2. The IMAP and Maildir folder names appear not to be kept distinct which leads to some odd behavior.

And as a bonus issue, which I do not believe is yours, but in imaplib2. The "SELECT command error"
appears to be a SQL injection issue in that the escaped string is being interpreted by the command processor, rather than just being passed as data. I'll defer to others' judgement if this is what is happening and whether an issue needs to be raised with that project.

Please feel free to reach out if you want additional debug information or if you'd like me to try another solution. (I'm a senior Emacs, SQL, Python dev, so I can provide some assistance; I did try to dig in a little, but the learning curve was significant with other responsibilities...)

General informations

  • system/distribution (with version): Fedora 40 GNU/Linux
  • offlineimap version (offlineimap -V): offlineimap v8.0.0, imaplib2 v3.06, Python v3.12.6, OpenSSL 3.2.2 4 Jun 2024
  • Python version: Python 3.12.6
  • server name or domain: protonmail.com with Protonmail Bridge
  • CLI options: none

Configuration file offlineimaprc

Validated with --info and --dry-run options

[general]
metadata = ~/.config/offlineimap
accounts = protonmail

[Account protonmail]
localrepository = myLocalProtonmail
remoterepository = myProtonmail

[Repository myLocalProtonmail]
type = Maildir
localfolders = ~/.mail/pm
sep = /

[Repository myProtonmail]
type = IMAP
remotehost = 127.0.0.1
remoteport = 1144
remoteuser = ....
remotepasseval = ....
sslcacertfile = ~/.protonmail-bridge.pem  # fetched from Protonmail Bridge
ssl = no
starttls = yes
nametrans = lambda foldername: re.sub(r'\\/', r'♢', foldername)
folderfilter = lambda foldername: foldername not in ['All Mail', 'Archive', 'Drafts', 'Spam', 'Starred']

Logs, error

2024-09-27 18:32:24 INFO: Syncing Folders/FSF \/ GNU \/ EFF: IMAP -> Maildir
2024-09-27 18:32:24 INFO: Copy message UID 413 (413/496) myProtonmail:Folders/Energy -> myLocalProtonmail:Folders/Energy
2024-09-27 18:32:24 ERROR: ERROR: ERROR in syncfolder for protonmail folder Folders/FSF ♢ GNU ♢ EFF: Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/offlineimap/accounts.py", line 652, in syncfolder
    check_uid_validity()
  File "/usr/lib/python3.12/site-packages/offlineimap/accounts.py", line 520, in check_uid_validity
    remotefolder.save_uidvalidity()
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/Base.py", line 300, in save_uidvalidity
    newval = self.get_uidvalidity()
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/IMAP.py", line 132, in get_uidvalidity
    self.__selectro(imapobj)
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/IMAP.py", line 80, in __selectro
    imapobj.select(self.getfullIMAPname(), force=force)
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplibutil.py", line 57, in select
    result = super(UsefulIMAPMixIn, self).select(mailbox, readonly)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplib2.py", line 1082, in select
    self._deliver_exc(self.error, '%s command error: %s %s. Data: %.100s' % (name, typ, dat, mailbox), kw)
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplib2.py", line 1496, in _deliver_exc
    raise exc(dat)
offlineimap.imaplib2.IMAP4.error: SELECT command error: BAD [b'[Error offset=31]: Expected \'\\\' or \'"\' after \'\\\' in quoted']. Data: "Folders/FSF \/ GNU \/ EFF"

  SELECT command error: BAD [b'[Error offset=31]: Expected \'\\\' or \'"\' after \'\\\' in quoted']. Data: "Folders/FSF \/ GNU \/ EFF"

...

2024-09-27 19:17:51 WARNING: ERROR: Exceptions occurred during the run!
2024-09-27 19:17:51 WARNING: ERROR: ERROR in syncfolder for protonmail folder Folders/FSF ♢ GNU ♢ EFF: Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/offlineimap/accounts.py", line 652, in syncfolder
    check_uid_validity()
  File "/usr/lib/python3.12/site-packages/offlineimap/accounts.py", line 520, in check_uid_validity
    remotefolder.save_uidvalidity()
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/Base.py", line 300, in save_uidvalidity
    newval = self.get_uidvalidity()
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/IMAP.py", line 132, in get_uidvalidity
    self.__selectro(imapobj)
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/IMAP.py", line 80, in __selectro
    imapobj.select(self.getfullIMAPname(), force=force)
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplibutil.py", line 57, in select
    result = super(UsefulIMAPMixIn, self).select(mailbox, readonly)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplib2.py", line 1082, in select
    self._deliver_exc(self.error, '%s command error: %s %s. Data: %.100s' % (name, typ, dat, mailbox), kw)
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplib2.py", line 1496, in _deliver_exc
    raise exc(dat)
offlineimap.imaplib2.IMAP4.error: SELECT command error: BAD [b'[Error offset=31]: Expected \'\\\' or \'"\' after \'\\\' in quoted']. Data: "Folders/FSF \/ GNU \/ EFF"

  SELECT command error: BAD [b'[Error offset=31]: Expected \'\\\' or \'"\' after \'\\\' in quoted']. Data: "Folders/FSF \/ GNU \/ EFF"
2024-09-27 19:17:51 WARNING: ERROR: ERROR in syncfolder for protonmail folder Folders/Swim ♢ Under: Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/offlineimap/accounts.py", line 652, in syncfolder
    check_uid_validity()
  File "/usr/lib/python3.12/site-packages/offlineimap/accounts.py", line 520, in check_uid_validity
    remotefolder.save_uidvalidity()
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/Base.py", line 300, in save_uidvalidity
    newval = self.get_uidvalidity()
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/IMAP.py", line 132, in get_uidvalidity
    self.__selectro(imapobj)
  File "/usr/lib/python3.12/site-packages/offlineimap/folder/IMAP.py", line 80, in __selectro
    imapobj.select(self.getfullIMAPname(), force=force)
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplibutil.py", line 57, in select
    result = super(UsefulIMAPMixIn, self).select(mailbox, readonly)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplib2.py", line 1082, in select
    self._deliver_exc(self.error, '%s command error: %s %s. Data: %.100s' % (name, typ, dat, mailbox), kw)
  File "/usr/lib/python3.12/site-packages/offlineimap/imaplib2.py", line 1496, in _deliver_exc
    raise exc(dat)

Steps to reproduce the error

  • Just ran offlineimap to capture my Protonmail emails.
@etbuira
Copy link

etbuira commented Sep 28, 2024

Hi,

Can you try with imapmirror (https://github.com/IMAPMirror/imapmirror)? It's better at encoding.

@thekix
Copy link
Member

thekix commented Sep 28, 2024

Dear @mmauger

I will check this problem ASAP. If the problem is with imaplib2, then we can open an issue in the library.

@etbuira imapmirror is not based in imaplib2? This problem is not present in imapmirror?

@etbuira
Copy link

etbuira commented Sep 28, 2024 •
edited
Loading

I don't know if imapmirror would sync correct on this.

But i know "\/" is not allowed in imap4 quoted strings, and there is no way to escape the mailbox separator. The part trying to "neutralise" any slash if "/" is mailbox separator is at fault (and it is not imaplib2).

@thekix
Copy link
Member

thekix commented Sep 29, 2024

Dear @mmauger

I created a folder (web interface). I synced without problems:

 Syncing Folders/Test\/Folder: IMAP -> Maildir
INFO:OfflineImap:Syncing Folders/Test\/Folder: IMAP -> Maildir

I moved a mail inside de the folder (using local email client). Then sync:

INFO:OfflineImap:Syncing Folders/Test\/Folder: IMAP -> Maildir
 Copy message UID -1 (1/1) Local-protonmail:Folders.Test\.Folder -> Remote-protonmail:Folders/Test\/Folder
INFO:OfflineImap:Copy message UID -1 (1/1) Local-protonmail:Folders.Test\.Folder -> Remote-protonmail:Folders/Test\/Folder

The folder in the filesystem is 'Folders.Test\.Folder' (slash are dots). I am not using nametrans.

Can you try without nametrans? What changes can I do to reproduce the issue?

Regards,
kix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thekix @mmauger @etbuira