Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bundle indexing and preview computation in a single listener #1482

Open
chibenwa opened this issue Jan 21, 2025 · 4 comments
Open

Bundle indexing and preview computation in a single listener #1482

chibenwa opened this issue Jan 21, 2025 · 4 comments
Assignees
Labels

Comments

@chibenwa
Copy link
Member

Why?

Mass delivering enail saturates network link.

Image

The pitfall is that for each mailbox we need to re-download the all message in order to...

  • Index it in OpenSearch
  • Pre-compute the preview

So 13 MB x 2 x 10.000 emails * 8 bit ~= 2 Tb of traffic
With a link speed of ~1Gb/s that's 40 minutes of link saturration.

Which is exactly what we see: incoming link saturated, outgoing link easy peasy.

How?

We can bundle Index it in OpenSearch + Pre-compute the preview as a single listener in Twake mail side and thus download the message for post processing only once and not twice.

Easy to do, gain a factor 2 (and reduces OVH cloud bill)

@chibenwa chibenwa added the perf label Jan 21, 2025
@quantranhong1999 quantranhong1999 self-assigned this Feb 3, 2025
@quantranhong1999
Copy link
Member

WIP: #1508

@chibenwa
Copy link
Member Author

chibenwa commented Feb 5, 2025

Proposal:

inferface MessageIndexer {
    Mono<Void> index(MailboxMessage message, Added event);
}

Write an OpenSearchMessageIndexer.

Write a ComputePreviewMessageIndexer.

(Extract some logic in ComputePreviewListener so that it gets reused into ComputePreviewMessageIndexer )

Inject Set<MessageIndexer> into the OpenSearchMailboxSearchIndex.

@quantranhong1999
Copy link
Member

apache/james-project#2631

I will create a dedicated JIRA ticket if this goes in the right direction.

@chibenwa
Copy link
Member Author

chibenwa commented Feb 6, 2025

Not really this is overly complex and I see many needless refactoring arising from the interface being wired in the wrong place.

Let me propose an alternative...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants