Screen Translation Feature for PowerToys - Real-time/Captured Text Translation

[ucs-marsala]

Description of the new feature / enhancement

I propose adding a screen translation feature to PowerToys that would allow users to:

1. Real-time Translation Mode:

• Hover over any text on screen to see instant translations
• Support for multiple source and target languages
• Customizable overlay display for translated text

2. Capture Translation Mode:

• Select specific areas of screen to translate text
• OCR capability to recognize text from images/screenshots
• Copy translated text to clipboard functionality

Key Features:

• Integration with major translation APIs (Google, Microsoft, DeepL)
• Language detection
• Customizable keyboard shortcuts
• History of recent translations
• Support for multiple monitor setups

This would seamlessly integrate with PowerToys' existing features like PowerToys Run and Text Extractor, enhancing the overall utility of the application.

Scenario when this would be used?

This feature would be invaluable in several common scenarios:

1. International Professional Work:

• Reading foreign language documentation
• Reviewing international communications
• Working with multilingual content

2. Research and Learning:

• Students reading academic papers in different languages
• Researchers accessing international resources
• Self-learners studying foreign language content

3. Content Creation:

• Developers working with international codebases
• Content creators localizing materials
• Translators needing quick reference tools

4. Daily Productivity:

• Quick translation of error messages
• Understanding foreign language software interfaces
• Reading international news or articles

This tool would significantly improve workflow efficiency by eliminating the need to copy/paste text into external translation services.

Supporting information

Similar features exist in other tools:

• Google Chrome's built-in translation
• Windows Store apps like "Translate Now"
• MacOS's built-in translation feature

However, integrating this into PowerToys would provide:

1. System-wide functionality rather than browser-only translation
2. Better privacy as it can be configured with preferred translation APIs
3. Seamless integration with existing PowerToys features
4. Open-source development allowing community contributions

Technical considerations:

• OCR capabilities could leverage Windows OCR API
• Translation could use Microsoft Translator API or user-configured services
• UI could follow existing PowerToys design patterns

[similar-issues-ai]

We've found some similar issues:

• On-screen translation #30771 , similarity score: 86%
• Translate selected text #22705 , similarity score: 84%

If any of the above are duplicates, please consider closing this issue out and adding additional context in the original issue.

Note: You can give me feedback by 👍 or 👎 this comment.