tracking issue: WebExtension port (aka. Firefox Quantum / Firefox 57+ support)

[myrdd]

Starting with Firefox 57+, only WebExtension Add-Ons will run in the browser. RequestPolicy needs to be ported to the WebExtension API.

You want to help with this transition? The first thing you can do is to install the “Development Channel” version [1] of RequestPolicy Continued and to report any bug you encounter with this specific channel. Thank you.

[1] RPC dev-channel: on the bottom of page https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/ [Edit, 2018-04-04] https://sslsites.de/requestpolicy.256k.de/requestpolicy-nightly.xpi [Edit, 2018-10-25] https://requestpolicy.256k.de/requestpolicy-nightly.xpi

[acrobat1]

In layman's terms, what would this mean for RP ?
Will it continue to work, or require major rewriting to do so ?
Can we users expect the same level of functionality, or will it be somehow restricted by this change?
Thanks.

[ivan-kolmychek]

@myrdd I agree, some explaination would be just wonderful, because not everyone has time and experience to get through all the docs and understand all the consequences.

In WebRequest documentation I've not seen anything that would actually prevent the RP from blocking a request, but I don't know how much of the requests will actually be exposed through it - for instance, can we block requests of plugins?

I'm also concerned about some details of Chrome's implementation:

Moreover, only the following schemes are accessible: http://, https://, ftp://, file://, or chrome-extension://. In addition, even certain requests with URLs using one of the above schemes are hidden, e.g., chrome-extension://other_extension_id where other_extension_id is not the ID of the extension to handle the request, https://www.google.com/chrome, and others (this list is not complete).

I've not found anything about that in firefox wiki, but I've not looked for it too much yet, just read the Chrome's verstion.

[myrdd]

TL;DR / Very short answer: There's no need to believe RP is going to discontinue to work. Some major rewriting might be necessary in the long term. I will make sure that RP won't lose functionality.

---

Mozilla's aim of the announcement was to start discussion with the community. The discussion is about the WebExtensions API, a „new browser extension API“.

Right now the technologies used to create extensions are XUL and XPCOM. Creating an API doesn't mean that the old technologies are dropped.

Still, we are not sure if XUL/XPCOM is dropped in a few years or not, so I think it's important that the API provides access to all functionality needed by add-on developers. I believe that someday Mozilla's future Layout Engine Servo, which is still experimental, will replace the current Layout Engine Gecko, which is used in Firefox and SeaMonkey. Servo is completely rewritten, and afaict it does not implement XPCOM. Now, what I believe is that Mozilla wants to develop the WebExtensions API independently from the underlying technology so that it can be supported by both Gecko and Servo extensions.

In my opinion Servo is a great project, but also the programming language it's written in, Rust, which has been developed by Mozilla Research as well. Having a browser with Servo/Rust someday in the future will be a great effort.

Now, what does this mean for RequestPolicy? It means we need to make sure that anything needed by RP will be supported by the WebExtensions API.

The XUL technology is not that important to RequestPolicy. However, the XPCOM technology is. One very important part of XPCOM used by RP is nsIContentPolicy.shouldLoad. The corresponding API to that function is WebRequest.onBeforeRequest. Still, the API isn't equivalent to shouldLoad yet. The parameters I need to request are definitely aRequestOrigin (the origin URI) and aContext (e.g. the HTMLElement in the DOM); maybe also aContentType and aRequestPrincipal.

[myrdd]

I've updated the first comment to represent my current knowledge. Input and investigation welcome.

[myrdd]

Just to give an update, I'm currently working on a WebExtension version of RPC.

[myrdd]

All those willing to help me with the transition to WebExtensions, please install the “Development Channel” version [1] of RPC, and report any bug you encounter. Thank you.

[1] on the bottom of page https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/

[genodeftest]

@myrdd Is it intended that Firefox still takes the extension as not-e10s-compatible?

[myrdd]

@genodeftest yes, RPC still isn't e10s-compatible. compatibility will come with the migration to webext.

[Pentarctagon]

Is it correct that the version in the dev channel is 1.0.beta12.4, while the currently released version is 1.0.beta13.0?

[myrdd]

This wasn't intended. Seems like I misread the documentation, so release/beta channels are really separated: users on the dev channel won't get an update if a new release version is released, even if the version number and release date are newer. However, the only difference between the newest dev-channel version and the beta13 is the version number (and a fixed UI test), as shows the diff. Thanks for the notice anyway.

[DJ-Leith]

@myrdd I hope you can port / rewrite RPC for Firefox 57.

Input and investigation welcome.

In my opinion, a good place to start reading is Aris (who created "Classic Theme Restorer")
clear explanation here:

"Legacy add-ons like CTR will stop working when Mozillas XUL/XPCOM support ends with Firefox 57
release"
Aris-t2/ClassicThemeRestorer#299

There are several very helpful links there.

In addition to the links in Aris' article see also these 3:

"Add-ons/Firefox57"
https://wiki.mozilla.org/Add-ons/Firefox57
This is Mozilla's Official wiki (but as it is a wiki it changes, so keep checking).

Andy McKay, who is very influential in the move to WebExtensions, has
posted on some Official Mozilla blogs (you can search for them).

This post, from his personal blog, gives a clear insight as to WHY
he is so keen on WebExtensions.
http://www.mckay.pub/2016-11-26-innovation/

"Stuff we can remove when XPCOM extensions are no longer supported"
https://bugzilla.mozilla.org/show_bug.cgi?id=1347507&format=default
See the "Depends on:" field, there are dozens of Bugs there.

As I am cautious, I moved all my important Profiles, in April 2017
(while Firefox Release was version 52), to the ESR 52.x.x

Nightly is already at version 57 and I expect 'things to break'.

DJ-Leith

[myrdd]

I've released a new pre-version on the development channel. It introduces many code changes necessary for WE migration. The pre-version does not contain new features, and it's not a WE yet. Please install and test the pre-version: https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/versions/beta

[genodeftest]

@myrdd : I'm running your pre-version for about a week without visible functionality issues.

On every start, I get these lines printed to console:

console.debug: 
  [RequestPolicy] starting up
console.info: [RequestPolicy] [POLICY] PolicyManager::loadUserRules loading user rules
console.info: [RequestPolicy] [POLICY] PolicyManager::loadSubscriptionRules: official / allow_extensions
console.info: [RequestPolicy] [POLICY] PolicyManager::loadSubscriptionRules: official / allow_sameorg
console.info: [RequestPolicy] [POLICY] PolicyManager::loadSubscriptionRules: official / allow_mozilla
console.info: [RequestPolicy] Will update subscription: official allow_extensions
console.info: [RequestPolicy] Will update subscription: official allow_sameorg
console.info: [RequestPolicy] Will update subscription: official allow_mozilla
console.info: [RequestPolicy] Will update list: official
console.info: [RequestPolicy] Updating [SubscriptionList official https://raw.githubusercontent.com/RequestPolicyContinued/subscriptions/master/official.json]

Also, since today I'm getting this error message printed to console/syslog:

console.error:
  [RequestPolicy] Fatal Error:
console.dir:
  Message: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIURI.asciiHost]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://rpcontinued/content/lib/request.js :: Request.prototype.isInternal :: line 76"  data: no]
  Stack:
    Request.prototype.isInternal@chrome://rpcontinued/content/lib/request.js:76:1
NormalRequest.prototype.isInternal@chrome://rpcontinued/content/lib/request.js:160:14
self.process@chrome://rpcontinued/content/lib/request-processor.js:262:17
self.shouldLoad@chrome://rpcontinued/content/main/content-policy.js:60:16

This happened 5 times today, but I cannot tell you when it happened, i.e. I don't have a reproducer.

[myrdd]

@genodeftest which browser and version are you using?

[myrdd]

Nevermind, I found the problem. It ought to be solved in the next pre-release.

[genodeftest]

Thanks!

@genodeftest which browser and version are you using?

That was Firefox 56.0 x86_64 from Fedora 26 repositories.

[shirishag75]

It took quite sometime to find this tracking bug as others are closed but still mentioned in bugs/tickets elsewhere. I did comment on one but not others.

[cheggerdev]

nightly build has version number 1.0.beta13.2.2462.r2f19aa957.pre - where is the 1.0.beta14.0 ?

[shirishag75]

@cheggerdev you have to build it yourself using nodejs/npm. I did it sometime back but it still falls short of being able to install on firefox 57 or/and later :( even after allowing unsigned extensions (which is bad policy security-wise. ) I think you will find my instructions on the issue referenced above, issue 900.

[iprok]

I care about getting this fixed, so I'm offering USD 50 via FreedomSponsors to the first person who fix it.

Offer link: https://freedomsponsors.org/issue/835/tracking-issue-webextension-port-aka-firefox-quantum-firefox-57-support

You can also join me and throw in a few bucks there and we'll get it fixed faster :)

If you fix this issue (see my acceptance criteria there) please use that site to request your payment.

[shirishag75]

One other thing you can try is to open the browser console as I had, see #900 if you still get the issue then you know the problems that are there. From what little I know of firefox anything from firefox 53 and later is/was supposed to be web-ext friendly .

I have shared one of the popular web-extensions that I use tree-style-tab as to how it is -

https://paste.debian.net/1047280/

the .xpi file that you have before you is nothing but an archive or more commonly a .zip file of sort. You could easily rename it from request-policy.xpi to request-policy.zip and see how the two extensions differ.

There are at least three bugs that you would need to fix/port over that I know that need fixing before it could be called a web-extension -

#863, #891, #893, #894 are the four bugs that need to be fixed . Martin has made some documentation so you or whoever wants to wet their feet wet in https://github.com/RequestPolicyContinued/requestpolicy/wiki/Setting-up-a-development-environment and https://github.com/RequestPolicyContinued/requestpolicy/wiki/Building , there are other pages but do not know how relevant they would be after Mozilla re-used web-extension add-ons which are used by other browser vendors.

See https://wiki.mozilla.org/WebExtensions#Status and they have put whole lot of documentation and care into giving budding extension developers a leg up. See https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Getting_started_with_web-ext and https://github.com/mdn/webextensions-examples

In a nutshell, if you succeed at making a web-extension porting it other browsers in theory should be a child's play. The benefit to users and developers is that dodgy extensions are easily called out, the reality may be a bit more complex.

[shirishag75]

if you look at his commits he is still fighting the fallout from Firefox 48.0.x still.

https://github.com/RequestPolicyContinued/requestpolicy/commits/develop

[shirishag75]

@wilkowy your best bet is to make a new issue, share how you built it, if possible share the SHA1sum of the built extension and the output in browser-console when you try installing it in firefox 48.0.2 . Do make sure you have his latest commit shared last around 12 days ago which results in a build version 1.14.0 and not any lower version. Hopefully @myrdd may be able to help you.

[myrdd]

@cheggerdev @wilkowy yes, please report your build & install problems in a new issue. Include the versions of the installed build requirements as well as the exact git commit you're using in your report. And, as @shirishag75 mentioned, you best add the sha1sum as well.

with Fx 48.0.2, no matter what is set in xpinstall.whitelist.required and xpinstall.signatures.required version 14.0 refuses to install.

depending on the firefox updating mechanism of your firefox installation you might need to lock some preferences to prevent updates: (you need to do this e.g. if you've installed any firefox release from here.)

// this file must start with a comment!
lockPref("app.update.auto", false);
lockPref("app.update.enabled", false);

[myrdd]

to keep this issue clean, from now on please use #902 for anything doc/wiki-related

btw @shirishag75 I use to read comments in my mailbox, and this way I don't get notified about edits on your comments, even if made few minutes after. pls try to put all info into the comment before publishing the comment.

[cheggerdev]

Is there any progress since October 2018?

[shirishag75]

Is there any progress since October 2018?

I don't think so, see https://github.com/RequestPolicyContinued/requestpolicy/commits/develop

[myrdd]

Sadly, no 😞 but thank you for passing by once again. This tells me I'm not the only person missing this add-on. 👀

I'm about to improve (once again) my personal time and priorities management.
BTW, friendly ”pings“ are often motivational to me 😉

[SiliconAlley]

I like to send a friendly ping to you Martin.

The time has come, as I'm still of FF56 only for RPC. But now they disabled a lot of my add-ons and at least this forces me to update to the latest FF. I think many are reading here, without logging on. I especially created an account for this comment.
I will miss RPC a lot. Maybe I switch my current profile to waterfox.

Thanks and cheers from Germany too.

[baptx]

@SiliconAlley I started using uMatrix which has similar features to block third-party requests, in addition to cookies and JavaScript. It also has a protection to avoid leaking the Referer HTTP header for better privacy. It may help you as an alternative addon to RequestPolicy.

[SiliconAlley]

Thanks @baptx, installed on my company notebook already, where I run the latest FF and miss RPC.
Just to mention here: Mozilla will fix add-on signing issue for older Firefox versions

[wilkowy]

@SiliconAlley try setting xpinstall.signatures.required to false. This disables signing requirement.

[genodeftest]

@SiliconAlley @wilkowy : This setting will only have an effect on nightly/alpha and some custom builds of Firefox (e.g. the version shipped with Fedora Linux).

[shirishag75]

Just to share @myrdd did release a new pre-release of RPC yesterday. I renamed it as 1.0.beta14.1.2485.r6d27d1e5.pre.xpi as that commit id was mentioned in manifest.json for the release. While it still doesn't get installed it I am sure @myrdd made some more changes, for us it would be usable only when the whole thing is ported to the webextension. FWIW, I did toggle xpinstall.signatures.required to false in about:config but still was unable to install it. Best of luck to @myrdd to getting the other pieces of the extension ported to changes in mozilla upstream. Hope it will work with Firefox 68.0.x the latest ESR stable from mozilla .

[jbuttery]

Hello @myrdd! I normally wouldn't post a low-value comment like this, but you did specifically say this: 😄

Sadly, no 😞 but thank you for passing by once again. This tells me I'm not the only person missing this add-on. 👀

I'm about to improve (once again) my personal time and priorities management.
BTW, friendly ”pings“ are often motivational to me 😉

Barely a day goes by (yes, still in December 2019) that I don't miss this extension. I've been using "Third-Party Request Blocker" (https://addons.mozilla.org/en-US/firefox/addon/tprb) but...it's just not the same.

Is there anything a non-developer can do to help make RPC a reality again? Pretty please? 😄

[baptx]

@SiliconAlley I started using uMatrix which has similar features to block third-party requests, in addition to cookies and JavaScript. It also has a protection to avoid leaking the Referer HTTP header for better privacy. It may help you as an alternative addon to RequestPolicy.

Since uMatrix is not supported anymore, I am using uBlock in advanced strict mode and NoScript to block JavaScript. For cookies blocking I am currently using the built-in cookies blocker of Firefox in strict mode. Blocking the Referer HTTP header can be done with the addon Header Editor (which supports request and response headers).

There are pages mentioning that uBlock can be used as a replacement to RequestPolicy:
https://github.com/gorhill/uBlock/wiki/Advanced-user-features
https://github.com/gorhill/uBlock/wiki/Dynamic-filtering