Greasemonkey: WebExtension compatibility

I've been thinking about this a bunch. I don't have a clear "decision", but some points:

• We _just_ got done with the process of porting for e10s compatibility. It was way harder when we started; e.g. Services.ppmm and .cpmm are nice shortcuts, which it would have been nice to rely on from the beginning, but they didn't exist when we (responsibly) started.
• That work was long and very painful, and I'm not looking forward to effectively repeating it.
• The e10s rollout has slipped from Firefox 36 (as of Sep 2014) to Firefox 42 (as of now, Sep 2015), or at least nine months.
• The announcement says webextensions-only is at least 1 or 2 years away; it will slip to 2, 3, 4 years from now?
• If we're going to do this, I think it's the right time to make a clean break and re-architect.
  
  
  
  • I find myself wishing we had good unit tests semi-often, even though we have a lot of issues they'd never catch, we have some they would have, too.
  
  
  • We can finally add Android support?
  
  
  • We don't have to actually re-write from scratch, but being thoughtful in which code we retain and which we drop may be warranted.
  
  
• I'd really love a much stronger Greasemonkey/Mozilla relationship to be in place before we start another task this large. I have only weak guesses at how to make that a reality.
  
  
  
  • I think we're effectively a torture test for porting to something like webextensions today; over the years we've built up quite a few advanced features. A two way communication as part of the plan will probably help a lot.
  
  

Starting with a design doc would be a great idea. Essentially we need to reverse engineer all of Greasemonkey as it exists into a plan for a good way, rather than an unplanned organically grown way, for all of it to be structured.

e.g. Services.ppmm and .cpmm are nice shortcuts, which it would have been nice to rely on from the beginning, but they didn't exist when we (responsibly) started.

I'm certainly not advocating using WebExtensions yet, they are way to immature for something like GM

We don't have to actually re-write from scratch, but being thoughtful in which code we retain and which we drop may be warranted.

Hrm, well, I was looking at it mostly from a technology POV. Right now the UI uses XUL overlays and direct script execution in the shared chrome environment.

So converting things to HTML and running each things in an independent window context and only communicate via message-passing seem to be how things are supposed to be done in the future.

The message manager is basically the lowest level where that's implemented. Higher level abstractions build on that. WebChannel.jsm/BroadcasstChannel/MessageChannel/WebExtension channels and such.

Starting with a design doc would be a great idea.

Would the GH wiki be the right place for that? Does it do notifications on edit to make collaborative work simple?

I think we mostly need a feature / internal plumbing list and how to implement each in a clean manner.

If you're interested in this issue's topic, please read:

https://groups.google.com/d/topic/greasemonkey-dev/K6IyDUWnTQc/discussion

Thanks!

Relevant:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1271553
• https://bugzilla.mozilla.org/show_bug.cgi?id=1334174

https://developer.mozilla.org/en-US/docs/Web/API/File_and_Directory_Entries_API/Introduction

This is a super interesting API, but "This feature is non-standard and is not on a standards track" and compatibility is very limited.

All my recent attempts (see #2483, #2484) at designing fully-featured Greasemonkey-under-WebExtensions have been frustrating. I'm starting to consider a more progressive style of development: pick a more limited set of features and support only that. Be a little bit useful, then hopefully find a path towards more functionality later.

Inspecting my 3.x install, I see that (for me) every single script is @grant none. Of 27, only six use @run-at document-start, and most of those would function at least somewhat gracefully if that was not supported. The @require feature is used heavily, and @resource quite a bit.

So that seems like a decent target to aim for first: Support plain user scripts in @grant none mode, no support for any GM_ APIs. Support @require. Hope to support @resource somehow, perhaps inefficiently.

(Side note, because I keep forgetting it: Plan to use sourceURL to make errors slightly more readable. Look into whether multiple sourceURLs are supported, or whether we'd have to (can we?) generate a sourceMap, factoring @require into consideration.)

We get access to plain Web APIs in addition to WebExtension ones, so:

https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API

? What's the storage limit of IndexedDB, for a WebExtension? This looks like a much better option than storage.local. The interface isn't the simplest, but it gives us more power to segregate scripts from each other and do selective reads. I think. Docs are also not the easiest to use.

https://github.com/mdn/webextensions-examples/pull/171 seems to have valuable discussion and example for IndexedDB

I've made some progress.

https://github.com/arantius/greasemonkey/tree/webbymonkey (at 88d53b4c67b7825858405eb2591f27c8487ce413 )

Reimplemented from scratch. Check out, go to about:debugging, press "Load Temporary Add-on" and select any file from the root location. You can install and just barely run user scripts. Absolutely no other features yet. (Well, there's the monkey menu, but it's an empty fake that doesn't do anything besides look OK.) Lots of TODOs scattered around the code for even this slight feature set. Can't guarantee any of this is the "right path" towards more features or not.

Since older versions of Tampermonkey, as well as Violentmonkey, are open source, could some of that code be used here, since WebExtensions is similar to Chromium Extensions?
EDIT: Actually, looking at it, I'm not sure about license compatibility with old Tampermonkey. But Violentmonkey is MIT-licensed, so it is compatible.

@PorygonZRocks: Violentmonkey becoming Greasemonkey?

I'm not very experienced with coding, but I was thinking it could at least have some guidelines on implementing things. Again, not very experienced/knowledgeable, so I may be wrong.

It is worth noting that FF56 disabled all non-multiprocess compatible addons so the deadline might need to change.

See my dev branch. Which is quite rough but is minimally functional. This is being done, there's nothing to track specifically in this broadly scoped issue.

@arantius Out of curiosity, are you writing new code or reusing the old?

Do you need help?

Again out of curiosity, for example, is the purpose of "parse-meta-line.js" is merely to parse the meta data into an object?

@arantius Out of curiosity, are you writing new code or reusing the old?

Mostly new. Copying when/where it's helpful. (So far, script parsing is a big example.)

Do you need help?

Help would be nice. Coordinating would be hard.

Again out of curiosity, for example, is the purpose of "parse-meta-line.js" is merely to parse the meta data into an object?

One line of it, yes.

One line of it, yes.
I meant, does it do anything else except parsing the meta data between these:

// ==UserScript==
....
// ==/UserScript==

It seems like a lot of code if that is the only objective.

Yes that's what it does. It's generated code. Please take this discussion to https://groups.google.com/d/topic/greasemonkey-dev .

I don't use Google groups :(
If it were me... I would probably do it differently.

The google group does not exist anymore.

Bad link: https://groups.google.com/forum/#!forum/greasemonkey-dev

Not entirely sure if this is the correct location for it, but here it is anyway. If you want me to open a separate issue @arantius, sure.
Isn't the 4.0.0 supposed to migrate the existing scripts? I updated to alpha 3 (coming from the latest non-beta) and noticed I had no scripts anymore.

@Phyxion, if you install 3.14 then the scripts should be migrated. Make sure you restart the browser after installing.

After that when you install 4.x you should have the scripts. If you don't then some detailed reproduction steps in a new issue would be helpful.

Greasemonkey 4 alpha is incompatible with Firefox 57.

@erkinalp, Could you elaborate? I've been using 4alpha2 for a lot of my testing and modifications, it works, although not all features in 3.x are available. I haven't pulled the changes in 4alpha3, so I don't know the few commits for that version break anything.

@Sxderp Well, reproducing it on my machine is easy. I have 3.14 installed with 10 user scripts. Go to AMO and download the latest alpha. Restart when asked. Then it just says there are no userscripts installed. Not sure how useful this for reproducing it.

I haven't tested this yet but I believe GM4 should lose its configuration after an uninstall, while GM3 should keep it, so I'd suggest you try:

1. Completely uninstall Greasemonkey
2. Restart Firefox
3. Install Greasemonkey 3.14 (including restart)
4. Restart Firefox for good measure
5. Install Greasemonkey 4 (dot anything)

Does that help?

I think the top level await - i.e. wrapping everything in an async function - is a not a good design choice since it restricts future implementations (e.g. if/when we get sandboxes back or es-future realms). It makes things inconsistent with vanilla JS execution where top level await is not available and scripts execute... well... at a top level.

@arantius
Still nothing here :(
BTW, is 4.0 supposed to look like this: https://i.imgur.com/CPuWWKM.png
There are no buttons or anything to add a script.

... wrapping everything in an async function ...

With what's available now I "have" to wrap in a function for scope purposes. I think I buy your reasoning for not making it async though. (At the least, adding an async wrapper function in the script itself is trivial.)

Yeah, it was mostly about the async/await, since that's currently intentionally not allowed in javascript, so enabling it in userscripts seems like a hazard for future changes. I know that the wrapper is currently unavoidable, but I hope that things could be moved back to the top level in the future.

@arantius Why the down vote?

@arantius commented on 2017. szept. 28. 14:57 CEST:

... wrapping everything in an async function ...

With what's available now I "have" to wrap in a function for scope purposes. I think I buy your reasoning for not making it async though. (At the least, adding an async wrapper function in the script itself is trivial.)

Try deleting the content of [mozilla profile]\storage\default\ (After backing it up)
Then try again.

Speaking of UserScript objects, I don't quite understand the design decision for having three UserScript classes. At the moment they have a singular inheritance tree which is a little silly.

Further, RemoteUserScript is only used once and for that it's only created in order to grab the id. And RunnableUserScript is never directly used.

Just for Info ...

I am on FF57.0a1 and running legacy addons and GM 3.13
Sadly, I dont get the updates (3.14, 3.15) since their max version is set to 56.*

It can be installed manually though..

GM 3.16 still hangs the browser on start-up (I guess it is updating the DB)