Mostly I’ve been really encouraged during the development of this at just how far our platform has come for developing restartless add-ons. When we first made it possible in Firefox 4 there was a whole list of things that were quite difficult to do but we’ve come a long way since then. While there are still things that are difficult there are lots of things that are now pretty straightforward. My add-on loads simple XUL overlays, style overlays, installs JS XPCOM components with category manager registration, all similar to older add-ons. In fact I’m struggling to think of things that it is still hard to do though I’m sure other more prolific developers will have plenty of comments on that!

The other thing I’ve been doing with this extension is experimenting with git and GitHub. I think it’s been an interesting experience, there are continual arguments over which is better between git and mercurial with many pros and cons listed. I think most of these were done some time ago before mercurial and git really matured because from what I’ve seen there is really little difference between the two. They have slightly different default branching styles, but both can do the same kind of branching that the other can if you want and there are a few other minor differences but nothing that would really make me all that bothered over deciding which to use. I think the only place where git has a bonus is with GitHub, and really as far as I can see there isn’t a reason why someone couldn’t develop a similar site backed by mercurial repositories, it’s just that no-one really has.

GitHub is pretty nice with built-in basic issue tracking and documentation though it still has some frustrating issues. It seems odd for example that you can’t fork your own project, only someone else’s, but that’s only a minor niggle really. As project hosting goes I can’t say I’ve come across anything better that I can remember.

Restartless add-ons change this around completely, the normal overlay and style-overlay mechanisms just aren’t available to restartless add-ons and this is likely to remain true for a while, these methods don’t clean up after themselves when the add-on is uninstalled.

This can make things hard, particularly for porting older add-ons to become restartless. I was in this situation earlier this weekend. I was working on porting David Ascher‘s WebTabs for Thunderbird to be restartless. I could have just moved all the script code over to bootstrap.js but in many ways it is nice to keep the code that works on the main UI separate to the code that runs for the preferences UI etc. Plus I like to play around with new ways of doing things so I developed a JS module I’m calling the OverlayManager.

The OverlayManager watches for new windows being opened and for every new window it can run JS script and apply CSS stylesheets to the window in a way that is easy to undo if the add-on is disabled at runtime. Although it can’t do any XUL modifications right now (I didn’t need any for this particular extension) it would be pretty easy to extend this to support a minimum about of XUL overlays.

Stylesheets are loaded by adding a HTML style tag to the XUL document, so they can be removed easily when the add-on is disabled. Scripts are handled in a way that may even be better than normal overlays. In the old system extension scripts all run in the same context as the window they overlay giving rise to the possibility of conflicts. Restartless add-ons shouldn’t do this since it makes removing the script code again much more troublesome. The OverlayManager handles it by creating a sandbox to run the script in. The sandbox’s prototype is set to the window the script is being run for meaning the script sees all the functions and objects of the window directly in its own scope but as long as it doesn’t modify any of the objects in the main window’s code all we have to do is throw away the sandbox to get rid of its JS.

There are a few things different of course. The script shouldn’t use load and unload event handlers for the window as it may get loaded well after the window does or unloaded well before. Instead the OverlayManager looks for an OverlayListener object in the script and calls load and unload methods on it, these are called either with the window’s real load and unload events or while the window is open normally. You also can’t reference code in the script from JS string blocks, so if you set onclick="myfunc()" on a XUL element it wouldn’t work because that would run in the main window scope which can’t see the sandbox code at that point. This tends to be pretty simple to get around by using addEventListener for all your events though.

You can see the existing state of the code on github and an example of the structure you’d pass to OverlayManager.addOverlays is in the bootstrap script for the same project. It is appropriately licensed so go nuts!

Update: I changed the stylesheets to use XML processing instructions to be more like they work currently and just for fun I implemented the very basics of document overlaying, almost totally untested though so YMMV.

The basic idea is simple. Create a XUL file containing a list of <setting> elements. Different types of settings are possible, checkboxes, input boxes, menulists, buttons, etc. Each one shows up as a row in the details view for an add-on in the add-ons manager. The XUL file can either be just added to your XPI (call it options.xul) or referenced by the optionsURL option in your install.rdf.

Get it right and you’ll see something like this:

I want to thank Geoff Lankow (darktrojan on IRC) for his awesome work getting this done. This feature is now in the Aurora builds and it’d be great to get add-on developers playing with it. Geoff even wrote up some detailed docs to help you out.

As a bonus Geoff also implemented support for in-tab preferences. This makes Firefox load an add-ons options UI in a new tab instead of a new window. Setting the optionsType property to 3 enables this.

Just call Components.utils.unload(uri) and the module loaded from that URI will be unloaded. Take care when you do this because something might still have references into it which will stop working. Firefox also caches the module’s code for fast loading the next time around. The add-ons manager clears this cache when your add-on is updated or uninstalled so you mostly don’t have to worry about it but if you do something strange like unload a module, manually alter the file and then import it again you won’t get the latest code.

This support is in the upcoming Aurora build.

Plugins, like Flash, Quicktime and Java for example, are a little more difficult to update in this way though. They tend to require explicit permission to install new versions and so we haven’t quite gotten to the point of doing this completely automatically. Instead we developed the plugin check page which can quickly and easily tell you which of the plugins you have installed are old and need updating. It will also tell you where to go to update them.

The latest version of Firefox currently in beta makes it easier to get to the plugin check page. Simply go to the Add-ons Manager, click on the Plugins section and there is a link at the top of the page to check if your plugins are up to date.

The crux of the matter is in how deeply we allow add-ons integrate with Firefox. Most browsers separate add-ons from their code using what can be called an add-on API. All the add-on can see and use is what the browser makes available through that API and this is normally a restricted set of functionality. Firefox has no such separation. Add-on code runs in exactly the same setting as the browser code, they can call any of the internal functionality that we write to make the browser work in the first place.

There are big advantages to the Firefox way:

• Add-ons can do basically anything, allowing really complicated add-ons like Firebug and NoScript to exist.
• Authors don’t have to wait for features to be exposed through some API to be able to use it, as soon as the feature is in the application add-ons can use it.

There are also some disadvantages:

• Add-ons can do basically anything so you should be wary about installing add-ons from people you don’t trust!
• Because we aren’t forced to provide an API specifically for add-ons sometimes what is there can be cumbersome for add-ons to use.
• Add-ons rely on internal functions, if we make a change to one then it breaks the add-on.

It is that last point which explains why Firefox updates break add-ons. Any time we add, remove or change some code we run a risk that some add-on depends on it and so will break. Some bits of code are more important than others but we have such a vast library of add-ons now that it’s probably getting to the point where almost everything is in use. When people say that Firefox updates shouldn’t break add-ons what they don’t realise is that they’re asking us to stop making changes to Firefox: no new features, no bug fixes.

How does the Add-ons SDK fit in?

With the most recent update to Firefox we also made the first official release of the new add-ons SDK available. These are tools designed to make developing add-ons for Firefox easier and faster and they largely serve as an add-ons API sitting between the add-on and Firefox with one difference, the API is actually a part of the add-on so you can actually write your own additional APIs to supplement those in the core SDK.

Add-ons written with the SDK (and only using the core APIs) gain a lot of the benefits of add-ons for other browsers. The APIs in the SDK can be far more stable than those in the browser itself, as the browser code changes the internal SDK code can adapt to match allowing add-ons to work just by rebuilding with the newer SDK. This makes the problem of Firefox updates much smaller for those that can move their add-ons to the SDK.

I don’t foresee a future where the only add-ons are SDK based though. We’d lose too many really important add-ons that way so the SDK doesn’t completely solve the problem.

How does rapid release affect the problem?

As I’ve said, when Firefox updates it breaks add-ons. Ask any add-on author how much fun it was fixing their add-ons to work from Firefox 2 to Firefox 3 or 3.6 to 4 and you’ll probably get some pained expressions. Now think about how in the past we would do a Firefox update about once a year, now we are talking about it being more than eight times a year. Having given up maintaining my own add-ons I can hardly blame authors who are concerned that they won’t be able to keep up with the pace.

It actually isn’t all bad though. It’s not like we are making more changes than we were before, we’re just releasing them sooner in bite-sized chunks. Instead of one update a year that probably changes every piece of functionality an add-on might use we do 8 smaller updates, each one touching smaller parts of the code. This means it is more likely that an add-on can survive an update than before.

The stability of updates is a key change too. It used to be that add-on authors might choose to wait for the RC stage of an update before updating their add-ons. We’d make so many changes even during the beta cycle that sometimes it wasn’t worth trying to keep up. In our new cycle though there is a full 6 weeks in the beta cycle before release during which time almost nothing changes, no new features, extremely limited bugfixes, just things to solve any stability issues that have been found. Before that there is 6 weeks in the aurora cycle which again should be largely change free. Some new features might get removed at this point and there is the possibility of larger bugfixes, but based on the two aurora cycles we’ve had so far there are far fewer changes going in than there were in the final beta stages of Firefox 4.

I actually think that rapid releases can make it easier for add-on authors to keep up. Sure you have to check your add-on more often but most of the time it will probably just work, if not the scope of change is likely to be smaller and you have 12 weeks of pretty stable code to build against.

Why not just mark add-ons as working when they are?

A pretty common question asked is why we don’t just mark add-ons as working in the new update of Firefox when we know they are. The good news is that we actually are starting to do this. As we make changes to Firefox we plan to keep track of which changes have the potential to affect add-on compatibility and automatically scan add-ons on AMO to find ones that might break.

In the run up to the most recent update of Firefox we found nearly 4,000 add-ons that were unlikely to be affected by the update and just marked them as compatible with no need for the add-on author to do anything. For the others we emailed the author to explain what problems we had found. You can read more about the compatibility checking process we’ll be using.

As well as automated tools the add-on compatibility reporter provides add-on authors valuable feedback on how their add-ons are performing in Firefox. Users can install this tool and then use normally incompatible add-ons and file reports on whether they are working or not.

Faster releases are good for add-ons

Rapid release is all about getting new features into users’ hands faster. It has a side effect though, it gets new features and APIs into add-on authors’ hands faster too. Back when I was still writing add-ons I would frequently do so in response to some fantastic new thing that had been added to the Firefox nightlies. It could then be up to a year before I’d be able to release an add-on that normal users could actually make use of, by which point I’d often have lost interest in it.

I think that on the whole faster releases will be a good thing for add-ons. Add-on developers can get exciting new features into users’ hands faster and should only need to make smaller updates with each new Firefox. The new automated tools we are building to help authors understand when and why their add-ons are no longer compatible are going to be a massive improvement, especially as we get better at identifying the changes we make in Firefox that will affect add-ons.

Time will tell of course, but I do believe that the overall work an author has to do to keep an add-on up to date with Firefox is going to drop with the new rapid release cycle even though they may have to do something more often.

A few days ago (and now in Aurora builds) I landed some improvements that simplify the hardest part of this, making the UI appear.

Registering a custom type

Previously in order to add a new type of add-on and have it appear in the add-ons manager you would have to do two things, first tell the add-ons manager module that you supported add-ons and secondly manually create some elements in the UI to allow the user to view them. These two things have now been combined:

AddonManagerPrivate.registerProvider(myProvider, [{
  id: "scripts",
  name: "User Scripts",
  uiPriority: 3500,
  viewType: AddonManager.VIEW_TYPE_LIST
}]);

This registers the provider as before but also tells the module that your provider supports an AddonType with the ID “scripts” and some other information. The documentation explains a little more about the properties. You’ll note it passes an array to allow for providers registering multiple types.

Currently you only need to do this if you want your add-on types to show up in the UI, however we may start requiring providers to register their types to help speed up some of the API calls. If you want to register your type and not have the UI automatically show it just don’t give it a viewType.

Breaking changes

Some of the changes we had to make may have broken some add-ons that have been doing this the hard way in the past. Here are the once I’ve identified and some ways to work around the problems:

Types must be registered

The UI now requires that if you’re displaying your add-ons in the regular list view then it must have been registered. If you only care about supporting Firefox 6 and higher then you can just remove your code for adding your type to the category list, if you want to support both then you should either only do that on versions before Firefox 6, or you could always manually add your type to the UI but also register your type with no viewType.

Some IDs changed in the UI

We made some changes to the IDs of elements in the add-ons manager UI, we dropped the “s” on the end of the type selectors. category-extensions became category-extension as an example.

Category items aren’t in the UI immediately

Because we now build the list of types to display in the UI (even the built-in types use the same type registration I’ve talked about here), the elements for those types aren’t in the UI until after the main initialize function for the window has completed. If you try to use overlays that refer to those elements or if you run script that tries to find these elements before the initialize function completes (it is a load event listener) then they won’t exist and you’ll probably find things broken.

We’ve now changed this and any nightly, try, or self-made builds (anything not on the aurora, beta or release channels) will instead use the single preference extensions.checkCompatibility.nightly from now on, regardless of version. This will be available from tonight’s nightly onwards.

Of course, we aren’t finished. There were a lot of things that we wanted to get done and didn’t get time for and I’m sure we’ll be getting lots of feedback from users of the final release to work on. Since the pressure has been off I’ve been skimming back over bug reports and notes to try to come up with an idea of what we want to work on over the next few releases of Firefox. As you may know the plan is to move to quarterly releases and not really block a release on any particular projects but we will still be getting an idea of what product drivers will prioritise work on and guess at what other things we can fit in.

UI refinements

In the closing stages of Firefox 4 there were a lot of small polish issues identified. We knocked down a lot of them but there are still plenty of tweaks that we’re going to be bringing to the UI to make sure it is as usable as we can make it. It isn’t really worth applying a fixed schedule to these as they are small one-off fixes that can just come as and when ready. We already have some that are basically ready to land.

Incremental API fixes

There are lots of things that I’d like to see done for the platform and APIs that developers use. Again there is no fixed schedule for these but the main things on my mind are changes that make developing restartless add-ons or extending the add-ons manager easier. I’ve already started work on a couple of these and expect to get them landed after the tree reopens.

Larger projects

There are of course also larger projects that will require more resources. I’ve started writing up short project pages for each of these. Each project should start with the people working on it coming to agreement on what the actual goals should be and an implementation plan so until the projects are actually picked up the project pages are mostly made up. I’ve put down a rough idea of when a project might make it into a Firefox release. These are just guesses, if the project turns out to be larger than expected they’ll come later, if people volunteer to work on projects or if the product drivers decide they are a priority then they could come sooner.

• Warning users when third-party applications install add-ons into Firefox (Firefox 5)
• Allowing users to uninstall add-ons installed by third-party applications (Firefox 5)
• Improvements to the compatibility checking UI after application upgrades (Firefox 5)
• Adding some kind of metric about add-on’s resource usage (Firefox 5/6)
• Including search engines in the add-ons manager (Firefox 5/6)
• Streamlining the installation UI (Firefox 6)
• Improved theme UI and allowing using personas at the same time as custom themes (Firefox 6)
• Embedding add-on preferences into the add-ons manager (Firefox 6/7)
• Managing plugin updates in the add-ons manager (Firefox 7)
• Improved plugin discovery (Firefox 7)

You can see that assuming a quarterly release schedule we have enough projects here to last till the end of 2011 at least, quite likely longer. Over that sort of time I guess it’s almost guaranteed that priorities and plans will change in the meantime but hopefully this gives an idea of what our thinking is at the moment. Expect to see progress on some of the things at the top of the list very soon!

When you’re developing an extension it’s almost certain that you’re going to want to interact with the main browser window in some way. With the XUL based extensions this was a pretty simple process, normally your extension would be overlaying browser.xul and so any script you add there would automatically run for every new browser window and would have direct access to it (in fact this sometimes confused new developers who assumed their script would run just once and be globally available, not for every browser window opened).

In restartless extensions things are different. You have one script that runs when your extension is started and runs in its own sandbox so you have to take steps to get access to windows. Complicating matters your extension may be started when Firefox starts up and there are no browser windows or after when some already exist. The best way to go is when your extension is started to look for any existing browser windows and make your necessary modifications and then wait for new windows to open and modify those too.

Since this is such a common case (and vlad was trying to figure it out on IRC) I put together a very simple bootstrap script that just does this. It calls the setupBrowserUI function for every window that needs to be initialised by your extension and tearDownBrowserUI for every window that must be cleaned up when your extension is being disabled. This code isn’t the only way to do all this but it is fairly straightforward and works, feel free to take it and modify it for use in your own extensions. I’ll probably get it put onto MDC too.

const Cc = Components.classes;
const Ci = Components.interfaces;

var WindowListener = {
  setupBrowserUI: function(window) {
    let document = window.document;

    // Take any steps to add UI or anything to the browser window
    // document.getElementById() etc. will work here
  },

  tearDownBrowserUI: function(window) {
    let document = window.document;

    // Take any steps to remove UI or anything from the browser window
    // document.getElementById() etc. will work here
  },

  // nsIWindowMediatorListener functions
  onOpenWindow: function(xulWindow) {
    // A new window has opened
    let domWindow = xulWindow.QueryInterface(Ci.nsIInterfaceRequestor)
                             .getInterface(Ci.nsIDOMWindowInternal);

    // Wait for it to finish loading
    domWindow.addEventListener("load", function listener() {
      domWindow.removeEventListener("load", listener, false);

      // If this is a browser window then setup its UI
      if (domWindow.document.documentElement.getAttribute("windowtype") == "navigator:browser")
        WindowListener.setupBrowserUI(domWindow);
    }, false);
  },

  onCloseWindow: function(xulWindow) {
  },

  onWindowTitleChange: function(xulWindow, newTitle) {
  }
};

function startup(data, reason) {
  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].
           getService(Ci.nsIWindowMediator);

  // Get the list of browser windows already open
  let windows = wm.getEnumerator("navigator:browser");
  while (windows.hasMoreElements()) {
    let domWindow = windows.getNext().QueryInterface(Ci.nsIDOMWindow);

    WindowListener.setupBrowserUI(domWindow);
  }

  // Wait for any new browser windows to open
  wm.addListener(WindowListener);
}

function shutdown(data, reason) {
  // When the application is shutting down we normally don't have to clean
  // up any UI changes made
  if (reason == APP_SHUTDOWN)
    return;

  let wm = Cc["@mozilla.org/appshell/window-mediator;1"].
           getService(Ci.nsIWindowMediator);

  // Get the list of browser windows already open
  let windows = wm.getEnumerator("navigator:browser");
  while (windows.hasMoreElements()) {
    let domWindow = windows.getNext().QueryInterface(Ci.nsIDOMWindow);

    WindowListener.tearDownBrowserUI(domWindow);
  }

  // Stop listening for any new browser windows to open
  wm.removeListener(WindowListener);
}