The WebExtensions API

In 2015, Mozilla announced that they would be deprecating XPCOM and XUL based addons in favor of their new WebExtensions API based on the Google Chrome Extension API. There were some vocal critics of this shift because it meant that some existing add-ons would be discontinued, but this was tremendously positive news for add-on and extension developers. Writing cross-browser extensions had previously been an absolutely miserable experience, and many developers understandably chose to only target Chrome due to its market share and relatively pleasant API. The Mozilla announcement was the first glimpse of a future where there would be a single standardized API for writing extensions that could run in any browser. That was an extremely exciting idea and one that would ultimately prove positive for both developers and users.

Fast forward a couple years and we’re living that future. You can now write extensions that pretty much run out of the box in Chrome, Edge, Firefox, Firefox for Android, and Opera. Unfortunately, while the future may already be here–it’s not very evenly distributed. The browser support is still a bit mixed, and some common the tooling only supports deprecated Firefox add-on formats.

One such example is Selenium, one of the handiest browser automation frameworks out there for UI testing and web scraping. Selenium might not be the new kid on the block these days, but it has some of the best cross-browser support out there. If you want to run or test a cross-browser WebExtension, then using a cross-browser automation framework is a must. That’s why it’s a bit disappointing that Selenium still has only partial support for running Firefox with WebExtensions years after Mozilla announced the transition.

Support was added very recently added to the Python client for installing addons using GeckoDriver’s addon manager, but this only supports installing packaged extensions and unsigned extensions can only be installed for a single session. The more robust FirefoxProfile.add_extension() method would side-step those issues, but it only supports the now-deprecated extension formats.

In this article, I’ll walk through the process of hacking support for Firefox WebExtensions into the Selenium Python client’s FirefoxProfile class. I’ve also put my money where my mouth is and submitted these changes as a pull request on the Selenium repository. This means that hopefully the workaround described in this article will eventually not be necessary. I’ll be sure to update this once that’s the case.

Making it Work

The Test Extension

Let’s start off by making a really simple extension that we can use to test whether it is being successfully installed or not before launching a browser. To do this, we’ll first make a directory

mkdir extension

and then place two files in it. The first is the extension manifest called extension/manifest.json with the following contents.

{
  "manifest_version": 2,
  "name": "Firefox Extensions With Selenium",
  "version": "1.0.0",
  "applications": {
    "gecko": {
      "id": "firefox-extensions-with-selenium@intoli.com"
    }
  },
  "content_scripts": [
    {
      "matches": ["*://*/*"],
      "js": ["content.js"],
      "run_at": "document_start"
    }
  ]
}

This defines the minimal required boilerplate and specifies that a file located at extension/content.js should be run in every page that the browser navigates to. If we create this file with these contents

// Wait for the DOM to completely load.
document.addEventListener("DOMContentLoaded", () => {
  // Overwrite the contents of the body.
  document.body.innerHTML = '<h1 id="installed" >Successfully Installed!</h1>';
});

then it should be pretty obvious whether or not the extension is installed; every page will consist of a large header that says “Successfully Installed!”

We’ll test this out using the Firefox driver’s install_addon() method which will require us to first package the extension.

cd extension/
zip /tmp/extension.xpi *
cd ../

You can see that I put the packaged extension in the /tmp/ directory. This can be put anywhere, but the full path will need to be provided to the driver.

from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException


# Set up the driver.
firefox_binary = '/usr/bin/firefox-developer'  # Must be the developer edition!!!
driver = webdriver.Firefox(firefox_binary=firefox_binary)
extension_path = '/tmp/extension.xpi'  # Must be the full path to an XPI file!
driver.install_addon(extension_path, temporary=True)


# Check if the extension worked and log the result.
try:
    header = driver.find_element_by_id('successfully-installed')
    print('Success! :-)')
except NoSuchElementException:
    print('Failure! :-(')
finally:
    # Clean up.
    driver.quit()

Running that script will let us know it worked by printing out

Success! :-)

and we should also briefly see

appear on the screen before the browser quits at the end of the script.

This method is sufficient for many use cases, but it can be very useful to be able to install extensions to a profile persistently if they need to store data between sessions. The FirefoxProfile.add_extension() method is better suited for this purpose. Less importantly, it also conveniently allows you to load unpackaged extensions and to specify them via relative paths.

Updating FirefoxProfile

The general approach to using a persistent extension with Firefox is to specify a Firefox profile containing the already-installed extension. Behind the scenes, Selenium normally creates a new profile in a temporary directory and then specifies it as a command-line argument when launching the Firefox process. You can also create this profile manually using the FirefoxProfile class and make your own modifications.

The FirefoxProfile class includes an add_extension() method which is supposed to automatically handle installing an extension within a profile. Here’s an example of how we would launch Firefox using a custom profile with an extension installed.

from selenium import webdriver


# Setup the profile.
profile = webdriver.FirefoxProfile()
# This is the directory again.
profile.add_extension('extension')

# Launch Firefox using the profile and navigate to a page.
firefox_binary = '/usr/bin/firefox-developer'  # Must be the developer edition!!!
driver = webdriver.Firefox(profile, firefox_binary=firefox_binary)
driver.get('https://intoli.com')

# Check if the extension worked and log the result.
try:
    header = driver.find_element_by_id('successfully-installed')
    print('Success! :-)')
except NoSuchElementException:
    print('Failure! :-(')
finally:
    # Clean up.
    driver.quit()

That would be sufficient if we were using the deprecated add-on format, but it will fail with our test extension because it is a WebExtension and Selenium doesn’t support them (yet). Running the script will instead produce the following error.

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 343, in _addon_details
    with open(os.path.join(addon_path, 'install.rdf'), 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'extension/install.rdf'


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run-firefox.py", line 33, in <module>
    profile.add_extension('extension')
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 92, in add_extension
    self._install_extension(extension)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 271, in _install_extension
    addon_details = self._addon_details(addon)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_profile.py", line 349, in _addon_details
    raise AddonFormatError(str(e), sys.exc_info()[2])
selenium.webdriver.firefox.firefox_profile.AddonFormatError: ("[Errno 2] No such file or directory: 'extension/install.rdf'", <traceback object at 0x7ff625c58848>)

It’s failing here because it’s expecting there to be a Resource Description Framework (RDF) file at extension/install.rdf. This file is basically the deprecated equivalent of our manifest.json file; it contains the metadata and settings for the extension.

Let’s take a peak at the docsrting for the _addon_details() method where this error occurs.

"""
Returns a dictionary of details about the addon.
:param addon_path: path to the add-on directory or XPI
Returns::
    {'id':      u'rainbow@colors.org', # id of the addon
     'version': u'1.4',                # version of the addon
     'name':    u'Rainbow',            # name of the addon
     'unpack':  False }                # whether to unpack the addon
"""

This clearly documents what information is supposed to be extracted from the install.rdf file: the name, version, and id. If we catch the AddonFormatError that’s thrown when parsing install.rdf fails, then we can load these in from the manifest.json file instead. The easiest way to do this is to make a new class that inherits from FirefoxProfile and overrides _addon_details().

import json
import os
import sys

from selenium.webdriver.firefox.firefox_profile import AddonFormatError


class FirefoxProfileWithWebExtensionSupport(webdriver.FirefoxProfile):
    def _addon_details(self, addon_path):
        try:
            return super()._addon_details(addon_path)
        except AddonFormatError:
            try:
                with open(os.path.join(addon_path, 'manifest.json'), 'r') as f:
                    manifest = json.load(f)
                    return {
                        'id': manifest['applications']['gecko']['id'],
                        'version': manifest['version'],
                        'name': manifest['name'],
                        'unpack': False,
                    }
            except (IOError, KeyError) as e:
                raise AddonFormatError(str(e), sys.exc_info()[2])

This code will first attempt to use the existing _addon_detaults() implementation and then fall back to manifest.json if that doesn’t work. If parsing the manifest fails too for some reason, then it will raise a new AddonFormatError.

It’s not obvious that there won’t be more problems encountered elsewhere in the code, but we can test this easily by swapping in our FirefoxProfileWithWebExtensionSupport class for FirefoxProfile in the script we just ran.

import webdriver


# Setup the profile, this obviously needs to be in scope.
profile = FirefoxProfileWithWebExtensionSupport()
# This is the directory again.
profile.add_extension('extension')

# Launch Firefox using the profile and navigate to a page.
firefox_binary = '/usr/bin/firefox-developer'  # Must be the developer edition!!!
driver = webdriver.Firefox(profile, firefox_binary=firefox_binary)
driver.get('https://intoli.com')

# Check if the extension worked and log the result.
try:
    header = driver.find_element_by_id('successfully-installed')
    print('Success! :-)')
except NoSuchElementException:
    print('Failure! :-(')
finally:
    # Clean up.
    driver.quit()

Running this prints out

Success! :-)

and displays our nice success message.

Tada!

Troubleshooting

Here are a couple of common issues that might come up when attempting to run Firefox in Selenium with extensions. If it’s not working for you, then check if any of these apply to you:

• You’re using a version of Selenium that doesn’t include these additions. You can try downloading the example project so that you’re absolutely sure that you’re running the correct code.
• You’re not using the Firefox Developer Edition and are using an unsigned extension. There is some outdated information online saying that it’s sufficient to set the xpinstall.signatures.required to False, but this is not true anymore. You must either use the developer edition or use a signed extension.
• You’re missing a required key in your manifest.json file, most likely applications.gecko.id. The id is not required when adding a temporary extension manually, but it is required when installing the extension using Selenium.

Conclusion

Hopefully you found this useful if you were trying to figure out how to run Firefox with an extension pre-installed using Selenium. The fix turned out to be pretty simple, but it took a bit of digging through the code to get everything sorted out. If you’re ever stuck on any browser automation related issues and looking for consulting services, then please keep us in mind!