WebExtensions are a frequently underappreciated tool for the purposes of web scraping and browser automation. They provide an easy way to access an extremely powerful API that’s cross browser compatible out of the box, and that API provides functionality that extends far beyond that of more specialized automation APIs like the Chrome DevTools Protocol or Firefox’s Marionnette. For example, the WebExtensions API provides a mechanism for containerizing individual tabs–Selenium and Puppeteer can’t do that!
Intoli Smart Proxies
Want to use the smartest web scraping proxies available?
Get started now and find out why Intoli is the best in the business!
Introduction Lead generation is a top priority for most successful companies and helping businesses find potential clients is a big part of what we do here at Intoli. Today, we’re pleased to announce a new open source marketing tool that makes it possible to find contact emails for any web domain with a single click. It’s called Email Spy and you can get the source on GitHub or install it directly as a Chrome extension or a Firefox addon.
The WebExtensions API In 2015, Mozilla announced that they would be deprecating XPCOM and XUL based addons in favor of their new WebExtensions API based on the Google Chrome Extension API. There were some vocal critics of this shift because it meant that some existing add-ons would be discontinued, but this was tremendously positive news for add-on and extension developers. Writing cross-browser extensions had previously been an absolutely miserable experience, and many developers understandably chose to only target Chrome due to its market share and relatively pleasant API.
Running Google Chrome with an extension installed is quite simple because Chrome supports a --load-extension command-line argument for exactly this purpose. This can be specified before launching Chrome with Selenium by creating a ChromeOptions instance and calling add_argument(). from selenium import webdriver from selenium.common.exceptions import NoSuchElementException # Configure the necessary command-line option. options = webdriver.ChromeOptions() options.add_argument('--load-extension=path/to/the/extension') # Initalize the driver with the appropriate options. driver = webdriver.Chrome(chrome_options=options) The above code will setup a Selenium driver for Chrome with the extension located at path/to/extension preinstalled.