Nightmare is a popular browser automation library specifically designed with ease of use in mind. A typical Nightmare script chains together semantically named user actions like goto and click to perform any given task, resulting in simple and readable code. These actions of course include a few methods for waiting on the page to fully load: you can wait for a selector to become available, for all static resources to load, or simply wait for a fixed amount of time. However, Nightmare does not have a built-in action for waiting until the network becomes idle, which can be a useful feature when you need even dynamically loaded resources to be available for a test.

In this post, I’ll write a waitUntilNetworkIdle Nightmare action which waits until there are no processed responses for a given amount of time—500 ms by default—before allowing the Nightmare chain to continue. After showing you the action and discussing some implementation details, I’ll write a test for the custom waitUntilNetworkIdle action. You should be able to easily modify this test to use the custom action for your own purposes.

Implementation of the waitUntilNetworkIdle Action

Nightmare makes it possible to define custom actions with the appropriately named Nightmare.action() call. The official documentation links to a few examples of this method in action, although you may need to look at the method’s implementation and learn about some of Nightmare’s internals to make the most out of it. Let’s first take a look at the new action as a whole, and then explain some of the details of how it works. By the way, the code described in this article is available in its entirety from our article materials GitHub repository, if you’d rather play with it yourself.

const Nightmare = require('nightmare');

Nightmare.action('waitUntilNetworkIdle',
  // The first callback defines the action on Electron's end,
  // making some internal objects available.
  function (name, options, parent, win, renderer, done) {

    // `parent` is Electron's reference to the object that
    // passes messages between Electron and Nightmare.
    parent.respondTo('waitUntilNetworkIdle', (waitTime, done) => {
      let lastRequestTime = Date.now();

      // win.webContents allows us to control the internal
      // Electron BrowserWindow instance.
      win.webContents.on('did-get-response-details', () => {
        lastRequestTime = Date.now();
      });

      const check = () => {
        const now = Date.now();
        const elapsedTime = now - lastRequestTime;
        if (elapsedTime >= waitTime) {
          done(); // Complete the action.
        } else {
          setTimeout(check, waitTime - elapsedTime);
        }
      }
      setTimeout(check, waitTime);
    });

    done(); // Complete the action's *creation*.
  },
  // The second callback runs on Nightmare's end and determines
  // the action's interface.
  function (waitTime, done) {
    // This is necessary because the action will only work if
    // action arguments are specified before `done`, and because
    // we wish to support calls without arguments.
    if (!done) {
      done = waitTime;
      waitTime = 500;
    }

    // `this.child` is Nightmare's reference to the object that
    // passes messages between Electron and Nightmare.
    this.child.call('waitUntilNetworkIdle', waitTime, done);
  });

module.exports = Nightmare;

The first thing you need to know is that Nightmare drives an underlying Electron instance, itself using Chromium. Communication between Nightmare and Electron is done through an IPC instance associated to the running process, and referred to as this.child from Nightmare and parent from Electron.

The action above uses two callbacks to coordinate Nightmare and Electron. The second, shorter callback is in charge of correctly setting arguments and notifying Electron that a waitUntilNetworkIdle action is initiated. The first callback is used to create an action on Electron’s end, so we need to register a separate handler to actually perform the action in the browser when called upon. This is where the callback to parent.respondTo() comes into play:

• When a waitUntilNetworkIdle action is triggered, we keep track of the last response time in

  let lastRequestTime = Date.now();
  

• We take advantage of Electron’s built-in events and update this variable whenever a did-get-response-details event is fired in the browser:

  win.webContents.on('did-get-response-details', () => {
    lastRequestTime = Date.now();
  });
  

• The remainder of the callback is a game of using timeouts to check whether there’s been no requests within the given waitTime. Here we use waitTime - elapsedTime instead of simply waitTime to schedule the next check, because an older timeout may complete after the last response came in.

Writing a Test for the waitUntilNetworkIdle Action

Let’s make sure that the custom action works by writing a test for it! I’ve created a demo page for just this purpose. A script on the page makes a sequence of three asynchronous requests, each one starting within 250 ms of the previous one. When the third request resolves, it updates the body of the page to read “All three requests received.”

The test will be a straightforward mix of Nightmare and Mocha. To get started, create the project directory and install the necessary requirements:

mkdir nightmare-network-idle
cd nightmare-network-idle
yarn add nightmare mocha

Save the customized Nightmare code from the previous section in a file called my-nightmare.js. Then, place the following into test.js.

const Nightmare = require('../my-nightmare.js');
const assert = require('assert');

describe('waitUntilNetworkIdle', function() {
  const waitTimes = [500, 1500, 5000];
  let startTime;
  waitTimes.forEach(function(waitTime) {
    it(`should wait for at least ${waitTime} ms after the last response`,
      function(done) {
        this.timeout(20000);

        const nightmare = new Nightmare({ show: true });
        startTime = Date.now();

        nightmare
          .on('did-get-response-details', () => {
            startTime = Date.now();
          })
          .goto('https://intoli.com/blog/nightmare-network-idle/demo.html')
          .waitUntilNetworkIdle(waitTime)
          .evaluate(() => {
            const body = document.querySelector('body');
            return body.innerText;
          })
          .end()
          .then((result) => {
            const elapsedTime = Date.now() - startTime;

            // Verify the requests completed as expected.
            assert.equal(result, 'All three requests received.');

            // Verify that the action caused Nightmare to wait long enough.
            assert(elapsedTime >= waitTime, 'Wait period too short');

            done();
          })
          .catch(done)
    });
  });
});

This test will navigate to the demo page, wait for a set amount of time after the last request, and finally verify that (1) the page’s body contains the expected text, and (2) the expected amount of time has passed. You can run the test from the project’s root directory with

./node_modules/.bin/mocha

which should print out something like the following in the terminal.

  waitUntilNetworkIdle
    ✓ should wait for at least 500 ms after the last response (1974ms)
    ✓ should wait for at least 1500 ms after the last response (2807ms)
    ✓ should wait for at least 5000 ms after the last response (6422ms)


  3 passing (11s)

Feel free to modify this test and use the waitUntilNetworkIdle action for your own purposes!

Conclusion

If you got here from a search engine looking for how to write custom Nightmare actions or the waitUntilNetworkIdle action in particular, I hope you found what you’re looking for! Our blog is full of informative automation and web scraping articles, and you can also subscribe to our mailing list. Finally, if you have an automation project in mind that you need some help on, please don’t hesitate to get in touch.