Skip to content

Latest commit

 

History

History
487 lines (353 loc) · 25.7 KB

NODELOADLIB.md

File metadata and controls

487 lines (353 loc) · 25.7 KB

OVERVIEW

nodeloadlib is a node.js library containing building blocks to programmatically create load tests for HTTP services. The components are:

  • High-level load testing interface
  • Distributed testing
  • A scheduler which executes functions at a given rate
  • Event-based loops
  • Statistics classes
  • HTTP-specific monitors
  • Web-based reports

QUICKSTART

Add require(./dist/nodeloadlib.js) and call runTest() or addTest()/startTests():

// Add to example.js:
require('./dist/nodeloadlib');

runTest({
    name: "Read",
    host: 'localhost',
    port: 8080,
    numClients: 20,
    timeLimit: 600,
    successCodes: [200],
    targetRps: 200,
    requestGenerator: function(client) {
        var url = '/data/object-' + Math.floor(Math.random()*10000);
        return traceableRequest(client, 'GET', url, { 'host': 'localhost' });
    }
});

This test will hit localhost:8080 with 20 concurrent connections for 10 minutes. During the test, a web server is started on localhost:8000, which displays requests per second and latency statistics. Non-200 responses will be logged to results-{timestamp}-err.log, statistics are regularly logged to results-{timestamp}-stats.log, and the summary page found at localhost:8000 is also written to results-{timestamp}-summary.html.

$ node example.js         ## while running, browse to http://localhost:8000 to track the test
Serving progress report on port 8000.
Opening log files.
......done.

Finishing...
Closed log files.
Shutdown report server.

Check out examples/nodeloadlib-ex.js for a example of a full read+write test.

CONFIGURATION

Define these global variables before including nodeloadlib to control its behavior:

  • QUIET: set to true to disable all console output (default is false)
  • HTTP_SERVER_PORT: set to the port to start the HTTP server on (default is 8000)
  • DISABLE_HTTP_SERVER: set to true to not start the HTTP server (default is false)
  • DISABLE_LOGS: set to true to not create the log or HTML summary files (default is false)
  • TEST_CONFIG: can be "long" or "short" which changes the test reporting interval to more appropriate settings (default is "short")

COMPONENTS

Load Test Functions

High-level functions useful for quickly building up complex load tests. See api.js.

Functions:

  • runTest(spec, callback, stayAliveAfterDone): Run a single test and call callback (see Test Definition below).
  • addTest(spec): Add a test to be run on startTests(). Tests are run concurrently.
  • addRamp(rampSpec): Gradually ramp up the load generated by a test (see Ramp Definition below).
  • startTests(callback, stayAliveAfterDone): Run tests added by addTest() and addRamp() and call callback.
  • traceableRequest(...): Used instead of built-in node.js http.Client.request() to allows proper tracking of unique URLs.

Usage:

A "test" represents requests being sent at a fixed rate over concurrent connections. Tests are run by calling runTest() or calling addTest() followed by startTests(). The parameters defining a test are detailed in Test Definition section. Issue requests using one of three methods:

  • Define method, path, and requestData, leaving requestGenerator and requestLoop as null. If method is 'PUT' or 'POST', nodeloadlib will send requestData in the request body.

  • Set requestGenerator to a function(http.Client) -> http.ClientRequest. Requests returned by this function are executed by nodeloadlib. For example, you can GET random URLs using a requestGenerator:

      addTest({
      requestGenerator: function(client) {
          return traceableRequest(client, 'GET', '/resource-' + Math.floor(Math.random()*10000));
      }
  });

  • Set requestLoop to a function(loopFun, http.Client) which calls loopFun({req: http.ClientRequest, res: http.ClientResponse}) after each request completes. This is the most flexibility, but the function must be sure to call loopFun(). For example, issue PUT requests with proper If-Match headers using a requestLoop:

      addTest({
      requestLoop: function(loopFun, client) {
          var req = traceableRequest(client, 'GET', '/resource');
          req.on('response', function(response) {
              if (response.statusCode != 200 && response.statusCode != 404) {
                  loopFun({req: req, res: response});
              } else {
                  var headers = { };
                  if (response.headers['etag'] != null)
                      headers['if-match'] = response.headers['etag'];
                  req = traceableRequest(client, 'PUT', '/resource', headers, "new value");
                  req.on('response', function(response) {
                      loopFun({req: req, res: response});
                  });
                  req.close();
              }
          });
          req.close();
      }
  });

A "ramp" increases the load of a particular test over some period of time. Schedule a ramp after scheduling a test by calling addRamp():

var test1 = addTest({
    targetRps: 100,
    requestGenerator: function(client) {
        return traceableRequest(client, 'GET', '/resource-' + Math.floor(Math.random()*10000));
    }
});

// Add 100 requests / second using 10 concurrent connections to test1 between minutes 1 and 2
addRamp({
    test: test1,
    numberOfSteps: 10,
    timeLimit: 60,
    rpsPerStep: 10,
    clientsPerStep: 1,
    delay: 60
});

Start all the tests by calling startTests(...). The script terminates 3 seconds after the tests complete unless the parameter stayAliveAfterDone==true.

Check out examples/nodeloadlib-ex.js for an example of a full read+write test.

Test Definition: The following object defines the parameters and defaults for a test, which is used by addTest() or runTest():

var TEST_DEFAULTS = {
    name: 'Debug test',                     // A descriptive name for the test

    host: 'localhost',                      // host and port specify where to connect
    port: 8080,                             //
    requestGenerator: null,                 // Specify one of: requestGenerator, requestLoop, or (method, path, requestData)
    requestLoop: null,                      //   - A requestGenerator is a function that takes a http.Client param
    method: 'GET',                          //     and returns a http.ClientRequest.
    path: '/',                              //   - A requestLoop is a function that takes two params (loopFun, http.Client).
    requestData: null,                      //     It should call loopFun({req: http.ClientRequest, res: http.ClientResponse})
                                            //     after each operation to schedule the next iteration of requestLoop.
                                            //   - (method, path, requestData) specify a single URL to test

    numClients: 10,                         // Maximum number of concurrent executions of request loop
    numRequests: Infinity,                  // Maximum number of iterations of request loop
    timeLimit: 120,                         // Maximum duration of test in seconds
    targetRps: Infinity,                    // Number of times per second to execute request loop
    delay: 0,                               // Seconds before starting test

    successCodes: null,                     // List of success HTTP response codes. Failures are logged to the error log.
    stats: ['latency', 'result-codes'],     // Specify list of: latency, result-codes, uniques, concurrency. Note that "uniques"
                                            // only shows up in summary report and requests must be made with traceableRequest().
                                            // Not doing so will result in reporting only 2 uniques.
    latencyConf: {percentiles: [.95,.99]},  // Set latencyConf.percentiles to percentiles to report for the 'latency' stat
    reportInterval: 2,                      // Seconds between each progress report
    reportFun: null,                        // Function called each reportInterval that takes a param, stats, which is a map of
                                            // { 'latency': Reportable(Histogram), 'result-codes': Reportable(ResultsCounter},
                                            // 'uniques': Reportable(Uniques), 'concurrency': Reportable(Peak) }
}

Ramp Definition: The following object defines the parameters and defaults for a ramp, which is used by addRamp():

var RAMP_DEFAULTS = {
    test: null,                         // The test to ramp up, returned from from addTest()
    numberOfSteps: 10,                  // Number of steps in ramp
    timeLimit: 10,                      // The total number of seconds to ramp up
    rpsPerStep: 10,                     // The rps to add to the test at each step
    clientsPerStep: 1,                  // The number of connections to add to the test at each step.
    delay: 0                            // Number of seconds to wait before ramping up. 
}

Distributed Testing

Functions to distributing tests across multiple slave nodeload instances. See remote.js

Functions:

  • remoteTest(spec): Return a test to be scheduled with remoteStart(...) (spec uses same format as addTest(spec)).
  • remoteStart(master, slaves, tests, callback, stayAliveAfterDone): Run tests on specified slaves.
  • remoteStartFile(master, slaves, filename, callback, stayAliveAfterDone): Execute a .js file on specified slaves.

Usage:

First, simply start nodeloadlib.js on each slave instances.

$ node dist/nodeloadlib.js       # Run on each slave machine

Then, create tests in using remoteTest(spec) with the same spec fields in the Test Definition section above. Pass the created tests as a list to remoteStart(...) to execute them on slave nodeload instances. master must be the "host:port" of the nodeload which is executing remoteStart(...). It will receive and aggregates statistics from the slaves, so the address should be reachable by the slaves. Or, use master=null to disable reports from the slaves.

// This script must be run on master:8000, which will aggregate results. Each slave 
// will GET http://internal-service:8080/ at 100 rps.
var t1 = remoteTest({
    name: "Distributed test",
    host: 'internal-service',
    port: 8080,
    timeLimit: 20,
    targetRps: 100
});
remoteStart('master:8000', ['slave1:8000', 'slave2:8000', 'slave3:8000'], [t1]);

Alternatively, an existing nodeload load test script file can be used:

// The file /path/to/load-test.js should contain valid javascript and can use any nodeloadlib functions
remoteStartFile('master:8000', ['slave1:8000', 'slave2:8000', 'slave3:8000'], '/path/to/load-test.js');

When the remote tests complete, the master instance will call the callback parameter if non-null. It then automatically terminates after 3 seconds unless the parameter stayAliveAfterDone==true.

Function Scheduler

The SCHEDULER object allows a function to be called at a desired rate and concurrency level.

Functions:

  • SCHEDULER.schedule(spec): Schedule a function to be executed (see the Schedule Definition below)
  • SCHEDULER.startAll(callback): Start running all the scheduled functions and execute callback when they complete
  • SCHEDULER.startSchedule(callback): Start a single scheduled function and execute callback when it completes
  • funLoop(fun): Wrap functions that do not perform IO so they can be used with SCHEDULER

Usage:

Call SCHEDULER.schedule(spec) to add a job. spec.fun must be a function(loopFun, args) and call loopFun(results) when it completes. Call SCHEDULER.startAll() to start running all scheduled jobs.

If spec.argGenerator is non-null, it is called spec.concurrency times on startup. One return value is passed as the second parameter to each concurrent execution of spec.fun. If null, the value of spec.args is passed to all executions of spec.fun instead.

A scheduled job finishes after its target duration or it has been called the maximum number of times. SCHEDULER stops all running jobs once all monitored jobs finish. For example, 1 monitored job is scheduled for 5 seconds, and 2 unmonitored jobs are scheduled with no time limits. SCHEDULER will start all 3 jobs when SCHEDULER.startAll() is called, and stop all 3 jobs 5 seconds later. Unmonitored jobs are useful for running side processes such as statistics gathering and reporting.

Example:

var t = 1;
SCHEDULER.schedule({
    fun: funLoop(function(i) { sys.puts("Thread " + i) }),
    argGenerator: function() { return t++; },
    concurrency: 5,
    rps: 10,
    duration: 10
});
SCHEDULER.startAll(function() { sys.puts("Done.") });

Job Definition: The following object defines the parameters and defaults for a job run by SCHEDULER:

var JOB_DEFAULTS = {
    fun: null,                  // A function to execute which accepts the parameters (loopFun, args).
                                // The value of args is the return value of argGenerator() or the args
                                // parameter if argGenerator is null. The function must call 
                                // loopFun(results) when it completes.
    argGenerator: null,         // A function which is called once when the job is started. The return
                                // value is passed to fun as the "args" parameter. This is useful when
                                // concurrency > 1, and each "thread" should have its own args.
    args: null,                 // If argGenerator is NOT specified, then this is passed to the fun as "args".
    concurrency: 1,             // Number of concurrent calls of fun()
    rps: Infinity,              // Target number of time per second to call fun()
    duration: Infinity,         // Maximum duration of this job in seconds
    numberOfTimes: Infinity,    // Maximum number of times to call fun()
    delay: 0,                   // Seconds to wait before calling fun() for the first time
    monitored: true             // Does this job need to finish in order for SCHEDULER.startAll() to end?
};

Event-based loops

The ConditionalLoop class provides a generic way to write a loop where each iteration is scheduled using process.nextTick(). This allows many long running "loops" to be executed concurrently by node.js.

Functions:

  • ConditionalLoop(fun, args, conditions, delay): Defines a loop (see Loop Definition below)
  • ConditionalLoop.start(callback): Starts executing and call callback on termination
  • ConditionalLoop.stop(): Terminate the loop
  • timeLimit(seconds), maxExecutions(numberOfTimes): useful ConditionalLoop conditions
  • rpsLoop(rps, fun): Wrap a function(loopFun, args) so ConditionalLoop calls it a set rate
  • funLoop(fun): Wrap a non-IO performing function(args) so it can be used with a ConditionalLoop

Usage:

Create a ConditionalLoop instance and call ConditionalLoop.start() to execute the loop. A function given to ConditionalLoop must be a function(loopFun, args) which ends by calling loopFun().

The conditions parameter is a list of functions. When any function returns false, the loop terminates. For example, the functions timeLimit(seconds) and maxExecutions(numberOfTimes) are conditions that limit the duration and number of iterations of a loop respectively.

The loop also terminates if ConditionalLoop.stop() is called.

Example:

var fun = function(loopFun, startTime) {
    sys.puts("It's been " + (new Date() - startTime) / 1000 + " seconds");
    loopFun();
};
var stopOnFriday = function() {
    return (new Date()).getDay() < 5;
}
var loop = new ConditionalLoop(rpsLoop(1, fun), new Date(), [stopOnFriday, timeLimit(604800 /*1 week*/)], 1);
loop.start(function() { sys.puts("It's Friday!") });

Loop Definition:

The ConditionalLoop constructor arguments are:

fun: Function that takes parameters (loopFun, args) and calls loopFun() after each iteration
args: The args parameter to pass to fun
conditions: A list of functions representing termination conditions. Terminate when any function returns `false`.
delay: Seconds to wait before starting the first iteration

Statistics

Implementations of various statistics.

Classes:

  • Histogram(numBuckets): A histogram of integers. If most of the items are between 0 and numBuckets, calculating percentiles and stddev is fast.
  • Accumulator: Calculates the sum of the numbers put in.
  • ResultsCounter: Tracks results which are be limited to a small set of possible choices. Tracks the total number of results, number of results by value, and results added per second.
  • Uniques: Tracks the number of unique items added.
  • Peak: Tracks the max of the numbers put in.
  • Rate: Tracks the rate at which items are added.
  • LogFile: Outputs to a file on disk.
  • NullLog: Ignores all items put in.
  • Reportable: Wraps any other statistic to store an interval and cumulative version of it.

Functions:

  • randomString(length): Returns a random string of ASCII characters between 32 and 126 of the requested length.
  • nextGaussian(mean, stddev): Returns a normally distributed number using the provided mean and standard deviation.
  • nextPareto(min, max, shape): Returns a Pareto distributed number between min and max inclusive using the provided shape.
  • roundRobin(list): Returns a copy of the list with a get() method. get() returns list entries round robin.

Usage:

All of the statistics classes support the methods:

  • .length: The total number of items put() into this object.
  • put(item): Include an item in the statistic.
  • get(): Get a specific value from the object, which varies depending on the object.
  • clear(): Clear out all items.
  • summary(): Get a object containing a summary of the object, which varies depending on the object. The fields returned are used to generate the trends of the HTML report graphs.

In addition, these other methods are supported:

  • Histogram.mean(): Calculate the mean of the numbers in the histogram.
  • Histogram.percentile(percentile): Calculate the given percentile, between 0 and 1, of the numbers in the histogram.
  • Histogram.stddev(): Standard deviation of the numbers in the histogram.
  • LogFile.open(): Open the file.
  • LogFile.close(): Close the file.
  • Reportable.next(): clear out the interval statistic for the next window.

Refer to the Statistics section near line 910 of nodeloadlib.js for the return value of the get() and summary() functions for the different classes.

HTTP-specific Monitors

A collection of wrappers for requestLoop functions that record statistics for HTTP requests. These functions can be run scheduled with SCHEDULER or run with a ConditionalLoop.

Functions:

  • monitorLatenciesLoop(latencies, fun): Call fun() and put the execution duration in latencies, which should be a Histogram.
  • monitorResultsLoop(results, fun): Call fun() and put the HTTP response code in results, which should be a ResultsCounter.
  • monitorByteReceivedLoop(bytesReceived, fun): Call fun() and put the number of bytes received in bytesReceived, usually an Accumulator.
  • monitorConcurrencyLoop(concurrency, fun): Call fun() and put the number of "threads" currently executing it into concurrency, usually a Peak.
  • monitorRateLoop(rate, fun): Call fun() and notify rate, which should be a Rate, that it was called.
  • monitorHttpFailuresLoop(successCodes, fun, log): Call fun() and put the HTTP request and response into log, which should be a LogFile, for every request that does not return an HTTP status code included in the list successCodes.
  • monitorUniqueUrlsLoop(uniqs, fun): Call fun() and put the HTTP request path into uniqs, which should be a Uniques.
  • loopWrapper(fun, start, finish): Create a custom loop wrapper by specifying a functions to execute before and after calling fun().

Usage:

All of these wrappers return a function(loopFun, args) which can be used by SCHEDULER and ConditionalLoop. The underlying function should have the same signature and execute an HTTP request. It must call loopFun({req: http.ClientRequest, res: http.ClientResponse}) when it completes the request.

Example:

// Issue GET requests to random objects at localhost:8080/data/obj-{0-1000} for 1 minute and
// track the number of unique URLs
var uniq = new Reportable(Uniques, 'Uniques');
var loop = monitorUniqueUrlsLoop(uniq, function(loopFun, client) {
    var req = traceableRequest(client, 'GET', '/data/obj-' + Math.floor(Math.random()*1000));
    req.on('response', function(response) {
        sys.puts('ah')
        loopFun({req: req, res: response});
    });
    req.close();
});
SCHEDULER.schedule({
    fun: loop,
    args: http.createClient(8080, 'localhost'),
    duration: 60
}).start(function() {
    sys.puts(JSON.stringify(uniq.summary()));
});

Web-based Reports

Functions for manipulating the report that is available during the test at http://localhost:8000/ and that is written to results-{timestamp}-summary.html. Note that the directory flot/ must be present for the charts to display properly.

Functions:

  • HTTP_REPORT.setText(text): Sets the text at the top of the report.
  • HTTP_REPORT.puts(text): Appends the line text to the text at the top of the report.
  • HTTP_REPORT.addChart(name): Adds a chart with the title name to the report and returns a Chart object. See Chart.put(data) below.
  • HTTP_REPORT.removeChart(name): Removes the chart with title name from the report.
  • HTTP_REPORT.clear(): Clears the text and removes all charts from the report.
  • Chart.put(data): Add the data, which is a map of { 'trend-1': value, 'trend-2': value, ... }, to the chart, which tracks the values for each trend over time.
  • SUMMARY_HTML_REFRESH_PERIOD: Change this global variable to set the auto-refresh period of the HTML page in milliseconds.

Usage:

An HTTP server is started on port HTTP_SERVER_PORT, which defaults to 8000, unless DISABLE_HTTP_SERVER=true when nodeloadlib is included. Likewise, the file results-{timestamp}-summary.html is written to the current directory when the test ends unless DISABLE_LOGS=true when nodeloadlib is included.

A chart is automatically added to HTTP_REPORT for each statistic requested in a test created by addTest() or runTest(), and for each Reportable object created with parameter addToHttpReport=true. Call HTTP_REPORT.addChart() to add additional charts to the report. Add data points to it manually by calling put() on the returned object.

The progress page automatically issues an AJAX request to refresh the text and chart data every SUMMARY_HTML_REFRESH_PERIOD milliseconds.

TIPS AND TRICKS

Some handy features worth mentioning.

  1. Examine and add to stats to the HTML page:

    addTest().stats and runTest().stats are maps:

     { 'latency': Reportable(Histogram), 
   'result-codes': Reportable(ResultsCounter},
   'uniques': Reportable(Uniques), 
   'concurrency': Reportable(Peak) }

    Put Reportable instances to this map to have it automatically updated each reporting interval. Create the Reportable with addToHttpReport=true to add a chart for it on the HTML status page. Or, set Report.disableIntervalReporting=true to only update Reportable.cumulative and not Reportable.interval each reporting interval.

  2. Post-process statistics:

    Use a startTests() callback to examine the final statistics in test.stats[name].cumulative at test completion.

     // GET random URLs of the form localhost:8080/data/object-#### for 10 seconds, then
 // print out all the URLs that were hit.
 var t = addTest({
     timeLimit: 10,
     targetRps: 10,
     stats: ['uniques'],
     requestGenerator: function(client) {
         return traceableRequest(client, 'GET', '/data/object-' + Math.floor(Math.random()*100));;
     }
 });
 function printAllUrls() {
     qputs(JSON.stringify(t.stats['uniques'].cumulative));
 }
 startTests(printAllUrls);

  3. Out-of-the-box file server:

    Just start nodeloadlib.js and it will serve files in the current directory.

     $ node dist/nodeloadlib.js
 $ curl -i localhost:8000/nodeloadlib.js     # executed in a separate terminal
 HTTP/1.1 200 OK
 Content-Length: 50763
 Connection: keep-alive
 
 var sys = require('sys');
 var http = require('http');
 ...

  4. Run arbitrary Javascript:

    POST any valid Javascript to /remote to have it eval()'d.

     $ node dist/nodeloadlib.js
 Serving progress report on port 8000.
 Opening log files.
 Starting remote test:
 sys.puts("hello!")
 hello!
 
 $ curl -i -d 'sys.puts("hello!")' localhost:8000/remote     # executed in a separate terminal