HTTP-LoadGen
view release on metacpan or search on metacpan
* Net::SSLeay
DESCRIPTION
This module implements a multi-process and multi-thread load generator
for HTTP. It uses Coro threads. So, in reality it does not use threads
but event-based IO.
Features
* limited support for SSL connections
* keep-alive connections
* configurable delay before and after each request
* run a list of URLs many times
* compute next URL based on the current request
* DNS cache can be preinitialized
* slow ramp up
'server' => ['Apache'],
}
RC_BODY (9)
the response body
RC_DNSCACHED (10)
boolean: has the DNS cache lookup resulted in a hit (1) or miss (0)?
RC_CONNCACHED (11)
boolean: has the has a kept-alive connection been used?
The %data hash
So, what can be specified in %data? Note, all keys here are case
sensitive.
NWorker (optional)
specifies the number of worker processes to be used. Default is 1.
RampUpStart (optional)
the number of threads to started up immediately (after the
The parameters $rc and $rq describe the previous request ($rq) and
its result ($rc).
For a description of the $rq and $new_rq format see URLList below.
Example:
InitURLs=>sub {
my $url=[qw!GET http foertsch.name 80 /-redir!,
{
keepalive=>KEEPALIVE,
headers=>[
'X-auth'=>1, # necessary to trigger 401 for that URL
], # it also shows a custom request header
}];
return sub {
my ($rc, $rq)=@_;
if( $rc->[RC_STATUS]==401 ) {
# redo with Authorization header
push @{$rq->[RQ_PARAM]->{headers}}, Authorization=>'Basic YmxhOmJsdWI=';
return $rq;
RQ_HOST == 2
RQ_PORT == 3
RQ_URI == 4
RQ_PARAM == 5
Example:
URLList=>[
[qw!GET http 109.73.51.50 80 /-redir!,
{
keepalive=>KEEPALIVE,
headers=>[
Authorization=>'Basic YmxhOmJsdWI=',
Host=>'foertsch.name',
],
}],
[qw!HUGO https www.kabatinte.net 443 /!,
{
keepalive=>KEEPALIVE,
predelay=>0.5,
prejitter=>1,
postdelay=>3,
postjitter=>1.5,
body=>'blablub',
}]
]
This "URLList" contains 2 requests, one for a server with the IP
address 109.73.51.50 and one for the host "www.kabatinte.net".
Although no "Host" header is specified in the request element one is
sent. If the request element does not contain a "Host" header one is
added automatically based on $host and $port.
You may also notice the "Content-Length" header. It is sent because
a request body is specified (the "body" item in $param).
So, what can be specified in the $param part?
keepalive
HTTP::LoadGen::Run exports 3 constants to be used as values.
"KEEPALIVE_USE" permits to use a previously kept alive
connection. "KEEPALIVE_STORE" allows to keep the connection
alive after the request. "KEEPALIVE" combines both of the above.
If you hate readability you can also use the numerical values:
KEEPALIVE_USE==1
KEEPALIVE_STORE==2
KEEPALIVE==3
predelay and prejitter
These statements define a period to wait before sending the
request. The wait is done after the request description has been
'www.kabatinte.net'=>'84.38.75.176',
'foertsch.name'=>'109.73.51.50',
},
times=>3, # run the URL list 3 times
InitURLs=>'random_start',
URLList=>do {
my $o={
keepalive=>KEEPALIVE,
qw!predelay 0.05 prejitter 0.1 postdelay 0.5 postjitter 1!,
};
[[qw!GET http foertsch.name 80 /-redir!, $o],
[qw!HUGO https www.kabatinte.net 443 /!, $o]
];
},
}
SEE ALSO
* HTTP::LoadGen::Run
config.sample view on Meta::CPAN
'www.kabatinte.net'=>'84.38.75.176',
'foertsch.name'=>'109.73.51.50',
},
times=>3, # run the URL list 3 times
InitURLs=>'random_start',
URLList=>do {
my $o={
keepalive=>KEEPALIVE,
qw!predelay 0.05 prejitter 0.1 postdelay 0.5 postjitter 1!,
};
[[qw!GET http foertsch.name 80 /-redir!, $o],
[qw!HUGO https www.kabatinte.net 443 /!, $o]
];
},
}
lib/HTTP/LoadGen.pm view on Meta::CPAN
my @h;
if( exists $el->[RQ_PARAM]->{headers} ) {
my $hdr=$el->[RQ_PARAM]->{headers};
for (my $i=0; $i<@$hdr; $i+=2) {
push @h, $hdr->[$i], $hdr->[$i+1] if exists $keep{lc $hdr->[$i]};
}
}
return ['GET', $scheme, $host, $port, $uri,
{keepalive=>KEEPALIVE, followed=>1, headers=>\@h}];
}
}
}
BEGIN {
%services=(http=>80, https=>443);
register_iterator '', default=>sub {
my $urls=options->{URLList};
my $nurls=@$urls;
lib/HTTP/LoadGen.pod view on Meta::CPAN
This module implements a multi-process and multi-thread load generator
for HTTP. It uses L<Coro> threads. So, in reality it does not
use threads but event-based IO.
=head2 Features
=over 4
=item * limited support for SSL connections
=item * keep-alive connections
=item * configurable delay before and after each request
=item * run a list of URLs many times
=item * compute next URL based on the current request
=item * DNS cache can be preinitialized
=item * slow ramp up
lib/HTTP/LoadGen.pod view on Meta::CPAN
=item RC_BODY (9)
the response body
=item RC_DNSCACHED (10)
boolean: has the DNS cache lookup resulted in a hit (C<1>) or miss (C<0>)?
=item RC_CONNCACHED (11)
boolean: has the has a kept-alive connection been used?
=back
=head3 The %data hash
So, what can be specified in C<%data>? Note, all keys here are case
sensitive.
=over 4
lib/HTTP/LoadGen.pod view on Meta::CPAN
and its result (C<$rc>).
For a description of the C<$rq> and C<$new_rq> format see
L<URLList|/URLList (either InitURLs or URLList or both must be present)> below.
Example:
InitURLs=>sub {
my $url=[qw!GET http foertsch.name 80 /-redir!,
{
keepalive=>KEEPALIVE,
headers=>[
'X-auth'=>1, # necessary to trigger 401 for that URL
], # it also shows a custom request header
}];
return sub {
my ($rc, $rq)=@_;
if( $rc->[RC_STATUS]==401 ) {
# redo with Authorization header
push @{$rq->[RQ_PARAM]->{headers}}, Authorization=>'Basic YmxhOmJsdWI=';
return $rq;
lib/HTTP/LoadGen.pod view on Meta::CPAN
RQ_HOST == 2
RQ_PORT == 3
RQ_URI == 4
RQ_PARAM == 5
Example:
URLList=>[
[qw!GET http 109.73.51.50 80 /-redir!,
{
keepalive=>KEEPALIVE,
headers=>[
Authorization=>'Basic YmxhOmJsdWI=',
Host=>'foertsch.name',
],
}],
[qw!HUGO https www.kabatinte.net 443 /!,
{
keepalive=>KEEPALIVE,
predelay=>0.5,
prejitter=>1,
postdelay=>3,
postjitter=>1.5,
body=>'blablub',
}]
]
This C<URLList> contains 2 requests, one for a server with the IP address
C<109.73.51.50> and one for the host C<www.kabatinte.net>.
lib/HTTP/LoadGen.pod view on Meta::CPAN
If the request element does not contain a C<Host> header one is added
automatically based on C<$host> and C<$port>.
You may also notice the C<Content-Length> header. It is sent because a
request body is specified (the C<body> item in C<$param>).
So, what can be specified in the C<$param> part?
=over 4
=item keepalive
L<HTTP::LoadGen::Run> exports 3 constants to be used as values.
C<KEEPALIVE_USE> permits to use a previously kept alive connection.
C<KEEPALIVE_STORE> allows to keep the connection alive after the request.
C<KEEPALIVE> combines both of the above.
If you hate readability you can also use the numerical values:
KEEPALIVE_USE==1
KEEPALIVE_STORE==2
KEEPALIVE==3
=item predelay and prejitter
lib/HTTP/LoadGen.pod view on Meta::CPAN
'www.kabatinte.net'=>'84.38.75.176',
'foertsch.name'=>'109.73.51.50',
},
times=>3, # run the URL list 3 times
InitURLs=>'random_start',
URLList=>do {
my $o={
keepalive=>KEEPALIVE,
qw!predelay 0.05 prejitter 0.1 postdelay 0.5 postjitter 1!,
};
[[qw!GET http foertsch.name 80 /-redir!, $o],
[qw!HUGO https www.kabatinte.net 443 /!, $o]
];
},
}
=head1 SEE ALSO
lib/HTTP/LoadGen/Run.pm view on Meta::CPAN
}
my %tls_cache;
sub tlscache () {\%tls_cache}
{
no warnings 'redefine';
*tlscache=\&HTTP::LoadGen::tlscache if exists $INC{'HTTP/LoadGen.pm'};
}
use constant {
KEEPALIVE_USE=>1, # use a kept alive connection if available
KEEPALIVE_STORE=>2, # keep the connection alive if possible
KEEPALIVE=>3, # USE|STORE combined
RQ_METHOD=>0, # req params see $el in run_urllist
RQ_SCHEME=>1,
RQ_HOST=>2,
RQ_PORT=>3,
RQ_URI=>4,
RQ_PARAM=>5,
RC_STATUS=>0, # indices into run_url's result
lib/HTTP/LoadGen/Run.pm view on Meta::CPAN
#D my ($lip, $lport); # only used when debugging
RESTART: {
#D warn "Restarting connection to $ip:$port\n" if $restart;
undef $restart;
undef $connh;
undef $store_time;
undef $eof;
my $key;
if( exists $param->{keepalive} and
$param->{keepalive}&KEEPALIVE_USE and
exists $conncache->{$key="$ip $port"} and
$connh=do{my $l=$conncache->{$key};
shift @$l while(@$l and !$l->[0]->[1]); # drop all unusables
shift @$l} ) {
#D ($lport, $lip)=@{$connh}[2,3];
$connh=$connh->[0];
$rc[RC_CONNCACHED]=1;
#D warn "Using kept-alive connection ".
#D $lip.':'.$lport." ==> $ip:$port\n";
$rc[RC_STARTTIME]=$rc[RC_CONNTIME]=AE::now;
config_handle $connh, $cb, \@err, \$eof, \$restart, 1;
} else {
$rc[RC_CONNCACHED]=0;
AnyEvent::Socket::tcp_connect $ip, $port, $cb, sub {
$rc[RC_STARTTIME]=AE::now;
$store_time=\$rc[RC_CONNTIME];
exists $param->{conn_timeout} ? $param->{conn_timeout} : 0;
lib/HTTP/LoadGen/Run.pm view on Meta::CPAN
#D warn "--Response Body------------------------------------------\n".
#D ($ENV{"HTTP__LoadGen__Run__dbg"}>1
#D ? do {my $s=$rc[RC_BODY]; $s=~s/\n?$/\n/; $s}
#D : "BODY omitted: set HTTP__LoadGen__Run__dbg>1 to get it\n")
#D unless(no_response_body $rc[RC_STATUS], $method);
#D warn "---------------------------------------------------------\n";
# update connection cache
if(!$eof and
exists $param->{keepalive} and
($param->{keepalive} & KEEPALIVE_STORE) and
($rc[RC_HTTPVERSION]>=1.1 &&
!(exists $headers{connection} and
$headers{connection}->[0]=~/close/i) or
$rc[RC_HTTPVERSION]<1.1 &&
(exists $headers{connection} and
$headers{connection}->[0]=~/keep-alive/i))) {
my $ccel=[$connh, 1];
#D push @$ccel, $lport, $lip;
$connh->on_starttls(undef);
$connh->on_read(undef);
$connh->on_eof(undef);
# EOF as well as any other error is now handled by on_error
$connh->on_error(sub {
#D warn "Connection ($ccel->[3]:$ccel->[2])=>($ip:$port) closed while cached: $_[2]\n";
$ccel->[1]=0;
});
lib/HTTP/LoadGen/Run.pm view on Meta::CPAN
=head3 $no_body=HTTP::LoadGen::Run::no_response_body $http_code, $method
asks if a pair of HTTP status and request method is expected to include
a response body.
returns true if the body is omitted.
=head3 HTTP::LoadGen::Run::conncache
the cache of kept-alive connections. Returns a hash ref.
=head3 HTTP::LoadGen::Run::build_req
internal use.
=head3 HTTP::LoadGen::Run::config_handle
internal use.
=head3 HTTP::LoadGen::Run::gen_cb
lib/HTTP/LoadGen/Run.pm view on Meta::CPAN
All of the following constants are exported by default.
See also L<HTTP::LoadGen>.
=head2 Keep-Alive specification
=over 4
=item KEEPALIVE_USE (C<1>)
it is permitted to use a kept-alive connection if available
=item KEEPALIVE_STORE (C<2>)
it is permitted to keep the connection alive for later usage
=item KEEPALIVE (C<3>)
both of the above
=back
=head2 Request descriptor
These constants are indices into an array returned by the URL iterator.
t/005-run.t view on Meta::CPAN
These tests may fail due to the structure of the internet. Hosts may
become unaccessible or may update to newer software versions.
That said ...
EOF
# http://science.ksc.nasa.gov/
# apache with http/1.1 but connection: close (no keep-alive)
#
# http://217.86.174.228:8080/axis-cgi/jpg/image.cgi
# axis webcam: http/1.0
#
# http://foertsch.name/
# apache http/1.1 with keep-alive
my $rc;
HTTP::LoadGen::Run::dnscache=\my %dns_cache;
my $conncache=HTTP::LoadGen::Run::conncache;
SKIP: {
skip 'set $ENV{ONLINETESTS}>1 to run tests to domains other than my own', 8
unless $ENV{ONLINETESTS}>1;
#########################################################################
($rc)=HTTP::LoadGen::Run::run_url
qw!GET http science.ksc.nasa.gov 80 /!, {keepalive=>KEEPALIVE};
#warn Dumper $rc->[RC_HEADERS];
ok exists($dns_cache{'science.ksc.nasa.gov'}),
'science.ksc.nasa.gov resolved to '.$dns_cache{'science.ksc.nasa.gov'};
is $rc->[RC_CONNCACHED], 0, 'no kept-alive connection available';
is_deeply [%$conncache], [], 'conncache still empty';
is length($rc->[RC_BODY]), $rc->[RC_HEADERS]->{'content-length'}->[0],
'Body length: '.$rc->[RC_HEADERS]->{'content-length'}->[0];
#########################################################################
($rc)=HTTP::LoadGen::Run::run_url
qw!GET http 217.86.174.228 8080 /axis-cgi/jpg/image.cgi!, {keepalive=>KEEPALIVE};
#warn Dumper $rc->[RC_HEADERS];
ok exists($dns_cache{'217.86.174.228'}),
'217.86.174.228 resolved to '.$dns_cache{'217.86.174.228'};
is $rc->[RC_CONNCACHED], 0, 'no kept-alive connection available';
is_deeply [%$conncache], [], 'conncache still empty';
is length($rc->[RC_BODY]), $rc->[RC_HEADERS]->{'content-length'}->[0],
'Body length: '.$rc->[RC_HEADERS]->{'content-length'}->[0];
#########################################################################
}
($rc)=HTTP::LoadGen::Run::run_url
qw!GET http foertsch.name 80 /!, {keepalive=>KEEPALIVE};
#warn Dumper $rc->[RC_HEADERS];
ok exists($dns_cache{'foertsch.name'}),
'foertsch.name resolved to '.$dns_cache{'foertsch.name'};
is $rc->[RC_CONNCACHED], 0, 'no kept-alive connection available';
is 0+keys %$conncache, 1, 'conncache with 1 element';
is length($rc->[RC_BODY]), $rc->[RC_HEADERS]->{'content-length'}->[0],
'Body length: '.$rc->[RC_HEADERS]->{'content-length'}->[0];
($rc)=HTTP::LoadGen::Run::run_url
qw!GET http foertsch.name 80 /!, {keepalive=>KEEPALIVE_USE};
is $rc->[RC_CONNCACHED], 1, 'conncache used';
is 0+@{$conncache->{$dns_cache{'foertsch.name'}.' 80'}}, 0,
'conncache empty again';
($rc)=HTTP::LoadGen::Run::run_url
qw!GET http foertsch.name 80 /!, {keepalive=>KEEPALIVE_STORE};
($rc)=HTTP::LoadGen::Run::run_url
qw!GET http foertsch.name 80 /!, {keepalive=>KEEPALIVE_STORE};
is 0+@{$conncache->{$dns_cache{'foertsch.name'}.' 80'}}, 2,
'2 connections cached for '.$dns_cache{'foertsch.name'}.':80';
($rc)=HTTP::LoadGen::Run::run_url
qw!GET https www.kabatinte.net 443 /!, {keepalive=>KEEPALIVE_STORE};
is $rc->[RC_STATUS], 303, 'https://www.kabatinte.net/ => 303';
ok length($rc->[RC_STATUSLINE])>0, 'STATUS_LINE';
ok length($rc->[RC_HTTPVERSION])>0, 'HTTPVERSION';
ok $rc->[RC_STARTTIME]>0, 'STARTTIME';
ok $rc->[RC_CONNTIME]>0, 'CONNTIME';
ok $rc->[RC_FIRSTTIME]>0, 'FIRSTTIME';
ok $rc->[RC_HEADERTIME]>0, 'HEADERTIME';
ok $rc->[RC_BODYTIME]>0, 'BODYTIME';
is $rc->[RC_DNSCACHED], 0, 'DNS cache miss';
( run in 0.757 second using v1.01-cache-2.11-cpan-5511b514fd6 )