Perlbal
view release on metacpan or search on metacpan
-- access control test
-- add option to AccessControl plugin to use observed_ip_string instead
-- add observed_ip_string method to perlbal sockets, allowing http
connections to set an observed ip string when an upstream proxy is
trusted.
-- add blind_proxy option, which disables appending to the end of the
X-Forwarded-For header when connections arrive from a trusted proxy.
-- make socket closing more verbose when Perlbal::DEBUG is set
-- verify_backend_path configuration option
-- don't overwrite $^P, allows use of perl debugger on perlbal.
1.60: 2007-10-23
-- accept LFLF to end HTTP request headers, instead of just
reload <plugin_name>
1.57: 2007-04-26
-- make ClientHTTP (webserver mode) support transfer-encoding
"chunked" PUTs, when PUTs are already enabled. useful
for things like MogileFS, where writing clients may not
know the final length ahead of time to predeclare in the
Content-Length request header.
-- Add the client_ip to the X-Forwarded-For header when the
upstream is "trusted" (Ask Bjoern Hansen).
1.56: 2007-04-16
-- make HTML proper in ClientManage interface Jacques Marneweck
<jacques@powertrip.co.za>
-- fix "make test" on OS X. Radu Greab <radu@yx.ro> and
Chuck Remes <cremes.devlist@mac.com> fixed & confirmed.
a connection: close header
-- added 'uptime' management command to track how long Perlbal has been up
-- new config commands: HEADER INSERT <svc> <header>: <value> and HEADER
REMOVE <svc> <header> which will insert and remove headers from user
requests before they're sent to backend proxy nodes.
-- add dependency to Net::Netmask; now you can specify trusted_upstream_proxies
on a service (SET service.trusted_upstream_proxies = 10.0.0.0/8, etc)
which will allow requests from that range to set X-Forwarded-For, X-Host,
and X-Forwarded-Host headers.
-- fixed a bug that caused connections to hang when the backend responded
before the user was done sending data
-- reset some variables that weren't being reset: read_buf, read_ahead, read_size
-- "proc" management command shows user and system CPU usage for Perlbal
this run, as well as a delta since the last time you ran "proc"
o Uses cookies to determine if a request should go to fast queue
(configurable)
o Highpri (high priority) plugin supports making requests high
priority by URI or Host
o Can specify a relief level to let low priority requests through to
prevent starvation
* Can allow X-Forwarded-For (and similar) headers from client based on
client IP
* Configurable header management before sending request to backend
* Internal redirection to file or URL(s)
o Big one for us; a backend can instruct Perlbal to fetch the user's
data from a completely separate server and port and URL, 100%
transparent to the user
doc/service-parameters.txt view on Meta::CPAN
DEFAULT <param> = <value>
For all services:
+----------------------------------------------------------------------------------+
| Param |type| Default | Description |
|---------------------------+----+---------------------+---------------------------|
| | | |Whether to trust all |
| | | |incoming requests' |
| | | |X-Forwarded-For and related|
|always_trusted |bool|false |headers. Set to true only |
| | | |if you know that all |
| | | |incoming requests from your|
| | | |own proxy servers that |
| | | |clean/set those headers. |
|---------------------------+----+---------------------+---------------------------|
|client_sndbuf_size |size|0 |How large to set the |
| | | |client's socket SNDBUF. |
|---------------------------+----+---------------------+---------------------------|
|enable_ssl |bool|false |Enable SSL to the client. |
doc/service-parameters.txt view on Meta::CPAN
|---------------------------+----+---------------------+---------------------------|
|ssl_verify_mode |int |0 |SSL verification mode |
|---------------------------+----+---------------------+---------------------------|
| | | |A comma separated list of |
| | | |Net::Netmask filters (e.g. |
| | | |10.0.0.0/24, see |
| | | |Net::Netmask) that |
|trusted_upstream_proxies | | |determines whether upstream|
| | | |clients are trusted or not,|
| | | |where trusted means their |
| | | |X-Forwarded-For/etc headers|
| | | |are not munged. |
+----------------------------------------------------------------------------------+
Only for 'reverse_proxy' services:
+-------------------------------------------------------------------------------+
| Param |type| Default | Description |
|-----------------------------+----+-------------------+------------------------|
| | | |The number of backend |
|backend_persist_cache |int |2 |connections to keep |
| | | |alive on reserve while |
| | | |there are no clients. |
|-----------------------------+----+-------------------+------------------------|
| | | |Flag to disable any |
| | | |modification of |
|blind_proxy |bool|false |X-Forwarded-For, X-Host,|
| | | |and X-Forwarded-Host |
| | | |headers. |
|-----------------------------+----+-------------------+------------------------|
| | | |How much content-body |
| | | |(POST/PUT/etc) data we |
| | | |read from a client |
| | | |before we start sending |
| | | |it to a backend web |
| | | |node. If |
|buffer_backend_connect |size|100k |'buffer_uploads' is |
lib/Perlbal/BackendHTTP.pm view on Meta::CPAN
$hds->header("X-Proxy-Capabilities", "reproxy-file");
}
# decide whether we trust the upstream or not, to give us useful
# forwarding info headers
if ($svc->trusted_ip($client_ip)) {
# yes, we trust our upstream, so just append our client's IP
# to the existing list of forwarded IPs, if we're a blind proxy
# then don't append our IP to the end of the list.
unless ($svc->{blind_proxy}) {
my @ips = split /,\s*/, ($hds->header("X-Forwarded-For") || '');
$hds->header("X-Forwarded-For", join ", ", @ips, $client_ip);
}
} else {
# no, don't trust upstream (untrusted client), so remove all their
# forwarding headers and tag their IP as the x-forwarded-for
$hds->header("X-Forwarded-For", $client_ip);
$hds->header("X-Host", undef);
$hds->header("X-Forwarded-Host", undef);
}
$self->tcp_cork(1);
$client->state('backend_req_sent');
$self->{content_length} = undef;
$self->{content_length_remain} = undef;
lib/Perlbal/ClientHTTPBase.pm view on Meta::CPAN
});
};
$self->{post_sendfile_cb}->();
}
sub check_req_headers {
my Perlbal::ClientHTTPBase $self = shift;
my Perlbal::HTTPHeaders $hds = $self->{req_headers};
if ($self->{service}->trusted_ip($self->peer_ip_string)) {
my @ips = split /,\s*/, ($hds->header("X-Forwarded-For") || '');
# This list may be empty, and that's OK, in that case we should unset the
# observed_ip_string, so no matter what we'll use the 0th element, whether
# it happens to be an ip string, or undef.
$self->observed_ip_string($ips[0]);
}
return;
}
lib/Perlbal/Manual/Internals.pod view on Meta::CPAN
Total bytes read from client, ever.
=item ditch_leading_rn
If true, the next header parsing will ignore a leading \r\n.
=item observed_ip_string
If defined, contains the observed IP string of the peer we're serving. This is intended for holding the value of the X-Forwarded-For and using it to govern ACLs.
=back
=head3 Perlbal::TCPListener
Descends from L<Perlbal::Socket>.
Very lightweight and fast connection accept class. Takes incoming connections as fast as possible and passes them off, instantiating one of the various Client* classes to handle it.
lib/Perlbal/Manual/Internals.pod view on Meta::CPAN
Array of L<Net::Netmask> objects containing netmasks for trusted upstreams.
=item always_trusted
Boolean; if true, always trust upstreams.
=item blind_proxy
Boolean; if true, do not modify C<X-Forwarded-For>, C<X-Host>, or C<X-Forwarded-Host> headers.
=item enable_reproxy
Boolean; if true, advertise that server will reproxy files and/or URLs.
=item reproxy_cache_maxsize
Maximum number of reproxy results to be cached. (0 is disabled and default).
lib/Perlbal/Manual/Plugins.pod view on Meta::CPAN
Add a custom header according to the SSL of a service.
=item * L<Perlbal::Plugin::FlvStreaming> (Perlbal core)
Enable FLV streaming with reverse proxy.
=item * L<Perlbal::Plugin::ForwardedFor>
Rename the X-Forwarded-For header in Perlbal.
=item * L<Perlbal::Plugin::Highpri> (Perlbal core)
Makes some requests high priority.
=item * L<Perlbal::Plugin::Include> (Perlbal core)
Allows multiple, nesting configuration files.
lib/Perlbal/Manual/ReverseProxy.pod view on Meta::CPAN
You can set parameters via commands of either forms:
SET <service-name> <param> = <value>
SET <param> = <value>
=over 8
=item B<always_trusted> = bool
Whether to trust all incoming requests' X-Forwarded-For and related headers. Set to true only if you know that all incoming requests from your own proxy servers that clean/set those headers.
Default is false.
=item B<backend_persist_cache> = int
The number of backend connections to keep alive on reserve while there are no clients.
Default is 2.
=item B<blind_proxy> = bool
Flag to disable any modification of X-Forwarded-For, X-Host, and X-Forwarded-Host headers.
Default is false.
=item B<buffer_backend_connect> = size
How much content-body (POST/PUT/etc) data we read from a client before we start sending it to a backend web node. If C<buffer_uploads> is enabled, this value is used to determine how many bytes are read before Perlbal makes a determination on whether...
Default is 100k.
lib/Perlbal/Manual/ReverseProxy.pod view on Meta::CPAN
=item B<ssl_key_file> = path/to/file
Path to private key PEM file for SSL.
Default is C<certs/server-key.pem>.
=item B<trusted_upstream_proxies> = Net::Netmask filter
A comma separated list of L<Net::Netmask> filters (e.g. 10.0.0.0/24, see L<Net::Netmask>) that determines whether upstream clients are trusted or not, where trusted means their X-Forwarded-For/etc headers are not munged.
=item B<upload_status_listeners> = comma separated list of hosts
Comma separated list of hosts in form 'a.b.c.d:port' which will receive UDP upload status packets no faster than once a second per HTTP request (PUT/POST) from clients that have requested an upload status bar, which they request by appending the URL ...
=item B<verify_backend> = bool
Whether Perlbal should send a quick OPTIONS request to the backends before sending an actual client request to them. If your backend is Apache or some other process-based webserver, this is highly recommended. All too often a loaded backend box will ...
lib/Perlbal/Service.pm view on Meta::CPAN
'max_backend_uses', # max requests to send per kept-alive backend (default 0 = unlimited)
'connect_ahead', # scalar: number of spare backends to connect to in advance all the time
'buffer_size', # int: specifies how much data a ClientProxy object should buffer from a backend
'buffer_size_reproxy_url', # int: same as above but for backends that are reproxying for us
'queue_relief_size', # int; number of outstanding standard priority
# connections to activate pressure relief at
'queue_relief_chance', # int:0-100; % chance to take a standard priority
# request when we're in pressure relief mode
'trusted_upstream_proxies', # Net::Netmask object containing netmasks for trusted upstreams
'always_trusted', # bool; if true, always trust upstreams
'blind_proxy', # bool: if true, do not modify X-Forwarded-For, X-Host, or X-Forwarded-Host headers
'enable_reproxy', # bool; if true, advertise that server will reproxy files and/or URLs
'reproxy_cache_maxsize', # int; maximum number of reproxy results to be cached. (0 is disabled and default)
'client_sndbuf_size', # int: bytes for SO_SNDBUF
'server_process' , # scalar: path to server process (executable)
'persist_client_idle_timeout', # int: keep-alive timeout in seconds for clients (default is 30)
'idle_timeout', # int: idle timeout outside of keep-alive time (default is 30)
# Internal state:
'waiting_clients', # arrayref of clients waiting for backendhttp conns
'waiting_clients_highpri', # arrayref of high-priority clients waiting for backendhttp conns
lib/Perlbal/Service.pm view on Meta::CPAN
check_role => "reverse_proxy",
setter => sub {
my ($self, $val, $set, $mc) = @_;
my $rv = $set->();
$self->spawn_backends if $self->{enabled};
return $rv;
},
},
'always_trusted' => {
des => "Whether to trust all incoming requests' X-Forwarded-For and related headers. Set to true only if you know that all incoming requests from your own proxy servers that clean/set those headers.",
default => 0,
check_type => "bool",
check_role => "*",
},
'blind_proxy' => {
des => "Flag to disable any modification of X-Forwarded-For, X-Host, and X-Forwarded-Host headers.",
default => 0,
check_type => "bool",
check_role => "reverse_proxy",
},
'high_priority_cookie' => {
des => "The cookie name to inspect to determine if the client goes onto the high-priority queue.",
check_role => "reverse_proxy",
},
'high_priority_cookie_contents' => {
des => "A string that the high_priority_cookie must contain to go onto the high-priority queue.",
check_role => "reverse_proxy",
},
'trusted_upstream_proxies' => {
des => "A comma separated list of Net::Netmask filters (e.g. 10.0.0.0/24, see Net::Netmask) that determines whether upstream clients are trusted or not, where trusted means their X-Forwarded-For/etc headers are not munged.",
check_role => "*",
check_type => sub {
my ($self, $val, $errref) = @_;
unless (my $loaded = eval { require Net::Netmask; 1; }) {
$$errref = "Net::Netmask not installed";
return 0;
}
my @val = split /\s*,\s*/, $val;
my @trusted_upstreams = ();
lib/Perlbal/Socket.pm view on Meta::CPAN
'do_die', # if on, die and do no further requests
'read_buf', # arrayref of scalarref read from client
'read_ahead', # bytes sitting in read_buf
'read_size', # total bytes read from client, ever
'ditch_leading_rn', # if true, the next header parsing will ignore a leading \r\n
'observed_ip_string', # if defined, contains the observed IP string of the peer
# we're serving. this is intended for hoding the value of
# the X-Forwarded-For and using it to govern ACLs.
);
use constant MAX_HTTP_HEADER_LENGTH => 102400; # 100k, arbitrary
use constant TRACK_OBJECTS => 0; # see @created_objects below
if (TRACK_OBJECTS) {
use Scalar::Util qw(weaken isweak);
}
# kick-off one cleanup
t/90-accesscontrol.t view on Meta::CPAN
my $url = "http://127.0.0.1:$port/foo.txt";
my $req = HTTP::Request->new(GET => $url, @_);
my $res = $ua->request($req);
return $res->is_success;
}
ok(check(), "Initial request succeeds");
ok(manage("ACCESS deny ip 127.0.0.1"), "ACCESS deny was set properly");
ok(!check(), "Denied");
ok(!check(["X-Forwarded-For" => "1.1.1.1"]), "Denied with XFF header");
ok(manage("SET always_trusted = 1"), "Turning always trusted on");
ok(!check(), "Denied");
ok(check(["X-Forwarded-For" => "1.1.1.1"]), "Allowed with XFF header");
ok(manage("SET always_trusted = 0"), "Turning always trusted off again");
ok(manage("SET trusted_upstream_proxies = 127.0.0.1"), "Turning trusted upstream proxies on for 127.0.0.1");
ok(!check(), "Denied");
ok(check(["X-Forwarded-For" => "1.1.1.1"]), "Allowed with XFF header");
ok(manage("SET trusted_upstream_proxies = 10.0.0.0/24, 127.0.0.1"), "Turning trusted upstream proxies on for multiple netmasks");
ok(!check(), "Denied");
ok(check(["X-Forwarded-For" => "1.1.1.1"]), "Allowed with XFF header");
ok(manage("SET test.AccessControl.use_observed_ip = 0"), "Turning off observed IP");
ok(!check(["X-Forwarded-For" => "1.1.1.1"]), "Denied with XFF header");
( run in 0.509 second using v1.01-cache-2.11-cpan-4e96b696675 )