Apache-Dynagzip
view release on metacpan or search on metacpan
Dynagzip.pm view on Meta::CPAN
__END__
=head1 NAME
Apache::Dynagzip - mod_perl extension for C<Apache-1.3.X> to compress the response with C<gzip> format.
=head1 ABSTRACT
This Apache handler provides dynamic content compression of the response data stream
for C<HTTP/1.0> and C<HTTP/1.1> requests.
Standard C<gzip> compression is optionally combined with an C<extra light> compression
that eliminates leading blank spaces and/or blank lines within the source document.
An C<extra light> compression could be applied even when the client (browser)
is not capable to decompress C<gzip> format.
Handler helps to compress the outbound
HTML content usually by 3 to 20 times, and provides a list of useful features.
This is particularly useful for compressing outgoing web content
that is dynamically generated on the fly (using templates, DB data, XML,
etc.), when at the time of the request it is impossible to determine the
length of the document to be transmitted. Support for Perl, Java, and C
source generators is provided.
Besides the benefits of reduced document size, this approach gains efficiency
from being able to overlap the various phases of data generation, compression,
transmission, and decompression. In fact, the browser can start to
decompress a document, which has not yet been completely generated.
=head1 SYNOPSIS
There is more then one way to configure Apache to use this handler...
=head2 Compress regular (static) HTML files
======================================================
Static html file (size=149208) no light compression:
======================================================
httpd.conf:
PerlModule Apache::Dynagzip
<Files ~ "*\.html">
SetHandler perl-script
PerlHandler Apache::Dynagzip
</Files>
client-side log:
C05 --> S06 GET /html/wowtmovie.html HTTP/1.1
C05 --> S06 Accept: */*
C05 --> S06 Referer: http://devl4.outlook.net/html/
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Pragma: no-cache
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Fri, 31 May 2002 17:36:57 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 X-Module-Sender: Apache::Dynagzip
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Expires: Friday, 31-May-2002 17:41:57 GMT
C05 <-- S06 Vary: Accept-Encoding
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 9411 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
1314 (hex) = 4884 (dec)
3ed (hex) = 1005 (dec)
354 (hex) = 852 (dec)
450 (hex) = 1104 (dec)
5e6 (hex) = 1510 (dec)
0 (hex) = 0 (dec)
== Latency = 0.170 seconds, Extra Delay = 0.440 seconds
== Restored Body was 149208 bytes ==
======================================================
Static html file (size=149208) with light compression:
======================================================
httpd.conf:
PerlModule Apache::Dynagzip
<Files ~ "*\.html">
SetHandler perl-script
PerlHandler Apache::Dynagzip
PerlSetVar LightCompression On
</Files>
client-side log:
C05 --> S06 GET /html/wowtmovie.html HTTP/1.1
C05 --> S06 Accept: */*
C05 --> S06 Referer: http://devl4.outlook.net/html/
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Pragma: no-cache
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Fri, 31 May 2002 17:49:06 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 X-Module-Sender: Apache::Dynagzip
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Expires: Friday, 31-May-2002 17:54:06 GMT
C05 <-- S06 Vary: Accept-Encoding
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 8515 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
119f (hex) = 4511 (dec)
3cb (hex) = 971 (dec)
472 (hex) = 1138 (dec)
736 (hex) = 1846 (dec)
0 (hex) = 0 (dec)
== Latency = 0.280 seconds, Extra Delay = 0.820 seconds
== Restored Body was 128192 bytes ==
Default values for the C<minChunkSizeSource> and C<minChunkSize> will be in effect in this case.
In order to overwrite them one can try for example
<IfModule mod_perl.c>
PerlModule Apache::Dynagzip
<Files ~ "*\.html">
SetHandler perl-script
PerlHandler Apache::Dynagzip
PerlSetVar minChunkSizeSource 36000
PerlSetVar minChunkSize 9
</Files>
</IfModule>
=head2 Compress the output stream of the Perl scripts
===============================================================================
GET dynamically generated (by perl script) html file with no light compression:
===============================================================================
httpd.conf:
PerlModule Apache::Filter
PerlModule Apache::Dynagzip
<Directory /var/www/perl/>
SetHandler perl-script
PerlHandler Apache::RegistryFilter Apache::Dynagzip
PerlSetVar Filter On
PerlSendHeader Off
PerlSetupEnv On
AllowOverride None
Options ExecCGI FollowSymLinks
Order allow,deny
Allow from all
</Directory>
client-side log:
C05 --> S06 GET /perl/start_example.cgi HTTP/1.1
C05 --> S06 Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/msword, */*
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Sat, 01 Jun 2002 16:59:47 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 X-Module-Sender: Apache::Dynagzip
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Expires: Saturday, 01-June-2002 17:04:47 GMT
C05 <-- S06 Vary: Accept-Encoding
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 758 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
2db (hex) = 731 (dec)
0 (hex) = 0 (dec)
== Latency = 0.220 seconds, Extra Delay = 0.050 seconds
== Restored Body was 1434 bytes ==
============================================================================
GET dynamically generated (by perl script) html file with light compression:
============================================================================
httpd.conf:
PerlModule Apache::Filter
PerlModule Apache::Dynagzip
<Directory /var/www/perl/>
SetHandler perl-script
PerlHandler Apache::RegistryFilter Apache::Dynagzip
PerlSetVar Filter On
PerlSetVar LightCompression On
PerlSendHeader Off
PerlSetupEnv On
AllowOverride None
Options ExecCGI FollowSymLinks
Order allow,deny
Allow from all
</Directory>
client-side log:
C05 --> S06 GET /perl/start_example.cgi HTTP/1.1
C05 --> S06 Accept: */*
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Pragma: no-cache
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Sat, 01 Jun 2002 17:09:13 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 X-Module-Sender: Apache::Dynagzip
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Expires: Saturday, 01-June-2002 17:14:14 GMT
C05 <-- S06 Vary: Accept-Encoding
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 750 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
2d3 (hex) = 723 (dec)
0 (hex) = 0 (dec)
== Latency = 0.280 seconds, Extra Delay = 0.000 seconds
== Restored Body was 1416 bytes ==
=head2 Compress the outgoing stream from the CGI binary
====================================================================================
GET dynamically generated (by C-written binary) html file with no light compression:
====================================================================================
httpd.conf:
PerlModule Apache::Dynagzip
<Directory /var/www/cgi-bin/>
SetHandler perl-script
PerlHandler Apache::Dynagzip
AllowOverride None
Options +ExecCGI
PerlSetupEnv On
PerlSetVar BinaryCGI On
Order allow,deny
Allow from all
</Directory>
client-side log:
C05 --> S06 GET /cgi-bin/mylook.cgi HTTP/1.1
C05 --> S06 Accept: */*
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Pragma: no-cache
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Fri, 31 May 2002 23:18:17 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 X-Module-Sender: Apache::Dynagzip
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Expires: Friday, 31-May-2002 23:23:17 GMT
C05 <-- S06 Vary: Accept-Encoding
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 1002 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
3cf (hex) = 975 (dec)
0 (hex) = 0 (dec)
== Latency = 0.110 seconds, Extra Delay = 0.110 seconds
== Restored Body was 1954 bytes ==
=================================================================================
GET dynamically generated (by C-written binary) html file with light compression:
=================================================================================
httpd.conf:
PerlModule Apache::Dynagzip
<Directory /var/www/cgi-bin/>
SetHandler perl-script
PerlHandler Apache::Dynagzip
AllowOverride None
Options +ExecCGI
PerlSetupEnv On
PerlSetVar BinaryCGI On
PerlSetVar LightCompression On
Order allow,deny
Allow from all
</Directory>
client-side log:
C05 --> S06 GET /cgi-bin/mylook.cgi HTTP/1.1
C05 --> S06 Accept: */*
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Pragma: no-cache
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Fri, 31 May 2002 23:37:45 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 X-Module-Sender: Apache::Dynagzip
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Expires: Friday, 31-May-2002 23:42:45 GMT
C05 <-- S06 Vary: Accept-Encoding
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 994 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
3c7 (hex) = 967 (dec)
0 (hex) = 0 (dec)
== Latency = 0.170 seconds, Extra Delay = 0.110 seconds
== Restored Body was 1862 bytes ==
=head1 INTRODUCTION
From a historical point of view this package was developed primarily in order to compress the output
of a proprietary CGI binary written in C that was
widely used by Outlook Technologies, Inc. in order to deliver uncompressed dynamically generated
HTML content over the Internet using C<HTTP/1.0> since the mid-'90s.
We were then presented with the challenge of using the content compression
features over C<HTTP/1.1> on busy production servers, especially those serving heavy traffic on virtual hosts
of popular American broadcasting companies.
The very first our attempts to implement a static gzip approach in order to compress the
dynamic content helped us to scale effectively the bandwidth at
the cost of significantly increased latency of the content delivery.
That was why I came up with an idea to use chunked data transmission of
the gzipped content, sharing a real time between the server side data
creation/compression, media data transmission, and the client side data
decompression/presentation in order to provide end users with the partially
displayed content as soon as it's possible in particular conditions of the
user's connection.
At the time we decided to go for dynamic compression there were no
appropriate software on the market. Even later in
February 2002 Nicholas Oxhøj wrote to the mod_perl mailing list about his
experience of finding Apache gzipper for the streaming outgoing content:
=for html
<blockquote>
I<"... I have been experimenting with all the different Apache compression modules
I have been able to find, but have not been able to get the desired result.
I have tried Apache::GzipChain, Apache::Compress, mod_gzip and mod_deflate, with
different results. One I cannot get to work at all. Most work, but seem to collect
all the output before compressing it and sending it to the browser...>
I<... Wouldn't it be nice to have some option to specify that the handler should flush
and send the currently compressed output every time it had received a certain amount
of input or every time it had generated a certain amount of output?..>
I<... So I am basically looking for anyone who has had any success in achieving this
kind of "streaming" compression, who could direct me at an appropriate Apache module.">
=for html
</blockquote>
Dynagzip.pm view on Meta::CPAN
in C<httpd.conf>. The value "On" is case-insensitive.
Any other value turns the C<extra light> compression off.
The main C<gzip> format is described in rfc1952.
This type of compression is applied when the client is recognized as one capable
to decompress C<gzip> format on the fly. In this version the decision is under the control
of whether the client sends the C<Accept-Encoding: gzip> HTTP header within the request, or not.
On C<HTTP/1.1>, when the C<gzip> compression is in effect, handler keeps the resonable control
over the size of the chunks and over the compression ratio
using the combination of two internal variables (those could be set in C<httpd.conf>):
minChunkSizeSource
minChunkSize
C<minChunkSizeSource> defines the minimum length of the source stream that C<zlib> may
accumulate in its internal buffer.
=over 4
=item Note:
The compression ratio depends on the length of the data
accumulated in that buffer;
More data we keep -- better ratio will be achieved...
=back
When the length defined by the C<minChunkSizeSource> is exceeded, the handler flushes the
internal buffer of C<zlib> and transfers the accumulated portion of the compressed data
into the own internal buffer in order to create a chunk when appropriate.
This buffer is not necessarily be fransfered to Appache immediately. The decision is
under the control of the C<minChunkSize> internal variable. When the size of the buffer
exceeds the value of C<minChunkSize> the handler chunks the internal buffer
and transfers the accumulated data to the Client.
This approach helps to create the effective compression combined with the limited latency.
For example, when I use
PerlSetVar minChunkSizeSource 16000
PerlSetVar minChunkSize 8
in my C<httpd.conf> in order to compress the dynamically generated content of the size of some
54,000 bytes, the client side log
C05 --> S06 GET /pipe/pp-pipe.pl/big.html?try=chunkOneMoreTime HTTP/1.1
C05 --> S06 Accept: */*
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
## Sockets 6 of 4,5,6 need checking ##
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Thu, 21 Feb 2002 20:01:47 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Vary: Accept-Encoding
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 6034 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
949 (hex) = 2377 (dec)
5e6 (hex) = 1510 (dec)
5c5 (hex) = 1477 (dec)
26e (hex) = 622 (dec)
0 (hex) = 0 (dec)
== Latency = 0.990 seconds, Extra Delay = 0.110 seconds
== Restored Body was 54655 bytes ==
shows that the first chunk consists of the gzip header only (10 bytes).
This chunk was sent back to web client as soon as the handler received the first portion of the data
generated by the CGI script. The data itself at that moment has been
storied in the zlib's internal buffer, because the C<minChunkSizeSource> is big enough.
=over 4
=item Note:
Longer we allow zlib to keep its internal buffer -- better compression ratio it makes for us...
=back
So far, in this example we have obtained the compression ratio at about 9 times.
In this version the handler provides defaults:
minChunkSizeSource = 32768
minChunkSize = 8
In case of C<gzip> compressed response to C<HTTP/1.0> request, handler uses C<minChunkSize>
and C<minChunkSizeSource> values in order to
limit the minimum size of internal buffers
providing appropriate compression ratio and avoiding multiple short outputs to the core Apache.
=head2 Chunking Features
On C<HTTP/1.1> this handler overwrites the default Apache behavior, and keeps own control over the
chunk-size when it is possible. In fact, handler provides the soft control over the chunk-size only:
It does never cut the incoming string in order to create a chunk of a particular size.
Instead, it controls the minimum size of the chunk only.
I consider this approach reasonable, because to date the HTTP chunk-size is not coordinated with the
packet-size on transport level.
In case of gzipped output the minimum size of the chunk is under the control of internal variable
minChunkSize
In case of uncompressed output, or the C<extra light> compression only,
the minimum size of the chunk is under the control of internal variable
minChunkSizePP
In this version handler provides defaults:
Dynagzip.pm view on Meta::CPAN
You usually should not care about this feature of C<Apache::Dynagzip>
unless you use it in your own chain of handlers for the various phases of the request processing.
=head2 Control over the Proxy Cache.
Control over the (possible) proxy cache is provided through the implementation of C<Vary>
HTTP header.
Within C<Apache::Dynagzip> this header is under the control of few simple rules:
=over 4
=item *
C<Apache::Dynagzip> does never generate this header unless C<gzip> compression is provided.
=item *
The value of C<Accept-Encoding> is always provided for this header, accompanying C<gzip> compression.
=item *
Advanced control over the proxy cache is provided since the version 0.07
with optional extension of Vary HTTP header.
This extension could be placed into your configuration file, using directive
C<PerlSetVar Vary E<lt>valueE<gt>>
Particularly, it might be helpful to indicate the content, which depends on some conditions,
other than just compression features.
For example, when the content is personalized, someone might wish to use
the "*" C<Vary> extension in order to prevent any proxy caching.
When the outgoing content is gzipped, this extension will be appended to the regular C<Vary> header,
like in the following example:
Using the following fragment within the C<httpd.conf>:
PerlModule Apache::Dynagzip
<Files ~ "*\.html">
SetHandler perl-script
PerlHandler Apache::Dynagzip
PerlSetVar LightCompression On
PerlSetVar Vary *
</Files>
We can observe the client-side log in the form of:
C05 --> S06 GET /devdoc/Dynagzip/Dynagzip.html HTTP/1.1
C05 --> S06 Accept: */*
C05 --> S06 Referer: http://devl4.outlook.net/devdoc/Dynagzip/
C05 --> S06 Accept-Language: en-us
C05 --> S06 Accept-Encoding: gzip, deflate
C05 --> S06 User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
C05 --> S06 Host: devl4.outlook.net
C05 --> S06 Pragma: no-cache
C05 --> S06 Accept-Charset: ISO-8859-1
== Body was 0 bytes ==
C05 <-- S06 HTTP/1.1 200 OK
C05 <-- S06 Date: Sun, 11 Aug 2002 21:28:43 GMT
C05 <-- S06 Server: Apache/1.3.22 (Unix) Debian GNU/Linux mod_perl/1.26
C05 <-- S06 X-Module-Sender: Apache::Dynagzip
C05 <-- S06 Expires: Sunday, 11-August-2002 21:33:43 GMT
C05 <-- S06 Vary: Accept-Encoding,*
C05 <-- S06 Transfer-Encoding: chunked
C05 <-- S06 Content-Type: text/html; charset=iso-8859-1
C05 <-- S06 Content-Encoding: gzip
C05 <-- S06 == Incoming Body was 11311 bytes ==
== Transmission: text gzip chunked ==
== Chunk Log ==
a (hex) = 10 (dec)
1c78 (hex) = 7288 (dec)
f94 (hex) = 3988 (dec)
0 (hex) = 0 (dec)
== Latency = 0.160 seconds, Extra Delay = 0.170 seconds
== Restored Body was 47510 bytes ==
=item *
Simple form
C<Vary: Accept-Encoding>
is provided as a default for the gzipped content.
=back
=head1 CUSTOMIZATION
C<Apache::Dynagzip> can be used in order
=over 4
=item *
to compress dynamic web content generated in C<Apache::Filter> chain;
=item *
to compress the output of CGI-compatible binary program;
=item *
to stream huge static files providing on the fly compression of the stream.
=back
These are the main regims, wich one can implement through the appropriate configuration of the handler.
Every main regim can be tuned with some specific settings and
can be accomplished with various control features.
All these specific settings and control features could be addressed through
additional configuration parameters unless provided defaults are sufficient.
=over 4
=item Note:
Do your best in order to avoid the implementation of this handler in internally redirected requests.
It does not help much in this case. Read your C<error_log> carefully in order to find appropriate
warnings. Tune your C<httpd.conf> carefully in order to take the most from opportunities offered
by this handler.
( run in 1.391 second using v1.01-cache-2.11-cpan-0bb4e1dffa6 )