view release on metacpan or  search on metacpan

lib/Apache/Hadoop/Config.pm  view on Meta::CPAN

            # secondary namenode 
            'dfs.namenode.secondary.http-address' => $self->{'secondary'}.':50090',
            'dfs.namenode.secondary.https-address'=> $self->{'secondary'}.':50091',
            },
        'yarn-site.xml' => {
            'yarn.nodemanager.aux-services' => 'mapreduce_shuffle',
            'yarn.nodemanager.aux-services.mapreduce.shuffle.class' => 'org.apache.hadoop.mapred.ShuffleHandler',
            'yarn.web-proxy.address' => $self->{'proxynode'}.':'.$self->{'proxyport'},
            },
        'mapred-site.xml' => {
            'mapreduce.framework.name' => 'yarn',
            }
        };
    
    $self->_copyconf ( $config );
}

#
# directory management
#
sub _mkdir {

lib/Apache/Hadoop/Config.pm  view on Meta::CPAN

          yarn.scheduler.maximum-allocation-mb: 2576
          yarn.nodemanager.aux-services.mapreduce.shuffle.class: org.apache.hadoop.mapred.ShuffleHandler
          yarn.nodemanager.resource.memory-mb: 2576
        core-site.xml
          hadoop.tmp.dir: /hdfs/tmp
          fs.defaultFS: http://localhost:9000
        mapred-site.xml
          mapreduce.reduce.java.opts: -Xmx588m
          mapreduce.map.memory.mb: 368
          mapreduce.map.java.opts: -Xmx294m
          mapreduce.framework.name: yarn
          mapreduce.reduce.memory.mb: 736
        ---------------
        -> writing to ./hdfs-site.xml ...
        -> writing to ./yarn-site.xml ...
        -> writing to ./core-site.xml ...
        -> writing to ./mapred-site.xml ...

If supplied with some arguments, basically for a different clusters, the configuration files
can still be generated:

lib/Apache/Hadoop/Config.pm  view on Meta::CPAN

          yarn.scheduler.maximum-allocation-mb: 57340
          yarn.nodemanager.aux-services.mapreduce.shuffle.class: org.apache.hadoop.mapred.ShuffleHandler
          yarn.nodemanager.resource.memory-mb: 57340
        core-site.xml
          hadoop.tmp.dir: /hdfs/tmp
          fs.defaultFS: http://localhost:9000
        mapred-site.xml
          mapreduce.reduce.java.opts: -Xmx9174m
          mapreduce.map.memory.mb: 5734
          mapreduce.map.java.opts: -Xmx4587m
          mapreduce.framework.name: yarn
          mapreduce.reduce.memory.mb: 11468
        ---------------
        -> writing to ./hdfs-site.xml ...
        -> writing to ./yarn-site.xml ...
        -> writing to ./core-site.xml ...
        -> writing to ./mapred-site.xml ...

Different customization can be done, using object's constructor arguments.