EAI-Wrap
view release on metacpan or search on metacpan
- fixed author test 3b_Wrap_Integration.t for automated tests, enhanced 2_File.t with excel format_skip (+ improved logging)
0.03 Wed Sep 13 23:00:00 2023
- completed getConn/setConn in EAI::DB and getHandle/setHandle in in EAI::FTP, added task{redoTimestampPatternPart} for redo improvement, added task{execOnly} and process{onlyExecFor} for selecting specific loads for execution, added checkParam for ...
0.04 Sat Sep 16 12:00:00 2023
- refactored DB.pm, FTP.pm and File.pm to decouple from process and execute configs, added Log::Dispatch::Email::LogSender and use instead of Log::Dispatch::Email::MIMELite to fix a Mime:Lite problem with diacritic chars, fixed problems with execOnl...
0.05 Mon Sep 18 23:00:00 2023
- fixed Issue #149802 (Testcase for DateUtil) by setting TZ explicitly in DateUtil.t and #149803 by including Text::Glob in Makefile.PL. replaced timelocal/gm with timelocal/gm_modern in DateUtil
0.06 Fri Sep 22 23:00:00 2023
- added use warnings, restricted imports from used modules.
0.07 Mon Sep 25 23:00:00 2023
- fixed trial Email::MIME leftover, added possibility to add custom calendars in DateUtil and refactored is_holiday.
0.08 Mon Sep 25 24:00:00 2023
- added possibility to add custom locales in DateUtil, removed exported locale hashes and replaced with accessor functions.
0.09 Tue Sep 26 23:00:00 2023
- refactored flat Dumper into function, moved site.config for automated tests into $ENV{EAI_WRAP_CONFIG_PATH}/t, decoupled EAI::DB completely from EAI::Wrap (no $execute dependency)
1.0 Sat Sep 30 12:00:00 2023
- improved API Doc, enabled overriding of redoDir, logRootPath, historyFolder and historyFolderUpload using commandline parameters, e.g. --config redoDir=/your/redo/folder, prevent error mail (logger) and uncaught exception in sendGeneralMail when n...
1.1 Thu Oct 05 23:00:00 2023
- fixed hadDBError setting in EAI::Wrap::dumpDataIntoDB, enhanced plannedUntil to allow/show seconds, added setDebugLevel Tool for really quick changing log.config
1.2 Sun Oct 08 23:00:00 2023
- fixed some problems in checkLogExist.pl, removed no warnings 'redundant' in DateUtil (causes problems with perl<5.20 tests), remove @previousline and @previoustempline in EAI::File as previous lines can anyway accessed using $data, fixed evalCusto...
lib/EAI/Common.pm view on Meta::CPAN
config => { # parameter category for site global settings, usually defined in site.config and other associated configs loaded at INIT
checkLogExistDelay => {}, # ref to following hash: {Test => 2, Dev => 3, "" => 0}, mapping to set delays for checkLogExist per environment in $execute{env}, this can be further overriden per job (and environment) in checkLookup.
checkLookup => {}, # ref to following datastructure: {"scriptname.pl + optional addToScriptName" => {errmailaddress => "",errmailsubject => "",timeToCheck =>"", freqToCheck => "", logFileToCheck => "", logcheck => "",logRootPath =>""},...} used for...
errmailaddress => "", # default mail address for central logcheck/errmail sending
errmailsubject => "", # default mail subject for central logcheck/errmail sending
executeOnInit => "", # code to be executed during INIT of EAI::Wrap to allow for assignment of config/execute parameters from commandline params BEFORE Logging!
folderEnvironmentMapping => {}, # ref to hash {Test => "Test", Dev => "Dev", "" => "Prod"}, mapping for $execute{envraw} to $execute{env}
fromaddress => "", # from address for central logcheck/errmail sending, also used as default sender address for sendGeneralMail
historyFolder => {}, # ref to following hash: {"scriptname.pl + optional addToScriptName" => "folder"}, folders where downloaded files are historized, lookup key as in checkLookup, default in "" => "defaultfolder". historyFolder, historyFolderUploa...
historyFolderUpload => {}, # ref to following hash: {"scriptname.pl + optional addToScriptName" => "folder"}, folders where uploaded files are historized, lookup key as in checkLookup, default in "" => "defaultfolder"
logCheckHoliday => "", # calendar for business days in central logcheck/errmail sending. builtin calendars are AT (Austria), TG (Target), UK (United Kingdom) and WE (for only weekends). Calendars can be added with EAI::DateUtil::addCalendar
logs_to_be_ignored_in_nonprod => qr//, # regular expression to specify logs to be ignored in central logcheck/errmail sending
logprefixForLastLogfile => sub {}, # prefix for previous (day) logs to be set in error mail (link), if not given, defaults to get_curdate(). In case Log::Dispatch::FileRotate is used as the File Appender in Log4perl config, the previous log is iden...
logRootPath => {}, # ref to following hash: {"scriptname.pl + optional addToScriptName" => "folder"}, paths to log file root folders (environment is added to that if non production), lookup key as checkLookup, default in "" => "defaultfolder"
prodEnvironmentInSeparatePath => 1, # set to 1 if the production scripts/logs etc. are in a separate Path defined by folderEnvironmentMapping (prod=root/Prod, test=root/Test, etc.), set to 0 if the production scripts/logs are in the root folder and...
redoDir => {}, # ref to following hash: {"scriptname.pl + optional addToScriptName" => "folder"}, folders where files for redo are contained, lookup key as checkLookup, default in "" => "defaultfolder"
sensitive => {}, # hash lookup table: {"prefix" => {user=>"",pwd =>"",hostkey=>"",privkey =>""},...} for sensitive access information in DB and FTP (lookup keys are set with DB{prefix} or FTP{prefix}), may also be placed outside of site.config; all...
smtpServer => "", # smtp server for den (error) mail sending
smtpTimeout => 60, # timeout for smtp response
testerrmailaddress => '', # error mail address in non prod environment
DB => {},
lib/EAI/Common.pm view on Meta::CPAN
ignoreNoTest => 0, # ignore the notest file in the process-script folder, usually preventing all runs that are not in production
plannedUntil => 2359, # latest time that planned repetition should start, this can be given either as HHMM (HourMinute) or HHMMSS (HourMinuteSecond), in case of HHMM the "Second" part is attached as 59
redoFile => 1, # flag for specifying a redo
redoTimestampPatternPart => "", # part of the regex for checking against filename in redo with additional timestamp/redoDir pattern (e.g. "redo", numbers and _), anything after files barename (and before ".$ext" if extension is defined) is regarded...
retryEndsAfterMidnight => 1, # if set, all retries should end after midnight
retrySecondsErr => 60, # retry period in case of error
retrySecondsErrAfterXfails => 600, # after fail count is reached this alternate retry period in case of error is applied. If 0/undefined then job finishes after fail count
retrySecondsXfails => 3, # fail count after which the retrySecondsErr are changed to retrySecondsErrAfterXfails
retrySecondsPlanned => 300, # retry period in case of planned retry
skipHolidays => "", # skip script execution on holidays
skipHolidaysDefault => "", # holiday calendar to take into account for skipHolidays
skipWeekends => 0, # skip script execution on weekends
skipForFirstBusinessDate => "", # used for "wait with execution for first business date", either this is a calendar or 1 (then calendar is skipHolidaysDefault), this cannot be used together with skipHolidays
},
);
# alternate type checking for these as they might have different types
my %alternateType = (
configexecuteOnInit => sub {}, # can be eval string or anonymous sub
DBpostDumpProcessing => sub {}, # can be eval string or anonymous sub
DBpostReadProcessing => sub {}, # can be eval string or anonymous sub
Filecolumns => [], # can be ref to hash or ref to array
Fileformat_skip => 1, # can be "skip until pattern" string or line number (int)
Fileformat_sep => qr//, # can be separator string or regex split
FilelineCode => sub {}, # can be eval string or anonymous sub
FTPremoteHost => "", # can also be string
taskskipHolidays => 1, # can be calendar string or true (1)
taskskipForFirstBusinessDate => 1, # can be calendar string or true (1)
);
our %common;our %config;our @loads;our %execute;our @optload;our %opt;
our $EAI_WRAP_CONFIG_PATH; our $EAI_WRAP_SENS_CONFIG_PATH;
my @coreConfig = ("DB","File","FTP","process");
my @commonCoreConfig = (@coreConfig,"task");
my @allConfig = (@commonCoreConfig,"config");
our $logConfig;
# read given config file (eval perl code)
lib/EAI/Common.pm view on Meta::CPAN
return $dump;
}
# check starting conditions and return 1 if met
sub checkStartingCond ($) {
my $arg = shift;
my $logger = get_logger();
my ($task) = extractConfigs("checking starting conditions",$arg,"task");
my $curdate = get_curdate();
$logger->debug("checkStartingCond for \$curdate: $curdate, task config:".dumpFlat($task,1));
# skipHolidays is either a calendar or 1 (then defaults to $task->{skipHolidaysDefault})
my $holidayCal = $task->{skipHolidays} if $task->{skipHolidays};
# skipForFirstBusinessDate is for "wait with execution for first business date", either this is a calendar or 1 (then calendar is skipHolidaysDefault), this cannot be used together with skipHolidays
$holidayCal = $task->{skipForFirstBusinessDate} if $task->{skipForFirstBusinessDate};
# default setting (1 becomes $task->{skipHolidaysDefault})
$holidayCal = $task->{skipHolidaysDefault} if ($task->{skipForFirstBusinessDate} eq "1" or $task->{skipHolidays} eq "1");
if ($holidayCal) {
if (is_holiday($holidayCal,$curdate) and !$task->{redoFile}) {
$logger->info("skip processing (skipHolidays = ".$task->{skipHolidays}.", skipForFirstBusinessDate = ".$task->{skipForFirstBusinessDate}.") as $curdate holiday in $holidayCal !");
return 1;
}
}
if (($task->{skipWeekends} or $task->{skipForFirstBusinessDate}) and is_weekend($curdate) and !$task->{redoFile}) {
$logger->info("skip processing (skipWeekends = ".$task->{skipWeekends}.", skipForFirstBusinessDate = ".$task->{skipForFirstBusinessDate}.") as $curdate is day of weekend !");
return 1;
}
# if there are were any business days (meaning that nonBusinessDays are less than calendar days) since the 1st of the month, then skip if $task->{skipForFirstBusinessDate}
if ($task->{skipForFirstBusinessDate} and !$task->{redoFile}) {
my $nonBusinessDays;
my $daysfrom1st = substr($curdate,-2)-1; # get the first decimal from day part of today, this is the number of days from the 1st
# count non business days since the 1st of the month
for (1..$daysfrom1st) {
$nonBusinessDays += (is_weekend(subtractDays($curdate,$_)) or is_holiday($holidayCal,subtractDays($curdate,$_)));
}
$logger->debug("\$nonBusinessDays: $nonBusinessDays,\$daysfrom1st: $daysfrom1st");
if ($nonBusinessDays < $daysfrom1st) {
$logger->info("skip processing (skipForFirstBusinessDate = ".$task->{skipForFirstBusinessDate}.") as processing already took place on a business day before $curdate!");
lib/EAI/DateUtil.pm view on Meta::CPAN
my %easterHol = ("BS"=>{"EM"=>1,"AS"=>1,"WM"=>1,"CC"=>1,"GF"=>1},
"BF"=>{"EM"=>1,"AS"=>1,"WM"=>1,"CC"=>1},
"AT"=>{"EM"=>1,"AS"=>1,"WM"=>1,"CC"=>1},
"TG"=>{"EM"=>1,"GF"=>1},
"UK"=>{"EM"=>1,"GF"=>1}
);
# reference to functions for special holiday calculations
my %specialHol = ("UK" => \&UKspecial);
# adds calendar to DateUtil, first arg name of $cal, second arg fixed holidays hash, third easter holidays hash and fourth special function for additional calculations
sub addCalendar ($$$$) {
my ($cal,$fixHol,$eastHol,$specialHolSub) = @_;
return undef if !$cal or !$fixHol or !$eastHol or !$specialHolSub;
if (defined($fixedHol{$cal})) {
cluck("calender <$cal> already implemented for fixed holidays !");
return 0;
}
if (defined($easterHol{$cal})) {
cluck("calender <$cal> already implemented for easter holidays !");
return 0;
lib/EAI/DateUtil.pm view on Meta::CPAN
if (defined($specialHol{$cal})) {
cluck("calender <$cal> already implemented for additional calculations !");
return 0;
}
$fixedHol{$cal} = ($fixHol ? $fixHol : "");
$easterHol{$cal} = ($eastHol ? $eastHol : "");
$specialHol{$cal} = ($specialHolSub ? $specialHolSub : "");
return 1;
}
# check whether second arg is easter in passed calendar (first arg).
# requires entry of calendar in %easterHol hash: "$Cal" => {"GF"=>1,"EM"=>1,"ES"=>1,"AS"=>1,"WM"=>1,"CC"=>1}
sub is_easter ($$) {
my ($cal) = $_[0];
return undef if !$cal or !$_[1];
my ($y,$m,$d) = $_[1] =~ /(.{4})(..)(..)/;
return undef if !$y or !$m or !$d;
# first find easter sunday using year
my $D = (((255 - 11 * ($y % 19)) - 21) % 30) + 21;
my $easter = timegm_modern(0,0,12,1,2,$y) + ($D + ($D > 48 ? 1 : 0) + 6 - (($y + int($y / 4) + $D + ($D > 48 ? 1 : 0) + 1) % 7))*86400;
return 1 if makeMD($easter) eq $d.$m and $easterHol{$cal}->{"EH"}; # easter sunday
# then the rest
return 1 if makeMD($easter-2*86400) eq $d.$m and $easterHol{$cal}->{"GF"}; # good friday
return 1 if makeMD($easter+1*86400) eq $d.$m and $easterHol{$cal}->{"EM"}; # easter monday
return 1 if makeMD($easter+39*86400) eq $d.$m and $easterHol{$cal}->{"AS"}; # ascension day
return 1 if makeMD($easter+50*86400) eq $d.$m and $easterHol{$cal}->{"WM"}; # whitmonday
return 1 if makeMD($easter+60*86400) eq $d.$m and $easterHol{$cal}->{"CC"}; # corpus christi day
return 0;
}
# check whether second arg is holiday in passed calendar (first arg)
sub is_holiday ($$) {
my ($cal) = $_[0];
return undef if !$cal or $cal eq "WE" or !$_[1]; # weekends are checked with "is_weekend", so no holiday passed here! Same for empty dates passed
my ($y,$m,$d) = $_[1] =~ /(.{4})(..)(..)/;
return undef if !$y or !$m or !$d;
return 1 if $fixedHol{$cal}->{$d.$m};
return 1 if is_easter($cal,$_[1]);
return 1 if $specialHol{$cal} and $specialHol{$cal}->($_[1]);
unless ($fixedHol{$cal} or $easterHol{$cal} or $specialHol{$cal}) {
cluck("calender <$cal> neither implemented in \$fixedHol{$cal} nor \$easterHol{$cal} nor \$specialHol{$cal} !");
lib/EAI/DateUtil.pm view on Meta::CPAN
returns 1 if first day of months, 0 else
$date .. date in format YYYYMMDD
=item is_last_day_of_month ($;$)
returns 1 if last day of month, 0 else
$date .. date in format YYYYMMDD
$cal .. optional, calendar for holidays used to get the last of month
=item get_last_day_of_month ($)
returns last day of month of passed date
$date .. date in format YYYYMMDD
=item weekday ($)
returns 1..sunday to 7..saturday
lib/EAI/DateUtil.pm view on Meta::CPAN
=item is_weekend ($)
returns 1 if saturday or sunday
$date .. date in format YYYYMMDD
=item is_holiday ($$)
returns 1 if weekend or holiday
$cal .. holiday calendar; currently supported: AT (Austria), TG (Target), UK (see is_holiday) and WE (for only weekends).
throws warning if calendar not supported (fixed lookups or additionally added). To add a calendar use addCalendar.
$date .. date in format YYYYMMDD
=item is_easter ($$)
returns 1 if date is an easter holiday for that calendar
$cal .. holiday calendar;
$date .. date in format YYYYMMDD
=item addCalendar ($$$$)
add an additional calendar for calendar holiday dependent calculations
$cal .. name of holiday calendar to be added, warns if already existing (builtin)
$fixedHol .. hash of fixed holiday dates for that calendar (e.g. {"0105"=>1,"2512"=>1} for may day and christmas day)
$easterHol .. hash of easter holidays for that calendar (possible: {"GF"=>1,"EM"=>1,"ES"=>1,"AS"=>1,"WM"=>1,"CC"=>1}) = good friday,easter monday, easter sunday, ascension day, whitmonday, corpus christi day
$specialFunction .. pass ref to sub used for additional calculations; this sub should receive a date (YYYYMMDD) and return 1 for holiday, 0 otherwise.
Example:
sub testCalSpecial {
my ($y,$m,$d) = $_[0] =~ /(.{4})(..)(..)/;
return 1 if $y eq "2002" and $m eq "09" and $d eq "08";
return 0;
}
addCalendar("TC",{"0101"=>1,"0105"=>1,"2512"=>1,"2612"=>1},{"EM"=>1,"GF"=>1},\&testCalSpecial);
lib/EAI/DateUtil.pm view on Meta::CPAN
arguments $day, $mon, $year are returned by ref as well if not passed as literal
$day .. day part
$mon .. month part
$year .. year part
$dayDiff .. days to be added
$locale .. optional locale as defined in monthsToInt (builtin "en" and "ge", can be added with addLocaleMonths)
=item subtractDays ($$)
subtracts $days actual calendar days from $date
$date .. date in format YYYYMMDD
$days .. calendar days to subtract
=item addDaysHol ($$;$$)
adds $days days to $date and regards weekends and holidays of passed calendar
$date .. date in format YYYYMMDD
$days .. calendar days to add
$template .. as in formatDate
$cal .. holiday calendar; currently supported: NO (no holidays = default if not given), rest as in is_holiday
=item subtractDaysHol ($$;$$)
subtracts $days days from $date and regards weekends and holidays of passed calendar
$date .. date in format YYYYMMDD
$days .. calendar days to subtract
$template .. as in formatDate
$cal .. holiday calendar; currently supported: NO (no holidays = default if not given), rest as in is_holiday
=item addDatePart ($$$;$)
adds $count dateparts to $date. when adding to months ends (>28 in february, >29 or >30 else), if the month end is not available in the target month, then date is moved into following month
$date .. date in format YYYYMMDD
$count .. count of dateparts to add
$datepart .. can be "d" or "day" for days, "m"/"mon"/"month" for months and "y" or "year" for years
$template .. as in formatDate
lib/EAI/DateUtil.pm view on Meta::CPAN
=item convertToThousendDecimal ($$)
converts $value into thousand separated decimal (german format) ignoring decimal places if wanted
$value .. number to be converted
$ignoreDecimal .. return number without decimal places (truncate)
=item get_dateseries ($$$)
returns date values (format YYYYMMMDD) starting at $fromDate until $toDate, if a holiday calendar is set in $cal (optional), these holidays (incl. weekends) are regarded as well.
$fromDate .. start date
$toDate .. end date
$cal .. holiday calendar
=item parseFromDDMMYYYY ($)
returns time epoch from given datestring (dd.mm.yyyy)
$dateStr .. datestring
=item parseFromYYYYMMDD ($)
returns time epoch from given datestring (yyyymmdd)
lib/EAI/Wrap.pm view on Meta::CPAN
=item historyFolder
ref to following hash: {"scriptname.pl + optional addToScriptName" => "folder"}, folders where downloaded files are historized, lookup key as in checkLookup, default in "" => "defaultfolder". historyFolder, historyFolderUpload, logRootPath and redoDi...
=item historyFolderUpload
ref to following hash: {"scriptname.pl + optional addToScriptName" => "folder"}, folders where uploaded files are historized, lookup key as in checkLookup, default in "" => "defaultfolder"
=item logCheckHoliday
calendar for business days in central logcheck/errmail sending. builtin calendars are AT (Austria), TG (Target), UK (United Kingdom) and WE (for only weekends). Calendars can be added with EAI::DateUtil::addCalendar
=item logs_to_be_ignored_in_nonprod
regular expression to specify logs to be ignored in central logcheck/errmail sending
=item logprefixForLastLogfile
prefix for previous (day) logs to be set in error mail (link), if not given, defaults to get_curdate(). In case Log::Dispatch::FileRotate is used as the File Appender in Log4perl config, the previous log is identified with <logname>.1
=item logRootPath
lib/EAI/Wrap.pm view on Meta::CPAN
=item retrySecondsPlanned
retry period in case of planned retry
=item skipHolidays
skip script execution on holidays
=item skipHolidaysDefault
holiday calendar to take into account for skipHolidays
=item skipWeekends
skip script execution on weekends
=item skipForFirstBusinessDate
used for "wait with execution for first business date", either this is a calendar or 1 (then calendar is skipHolidaysDefault), this cannot be used together with skipHolidays
=back
=back
=head1 COPYRIGHT
Copyright (c) 2025 Roland Kapl
All rights reserved. This program is free software; you can
t/1_DateUtil.t view on Meta::CPAN
is(convertEpochToYYYYMMDD(Time::Piece->strptime("20010131","%Y%m%d")),"20010131",'convertEpochToYYYYMMDD Time::Piece 20010131');
is(get_last_day_of_month("20011215"),"20011231",'get_last_day_of_month 20011231');
is(get_last_day_of_month("20010115"),"20010131",'get_last_day_of_month 20010131');
is(get_last_day_of_month("20010215"),"20010228",'get_last_day_of_month 20010228');
is(get_last_day_of_month("20040215"),"20040229",'get_last_day_of_month 20040229');
sub testCalSpecial {
my ($y,$m,$d) = $_[0] =~ /(.{4})(..)(..)/;
return 1 if $y eq "2002" and $m eq "09" and $d eq "08";
return 0;
}
is(addCalendar("TC",{"0101"=>1,"0105"=>1,"2512"=>1,"2612"=>1},{"EM"=>1,"GF"=>1},\&testCalSpecial),1,'added test calendar');
is(is_holiday("TC","20020908"),1,'special holiday testcalendar');
is(is_holiday("TC","20120406"),1,'good friday testcalendar');
is(is_holiday("TC","20120501"),1,'may day testcalendar');
is(addLocaleMonths("FR",["jan","fév","mars","avr","mai","juin","juil","août","sept","oct","nov","déc"]),1,'added french short months');
is(monthsToInt("jan","FR"),"01",'1 from french january');
is(monthsToInt("jan","EN"),"01",'1 from english january');
is(monthsToInt("fév","FR"),"02",'2 from french february');
is(intToMonths(8,"FR"),"août",'french august from 8');
is(monthsToInt("mär","GE"),"03",'3 from german march');
is(intToMonths(3,"GE"),"Mär",'german march from 3');
is(formatDate(2019,3,1,"D.mmm.Y[fr]"),"01.mars.2019",'formatDate D.mmm.Y french');
is(formatDate(2019,3,1,"D.mmm.Y"),"01.Mär.2019",'formatDate D.mmm.Y german');
is(formatTime(make_time("122003")),"12:20:03",'formatTime from make_time("122003")');
( run in 0.737 second using v1.01-cache-2.11-cpan-5dc5da66d9d )