Algorithm-CRF

 view release on metacpan or  search on metacpan

common.h  view on Meta::CPAN

#define BUF_SIZE 8192

namespace CRFPP {
  template <class T> inline T _min(T x, T y) { return(x < y) ? x : y; }
  template <class T> inline T _max(T x, T y) { return(x > y) ? x : y; }

  template <class Iterator>
  inline size_t tokenizeCSV(char *str,
                            Iterator out, size_t max) {
    char *eos = str + std::strlen(str);
    char *start = 0;
    char *end = 0;
    size_t n = 0;

    for (; str < eos; ++str) {
      while (*str == ' ' || *str == '\t') ++str;  // skip white spaces
      bool inquote = false;
      if (*str == '"') {
        start = ++str;
        end = start;
        for (; str < eos; ++str) {
          if (*str == '"') {
            str++;
            if (*str != '"')
              break;
          }
          *end++ = *str;
        }
        inquote = true;
        str = std::find(str, eos, ',');
      } else {
        start = str;
        str = std::find(str, eos, ',');
        end = str;
      }
      if (max-- > 1) *end = '\0';
      *out++ = start;
      ++n;
      if (max == 0) break;
    }

    return n;
  }

  template <class Iterator>
  inline size_t tokenize(char *str, const char *del,
                         Iterator out, size_t max) {

t/test.data  view on Meta::CPAN

managed VBD O
Mr. NNP B
Noriega NNP I
and CC O
his PRP$ B
operation NN I
. . O

`` `` O
He PRP B
started VBD O
building VBG O
the DT B
files NNS I
that WDT B
helped VBD O
him PRP B
gain VBP O
power NN B
. . O
'' '' O

t/test.data  view on Meta::CPAN

and CC O
then RB O
retreating VBG O
to TO O
his PRP$ B
office NN I
. . O

`` `` O
People NNS B
started VBD O
wondering VBG O
if IN O
something NN B
was VBD O
wrong JJ O
with IN O
him PRP B
, , O
'' '' O
Mr. NNP B

t/test.data  view on Meta::CPAN

Old NNP B
Environmentalism NNP I
involved VBD O
microbe NN B
hunters NNS I
and CC O
sanitationists NNS B
. . O

It PRP B
started VBD O
with IN O
improvements NNS B
in IN O
hygiene NN B
made VBD O
possible JJ O
by IN O
affordable JJ B
soap NN I
and CC O

t/test.data  view on Meta::CPAN

handled VBN O
at IN O
the DT B
local JJ I
level NN I
. . O

New NNP B
Environmentalism NNP I
probably RB O
started VBD O
in IN O
1962 CD B
with IN O
the DT B
publication NN I
of IN O
Rachel NNP B
Carson NNP I
's POS B
book NN I

t/test.data  view on Meta::CPAN

ago RB O
. . O

Rohm NNP B
& CC I
Haas NNP I
, , O
which WDT B
plans VBZ O
to TO O
start VB O
operating VBG O
seven CD B
new JJ I
production NN I
units NNS I
this DT B
year NN I
, , O
attributed VBD O
the DT B
profit NN I
slide NN I
partly RB O
to TO O
higher JJR B
start-up JJ I
expense NN I
. . O

The DT B
company NN I
also RB O
cited VBD O
the DT B
stronger JJR I
dollar NN I

t/train.data  view on Meta::CPAN

of IN O
certain JJ B
types NNS I
of IN O
freight NN B
transportation NN I
, , O
and CC O
rates NNS B
are VBP O
starting VBG O
to TO O
move VB O
up IN O
'' '' O
at IN O
a DT B
rate NN I
`` `` O
close RB O
to TO O

t/train.data  view on Meta::CPAN

of IN O
deregulation NN B
in IN O
freight-cost JJ B
reductions NNS I
. . O

Now RB O
we PRP B
are VBP O
starting VBG O
to TO O
see VB O
real JJ B
freight-rate JJ I
increases NNS I
as IN O
carriers NNS B
replace VBP O
equipment NN B
, , O