DBD-SQLcipher
view release on metacpan or search on metacpan
1.43_06 2014-07-22
- Fixed compile error/warning for older perls (reported by ribasushi)
(ISHIGAKI)
1.43_05 2014-07-21
- No significant code changes; removed unnecessary dependencies.
1.43_04 2014-07-21
*** CHANGES THAT MAY POSSIBLY BREAK YOUR OLD APPLICATIONS ***
- Resolved #96877: sql statements should be converted to utf8 (DAMI)
If you set sqlite_unicode to true, SQL statements will be upgraded
to avoid inconsistency between embedded params and bind params.
- Resolved #96494: [PATCH] add SYSTEM TABLE to table_info() type
list (MJP)
- Supported virtual tables in Perl, and added two sample tables
(DAMI++)
1.43_03 2014-06-12
- Updated to SQLite 3.8.5, which should fix query planner's
issues in SQLite (ISHIGAKI)
1.40 2013-07-28
- NetBSD also doesn't like the _XOPEN_SOURCE hack (ISHIGAKI)
- Resolved #86080: PATCH: statistics_info support (DDICK)
1.39 2013-05-31
- Production release, no changes from 1.38_05
1.38_05 2013-05-31
- OpenBSD doesn't like the previous _XOPEN_SOURCE hack (ISHIGAKI)
- Disabled a unicode-related test for older perls (ISHIGAKI)
1.38_04 2013-05-29
- Tentatively defined _XOPEN_SOURCE under *BSD systems to see
if it solves a compilation issue for threaded perls (ISHIGAKI)
1.38_03 2013-05-20
*** NOTICE ON QUERY OPTIMIZER ENHANCEMENT ***
- As of SQLite 3.7.15, SQLite's query optimizer was enhanced
and the result order of a SELECT statement without an ORDER
BY clause may be different from the one of the previous
it actually broke things. This feature probably will be
enabled by default by the sqlite team in the future, and
eventually you'll need to cope with it, but right now we
agreed with some discussion to give you more time to be
prepared. If you use referential stuff in your schema
(which sqlite ignores now) should do extensive testing
to ensure that they will work when you issue "PRAGMA
foreign_keys = ON". (ISHIGAKI)
- Updated to SQLite 3.6.20 (DUNCAND)
- Resolved #50935: there remained old "unicode" attribute usage
in the pod, spotted by ASHLEY. (ISHIGAKI)
1.26_06 2009-10-28
*** CHANGES THAT MAY POSSIBLY BREAK YOUR OLD APPLICATIONS ***
- Removed undocumented (and most probably unused) reset method
from a statement handle (which was only accessible via func().)
Simply use "$sth->finish" instead. (ISHIGAKI)
- Now DBD::SQLite supports foreign key constraints by default.
Long-ignored foreign keys (typically written for other DB
engines) will start working. If you don't want this feature,
issue a pragma to disable foreign keys. (ISHIGAKI)
- Renamed "unicode" attribute to "sqlite_unicode" for integrity.
Old "unicode" attribute is still accessible but will be
deprecated in the near future. (ISHIGAKI)
- You can see file/line info while tracing even if you compile
with a non-gcc compiler. (ISHIGAKI)
- Major code refactoring. (ISHIGAKI)
- Pod reorganized, and some of the missing bits (including
pragma) are added. (ISHIGAKI)
1.26_05 2009-10-15
- Updated to SQLite 3.6.19 (ISHIGAKI)
- Added access to Online Backup functionality. (TJC)
- Added enable_load_extension pod (ISHIGAKI)
- Now private methods/functions return true after successful
calls (#44871) (ISHIGAKI)
- Removed all of the "croak"s (#44871) (ISHIGAKI)
1.26_01 2009-05-05
- Added ORDINAL_POSITION support for $dbh->column_info (ADAMK)
- Applied several fixes from GFUJI to clean up code (#45578)
(ISHIGAKI)
- Skipped some of the unicode path tests under cygwin (#45166)
(JDHEDDEN)
- Added some explanation and workarounds for a SQL that
compares a return value of a function with a numeric bind
value (ISHIGAKI)
1.25 2009-04-23
- Amalgamation conversion turned out to be quicker than expected.
- Changing to a production release. (ADAMK)
1.24_02 2009-04-22
- Merging various externally-contributed annotations from
annocpan.org (ADAMK)
- Created the beginnings of a DBD::SQLite::Cookbook (ADAMK)
1.24_01 2009-04-22
- Moved getsqlite.pl into util (ADAMK)
- Switching to the RT queue instead of the RT report page that
does nothing and just refers you to email (ADAMK)
- Now DBD::SQLite also uses amalgamated source recommended at sqlite.org (ISHIGAKI)
- Resolved #45166: better unicode path handling under cygwin (ISHIGAKI)
- Resolved #45171: test failure on CentOS 4.6 (ISHIGAKI)
1.23 2009-04-19
- No changes from 1.22_08, just switched to production release (ADAMK)
1.22_08 2009-04-17
- Completed the migration of all tests and deleted lib.pl (ADAMK)
- Prevented a double "commit is innefective" warning (ADAMK)
- Adding a version for the Win32.pm dependency (ADAMK)
1.22_07 2009-04-16
- Improved non-latin unicode filename support/test
on Windows (SZABGAB/ISHIGAKI)
- Removed the table name generator from t/lib.pl,
getting us closer to removing t/lib.pl entirely (ADAMK)
- Increased use of Test::NoWarnings (ADAMK)
- Converted half the remaining lib.pl tests to t::lib::Test (ADAMK)
- Require Win32.pm on Windows (CHORNY)
1.22_06 2009-04-15
- Simplifying various miscellaneous code (ADAMK)
- Adding support for non-latin unicode filenames on Windows (ADAMK)
1.22_05 2009-04-15
- Hopefully the last dev release before the next production release.
- Updated to SQLite 3.6.13 (DUNCAND)
- Setting svn:eol-style to native to prevent EOL issues (ADAMK)
- Resolved #44861: tweaked Makefile.PL to support older HP-UX (ISHIGAKI)
1.22_04 2009-04-11
- Adding support parsing attributes out of the DSN (ADAMK)
- Inserted pTHX_/aTHX_ for better efficiency (suggested in #44884 by TIMB) (ISHIGAKI)
1.22_03 2009-04-10
- Resolved #44876: Patch to fix includes in the SQLITE_LOCATION case by janus (ISHIGAKI)
- Added PERL_NO_GET_CONTEXT for efficiency (suggested in #44884 by TIMB) (ISHIGAKI)
- Refactored error handling (suggested in #44884, #44871 by TIMB) (ISHIGAKI)
1.22_02 2009-04-09
- Added missing documentation bits for 'create_collation'
and 'progress_handler' (DAMI)
- Resolved RT#25924 (Arguments to user-defined functions do not
respect unicode setting) (DAMI)
- Added comments on the return values on error, and fixed another
wrong return value in execute (ISHIGAKI)
- Added SQL_NULLABLE_UNKNOWN; still wonders if the error above
should be ignored or not (ISHIGAKI)
1.22_01 2009-04-09
- Resolved #25371: Calls sv_utf8_upgrade on strings going into
the database to make sure latin-1 strings are not saved as
Malformed UTF-8 character in the SQLite TEXT column (MIYAGAWA)
- Improved error handling
(many MANY thanks to Tim for all these patches!)
- Updated to sqlite 2.8.0
0.24
- Fixed major crash bug affecting Mac OS X
- Removed test.pl from distribution
- Upgraded to sqlite 2.7.6
0.23
- Fixed unicode tests
0.22
- Merge with sqlite 2.7.4
0.21
- Ooops - forgot new opcodes files from MANIFEST
0.20
- Port to SQLite 2.7.2
- Fixed bug in not freeing memory if you re-execute a $sth
- Fixed getsqlite.pl
0.16
- Upgraded to SQLite 2.5.0
0.15
- Upgraded to SQLite 2.4.5
0.14
- Added NoUTF8Flag option, so that returned strings don't get flagged
with SvUTF8_on() - needed when you're storing non-unicode in the database
0.13
- Upgraded to SQLite 2.4.3
- Added script to download sqlite core library when it's upgraded
0.12
- Upgraded to SQLite 2.4.2
0.11
- Upgraded to SQLite 2.4.0, which adds views, subqueries, new builtin
t/01_compile.t
t/02_logon.t
t/03_create_table.t
t/04_insert.t
t/05_select.t
t/06_tran.t
t/07_error.t
t/08_busy.t
t/09_create_function.t
t/10_create_aggregate.t
t/12_unicode.t
t/13_create_collation.t
t/14_progress_handler.t
t/15_ak_dbd.t
t/16_column_info.t
t/17_createdrop.t
t/18_insertfetch.t
t/19_bindparam.t
t/20_blobs.t
t/21_blobtext.t
t/22_listfields.t
t/54_literal_txn.t
t/55_statistics_info.t
t/56_open_flags.t
t/57_uri_filename.t
t/58_see_if_its_a_number_and_explicit_binding.t
t/59_extended_result_codes.t
t/cookbook_variance.t
t/lib/Test.pm
t/rt_15186_prepcached.t
t/rt_21406_auto_finish.t
t/rt_25371_asymmetric_unicode.t
t/rt_25460_numeric_aggregate.t
t/rt_25924_user_defined_func_unicode.t
t/rt_26775_distinct.t
t/rt_27553_prepared_cache_and_analyze.t
t/rt_29058_group_by.t
t/rt_29629_sqlite_where_length.t
t/rt_31324_full_names.t
t/rt_32889_prepare_cached_reexecute.t
t/rt_36836_duplicate_key.t
t/rt_36838_unique_and_bus_error.t
t/rt_40594_nullable.t
t/rt_48393_debug_panic_with_commit.t
t/rt_50503_fts3.t
t/rt_52573_manual_exclusive_lock.t
t/rt_53235_icu_compatibility.t
t/rt_62370_diconnected_handles_operation.t
t/rt_64177_ping_wipes_out_the_errstr.t
t/rt_67581_bind_params_mismatch.t
t/rt_71311_bind_col_and_unicode.t
t/rt_73159_fts_tokenizer_segfault.t
t/rt_73787_exponential_buffer_overflow.t
t/rt_76395_int_overflow.t
t/rt_77724_primary_key_with_a_whitespace.t
t/rt_78833_utf8_flag_for_column_names.t
t/rt_81536_multi_column_primary_key_info.t
t/rt_88228_sqlite_3_8_0_crash.t
t/rt_96050_db_filename_for_a_closed_database.t
t/rt_96877_unicode_statements.t
t/rt_96878_fts_contentless_table.t
t/rt_97598_crash_on_disconnect_with_virtual_tables.t
t/virtual_table/00_base.t
t/virtual_table/01_destroy.t
t/virtual_table/02_find_function.t
t/virtual_table/10_filecontent.t
t/virtual_table/11_filecontent_fulltext.t
t/virtual_table/20_perldata.t
t/virtual_table/21_perldata_charinfo.t
t/virtual_table/rt_99748.t
The above will allocate 800M for DB cache; the default is 2M. Your sweet
spot probably lies somewhere in between.
DRIVER PRIVATE ATTRIBUTES
Database Handle Attributes
sqlite_version
Returns the version of the SQLcipher library which DBD::SQLcipher is
using, e.g., "2.8.0". Can only be read.
sqlite_unicode
If set to a true value, DBD::SQLcipher will turn the UTF-8 flag on for
all text strings coming out of the database (this feature is
currently disabled for perl < 5.8.5). For more details on the UTF-8
flag see perlunicode. The default is for the UTF-8 flag to be turned
off.
Also note that due to some bizarreness in SQLcipher's type system (see
<http://www.sqlite.org/datatype3.html>), if you want to retain
blob-style behavior for some columns under "$dbh->{sqlite_unicode} =
1" (say, to store images in the database), you have to state so
explicitly using the 3-argument form of "bind_param" in DBI when
doing updates:
use DBI qw(:sql_types);
$dbh->{sqlite_unicode} = 1;
my $sth = $dbh->prepare("INSERT INTO mytable (blobcolumn) VALUES (?)");
# Binary_data will be stored as is.
$sth->bind_param(1, $binary_data, SQL_BLOB);
Defining the column type as "BLOB" in the DDL is not sufficient.
This attribute was originally named as "unicode", and renamed to
"sqlite_unicode" for integrity since version 1.26_06. Old "unicode"
attribute is still accessible but will be deprecated in the near
future.
sqlite_allow_multiple_statements
If you set this to true, "do" method will process multiple
statements at one go. This may be handy, but with performance
penalty. See above for details.
sqlite_use_immediate_transaction
If you set this to true, DBD::SQLcipher tries to issue a "begin
txt1 COLLATE perl,
txt2 COLLATE perllocale,
txt3 COLLATE nocase
)
or
SELECT * FROM foo ORDER BY name COLLATE perllocale
Unicode handling
If the attribute "$dbh->{sqlite_unicode}" is set, strings coming from
the database and passed to the collation function will be properly
tagged with the utf8 flag; but this only works if the "sqlite_unicode"
attribute is set before the first call to a perl collation sequence .
The recommended way to activate unicode is to set the parameter at
connection time :
my $dbh = DBI->connect(
"dbi:SQLcipher:dbname=foo", "", "",
{
RaiseError => 1,
sqlite_unicode => 1,
}
);
Adding user-defined collations
The native SQLcipher API for adding user-defined collations is exposed
through methods "sqlite_create_collation" and "sqlite_collation_needed".
To avoid calling these functions every time a $dbh handle is created,
"DBD::SQLcipher" offers a simpler interface through the
%DBD::SQLcipher::COLLATION hash : just insert your own collation functions
return SQLITE_BLOB;
default:
return SQLITE_TEXT;
}
}
void
init_cxt() {
dTHX;
MY_CXT_INIT;
MY_CXT.last_dbh_is_unicode = 0;
}
SV *
stacked_sv_from_sqlite3_value(pTHX_ sqlite3_value *value, int is_unicode)
{
STRLEN len;
sqlite_int64 iv;
int type = sqlite3_value_type(value);
SV *sv;
switch(type) {
case SQLITE_INTEGER:
iv = sqlite3_value_int64(value);
if ( iv >= IV_MIN && iv <= IV_MAX ) {
else {
/* warn("integer overflow, cast to NV"); */
return sv_2mortal(newSVnv((NV)iv));
}
case SQLITE_FLOAT:
return sv_2mortal(newSVnv(sqlite3_value_double(value)));
break;
case SQLITE_TEXT:
len = sqlite3_value_bytes(value);
sv = newSVpvn((const char *)sqlite3_value_text(value), len);
if (is_unicode) {
SvUTF8_on(sv);
}
return sv_2mortal(sv);
case SQLITE_BLOB:
len = sqlite3_value_bytes(value);
return sv_2mortal(newSVpvn(sqlite3_value_blob(value), len));
default:
return &PL_sv_undef;
}
}
val = hv_fetch(hv, "sqlite_open_flags", 17, 0);
flag = (val && SvOK(*val)) ? SvIV(*val) : 0;
}
}
rc = sqlite_open2(dbname, &(imp_dbh->db), flag, extended);
if ( rc != SQLITE_OK ) {
return FALSE; /* -> undef in lib/DBD/SQLcipher.pm */
}
DBIc_IMPSET_on(imp_dbh);
imp_dbh->unicode = FALSE;
imp_dbh->functions = newAV();
imp_dbh->aggregates = newAV();
imp_dbh->collation_needed_callback = newSVsv( &PL_sv_undef );
imp_dbh->timeout = SQL_TIMEOUT;
imp_dbh->handle_binary_nulls = FALSE;
imp_dbh->allow_multiple_statements = FALSE;
imp_dbh->use_immediate_transaction = TRUE;
imp_dbh->see_if_its_a_number = FALSE;
imp_dbh->extended_result_codes = extended;
imp_dbh->stmt_list = NULL;
int rc = 0;
int i;
char *statement;
if (!DBIc_ACTIVE(imp_dbh)) {
sqlite_error(dbh, -2, "attempt to do on inactive database handle");
return -2; /* -> undef in SQLcipher.xsi */
}
/* sqlite3_prepare wants an utf8-encoded SQL statement */
if (imp_dbh->unicode) {
sv_utf8_upgrade(sv_statement);
}
statement = SvPV_nolen(sv_statement);
sqlite_trace(dbh, imp_dbh, 3, form("do statement: %s", statement));
croak_if_db_is_null();
if (sqlite3_get_autocommit(imp_dbh->db)) {
}
if (strEQ(key, "sqlite_see_if_its_a_number")) {
imp_dbh->see_if_its_a_number = !(! SvTRUE(valuesv));
return TRUE;
}
if (strEQ(key, "sqlite_extended_result_codes")) {
imp_dbh->extended_result_codes = !(! SvTRUE(valuesv));
sqlite3_extended_result_codes(imp_dbh->db, imp_dbh->extended_result_codes);
return TRUE;
}
if (strEQ(key, "sqlite_unicode")) {
#if PERL_UNICODE_DOES_NOT_WORK_WELL
sqlite_trace(dbh, imp_dbh, 3, form("Unicode support is disabled for this version of perl."));
imp_dbh->unicode = 0;
#else
imp_dbh->unicode = !(! SvTRUE(valuesv));
#endif
return TRUE;
}
if (strEQ(key, "unicode")) {
if (DBIc_has(imp_dbh, DBIcf_WARN))
warn("\"unicode\" attribute will be deprecated. Use \"sqlite_unicode\" instead.");
#if PERL_UNICODE_DOES_NOT_WORK_WELL
sqlite_trace(dbh, imp_dbh, 3, form("Unicode support is disabled for this version of perl."));
imp_dbh->unicode = 0;
#else
imp_dbh->unicode = !(! SvTRUE(valuesv));
#endif
return TRUE;
}
return FALSE;
}
SV *
sqlite_db_FETCH_attrib(SV *dbh, imp_dbh_t *imp_dbh, SV *keysv)
{
dTHX;
}
if (strEQ(key, "sqlite_use_immediate_transaction")) {
return sv_2mortal(newSViv(imp_dbh->use_immediate_transaction ? 1 : 0));
}
if (strEQ(key, "sqlite_see_if_its_a_number")) {
return sv_2mortal(newSViv(imp_dbh->see_if_its_a_number ? 1 : 0));
}
if (strEQ(key, "sqlite_extended_result_codes")) {
return sv_2mortal(newSViv(imp_dbh->extended_result_codes ? 1 : 0));
}
if (strEQ(key, "sqlite_unicode")) {
#if PERL_UNICODE_DOES_NOT_WORK_WELL
sqlite_trace(dbh, imp_dbh, 3, "Unicode support is disabled for this version of perl.");
return sv_2mortal(newSViv(0));
#else
return sv_2mortal(newSViv(imp_dbh->unicode ? 1 : 0));
#endif
}
if (strEQ(key, "unicode")) {
if (DBIc_has(imp_dbh, DBIcf_WARN))
warn("\"unicode\" attribute will be deprecated. Use \"sqlite_unicode\" instead.");
#if PERL_UNICODE_DOES_NOT_WORK_WELL
sqlite_trace(dbh, imp_dbh, 3, "Unicode support is disabled for this version of perl.");
return sv_2mortal(newSViv(0));
#else
return sv_2mortal(newSViv(imp_dbh->unicode ? 1 : 0));
#endif
}
return NULL;
}
SV *
sqlite_db_last_insert_id(SV *dbh, imp_dbh_t *imp_dbh, SV *catalog, SV *schema, SV *table, SV *field, SV *attr)
{
dTHX;
sqlite_st_prepare_sv(SV *sth, imp_sth_t *imp_sth, SV *sv_statement, SV *attribs)
{
dTHX;
dMY_CXT;
int rc = 0;
const char *extra;
char *statement;
stmt_list_s * new_stmt;
D_imp_dbh_from_sth;
MY_CXT.last_dbh_is_unicode = imp_dbh->unicode;
if (!DBIc_ACTIVE(imp_dbh)) {
sqlite_error(sth, -2, "attempt to prepare on inactive database handle");
return FALSE; /* -> undef in lib/DBD/SQLcipher.pm */
}
/* sqlite3_prepare wants an utf8-encoded SQL statement */
if (imp_dbh->unicode) {
sv_utf8_upgrade(sv_statement);
}
statement = SvPV_nolen(sv_statement);
#if 0
if (*statement == '\0') {
sqlite_error(sth, -2, "attempt to prepare empty statement");
return FALSE; /* -> undef in lib/DBD/SQLcipher.pm */
}
else if (sql_type == SQLITE_BLOB) {
STRLEN len;
char * data = SvPVbyte(value, len);
rc = sqlite3_bind_blob(imp_sth->stmt, i+1, data, len, SQLITE_TRANSIENT);
}
else {
STRLEN len;
const char *data;
int numtype = 0;
if (imp_dbh->unicode) {
sv_utf8_upgrade(value);
}
data = SvPV(value, len);
/*
* XXX: For backward compatibility, it'd be better to
* accept a value like " 4" as an integer for an integer
* type column (see t/19_bindparam.t), at least when
* we explicitly specify its type. However, we should
* keep spaces when we just guess.
sqlite_trace(sth, imp_sth, 5, form("fetch column %d as text", i));
val = (char*)sqlite3_column_text(imp_sth->stmt, i);
len = sqlite3_column_bytes(imp_sth->stmt, i);
if (chopBlanks) {
while((len > 0) && (val[len-1] == ' ')) {
len--;
}
}
sv_setpvn(AvARRAY(av)[i], val, len);
if (imp_dbh->unicode) {
SvUTF8_on(AvARRAY(av)[i]);
} else {
SvUTF8_off(AvARRAY(av)[i]);
}
break;
case SQLITE_BLOB:
sqlite_trace(sth, imp_sth, 5, form("fetch column %d as blob", i));
len = sqlite3_column_bytes(imp_sth->stmt, i);
sv_setpvn(AvARRAY(av)[i], sqlite3_column_blob(imp_sth->stmt, i), len);
SvUTF8_off(AvARRAY(av)[i]);
retsv = sv_2mortal(newRV_noinc((SV*)av));
for (n = 0; n < i; n++) {
/* warn("Fetch col name %d\n", n); */
const char *fieldname = sqlite3_column_name(imp_sth->stmt, n);
if (fieldname) {
/* warn("Name [%d]: %s\n", n, fieldname); */
/* char *dot = instr(fieldname, "."); */
/* if (dot) drop table name from field name */
/* fieldname = ++dot; */
SV *sv_fieldname = newSVpv(fieldname, 0);
if (imp_dbh->unicode)
SvUTF8_on(sv_fieldname);
av_store(av, n, sv_fieldname);
}
}
}
else if (strEQ(key, "PRECISION")) {
AV *av = newAV();
retsv = sv_2mortal(newRV_noinc((SV*)av));
}
else if (strEQ(key, "TYPE")) {
if (!DBIc_ACTIVE(imp_dbh)) {
sqlite_error(dbh, -2, "attempt to set busy timeout on inactive database handle");
return -2;
}
sqlite3_busy_timeout(imp_dbh->db, imp_dbh->timeout);
}
return imp_dbh->timeout;
}
static void
sqlite_db_func_dispatcher(int is_unicode, sqlite3_context *context, int argc, sqlite3_value **value)
{
dTHX;
dSP;
int count;
int i;
SV *func;
func = sqlite3_user_data(context);
ENTER;
SAVETMPS;
PUSHMARK(SP);
for ( i=0; i < argc; i++ ) {
XPUSHs(stacked_sv_from_sqlite3_value(aTHX_ value[i], is_unicode));
}
PUTBACK;
count = call_sv(func, G_SCALAR|G_EVAL);
SPAGAIN;
/* Check for an error */
if (SvTRUE(ERRSV) ) {
sqlite_set_result(aTHX_ context, ERRSV, 1);
} else {
sqlite_set_result(aTHX_ context, POPs, 0 );
}
PUTBACK;
FREETMPS;
LEAVE;
}
static void
sqlite_db_func_dispatcher_unicode(sqlite3_context *context, int argc, sqlite3_value **value)
{
sqlite_db_func_dispatcher(1, context, argc, value);
}
static void
sqlite_db_func_dispatcher_no_unicode(sqlite3_context *context, int argc, sqlite3_value **value)
{
sqlite_db_func_dispatcher(0, context, argc, value);
}
int
sqlite_db_create_function(pTHX_ SV *dbh, const char *name, int argc, SV *func, int flags)
{
D_imp_dbh(dbh);
int rc;
SV *func_sv;
/* Copy the function reference */
func_sv = newSVsv(func);
av_push( imp_dbh->functions, func_sv );
croak_if_db_is_null();
/* warn("create_function %s with %d args\n", name, argc); */
rc = sqlite3_create_function( imp_dbh->db, name, argc, SQLITE_UTF8|flags,
func_sv,
imp_dbh->unicode ? sqlite_db_func_dispatcher_unicode
: sqlite_db_func_dispatcher_no_unicode,
NULL, NULL );
if ( rc != SQLITE_OK ) {
sqlite_error(dbh, rc, form("sqlite_create_function failed with error %s", sqlite3_errmsg(imp_dbh->db)));
return FALSE;
}
return TRUE;
}
#ifndef SQLITE_OMIT_LOAD_EXTENSION
return;
}
static void
sqlite_db_aggr_step_dispatcher(sqlite3_context *context,
int argc, sqlite3_value **value)
{
dTHX;
dSP;
int i, is_unicode = 0; /* TODO : find out from db handle */
aggrInfo *aggr;
aggr = sqlite3_aggregate_context(context, sizeof (aggrInfo));
if ( !aggr )
return;
ENTER;
SAVETMPS;
/* initialize on first step */
sqlite_db_aggr_new_dispatcher(aTHX_ context, aggr);
}
if ( aggr->err || !aggr->aggr_inst )
goto cleanup;
PUSHMARK(SP);
XPUSHs( sv_2mortal( newSVsv( aggr->aggr_inst ) ));
for ( i=0; i < argc; i++ ) {
XPUSHs(stacked_sv_from_sqlite3_value(aTHX_ value[i], is_unicode));
}
PUTBACK;
call_method ("step", G_SCALAR|G_EVAL|G_DISCARD);
/* Check for an error */
if (SvTRUE(ERRSV) ) {
aggr->err = newSVpvf("error during aggregator's step(): %s",
SvPV_nolen(ERRSV));
POPs;
sqlite_trace(dbh, imp_dbh, 3, form("improper collation function: '%s' is not symmetric", name));
}
/* Copy the func reference so that it can be deallocated at disconnect */
av_push( imp_dbh->functions, func_sv );
/* Register the func within sqlite3 */
rv = sqlite3_create_collation(
imp_dbh->db, name, SQLITE_UTF8,
func_sv,
imp_dbh->unicode ? sqlite_db_collation_dispatcher_utf8
: sqlite_db_collation_dispatcher
);
if ( rv != SQLITE_OK ) {
sqlite_error(dbh, rv, form("sqlite_create_collation failed with error %s", sqlite3_errmsg(imp_dbh->db)));
return FALSE;
}
return TRUE;
}
/* allocate and initialize the cursor struct */
perl_tokenizer_cursor *c;
c = (perl_tokenizer_cursor *) sqlite3_malloc(sizeof(*c));
memset(c, 0, sizeof(*c));
*ppCursor = &c->base;
/* flags for creating the Perl SV containing the input string */
flags = SVs_TEMP; /* will call sv_2mortal */
/* special handling if working with utf8 strings */
if (MY_CXT.last_dbh_is_unicode) {
/* data to keep track of byte offsets */
c->lastByteOffset = c->pInput = pInput;
c->lastCharOffset = 0;
/* string passed to Perl needs to be flagged as utf8 */
flags |= SVf_UTF8;
}
ENTER;
return SQLITE_OK;
}
static int perl_vt_Filter( sqlite3_vtab_cursor *pVtabCursor,
int idxNum, const char *idxStr,
int argc, sqlite3_value **argv ){
dTHX;
dSP;
dMY_CXT;
int i, count;
int is_unicode = MY_CXT.last_dbh_is_unicode;
ENTER;
SAVETMPS;
/* call the FILTER() method with ($idxNum, $idxStr, @args) */
PUSHMARK(SP);
XPUSHs(((perl_vtab_cursor *) pVtabCursor)->perl_cursor_obj);
XPUSHs(sv_2mortal(newSViv(idxNum)));
XPUSHs(sv_2mortal(newSVpv(idxStr, 0)));
for(i = 0; i < argc; i++) {
XPUSHs(stacked_sv_from_sqlite3_value(aTHX_ argv[i], is_unicode));
}
PUTBACK;
count = call_method("FILTER", G_VOID);
SPAGAIN;
SP -= count;
PUTBACK;
FREETMPS;
LEAVE;
return rc;
}
static int perl_vt_Update( sqlite3_vtab *pVTab,
int argc, sqlite3_value **argv,
sqlite3_int64 *pRowid ){
dTHX;
dSP;
dMY_CXT;
int count, i;
int is_unicode = MY_CXT.last_dbh_is_unicode;
int rc = SQLITE_ERROR;
SV *rowidsv;
ENTER;
SAVETMPS;
/* call the _SQLITE_UPDATE() method */
PUSHMARK(SP);
XPUSHs(((perl_vtab *) pVTab)->perl_vtab_obj);
for(i = 0; i < argc; i++) {
XPUSHs(stacked_sv_from_sqlite3_value(aTHX_ argv[i], is_unicode));
}
PUTBACK;
count = call_method ("_SQLITE_UPDATE", G_SCALAR);
SPAGAIN;
if (count != 1) {
warn("cursor->_SQLITE_UPDATE() returned %d vals instead of 1", count);
SP -= count;
}
else {
if (argc > 1 && sqlite3_value_type(argv[0]) == SQLITE_NULL
coderef = newSVsv(result);
}
/* store result in cache */
hv_store(functions, func_name, len, coderef ? coderef : &PL_sv_undef, 0);
}
/* return function information for sqlite3 within *pxFunc and *ppArg */
is_overloaded = coderef && SvTRUE(coderef);
if (is_overloaded) {
*pxFunc = MY_CXT.last_dbh_is_unicode ? sqlite_db_func_dispatcher_unicode
: sqlite_db_func_dispatcher_no_unicode;
*ppArg = coderef;
}
cleanup:
PUTBACK;
FREETMPS;
LEAVE;
sqlite3_free(func_name);
return is_overloaded;
}
#ifndef _DBDIMP_H
#define _DBDIMP_H 1
#include "SQLcipherXS.h"
#include "sqlite3.h"
#define MY_CXT_KEY "DBD::SQLcipher::_guts" XS_VERSION
typedef struct {
int last_dbh_is_unicode;
} my_cxt_t;
#define PERL_UNICODE_DOES_NOT_WORK_WELL \
(PERL_REVISION <= 5) && ((PERL_VERSION < 8) \
|| (PERL_VERSION == 8 && PERL_SUBVERSION < 5))
/* 30 second timeout by default */
#define SQL_TIMEOUT 30000
#ifndef sqlite3_int64
struct imp_drh_st {
dbih_drc_t com;
/* sqlite specific bits */
};
/* Database Handle */
struct imp_dbh_st {
dbih_dbc_t com;
/* sqlite specific bits */
sqlite3 *db;
bool unicode;
bool handle_binary_nulls;
int timeout;
AV *functions;
AV *aggregates;
SV *collation_needed_callback;
bool allow_multiple_statements;
bool use_immediate_transaction;
bool see_if_its_a_number;
int extended_result_codes;
stmt_list_s * stmt_list;
lib/DBD/SQLcipher.pm view on Meta::CPAN
}
}
}
if (my $flags = $attr->{sqlite_open_flags}) {
unless ($flags & (DBD::SQLcipher::OPEN_READONLY() | DBD::SQLcipher::OPEN_READWRITE())) {
$attr->{sqlite_open_flags} |= DBD::SQLcipher::OPEN_READWRITE() | DBD::SQLcipher::OPEN_CREATE();
}
}
# To avoid unicode and long file name problems on Windows,
# convert to the shortname if the file (or parent directory) exists.
if ( $^O =~ /MSWin32/ and $real ne ':memory:' and $real ne '' and $real !~ /^file:/ and !-f $real ) {
require File::Basename;
my ($file, $dir, $suffix) = File::Basename::fileparse($real);
# We are creating a new file.
# Does the directory it's in at least exist?
if ( -d $dir ) {
require Win32;
$real = join '', grep { defined } Win32::GetShortPathName($dir), $file, $suffix;
} else {
lib/DBD/SQLcipher.pm view on Meta::CPAN
=head2 Database Handle Attributes
=over 4
=item sqlite_version
Returns the version of the SQLcipher library which B<DBD::SQLcipher> is using,
e.g., "2.8.0". Can only be read.
=item sqlite_unicode
If set to a true value, B<DBD::SQLcipher> will turn the UTF-8 flag on for all
text strings coming out of the database (this feature is currently disabled
for perl < 5.8.5). For more details on the UTF-8 flag see
L<perlunicode>. The default is for the UTF-8 flag to be turned off.
Also note that due to some bizarreness in SQLcipher's type system (see
L<http://www.sqlite.org/datatype3.html>), if you want to retain
blob-style behavior for B<some> columns under C<< $dbh->{sqlite_unicode} = 1
>> (say, to store images in the database), you have to state so
explicitly using the 3-argument form of L<DBI/bind_param> when doing
updates:
use DBI qw(:sql_types);
$dbh->{sqlite_unicode} = 1;
my $sth = $dbh->prepare("INSERT INTO mytable (blobcolumn) VALUES (?)");
# Binary_data will be stored as is.
$sth->bind_param(1, $binary_data, SQL_BLOB);
Defining the column type as C<BLOB> in the DDL is B<not> sufficient.
This attribute was originally named as C<unicode>, and renamed to
C<sqlite_unicode> for integrity since version 1.26_06. Old C<unicode>
attribute is still accessible but will be deprecated in the near future.
=item sqlite_allow_multiple_statements
If you set this to true, C<do> method will process multiple
statements at one go. This may be handy, but with performance
penalty. See above for details.
=item sqlite_use_immediate_transaction
lib/DBD/SQLcipher.pm view on Meta::CPAN
txt2 COLLATE perllocale,
txt3 COLLATE nocase
)
or
SELECT * FROM foo ORDER BY name COLLATE perllocale
=head2 Unicode handling
If the attribute C<< $dbh->{sqlite_unicode} >> is set, strings coming from
the database and passed to the collation function will be properly
tagged with the utf8 flag; but this only works if the
C<sqlite_unicode> attribute is set B<before> the first call to
a perl collation sequence . The recommended way to activate unicode
is to set the parameter at connection time :
my $dbh = DBI->connect(
"dbi:SQLcipher:dbname=foo", "", "",
{
RaiseError => 1,
sqlite_unicode => 1,
}
);
=head2 Adding user-defined collations
The native SQLcipher API for adding user-defined collations is
exposed through methods L</"sqlite_create_collation"> and
L</"sqlite_collation_needed">.
To avoid calling these functions every time a C<$dbh> handle is
lib/DBD/SQLcipher/Fulltext_search.pod view on Meta::CPAN
=item icu
The I<icu> tokenizer uses the ICU library to decide how to
identify word characters in different languages; however, this
requires SQLcipher to be compiled with the C<SQLITE_ENABLE_ICU>
pre-processor symbol defined. So, to use this tokenizer, you need
edit F<Makefile.PL> to add this flag in C<@CC_DEFINE>, and then
recompile C<DBD::SQLcipher>; of course, the prerequisite is to have
an ICU library available on your system.
=item unicode61
The I<unicode61> tokenizer works very much like "simple" except that it
does full unicode case folding according to rules in Unicode Version
6.1 and it recognizes unicode space and punctuation characters and
uses those to separate tokens. By contrast, the simple tokenizer only
does case folding of ASCII characters and only recognizes ASCII space
and punctuation characters as token separators.
By default, "unicode61" also removes all diacritics from Latin script
characters. This behaviour can be overridden by adding the tokenizer
argument C<"remove_diacritics=0">. For example:
-- Create tables that remove diacritics from Latin script characters
-- as part of tokenization.
CREATE VIRTUAL TABLE txt1 USING fts4(tokenize=unicode61);
CREATE VIRTUAL TABLE txt2 USING fts4(tokenize=unicode61 "remove_diacritics=1");
-- Create a table that does not remove diacritics from Latin script
-- characters as part of tokenization.
CREATE VIRTUAL TABLE txt3 USING fts4(tokenize=unicode61 "remove_diacritics=0");
Additional options can customize the set of codepoints that unicode61
treats as separator characters or as token characters -- see the
documentation in L<http://www.sqlite.org/fts3.html#unicode61>.
=back
If a more complex tokenizing algorithm is required, for example to
implement stemming, discard punctuation, or to recognize compound words,
use the perl tokenizer to implement your own logic, as explained below.
=head2 Perl tokenizers
=head3 Declaring a perl tokenizer
lib/DBD/SQLcipher/VirtualTable/PerlData.pm view on Meta::CPAN
USING perl(path, dev, ino, mode, nlink, uid, gid, rdev, size,
atime, mtime, ctime, blksize, blocks,
arrayrefs="main::file_stats");
# search files
my $sth = $dbh->prepare(<<"");
SELECT * FROM file_stats
WHERE mtime BETWEEN ? AND ?
AND uid IN (...)
=head2 Hashref example : unicode characters
Given any unicode character, the L<Unicode::UCD/charinfo> function
returns a hashref with various bits of information about that character.
So this can be exploited in a virtual table :
use Unicode::UCD 'charinfo';
our $chars = [map {charinfo($_)} 0x300..0x400]; # arbitrary subrange
# create a temporary virtual table
$dbh->do(<<"");
CREATE VIRTUAL TABLE charinfo USING perl(
code, name, block, script, category,
parse_arithexpr||5.013008|
parse_barestmt||5.013007|
parse_block||5.013007|
parse_body|||
parse_fullexpr||5.013008|
parse_fullstmt||5.013005|
parse_label||5.013007|
parse_listexpr||5.013008|
parse_stmtseq||5.013006|
parse_termexpr||5.013008|
parse_unicode_opts|||
parser_dup|||
parser_free|||
path_is_absolute|||n
peep|||
pending_Slabs_to_ro|||
perl_alloc_using|||n
perl_alloc|||n
perl_clone_using|||n
perl_clone|||n
perl_construct|||n
#endif
#ifndef PERL_PV_PRETTY_DUMP
# define PERL_PV_PRETTY_DUMP PERL_PV_PRETTY_ELLIPSES|PERL_PV_PRETTY_QUOTE
#endif
#ifndef PERL_PV_PRETTY_REGPROP
# define PERL_PV_PRETTY_REGPROP PERL_PV_PRETTY_ELLIPSES|PERL_PV_PRETTY_LTGT|PERL_PV_ESCAPE_RE
#endif
/* Hint: pv_escape
* Note that unicode functionality is only backported to
* those perl versions that support it. For older perl
* versions, the implementation will fall back to bytes.
*/
#ifndef pv_escape
#if defined(NEED_pv_escape)
static char * DPPP_(my_pv_escape)(pTHX_ SV * dsv, char const * const str, const STRLEN count, const STRLEN max, STRLEN * const escaped, const U32 flags);
static
#else
extern char * DPPP_(my_pv_escape)(pTHX_ SV * dsv, char const * const str, const STRLEN count, const STRLEN max, STRLEN * const escaped, const U32 flags);
#endif
};
/* One or more of the following flags are set to indicate the validOK
** representations of the value stored in the Mem struct.
**
** If the MEM_Null flag is set, then the value is an SQL NULL value.
** No other flags may be set in this case.
**
** If the MEM_Str flag is set then Mem.z points at a string representation.
** Usually this is encoded in the same unicode encoding as the main
** database (see below for exceptions). If the MEM_Term flag is also
** set, then the string is nul terminated. The MEM_Int and MEM_Real
** flags may coexist with the MEM_Str flag.
*/
#define MEM_Null 0x0001 /* Value is NULL */
#define MEM_Str 0x0002 /* Value is a string */
#define MEM_Int 0x0004 /* Value is an integer */
#define MEM_Real 0x0008 /* Value is a real number */
#define MEM_Blob 0x0010 /* Value is a BLOB */
#define MEM_AffMask 0x001f /* Mask of affinity bits */
c = ((*zIn++)<<8); \
c += (*zIn++); \
if( c>=0xD800 && c<0xE000 && TERM ){ \
int c2 = ((*zIn++)<<8); \
c2 += (*zIn++); \
c = (c2&0x03FF) + ((c&0x003F)<<10) + (((c&0x03C0)+0x0040)<<10); \
} \
}
/*
** Translate a single UTF-8 character. Return the unicode value.
**
** During translation, assume that the byte that zTerm points
** is a 0x00.
**
** Write a pointer to the next unread byte back into *pzNext.
**
** Notes On Invalid UTF-8:
**
** * This routine never allows a 7-bit character (0x00 through 0x7f) to
** be encoded as a multi-byte character. Any multi-byte character that
** * This routine never allows a UTF16 surrogate value to be encoded.
** If a multi-byte character attempts to encode a value between
** 0xd800 and 0xe000 then it is rendered as 0xfffd.
**
** * Bytes in the range of 0x80 through 0xbf which occur as the first
** byte of a character are interpreted as single-byte characters
** and rendered as themselves even though they are technically
** invalid characters.
**
** * This routine accepts over-length UTF8 encodings
** for unicode values 0x80 and greater. It does not change over-length
** encodings to 0xfffd as some systems recommend.
*/
#define READ_UTF8(zIn, zTerm, c) \
c = *(zIn++); \
if( c>=0xc0 ){ \
c = sqlite3Utf8Trans1[c-0xc0]; \
while( zIn!=zTerm && (*zIn & 0xc0)==0x80 ){ \
c = (c<<6) + (0x3f & *(zIn++)); \
} \
if( c<0x80 \
pMem->z[pMem->n+1] = '\0';
pMem->flags |= MEM_Term;
pMem->enc = bom;
}
}
return rc;
}
#endif /* SQLITE_OMIT_UTF16 */
/*
** pZ is a UTF-8 encoded unicode string. If nByte is less than zero,
** return the number of unicode characters in pZ up to (but not including)
** the first 0x00 byte. If nByte is not less than zero, return the
** number of unicode characters in the first nByte of pZ (or up to
** the first 0x00, whichever comes first).
*/
SQLITE_PRIVATE int sqlite3Utf8CharLen(const char *zIn, int nByte){
int r = 0;
const u8 *z = (const u8*)zIn;
const u8 *zTerm;
if( nByte>=0 ){
zTerm = &z[nByte];
}else{
zTerm = (const u8*)(-1);
sqlite3VdbeMemRelease(&m);
m.z = 0;
}
assert( (m.flags & MEM_Term)!=0 || db->mallocFailed );
assert( (m.flags & MEM_Str)!=0 || db->mallocFailed );
assert( m.z || db->mallocFailed );
return m.z;
}
/*
** zIn is a UTF-16 encoded unicode string at least nChar characters long.
** Return the number of bytes in the first nChar unicode characters
** in pZ. nChar must be non-negative.
*/
SQLITE_PRIVATE int sqlite3Utf16ByteLen(const void *zIn, int nChar){
int c;
unsigned char const *z = zIn;
int n = 0;
if( SQLITE_UTF16NATIVE==SQLITE_UTF16BE ){
while( n<nChar ){
READ_UTF16BE(z, 1, c);
}
default: {
assert( sqlite3_value_type(argv[0])==SQLITE_NULL );
sqlite3_result_text(context, "NULL", 4, SQLITE_STATIC);
break;
}
}
}
/*
** The unicode() function. Return the integer unicode code-point value
** for the first character of the input string.
*/
static void unicodeFunc(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
const unsigned char *z = sqlite3_value_text(argv[0]);
(void)argc;
if( z && z[0] ) sqlite3_result_int(context, sqlite3Utf8Read(&z));
}
/*
** The char() function takes zero or more arguments, each of which is
** an integer. It constructs a string where each character of the string
** is the unicode character for the corresponding integer argument.
*/
static void charFunc(
sqlite3_context *context,
int argc,
sqlite3_value **argv
){
unsigned char *z, *zOut;
int i;
zOut = z = sqlite3_malloc( argc*4+1 );
if( z==0 ){
FUNCTION(max, -1, 1, 1, minmaxFunc ),
FUNCTION(max, 0, 1, 1, 0 ),
AGGREGATE2(max, 1, 1, 1, minmaxStep, minMaxFinalize,
SQLITE_FUNC_MINMAX ),
FUNCTION2(typeof, 1, 0, 0, typeofFunc, SQLITE_FUNC_TYPEOF),
FUNCTION2(length, 1, 0, 0, lengthFunc, SQLITE_FUNC_LENGTH),
FUNCTION(instr, 2, 0, 0, instrFunc ),
FUNCTION(substr, 2, 0, 0, substrFunc ),
FUNCTION(substr, 3, 0, 0, substrFunc ),
FUNCTION(printf, -1, 0, 0, printfFunc ),
FUNCTION(unicode, 1, 0, 0, unicodeFunc ),
FUNCTION(char, -1, 0, 0, charFunc ),
FUNCTION(abs, 1, 0, 0, absFunc ),
#ifndef SQLITE_OMIT_FLOATING_POINT
FUNCTION(round, 1, 0, 0, roundFunc ),
FUNCTION(round, 2, 0, 0, roundFunc ),
#endif
FUNCTION(upper, 1, 0, 0, upperFunc ),
FUNCTION(lower, 1, 0, 0, lowerFunc ),
FUNCTION(coalesce, 1, 0, 0, 0 ),
FUNCTION(coalesce, 0, 0, 0, 0 ),
nBytes = sz;
}
sqlite3_mutex_enter(db->mutex);
zSql8 = sqlite3Utf16to8(db, zSql, nBytes, SQLITE_UTF16NATIVE);
if( zSql8 ){
rc = sqlite3LockAndPrepare(db, zSql8, -1, saveSqlFlag, 0, ppStmt, &zTail8);
}
if( zTail8 && pzTail ){
/* If sqlite3_prepare returns a tail pointer, we calculate the
** equivalent pointer into the UTF-16 string by counting the unicode
** characters between zSql8 and zTail8, and then returning a pointer
** the same number of characters into the UTF-16 string.
*/
int chars_parsed = sqlite3Utf8CharLen(zSql8, (int)(zTail8-zSql8));
*pzTail = (u8 *)zSql + sqlite3Utf16ByteLen(zSql, chars_parsed);
}
sqlite3DbFree(db, zSql8);
rc = sqlite3ApiExit(db, rc);
sqlite3_mutex_leave(db->mutex);
return rc;
Fts3Table*, Fts3MultiSegReader*, int, const char*, int);
SQLITE_PRIVATE int sqlite3Fts3MsrIncrNext(
Fts3Table *, Fts3MultiSegReader *, sqlite3_int64 *, char **, int *);
SQLITE_PRIVATE int sqlite3Fts3EvalPhrasePoslist(Fts3Cursor *, Fts3Expr *, int iCol, char **);
SQLITE_PRIVATE int sqlite3Fts3MsrOvfl(Fts3Cursor *, Fts3MultiSegReader *, int *);
SQLITE_PRIVATE int sqlite3Fts3MsrIncrRestart(Fts3MultiSegReader *pCsr);
/* fts3_tokenize_vtab.c */
SQLITE_PRIVATE int sqlite3Fts3InitTok(sqlite3*, Fts3Hash *);
/* fts3_unicode2.c (functions generated by parsing unicode text files) */
#ifndef SQLITE_DISABLE_FTS3_UNICODE
SQLITE_PRIVATE int sqlite3FtsUnicodeFold(int, int);
SQLITE_PRIVATE int sqlite3FtsUnicodeIsalnum(int);
SQLITE_PRIVATE int sqlite3FtsUnicodeIsdiacritic(int);
#endif
#endif /* !SQLITE_CORE || SQLITE_ENABLE_FTS3 */
#endif /* _FTSINT_H */
/************** End of fts3Int.h *********************************************/
}else{
sqlite3Fts3HashInit(pHash, FTS3_HASH_STRING, 1);
}
/* Load the built-in tokenizers into the hash table */
if( rc==SQLITE_OK ){
if( sqlite3Fts3HashInsert(pHash, "simple", 7, (void *)pSimple)
|| sqlite3Fts3HashInsert(pHash, "porter", 7, (void *)pPorter)
#ifndef SQLITE_DISABLE_FTS3_UNICODE
|| sqlite3Fts3HashInsert(pHash, "unicode61", 10, (void *)pUnicode)
#endif
#ifdef SQLITE_ENABLE_ICU
|| (pIcu && sqlite3Fts3HashInsert(pHash, "icu", 4, (void *)pIcu))
#endif
){
rc = SQLITE_NOMEM;
}
}
#ifdef SQLITE_TEST
sqlite3_result_error_code(pContext, rc);
}else{
int n = pCsr->nMatchinfo * sizeof(u32);
sqlite3_result_blob(pContext, pCsr->aMatchinfo, n, SQLITE_TRANSIENT);
}
}
#endif
/************** End of fts3_snippet.c ****************************************/
/************** Begin file fts3_unicode.c ************************************/
/*
** 2012 May 24
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
******************************************************************************
**
** Implementation of the "unicode" full-text-search tokenizer.
*/
#ifndef SQLITE_DISABLE_FTS3_UNICODE
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
/* #include <assert.h> */
/* #include <stdlib.h> */
/* #include <stdio.h> */
/* #include <string.h> */
}else{ \
*zOut++ = 0xF0 + (u8)((c>>18) & 0x07); \
*zOut++ = 0x80 + (u8)((c>>12) & 0x3F); \
*zOut++ = 0x80 + (u8)((c>>6) & 0x3F); \
*zOut++ = 0x80 + (u8)(c & 0x3F); \
} \
}
#endif /* ifndef SQLITE_AMALGAMATION */
typedef struct unicode_tokenizer unicode_tokenizer;
typedef struct unicode_cursor unicode_cursor;
struct unicode_tokenizer {
sqlite3_tokenizer base;
int bRemoveDiacritic;
int nException;
int *aiException;
};
struct unicode_cursor {
sqlite3_tokenizer_cursor base;
const unsigned char *aInput; /* Input text being tokenized */
int nInput; /* Size of aInput[] in bytes */
int iOff; /* Current offset within aInput[] */
int iToken; /* Index of next token to be returned */
char *zToken; /* storage for current token */
int nAlloc; /* space allocated at zToken */
};
/*
** Destroy a tokenizer allocated by unicodeCreate().
*/
static int unicodeDestroy(sqlite3_tokenizer *pTokenizer){
if( pTokenizer ){
unicode_tokenizer *p = (unicode_tokenizer *)pTokenizer;
sqlite3_free(p->aiException);
sqlite3_free(p);
}
return SQLITE_OK;
}
/*
** As part of a tokenchars= or separators= option, the CREATE VIRTUAL TABLE
** statement has specified that the tokenizer for this table shall consider
** all characters in string zIn/nIn to be separators (if bAlnum==0) or
** token characters (if bAlnum==1).
**
** For each codepoint in the zIn/nIn string, this function checks if the
** sqlite3FtsUnicodeIsalnum() function already returns the desired result.
** If so, no action is taken. Otherwise, the codepoint is added to the
** unicode_tokenizer.aiException[] array. For the purposes of tokenization,
** the return value of sqlite3FtsUnicodeIsalnum() is inverted for all
** codepoints in the aiException[] array.
**
** If a standalone diacritic mark (one that sqlite3FtsUnicodeIsdiacritic()
** identifies as a diacritic) occurs in the zIn/nIn string it is ignored.
** It is not possible to change the behavior of the tokenizer with respect
** to these codepoints.
*/
static int unicodeAddExceptions(
unicode_tokenizer *p, /* Tokenizer to add exceptions to */
int bAlnum, /* Replace Isalnum() return value with this */
const char *zIn, /* Array of characters to make exceptions */
int nIn /* Length of z in bytes */
){
const unsigned char *z = (const unsigned char *)zIn;
const unsigned char *zTerm = &z[nIn];
int iCode;
int nEntry = 0;
assert( bAlnum==0 || bAlnum==1 );
p->aiException = aNew;
p->nException = nNew;
}
return SQLITE_OK;
}
/*
** Return true if the p->aiException[] array contains the value iCode.
*/
static int unicodeIsException(unicode_tokenizer *p, int iCode){
if( p->nException>0 ){
int *a = p->aiException;
int iLo = 0;
int iHi = p->nException-1;
while( iHi>=iLo ){
int iTest = (iHi + iLo) / 2;
if( iCode==a[iTest] ){
return 1;
}else if( iCode>a[iTest] ){
}
}
return 0;
}
/*
** Return true if, for the purposes of tokenization, codepoint iCode is
** considered a token character (not a separator).
*/
static int unicodeIsAlnum(unicode_tokenizer *p, int iCode){
assert( (sqlite3FtsUnicodeIsalnum(iCode) & 0xFFFFFFFE)==0 );
return sqlite3FtsUnicodeIsalnum(iCode) ^ unicodeIsException(p, iCode);
}
/*
** Create a new tokenizer instance.
*/
static int unicodeCreate(
int nArg, /* Size of array argv[] */
const char * const *azArg, /* Tokenizer creation arguments */
sqlite3_tokenizer **pp /* OUT: New tokenizer handle */
){
unicode_tokenizer *pNew; /* New tokenizer object */
int i;
int rc = SQLITE_OK;
pNew = (unicode_tokenizer *) sqlite3_malloc(sizeof(unicode_tokenizer));
if( pNew==NULL ) return SQLITE_NOMEM;
memset(pNew, 0, sizeof(unicode_tokenizer));
pNew->bRemoveDiacritic = 1;
for(i=0; rc==SQLITE_OK && i<nArg; i++){
const char *z = azArg[i];
int n = (int)strlen(z);
if( n==19 && memcmp("remove_diacritics=1", z, 19)==0 ){
pNew->bRemoveDiacritic = 1;
}
else if( n==19 && memcmp("remove_diacritics=0", z, 19)==0 ){
pNew->bRemoveDiacritic = 0;
}
else if( n>=11 && memcmp("tokenchars=", z, 11)==0 ){
rc = unicodeAddExceptions(pNew, 1, &z[11], n-11);
}
else if( n>=11 && memcmp("separators=", z, 11)==0 ){
rc = unicodeAddExceptions(pNew, 0, &z[11], n-11);
}
else{
/* Unrecognized argument */
rc = SQLITE_ERROR;
}
}
if( rc!=SQLITE_OK ){
unicodeDestroy((sqlite3_tokenizer *)pNew);
pNew = 0;
}
*pp = (sqlite3_tokenizer *)pNew;
return rc;
}
/*
** Prepare to begin tokenizing a particular string. The input
** string to be tokenized is pInput[0..nBytes-1]. A cursor
** used to incrementally tokenize this string is returned in
** *ppCursor.
*/
static int unicodeOpen(
sqlite3_tokenizer *p, /* The tokenizer */
const char *aInput, /* Input string */
int nInput, /* Size of string aInput in bytes */
sqlite3_tokenizer_cursor **pp /* OUT: New cursor object */
){
unicode_cursor *pCsr;
pCsr = (unicode_cursor *)sqlite3_malloc(sizeof(unicode_cursor));
if( pCsr==0 ){
return SQLITE_NOMEM;
}
memset(pCsr, 0, sizeof(unicode_cursor));
pCsr->aInput = (const unsigned char *)aInput;
if( aInput==0 ){
pCsr->nInput = 0;
}else if( nInput<0 ){
pCsr->nInput = (int)strlen(aInput);
}else{
pCsr->nInput = nInput;
}
*pp = &pCsr->base;
UNUSED_PARAMETER(p);
return SQLITE_OK;
}
/*
** Close a tokenization cursor previously opened by a call to
** simpleOpen() above.
*/
static int unicodeClose(sqlite3_tokenizer_cursor *pCursor){
unicode_cursor *pCsr = (unicode_cursor *) pCursor;
sqlite3_free(pCsr->zToken);
sqlite3_free(pCsr);
return SQLITE_OK;
}
/*
** Extract the next token from a tokenization cursor. The cursor must
** have been opened by a prior call to simpleOpen().
*/
static int unicodeNext(
sqlite3_tokenizer_cursor *pC, /* Cursor returned by simpleOpen */
const char **paToken, /* OUT: Token text */
int *pnToken, /* OUT: Number of bytes at *paToken */
int *piStart, /* OUT: Starting offset of token */
int *piEnd, /* OUT: Ending offset of token */
int *piPos /* OUT: Position integer of token */
){
unicode_cursor *pCsr = (unicode_cursor *)pC;
unicode_tokenizer *p = ((unicode_tokenizer *)pCsr->base.pTokenizer);
int iCode = 0;
char *zOut;
const unsigned char *z = &pCsr->aInput[pCsr->iOff];
const unsigned char *zStart = z;
const unsigned char *zEnd;
const unsigned char *zTerm = &pCsr->aInput[pCsr->nInput];
/* Scan past any delimiter characters before the start of the next token.
** Return SQLITE_DONE early if this takes us all the way to the end of
** the input. */
while( z<zTerm ){
READ_UTF8(z, zTerm, iCode);
if( unicodeIsAlnum(p, iCode) ) break;
zStart = z;
}
if( zStart>=zTerm ) return SQLITE_DONE;
zOut = pCsr->zToken;
do {
int iOut;
/* Grow the output buffer if required. */
if( (zOut-pCsr->zToken)>=(pCsr->nAlloc-4) ){
/* Write the folded case of the last character read to the output */
zEnd = z;
iOut = sqlite3FtsUnicodeFold(iCode, p->bRemoveDiacritic);
if( iOut ){
WRITE_UTF8(zOut, iOut);
}
/* If the cursor is not at EOF, read the next character */
if( z>=zTerm ) break;
READ_UTF8(z, zTerm, iCode);
}while( unicodeIsAlnum(p, iCode)
|| sqlite3FtsUnicodeIsdiacritic(iCode)
);
/* Set the output variables and return. */
pCsr->iOff = (int)(z - pCsr->aInput);
*paToken = pCsr->zToken;
*pnToken = (int)(zOut - pCsr->zToken);
*piStart = (int)(zStart - pCsr->aInput);
*piEnd = (int)(zEnd - pCsr->aInput);
*piPos = pCsr->iToken++;
return SQLITE_OK;
}
/*
** Set *ppModule to a pointer to the sqlite3_tokenizer_module
** structure for the unicode tokenizer.
*/
SQLITE_PRIVATE void sqlite3Fts3UnicodeTokenizer(sqlite3_tokenizer_module const **ppModule){
static const sqlite3_tokenizer_module module = {
0,
unicodeCreate,
unicodeDestroy,
unicodeOpen,
unicodeClose,
unicodeNext,
0,
};
*ppModule = &module;
}
#endif /* !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3) */
#endif /* ifndef SQLITE_DISABLE_FTS3_UNICODE */
/************** End of fts3_unicode.c ****************************************/
/************** Begin file fts3_unicode2.c ***********************************/
/*
** 2012 May 25
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
/*
** DO NOT EDIT THIS MACHINE GENERATED FILE.
*/
#ifndef SQLITE_DISABLE_FTS3_UNICODE
#if defined(SQLITE_ENABLE_FTS3) || defined(SQLITE_ENABLE_FTS4)
/* #include <assert.h> */
/*
** Return true if the argument corresponds to a unicode codepoint
** classified as either a letter or a number. Otherwise false.
**
** The results are undefined if the value passed to this function
** is less than zero.
*/
SQLITE_PRIVATE int sqlite3FtsUnicodeIsalnum(int c){
/* Each unsigned integer in the following array corresponds to a contiguous
** range of unicode codepoints that are not either letters or numbers (i.e.
** codepoints for which this function should return 0).
**
** The most significant 22 bits in each 32-bit value contain the first
** codepoint in the range. The least significant 10 bits are used to store
** the size of the range (always at least 1). In other words, the value
** ((C<<22) + N) represents a range of N codepoints starting with codepoint
** C. It is not possible to represent a range larger than 1023 codepoints
** using this format.
*/
static const unsigned int aEntry[] = {
}else{
iHi = iTest-1;
}
}
assert( key>=aDia[iRes] );
return ((c > (aDia[iRes]>>3) + (aDia[iRes]&0x07)) ? c : (int)aChar[iRes]);
}
/*
** Return true if the argument interpreted as a unicode codepoint
** is a diacritical modifier character.
*/
SQLITE_PRIVATE int sqlite3FtsUnicodeIsdiacritic(int c){
unsigned int mask0 = 0x08029FDF;
unsigned int mask1 = 0x000361F8;
if( c<768 || c>817 ) return 0;
return (c < 768+32) ?
(mask0 & (1 << (c-768))) :
(mask1 & (1 << (c-768-32)));
}
/*
** Interpret the argument as a unicode codepoint. If the codepoint
** is an upper case character that has a lower case equivalent,
** return the codepoint corresponding to the lower case version.
** Otherwise, return a copy of the argument.
**
** The results are undefined if the value passed to this function
** is less than zero.
*/
SQLITE_PRIVATE int sqlite3FtsUnicodeFold(int c, int bRemoveDiacritic){
/* Each entry in the following array defines a rule for folding a range
** of codepoints to lower case. The rule applies to a range of nRange
** to all nRange codepoints (i.e. all nRange codepoints are upper case and
** need to be folded). Or, if it is set, then the rule only applies to
** every second codepoint in the range, starting with codepoint C.
**
** The 7 most significant bits in flags are an index into the aiOff[]
** array. If a specific codepoint C does require folding, then its lower
** case equivalent is ((C + aiOff[flags>>1]) & 0xFFFF).
**
** The contents of this array are generated by parsing the CaseFolding.txt
** file distributed as part of the "Unicode Character Database". See
** http://www.unicode.org for details.
*/
static const struct TableEntry {
unsigned short iCode;
unsigned char flags;
unsigned char nRange;
} aEntry[] = {
{65, 14, 26}, {181, 64, 1}, {192, 14, 23},
{216, 14, 7}, {256, 1, 48}, {306, 1, 6},
{313, 1, 16}, {330, 1, 46}, {376, 116, 1},
{377, 1, 6}, {383, 104, 1}, {385, 50, 1},
else if( c>=66560 && c<66600 ){
ret = c + 40;
}
return ret;
}
#endif /* defined(SQLITE_ENABLE_FTS3) || defined(SQLITE_ENABLE_FTS4) */
#endif /* !defined(SQLITE_DISABLE_FTS3_UNICODE) */
/************** End of fts3_unicode2.c ***************************************/
/************** Begin file rtree.c *******************************************/
/*
** 2001 September 15
**
** The author disclaims copyright to this source code. In place of
** a legal notice, here is a blessing:
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
** May you do good and not evil.
** May you find forgiveness for yourself and forgive others.
** May you share freely, never taking more than you give.
**
*************************************************************************
** $Id: icu.c,v 1.7 2007/12/13 21:54:11 drh Exp $
**
** This file implements an integration between the ICU library
** ("International Components for Unicode", an open-source library
** for handling unicode data) and SQLite. The integration uses
** ICU to provide the following to SQLite:
**
** * An implementation of the SQL regexp() function (and hence REGEXP
** operator) using the ICU uregex_XX() APIs.
**
** * Implementations of the SQL scalar upper() and lower() functions
** for case mapping.
**
** * Integration of ICU and SQLite collation sequences.
**
** * An implementation of the LIKE operator that uses ICU to
** provide case-independent matching.
*/
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_ICU)
/* Include ICU headers */
#include <unicode/utypes.h>
#include <unicode/uregex.h>
#include <unicode/ustring.h>
#include <unicode/ucol.h>
/* #include <assert.h> */
#ifndef SQLITE_CORE
SQLITE_EXTENSION_INIT1
#else
#endif
/*
** Maximum length (in bytes) of the pattern in a LIKE or GLOB
**
*************************************************************************
** This file implements a tokenizer for fts3 based on the ICU library.
*/
#if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_FTS3)
#ifdef SQLITE_ENABLE_ICU
/* #include <assert.h> */
/* #include <string.h> */
#include <unicode/ubrk.h>
/* #include <unicode/ucol.h> */
/* #include <unicode/ustring.h> */
#include <unicode/utf16.h>
typedef struct IcuTokenizer IcuTokenizer;
typedef struct IcuCursor IcuCursor;
struct IcuTokenizer {
sqlite3_tokenizer base;
char *zLocale;
};
struct IcuCursor {
t/02_logon.t view on Meta::CPAN
ok( defined $dbh->$call_func(0, 'busy_timeout') );
is( $dbh->$call_func('busy_timeout'), 0, 'Set busy_timeout to 0' );
}
# Attributes in the connect string
SKIP: {
unless ( $] >= 5.008005 ) {
skip( 'Unicode is not supported before 5.8.5', 2 );
}
my $file = 'foo'.$$;
my $dbh = DBI->connect( "dbi:SQLcipher:dbname=$file;sqlite_unicode=1", '', '' );
isa_ok( $dbh, 'DBI::db' );
is( $dbh->{sqlite_unicode}, 1, 'Unicode is on' );
$dbh->disconnect;
unlink $file;
}
# dbname, db, database
SCOPE: {
for my $key (qw/database db dbname/) {
my $file = 'foo'.$$;
unlink $file if -f $file;
ok !-f $file, 'database file does not exist';
t/12_unicode.t view on Meta::CPAN
#!/usr/bin/perl
# This is a test for correct handling of the "unicode" database
# handle parameter.
use strict;
BEGIN {
$| = 1;
$^W = 1;
}
use t::lib::Test;
use Test::More;
t/12_unicode.t view on Meta::CPAN
! is_utf8($bytestring),
'$bytestring should *NOT* be marked as UTF-8 by Perl',
);
# Sends $ain and $bin into TEXT resp. BLOB columns the database, then
# reads them again and returns the result as a list ($aout, $bout).
### Real DBD::SQLcipher testing starts here
my ($textback, $bytesback);
SCOPE: {
my $dbh = connect_ok( dbfile => 'foo', RaiseError => 1 );
is( $dbh->{sqlite_unicode}, 0, 'Unicode is off' );
ok(
$dbh->do("CREATE TABLE table1 (a TEXT, b BLOB)"),
'CREATE TABLE',
);
($textback, $bytesback) = database_roundtrip($dbh, $bytestring, $bytestring);
ok(
! is_utf8($bytesback),
"Reading blob gives binary",
t/12_unicode.t view on Meta::CPAN
ok(
! is_utf8($textback),
"Reading text gives binary too (for now)",
);
is($bytesback, $bytestring, "No blob corruption");
is($textback, $bytestring, "Same text, different encoding");
}
# Start over but now activate Unicode support.
SCOPE: {
my $dbh = connect_ok( dbfile => 'foo', sqlite_unicode => 1 );
is( $dbh->{sqlite_unicode}, 1, 'Unicode is on' );
($textback, $bytesback) = database_roundtrip($dbh, $utfstring, $bytestring);
ok(! is_utf8($bytesback), "Reading blob still gives binary");
ok(is_utf8($textback), "Reading text returns UTF-8");
ok($bytesback eq $bytestring, "Still no blob corruption");
ok($textback eq $utfstring, "Same text");
my $lengths = $dbh->selectall_arrayref(
"SELECT length(a), length(b) FROM table1"
t/13_create_collation.t view on Meta::CPAN
delete $tied->{foo};
$DBD::SQLcipher::COLLATION{foo} = \&by_num_desc; # override, no longer dies
is($DBD::SQLcipher::COLLATION{foo}, \&by_num_desc, "overridden collation");
# now really test the collation functions
foreach my $call_func (@CALL_FUNCS) {
for my $use_unicode (0, 1) {
# connect
my $dbh = connect_ok( RaiseError => 1, sqlite_unicode => $use_unicode );
# populate test data
my @words = qw{
berger Bergèòe bergèòe Bergere
HOT hôôe
héôéòoclite héôaïòe hêôre héòaut
HAT hâôer
féôu fêôe fèöe ferme
};
if ($use_unicode) {
utf8::upgrade($_) foreach @words;
}
$dbh->do( 'CREATE TEMP TABLE collate_test ( txt )' );
$dbh->do( "INSERT INTO collate_test VALUES ( '$_' )" ) foreach @words;
# test builtin collation "perl"
my @sorted = sort @words;
my $db_sorted = $dbh->selectcol_arrayref("$sql COLLATE perl");
is_deeply(\@sorted, $db_sorted, "collate perl (@sorted // @$db_sorted)");
t/33_non_latin_path.t view on Meta::CPAN
RaiseError => 1,
PrintError => 0,
} );
isa_ok( $dbh, 'DBI::db' );
};
is( $@, '', "Could connect to database in $subdir" );
diag( $@ ) if $@;
unlink(_path($dbfile)) if -e _path($dbfile);
# Repeat with the unicode flag on
my $ufile = $dbfile;
eval {
my $dbh = DBI->connect("dbi:SQLcipher:dbname=$dbfile", undef, undef, {
RaiseError => 1,
PrintError => 0,
sqlite_unicode => 1,
} );
isa_ok( $dbh, 'DBI::db' );
};
is( $@, '', "Could connect to database in $subdir" );
diag( $@ ) if $@;
# Reopen the database
eval {
my $dbh = DBI->connect("dbi:SQLcipher:dbname=$dbfile", undef, undef, {
RaiseError => 1,
PrintError => 0,
sqlite_unicode => 1,
} );
isa_ok( $dbh, 'DBI::db' );
};
is( $@, '', "Could connect to database in $subdir" );
diag( $@ ) if $@;
unlink(_path($ufile)) if -e _path($ufile);
# when the name of the database file has non-latin characters
my $dbfilex = catfile($dir, "$subdir.db");
t/37_regexp.t view on Meta::CPAN
}
}
use locale;
use DBD::SQLcipher;
foreach my $call_func (@CALL_FUNCS) {
for my $use_unicode (0, 1) {
# connect
my $dbh = connect_ok( RaiseError => 1, sqlite_unicode => $use_unicode );
# The following tests are about ordering, so don't reverse!
if ($dbh->selectrow_array('PRAGMA reverse_unordered_selects')) {
$dbh->do('PRAGMA reverse_unordered_selects = OFF');
}
# populate test data
my @vals = @words;
if ($use_unicode) {
utf8::upgrade($_) foreach @vals;
}
$dbh->do( 'CREATE TEMP TABLE regexp_test ( txt )' );
$dbh->do( "INSERT INTO regexp_test VALUES ( '$_' )" ) foreach @vals;
foreach my $regex (@regexes) {
my @perl_match = grep {/$regex/} @vals;
my $sql = "SELECT txt from regexp_test WHERE txt REGEXP '$regex' "
. "COLLATE perllocale";
t/43_fts3.t view on Meta::CPAN
plan skip_all => 'Unicode is not supported before 5.8.5';
}
if (!grep /ENABLE_FTS3/, DBD::SQLcipher::compile_options()) {
plan skip_all => 'FTS3 is disabled for this DBD::SQLcipher';
}
}
use Test::NoWarnings;
my $num = has_sqlite('3.7.4') ? 4 : 2;
plan tests => $num * @tests # each test with unicode y/n and with fts3/fts4
+ 2 # connect_ok with unicode y/n
+ 1; # Test::NoWarnings
BEGIN {
# Sadly perl for windows (and probably sqlite, too) may hang
# if the system locale doesn't support european languages.
# en-us should be a safe default. if it doesn't work, use 'C'.
if ( $^O eq 'MSWin32') {
use POSIX 'locale_h';
setlocale(LC_COLLATE, 'en-us');
}
t/43_fts3.t view on Meta::CPAN
};
};
}
use DBD::SQLcipher;
for my $use_unicode (0, 1) {
# connect
my $dbh = connect_ok( RaiseError => 1, sqlite_unicode => $use_unicode );
for my $fts (qw/fts3 fts4/) {
next if $fts eq 'fts4' && !has_sqlite('3.7.4');
# create fts table
$dbh->do(<<"") or die DBI::errstr;
CREATE VIRTUAL TABLE try_$fts
USING $fts(content, tokenize=perl 'main::locale_tokenizer')
# populate it
my $insert_sth = $dbh->prepare(<<"") or die DBI::errstr;
t/43_fts3.t view on Meta::CPAN
SKIP: {
skip "These tests require SQLcipher compiled with "
. "ENABLE_FTS3_PARENTHESIS option", scalar @tests
unless DBD::SQLcipher->can('compile_options') &&
grep /ENABLE_FTS3_PARENTHESIS/, DBD::SQLcipher::compile_options();
my $sql = "SELECT docid FROM try_$fts WHERE content MATCH ?";
for my $t (@tests) {
my ($query, @expected) = @$t;
@expected = map {$doc_ids[$_]} @expected;
my $results = $dbh->selectcol_arrayref($sql, undef, $query);
is_deeply($results, \@expected, "$query ($fts, unicode=$use_unicode)");
}
}
}
}
t/rt_25371_asymmetric_unicode.t view on Meta::CPAN
use Test::More;
BEGIN {
if ( $] >= 5.008005 ) {
plan( tests => 23 );
} else {
plan( skip_all => 'Unicode is not supported before 5.8.5' );
}
}
use Test::NoWarnings;
my $dbh = connect_ok( sqlite_unicode => 1 );
is( $dbh->{sqlite_unicode}, 1, 'Unicode is on' );
ok( $dbh->do(<<'END_SQL'), 'CREATE TABLE' );
CREATE TABLE foo (
bar varchar(255)
)
END_SQL
foreach ( "\0", "A", "\xe9", "\x{20ac}" ) {
ok( $dbh->do("INSERT INTO foo VALUES ( ? )", {}, $_), 'INSERT' );
my $foo = $dbh->selectall_arrayref("SELECT bar FROM foo");
t/rt_25924_user_defined_func_unicode.t view on Meta::CPAN
BEGIN {
if ( $] >= 5.008005 ) {
plan( tests => 15 * @CALL_FUNCS + 1);
} else {
plan( skip_all => 'Unicode is not supported before 5.8.5' );
}
}
use Test::NoWarnings;
foreach my $call_func (@CALL_FUNCS) {
my $dbh = connect_ok( sqlite_unicode => 1 );
ok($dbh->$call_func( "perl_uc", 1, \&perl_uc, "create_function" ));
ok( $dbh->do(<<'END_SQL'), 'CREATE TABLE' );
CREATE TABLE foo (
bar varchar(255)
)
END_SQL
my @words = qw{Bergère hôte hétaïre hêtre};
foreach my $word (@words) {
# rt48048: don't need to "use utf8" nor "require utf8"
utf8::upgrade($word);
ok( $dbh->do("INSERT INTO foo VALUES ( ? )", {}, $word), 'INSERT' );
my $foo = $dbh->selectall_arrayref("SELECT perl_uc(bar) FROM foo");
is_deeply( $foo, [ [ perl_uc($word) ] ], 'unicode upcase ok' );
ok( $dbh->do("DELETE FROM foo"), 'DELETE ok' );
}
$dbh->disconnect;
}
sub perl_uc {
my $string = shift;
return uc($string);
}
t/rt_71311_bind_col_and_unicode.t view on Meta::CPAN
BEGIN {
if ( $] >= 5.008005 ) {
plan( tests => 50 );
} else {
plan( skip_all => 'Unicode is not supported before 5.8.5' );
}
}
use Test::NoWarnings;
use DBI qw/:sql_types/;
my $dbh = connect_ok(sqlite_unicode => 1);
$dbh->do('create table test1 (id integer, b blob)');
my $blob = "\x{82}\x{A0}";
my $str = "\x{20ac}";
{
my $sth = $dbh->prepare('insert into test1 values (?, ?)');
$sth->execute(1, $blob);
t/rt_71311_bind_col_and_unicode.t view on Meta::CPAN
{
my $sth = $dbh->prepare('select * from test1 order by id');
$sth->execute;
my $expected = [undef, 1, 0, 0, 1, 1, 1];
for (1..6) {
my $row = $sth->fetch;
ok $row && $row->[0] == $_;
ok $row && utf8::is_utf8($row->[1]) == $expected->[$_],
"row $_ is ".($expected->[$_] ? "unicode" : "not unicode");
}
$sth->finish;
}
{
my $sth = $dbh->prepare('select * from test1 order by id');
$sth->bind_col(1, \my $col1);
$sth->bind_col(2, \my $col2);
$sth->execute;
my $expected = [undef, 1, 0, 0, 1, 1, 1];
for (1..6) {
$sth->fetch;
ok $col1 && $col1 == $_;
ok $col1 && utf8::is_utf8($col2) == $expected->[$_],
"row $_ is ".($expected->[$_] ? "unicode" : "not unicode");
}
$sth->finish;
}
{
my $sth = $dbh->prepare('select * from test1 order by id');
$sth->bind_col(1, \my $col1);
$sth->bind_col(2, \my $col2, SQL_BLOB);
$sth->execute;
my $expected = [undef, 0, 0, 0, 0, 0, 0];
for (1..6) {
$sth->fetch;
ok $col1 && $col1 == $_;
ok $col2 && utf8::is_utf8($col2) == $expected->[$_],
"row $_ is ".($expected->[$_] ? "unicode" : "not unicode");
}
$sth->finish;
}
{
my $sth = $dbh->prepare('select * from test1 order by id');
$sth->bind_col(1, \my $col1);
$sth->bind_col(2, \my $col2, {TYPE => SQL_BLOB});
$sth->execute;
my $expected = [undef, 0, 0, 0, 0, 0, 0];
for (1..6) {
$sth->fetch;
ok $col1 && $col1 == $_;
ok $col2 && utf8::is_utf8($col2) == $expected->[$_],
"row $_ is ".($expected->[$_] ? "unicode" : "not unicode");
}
$sth->finish;
}
t/rt_78833_utf8_flag_for_column_names.t view on Meta::CPAN
my $tests = 27;
$tests += 2 if has_sqlite('3.6.14');
plan( tests => $tests * 2 + 1 );
} else {
plan( skip_all => 'Unicode is not supported before 5.8.5' );
}
}
use Test::NoWarnings;
use Encode;
unicode_test("\x{263A}"); # (decoded) smiley character
unicode_test("\x{0100}"); # (decoded) capital A with macron
sub unicode_test {
my $unicode = shift;
ok Encode::is_utf8($unicode), "correctly decoded";
my $unicode_encoded = encode_utf8($unicode);
{ # tests for an environment where everything is encoded
my $dbh = connect_ok(sqlite_unicode => 0);
$dbh->do("pragma foreign_keys = on");
my $unicode_quoted = $dbh->quote_identifier($unicode_encoded);
$dbh->do("create table $unicode_quoted (id, $unicode_quoted primary key)");
$dbh->do("create table bar (id, ref references $unicode_quoted ($unicode_encoded))");
ok $dbh->do("insert into $unicode_quoted values (?, ?)", undef, 1, "text"), "insert successfully";
ok $dbh->do("insert into $unicode_quoted (id, $unicode_quoted) values (?, ?)", undef, 2, "text2"), "insert with unicode name successfully";
{
my $sth = $dbh->prepare("insert into $unicode_quoted (id) values (:$unicode_encoded)");
$sth->bind_param(":$unicode_encoded", 5);
$sth->execute;
my ($id) = $dbh->selectrow_array("select id from $unicode_quoted where id = :$unicode_encoded", undef, 5);
is $id => 5, "unicode placeholders";
}
{
my $sth = $dbh->prepare("select * from $unicode_quoted where id = ?");
$sth->execute(1);
my $row = $sth->fetchrow_hashref;
is $row->{id} => 1, "got correct row";
is $row->{$unicode_encoded} => "text", "got correct (encoded) unicode column data";
ok !exists $row->{$unicode}, "(decoded) unicode column does not exist";
}
{
my $sth = $dbh->prepare("select $unicode_quoted from $unicode_quoted where id = ?");
$sth->execute(1);
my $row = $sth->fetchrow_hashref;
is $row->{$unicode_encoded} => "text", "got correct (encoded) unicode column data";
ok !exists $row->{$unicode}, "(decoded) unicode column does not exist";
}
{
my $sth = $dbh->prepare("select id from $unicode_quoted where $unicode_quoted = ?");
$sth->execute("text");
my ($id) = $sth->fetchrow_array;
is $id => 1, "got correct id by the (encoded) unicode column value";
}
{
my $sth = $dbh->column_info(undef, undef, $unicode_encoded, $unicode_encoded);
my $column_info = $sth->fetchrow_hashref;
is $column_info->{COLUMN_NAME} => $unicode_encoded, "column_info returns the correctly encoded column name";
}
{
my $sth = $dbh->primary_key_info(undef, undef, $unicode_encoded);
my $primary_key_info = $sth->fetchrow_hashref;
is $primary_key_info->{COLUMN_NAME} => $unicode_encoded, "primary_key_info returns the correctly encoded primary key name";
}
if (has_sqlite('3.6.14')) {
my $sth = $dbh->foreign_key_info(undef, undef, $unicode_encoded, undef, undef, 'bar');
my $foreign_key_info = $sth->fetchrow_hashref;
is $foreign_key_info->{PKCOLUMN_NAME} => $unicode_encoded, "foreign_key_info returns the correctly encoded foreign key name";
}
{
my $sth = $dbh->table_info(undef, undef, $unicode_encoded);
my $table_info = $sth->fetchrow_hashref;
is $table_info->{TABLE_NAME} => $unicode_encoded, "table_info returns the correctly encoded table name";
}
}
{ # tests for an environment where everything is decoded
my $dbh = connect_ok(sqlite_unicode => 1);
$dbh->do("pragma foreign_keys = on");
my $unicode_quoted = $dbh->quote_identifier($unicode);
$dbh->do("create table $unicode_quoted (id, $unicode_quoted primary key)");
$dbh->do("create table bar (id, ref references $unicode_quoted ($unicode_quoted))");
ok $dbh->do("insert into $unicode_quoted values (?, ?)", undef, 1, "text"), "insert successfully";
ok $dbh->do("insert into $unicode_quoted (id, $unicode_quoted) values (?, ?)", undef, 2, "text2"), "insert with unicode name successfully";
{
my $sth = $dbh->prepare("insert into $unicode_quoted (id) values (:$unicode)");
$sth->bind_param(":$unicode", 5);
$sth->execute;
my ($id) = $dbh->selectrow_array("select id from $unicode_quoted where id = :$unicode", undef, 5);
is $id => 5, "unicode placeholders";
}
{
my $sth = $dbh->prepare("select * from $unicode_quoted where id = ?");
$sth->execute(1);
my $row = $sth->fetchrow_hashref;
is $row->{id} => 1, "got correct row";
is $row->{$unicode} => "text", "got correct (decoded) unicode column data";
ok !exists $row->{$unicode_encoded}, "(encoded) unicode column does not exist";
}
{
my $sth = $dbh->prepare("select $unicode_quoted from $unicode_quoted where id = ?");
$sth->execute(1);
my $row = $sth->fetchrow_hashref;
is $row->{$unicode} => "text", "got correct (decoded) unicode column data";
ok !exists $row->{$unicode_encoded}, "(encoded) unicode column does not exist";
}
{
my $sth = $dbh->prepare("select id from $unicode_quoted where $unicode_quoted = ?");
$sth->execute("text2");
my ($id) = $sth->fetchrow_array;
is $id => 2, "got correct id by the (decoded) unicode column value";
}
{
my $sth = $dbh->column_info(undef, undef, $unicode, $unicode);
my $column_info = $sth->fetchrow_hashref;
is $column_info->{COLUMN_NAME} => $unicode, "column_info returns the correctly decoded column name";
}
{
my $sth = $dbh->primary_key_info(undef, undef, $unicode);
my $primary_key_info = $sth->fetchrow_hashref;
is $primary_key_info->{COLUMN_NAME} => $unicode, "primary_key_info returns the correctly decoded primary key name";
}
if (has_sqlite('3.6.14')) {
my $sth = $dbh->foreign_key_info(undef, undef, $unicode, undef, undef, 'bar');
my $foreign_key_info = $sth->fetchrow_hashref;
is $foreign_key_info->{PKCOLUMN_NAME} => $unicode, "foreign_key_info returns the correctly decoded foreign key name";
}
{
my $sth = $dbh->table_info(undef, undef, $unicode);
my $table_info = $sth->fetchrow_hashref;
is $table_info->{TABLE_NAME} => $unicode, "table_info returns the correctly decoded table name";
}
}
}
t/rt_96877_unicode_statements.t view on Meta::CPAN
#!/usr/bin/perl
# According to the sqlite doc, the SQL argument to sqlite3_prepare_v2
# should be in utf8, but DBD::SQLcipher does not ensure this (even with
# sqlite_unicode => 1). Only bind values are properly converted.
use strict;
BEGIN {
$| = 1;
$^W = 1;
}
use t::lib::Test;
use Test::More;
BEGIN {
if ( $] >= 5.008005 ) {
plan( tests => 16 );
} else {
plan( skip_all => 'Unicode is not supported before 5.8.5' );
}
}
use Test::NoWarnings;
my $dbh = connect_ok( sqlite_unicode => 1 );
is( $dbh->{sqlite_unicode}, 1, 'Unicode is on' );
ok( $dbh->do(<<'END_SQL'), 'CREATE TABLE' );
CREATE TABLE foo (
bar varchar(255)
)
END_SQL
foreach ( "A", "\xe9", "\x{20ac}" ) {
note sprintf "testing \\x{%x}", ord($_);
ok( $dbh->do("INSERT INTO foo VALUES ( ? )", {}, $_), 'INSERT with bind' );
t/virtual_table/21_perldata_charinfo.t view on Meta::CPAN
#!/usr/bin/perl
use strict;
BEGIN {
$| = 1;
$^W = 1;
}
# test the example described in
# L<DBD::SQLcipher::VirtualTable::PerlData/"Hashref example : unicode characters">
use t::lib::Test qw/connect_ok $sqlite_call/;
use Test::More;
BEGIN {
# check for old Perls which did not have Unicode::UCD in core
if (eval "use Unicode::UCD 'charinfo'; 1") {
plan tests => 10;
}
else {
( run in 0.478 second using v1.01-cache-2.11-cpan-f29a10751f0 )