Sunday, 11 September 2011

Minimal Sqlite library size

The Sqlite website states that the library size should be around 300KB, and that by omitting optional features it was possible to get the file size down to 180KB.

SQLite is a compact library. With all features enabled, the library size can be less than 300KiB, depending on compiler optimization settings. (Some compiler optimizations such as aggressive function inlining and loop unrolling can cause the object code to be much larger.) If optional features are omitted, the size of the SQLite library can be reduced below 180KiB.
There are also disclaimers that omitting optional features is unsupported and not guaranteed to work.
Important Note: The SQLITE_OMIT_* compile-time options are unsupported.
And there are three out of the many which do not work. When they are specified, compilation errors due to other external code depending on parts of them.
  • SQLITE_OMIT_COMPLETE
  • SQLITE_OMIT_DISKIO
  • SQLITE_OMIT_GET_TABLE
Which leaves the following options that do work:
nmake "OPTS=-DSQLITE_OMIT_ALTERTABLE -DSQLITE_OMIT_ANALYZE -DSQLITE_OMIT_ATTACH -DSQLITE_OMIT_AUTHORIZATION -DSQLITE_OMIT_AUTOINCREMENT -DSQLITE_OMIT_AUTOINIT -DSQLITE_OMIT_AUTOMATIC_INDEX -DSQLITE_OMIT_AUTORESET -DSQLITE_OMIT_AUTOVACUUM -DSQLITE_OMIT_CAST -DSQLITE_OMIT_BETWEEN_OPTIMIZATION -DSQLITE_OMIT_BLOB_LITERAL -DSQLITE_OMIT_BUILTIN_TEST -DSQLITE_OMIT_BTREE_COUNT -DSQLITE_OMIT_CHECK -DSQLITE_OMIT_COMPILEOPTION_DIAGS -DSQLITE_OMIT_COMPOUND_SELECT -DSQLITE_OMIT_DATETIME_FUNCS -DSQLITE_OMIT_DECLTYPE -DSQLITE_OMIT_EXPLAIN -DSQLITE_OMIT_FLAG_PRAGMAS -DSQLITE_OMIT_FLOATING_POINT -DSQLITE_OMIT_FOREIGN_KEY -DSQLITE_OMIT_INCRBLOB -DSQLITE_OMIT_INTEGRITY_CHECK -DSQLITE_OMIT_LIKE_OPTIMIZATION -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_OMIT_LOCALTIME -DSQLITE_OMIT_LOOKASIDE -DSQLITE_OMIT_MEMORYDB -DSQLITE_OMIT_OR_OPTIMIZATION -DSQLITE_OMIT_PAGER_PRAGMAS -DSQLITE_OMIT_PRAGMA -DSQLITE_OMIT_PROGRESS_CALLBACK -DSQLITE_OMIT_QUICKBALANCE -DSQLITE_OMIT_REINDEX -DSQLITE_OMIT_SCHEMA_PRAGMAS -DSQLITE_OMIT_SCHEMA_VERSION_PRAGMAS -DSQLITE_OMIT_SHARED_CACHE -DSQLITE_OMIT_SUBQUERY -DSQLITE_OMIT_TCL_VARIABLE -DSQLITE_OMIT_TEMPDB -DSQLITE_OMIT_TRACE -DSQLITE_OMIT_TRIGGER -DSQLITE_OMIT_TRUNCATE_OPTIMIZATION -DSQLITE_OMIT_VACUUM -DSQLITE_OMIT_UTF16 -DSQLITE_OMIT_VIEW -DSQLITE_OMIT_VIRTUALTABLE -DSQLITE_OMIT_WAL -DSQLITE_OMIT_XFER_OPT" -f makefile.msc all dll
Additionally there are some other features that can be disabled by modifying the makefile before compilation.
MAKEFILE_NAME = "Makefile.msc"
s = open(MAKEFILE_NAME, "r").read()
s = s.replace("OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_FTS3=1",
    "# OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_FTS3=1")
s = s.replace("OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_RTREE=1",
    "# OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_RTREE=1")
s = s.replace("OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_COLUMN_METADATA=1",
    "# OPT_FEATURE_FLAGS = $(OPT_FEATURE_FLAGS) -DSQLITE_ENABLE_COLUMN_METADATA=1")
s = s.replace("-O2", "-Os")
open(MAKEFILE_NAME, "w").write(s)
And gives the minimal file sizes of:
381,162 libsqlite3.lib
373,588 sqlite3.lo
382,464 sqlite3.exe
327,680 sqlite3.dll
This is not quite as simple (or perhaps achievable) as advertised. My current assumption is that the Sqlite website is four or five years out of date in describing what is possible.

4 comments:

  1. Hi,

    can you do the same with pythonXX.dll?

    I just found that python23.dll is only 975KB. Very small runtime, while python24.dll is 1.8MB.

    ReplyDelete
  2. Python is pretty hefty. If small file size is important, you should perhaps look at tinypy.

    ReplyDelete
  3. This is not the correct way to do omits as http://www.sqlite.org/compile.html#omitfeatures tells you.

    The SQLite code includes a SQL parser and virtual machine query execution engine amongst other things. You have to regenerate the parser, VM and supporting code with the OMIT options in order to get the actual size reductions.

    This is best done by producing a new amalgamation with the OMIT options (and a lot less code). Making a new amalgamation is easiest done on a Linux box as it needs various scripting tools (eg TCL, awk), but the resulting amalgamation can be used on any platform.

    ReplyDelete
  4. Excellent news Roger, thanks. I will give it another shot.

    ReplyDelete