Deploying android apps with large databases Mar 13

One issue that took some time to solve while developing my last android app is that in this application there is a large database that I've prepopulated before deploying to the market. Even on the emulator I was typically seeing the initial deployment take 5-10 minutes to populate the database (roughly 8-10K rows including a large amount of textual data). This was unacceptable and needed to be resolved before I could deploy the application.

I began searching the internet for a solution and came across a few partials. I found a Stack Overflow post on shipping databases with android applications and a blog with slightly more detail talking about pre-packaging the database. I began implementing the solution discussed in the second reference when I discovered android prevents raw files larger than 1MB from being opened. Well my database after being prepopulated was about 2MB. I began trying to open it as an asset then as a raw binary file, nothing worked.

I thought about it and realized that I've run into a similar issue earlier on in my development. I was initially loading the database from a text file by parsing and making individual queries (this is what took the 5-10 minutes). In this case the text file couldn't be opened either, instead I had to break it into two parts and open each part separately. I thought this solution may also work on the database. began by splitting my database file into pieces.

Both linux and cygwin provide the split utility to break a file into multiple pieces. For more information on split see the man page.

split inputFile -b 1048576 outputPrefix

This will restrict each piece to be 1MB or less which is the limit enforced by android. In my case it split the database into two pieces. Now the code! I've only modified a few parts to the second example previously referenced so I'll spare you the duplication.

Move the pieces to the res/raw directory of your application. This is the first change, I like being able to access the files using their generated ids which isn't possible from the assets directory. Just remember when the class loads to grab a reference to the resources object.

public DataBaseHelper(Context context) {
    super(context, DB_NAME, null, 1);
    resources = context.getResources();
}

Now the copyDataBase method which is the second change.

private void copyDataBase() throws IOException{
    InputStream databaseInput = null;
    String outFileName = DB_PATH + DbConsts.DATABASE_NAME;
    OutputStream databaseOutput = new FileOutputStream(outFileName);

    byte[] buffer = new byte[1024];
    int length;

    databaseInput = resources.openRawResource(R.raw.databaseFileA);
    while((length = databaseInput.read(buffer)) > 0) {
        databaseOutput.write(buffer, 0, length);
        databaseOutput.flush();
    }
    databaseInput.close();

    databaseInput = resources.openRawResource(R.raw.databaseFileB);
    while((length = databaseInput.read(buffer)) > 0) {
        databaseOutput.write(buffer);
        databaseOutput.flush();
    }
    databaseInput.close();
    databaseOutput.flush();
    databaseOutput.close();
}

tags: android databases tutorial

Comments

Ovidiu Aug 27

Thanks man!!! Awesome post.

Kiran Nov 23

wasted couple of days ... I saw this post before but didn't read carefully.. and thought it would never work... I take my words back..

Thanks

Kiran Nov 23

I think I celebrated little early... I am seeing file size is almost 20mb ... and I just have android_metadata table.. whats wrong ??

Petar Jul 9

Hi I tried the method presented however when I ran my app I got a "android.database.sqlite.SQLiteDatabaseCorruptException: database disk image is malformed" Error. Has anyone come across this and if so how can it be fixed?

Thanks

bosha Sep 14

Great post. Thanks.

bosha Sep 14

Great post. Thanks.

halilenver Mar 22

Petar, I have same problem

here is my logs: 03-22 20:52:49.401 I/Database( 2487): sqlite returned: error code = 11, msg = database corruption at line 46886 of [42537b6056] 03-22 20:52:49.401 I/Database( 2487): sqlite returned: error code = 11, msg = database disk image is malformed 03-22 20:52:49.401 I/FFLOG ( 2487): FF_CORRUPTION:database disk image is malformed: , while compiling: SELECT Names FROM MemberList WHERE PageNo = 1

It works in some phones just perfectly. but does not work in others. I copies database, then when I try to get a query it fails. please help me.

btw, I do not think it is about query's sql codes. if it were, i would not work in any phone.

Ayon Jul 21

Awesome tutorial , I am greately benifited....thanks a lot

Mehdi Zonjy May 9

I love your approach, however, there's a major problem with it. If i'm not mistaken after you finish merging the database files, you're gonna end up consuming twice the storage space, since now you are keeping two versions of the database. The first is the split version which is stored in the RAW directory; The second is the merged version, which is stored in the DATA directory. this could be a problem if the database is huge such as in my case. I've been trying to find a away to remove files from the Asset,Resource folders, however it seems you only have a write access on these directories.

(Required)

(Optional)

(Optional)

Are you human (yes/no)?



PREVIEW