Deploying android apps with large databases Mar 13

One issue that took some time to solve while developing my last android app is that in this application there is a large database that I've prepopulated before deploying to the market. Even on the emulator I was typically seeing the initial deployment take 5-10 minutes to populate the database (roughly 8-10K rows including a large amount of textual data). This was unacceptable and needed to be resolved before I could deploy the application.

I began searching the internet for a solution and came across a few partials. I found a Stack Overflow post on shipping databases with android applications and a blog with slightly more detail talking about pre-packaging the database. I began implementing the solution discussed in the second reference when I discovered android prevents raw files larger than 1MB from being opened. Well my database after being prepopulated was about 2MB. I began trying to open it as an asset then as a raw binary file, nothing worked.

I thought about it and realized that I've run into a similar issue earlier on in my development. I was initially loading the database from a text file by parsing and making individual queries (this is what took the 5-10 minutes). In this case the text file couldn't be opened either, instead I had to break it into two parts and open each part separately. I thought this solution may also work on the database. began by splitting my database file into pieces.

Both linux and cygwin provide the split utility to break a file into multiple pieces. For more information on split see the man page.

split inputFile -b 1048576 outputPrefix

This will restrict each piece to be 1MB or less which is the limit enforced by android. In my case it split the database into two pieces. Now the code! I've only modified a few parts to the second example previously referenced so I'll spare you the duplication.

Move the pieces to the res/raw directory of your application. This is the first change, I like being able to access the files using their generated ids which isn't possible from the assets directory. Just remember when the class loads to grab a reference to the resources object.

public DataBaseHelper(Context context) {
    super(context, DB_NAME, null, 1);
    resources = context.getResources();
}

Now the copyDataBase method which is the second change.

private void copyDataBase() throws IOException{
    InputStream databaseInput = null;
    String outFileName = DB_PATH + DbConsts.DATABASE_NAME;
    OutputStream databaseOutput = new FileOutputStream(outFileName);

    byte[] buffer = new byte[1024];
    int length;

    databaseInput = resources.openRawResource(R.raw.databaseFileA);
    while((length = databaseInput.read(buffer)) > 0) {
        databaseOutput.write(buffer, 0, length);
        databaseOutput.flush();
    }
    databaseInput.close();

    databaseInput = resources.openRawResource(R.raw.databaseFileB);
    while((length = databaseInput.read(buffer)) > 0) {
        databaseOutput.write(buffer);
        databaseOutput.flush();
    }
    databaseInput.close();
    databaseOutput.flush();
    databaseOutput.close();
}

tags: android databases tutorial

Accessing the android emulator database Mar 11

I've begun developing on the android platform and I thought I'd write a series of articles on things I've learned since I started. Today's is going to be on command line access to the embedded sqlite database. Access to the database can be obtained using the adb tool (which is a pretty amazing tool if you haven't tried it). The adb tool lets you interact with an emulator while it's running. Two of the uses I'll be talking about are its ability to navigating the internal file system through the shell and transferring files between your local file system and the emulator file system.

To use the adb tool, add the android SDK tools directory to your path. Once the tools directory is part of the path open a command prompt. On the command prompt open the shell by executing:

adb shell

For this command to be successful the emulator must be running. Adb attaches itself to the emulator and provides command line access within the emulator. Once attached, navigate to your application's directory by typing

cd data/data/<application package>

This directory is where your application stores it's data.

From here enter the databases directory

cd databases

Any databases created by your application will be present in this directory. The databases directory is only created once your application has run within the emulator and created the database. If you have not run your app or if it doesn't interact with the database then this directory will not exist. Once in the database directory, you should see a database file named by your application. This is your application's database, now lets begin interacting with it!

Command-line access to the database is provided by sqlite3. On the command line try executing

sqlite3 myapp.db

This will open the database and allow any typical sql commands to be executed. To see a listing of sqlite commands, type '.help' which will display a usage of all commands. A couple of the more useful commands to begin with are .tables to list all tables in the database and '.schema table' to display the schema used to create the table.

If you prefer one of the many sqlite graphical interfaces instead of the command line access. The following describes steps to extract the sqlite database from the emulator to use on your PC with your favorite tools. The adb tool can push and pull files between your local file system and the one on the emulator. To pull the database out of the emulator, the adb tool provides a pull command with the following syntax

adb pull <remote> <local>  
For example: adb pull data/data/<application package>/databases/app.db local_file.db

Once the database file is pulled from the emulator it can be opened with any of your favorite sqlite tools.

tags: android databases sqlite tutorial

Auto-Discover JAVA_HOME on Windows Jun 15

It's bothered me for a while now that many open source distributions and software packages include batch files that depend on JAVA_HOME being defined and pointing to the correct location. There must be a way around this dependency since people may not have the JAVA_HOME environment variable defined properly. The following is a small tutorial describing a method for automatically discovering JAVA_HOME on a user's machine from a batch file. This may be useful for anyone wanting to setup JAVA_HOME automatically before starting their java application.

After doing a bit of research I discovered that Java, during its installation, creates several useful registry entries. The section of interest in this case is:

HKLM > Software > JavaSoft > Java Runtime Environment

Within the 'Java Runtime Environment' are several registry keys along with directories for each version of Java that has been installed on the user's machine. Among the keys includes one called CurrentVersion. This key points to the current version of Java that has been installed on the machine. By this I mean it references one of the directories under 'Java Runtime Environment' by name. That directory contains a key called JavaHome. The value of this key is a filesystem path to an installation of Java.

Now that I have found a way to determine the current version of Java installed on the user's machine and the path to that installation there must be a way to write it up within a batch file to automatically discover JAVA_HOME if it is undefined.

The batch command that accomplishes just this is 'FOR' which is able to execute a command and store its return in a variable. Combining 'FOR' with 'REG QUERY' allows registry keys to be queried for their values, parses the response and stores the result in a variable.

FOR /F "skip=2 tokens=2*" %%A IN 
    ('REG QUERY "HKLM\Software\JavaSoft\Java Runtime Environment" 
    /v CurrentVersion') DO set CurVer=%%B

FOR /F "skip=2 tokens=2*" %%A IN 
    ('REG QUERY "HKLM\Software\JavaSoft\Java Runtime Environment\%CurVer%" 
    /v JavaHome') DO set JAVA_HOME=%%B

Now taking advantage of those two commands we can put together a simple batch file that checks the contents of JAVA_HOME and if empty runs the automatic discovery.

@echo off
if "%JAVA_HOME%"=="" call:FIND_JAVA_HOME
echo "Java home: %JAVA_HOME%"
rem Start your java application or perform other actions here
goto:END

:FIND_JAVA_HOME
FOR /F "skip=2 tokens=2*" %%A IN 
    ('REG QUERY "HKLM\Software\JavaSoft\Java Runtime Environment" 
    /v CurrentVersion') DO set CurVer=%%B

FOR /F "skip=2 tokens=2*" %%A IN 
    ('REG QUERY "HKLM\Software\JavaSoft\Java Runtime Environment\%CurVer%" 
    /v JavaHome') DO set JAVA_HOME=%%B
goto:EOF

:END

My batch scripting knowledge is limited so there may be more efficient ways of writing this script. If that is the case please comment.

tags: batch java script tutorial

Simple Django Commenting System with Ajax May 10

I have begun incorporating ajax techniques using the script.aculo.us and prototype libraries and thought I'd share a basic example that combines ajax with django. The snippets shown here have been taken and simplified for the purpose of this article. Making good use of ajax can significantly improve the usability of a website as it introduces a more user friendly experience. Granted sites can go overboard especially with all the effects. In this article I thought I would demonstrate a simple commenting system that uses the Prototype library to create asynchronous calls necessary for submitting a form while remaining integrated with the django framework.

Model

Lets start with the model describing the commenting class. It contains the basics for user submitted comments, name, email, website, content, reference, etc. The key represents an attribute for retrieving a set of comments. For example, combining the comments with articles the key could store the article's title or slug. When an article is loaded the comments can easily be retrieved based on their title. The name, email and website contain information on the user submitting a comment. The email and website fields are optional in this case. The text refers to the comment being submitted by the user and created date is a timestamp automatically assigned when the user performs a submission.

class Comment(models.Model):
    key = models.CharField(max_length=200)
    name = models.CharField(max_length=50)
    email = models.CharField(max_length=100, blank=True)
    website = models.CharField(max_length=200, blank=True)
    text = models.TextField()
    created_date = models.DateTimeField(auto_now_add=True)

Form

Extending the ModelForm provides some built in features when processing comments that are submitted. In this case several fields are excluded to construct a form only displaying the name, email website and text fields of the comment. Notice the key and created_date fields have been left out. The created_date will be automatically filled when the record is added to the database and the key field will be filled by the view prior to committing the record.

class CommentForm(ModelForm):
    class Meta:
        model = Comment
        fields = ('name', 'email', 'website', 'text')

URL

The URL used to direct comment submissions to the correct callback method.

(r'^comment/(?P<key>\d+)/$','comments.views.add_comment')

HTML

The HTML used to display the comment form. In this case the key being associated with comments is the article's slug which is described in the URL used to submit the comment.

<div id="response">
    <form id="comment_form" action="/comment/{{ article.slug }}/"
        method="post" onSubmit="ajaxSendComment(); return false;">
        <p>{{ comment_form.name }} {{ comment_form.name.label_tag }}(Required)</p>
        <p>{{ comment_form.name }} {{ comment_form.email.label_tag }}(Optional)</p>
        <p>{{ comment_form.name }} {{ comment_form.website.label_tag }}(Optional)</p>
        <p>{{ comment_form.name }} </p>
        <p><input id="submit" type="Submit" value="Submit" /></p>
    </form>
</div>

This is basic HTML to display the list of comments. It can be encapsulated in an HTML file which is then called by the View to return the updated comments.

<div id="comment_texts">
    <h1>Comments</h1>
    {% if object.get_comments %}
    {% for comment in object.get_comments %}
        <div id="comment">
            <div id="comment-title">
                <p class="comment-name">
            {{ comment.name }} <span>{{ comment.created_date|date:"M j" }}</span>
        </p>
        </div>
        <div id="comment-body">
            <p class="comment-text">{{ comment.text|safe }}</p>
        </div>
        </div>
    {% endfor %}
    {% else %}
    <p> No comments have been added </p>
    {% endif %}
</div>

Javascript

The javascript used in this example makes use of the prototype library. The html headers should import the prototype library.

Here is the meat of the code running on the client side to initiate the asynchronous operations. The form contains an attribute that onSubmit calls the ajaxSendComment method. The method obtains the form object using the $('form_id') notation provided by the prototype library. Using the form object the request method can be invoked. The request submits the form; the onSuccess callback processes the response. The callback updates the existing user submitted comments to include the user's comment. The code also disables the submit button while processing the request and on successful completion clears the form allowing a new comment to be entered.

<script type="text/javascript">
    function ajaxSendComment() {
        var form = $('comment_form');

        form.request({
            onSuccess: function(transport) {
                var result_field = $('comment_texts');
                var submit_button = $('submit');

                submit_button.disabled = true;
                submit_button.value = "Processing.  Please Wait...";

                if(transport.status == 200) {
                    form.reset();
                }
                result_field.update(transport.responseText);

                submit_button.value = 'Submit';
                submit_button.disabled = false;

            },
        });
    }
</script>

View

Now the server side meat. When the comment is posted, part of the URL describes the key that should be associated with the comment. The key is captured by the URL parser and passed into the add_comment view. The form is loaded from the post arguments and validated. The comment is initially saved with the parameter commit=false to generate the Comment object but not actually saving it into the database. This is done to obtain the object but still allows post processing of the object prior to database insertion. Once the object is obtained, the key is set to allow retrieval. The comment is then saved again committing it to the database. The set of new comments is then passed back as the response allowing the user to see the comment that was just submitted.

def add_comment(request, key):
    if request.method == 'POST':
        form = CommentForm(request.POST, request.FILES)
        if form.is_valid():
            cmt = form.save(commit=False)
            cmt.key = key
            cmt.save()
comments = Comment.objects.filter(key__exact=key)
    return render_to_response('comments/comment_listing.html', {'comments-all': comments})

tags: ajax django javascript prototype

Great Students Are Great Employees Not Founders Apr 8

I have read several articles recently, one from the New York Times and another from Forbes that discuss the necessity (or lack of) an advanced education. The article from the NYTimes is primarily on Marissa Mayer's life and work at Google. If you skip ahead to page 3 it gets into her hiring practices and what she looks for and considers when hiring. The article from Forbes is on the world's billionaires and in part summarizes the level of education for people appearing on the list.

The hiring practices of Marissa Mayer are described as:

She homes in on grade-point averages and SAT scores to narrow a list of candidates, many having graduated from Ivy League schools, whom she wanted to meet as part of a program to foster in-house talent.

And in the case of one candidate's resume brought on the following remark.

One candidate got a C in macroeconomics. "That's troubling to me," Ms. Mayer says. "Good students are good at all things."

This mentality resembles that of top college recruitment processes. Colleges want to recruit students that are best able to follow the rules. What I mean here is that there are many people that have great grades and exceptional SAT scores, but they don't immediately get accepted to Ivy League schools. It's the people that write the essays that the recruitment panels want to read, join groups and take positions that look good on an application. The hiring process with large companies like Google, Microsoft, Apple, etc. want these same qualities in their employees. That being, employees that are able to do exactly what is asked of them.

This selection process differs from the selection taken to produce the people appearing on Forbes billionaires list. The selection of great founders doesn't depend on schooling, good grades and following the rules. In this case success depends on the ability to think outside the box, to see improvements in society and act on those ideas to build something great. One comment from the article is that many people appearing in the list never had any advanced education.

More than 20% of the 292 of the self-made American billionaires on the most recent list of the World's Billionaires have either never started or never completed college. This is especially true of those destined for careers as technology entrepreneurs.

These characteristics are less likely to work at a larger company because to those companies change happens slowly. They are less likely to lead the way breaking new ground, waiting instead for ideas to be proven. Especially ideas presented by entry level employees that have just graduated college wanting to change the world.

Characteristics of great founders versus great employees obviously doesn't hold true for everyone. I have met people from both sides that didn't fit the norm. Employees that graduated with 4.0s but were so brainwashed with a college mentality that thinking abstractly where tasks aren't just bulleted lists doesn't work. However, I can also understand why larger companies prefer graduates from prestigious schools. And I say good! I'd much rather be working along side people wanting to change the world than someone good at completing a task regardless if it is right or wrong.

My view is biased on the software and technology industries. I wonder how accurate this is within other industries.

tags: founders students wealth

Selecting Web Hosting Mar 16

I've been interested in finding out what it takes to attract traffic to a site and what type of money it draws in with Google's Adsense program. So along with building a website I'm also going to learn python and django to develop the site. So recently I've been spending some of my free time going through django tutorials and writing small applications to get the hang of python. I've also discovered not to enter python into a browser and push ctrl-enter at work because python.com is very different from python.org. What I found while performing the initial implementation and developing a basic blogging application is that it's not the python and django programming where I ran into most of the problems rather all of the setup of apache, the database, finding web hosting, etc. The following is going to be talking about my experience searching for a good host for my django applications.

My initial criteria when selecting a company to provide hosting is first to be inexpensive. This criteria was key since I'll be primarily using the hosting for a blog and any experimentation so I didn't want to start dumping buckets of money into this project. Along with the financial aspect, the web host must support python and django (obviously!), and is able to provide useful support with quick response.

Given that I wanted an inexpensive option that supported python and django, my selection of hosting sites was limited. Since this is is my first experience with web hosting I also wanted a company that would be supportive of any issues I encountered. I began my search, where else, with Google and found two (DjangoProject and DjangoFriendly) lengthy listings of web hosts that support django to some extent. Although these provided lengthy listings there was very little covering their support, customer service or ease of use with django.

I discovered on StackOverflow and HackerNews several helpful articles that narrowed down the listings with good recommendations. From those I gathered WebFaction, SliceHost, and MochaHost. SliceHost was eliminated quickly because the cost of their cheapest plan started at $20 dollars a month. Now left with WebFaction and MochaHost the choice became much more difficult.

The benefits I found with WebFaction were that it gave complete shell access to the server and applications could be installed in the user's home directory. The cost of hosting is $8.50 a month with a yearly plan. With MochaHost only their second and third tiered plans support django. The cost of their second tiered plan is $6.62 with purchase of a yearly plan. Purchase of this plan also includes a lifetime domain name but MochaHost does not allow shell access. I contacted both company's support team with questions on their service, instructions on what is required to support django, which plan is necessary, how difficult is it to setup to figure out their knowledge of django and gain an understanding of their response time requesting support. Both companies had excellent response time of several hours and responded with links to forum articles and detailed explanations to my questions.

After fully reviewing both companies I decided on the MochaHost service. Both companies seemed exceptional however the cost savings and the convenience having the domain registration and web hosting all maintained by one company was the deciding factor. Since my purchase the service I've received has been amazing. From the time I signed up and submitted the request for service my domain was purchased and hosting setup within the hour. It was simple to take the website I had been developing locally and transition it to their servers using following written tutorials and documentation that MochaHost provided.

tags: django hosting mochahost python web hosting

The site is running! Mar 15

Welcome, here you can find my personal site. I'm not sure exactly how I'll be using this space yet but there is more to come.

Check back shortly for updates.