Blog posts tagged: python

Since my company was acquired, I needed to get my website off of the new company's servers. My site was built with Django, and I didn't want to have to pay for hosting.

I was at the same time reading a book on Google App Engine (GAE), and realized this would be a great fit. GAE offers generous initial quotas for free and I think I can fit within the free quota limits of GAE for a while. The free quotas equate to 1 GB of persistent storage and enough CPU and bandwidth for about 5 million page views a month.

As of a few days ago, the site was fully converted to a live GAE site. The page you are reading was served from GAE.

The entire conversion process only took a day and a bit for bug fixing, including coding an administrative back-end. To import the data from the old site I used the GAE shell which interfaced into sqlite and my local GAE instance, and then used the GAE bulk exporter from my local datastore and bulk importer to the remote datastore. Alternatively, I could have used the Remote API if I wanted to go directly to the remote site. If this all sounds foreign to you, and you are interested in GAE, buy the book from the review below!

As most Django websites, my existing site was based on a relational database (in particular sqlite). You can't use relational databases with GAE, so most of the work in converting the site was in converitng the Django models and queries to the datastore's equivalents. In particular I had to re-code the blogs database, comments, tags, and syndication using Google's datastore and GQL.

In addition, the Django administration console didn't work since it relies heavily on the database back-end. I made a small administration section myself using Django forms.

GQL by the way looks exactly like SQL, but doesn't allow joins, and can't select partial entities which exist in your datastore.

GAE doesn't support Django 1.3 yet, so I used 1.2 and the Google App Engine Django project. The Google App Engine Django project interops GAE with Django easily. I was using Django 0.96 on my old site so this was already an improvement. The Google App Engine Django project is a great project which allows access to many Django features including using manage.py, sending mail, and Django's test framework.

The only downside of GAE is that if Google ever stops offering their service, I'd need to go through another conversion. There are already several replacement systems though which implement the same platform that you can host yourself, although I've heard all are still missing some functionality.

After watching the launch videos of Google App Engine at Google Campfire (6 part video), I was surprised to hear that Guido van Rossum works on the Google App Engine team. Apparently 50% of his time is still spent working on managing the Python language itself.

★★★★☆ (4 stars out of 5)

I purchased the Programming Google App Engine book by Dan Sanderson because I wanted to learn more about Google App Engine (GAE) on Python.

GAE is a platform and set of services for building web applications.
GAE scales your web application automatically as long as you work within their set of restrictions such as using a non relational datastore.

Although the online GAE doc has great coverage and documentation, I wanted something which could provide a little more depth. Looking up things as you need them in the GAE docs doesn't give you a high level global understanding of what's possible and how to do it.

I am left feeling confident coding for GAE after reading this book. The author has a great understanding of GAE and does a great job of explaining everything clearly. I was never bored when reading this book and the examples were great dealing with game avatars.

After reading this book you will be ready to take on a GAE project to solidify the knowledge you learnt. You will have a great understanding of GAE runtime, datastore, index design, datastore transactions, memcache, queues, scheduled tasks, webapp module, django on GAE, remote API, bulk data operations, incoming/outgoing email, incoming/outgoing XMPP and more.

You will also understand exactly what will never be possible on this platform (such as writing to the filesystem), and what will one day be possible (such as different languages and runtimes).

This book is a quick read, it is 333 pages not including the index, and if you are only interested in Python you will be skipping over about 1/3 of the book. I would have preferred if the book only focused on Python and didn't even mention Java. I would have preferred the book to be split into 2; a python version and a Java version.

In the one and a half years since the book was released, and since GAE was released April 7, 2008, there are already several additions to GAE, but everything mentioned in the book is still sound. The book will give you enough of a foundation of Google App Engine to continue on.

I hope future editions will include coverage on OAuth, Prospective Search, DoS Protection, the django nonrel project, and updated content throughout

This post will use Fibonacci numbers as a tool to build up to explaining the usefulness of Python generators.

This post will feature both C++ and Python code.

Fibonacci numbers are defined as the sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ....

Or in general:

F0 = 0
F1 = 1
Fn = Fn-1 + Fn-2

This can be transferred into a C++ function extremely easily:

size_t Fib(size_t n)
{ 
    //Fib(0) = 0
    if(n == 0)
        return 0;

    //Fib(1) = 1
    if(n == 1)
        return 1;

    //Fib(N) = Fib(N-2) + Fib(N-1)
    return Fib(n-2) + Fib(n-1);
}

But if you want to print the first 6 Fibonacci numbers, you will be recalculating a lot of the values with the above function.

For example: Fib(3) = Fib(2) + Fib(1), but Fib(2) also recalculates Fib(1). The higher the value you want to calculate, the worse off you will be.

So one may be tempted to re-write the above by keeping track of the state in main.

//Not supported for the first 2 elements of Fib
size_t GetNextFib(size_t &pp, size_t &p)
{
    int result = pp + p;
    pp = p;
    p = result;
    return result;
}

int main(int argc, char *argv[])
{
    size_t pp = 0;
    size_t p = 1;
    std::cout << "0 " << "1 ";
    for(size_t i = 0; i <= 4; ++i)
    {
        size_t fibI = GetNextFib(pp, p);            
        std::cout << fibI << " ";
    }   
return 0;
}

But this is very ugly, and it complicates our logic in main, it would be better to not have to worry about state in our main function.

We could return a vector of values and use an iterator to iterate over that set of values, but this requires a lot of memory all at once for a large number of return values.

So back to our old approach, what happens if we wanted to do something else besides print the numbers? We'd have to copy and paste the whole block of code in main and change the output statements to whatever else we wanted to do. And if you copy and paste code, then you should be shot. You don't want to get shot do you?

To solve these problems, and to avoid getting shot, we may re-write this block of code using a callback function. Every time a new Fibonacci number is encountered, we would call the callback function.

void GetFibNumbers(size_t max, void(*FoundNewFibCallback)(size_t))
{    
    if(max-- == 0) return;        
    FoundNewFibCallback(0);        
    if(max-- == 0) return;
    FoundNewFibCallback(1);

    size_t pp = 0;
    size_t p = 1;
    for(;;)
    {
        if(max-- == 0) return;
        int result = pp + p;
        pp = p;
        p = result;
        FoundNewFibCallback(result);
    }
}

void foundNewFib(size_t fibI)
{
    std::cout << fibI << " ";
}

int main(int argc, char *argv[])
{
    GetFibNumbers(6, foundNewFib);  
    return 0;
}

This is clearly an improvement, your logic in main is not as cluttered, and you can do anything you want with the Fibonacci numbers, simply define new callbacks.

But this is still not perfect. What if you wanted to only get the first 2 Fibonacci numbers, and then do something, then get some more, then do something else.

Well we could go on like we have been, and we could start adding state again into main, allowing GetFibNumbers to start from an arbitrary point. But this will further bloat our code, and it already looks too big for a simple task like printing Fibonacci numbers.

We could implement a producer and consumer model via a couple threads. But this complicates the code even more.

Instead let's talk about generators.

Python has a very nice language feature that solves problems like these called generators.

A generator allows you to execute a function, stop at an arbitrary point, and then continue again where you left off. Each time returning a value.

Consider the following code that uses a generator:

def fib():
    pp, p = 0, 1  
    while 1:
        yield pp
        pp, p = p, pp+p

g = fib()
for i in range(6):
    g.next()

Which gives us the results:

0
1
1
2
3
5

The yield statement is used in conjuction with Python generators. It saves the state of the function and returns the yeilded value. The next time you call the next() function on the generator, it will continue where the yield left off.

This is by far more clean than the callback function code. We have cleaner code, smaller code, and not to mention much more functional code (Python allows arbitrarily large integers).

I'm currently reading:

1. Gray Hat Python: Python Programming for hackers and reverse engineers by Justin Seitz
2. C# in depth 2nd edition by John Skeet

I'm about half way through both books. They're both great books so far.

[Creator of Python talks about Python 3000[1] (Python 3.x series)