Preparing for the talk in PyCon India 2010

So, my talk on 'Building Scalable Apps using AppEngine' in PyCon India 2010 has been approved. Although I have mentioned in the talk details as to how will my talk proceed, I would like to elaborate more on this.

Building Scalable Apps using Google App Engine (Python) (45 minutes )

Now there are a couple of keywords that must be understood clearly. We will proceed one by one on it.

Building Scalable Apps

So, we will be talking about building scalable applications. Building, as in starting from scratch. I will be presuming that the audience has some knowledge of web application development using a framework in any programming language. However acquaintance with Django, TurboGears is highly preferred. I am not expecting a lot of people to be used with Google App Engine, so I will be talking about Google App Engine in some details. What is Google App Engine, What is the purpose of it's existence, what are its chief features, how is it different from other web application development framework. Then we will be diving into the greatest advantage of Google App Engine over others - Scalability. In some of the talks that I have taken on App Engine, I have seen developers with a blurred view on Scalability. Most of the times, people are mistaken for what in true sense is scaling of application/infrastructure. Scaling is so important to understand because we have been living a world where most of the things we see around are non scalable. Look around yourself, we have the same roads that existed years ago but traffic has been growing exponentially. The cities are the same, railways are the same and so is everything else. It is getting day by day difficult to sustain the burden we put on our limited resources. But nature and we do scale to certain limits. You heart beats normally at 72 beats per minute. While you are running or playing, your body needs more oxygen and so your heart stats to pump at a faster rate in order to cope up with the increased demand. This is called as scaling up. On the other hand consider your brain. Whatever we see, sense, smell are stored in neurons as patterns. The things we need to recall more frequently are stored at much larger neural networks as compared to things that we need infrequently. This is called as scaling out. I hope this explains the difference between a non-scalable architecture, a scale-up architecture and a scale-out architecture. Having understood that, we must know that Google App Engine is designed to scale out. This means that Google App Engine is very efficient at performing tasks at parallel. And this is what we must keep in mind while developing applications on App Engine.

Using Google App Engine

What differentiates App Engine is that it provides us the same infrastructure that is used by Google. Apart from many other things, it implies that we will be using Google BigTable for storing data. AppEngine provides a DataStore environment that is build on top of BigTable. DataStore is not SQL, it is not FlatFile, it is not RDBMS. It is BigTable. And it has its own rules, own ways to read/write from datastore, own ways to maintain concurrency and all other things that should be there in a modern database system. And there are certain restrictions, certain flexibility onto it. In order to be able to develop a scalable application using App Engine, it is very necessary that we know what is datastore capable of and what is outside the limits of datastore. In any application, there are more reads as compared to writes, so we need to make sure that our read latency is very low. And here comes the concept of write once, read many. We take care of all the possible ways in which our data might be needed to read, and we store them at the write time only. We also need to take care in how we write data as there are again a lot of overheads involved in Index building, searching, and quering. Apart from all these things we can do with datastore, we have a high speed, in memory cache called as 'Memcache'. Used with a perfect blend of Datastore and memcache, the application can be quite fast and scalable.

I am also working on gathering the measurable statistics for a scalable code and a non-scalable code. I hope to complete it by this weekend, and i will upload it here soon.

Apart from this, I will be talking about a little technical background of Oxylabs Networks, India's leading social apps and game developing company. How we began with MySQL, where MySQL lagged and why we chose AppEngine. How we cater to the needs of our more than 20 million player base with high scalability using App Engine.

And if all of this excites you, creates a chill down your spine, we are looking forward to give you an opportunity to be a part of our team. Attend the talk for more details on this.

PyCon India 2010,
25th and 26th Sept, 2010
M S Ramaiah Institute of Technology, Bangalore, India (map)

Update

Presentation Slide for PyCon India is shared here