End of GSoC

I haven't posted in a while. But now that GSoC is over I think I need to post something to mark the end. Some screenshots of the final software.

Biographer Simulator Network Graph

Biographer Simulator State Transition Graph

It's been a great journey and I'm satisfied with my performance. Here are some thoughts I had put  in the final evaluation at google melange.

Initially when I was familiarising myself with codebase I had to decide which of the subprojects I would use as libraries. At that time due to my limited knowledge of the intricacies of the code, I never realised that I would be using a library being created by a co-GSoCer for my organisation. So I just skimmed through the emails that he sent to the mailing list. After one month into the project it hit me that if I used his library it would greatly simplify my code. So I had wasted a lot of time writing my own code which I could have spent on something else. In hindsight, I should have participated in it's design process and discussions. One thing I've learnt is that I should read every mail that comes to the list thoroughly, you may never know what you may need in the future.

As a programmer I've worked on large software projects before, but the code I wrote for these was mainly written solitarily. Hence there was no real incentive to document the code and make it more readable. By participating in this year's GSoC I realised the importance of other people reading your code and commenting on it and the benefits of collaborative programming. Also I learnt the philosophy of testing and why it is so significant. I spent the better part of post-midterm time on constructing test cases and ensuring that when a new feature is added in the future the test cases can check if the existing ones don't break.

I really liked the way my organisation, biographer, allowed me to decide my own timeline and the flexibility in reporting progress that I've made in the project. I know some organisations that have daily meetings/calls with their students. Though this technique helps ensure that the students complete their work on time, I think it's better that the student be given the freedom to interact when they feel the need to.

Advice for future GSoC Applicants
Choose a topic/organisation that you have a genuine interest in and are also qualified for. And it's good to apply to multiple organisations as the competition is tight. But still spend enough time with each one of them during the application phase.

Cleanup of the Simulator and jQuery UI beautification

It's been a while since I last posted. Been busy with a conference that I had to attend. Still suffering from jet lag, but at least the headache has gone.

In the first week back to work, I got on to cleaning up the simulator code. Removed libraries that duplicated functionality already existing in jQuery and jQuery-UI. The reason for this was that these libraries were anyways required for bui, hence it would make sense to use their functions. jQuery is really awesome, makes selections so simple and we can concentrate on getting stuff done rather than fiddling with the DOM.

Another important change is that the 2 Boolean Network importers have been unified using libSBGN.js. Quite a nice library, well documented and clean. Thanks Lian.

Upcoming Tasks:
1. Layout: The d3.force layouter requires the edges to be straight, whereas in graphviz the edges can be curved. This restriction results in overlapping of edges with nodes and also with other edges. I found a nice thread that listed various js graph layout libraries
http://stackoverflow.com/questions/7034/graph-visualization-code-in-javascript
2. More importers: There are a few formats for which importers need to be written. GINML, SBML-LRG, BLIF? These formats are much more complex than the ones being used currently. Writing these importers will take time.
3. SBGN Logical Operators: The graph that is currently displayed in bui from the Boolean Network file formats only have the chemicals as nodes, if we have to conform with SBGN, then the AND/OR/NOT nodes also need to be displayed. For this the update rule needs to be parsed. Have to start searching for a expression parser library for js.

bui integration

After completing work on libscopes I moved on to bui integration. The task was to make the simulator use bui to draw the boolean network graph. I learnt from Matthias that the bui simulator code would be quite similar to graphviz and there would be no need to rewrite from scratch. That turned out to be true and there are hardly 4-5 lines added in simulator.js to draw the SVG using bui and manipulate the color of the nodes.

The Graph is imported into bui using the graph.importFromJSON, this function is really helpful as all the import code had been written to convert the boolean network formats to jSBGN objects. The importFromJSON directly imports a jSBGN object into the bui graph.

For changing the color of the nodes I obtained the SVG node element by accessing the drawables method of the graph. This gave the list of nodes each of whose SVG group element can be extracted using the nodeGroup method.

Initially I expected the color change code to work out of the box(without modifying any part of the graphviz code), but for some reason it didn't work. After debugging the conclusion was that the setAttribute method wasn't working for 'fill'. So I used element.style.fill instead.

The initial states can also be modified by clicking on the nodes. I've added a start simulation button so that the user can first set the seed and then start the simulation. The code is in the 'bui-devel' branch of the simulator repo.

The bui code isn't entirely function due a missing js layout library. I've asked Falko for including the lib in bui. Once that's done it won't take much time for the simulator to become functional.

Server:

http://goo.gl/X7Xlr

libScopes interfacing in Python

After finishing off an easy task for the biographer project, it was time to move onto more challenging stuff.

At the outset interfacing libScopes with Python seemed straightforward. Just use ctypes and the library will be magically imported into Python. But soon I realised that ctypes works well only with C libraries and not C++. There are ways to get around issues like name mangling by using extern "C",  but then I ran into many problems with classes.

After spending a considerable amount of time on this method I finally decided to abandon ship and choose SWIG over ctypes. SWIG for Python converts your C/C++ code into Python extensions. It does this by generating a lot of wrapper code. Though this may potentially slow down processing, at least SWIG supports C++.

SWIG took quite a while to get used to. SWIG uses .i files to define the Python interfaces to your C++ code. I just %include'd all the C++ headers and thankfully it worked. Just a few issues with some typedefs that had to be redefined. A few modifications also had to be made to the libScopes library, mainly commenting out functions that had been defined in the header file but the function source was missing. Also included definitions in the Makefile to use SWIG to generate the Python Extension. Takes a bit of time to compile.

Phew! The Python library is ready, but what was the point of all this? Well, to run simulations of SBML files, libScopes had to be run on the server. I chose to run it as a module in Python. The code to import the SBML file and run a single iteration(dScopeStep) can be found in the models/scopes.py file in the server repository. An additional controller has been added for the simulator. Also JSON exchange(import/export) code had to be written on the server and client to communicate the states of the boolean network.

There is one small flaw in the method that I've used. The SBML is imported for every single iteration on the server. I could not avoid this as the session variable in web2py can only store objects that can be pickled(can be converted to a string). Apparently SwigPyObject cannot be pickled. The network is stored in a Net class object whose type in Python is SwigPyObject. I'm looking into alternatives like mmap.

A live server can be tested at

http://goo.gl/dBCfJ

 

A bit of lag can be seen at every iteration. This is due to the SBML file being imported every time. Also the reaction nodes are also turning green. 

Quite a long post! Now I have to look into Lian's libSBGN.js code and check how it can be integrated. Also the simulator using libScopes output has to be verified.

R BoolNet import done!

It has been an exciting week. Completed the first task of my timeline, that is importing R's BoolNet files into jSBGN. The coding part wasn't difficult, reused most of the code written for booleannet import. Most of the time was spent in navigating HTML, including js files and reading chromium's console logs. It is quite surprising that to find one small bug I had to write multiple console.log statements across 2-3 files. Finally caught the bug and the fix was just 9 characters! Desperately need to find a better method of debugging js code.

Also learnt two new tools this week, jslint and jsdoc. Both are pretty cool and do their job well. Though, jslint does not come across as a likeable person. Also had some discussion on the dev mailing list about reorganising the repos. That too is an important task. Workload for the weekend: thinking about the repo layout and getting started with libscopes in preparation for next week.

PS: Got my GSoC package. Don't have it in my hands, can't wait to get back to the institute!

Importing Boolean Network formats and libscopes

On to the second week of GSoC, started with editing the ServerInstallation page of the wiki. Modified it to support Ubuntu. The installation scripts present in the server repository are a bit outdated and need to be modified to run on the latest Debian/Ubuntu systems.

Then I moved on to studying the source code of BooleanNet import module. It has been written in javascript and converts BooleanNet files to jSBGN objects which consists of nodes, edges and update rules. The layouting is done by graphviz currently(on the server side) which generates an SVG of the network. The simulator runs on the client side and consists of code to modify the SVG's at every  iteration.

I also looked into R's BoolNet. It's file format closely matches BooleanNet's and the existing function for importing BooleanNet only has to slightly modified to support BoolNet. I'll start writing code for this next week.

I had a discussion on the biographer-dev list about how to incorporate libscopes in the boolean network simulation module. It was decided that libscopes can be used for simulating SBGN PD networks and the simulation code would utilise libscopes API. Hence the boolean network simulation for PD has to be executed on the server side. For the libscopes API ctypes seems a better option as it has lesser overhead and provides a more direct route.

Finally, the remaining task for this week is to analyse the biographer-ui.

Biographer: Boolean Network Simulation

I know this is a bit late, but still here it is. I got selected into GSoC 2012! My project is on Boolean Network Simulation and the organisation is biographer. I'm totally psyched to work on this project.

On the first day of work, I updated my system to Ubuntu 12.04, got all the necessary softwares installed, took a lot of bandwidth and time on my 3G dongle. But it was worth it. Ubuntu 12.04 is really awesome, Unity is finally usable, but my laptop being a bit old I need to run Unity2D without all the flashy effects.

Ok. So back to the topic of the post. I needed to get the biographer server up and running. So installed all the dependencies including web2py, python-simplejson, graphviz, libsbml, the layouter and the ui. I had to use the latest version of libsbml as the old version wasn't compiling. The scripts were a bit out outdated so fixed them for Ubuntu, replaced the experimental layouter with the normal one. Then copied the simulator code to the static directory. Modified Get.py to use my local installation of the simulator. And voila, it worked, a huge network popped up in my browser(which I had to change from my usual Firefox to Chrome).

That was a bit detailed, but as it was the first day I still have the enthuthasiasm to blabber about every small thing that I'm doing. Next few days understanding the existing tools in Boolean network simulation like the python booleannet and R's BoolNet. Followup posts would hopefully focus more on the ideas and a shorter summary of the work.