The Intellog Blog

In 2008 and 2009, Intellog submitted proposals to The University of Calgary’s Software Engineering Project (CPSC 594) program led by  Dr. Guenther Ruhe. This year, we are once again submitting a proposal. The abstract;

“There are a wide variety of use cases for information search functionality which can accept a combination of content and geospatial criteria. Open source software is available to address both of these individual requirements; Solr for content-based search and PostGIS for processing geospatial queries. Intellog Inc. proposes the development of a new open source software component (“Blanco”) which can take search requests with both types of criteria, dispatch them to the respective systems for processing, integrate the returned result sets and finally return the rationalized result set to the user or calling application.“

But in addition to submitting a proposal to the CPSC 594 program, we are simultaneously submitting the identical proposal to the University of Helsinki’s Software Factory program. There are many similarities between the two programs, and some differences. The intended objective is to have the proposal accepted by both programs, in which case, the student teams will have the added challenge of collaborating over a distance of over 7000 kilometres and nine time zones.

The proposal can be found here, or by clicking the Adobe Reader logo bottom right. Your comments are welcome and encouraged. Hopefully, the third time is the charm.

We are almost a week away from the completion of our second iteration and consequently having our application deployed.  There is still a fair bit of work to be done, however we seem to be at a very good spot.  I attribute this to the development methodologies and processes we have been using.  Although we are following Agile methodology pretty closely, nevertheless we have been quite conscious of tailoring it to our specific needs.  For instance in order to make effective use of our time in this sprint and ensure we are on the right track, we decided to have short meetings (approximately 20 minutes each)  everyday.  The main objectives of these meetings were to touch base, go over the tasks done each day and address any issues or obstacles that might exist.  Moreover, as mentioned in my previous posts, using Basecamp as our project management tool went a long way towards assisting with our collaboration.

As it stands right now, we have completed the functional portion of this iteration and just working our way through the testing.  What remains is mostly designing the actual web pages,  therefore I am quite confident that we will meet our deadline.  However right now I wanted to take this opportunity and write a few lines about my experiences while developing this code.

Rails provides a powerful framework for the development of web applications and as such is great to work with.  As mentioned previously, Rails not only follows, but strictly enforces the use of MVC (Model-View-Controller) software architecture.  Thus it is absolutely crucial for the developer to clearly understand the roles and responsibilities associated with each of the layers.   Looking back now, this was one area I wish I would have pondered more carefully prior to the start of development.   Another challenge that I faced while working with Rails was ironically because of its tendencies to make things easier for the developer by automating much of the development process.  This was due to the fact that for instance I would plan to do a certain task, only to find out that either that task has already been done by Rails or there was a much simpler way of doing it! off course this is nothing that can be overcome by more reading and experience.

Last but not least I found myself being challenged almost every step of the way by my lack of knowledge of the Ruby language.  This posed a real challenge especially when I was trying to deal with more advanced code; whether it was reading a book on Rail or writing the code myself.  I did find some good materials on this topic online (http://api.rubyonrails.org/) but I am also looking for a good Ruby book to buy as well.

One of the important tasks to be completed in this iteration was establishing our code repository.  Having a repository would provide us with a reliable back up for our work as well as the opportunity for multiple programmers to share code efficiently and work simultaneously. There are a number of code repositories to choose from; but we have selected Git for our project.  Git is a powerful and flexible version control tool which enables collaborative development with low overhead.  Git is a Linux based software which was actually first invented to support the development of the Linux Kernel; however it has since been successfully used in a variety of different projects.

Working with Git can seem complicated at first, but after doing some reading and getting some hands on practice it proves to be quite straight forward.  In order to use Git, we need a server (i.e. remote code repository) and a client application to send our code over to the server using SSH connection.  For the server side, we are using github.com which is an interactive, easy to use tool to manage the code.  As for the client application, it depends on the operating system.  Since I am running windows, I could either use Git through Cygwin or by installing a standalone Git software called “msysGit”.  I didn’t have Cygwing installed on my machine and don’t really have a lot of use for it at this point of time, so I decided to go with msysGit.

Git is quite sophisticated and has a wide range of functionalities which are useful in managing a project of any size.  For instance Git allows you to keep track of all changes that have been made to a given project and enables version control management in a very elegant manner.  There are a lot of material online to help with learning all the capabilities of Git; but the basic idea is quite straight forward as follows: The client application runs on the local machine and is used to create repositories.  The client application communicates with the server through a generated SSH key; thus the client can upload code unto the server repository and download it.

The client application which I am using (i.e. msysGit) is quite easy to work with.  It contains both a command line and a GUI application to make repositories and send them over to the server; in this case github.com.  I started learning Git, and in particular msysGit, using a book called “Version Control with Git” by Jon Leoliger.  Although an insightful book, I found that it was more concentrated on working with the command line.  I also found the following tutorial a real asset in learning Git:

http://nathanj.github.com/gitguide/tour.html

Although I followed a different direction at times from what was suggested in this tutorial, (in particular about using Putty for establishing the SSH connection with the server) nevertheless I found it extremely useful.

Perhaps one of the most satisfying experiences for a software developer is to see his or her programs deployed and used by others.  It is a great sense of pride and accomplishment which is much deserved after relatively long periods of hard work and in fact it’s one of the main reasons I decided to become a software engineer.  I am excited to say that we are just a month-long-iteration away from deploying the first version of our application.  Needless to say that there is a lot of work to be done at this point which requires a great deal of precise planning.

The last iteration has left us with the general logic of the application; the nuts and bolts of the overall product if you will, and a pretty detailed knowledge of Rails framework.  Although this is a solid foundation to build upon,  we are still far away from having an application which is ready for deployment.  Not only there is still a substantial amount of testing to be done, but the layout of the web pages remain to be agreed upon and completed.  To add to the challenges at hand, I have to go out of town for a software engineering conference this week which means there is less time to work on this iteration.

To plan for this, we had a good meeting on Friday (July 30th) to discuss the last  sprint as well as what we need to do to achieve our goals in the current iteration.  The good news is that everything in terms of where we are now and where we want to be at the end of this month is crystal clear; the challenge is the time constraint.  However having time constraint is definitely not something new to software development projects and can be properly managed with some good planning.

Following our discussions; I am currently in the process of analyzing the tasks at hand as well as the resources (mainly time) needed for their successful completion.  As it stand right now, I believe I am going to start by ensuring the functionalities developed in the previous iteration are completed and work properly.  This step should give me a nice head start in the process of establishing automated testing for our application hereon forth.    Almost parallel with these steps I plan to establish a code repository which is a bit overdue at this point.   From there I plan to spend most of the next iteration working on the layout of the web pages and ensure they are tailored to expectations. Furthermore to make sure we are on track and monitor our progress, we are utilizing a project management tool called Basecamp.

The first step towards incorporating Manitoba well data into Onramp was to set aside seemingly-centuries-old notions regarding the normalization of data. This exercise involved combining two very different types of information into what is effectively one table. That rustling sound I kept hearing in the distance must have been Edgar Codd rolling over in his grave.

The current content of Onramp is entirely document-centric; either complete documents like the ERCB’s ST1, ST49 and ST96 reports, or individual pages of documents, such as the ERCB Directives. The Manitoba well data is entirely different. It’s series of rows in a table of data; each row containing a series of discrete pieces of information each of which describe the well in some way. The challenge is to transmogrify each row so it resembles the existing documents in the index. This is because an Onramp user may not care where a particular search term is found, only that it was found, and now they want to look at it in context. For example, if they use the search term Tundra, they may be looking for documents containing references to ‘drilling considerations with respect to arctic tundra‘ . On the other hand, they may be looking for wells, the names of which include Tundra the Manitoba-based oil and gas company. But most importantly, they don’t want to have to worry about it, so the the index will need to reflect both. Once the user sees the term in context, they can then decide how to further refine the search to find the information for which they are looking.

To work through the changes required to Solr’s schema.xml, the data available on each well was systematically examined and compared to the fields already defined in schema.xml for the existing documents. If a particular piece of well data closely resembled an existing field, than the existing field would be used. If a given piece of data was substantially different from any existing field in schema.xml, a field would be added. The resulting combination of the existing and new fields was then reviewed and their use with both the existing and new documents described in the document above. There were a few fields which were just too problematic to try and reconcile between the old and new documents, and were therefore declared out of scope or this particular iteration. That means they will be not available in the index, which is a relatively small price to pay or the time being.

Click on the document at the top left for a PDF version of the complete analysis. Fields added to schema.xml as a result of this process are shaded green. As always, questions and comments are welcome. Thank you very much for reading, for your interest and your support.

Code Shavings On the bright side, the data which is available for Saskatchewan and BC is very similar to the Manitoba data being incorporated at this time. So indexing BC and Saskatchewan wells can be similarly indexed fairly quickly. As for Alberta, well that’s a whole other story.

As I have mentioned in my previous posts, we are using the agile development methodology towards this project.  The main reasons are that agile allows for changes to the requirements, incorporates iterative development and it’s quite compatible with Rails.  Using iterative development enables us to build our system in steps; providing a useful prototype at the end of each iteration cycle.  This approach provides us with the ability to incorporate changes to the requirements at each step of the way; which is both comforting and exciting for us as developers as well as for our customers.

Speaking of iterative development, this week happens to be the end of my first iteration.  I have received a great amount of support during this task and I am excited to say that the first iteration has indeed been a success; although there is still work to do!  Looking back, we started this iteration with a fairly concise and straight forward way.  The requirements were explained to me using simple diagrams and scenarios of the system and the expectations were made quite clear.  I made sure I got a head start on learning and hit the books pretty hard from the start.  Although I think I made good progress in this iteration, I do believe I experimented a little too much with the technology rather than sticking to the straight and narrow path of development, which was costly in terms of time but educational nevertheless.  I attribute this mostly to my lack of knowledge and experience with Rails as well as being in my first iteration so it is pretty justifiable.  The good news is that as the result, I have become more experienced and feel more confident towards future iterations.

In this post I wanted to briefly explain the application we are currently building.  Simply put, we are developing a web application which enables petroleum related information sharing among users from different organizations and backgrounds.  This application will contain technical and geographical information about all wells in the country and allows users to search for them and provide information of their own.  During this iteration I worked on building a general framework, a system skeleton if you will, for the system with the basic functionality.  The hope is that our first iteration will provide us with a primary prototype which will be the stepping stone in the development of this application.

There seems to be a relatively new trend among software application development technologies, languages and frameworks which involves taking care of the lower level details and helping to avoid repetition.  Emergence of C# in the .NET framework for instance is a good example of this trend.  Enforcing object orientated implementation, lots of built of functionality and a very easy to use utility for building graphical user interface (GUI) are among a few properties of C#.  In terms of web development Rails seems to be after the same thing!

For instance, C++ is a very popular and powerful language for developing object oriented applications.  However C++ does not impose object oriented programming (OOP) and it is left up to the programmer to choose whereas C# enforces the use of OOP for the most part. Similarly, the model-view-controller (MVC) architecture, can be followed using PHP for instance, but MVC is actually enforced and stricly followed in Rails.  A nice overview of MVC can be found here: http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller

Now without going into deep technical debates, we can safely assume MVC  is a reasonable software architecture, in fact it’s a very good one.  MVC helps separate the different layers of the software, such as the data handling layer (model) from the presentation layer (view) and those from the business logic (controller).  It’s a perfect architecture in terms of producing maintainable code which I always found too difficult to build using PHP.  Having said that, I am not claiming that MVC can’t be built using PHP, just that it needs a lot of work, whereas Rails literally guides the programmer through building an MVC architecture.  I have some experience with building applications with J2EE and in the weeks to come I plan to point out (at least in my view) how it differs from Rails.

Aside from the purely technical side of things, using Rails is favorable in terms of enabling the developers to provide exactly what the customer wants.  As agile style development has been embedded in the very structure of Rails, it makes it very flexible towards change.  For instance, Rails strictly follows the DRY (don’t repeat yourself) principle.  This along side of the very structured and modular design of Rails applications make them extremely maintainable.  I will talk more about this in the next post as I actually go into explaining the application we are building and our agile approach towards the project.

The second round of interviews are going very well and I am absolutely happy with the feedback I am receiving. The feedbacks are positively constructive and I think we are on the right track. Most interviewees find the system intuitive and easy to use with very few stumbles, which will be fixed. They also provide us with very useful information about what functionality would be useful for them. A few interviews are remaining and I will develop a low fidelity prototype after that.

We have started the second round of interviews. The first goal of this round of interviews is to evaluate the prototype usability. For this purpose we have defined several tasks and watch how users accomplish them. We also evaluate whether each user interface concept is clear.The second and more important goal of this round of interviews is to elicit more requirements from the industry. In order to discover the functionalities that would be useful for the industry, we are asking them: whether they would use each feature of the program; how they would use it; and what more they would need. We also ask them how each feature and the overall software would fit in their work. So far, we have conducted a few interviews and received positive and constructive feedback. We are happy with the project progress.

The first week on the job is always memorable. It can be exciting, stressful and even overwhelming at times. As for my first week with Intellog it has been an exciting experience as well as a comfortable transition. I just started with the company this past Monday. I am a University of Calgary graduate in software engineering as well as computer science. I was hired as a software engineer to develop applications using Ruby on Rails framework. We have gone through most of the administration work and I am well on my way to discovering Rails! Aside from landing the employment position of my choosing, this is the source of most my excitement.

I have been coding with programming languages such as C++ and Java for years. In the last year or so I have been moving to C# because of its GUI development possibilities. In terms of web development, I have done both J2EE with Oracle and also PHP with SQL. So needless to say I was particularly excited to have a go at Rails. I was particularly intrigued when I was informed of all the capabilities of this particular framework. It seems that Rails takes care of a lot of low level details particularly with regards to the system’s back end. Now I know that hardcore coders often oppose languages that seem to do much of the work and in return limit the authority of the programmer, but I do believe there is something to be said for the overhead which is saved by code automation.

In the weeks to come I will keep you posted on my experience with Rails and will talk about, at least in my view, how it compares with other frameworks. But for the time being I do want to take the opportunity and share some of the other aspects of this position. As I mentioned previously it has been a smooth transition largely due to the great amount of support I have received from the company, for which I am grateful. I have been given access to all the resources that I would want and need in learning the technology as well as reasonable time frames. We are following an agile development approach which makes perfect sense; given our project and the Rails structure. I will have lots more to say as I move forward with everything, particularly about the project itself as well as Rails!