The Intellog Blog

The first step towards incorporating Manitoba well data into Onramp was to set aside seemingly-centuries-old notions regarding the normalization of data. This exercise involved combining two very different types of information into what is effectively one table. That rustling sound I kept hearing in the distance must have been Edgar Codd rolling over in his grave.

The current content of Onramp is entirely document-centric; either complete documents like the ERCB’s ST1, ST49 and ST96 reports, or individual pages of documents, such as the ERCB Directives. The Manitoba well data is entirely different. It’s series of rows in a table of data; each row containing a series of discrete pieces of information each of which describe the well in some way. The challenge is to transmogrify each row so it resembles the existing documents in the index. This is because an Onramp user may not care where a particular search term is found, only that it was found, and now they want to look at it in context. For example, if they use the search term Tundra, they may be looking for documents containing references to ‘drilling considerations with respect to arctic tundra‘ . On the other hand, they may be looking for wells, the names of which include Tundra the Manitoba-based oil and gas company. But most importantly, they don’t want to have to worry about it, so the the index will need to reflect both. Once the user sees the term in context, they can then decide how to further refine the search to find the information for which they are looking.

To work through the changes required to Solr’s schema.xml, the data available on each well was systematically examined and compared to the fields already defined in schema.xml for the existing documents. If a particular piece of well data closely resembled an existing field, than the existing field would be used. If a given piece of data was substantially different from any existing field in schema.xml, a field would be added. The resulting combination of the existing and new fields was then reviewed and their use with both the existing and new documents described in the document above. There were a few fields which were just too problematic to try and reconcile between the old and new documents, and were therefore declared out of scope or this particular iteration. That means they will be not available in the index, which is a relatively small price to pay or the time being.

Click on the document at the top left for a PDF version of the complete analysis. Fields added to schema.xml as a result of this process are shaded green. As always, questions and comments are welcome. Thank you very much for reading, for your interest and your support.

Code Shavings On the bright side, the data which is available for Saskatchewan and BC is very similar to the Manitoba data being incorporated at this time. So indexing BC and Saskatchewan wells can be similarly indexed fairly quickly. As for Alberta, well that’s a whole other story.

In the post But They Didn’t Even Call? a commitment was made to file a Request for Inquiry with the Office of Information and Privacy Commissioner (OIPC). This commitment has now been made good, and the document can be found here, or by clicking the Adobe Reader® logo, bottom right. Please feel free to contribute your thoughts and comments elow, or contact us. Thank you very much for your interest and support.

As I have mentioned in my previous posts, we are using the agile development methodology towards this project.  The main reasons are that agile allows for changes to the requirements, incorporates iterative development and it’s quite compatible with Rails.  Using iterative development enables us to build our system in steps; providing a useful prototype at the end of each iteration cycle.  This approach provides us with the ability to incorporate changes to the requirements at each step of the way; which is both comforting and exciting for us as developers as well as for our customers.

Speaking of iterative development, this week happens to be the end of my first iteration.  I have received a great amount of support during this task and I am excited to say that the first iteration has indeed been a success; although there is still work to do!  Looking back, we started this iteration with a fairly concise and straight forward way.  The requirements were explained to me using simple diagrams and scenarios of the system and the expectations were made quite clear.  I made sure I got a head start on learning and hit the books pretty hard from the start.  Although I think I made good progress in this iteration, I do believe I experimented a little too much with the technology rather than sticking to the straight and narrow path of development, which was costly in terms of time but educational nevertheless.  I attribute this mostly to my lack of knowledge and experience with Rails as well as being in my first iteration so it is pretty justifiable.  The good news is that as the result, I have become more experienced and feel more confident towards future iterations.

In this post I wanted to briefly explain the application we are currently building.  Simply put, we are developing a web application which enables petroleum related information sharing among users from different organizations and backgrounds.  This application will contain technical and geographical information about all wells in the country and allows users to search for them and provide information of their own.  During this iteration I worked on building a general framework, a system skeleton if you will, for the system with the basic functionality.  The hope is that our first iteration will provide us with a primary prototype which will be the stepping stone in the development of this application.

It’s July, and that means that once again, the Tour de France is on TV, and for just one time each year, I’m watching Versus. During one of the commercials breaks I didn’t fast forward through, the Versus Sports Montage commercial came on, and I was horrified by what they had done to it. I was reminded of the fact that last year’s equivalent of the ad is very likely some of the best commercial footage you are ever going to see.

“The odds may be stacked against you. Fair enough.“

The images combined with the gruff old coach voice over made me think about the little endeavour we’ve got underway here. Loyal followers of this blog will have seen a lot over the last couple of years, and some have even have noted the seemingly futile efforts to get things done. Important things. Things which need doing. For us. For all of us.

“Don’t be surprised if somebody decides to flip the script, and take a pass on yelling uncle.“

But when the little successes come, I’m reminded how much I love this job and this entrepreneur’s life I have chosen. There’s no guarantee of success, and the odds are long, but the rewards are truly worth the effort. I’m truly proud, not yet for having succeeded, but simply for having tried.

“We got ourselves a game.“

Note: The fabulous soundtrack for this ad is First Breath After Coma by Explosions in the Sky, available on iTunes. The voice over is actor John Doman, most recently of The Wire. Also, I only wish I could get a version of this ad without the hunting, which I simply cannot abide.

Alberta Petroleum & Natural Gas Public Offerings for 2010-09-15 are now available. This data is formatted in CSV and is ready for upload to Accumap. The file can be downloaded here or by clicking the CSV icon, bottom right.

If you would like to be notified by email as soon as the reformatted CSV files are available in the future, please contact us or leave a comment below, and we will be happy to add you to our guaranteed spam-free email notification list. There is no charge for this service.

Background information can be found here, and please let us know what you think. Thanks for your interest and support.

Back in May of this year, a post to this blog described an effort to obtain support for free well data from various oil & gas industry associations. To date, all of these letters have gone unanswered, which is not all that surprising given that Intellog in not a constituent member of any of these organizations. However, it’s still disappointing that none of them felt it was necessary to reply, even if it was to say they were not interested.

Undeterred, the decision has been taken to go directly after broad support with a series of letters written directly to industry participants. Copies of the first 25 letters in this regard can be found here, or by clicking the Adobe Reader® logo at the bottom right. The general selection criteria for the recipients was to identify larger organizations on the producer side, along with higher profile service companies with which they do business. Ultimately, all of these companies stand to directly benefit from this initiative. Shared well identification information enabled by free well data will make transferring data between these industry participants dramatically easier and much more efficient.

As additional batches of recipients are identified, similar letters will go out to them, and these will be documented in the comments below. Also, replies received in response to these letters will be posted as well. Comments or questions are welcome, or if you prefer, you can get directly in contact with us. We appreciate and highly value your support!

There seems to be a relatively new trend among software application development technologies, languages and frameworks which involves taking care of the lower level details and helping to avoid repetition.  Emergence of C# in the .NET framework for instance is a good example of this trend.  Enforcing object orientated implementation, lots of built of functionality and a very easy to use utility for building graphical user interface (GUI) are among a few properties of C#.  In terms of web development Rails seems to be after the same thing!

For instance, C++ is a very popular and powerful language for developing object oriented applications.  However C++ does not impose object oriented programming (OOP) and it is left up to the programmer to choose whereas C# enforces the use of OOP for the most part. Similarly, the model-view-controller (MVC) architecture, can be followed using PHP for instance, but MVC is actually enforced and stricly followed in Rails.  A nice overview of MVC can be found here: http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller

Now without going into deep technical debates, we can safely assume MVC  is a reasonable software architecture, in fact it’s a very good one.  MVC helps separate the different layers of the software, such as the data handling layer (model) from the presentation layer (view) and those from the business logic (controller).  It’s a perfect architecture in terms of producing maintainable code which I always found too difficult to build using PHP.  Having said that, I am not claiming that MVC can’t be built using PHP, just that it needs a lot of work, whereas Rails literally guides the programmer through building an MVC architecture.  I have some experience with building applications with J2EE and in the weeks to come I plan to point out (at least in my view) how it differs from Rails.

Aside from the purely technical side of things, using Rails is favorable in terms of enabling the developers to provide exactly what the customer wants.  As agile style development has been embedded in the very structure of Rails, it makes it very flexible towards change.  For instance, Rails strictly follows the DRY (don’t repeat yourself) principle.  This along side of the very structured and modular design of Rails applications make them extremely maintainable.  I will talk more about this in the next post as I actually go into explaining the application we are building and our agile approach towards the project.

The second round of interviews are going very well and I am absolutely happy with the feedback I am receiving. The feedbacks are positively constructive and I think we are on the right track. Most interviewees find the system intuitive and easy to use with very few stumbles, which will be fixed. They also provide us with very useful information about what functionality would be useful for them. A few interviews are remaining and I will develop a low fidelity prototype after that.

When a previous post in this series reported Portfolio Officer Hendy having declined Intellog’s offer for more information, it was assumed that was for the time being — surely a meeting would be eventually be set up, and Intellog would be provided the opportunity to make our case. It was a shock, therefore, when the final report was received, in which Ms. Hendy provides the findings of her review. To save you the trouble, skip to page seven where she states; “the ERCB correctly applied section 29 of the FOIP Act to your request for access to the General Well Data File and monthly updates to this file” and goes on to say “the ERCB was correct in stating section 93(1) does not apply to your request for access to records/information as the fees you requested waived were not fees for services relating to your access request.“

So it would appear the Freedom of Information and Protection of Privacy Act (FOIPPA) has a hole you could drive an oil sands dump truck through. For any data an agency of government does not want to make public, they simply need to publish a prohibitively high price for it. They can then invoke 29(1)(a.1) of the Act which states “[t]he head of a public body may refuse to disclose to an applicant information…that is available for purchase by the public.” Don’t want to disclose the budget for the next fiscal year? Simply have the Minister of Finance declare it’s available for purchase for $1,000,000 and they don’t have to.

Of course, the latter is ridiculous — the public outcry would simply be overwhelming. But once you’ve crossed that threshold, then what amount for this data is reasonable? $750,000? $500,000? Or how about $1? Or even $0.01? Similarly, if it is unreasonable to charge for data in which the public has a deep and abiding interest, then what is the mechanism by which it’s determined which data should be free, and which data should have a price?

Fortunately, Ms. Hendy concludes her report by outlining the procedure by which a formal hearing can be requested. That is precisely what Intellog intends to do next, and progress will be reported on this blog. The complete text of Ms. Hendy’s report can be obtained here or by clicking the Adobe Reader® logo at the bottom right. Thanks very much for your continued interest and support, and please feel free to leave comments and questions below or contact us.

New Intellog business cards are now being issued with a 2D bar code similar to the one shown to the immediate left. The bar code on the business card is a link to the home page for the team member to whom the card belongs. Reading the code is simply a matter of downloading some scanning software to your iPhone or Blackberry, pointing the phone’s camera at the code, and you will automatically be redirected to their home page. To try it out, scan the code to the left, and you will be taken the main Intellog landing page.

There are many different types of these codes and Intellog has standardized on the QR Code developed in Japan by Denso-Wave. The code images themselves can be generated easily, and for free, with one of the many online generators. Our codes are generated with the utility found at Zerbra Crossing (ZXing).    

There are a number of different readers available, of varying capabilities and quality. The one that seems most robust and has gained some traction with mainstream media is NeoReader. It’s free, and can be installed by simply pointing your smartphone’s browser to http://get.neoreader.com. It will automatically sense the particular make/model of smartphone you are using and lead you through the installation of the correct version.

Intellog has some neat projects we hope to bring to you in the near future which involve fairly extensive use of these bar codes. We thought using them for this internal application was a good way to introduce them, and gain some valuable experience ourselves. If you have any comments or questions, please do not hesitate to leave a comment below, or contact us.