SortaLogic

The new job is different from my former one in the kind of work so far, the organization, and the environment. The responsibility for other systems - customer service, outage management, geographic information - is not on "some other team" in some other department in the company, but your responsibility.

There is only one technical team, and while other departments have personnel that are technically competent, ultimately your team is charged with making everything run on a day to day basis.

A good example here is on a rush job today - pulling data out of what we'll call a relational spatial database (CertaLogic) and putting it into an industry standard XML format (Multispeak). I am not familiar enough with CertaLogic to do an extensive complaint, but it is sufficient to say that whatever flavor we have is not easily mapped to said industry standard.

In my previous job, I would have seen this requirement coming. The confusion over whether or not it was a task I was responsible for comes from the fact that at a smaller firm, more work appears to be contracted out to 3rd parties, and one of them was charged with doing the "migration". Much of my department was surprised that this didn't include some specific parts of our GIS.

As with all unexpected requirements, there was some pressure to get it done fast.

Short answer for those that in the same situation, you have three options. Roll your own solution using a standard xml library (lxml for Python, System.Xml for .NET, etc.), XmlSpy/MapForce, or use SQL's built in XML functionality.

(Edit: hat tip Scott Hanselman for XmlSpy)

New State, New Faces

So, there has been a little change of scenery since I last posted. I got a job in Washington and didn't have to be in too much of a rush to move (unlike when we moved down to Arizona). It went something like this:

Northern Arizona

Hoover Dam Some part of California Some other part of California (you'll never guess). We traveled up the Oregon coast. I would heartily recommend this. Nearly as pretty as that girl. Now we are living in Port Orchard. It is rather nice.

Still settling in, so I will leave more details for another post. The summary: doing GIS work for Peninsula Light Co, a nonprofit coop power utility, enjoying the temperate climate, and taking the ferry to west Seattle on the weekends to see friends. I've been rather terrible about updating this blog recently due to the effort required in the move, but I intend to talk more about the area and the new job in more frequent posts.

More graph tracing - this time for water quality.

My last interlude with tracing the water system involved looking for hydrologic hazards - specifically other potential sources of water that could confound maintenance efforts. Apparently I impressed someone enough with it to get a new trace-oriented project having to do with water quality. I'll have a second part to this talking about my first run at it and a subsequent refactoring that I'm extremely happy with, but first I wanted to mention how I was documenting it. As far as I can tell, the corporate standard for documentation is Microsoft Word documents. At best, these have a relatively easy to navigate table of contents and the document is stored at the same location as the topic. At worst, it has neither attribute or doesn't exist - there is no practical difference between those two situations really. No one will ever find them - which for development projects is especially problematic. The Don't Repeat Yourself (DRY) principle applies outside single projects. There were some things I really wanted for the documentation I was going to produce for the product.

1. Fast, built in search. Amazing the difference this makes.
2. Something you could put up on the web with little to no fuss, but wasn't actually a public facing website.
3. Allowed all kinds of markup, images, other resources.
4. Wiki-style editing - who changed what, when, and some measure of version control.
5. Plaintext or in some easily parsed format.
6. I could quickly convert it into Word if I caught too much flak for not using it

What I eventually settled on is called TiddlyWiki (Google, Bing, Yahoo!..I'm just certain there is a rule Web2.0 stuff requires a childish name for success). Its one HTML document, thus very portable, with a bunch of JavaScript that implements all the functionality I wanted above. It also apparently has a lively plugin community that I haven't had time to peruse.

Documentation is important to preserving the intent of any project.

Stored Procedure Hell

I fancy myself as a very competent user of SQL (particularly T-SQL), and from school, on-the-job training and experience I've come to understand a comprehensive list of things SQL does well:

1. Basic CRUD operations
2. See number 1
3. Some very basic data manipulation - some of the stuff found here.

And that's it. I've found a problem begins to occur when people start going outside this basic mold and attempt to put in real application and business logic into stored procedures. Say, calculating a custom coordinate by referencing a seldom updated flat table containing coordinate grid intersections. Or tracing a major road or utility network. It isn't too odd this happens - a lot more people know or can quickly learn SQL. Compiled on the database, stored procedures can have impressive performance. The inevitable result, however, is not pretty. Half of your application logic is living on the database, the other half hangs out on (hopefully) a real source control system. You become heavily tied to that database vendor, since they all seem to have their own SQL dialect, and undoubtedly you've had to use their more advanced and esoteric features to implement something. And I've found the stored procedures are rarely extensible for multiple applications. They can be, but in practice they are built for very specific purposes and are tightly coupled to the applications that need them. Example: a stored procedure that calculates a custom coordinate, like the example mentioned above, is entirely coupled to its table. There would be no easy way to change the data source to a more commonly updated table or, ideally, a standardized web service. Also, the calculation itself isn't broken up into discrete methods, which means modifying one part of the long procedure will quite easily break another part in ways not necessarily easy to discover.

It's for storing data, not making a calculator for it.