A Programmer By Any Other Name

If you follow economics at all, the news has been awash in a sizable scandal involving a pair of economists and their bold claim - a specific level of government debt causes economic slowdowns - being found to have several serious problems. One of these problems was a critical error made in their Excel spreadsheet.

That's right. They were using Excel for high level, academic work that had direct policy impacts. This is the cue for everyone technical in the audience to feel smug.

Turns out this isn't uncommon - the use of Excel for this kind of work or the errors. In another instance, a very massive trade by JPMorgan Chase blew up spectacularly in part because of an error. In this instance, too, the flaw appeared to (temporary) benefit the party who wrote the formula.

I would argue using Excel for this kind of work isn't necessarily the biggest problem. That would be...

They Don't Think They Are Programmers, But It Is Likely They Are.

Sufficiently complex Excel formula/models are indistinguishable from a lot of other programming. There is some question as to whether the formula language by itself is actually Turing complete, but given the presence of VBA it is made trivially so anyway. Other complex business analysis/design tools are in a similar boat.

Given this, such users should be considered programmers. But since they are not, there is little encouragement or recognition that they should aspire toward the semblance of rigor that some software developers actually practice. Which isn't a lot, truth be told.

What We Actually Know About Software Development, and Why We Believe It's True

Shadow Programmers

About 40% of those employed in computer and mathematical occupations are actually software developers, but looking at the other job descriptions in that category, I would be shocked if they weren't actually programming - writing elaborate models worked up in Excel or maddening cursor-heavy SQL business logic.

And this category doesn't include economists. Undoubtedly there are even more occupations that act as shadow programmers. It is an open question if all white collar/academic occupations are or will be writing code. Certainly anyone who is doing anything like a proof is doing so.

Given the numbers, just like the shadow economy/shadow banking system, the number of shadow programmers almost certainly exceeds anyone with the title.

Possible Solutions

Professional certification/licensing

This is a stupid idea for a variety of reasons. First, the excessive number and nature of professional certifications are already a problem in the United States. They raise the barrier to entry and are used to kneecap potential competition. It is questionable they do anything to prevent catastrophes. And even if they did, there isn't a compelling state interest for a lot of programming to be of very high quality. A bug in a photo sharing app that trashes the user experience will have an appropriate market effect.

Outreach

This might actually work. If you see someone who is clearly programming - aka:

Oh no it's just this simple query…yes it spans 1000 lines and updates based on business logic with a trigger, why do you ask?

You call it out.

Yes that describes more or less the maximum complexity of any business/enterprise procedure. Let me give you some books. Let's write tests. Let's have reviews. Let's use version control.

Then you convince management. Either bite the bullet and (1) train this person to program in a responsible manner, which is not terribly expensive, (2) have a software developer take on the task, which can be expensive as heck if you need more of them, or (3) deal with the fact their business is built on bad code. That last one is cheap now, and potentially crippling later.

AI/Machine Learning Python Samples

I have a new repository on github that demonstrates a couple of basic machine learning and AI techniques, principally picked up from CS_373 and Stanford's Introduction to AI. It's all explained there, and I intend to add to it as I continue my eduction in the field.

Machine learning is something I rarely hear talked about in the spatial developer field. This is unfortunate, as machine learning can be an effecitive means of analyzing, managing, and generating spatial information.

Another cool thing is the documentation I put together for this. I used pycco, a very easy to use annotated code document generator (port of Docco). Here is a particle filter. Here is a Kalman filter.

Posted via email from The Pragmatic Geographer

Spatial correction using a particle filter (applications of the AI class)

The Problem - old data, important data

Some of the most important spatial data is old. It was built up and maintained over decades by paper and early computer systems, and it represents power lines, roads, water pipes, and property lines. It would be good to know the precise location of this stuff.

The PLC power system was designed on in-house drawn lotlines. Today, the difference between those lotlines and the actual parcel locations is as much as 100ft, and in no consistent direction. What follows are attempts to correct the location of more than 20,000 structures without doing a significant portion by hand, using some techniques picked up in Stanford's Free AI class.

The correct location is the "Hidden" bit

Education in some very advanced and useful algorithms are now within the grasp of anyone with an internet connection and a decade old computer. More than a hundred thousand participated in the recently completed Stanford AI course, including myself. One particular technique caught my eye:

The problem being solved above is one of location - that is the hidden variable that needs to be estimated in continuous space. Why couldn't I do something similar for static assets like poles and underground vaults? With enough control points I could then move everything else relative to them (inverse distance weighted rubbersheeting) and vastly improve the data.

A Naive Approach

I wanted to start with the simplest possible implementation. I loaded the lotlines (old, hand-drawn), parcel polygons, and the poles into PostGIS. I then converted the lines and polygons to points, and decided to use the total sum distance as the mechanism for comparing candidate particles to the poles.

Again, very naive (and the data is too noisy for it to work), but it served a purpose - getting everything set up for my next iteration: comparing candidates based on tangent and distance as the robot sensors above undoubtebly do.

Posted via email from The Pragmatic Geographer

TileMill - what it does and some reasons to try it

We were promised jetpacks, but I'll take Tilemill as a temporary replacement: http://mapbox.com/tilemill/

The MapBox/DevelopmentSeed team has created one of the the last pieces really needed for mainstream open source GIS to gain really massive appeal

TileMill is used for making web maps - or more specifically - for generating tiles that make up the now-ubiquiteous slippy maps we see online.

There are other desktop applications that do this, the most notable being ArcGIS Desktop. But Desktop was built for other things first: advanced analysis tools, some pretty powerful editing capabilities, and authoring paper maps.

TileMill does one thing and it does it well. It costs nothing (compared to several thousand for some flavor of ArcMap), and outputs an open tile format that you can wire up to a webmap or iPad in less time than it takes to install ArcMap.

And it is smooth. The user experience is the best I have had with a desktop application in a long while.

It also has sane, plaintext css-like styling (MSS). This may sound like a no-brainer, but your options before this were basically some proprietary binary format from ESRI (not extensible, difficult to automate, limiting, vendor specific) or SLD, which is open source but widely regarded as something of a mess for other reasons.

There is also the training issue. ArcMap is giant and powerful - and extremely complex. The market for "GIS Analysts" is still strong in a large part because of this complexity. Less experienced users will find TillMill easier to pick up and web designers (of which there is a large pool of talent) will find it very easy.

It is out for every operating system of note.

Seriously, go give it a try.

What else is needed

Conversion - minimal, well documented, mostly automated steps from ESRI - TileMill/FOSS

Samples like crazy - more or less emulate the ESRI samples, complete with documentation

Posted via email from The Pragmatic Geographer

HTML5 File System API - Basic Tips

Recently went to an HTML5 Hackathon at Google Kirkland. My group's project was an in-browser IDE Chrome extension that zipped up a user-provided series of HTML/CSS/JS files into a package that could be uploaded to the Chrome Store. Issac Lewis came up with the idea after trying to develop chrome extensions on his chromebook and finding it basically impossible to do. Storing the files was a perfect use case for the FileSystem API, but I spent most of my time beating my head against the wall to get it working. Here are some of the things I wish I knew going in.

1. The FileSystem API is not LocalStorage.

   LocalStorage is a key-value store, the FileSystem API really is an entire virtual file system, sandboxed on a user's local file system. You write, read, and create files async. It's also only implemented currently in Chrome. The documentation says 9+, but I hit errors until I switched from Chromium 12 to Chrome 13.

2. There's no limit to the storage, currently.

   Hell yeah, cache all your map data on the user's local file system without needing an explicit download or local client built for it. That's a big deal for conditions or places with little to no connectivity. Also a big deal for massive games with a ton of art assets. They go through some good use cases here.

3. Debugging is a pain.

   You will hit the dreaded SECURITY_ERR or QUOTA_EXCEEDED_ERR at some point, and it will be because debugging locally (file://) doesn't work well in my experience. The documentation suggests it's possible by opening Chrome with the --unlimited-quota-for-files and --allow-file-access-from-files flags, but my problems were only resolved when I started debugging as an extension rather than as a local file.

   You also need to be careful about the flux the API is in. Throwing around BlobBuilder() and other pieces of the newer APIs can throw errors that can be difficult to track down. BlobBuilder didn't work for me, I needed window.WebKitBlobBuilder. That webkit prefixing shows up elsewhere as well (like window.webkitRequestFileSystem).

4. Feel no guilt in lifting gratuitously from the sample docs when starting out.

   Async file access isn't really any wierder than any other browser async work, but there is some boilerplate code that is worth snapping up. Example:

    //error handling 
   function errorHandler(e) {
     var msg = '';
   
   
     switch (e.code) {
       case FileError.QUOTA_EXCEEDED_ERR:
         msg = 'QUOTA_EXCEEDED_ERR';
         break;
       case FileError.NOT_FOUND_ERR:
         msg = 'NOT_FOUND_ERR';
         break;
       case FileError.SECURITY_ERR:
         msg = 'SECURITY_ERR';
         break;
       case FileError.INVALID_MODIFICATION_ERR:
         msg = 'INVALID_MODIFICATION_ERR';
         break;
       case FileError.INVALID_STATE_ERR:
         msg = 'INVALID_STATE_ERR';
         break;
       default:
         msg = 'Unknown Error';
         break;
     };
   
   
     console.log('Error: ' + msg);
   }
   //file system instantiation
   window.requestFileSystem(window.PERSISTENT, 5*1024*1024 /*5MB*/, FSCreatedSuccess, errorHandler);

   This kind of thing is okay starting out, but you'll want a lot more out of the error handling eventually. The message is fine, but the code tells you nothing about where the error occurred and in reference to what object or operation.

5. It's not CRUD, mostly.

   Don't look for an explicit create method somewhere, the default is get or create via [filesystem_obj].[directory].get[Directory|File]. All reading, writing, and updating is probably going to live in a closure that starts with that first get.

6. Don't rush.

   I made the mistake of looking at the limited time allocated and starting just throwing the example code in willy-nilly. This is not what you do with an unfamiliar and very new API. The typical help online is not there yet because it hasn't been used yet in a widespread way, throwing those error messages into google is not going to help you (unless that is how you got to this page, naturally). Start with the example code, sure, but I would carefully read the entirety of the short intro before trying random things to get it to work.

Posted via email from The Pragmatic Geographer

Some Must-See Development Summit videos

In no particular order:

• Making Apps that Don't Suck: UX Basics for GeoNerds (video)
  
By Kirk Van Gorkum
I only caught the last half of this because I didn't know Kirk was doing this one and there was another good talk going on. User experience is everyone's job, and some GIS developers are behind their web developer cousins/alter-egos in understanding this.
• Using the ArcGIS Flex API to Build Collaborative Mobile Applications Deployed on Multiple Platforms (Android and iOS)
By Mansour Raad
There is a sick combination here. First, it is being done by Mansour Raad, who is easily the most entertaining ESRI presenter I found during the conference. Second, any portion of the title has some interesting stuff for just about everyone - building cross platform mobile apps (the Android and iOS bit), collaborative mobile applications, and apparently there are some people that really like Flex. After this presentation you can count me among them. Much of the content of this talk is from one of his blog posts, but you're cheating yourself if you don't give this a watch.
• HTML5: Not Just for Breakfast Anymore! (video)
By Brian Noyle, Dave Bouwman, Mike Juniper
Great stuff here on the state of HTML5 - in and out of the geo world - and some cool demos showing off applications working well on mobile/tablet/laptop devices with relatively little additional work (in these demos, a custom view engine for ASP MVC and Modernizr).
• You Are Legend (with jQuery and the ArcGIS API for JavaScript) (video)
By Glenn Goodrich
An important thing to keep in mind about every presentation you see is that the given specific technology typically being demonstrated isn't that important the long term - the field just evolves too rapidly. More important is the general techniques, thought processes, and tricks/hacks you can pick up from the presenter(s). Glenn's presentation is full of this stuff, even if the legend bit is more or less now done by later versions of ArcServer.
• Creating (and sharing with you) a Vector Tile Cache for ArcGIS Server (video)
By Dave Bouwman and Mike Juniper
Grassroots open source development can drive a lot of innovation on a platform (gems for Ruby, easy_install/pip apps for Python, etc), but it doesn't seem to be as common in closed-source commercial software - software improvements instead tend to come top-down. It is thus encouraging to see the DTS folks setting their sights on vector tile caching, which is kinda a big deal for those of us that want to do client side vector manipulation/analysis with big data sets (like say an entire electric system).

Posted via email from The Pragmatic Geographer

Side Projects, Green Projects

I love doing side projects. There are many reasons:

1. No better way to learn some technology than to put it into practice.
2. Can actually turn into a real thing, even a day job (a good example is Instapaper, which is the "killer app" for ebook readers like the Kindle as far as I am concerned).
3. Good for showing off to current and future employers, particularly if it is semi-related to your current domain.

I had read quite a bit of gushing on Ruby on Rails from Dave Bouwman, and I had a stupid idea for a web app that probably already existed, so I thought "why not"? I searched around and couldn't find what I had a picture in my head of.

This is what I wanted:

Simple two/three line webform where a user enters in where they are, what home improvement project they were looking at. They get back a list of what their local utility company would pay for and what contractors were available for it.

The utility information was basically public and there are actually quite a few conservation programs - every kilowatt hour saved is less that needs to be generated by building new, usually dirty power plants.

Contractors could add themselves and maybe pay to show up first. I wired up a prototype last weekend and showed a coworker who deals with conservation credits like these.

Her: "I like it."

Me: "Yeah I was going for something simple and direct like Hipmunk"

Her: *looks at Hipmunk* "Yeah, you just need an adorable mascot like they do"

Me: "Exactly! How about a big cartoony green brontosaurus with a bulbous nose?"

The next day, of course, I see a link via Twitter of a website that not only does more or less exactly what I was planning, but even had the mascot I was thinking of; only orange instead of green. The mascot bit might have been subconcious, because I remember reading a blog post from the same site (via hackernews) without ever looking at their homepage.

I'm not an expert in web design (mostly backend stuff), but I think I would prefer the simple form rather than their current landing page. I think it would be more focused on responding to what a homeowner is looking for. The energy savings calculator though, is probably the best I've seen in terms of visual style.

I might keep going with this project. The fact I couldn't easily find it on my initial search (my conservation coworkers hadn't heard of it either), the landing page, and the low barrier to entry (what do I have to lose, exactly?) means I could build something pretty quick I think.

Maybe not. I have a lot of other ideas for side projects - an open source outage management system has been rattling in my head since I've had to deal with a wonky one at work. That would be a great opportunity to do some high scalability/performance stuff outside of work.

Posted via email from The Pragmatic Geographer

Introducing (belatedly) nx_spatial

It's been more or less done a while, but here is finally a blog post about it.

nx_spatial is a collection of addon functions for the networkx python graph library. What can you do with it?

1. Load GIS formats into networkx graphs (where you can do all sorts of crazy analytics on them)
2. Perform upstream and downstream traces with stopping points.
3. Set sources and find/repair edges that don't have the correct to/from nodes.

Example from the wiki:

>>> import nx_spatial as ns 
>>> net = ns.read_shp('/shapes/lines.shp') 
>>> net.edges() [[(1.0, 1.0), (2.0, 2.0)], [(2.0, 2.0), (3.0, 3.0)], [(0.9, 0.9), (4.0, 2.0)]] 
>>> net.nodes() [(1.0, 1.0), (2.0, 2.0), (3.0, 3.0), (0.9, 0.9), (4.0, 2.0)] 
>>> source = (2.0, 2.0) 
>>> ns.setdirection(net, source) 
>>> net.edges() [[(2.0, 2.0), (1.0, 1.0)], [(2.0, 2.0), (3.0, 3.0)], [(0.9, 0.9), (4.0, 2.0)]]

Available on pypi or bitbucket. Eventually I want to integrate it with networkx trunk (loading shapefiles is already in 1.4).

Posted via email from The Pragmatic Geographer

Random Hacks of Kindness

I really didn't know what to expect.

Information on the official site and wiki was limited. The location, percise problem definitions, available resources, what we should bring - all quite hazy.

I don't think I could be much happier with how it turned out.

Random Hacks of Kindness (RHOK) is a weekend-long event in dozens of cities that puts volunteer programmers to problems identified by disaster relief experts. In Seattle, this took the form of around 20 people going after (1) mobile damage assessment and (2) identifying and providing information on populated places during disasters. Splitting off into groups, I joined the group working on the latter problem.

A decent portion of our first day was taken up with research and getting set up. The problem asked to pull information from Wikipedia for the populated regions, but the GeoNames API was vastly more suited for the task. It was decided to create a very simple Rails app that took as its input a know populated place. The user would then be prompted with a disambiguation page that includes a search radius option. The final output drops some very basic data about nearby populated places (at the moment, just the name and population), hopefully later to be expanded (haven't quite gotten around to that).

Lion's share of the credit goes to Kevin Moore and Ezekiel for doing much of the hacking. Being fairly new to Ruby I mostly helped with the spatial stuff - talking up GeoNames, OGC, and database help at the end (and in some controller tests we never really used).

Source code is up on github here.

 

Posted via email from The Pragmatic Geographer

Location based services are going to compete on trust, except Facebook

Two articles recently came to my attention - one describing how location based services like FourSquare are used by some vanishingly small percentage of young people and basically no one else. This could be rather bad news for such services, as young men are a group that will try anything at least once regardless of its practicality or common sense.

The other article described how some mobile applications are secretly sending GPS coordinates to advertisers. This is both good and terrible news for developers of location based services. To me, it suggests two things:

1. They'll be competing on trust, which is remarkably difficult to do if you get big. Even if you are internally careful about the sharing of information there are third parties that can make your whole idea look like a bad one.
2. Quite a lot of money is being invested into location based services - startups like CloudMade and Facebook (Places) for example. Is it potentially because things like sending GPS coordinates and other personal information was what the funders were ultimately expecting in the future?

People will still use Facebook even if they find out it's been sending GPS coordinates and personal information to advertisers because everyone knows what they are getting into (or should). Everyone else has to worry about the incompatibility of competing on trust and potentially being funded on the notion that in the future you will be asked to betray your users.

Posted via email from The Pragmatic Geographer

WhereCampPDX 2010: Some of the Sessions

It's going to take me some time to fully decompress and write this up, as a good portion of what I saw in Portland was new to me in some form. It is easy enough to read about the hacker community and fiddle with the cutting edge stuff like GeoCouch: it is quite another to actually meet the people and see what they are cooking up.

QTDesigner (Holly Glaser), GeoCouch/GeoJson (Max Ogden)

I hopped between sessions on this one, so I missed about half of Max's talk. The QT talk was small and had David Turner walking Holly through making a survey plugin.

The latter session was showed off an interop tool for converting shapefiles into GeoCouch and why you might want to use that datastore over say PostGIS. The larger point here, which was only briefly mentioned and which I'd like to examine more closely at a later date, is the assertion that simplicity in this sphere breeds adoption. I don't know I necessarily agree.

Domination of the Shapefile (James Fee)

I don't recall the real title, but the alt title was "Why I was Wrong in my last Keynote". It turns out Spatialite, though it was an improvement over the traditional method of bulk data delivery (masses of shapefiles), didn't have enough going for it to actually see mainstream adoption. There just isn't enough wrong with shapefiles to bother switching for data providers. Worse is simply better, at least in this case.

Still, there are problems in data interchange that need to be addressed, the big ones being (1) styling, (2) attribute relationships. I honestly don't see too big a problem with the shapefile being multiple files, since people have a tendency to zip up such things anyway and you probably want more than one shapefile at a time.

This was an interesting discussion and probably merits its own post, the examples were particularly illuminating (FGDC, GML, Gulf Spill map styling).

Keynote (Nathaniel Kelso)

Nathaniel works for the Washington Post making beautiful maps. He showed off examples - the accuracy of the coordinates for WhereCamp, Top Secret America, people estimation at the Obama inauguration - around the theme of "filling in" the map. Of an estimated 2.3 million habitated places we have good maps for a small fraction of them.

Multimodal Trip Planner (David Turner)

This talk is an odd intersection of network topology, real-time scheduling, dynamic mobile resources, and human behavior. When people say they only want to travel on bike lanes, does that mean they'll go five miles out of their way to do it? What if they want to bike only downhill by cleverly leveraging Portland's excellent mass transit system?

One thing I didn't bring up, and wish I had, is that this is an app that essentially lets people plan their travel around a relatively fixed mass transit schedule. What happens when everyone is using such information all the time? Would it be possible to start dynamically modifying the transit system itself to best integrate with user demands?

Geodata for the Masses (various)

I caught part of this talk late after briefly stopping in on the OSM editing and civic apps sessions. By the time I got there it appeared to have devolved into a talk on DIY data collection by kite and balloon.

Posted via email from The Pragmatic Geographer

"How do I get a GIS job?"

I got this question from a recent grad and I'm not sure if I gave a good answer. Or, for that matter, if there is much advice that would help someone graduating into this job market. One theory going around is that timing like that can hurt forever.

This was my response:

What kind of GIS job do you want? I see the GIS field as roughly divided into administrative, technical, and special domain-specific knowledge areas. You can specialize and in some cases earn more, or diversify and be able to potentially apply for a larger number of positions. Everyone falls a little into all of these categories. Decide to what extent you wish to specialize, and in what area.

1. Administrative
Your project managers, leaders, and communicators. Even the nerdiest technical expert needs to be able to properly gather objectives and communicate requirements. The deeper you go into this, the more likely you will be delegating the actual GIS work.

You get better at this by getting better at written and verbal communication in general. I don't feel exceptionally qualified to talk about leadership, though increasingly I have been thrust into project management work.

2. Technical
GIS is a technical field. If you lack technical skills in a technical field you should fully expect to be mocked. This area can actually be further subdivided into the people leaning toward system admin/computer janitor work and dedicated developers.

I can recommend some fantastic books if you were interested in development - starting with The Pragmatic Programmer, CODE, Code Complete, and Programming Pearls. Just reading isn't enough, I would look into contributing to open source projects.

For the GIS admin/analyst types, I'd recommend working through the entirety of the book GIS for Web Developers. You can go from nothing to managing an entire open source GIS stack (which is fairly similar to ESRI's in terms of architecture).

3. Domain knowledge
Geographic information systems are usually serving some greater business need such as the mandates of public agencies or the profit center of a private corporation. Specialized knowledge of those needs are a career in themselves, and the more you specialize in this the more that career becomes yours. People that fall closer to this area exist on the fuzzy line between actual GIS professional and being a professional in something else.

I'm not going to claim I am the best authority on this particular topic, but I've had success with this particular mindset. A good discussion on what skills you need for each can be found on the new GIS -StakeExchange website here: http://gis.stackexchange.com/questions/883/gis-development-skills

Posted via email from The Pragmatic Geographer

Geometric Network Geoprocessing

I've been hammering away at a GIS migration project that has been going on far longer than anyone expected it might take, but I've gotten the opportunity to hack away at some Python to more or less fix a problem I found on TheGISForum:

I currently have a geometric network that I've been working with using the Utility Network Analyst tools. However, I've been trying to perform a few tasks that I've noticed the Network Analyst tools provide, but the Utility Network Analyst does not.

Specifically, I want to place barriers at all locations specified by a point feature class - a functionality provided by the "Add Locations" tool in the Network Analyst Toolset.

Is there a way to use the network analyst tools on a geometric network?

The answer, sadly, is no. As far as I can tell, ESRI has not released any geoprocessing tools for their Geometric Network.

But it is possible to roll-your-own solution to get the same kind of tools by using existing open source libraries. I have a newbish work-in-progress project along these exact lines.

Posted via email from The Pragmatic Geographer

WAURISA 2010 Conference

The Washington URISA conference is just about over, and it's made me realize how long it has been since I've gotten a chance to interact with the local GIS community. Some impressions:

• The panels were probably the best part. I see now why the Barcamp/Unconference model has become so popular. Unfortunate no one from ESRI/AutoDesk/etc participated in these as far as I can tell - just the local/state government representatives and open source guys.

• Best talk I heard was from Aaron Racicot on haiticrisismap.org and OSM. Even if you came in knowing the story, the energy of the speaker, quality of maps, and audience participation was excellent.

• I was happy to see there wasn't too much stigma hopping between rooms to hit particular talks as long as you were quiet and professional about it.

• It was nice to shake hands with some Well Known Nerds (WKN) of the open source GIS community.

• Some excellent balance in conversations about the variety of solutions to a given problem. No one was especially partisan about a particular product, software, or method.

• Vendors were not as aggressive as I had been warned.

• The turnout of the CUGOS group was good. I didn't seen any equivalent ESRI or AutoDesk user group wandering around in a similar fashion.

• Smart people giving poor presentations is heartbreaking. I know this might ring hollow coming from someone who didn't give a talk, but some folks should thumb through Presentation Zen again. Walls of text all over the place.

(credit)

Link Cleanup Time

First, an adorable picture General nerd stuff
Android iTouch killer? $155 is cheap. Looking seriously at this or the Dell Mini 5. Vendor's site.
MIX 10: what Microsoft is doing this year.
AI book worthy of a read.
The Hacker, Architect, and Superhero. Only problem with this is the sushi pictures making me hungry.

Some programming stuff
Function Dependency Inversion - something about currying. Something else about F#.
Usability and Python; why it is a pretty good language for beginners even though it has some conventions not used in too many other languages. Also here is PyYAML. I haven't looked at it but people in irc were all hyper about it.
Parallel Programming is hard? I was certainly under that impression.
Let them eat code cake; Ruby ActiveRecord stuff.
Google Clean Code talks.
Some design patterns from MS: Data Persistence, Unit Of Work. Also some LINQ for beginners.
Regular expressions for people that don't know them or could use a reminder.

It's me, I'm a data janitor
Database change management.

The Web!
REST and preventing race conditions.

Local Open source geo in the northwest.

Spatial Metadata: Why is it so hard?

Why can't my GIS data be as easy to organize/provide metadata for as my music collection?

I want a clean, detailed interface to explore my files. I want an indexed library for fast searches and simple playlist dataset/layer package/group/namespace creation. I want to change the owner, license, one-line description of files the same way I might change them in a Music folder in Windows Explorer, not by diving into some ugly wizard.

ESRI shouldn't throw out ArcCatalog, they should make it the iTunes of geospatial information. Move it from the smaller segmented market of GIS professionals to the mass market. Clean up the UI, throw up a "store" that connects with ArcGIS Online and the Resource Centers. Refashion the toolbox into components available for purchase from an in-application repository and open that repository to 3rd party developers (with a trusted [reviewed] and untrusted section).

Have it read every damn spatial format possible. Embed Webkit so you can jump to Bing, Google MyMaps, OpenStreetMap, WeoGeo, and everything else out there of geospatial interest. Metadata is largely a solved problem with my music in Foobar (music player). Why? Because of freedb, a license-free database of song metadata. There isn't any reason why we couldn't do something similar for spatial data.

Ideas are cheap and this is a lot of hard work, but it is something I'd like to see. There is a lot of moving parts to spatial data - a lot of sources, metadata, and datastores. It would be nice if we could abstract away all the stuff you currently have no interest in. The fact your spatial data is in PostGIS, or AGS, or in the cloud somewhere is unimportant when all you want to do is supply the key information of when it was collected and why.

Fun with SQL Spatial

This is old hat by now, but I love how much cruft can be killed by outright avoiding older APIs.

A unique location number needed to be generated for an engineering design tool. A vendor contact sent an example to work off that was about 400 lines of C# code. This little stored procedure replaced basically all of it.

CREATE PROCEDURE [sde].[IntersectGrid]
@x FLOAT, @y FLOAT
AS SET NOCOUNT ON
BEGIN

DECLARE @g geometry;
DECLARE @grid int;
SET @g = geometry::STPointFromText('POINT ('+str(@x)+' '+ STR(@y)+')', 2);
SET @grid = (
  SELECT TOP 1 [areaName]
  FROM GridTable
  Where @g.STWithin(Shape) <> 0
 )
return @grid
END

How do you make this 400 lines? Easy, use the ArcObject API to do the intersect. Instantiating dozens of objects, checking out/in licenses, and using reflection to read a config file (not sure why they didn't just use AppSettings) adds up fast.

I could probably even do it with even less effort using Shapely, but no one else is really familiar with Python in this situation (vendors or coworkers).

Use Source Control For Your Thesis

Why?

• Back up old versions without scattering dozens of files everywhere.
• Free online hosting with Google - create a message board, wiki, group commenting section
• Immediately push in revisions from the committee or other trusted contributers
• One central repository for your thesis, no need to back it up a dozen places

I found out how to do this too late for my own thesis, so here are the simple steps you can take to do it yourself.

1. Download and install Subversion client and TortioseSVN.
The client is hosted at CollabNet, you probably want the Windows binaries, Server and Client.

You have two choices here. If you want to make your own server (so you can easily get at it anywhere) follow this guide. For a more simple, file based approach, uncheck the SVNSERV and Apache_Mod check boxes in the installer and install it in whatever directory you want.

TortoiseSVN is a handy front-end that plugs into Windows Explorer. You can find it here.

2. Make a repository.
Right click on a folder where you want to put the repository (the database containing your thesis) and select TortoiseSVN-->Create repository here.


3. Make up some folders for the database.
Right click TortoiseSVN-->Repo-browser. The URL it asks for can be your server (if you made one) or "file:///C:/svn", without those quotes, where the folder of your repository is C:\svn.

The Repo-browser shows the contents of your database. Right now, it is empty. You pull stuff in by right-clicking (starting to see a pattern?) in the right hand panel of the browser and selecting "Add folder...", "Add Folder...", or "Create folder...".

You'll notice that everytime you add something, you can put in a note saying what you added, changed, or deleted. Very handy if you want to roll back to a previous change you made.

4. Add your thesis into the database.
This is easy, simply right-click as above and Add File. This will be your authoritative copy from here on in. You can check out local versions, modify them, then push the modifications to the database.


5. Check out your thesis.
Another right-click operation, this time target open folder with the files you want and put it in a local folder.

The place you put it becomes your "working copy". You can make as many of these in as many different locations as you want. Subversion handles how up-to-date a current working copy is, and lets you chose if you want to keep parts of the repository copy ("theirs") or your local copy ("mine") when you check it back in to make changes.

6. Make some change to your local copy.
The nature of your change is basically unimportant. Any change to the file will change the icon next to the file in Explorer to a red "!" rather than a green . This shows that changes are pending - there is a difference between your local copy and the server. You can check what exactly changed by Right Click-->Check For Modifications-->Right Click on a file--> Compare With Base.

If you are using a Microsoft Word document for your thesis, stop doing that. Seriously. Pick up LaTeX and save yourself six months of infuriating style editing in the future for this and every subsequent paper you do. Anyway, if you are using a Word document you can still use Subversion, you just can't use the nifty diff tool with it. Only plaintext works, Word documents are binary files.

7. Check in your change.
Checking in your change is simple, just Right Click-->Commit. You can also check what has changed compared to the stuff in the repository by right clicking as in the above step.

You can use this for all kinds of important documents. Anyone you give permission to can post changes, and you can see exactly what changed and whether to keep the changes (look for the Update To Revision in the right click context menu). Finally, if you do any programming or scripting at all, you are doing yourself a great disservice by not using source control. Plus, it is a great skill to put on a resume/CV.

SortaLogic

The new job is different from my former one in the kind of work so far, the organization, and the environment. The responsibility for other systems - customer service, outage management, geographic information - is not on "some other team" in some other department in the company, but your responsibility.

There is only one technical team, and while other departments have personnel that are technically competent, ultimately your team is charged with making everything run on a day to day basis.

A good example here is on a rush job today - pulling data out of what we'll call a relational spatial database (CertaLogic) and putting it into an industry standard XML format (Multispeak). I am not familiar enough with CertaLogic to do an extensive complaint, but it is sufficient to say that whatever flavor we have is not easily mapped to said industry standard.

In my previous job, I would have seen this requirement coming. The confusion over whether or not it was a task I was responsible for comes from the fact that at a smaller firm, more work appears to be contracted out to 3rd parties, and one of them was charged with doing the "migration". Much of my department was surprised that this didn't include some specific parts of our GIS.

As with all unexpected requirements, there was some pressure to get it done fast.

Short answer for those that in the same situation, you have three options. Roll your own solution using a standard xml library (lxml for Python, System.Xml for .NET, etc.), XmlSpy/MapForce, or use SQL's built in XML functionality.

(Edit: hat tip Scott Hanselman for XmlSpy)

New State, New Faces

So, there has been a little change of scenery since I last posted. I got a job in Washington and didn't have to be in too much of a rush to move (unlike when we moved down to Arizona). It went something like this:

Northern Arizona

Hoover Dam Some part of California Some other part of California (you'll never guess). We traveled up the Oregon coast. I would heartily recommend this. Nearly as pretty as that girl. Now we are living in Port Orchard. It is rather nice.

Still settling in, so I will leave more details for another post. The summary: doing GIS work for Peninsula Light Co, a nonprofit coop power utility, enjoying the temperate climate, and taking the ferry to west Seattle on the weekends to see friends. I've been rather terrible about updating this blog recently due to the effort required in the move, but I intend to talk more about the area and the new job in more frequent posts.