Spatial correction using a particle filter (applications of the AI class)

The Problem - old data, important data

Some of the most important spatial data is old. It was built up and maintained over decades by paper and early computer systems, and it represents power lines, roads, water pipes, and property lines. It would be good to know the precise location of this stuff.

The PLC power system was designed on in-house drawn lotlines. Today, the difference between those lotlines and the actual parcel locations is as much as 100ft, and in no consistent direction. What follows are attempts to correct the location of more than 20,000 structures without doing a significant portion by hand, using some techniques picked up in Stanford's Free AI class.

The correct location is the "Hidden" bit

Education in some very advanced and useful algorithms are now within the grasp of anyone with an internet connection and a decade old computer. More than a hundred thousand participated in the recently completed Stanford AI course, including myself. One particular technique caught my eye:

The problem being solved above is one of location - that is the hidden variable that needs to be estimated in continuous space. Why couldn't I do something similar for static assets like poles and underground vaults? With enough control points I could then move everything else relative to them (inverse distance weighted rubbersheeting) and vastly improve the data.

A Naive Approach

I wanted to start with the simplest possible implementation. I loaded the lotlines (old, hand-drawn), parcel polygons, and the poles into PostGIS. I then converted the lines and polygons to points, and decided to use the total sum distance as the mechanism for comparing candidate particles to the poles.

Again, very naive (and the data is too noisy for it to work), but it served a purpose - getting everything set up for my next iteration: comparing candidates based on tangent and distance as the robot sensors above undoubtebly do.

Posted via email from The Pragmatic Geographer

TileMill - what it does and some reasons to try it

We were promised jetpacks, but I'll take Tilemill as a temporary replacement: http://mapbox.com/tilemill/

The MapBox/DevelopmentSeed team has created one of the the last pieces really needed for mainstream open source GIS to gain really massive appeal

TileMill is used for making web maps - or more specifically - for generating tiles that make up the now-ubiquiteous slippy maps we see online.

There are other desktop applications that do this, the most notable being ArcGIS Desktop. But Desktop was built for other things first: advanced analysis tools, some pretty powerful editing capabilities, and authoring paper maps.

TileMill does one thing and it does it well. It costs nothing (compared to several thousand for some flavor of ArcMap), and outputs an open tile format that you can wire up to a webmap or iPad in less time than it takes to install ArcMap.

And it is smooth. The user experience is the best I have had with a desktop application in a long while.

It also has sane, plaintext css-like styling (MSS). This may sound like a no-brainer, but your options before this were basically some proprietary binary format from ESRI (not extensible, difficult to automate, limiting, vendor specific) or SLD, which is open source but widely regarded as something of a mess for other reasons.

There is also the training issue. ArcMap is giant and powerful - and extremely complex. The market for "GIS Analysts" is still strong in a large part because of this complexity. Less experienced users will find TillMill easier to pick up and web designers (of which there is a large pool of talent) will find it very easy.

It is out for every operating system of note.

Seriously, go give it a try.

What else is needed

Conversion - minimal, well documented, mostly automated steps from ESRI - TileMill/FOSS

Samples like crazy - more or less emulate the ESRI samples, complete with documentation

Posted via email from The Pragmatic Geographer

HTML5 File System API - Basic Tips

Recently went to an HTML5 Hackathon at Google Kirkland. My group's project was an in-browser IDE Chrome extension that zipped up a user-provided series of HTML/CSS/JS files into a package that could be uploaded to the Chrome Store. Issac Lewis came up with the idea after trying to develop chrome extensions on his chromebook and finding it basically impossible to do. Storing the files was a perfect use case for the FileSystem API, but I spent most of my time beating my head against the wall to get it working. Here are some of the things I wish I knew going in.

1. The FileSystem API is not LocalStorage.

   LocalStorage is a key-value store, the FileSystem API really is an entire virtual file system, sandboxed on a user's local file system. You write, read, and create files async. It's also only implemented currently in Chrome. The documentation says 9+, but I hit errors until I switched from Chromium 12 to Chrome 13.

2. There's no limit to the storage, currently.

   Hell yeah, cache all your map data on the user's local file system without needing an explicit download or local client built for it. That's a big deal for conditions or places with little to no connectivity. Also a big deal for massive games with a ton of art assets. They go through some good use cases here.

3. Debugging is a pain.

   You will hit the dreaded SECURITY_ERR or QUOTA_EXCEEDED_ERR at some point, and it will be because debugging locally (file://) doesn't work well in my experience. The documentation suggests it's possible by opening Chrome with the --unlimited-quota-for-files and --allow-file-access-from-files flags, but my problems were only resolved when I started debugging as an extension rather than as a local file.

   You also need to be careful about the flux the API is in. Throwing around BlobBuilder() and other pieces of the newer APIs can throw errors that can be difficult to track down. BlobBuilder didn't work for me, I needed window.WebKitBlobBuilder. That webkit prefixing shows up elsewhere as well (like window.webkitRequestFileSystem).

4. Feel no guilt in lifting gratuitously from the sample docs when starting out.

   Async file access isn't really any wierder than any other browser async work, but there is some boilerplate code that is worth snapping up. Example:

    //error handling 
   function errorHandler(e) {
     var msg = '';
   
   
     switch (e.code) {
       case FileError.QUOTA_EXCEEDED_ERR:
         msg = 'QUOTA_EXCEEDED_ERR';
         break;
       case FileError.NOT_FOUND_ERR:
         msg = 'NOT_FOUND_ERR';
         break;
       case FileError.SECURITY_ERR:
         msg = 'SECURITY_ERR';
         break;
       case FileError.INVALID_MODIFICATION_ERR:
         msg = 'INVALID_MODIFICATION_ERR';
         break;
       case FileError.INVALID_STATE_ERR:
         msg = 'INVALID_STATE_ERR';
         break;
       default:
         msg = 'Unknown Error';
         break;
     };
   
   
     console.log('Error: ' + msg);
   }
   //file system instantiation
   window.requestFileSystem(window.PERSISTENT, 5*1024*1024 /*5MB*/, FSCreatedSuccess, errorHandler);

   This kind of thing is okay starting out, but you'll want a lot more out of the error handling eventually. The message is fine, but the code tells you nothing about where the error occurred and in reference to what object or operation.

5. It's not CRUD, mostly.

   Don't look for an explicit create method somewhere, the default is get or create via [filesystem_obj].[directory].get[Directory|File]. All reading, writing, and updating is probably going to live in a closure that starts with that first get.

6. Don't rush.

   I made the mistake of looking at the limited time allocated and starting just throwing the example code in willy-nilly. This is not what you do with an unfamiliar and very new API. The typical help online is not there yet because it hasn't been used yet in a widespread way, throwing those error messages into google is not going to help you (unless that is how you got to this page, naturally). Start with the example code, sure, but I would carefully read the entirety of the short intro before trying random things to get it to work.

Posted via email from The Pragmatic Geographer

Some Must-See Development Summit videos

In no particular order:

• Making Apps that Don't Suck: UX Basics for GeoNerds (video)
  
By Kirk Van Gorkum
I only caught the last half of this because I didn't know Kirk was doing this one and there was another good talk going on. User experience is everyone's job, and some GIS developers are behind their web developer cousins/alter-egos in understanding this.
• Using the ArcGIS Flex API to Build Collaborative Mobile Applications Deployed on Multiple Platforms (Android and iOS)
By Mansour Raad
There is a sick combination here. First, it is being done by Mansour Raad, who is easily the most entertaining ESRI presenter I found during the conference. Second, any portion of the title has some interesting stuff for just about everyone - building cross platform mobile apps (the Android and iOS bit), collaborative mobile applications, and apparently there are some people that really like Flex. After this presentation you can count me among them. Much of the content of this talk is from one of his blog posts, but you're cheating yourself if you don't give this a watch.
• HTML5: Not Just for Breakfast Anymore! (video)
By Brian Noyle, Dave Bouwman, Mike Juniper
Great stuff here on the state of HTML5 - in and out of the geo world - and some cool demos showing off applications working well on mobile/tablet/laptop devices with relatively little additional work (in these demos, a custom view engine for ASP MVC and Modernizr).
• You Are Legend (with jQuery and the ArcGIS API for JavaScript) (video)
By Glenn Goodrich
An important thing to keep in mind about every presentation you see is that the given specific technology typically being demonstrated isn't that important the long term - the field just evolves too rapidly. More important is the general techniques, thought processes, and tricks/hacks you can pick up from the presenter(s). Glenn's presentation is full of this stuff, even if the legend bit is more or less now done by later versions of ArcServer.
• Creating (and sharing with you) a Vector Tile Cache for ArcGIS Server (video)
By Dave Bouwman and Mike Juniper
Grassroots open source development can drive a lot of innovation on a platform (gems for Ruby, easy_install/pip apps for Python, etc), but it doesn't seem to be as common in closed-source commercial software - software improvements instead tend to come top-down. It is thus encouraging to see the DTS folks setting their sights on vector tile caching, which is kinda a big deal for those of us that want to do client side vector manipulation/analysis with big data sets (like say an entire electric system).

Posted via email from The Pragmatic Geographer