A Programmer By Any Other Name

If you follow economics at all, the news has been awash in a sizable scandal involving a pair of economists and their bold claim - a specific level of government debt causes economic slowdowns - being found to have several serious problems. One of these problems was a critical error made in their Excel spreadsheet.

That's right. They were using Excel for high level, academic work that had direct policy impacts. This is the cue for everyone technical in the audience to feel smug.

Turns out this isn't uncommon - the use of Excel for this kind of work or the errors. In another instance, a very massive trade by JPMorgan Chase blew up spectacularly in part because of an error. In this instance, too, the flaw appeared to (temporary) benefit the party who wrote the formula.

I would argue using Excel for this kind of work isn't necessarily the biggest problem. That would be...

They Don't Think They Are Programmers, But It Is Likely They Are.

Sufficiently complex Excel formula/models are indistinguishable from a lot of other programming. There is some question as to whether the formula language by itself is actually Turing complete, but given the presence of VBA it is made trivially so anyway. Other complex business analysis/design tools are in a similar boat.

Given this, such users should be considered programmers. But since they are not, there is little encouragement or recognition that they should aspire toward the semblance of rigor that some software developers actually practice. Which isn't a lot, truth be told.

What We Actually Know About Software Development, and Why We Believe It's True

Shadow Programmers

About 40% of those employed in computer and mathematical occupations are actually software developers, but looking at the other job descriptions in that category, I would be shocked if they weren't actually programming - writing elaborate models worked up in Excel or maddening cursor-heavy SQL business logic.

And this category doesn't include economists. Undoubtedly there are even more occupations that act as shadow programmers. It is an open question if all white collar/academic occupations are or will be writing code. Certainly anyone who is doing anything like a proof is doing so.

Given the numbers, just like the shadow economy/shadow banking system, the number of shadow programmers almost certainly exceeds anyone with the title.

Possible Solutions

Professional certification/licensing

This is a stupid idea for a variety of reasons. First, the excessive number and nature of professional certifications are already a problem in the United States. They raise the barrier to entry and are used to kneecap potential competition. It is questionable they do anything to prevent catastrophes. And even if they did, there isn't a compelling state interest for a lot of programming to be of very high quality. A bug in a photo sharing app that trashes the user experience will have an appropriate market effect.

Outreach

This might actually work. If you see someone who is clearly programming - aka:

Oh no it's just this simple query…yes it spans 1000 lines and updates based on business logic with a trigger, why do you ask?

You call it out.

Yes that describes more or less the maximum complexity of any business/enterprise procedure. Let me give you some books. Let's write tests. Let's have reviews. Let's use version control.

Then you convince management. Either bite the bullet and (1) train this person to program in a responsible manner, which is not terribly expensive, (2) have a software developer take on the task, which can be expensive as heck if you need more of them, or (3) deal with the fact their business is built on bad code. That last one is cheap now, and potentially crippling later.

HTML5 File System API - Basic Tips

Recently went to an HTML5 Hackathon at Google Kirkland. My group's project was an in-browser IDE Chrome extension that zipped up a user-provided series of HTML/CSS/JS files into a package that could be uploaded to the Chrome Store. Issac Lewis came up with the idea after trying to develop chrome extensions on his chromebook and finding it basically impossible to do. Storing the files was a perfect use case for the FileSystem API, but I spent most of my time beating my head against the wall to get it working. Here are some of the things I wish I knew going in.

1. The FileSystem API is not LocalStorage.

   LocalStorage is a key-value store, the FileSystem API really is an entire virtual file system, sandboxed on a user's local file system. You write, read, and create files async. It's also only implemented currently in Chrome. The documentation says 9+, but I hit errors until I switched from Chromium 12 to Chrome 13.

2. There's no limit to the storage, currently.

   Hell yeah, cache all your map data on the user's local file system without needing an explicit download or local client built for it. That's a big deal for conditions or places with little to no connectivity. Also a big deal for massive games with a ton of art assets. They go through some good use cases here.

3. Debugging is a pain.

   You will hit the dreaded SECURITY_ERR or QUOTA_EXCEEDED_ERR at some point, and it will be because debugging locally (file://) doesn't work well in my experience. The documentation suggests it's possible by opening Chrome with the --unlimited-quota-for-files and --allow-file-access-from-files flags, but my problems were only resolved when I started debugging as an extension rather than as a local file.

   You also need to be careful about the flux the API is in. Throwing around BlobBuilder() and other pieces of the newer APIs can throw errors that can be difficult to track down. BlobBuilder didn't work for me, I needed window.WebKitBlobBuilder. That webkit prefixing shows up elsewhere as well (like window.webkitRequestFileSystem).

4. Feel no guilt in lifting gratuitously from the sample docs when starting out.

   Async file access isn't really any wierder than any other browser async work, but there is some boilerplate code that is worth snapping up. Example:

    //error handling 
   function errorHandler(e) {
     var msg = '';
   
   
     switch (e.code) {
       case FileError.QUOTA_EXCEEDED_ERR:
         msg = 'QUOTA_EXCEEDED_ERR';
         break;
       case FileError.NOT_FOUND_ERR:
         msg = 'NOT_FOUND_ERR';
         break;
       case FileError.SECURITY_ERR:
         msg = 'SECURITY_ERR';
         break;
       case FileError.INVALID_MODIFICATION_ERR:
         msg = 'INVALID_MODIFICATION_ERR';
         break;
       case FileError.INVALID_STATE_ERR:
         msg = 'INVALID_STATE_ERR';
         break;
       default:
         msg = 'Unknown Error';
         break;
     };
   
   
     console.log('Error: ' + msg);
   }
   //file system instantiation
   window.requestFileSystem(window.PERSISTENT, 5*1024*1024 /*5MB*/, FSCreatedSuccess, errorHandler);

   This kind of thing is okay starting out, but you'll want a lot more out of the error handling eventually. The message is fine, but the code tells you nothing about where the error occurred and in reference to what object or operation.

5. It's not CRUD, mostly.

   Don't look for an explicit create method somewhere, the default is get or create via [filesystem_obj].[directory].get[Directory|File]. All reading, writing, and updating is probably going to live in a closure that starts with that first get.

6. Don't rush.

   I made the mistake of looking at the limited time allocated and starting just throwing the example code in willy-nilly. This is not what you do with an unfamiliar and very new API. The typical help online is not there yet because it hasn't been used yet in a widespread way, throwing those error messages into google is not going to help you (unless that is how you got to this page, naturally). Start with the example code, sure, but I would carefully read the entirety of the short intro before trying random things to get it to work.

Posted via email from The Pragmatic Geographer