A Programmer By Any Other Name

If you follow economics at all, the news has been awash in a sizable scandal involving a pair of economists and their bold claim - a specific level of government debt causes economic slowdowns - being found to have several serious problems. One of these problems was a critical error made in their Excel spreadsheet.

That's right. They were using Excel for high level, academic work that had direct policy impacts. This is the cue for everyone technical in the audience to feel smug.

Turns out this isn't uncommon - the use of Excel for this kind of work or the errors. In another instance, a very massive trade by JPMorgan Chase blew up spectacularly in part because of an error. In this instance, too, the flaw appeared to (temporary) benefit the party who wrote the formula.

I would argue using Excel for this kind of work isn't necessarily the biggest problem. That would be...

They Don't Think They Are Programmers, But It Is Likely They Are.

Sufficiently complex Excel formula/models are indistinguishable from a lot of other programming. There is some question as to whether the formula language by itself is actually Turing complete, but given the presence of VBA it is made trivially so anyway. Other complex business analysis/design tools are in a similar boat.

Given this, such users should be considered programmers. But since they are not, there is little encouragement or recognition that they should aspire toward the semblance of rigor that some software developers actually practice. Which isn't a lot, truth be told.

What We Actually Know About Software Development, and Why We Believe It's True

Shadow Programmers

About 40% of those employed in computer and mathematical occupations are actually software developers, but looking at the other job descriptions in that category, I would be shocked if they weren't actually programming - writing elaborate models worked up in Excel or maddening cursor-heavy SQL business logic.

And this category doesn't include economists. Undoubtedly there are even more occupations that act as shadow programmers. It is an open question if all white collar/academic occupations are or will be writing code. Certainly anyone who is doing anything like a proof is doing so.

Given the numbers, just like the shadow economy/shadow banking system, the number of shadow programmers almost certainly exceeds anyone with the title.

Possible Solutions

Professional certification/licensing

This is a stupid idea for a variety of reasons. First, the excessive number and nature of professional certifications are already a problem in the United States. They raise the barrier to entry and are used to kneecap potential competition. It is questionable they do anything to prevent catastrophes. And even if they did, there isn't a compelling state interest for a lot of programming to be of very high quality. A bug in a photo sharing app that trashes the user experience will have an appropriate market effect.

Outreach

This might actually work. If you see someone who is clearly programming - aka:

Oh no it's just this simple query…yes it spans 1000 lines and updates based on business logic with a trigger, why do you ask?

You call it out.

Yes that describes more or less the maximum complexity of any business/enterprise procedure. Let me give you some books. Let's write tests. Let's have reviews. Let's use version control.

Then you convince management. Either bite the bullet and (1) train this person to program in a responsible manner, which is not terribly expensive, (2) have a software developer take on the task, which can be expensive as heck if you need more of them, or (3) deal with the fact their business is built on bad code. That last one is cheap now, and potentially crippling later.

Use Source Control For Your Thesis

Why?

• Back up old versions without scattering dozens of files everywhere.
• Free online hosting with Google - create a message board, wiki, group commenting section
• Immediately push in revisions from the committee or other trusted contributers
• One central repository for your thesis, no need to back it up a dozen places

I found out how to do this too late for my own thesis, so here are the simple steps you can take to do it yourself.

1. Download and install Subversion client and TortioseSVN.
The client is hosted at CollabNet, you probably want the Windows binaries, Server and Client.

You have two choices here. If you want to make your own server (so you can easily get at it anywhere) follow this guide. For a more simple, file based approach, uncheck the SVNSERV and Apache_Mod check boxes in the installer and install it in whatever directory you want.

TortoiseSVN is a handy front-end that plugs into Windows Explorer. You can find it here.

2. Make a repository.
Right click on a folder where you want to put the repository (the database containing your thesis) and select TortoiseSVN-->Create repository here.


3. Make up some folders for the database.
Right click TortoiseSVN-->Repo-browser. The URL it asks for can be your server (if you made one) or "file:///C:/svn", without those quotes, where the folder of your repository is C:\svn.

The Repo-browser shows the contents of your database. Right now, it is empty. You pull stuff in by right-clicking (starting to see a pattern?) in the right hand panel of the browser and selecting "Add folder...", "Add Folder...", or "Create folder...".

You'll notice that everytime you add something, you can put in a note saying what you added, changed, or deleted. Very handy if you want to roll back to a previous change you made.

4. Add your thesis into the database.
This is easy, simply right-click as above and Add File. This will be your authoritative copy from here on in. You can check out local versions, modify them, then push the modifications to the database.


5. Check out your thesis.
Another right-click operation, this time target open folder with the files you want and put it in a local folder.

The place you put it becomes your "working copy". You can make as many of these in as many different locations as you want. Subversion handles how up-to-date a current working copy is, and lets you chose if you want to keep parts of the repository copy ("theirs") or your local copy ("mine") when you check it back in to make changes.

6. Make some change to your local copy.
The nature of your change is basically unimportant. Any change to the file will change the icon next to the file in Explorer to a red "!" rather than a green . This shows that changes are pending - there is a difference between your local copy and the server. You can check what exactly changed by Right Click-->Check For Modifications-->Right Click on a file--> Compare With Base.

If you are using a Microsoft Word document for your thesis, stop doing that. Seriously. Pick up LaTeX and save yourself six months of infuriating style editing in the future for this and every subsequent paper you do. Anyway, if you are using a Word document you can still use Subversion, you just can't use the nifty diff tool with it. Only plaintext works, Word documents are binary files.

7. Check in your change.
Checking in your change is simple, just Right Click-->Commit. You can also check what has changed compared to the stuff in the repository by right clicking as in the above step.

You can use this for all kinds of important documents. Anyone you give permission to can post changes, and you can see exactly what changed and whether to keep the changes (look for the Update To Revision in the right click context menu). Finally, if you do any programming or scripting at all, you are doing yourself a great disservice by not using source control. Plus, it is a great skill to put on a resume/CV.