Fun with SQL Spatial

This is old hat by now, but I love how much cruft can be killed by outright avoiding older APIs.

A unique location number needed to be generated for an engineering design tool. A vendor contact sent an example to work off that was about 400 lines of C# code. This little stored procedure replaced basically all of it.

CREATE PROCEDURE [sde].[IntersectGrid]
@x FLOAT, @y FLOAT
AS SET NOCOUNT ON
BEGIN

DECLARE @g geometry;
DECLARE @grid int;
SET @g = geometry::STPointFromText('POINT ('+str(@x)+' '+ STR(@y)+')', 2);
SET @grid = (
  SELECT TOP 1 [areaName]
  FROM GridTable
  Where @g.STWithin(Shape) <> 0
 )
return @grid
END

How do you make this 400 lines? Easy, use the ArcObject API to do the intersect. Instantiating dozens of objects, checking out/in licenses, and using reflection to read a config file (not sure why they didn't just use AppSettings) adds up fast.

I could probably even do it with even less effort using Shapely, but no one else is really familiar with Python in this situation (vendors or coworkers).

More graph tracing - this time for water quality.

My last interlude with tracing the water system involved looking for hydrologic hazards - specifically other potential sources of water that could confound maintenance efforts. Apparently I impressed someone enough with it to get a new trace-oriented project having to do with water quality. I'll have a second part to this talking about my first run at it and a subsequent refactoring that I'm extremely happy with, but first I wanted to mention how I was documenting it. As far as I can tell, the corporate standard for documentation is Microsoft Word documents. At best, these have a relatively easy to navigate table of contents and the document is stored at the same location as the topic. At worst, it has neither attribute or doesn't exist - there is no practical difference between those two situations really. No one will ever find them - which for development projects is especially problematic. The Don't Repeat Yourself (DRY) principle applies outside single projects. There were some things I really wanted for the documentation I was going to produce for the product.

1. Fast, built in search. Amazing the difference this makes.
2. Something you could put up on the web with little to no fuss, but wasn't actually a public facing website.
3. Allowed all kinds of markup, images, other resources.
4. Wiki-style editing - who changed what, when, and some measure of version control.
5. Plaintext or in some easily parsed format.
6. I could quickly convert it into Word if I caught too much flak for not using it

What I eventually settled on is called TiddlyWiki (Google, Bing, Yahoo!..I'm just certain there is a rule Web2.0 stuff requires a childish name for success). Its one HTML document, thus very portable, with a bunch of JavaScript that implements all the functionality I wanted above. It also apparently has a lively plugin community that I haven't had time to peruse.

Documentation is important to preserving the intent of any project.

Google Wave

Yesterday at their I/O conference, Google demonstrated a new communication/collaboration tool that appears to be intended to kill email as the standard form of personal web communication. My first feeling is: Good! Have you ever tried to use email for long, branching communications? The flat, threaded view is trash, and a tree view is not much better. I would love to see this get mainstream attention because it might not only kill email, but also kill off less collaboration tools I personally find less than enjoyable to use. Sharepoint, for instance, I have found to be clunky and have poor integration with email. That isn't really the fault of Sharepoint - email was never meant to be an issue tracker or dynamically update websites. Nor would you want it to, it doesn't have the tools. One of the first things they show off with Wave is how easy it is to do just that (particularly bugtracking). Wave is also free, open source (in some fashion or another), and federated. That last bit means your Waves don't necessarily need to live or interact with Google at all - just like internal email servers you can whip up a Wave server and apparently quite easily. The motivation for Google doesn't immediately occur to me. Best, moderately educated guess is that it might have something to do with HTML5 - the next version of HTML currently supported by roughly zero browsers yet - will be easier to index and search than Flash/Flex/Silverlight. Is Wave supposed to be HTML5's killer app, driving the browsers to pick up support for it faster? As a side note, there appears to be a conspicuous number of references to the television show Firefly in the above presentation and in the project. In the show, the standard communication term used was 'wave', the poll robot they demonstrated included an option for Serenity as the Best Movie Ever, etc. This isn't too wild for them. Keep in mind that Google Earth was apparently inspired by the "Earth" program described in Snowcrash.

Future of GIS Analysts, Part 2

In the first part of this post, I tried to put together a simplified list of some of the activities that I have found GIS analysts doing as a part of their job. The goal was to get folks talking about future roles and if GIS analysts have a future ten years from now. I was happy commentators posted some things I missed. I hadn't included, but was helpfully mentioned by geographygeek in the comments, the role of an analyst after the data has been collected and processed. Some (hopefully most) are trained to answer the question, "Well, what does that mean?" Another commenter, KindaSpatial (who puts out a rather good geography podcast), wanted me to talk a bit about some of the newer 3D and hyperlocal data and interpretation. I'm honestly not sure if I am qualified to, but I'll give it a shot and have people correct me later. I like to think of blogging as more of a dialog. On closer inspection, each of these should be their own blog post, so that is what I will do. For now, I will give my initial impressions by going down the list and examine each item, asking the same questions: Can this be automated? Is it easily outsourced Is another profession largely absorbing it? Here are my thoughts. They are not quite fleshed out, feedback greatly encouraged.

1. Map production: Outsourcing: try to make or get a good, topic specific map via phone conversation. Automation: still a lot of overhead software knowledge required for the kind of quality maps necessary for professional reports. Professional designers have all of the aesthetic abilities necessary for this, but little knowledge of the pitfalls of cartography - maps are like statistics that are even easier to lie with.
2. Requirements gathering: Probably impossible to automate, and you can read the hilarious results of trying to outsource it elsewhere. Increasingly the realm of project managers with enough GIS experience to know what is available/feasible.
3. Feature creation and maintenance: The simple stuff can and will be outsourced or automated. Stuff that requires boots on the ground simply can't, and yes it is hard to tell the difference. I don't think the dimensionality of the data makes a significant difference here.
4. General IT/helpdesk support: This really is the work of IT professionals, but for smaller firms or feudal departments it isn't going anywhere.
5. Database/content management: All information has a location-based component, and this function exists as only so long as professional database administrators are uninterested in how the middleware (SDE and PostGIS I believe) works.
6. Minor automation tasks: Prime candidate for professionals to do the automation - it ends up being cleaner and reusable.
7. Post-processing interpretation: (hat tip: geographygeek) This dovetails nicely which what I think is the core of GIS - the visual display of quantitative information. What you replace a geostatistician with? It doesn't seem to easily fit in with other professions, the methods are unique enough to be difficult to automate for easy public use.
8. Hyperlocal data: (hat tip KindaSpatial) The mass of data associated with this seems to require, not merely lend itself to automation. Turning that stuff into interesting visual works or meaningful statistics sort of falls under the previous point.

Based on any comments/corrections I get, I'll be going though these points individually. I haven't been at the GIS game too long, so I'd love to hear from some people from different backgrounds.

Future of GIS Analysts, Part 1

A question occurred to me as a result of a comment made by David Bouwman on Twitter. Folks were congratulating James Fee, who had just been offered an opportunity to teach GIS at the Arizona State University masters program.

Teach them that "GIS Analyst" will be a rare job in 10 years - just like "Database Analyst" is today.

My question(s): what do we mean by GIS Analyst in this context - what functions do they provide today that will be unnecessary or absorbed by other jobs? My official title is something like GIS analyst, though this kind of talk might be more disconcerting if I hadn't already oriented my career towards application development. But I know people I work with, professors, and certainly students today would be interested in discovering if this was in fact the case. If we were to properly investigate this matter, lets consider first what GIS analysts (which we can probably group with specialists, technicians, etc) do that makes them necessary today, and then, in part two, we can examine what that role might look like - if it exists at all - in the future. Here is a quick list of the roles I've seen played by GIS analysts:

1. Map production: Your standard cartography work. Organizing the layers, layouts, titles, etc into an aesthetically pleasing package and either plotting/printing it off, or, more recently, publishing it as a service.
2. Requirements gathering: Client communication, identifying potential solutions from user stories, project specification and some project management.
3. Feature creation and maintenance: Gathering and organizing data from disparate sources, digitizing/COGO work. Associated documentation/metadata probably falls in here too.
4. General IT/helpdesk support: Particularly the case when it is a small firm without a real IT department or person, or if that person/department is swamped, or doesn't know anything about GIS software, or IT's grasp on individual departments is tenuous.
5. Database/content management: Organizing databases, particularly ESRI geodatabases - what belongs in a given dataset, should it be part of the network, etc. File management of documentation, supplementary data.
6. Minor automation tasks: Modelbuilder, Python, VBA stuff. Almost any programming task where not knowing the basics of object oriented programming is not much of a hindrance (though it makes for terrible code).

I'm missing probably a hundred more things analysts do and I'd like to invite everyone to help me add to the list.

From GIS User to Developer - Part 3.

GIS college education typically prepares students for the very basic entry level positions. Assuming you want to have a job five years later that isn't the subject of ridicule or eliminated largely by automation, you should consider yourself a lifelong student. And just what are these entry level GIS positions? Again, my experience in the entire field is limited to just a few years, but talking to people, it seems pretty standard. Many of them are georectification/image processing monkeys. Some are brought on as folks to help maintain larger datasets like municipal water/power lines, parcel boundaries, etc, assuming there isn't a process for the drafters to input in their designs manually. Still others are cartographers in the sense they make custom stylized maps on demand. There is absolutely an art to this and, with the proper motivation, you can make a real statement (or lie) by including, omitting, or tweaking the standard map components. The proper visual display of quantitative data can shift public opinion, lead to a medical breakthroughs, and are sometimes the cause of the international dispute. It really shouldn't be underestimated. The Advice If you find yourself in the standard entry level GIS position, or applying for them, you should be looking for the means to specialize. As far as I know there are three primary ways of doing this.

1. Not Business
Environmental analysts, land use planners, medical researchers; you name it, they are probably using some form of GIS. Increasingly even the analysis stuff is done by the actual specialists rather than contracted out to a GIS person. This is the result of progressively easier to use tools and their acceptance into mainstream use. You don't necessarily need to go back to school to do this; realistically, the extent of what you can do with GIS software is such that there is a place for people with just enough business knowledge to be the interface between those specialists and the tools. At least for now.
2. The Data
What features do you have, which ones do you need, how do they relate to each other, how will you organize them? Depending on the size of the organization, there might be a taskforce that is dedicated to answering these questions. It is likely they will be IT professionals, so that is the kind of secondary activity I would recommend.
3. Applications
Create the tools the other two groups use. The bleeding edge of this is the Rich Internet Applications (RIA) using Silverlight or Flash to make extremely fancy web based mapping applications. And of course, there are probably countless other developers working on in-house custom applications (via ArcEngine) or simply extensions to existing GIS software (usually ESRI).

At the moment I am sort of doing all three of these things, but pushing myself more towards the applications side. I'll have a much bigger chance to do so when I've wrapped up my thesis (Feb 13th defense, March 6th text deadline). Really, I am still new to a much wider GIS world and it would behoove people still doing the entry level stuff to check out some blogs written by people more knowledgeable than myself.

From GIS User to Developer - Part 2

So School I finished up a GIS oriented Geography degree from CWU a few years back. From what I have heard about other, similar programs around the United States, my experience was typical - some general GIS/Computer Science history, labs following relatively closely to ESRI published workbooks using the ubiquitous ArcInfo suite, some quantitative/statistical methods, and some remote sensing stuff. I was lucky enough to do some special projects for actual state and local agencies who wanted some free/cheap maps with perhaps a smattering of analysis work. I thought I would just walk out of the college doors and fall into a low/entry level GIS job, and I might have been able to. But I met a lovely girl, who I ended up marrying, so luckily I found a pretty good excuse to stay around the college until she finished.

I don't really have a relevant picture for this section so here is CNN embarrassing themselves.

And Yet More School Up until I came down to Phoenix, everything I learned about GIS came from Dr. Robert Hickey. He was, at the time, the only member of the faculty to know much about GIS and apparently I had annoyed him enough with not-always-easy to answer questions about his courses. This left the probably mistaken impression I was more intelligent than I actually was at the time. I was curious and motivated. I liked the work and wanted to master it to the extent that was possible. This basically remains true. Anyway, this annoyance/mistaken impression factor got me an assistantship running the GIS lab at CWU. This was basically a helpdesk/IT role that, along with some independent contracting, paid my way through grad school. I get to defend in a few weeks. The Advice The only information I left the school with in terms of automating common GIS tasks was using ModelBuilder. Don't let this happen to you. At the very least, pick up SQL. Even if you don't want to become a developer and intend to live the haunted existence of a business analyst. If you are doing GIS classes you pretty much already have the necessary logic to grasp SQL. Done an attribute query? Then you've basically done half of an SQL SELECT statement. It is a "language" with a ridiculously small amount of actual syntax but can be extremely powerful. Also, it exists in every database in the world. Also, you can be justifiably ridiculed for not knowing it. Assuming you don't take a minor in Computer Science, which I would highly recommend at this stage, try this

1. Make your ModelBuilder runs. Make them as complicated as you want.
2. When finished editing, go to File:Export:To Script. Choose Python, you'll be happier learning JavaScript later
3. Read this basic tutorial and try to make small changes to your script. I know I complained about documentation just a post ago, but mostly because I have otherwise been spoiled rotten by how good ESRI's usually is. Read that too.

You can learn a lot of the basics of object oriented programming from this stuff. There is an alternative, darker route involving the use of the VBA editor in ArcMap. Nothing you make there will make you proud later, and it is possible your future children will be subject to your subsequent crimes.

Some computer science is highly recommended.

Brief interlude: What the hell ESRI?

Listen this makes no sense:

1. The documentation for your VE and GMap JS extensions - which appear to be roughly identical in function - have documentation that contains very different information. To figure out details on the VE.Geometry.ProjectToVEShapes method, I had to hunt down the similar function in the GMaps documentation.
2. Stop pretending these are really REST functions. Be a little more clear about which JS functions are just SOAP requests you later make into JSON clientside. ProjectToVEShapes demands a proxy for larger requests, which means it is running into browser security features meant to prevent cross-domain scripting attacks (detailed a bit here). Oddly the demand for a proxy was only returned when I tried it in IE, in Firefox I had trouble finding it even with Firebug.
3. The JS objects you return from ArcServer queries differ in format to the JS objects you require for functions in the aforementioned extensions. If I make an ArcServer Query for some feature class, the object it returns is not valid as an input to Project, for example.

These API extensions, and ArcServer itself, is pretty new. I can understand some of the documentation stuff, but could you be a little more consistant about the JSON formatting? Converting between the two is a bunch of unnecessary work. Am I wrong here? I'm pretty new to this, I could be missing a simple solution or not understand the problem.

Government Jobs

Some folks, particularly in this economy, tend to look more for government positions because of the perceived job security. The problem is you have to contend with stuff like this, sent to me by a friend. She told me it took them six months to hear back from them, and she had forgotten she had applied to the job at all. Everything is stripped of personal/identifiable information to protect the stupid or annoyed. Red highlighting is mine.

*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----* Replies to this email will be sent to Joe Sixpack *----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----* December 15, 2008 Jane Doe 1243 Fake Street Springfield, IL 78841 Dear Candidate: Thank you for applying for the position of Systems Analyst with the Government Department. As part of the evaluation process, you are being invited to participate in a performance examination. The exam will be conducted using computer based testing programs. The exam may consist of data entry, clerical proofreading, basic office skills, and customer service representative skills. EXAM CHECK-IN Check-in will begin ten minutes prior to your scheduled exam time. Do not arrive any earlier, as there is no waiting area outside of the testing room. Your exam is scheduled for the following time and location: Date: 12/23/2008 Time: 01:30 PM Duration: 1 hour, 45 minutes Location: County Building, Room 120 123 10th Ave Makin, GA 62479 You must bring official picture identification, such as a driver’s license, to be admitted to the test (no exceptions). Instructions for Test Confirmation To confirm your appointment, please e-mail test@county.gov. no later than 4:30PM, DATE, 2008. In the subject line, write: Systems Analyst, Yes I will attend testing session *OR* Systems Analyst, No I will not attend testing session. Include your first and last name in the body of the e-mail. Due to the volume of responses, we are unable to reply to email confirmations. If you are unable to attend your scheduled appointment time and need an alternate test time, please leave a message at 555-1234. We will try and accommodate your request. However, due to limited testing facilities and time constraints we cannot guarantee an alternate time. Parking There is limited on-street parking (applicable fees charged) off 10th Ave between Parkplace and Broadway. There are several parking lots and garages nearby that provide parking for a fee. We look forward to seeing you. Directions to this location are provided at http://www.someurl.gov Sincerely, Joe Sixpack Human Resources Division PLEASE DISREGARD THE FOLLOWING MESSAGE: If you applied online, please verify you have received this message by visiting: https://www.wtf.com *----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*----*

Lets list the hilarious parts, in order:

The exam will be conducted using computer based testing programs

Whoa, tests on computers now! Fancy. I bet it uses the same fantastic generic form program this letter was (still)born from.

The exam may consist of data entry, clerical proofreading, basic office skills, and customer service representative skills.

Great, a basic office skills and data entry test. Not at all a waste of time for a Systems Analyst.

Do not arrive any earlier, as there is no waiting area outside of the testing room

"People arriving early will be forced to run on a treadmill until the time has elapsed." I don't get it. Do people cheat through walls now? Has the office banned chairs because of the tightening economy?

12/23/2008

I assume the six month wait was so they could have the test on the second to worst day of the year.

no later than 4:30PM, DATE, 2008

"I am too lazy and/or incompetent to actually fill in all the fields in this form letter."

PLEASE DISREGARD THE FOLLOWING MESSAGE

On second thought maybe this isn't sheer stupidity - perhaps it is the test itself! Anyone stupid enough to click on a link in an email from a vaguely trusted source (that even goes to the trouble of telling you not to click it) would be a useless Systems Administrator. Actually anyone given a computer that does this is a network outage waiting to happen. The people qualified for the job, assuming they didn't get one already (my friend did) should answer it like this: "Subject: Systems Analyst, No I will not attend testing session. Is this a joke? Is a form letter really that hard to fill out? Why do I want to work with such an HR department? You wait six months to assign me to a test for which anyone applying for this position will be embarrassingly overqualified for the day before Christmas Eve. If this isn't a joke, get bent."

Flex vs JavaScript for created online maps

An interesting discussion over at James Fee's blog on this subject. It looks like more folks are going with Flex for some very simple reasons: 1. Sure, its dedicated IDE costs money, but so does Visual Studio. 2. Adobe Air allows for movement of the app to the desktop. 3. Browser compatibility, particularly with IE, is a massive problem and Flex sidesteps it. 4. Reported better performance. Not to repeat the whole thread, but the most interesting comment was revolving around how Adobe has done what Java was designed to do, but couldn't - get people to install a plugin that was basically browser agnostic and had more functionality than JavaScript (credit to Matt Giger). Update: More commentary from the developers of the Geocommons website. Geocommons is worth a whole post I'll probably do sometime soon, it is pretty nifty and uses Flex.

GIS In the Cloud

Cloud computing has been the hot topic in many GIS circle for the last year or so, largely for the same reason it is building steam more recently in most IT circles in general - datacenters and bandwidth speeds are nearing the point where the promise of mass cloud computing is feasible for corporate users (which is where the money is). For anyone previously not versed in this topic, cloud computing is basically the movement of applications, services, and data from local storage to massive datacenters run by people like Google, Yahoo, Amazon, and Microsoft. You probably use it already - say if you use GMail rather than a local application like Outlook or Windows Mail. Maps have obviously moved there too. If you own a computer you probably use some kind of online map for directions. It doesn't necessarily need to be in a browser either - Google Earth and NASA's WorldWind might be local applications, but all of the data and services are running off of datacenters somewhere else. It is believed this somewhat slow progression is going to accelerate as activities typically preformed by local IT departments for small and medium-sized businesses are increasingly replaced by cloud apps offered for basically free by the above organizations. It might seem foolish to marry yourself to a particular platform or business in this fashion but (1) a lot of these things are built on open standards like LAMP anyway and can be transferred around and (2) companies marry themselves to a vendor all the time (see SAP). Traditional desktop software vendors are shifting to do their stuff at least partially as a service (and thus online). Windows 7 isn't going to have a mail program, it is going to fill that functionality with Live Mail. ESRI, the biggest GIS software vendor, has made it a point to make it extremely easy to put online data services into map documents just as you would add in layers on a local computer. A great number of GIS data providers, largely governmental, are not by and large going to venture into the cloud all that soon, not without intervention by legislators. Why? Liability, tradition, data sensitivity. What data public exists is often of variable quality, especially when overlaid with other information from other sources. Throw some bad data out there, even if you include metadata that includes a hefty disclaimer, morons will still use it to hike into a blizzard and and sue you for having to eat their children. Even with good data, there is the problem of interpretation. Take a parcel layer from any given city/county government in the United States and throw it on Google Earth. GE does a pretty good job, but I would wager good money the satellite imagery isn't accurate to a quarter of an inch. The parcel layer is, by law. The number of people who will go on to Google Earth and stir up property disputes without this knowledge is probably substantial enough to be a factor in deciding to release it. And of course there is the security issue. Knowing what substation can black out a particular city block, what water main is feeding that block, communication lines, emergency vehicle GPS locations - this stuff could be used not just by some existential terrorism threat but by normal criminals to cause all sorts of mischief and evade capture.

Programmatically tracing a network in ArcGIS

Recently I was tasked with tracing a water network, and after studying a previous programmer's work on the subject, I realized the solution is just a modified tree algorithm that is largely language independent. The trick is a recursive function that continues to call itself until it has gone down the totality of a single path. When it can't keep going - because it reaches the end of the line or because you convince the program it is the end of the line - it hops back one function call and tries to take an unvisited path. If it finds one, it repeats the whole process. The ArcObjects API contains the extensions for Network and Utility Network tools, but as far as I can tell,those are mostly for listening for trace events rather than starting your own. Since a network like that is just a modified topology (it inherits from ESRI's topology object), you can use ITopology to create a TopologyGraph which allows access to the Edges and Nodes required. But you don't need to use C#/VB.NET/VBA to pull this off. It would be possible with the API ESRI provides with its Python scripting object (IDispatch), as that contains start and end points in its Geometry object, or by creating your own node/tree structure. I understand this is a fairly elementary use of recursion, but as a novice programmer its a lot of fun putting it into action for a real project.

Poor design

Sorry I haven't updated in a while, I have been busying myself with some reading and general related work. More and more I am putting myself into the position of a GIS developer rather than an operator - creating tools with ESRI's ArcObjects API. Whether this is "real" programming or not depends, I suppose, on your definition. I still consider myself something of a novice to the actual field of computer science, but I feel comfortable creating custom scripts and operator tools for ArcGIS in C#, Python, VB.NET, and T-SQL. I could say I am going to the trouble of learning all of this because it increases my earning potential, but it's actually a lot of fun. Computers are rather poor at what the human mind generally does - analysis and higher order modeling - and human minds simply cannot match the power of computers to instantly recall and organize memory (not to mention their potential for networking with the vast knowledge of the internet). Interfacing with a computer at progressively lower levels greatly increases the collaborative power that comes as a result of normal use, and it is a joy to be working in such a way.

HeyWhat'sThat: The coolest mapping site you've never heard of

What happens when someone has too much free time, programming talent, and a burning question? This.



Summary: HeyWhatsThat is basically a rather advanced map hack site which includes viewsheds linked with government/public name systems (names with xy data) so from a given point you can see all the major peaks nearby. The proprietor and designer, Michael Kosowsky, has steadily been adding stuff to it and it now includes pathing (which gives you an elevation profile), and a night view.

I'm moderately curious why Google hasn't bought him out/hired him/stolen the idea. As far as I can tell, Kosowsky built all the tools (minus the Google Map of course) from the ground up. It includes some interesting little quirks based on his response to user comments, like the way it handles the curvature of the earth and the fact it switches to metric if outside the United States.

(To be completely fair, you might have heard about this site. Several other bloggers, including Ogle Earth, have wrote about it.)

The viable solution to internet piracy

The sad fact of the matter is piracy as it is currently defined cannot be stopped without enormous breaches of privacy. Absent the restriction of tools to even create programs that can effortlessly defeat protection methods, you can't stop people from copying digital information. Mostly because the mere act of being on the internet means you are required to copy things just to view/listen to them. This is a sample of some of the points of a paper I recently read at Cato called The Future of Copyright. My personal favorite part:

When American troops liberated the city of Luxembourg in 1944, they made a strange capture: a machine capable of recording sound on magnetic tapes. Shortly after the war, this German military invention made its appearance in private homes. Tape recorders integrated listening and reproduction in one device, but as separate functions. That’s no longer the case with digital technology. Today, to use digital information is to copy it.

Computers operate by copying. They couldn’t care less whether the physical distance between original and copy is measured in micrometers or in miles; both work equally well for them. Copyright law, on the other hand, must somehow draw a line between use and distribution. That means putting an imaginary grid over the chaotic myriad of network nodes, delineating clusters of devices that can be attributed to individuals or households.

Every person reading this article is actually copying it illegally. You can't help but do it. As you surf the internet your computer is constantly caching (saving locally) data that you are viewing. Anyone who has ever used the internet for anything is almost certainly a pirate. I'll let you all off with a warning this time. Humor aside, the problem is one that the music industry found itself in a long while ago with the invention and mass adoption of radio. John Philip Sousa was convinced anything but live shows would completely destroy music. The solution, after much complaining, was to just license the distributor. Now of course the distributor and consumer and creator is anyone and everyone on the internet. The compromise is an idea that is not new and is even talked about by both sides of the issue with increasing interest: mass public electronic media licensing. If everyone on the internet is a pirate, license everyone. Everyone who wants to buy in at least. You've solved 99% of your piracy problems by facing the reality of a system, the internet, that requires copying. You also don't even have to host the files themselves, users are more than willing to do so. The idea has merit but the problem is the pie to be divided. We need an organization as trusted as the Nielsen statistics are for downloaded media content to correctly award creators and their labels/studios/whatever. The only alternatives are to entire restrict piracy with the kind of locking down that would destroy the internet and privacy in general (people could still just rip stuff from friends/renters) OR to see an attempt to destroy internet and privacy while the major content holders die a slow and painful (for everyone involved) death at the hands of unrelenting technological innovation. Edit: Hahaha. Bonus quote from a Guardian article linked from the Cato piece:

"For somebody who has spent 30 years in the music industry, you instinctively know this stuff is going on. But when you actually sit looking at your computer and see a number that says 95% of people are copying music at home, you suddenly go, 'Bloody hell'," he said.

Turns out you could nuke the internet from orbit and the current copyright model is doomed. Note: Copying this work is no longer actually illegal in any sense, since I have licensed everything with Creative Commons.

Monday Links

MS Server 2008 as an OS: Amusingly enough, for all the hatred generated for Windows Vista, a product with supposedly a virtually identical code base is whooping up on it as far as performance goes. Some folks in the comments have suggested this might be because Server 2008 doesn't have the more intrusive DRM that currently is the biggest reason I have no interest in upgrading ("Oh, so I get a performance hit, maybe two useful new features total, and one day you can just take over my computer to prevent me from copying stuff I bought legitimately? SIGN ME UP"). Server 2008 with some minor modifications can do just about everything that Vista can do (music, movies, games, file management) and still move faster. If this forces updates like Vista does and gets at all popular, I expect Microsoft will toss in the DRM stuff. They can't seem to help themselves, they really really want control over your computer and all the software on it. There is a lot of money in it, so I can see the motivation. Brain Rules: John Medina is entertaining in this authors@google video, explaining his new book on some of the things you can do to aid cognitive function, help in relationships, and live longer without dementia. Strange Maps: Exactly what it sounds like. Want a cheese map of Canada or a 1942 map of the "new world order" (a vision of a post war world from the very depths of the war)? This is the place to find it. Gas Prices relative to income: Who is getting hit hardest by gas prices (which is to say, what percentage of your income is being eaten up by it)? I am curious if this map takes into account the average (or median, as one might prefer) gasoline consumption of the areas. It doesn't appear to, but I will attempt to recreate it when I have time to see. Map of School Vaccinations: These are the areas to point fingers to when we see a resurgence of childhood diseases. Normally I would be live and let live about this kind of bottomless stupidity, but a population that will sustain a stable habitat for such diseases will allow said diseases to adapt and render the vaccines ineffective. Also, children should not be responsible to the mass idiocy of their parents. I believe one day I will make a density map of stupidity and this will certainly be one of the variables. Top Five Programming Languages to Learn: Just like learning a foreign language but probably worth far more money if you are really good at speaking it. C# tops the list and I would agree. You can use it to add on to anything having to do with Windows, use it for web applications, Silverlight (a .NET version of the ubiquitous Flash player) and understanding the libraries will come in handy even in an open source/Linux environment with Mono.

The benefits of being a late bloomer

This isn't a post about an embarrassing teenage angst. It is actually a story about natural resources, the industrial revolution, and programming for Windows. Ars Technica, one of the better tech sites on the internet, recently did a series on the lead up to the most recent malaise we've seen with Windows programming. As I read the piece, I kept noticing parallels between Microsoft's difficulties and the early innovators in the industrial revolution. Despite the fact that you probably know nothing about the 19th century about Britain besides the fact the nascent United States kicked its ass off the better parts of North America, it is actually pretty relevant. Rule Britannia Britain ran the 19th century. It controlled all of the sea lanes, dominated world trade/finance, and ran nations which were ten times its landmass and population by occasionally machinegunning down crazy-brave hordes of natives who were attacking them with spears. It got this way because it not only was the first to recognize and reap the massive benefits from new industrial machinery, but also because it ruthlessly closed itself off in terms of technology to retain their advantage. A lot of the first major industries that started in America were based on stealing the technology anyway - not completely unlike piracy in China at the moment. In the late 19th century this initial advantage actually started to hurt them in places. A "second" industrial revolution started when better techniques for creating stuff like steel were developed. Nations a little behind the times compared to Britain could buy better stuff fresh without the perceived cost of replacing older equipment. A newly united Germany bears mention here. Relevance to Windows development Programming in Windows is a bunch of old equipment that needs replacing according to the Ars series (they are not at all alone in this suggestion). Old 16bit windows stuff will still work in the new 64 bit systems, but the odd things you have to account for to maintain this backwards compatibility makes programming new stuff for it a tremendous pain. Relatively late comers (in terms of popularity) like Apple and the more user friendly Linux distributions like Ubuntu can and do start from scratch in order to build better tools and not incur the kind of fallout that would happen if MS did the same. Apple pulled this of by creating a virtual machine for their old apps. Desktop Linux distributions don't really have to worry about this because their user base really isn't so large as to cause mass customer loss at breaking compatibility. The apparently sexier place to do development is for the web, which is OS neutral and can increasingly do stuff that was once limited only to the desktop. That stuff that still needs to be done on a local machine - heavy audio/image/video processing - have a lot of good tools on alternatives to Windows. The only exception here seems to be in the gaming industry, which in general steadfastly refuses to develop for multiple operating systems because, from what I understand, the open tools for rendering graphics (OpenGL) are more difficult to use than Microsoft's DirectX.

In a decade or two, anything less will be like being illiterate

I casually linked a 12 year old who is doing a lot of development on a major Javascript framework in the previous post. Here he is giving a talk at Google on jQuery. The tools will only get easier and the next generation will have no problem picking it all up with their sponge-like brains. There are four consequences of this that I can immediately think of:

• More open source development. It looks great on a resume, there is a possibility of making a big name for yourself (publish something in academic journals, giving talks at Google, etc.) the satisfaction derived from contributing, and naturally you, like everyone else, get the benefits of the improvements.
• Possibly a citizenry that can more easily identify logical fallacies. Perhaps nothing is more important for a democracy besides free speech. As with the open source stuff, people should demand more transparency from their elected officials, and even if they don't get it though official channels, they will always find ways to fulfill this need.
• A greater appreciation for mathematics. Hopefully higher level math taught earlier, when the kids can absorb basically any information.
• More rational discussion of basic scientific facts about the world around us. Less of this tragic, mortifying stupidity.

What language should you learn first?

The breakdown Read some stuff on the internet and you will see everyone has an opinion on which language you should learn if you are just starting to program. 1. Hippies: Python, Smalltalk, Visual Basic, Java. This school assumes most beginners don't need to know anything about pointers or garbage collection, and a lot won't at all. If they ever have to learn these things, they are better off picking it up after getting the basics of object oriented design and general logic down. 2. Masochists: C++, C. Learn the hard way and everything else will be easy. This school weeds out anyone not wholly committed to the idea of coding their whole lives. They'd have you do assembly if they had the time to proofread it and comment how terrible and hopelessly inept you are, assuming you can get anything to work. This method works best if you are a wunderkind that heads development on major Javascript frameworks at aged goddamned 12. 3. Pragmatists: C# or .NET in general, Javascript, maybe Java, whatever your company/school uses. It's possible you'll pick up the concepts necessary to be a decent programmer, but you need to be able to code several things right now and the theories of software architecture can take a back seat until you have more time. An of course there is mixing and matching, which is probably more common than any complete adherence to any school. My story I cut my teeth on C++ back when all the cool kids were doing it in high school, but when I started doing Running Start classes at the local community college in my junior year my embarrassingly poor math skills at the time precluded taking basically any CS course. I did manage to make a very inelegant Tic-Tac-Toe program with a pretty decent AI, but that was the sum total of my experience. The book I had probably wasn't a good idea to use (I picked it up at the remainder book store I worked for at the time). Later, when I got into Geography (and GIS specifically) I went through a slow evolution: from the moderately crummy visual ModelBuilder program in ArcGIS to VB.NET - the latter via a course my wife took before I did and aced - to Python, and finally now fooling around with .NET stuff but looking back at C++ again. Where to go now At the moment I am kind of between #2 and #3. The one professional coder I know insists I do some lower level (meaning, closer to raw machine language; meaning harder in general) language. Specifically C++, which he could then critique. This would be a really really good idea and is extremely nice of him to offer to do. Long run, I'll probably be a better programmer for doing it. At the same time though, I know a bit of C# already, everything in the office is .NET, and there is a really good chance I'll be using it very soon. Particularly because the small stuff I have written at work (mostly Python) is likely going to be kept around and maintained in some way. And that means moving it to .NET because no one really uses Python here. I'd have real projects to do as opposed to homework from my friend. Also, there are free in-house training programs for it. I'm almost certain to go with #2 because of the long term prospects of actually being good at this (which means versatility) and also because if I am hired on here as a GIS Developer they'll train me anyway. "Specific language is actually unimportant," I've been told. What matters is understanding what you are doing when you give a computer instructions, what object oriented design means, and good practices & algorithms. Anyone who knows the basics, logic, and elegance behind programming can pick up just about any major language in a day or two. My largely uninformed opinion on language C++ is actually scary as hell though, and never something I'd give a beginner. As other people have observed better than I could, C makes you do more work (garbage collection and memory management), higher level languages and .NET does it for you, and C++ appears to do it for you but can fail catastrophically if you don't know what is going on under the hood. Had I known about Python in my high school days I would have started with that, no question.

Mapping Science Itself

This is a Google Tech Talk called Scholarly Data, Network Science, and (Google) Maps. The creative use of maps in this presentation is impressive, but it is interesting to see that the vast majority of the time and effort to produce them consisted solely of data cleanup. This kind of data cleanup - specifically the kind that requires some expertise in the subject (in this case, the metadata associated with huge datasets of high variability), some technical skill to automate processes when you can, and recognizing when it is prohibitive to do so - is likely to be steady, if sometimes boring, work for as long as such data is in demand. I would be curious how much of it they have considered handing off to the crowd (though scholarly work may not really lend itself to the concept as well as, say, images do via something like Google's Image Labeler) Another interesting thing regarding this presentation is the composition of the team responsible for the maps. Geography is always billed as a highly interdisciplinary field and this is a perfect example (though you will notice an abundance of mathematicians, graphic designers/cartographers, and some computer science folks). These guys sell their maps here. All of the money goes to getting such maps into schools where they can do some real good.