Programmatically tracing a network in ArcGIS

Recently I was tasked with tracing a water network, and after studying a previous programmer's work on the subject, I realized the solution is just a modified tree algorithm that is largely language independent. The trick is a recursive function that continues to call itself until it has gone down the totality of a single path. When it can't keep going - because it reaches the end of the line or because you convince the program it is the end of the line - it hops back one function call and tries to take an unvisited path. If it finds one, it repeats the whole process. The ArcObjects API contains the extensions for Network and Utility Network tools, but as far as I can tell,those are mostly for listening for trace events rather than starting your own. Since a network like that is just a modified topology (it inherits from ESRI's topology object), you can use ITopology to create a TopologyGraph which allows access to the Edges and Nodes required. But you don't need to use C#/VB.NET/VBA to pull this off. It would be possible with the API ESRI provides with its Python scripting object (IDispatch), as that contains start and end points in its Geometry object, or by creating your own node/tree structure. I understand this is a fairly elementary use of recursion, but as a novice programmer its a lot of fun putting it into action for a real project.

Taking a picture of an image without using light that interacted with it.

When you look at something - any object - your eyes are reading information from all of the light reflected off that object. A leaf appears green because it absorbs or scatters all other kinds of light.

Light actually consists of individual particles called photons. An odd thing about particles - if you "entangle" one particle with another, they still are effected by one another over great distances. Change the spin of one, and the other one reacts. It doesn't seem to make too much sense, but that is quantum physics.

So what happens if you could capture the information of light particles that are hitting an object, but you only have contact with their entangled friend? Turns out you can make an image out of it anyway. The information is, I am told, only useful if you get the other one back too, but it is still pretty nifty. Below is an image of a toy soldier they viewed by this indirect method:



This will be sort of a big deal, as in the future, it might allow for the ability to see through all sorts of things. I know is seems strange, since we only see a cloudless Google Earth whenever we want, but satellites still need clear days for good imagery. This might be another way of mitigating those pesky clouds.

HeyWhat'sThat: The coolest mapping site you've never heard of

What happens when someone has too much free time, programming talent, and a burning question? This.



Summary: HeyWhatsThat is basically a rather advanced map hack site which includes viewsheds linked with government/public name systems (names with xy data) so from a given point you can see all the major peaks nearby. The proprietor and designer, Michael Kosowsky, has steadily been adding stuff to it and it now includes pathing (which gives you an elevation profile), and a night view.

I'm moderately curious why Google hasn't bought him out/hired him/stolen the idea. As far as I can tell, Kosowsky built all the tools (minus the Google Map of course) from the ground up. It includes some interesting little quirks based on his response to user comments, like the way it handles the curvature of the earth and the fact it switches to metric if outside the United States.

(To be completely fair, you might have heard about this site. Several other bloggers, including Ogle Earth, have wrote about it.)

The viable solution to internet piracy

The sad fact of the matter is piracy as it is currently defined cannot be stopped without enormous breaches of privacy. Absent the restriction of tools to even create programs that can effortlessly defeat protection methods, you can't stop people from copying digital information. Mostly because the mere act of being on the internet means you are required to copy things just to view/listen to them. This is a sample of some of the points of a paper I recently read at Cato called The Future of Copyright. My personal favorite part:

When American troops liberated the city of Luxembourg in 1944, they made a strange capture: a machine capable of recording sound on magnetic tapes. Shortly after the war, this German military invention made its appearance in private homes. Tape recorders integrated listening and reproduction in one device, but as separate functions. That’s no longer the case with digital technology. Today, to use digital information is to copy it.

Computers operate by copying. They couldn’t care less whether the physical distance between original and copy is measured in micrometers or in miles; both work equally well for them. Copyright law, on the other hand, must somehow draw a line between use and distribution. That means putting an imaginary grid over the chaotic myriad of network nodes, delineating clusters of devices that can be attributed to individuals or households.

Every person reading this article is actually copying it illegally. You can't help but do it. As you surf the internet your computer is constantly caching (saving locally) data that you are viewing. Anyone who has ever used the internet for anything is almost certainly a pirate. I'll let you all off with a warning this time. Humor aside, the problem is one that the music industry found itself in a long while ago with the invention and mass adoption of radio. John Philip Sousa was convinced anything but live shows would completely destroy music. The solution, after much complaining, was to just license the distributor. Now of course the distributor and consumer and creator is anyone and everyone on the internet. The compromise is an idea that is not new and is even talked about by both sides of the issue with increasing interest: mass public electronic media licensing. If everyone on the internet is a pirate, license everyone. Everyone who wants to buy in at least. You've solved 99% of your piracy problems by facing the reality of a system, the internet, that requires copying. You also don't even have to host the files themselves, users are more than willing to do so. The idea has merit but the problem is the pie to be divided. We need an organization as trusted as the Nielsen statistics are for downloaded media content to correctly award creators and their labels/studios/whatever. The only alternatives are to entire restrict piracy with the kind of locking down that would destroy the internet and privacy in general (people could still just rip stuff from friends/renters) OR to see an attempt to destroy internet and privacy while the major content holders die a slow and painful (for everyone involved) death at the hands of unrelenting technological innovation. Edit: Hahaha. Bonus quote from a Guardian article linked from the Cato piece:

"For somebody who has spent 30 years in the music industry, you instinctively know this stuff is going on. But when you actually sit looking at your computer and see a number that says 95% of people are copying music at home, you suddenly go, 'Bloody hell'," he said.

Turns out you could nuke the internet from orbit and the current copyright model is doomed. Note: Copying this work is no longer actually illegal in any sense, since I have licensed everything with Creative Commons.

Monday Links

MS Server 2008 as an OS: Amusingly enough, for all the hatred generated for Windows Vista, a product with supposedly a virtually identical code base is whooping up on it as far as performance goes. Some folks in the comments have suggested this might be because Server 2008 doesn't have the more intrusive DRM that currently is the biggest reason I have no interest in upgrading ("Oh, so I get a performance hit, maybe two useful new features total, and one day you can just take over my computer to prevent me from copying stuff I bought legitimately? SIGN ME UP"). Server 2008 with some minor modifications can do just about everything that Vista can do (music, movies, games, file management) and still move faster. If this forces updates like Vista does and gets at all popular, I expect Microsoft will toss in the DRM stuff. They can't seem to help themselves, they really really want control over your computer and all the software on it. There is a lot of money in it, so I can see the motivation. Brain Rules: John Medina is entertaining in this authors@google video, explaining his new book on some of the things you can do to aid cognitive function, help in relationships, and live longer without dementia. Strange Maps: Exactly what it sounds like. Want a cheese map of Canada or a 1942 map of the "new world order" (a vision of a post war world from the very depths of the war)? This is the place to find it. Gas Prices relative to income: Who is getting hit hardest by gas prices (which is to say, what percentage of your income is being eaten up by it)? I am curious if this map takes into account the average (or median, as one might prefer) gasoline consumption of the areas. It doesn't appear to, but I will attempt to recreate it when I have time to see. Map of School Vaccinations: These are the areas to point fingers to when we see a resurgence of childhood diseases. Normally I would be live and let live about this kind of bottomless stupidity, but a population that will sustain a stable habitat for such diseases will allow said diseases to adapt and render the vaccines ineffective. Also, children should not be responsible to the mass idiocy of their parents. I believe one day I will make a density map of stupidity and this will certainly be one of the variables. Top Five Programming Languages to Learn: Just like learning a foreign language but probably worth far more money if you are really good at speaking it. C# tops the list and I would agree. You can use it to add on to anything having to do with Windows, use it for web applications, Silverlight (a .NET version of the ubiquitous Flash player) and understanding the libraries will come in handy even in an open source/Linux environment with Mono.

The real future of mobile GIS

This. This is it. Here are a pair of guys that have integrated spatial objects into video, allowing a user to pan around with a Android phone (or any camera probably) and have that information appear. Next step, combining that functionality with an artificial contact lens and, if we felt like being really sci-fi, possibly thought activated controls. The technology basically exists but requires the engineering for the best practical applications and integration. All of it would be noninvasive - no need for Matrix-style headjacks or major surgery - and it could just be taken off when you don't want to expend the effort. Then, integrate this kind of thing with existing applications like Whereyougonnabe and Twitter. Never have trouble finding your friends at a social event again, always know the efficient bike/walk/car path to follow, and the ability to immediately search and find anything interesting you want nearby. You could search around you in a fashion that you would otherwise need the internet for, finding friends in a wide-Facebook/MySpace network style and nearby interests by drawing resources from the largest repository of human knowledge ever conceived. Interaction wouldn't be far away. See a bad driver on the road? Vote them down and people will know to avoid him/her. Some area particularly dangerous and you will know. Which bar/club is currently overcapacity and not worth going to visit until later? What food on the menu is actually good? The simple fact is that eventually the concept of "spatial data" will not be discernible as something independent because nonspatial data will be considered so worthless by comparison.

In a decade or two, anything less will be like being illiterate

I casually linked a 12 year old who is doing a lot of development on a major Javascript framework in the previous post. Here he is giving a talk at Google on jQuery. The tools will only get easier and the next generation will have no problem picking it all up with their sponge-like brains. There are four consequences of this that I can immediately think of:

• More open source development. It looks great on a resume, there is a possibility of making a big name for yourself (publish something in academic journals, giving talks at Google, etc.) the satisfaction derived from contributing, and naturally you, like everyone else, get the benefits of the improvements.
• Possibly a citizenry that can more easily identify logical fallacies. Perhaps nothing is more important for a democracy besides free speech. As with the open source stuff, people should demand more transparency from their elected officials, and even if they don't get it though official channels, they will always find ways to fulfill this need.
• A greater appreciation for mathematics. Hopefully higher level math taught earlier, when the kids can absorb basically any information.
• More rational discussion of basic scientific facts about the world around us. Less of this tragic, mortifying stupidity.

What language should you learn first?

The breakdown Read some stuff on the internet and you will see everyone has an opinion on which language you should learn if you are just starting to program. 1. Hippies: Python, Smalltalk, Visual Basic, Java. This school assumes most beginners don't need to know anything about pointers or garbage collection, and a lot won't at all. If they ever have to learn these things, they are better off picking it up after getting the basics of object oriented design and general logic down. 2. Masochists: C++, C. Learn the hard way and everything else will be easy. This school weeds out anyone not wholly committed to the idea of coding their whole lives. They'd have you do assembly if they had the time to proofread it and comment how terrible and hopelessly inept you are, assuming you can get anything to work. This method works best if you are a wunderkind that heads development on major Javascript frameworks at aged goddamned 12. 3. Pragmatists: C# or .NET in general, Javascript, maybe Java, whatever your company/school uses. It's possible you'll pick up the concepts necessary to be a decent programmer, but you need to be able to code several things right now and the theories of software architecture can take a back seat until you have more time. An of course there is mixing and matching, which is probably more common than any complete adherence to any school. My story I cut my teeth on C++ back when all the cool kids were doing it in high school, but when I started doing Running Start classes at the local community college in my junior year my embarrassingly poor math skills at the time precluded taking basically any CS course. I did manage to make a very inelegant Tic-Tac-Toe program with a pretty decent AI, but that was the sum total of my experience. The book I had probably wasn't a good idea to use (I picked it up at the remainder book store I worked for at the time). Later, when I got into Geography (and GIS specifically) I went through a slow evolution: from the moderately crummy visual ModelBuilder program in ArcGIS to VB.NET - the latter via a course my wife took before I did and aced - to Python, and finally now fooling around with .NET stuff but looking back at C++ again. Where to go now At the moment I am kind of between #2 and #3. The one professional coder I know insists I do some lower level (meaning, closer to raw machine language; meaning harder in general) language. Specifically C++, which he could then critique. This would be a really really good idea and is extremely nice of him to offer to do. Long run, I'll probably be a better programmer for doing it. At the same time though, I know a bit of C# already, everything in the office is .NET, and there is a really good chance I'll be using it very soon. Particularly because the small stuff I have written at work (mostly Python) is likely going to be kept around and maintained in some way. And that means moving it to .NET because no one really uses Python here. I'd have real projects to do as opposed to homework from my friend. Also, there are free in-house training programs for it. I'm almost certain to go with #2 because of the long term prospects of actually being good at this (which means versatility) and also because if I am hired on here as a GIS Developer they'll train me anyway. "Specific language is actually unimportant," I've been told. What matters is understanding what you are doing when you give a computer instructions, what object oriented design means, and good practices & algorithms. Anyone who knows the basics, logic, and elegance behind programming can pick up just about any major language in a day or two. My largely uninformed opinion on language C++ is actually scary as hell though, and never something I'd give a beginner. As other people have observed better than I could, C makes you do more work (garbage collection and memory management), higher level languages and .NET does it for you, and C++ appears to do it for you but can fail catastrophically if you don't know what is going on under the hood. Had I known about Python in my high school days I would have started with that, no question.

Mapping Science Itself

This is a Google Tech Talk called Scholarly Data, Network Science, and (Google) Maps. The creative use of maps in this presentation is impressive, but it is interesting to see that the vast majority of the time and effort to produce them consisted solely of data cleanup. This kind of data cleanup - specifically the kind that requires some expertise in the subject (in this case, the metadata associated with huge datasets of high variability), some technical skill to automate processes when you can, and recognizing when it is prohibitive to do so - is likely to be steady, if sometimes boring, work for as long as such data is in demand. I would be curious how much of it they have considered handing off to the crowd (though scholarly work may not really lend itself to the concept as well as, say, images do via something like Google's Image Labeler) Another interesting thing regarding this presentation is the composition of the team responsible for the maps. Geography is always billed as a highly interdisciplinary field and this is a perfect example (though you will notice an abundance of mathematicians, graphic designers/cartographers, and some computer science folks). These guys sell their maps here. All of the money goes to getting such maps into schools where they can do some real good.