TileMill - what it does and some reasons to try it

We were promised jetpacks, but I'll take Tilemill as a temporary replacement: http://mapbox.com/tilemill/

The MapBox/DevelopmentSeed team has created one of the the last pieces really needed for mainstream open source GIS to gain really massive appeal

TileMill is used for making web maps - or more specifically - for generating tiles that make up the now-ubiquiteous slippy maps we see online.

There are other desktop applications that do this, the most notable being ArcGIS Desktop. But Desktop was built for other things first: advanced analysis tools, some pretty powerful editing capabilities, and authoring paper maps.

TileMill does one thing and it does it well. It costs nothing (compared to several thousand for some flavor of ArcMap), and outputs an open tile format that you can wire up to a webmap or iPad in less time than it takes to install ArcMap.

And it is smooth. The user experience is the best I have had with a desktop application in a long while.

It also has sane, plaintext css-like styling (MSS). This may sound like a no-brainer, but your options before this were basically some proprietary binary format from ESRI (not extensible, difficult to automate, limiting, vendor specific) or SLD, which is open source but widely regarded as something of a mess for other reasons.

There is also the training issue. ArcMap is giant and powerful - and extremely complex. The market for "GIS Analysts" is still strong in a large part because of this complexity. Less experienced users will find TillMill easier to pick up and web designers (of which there is a large pool of talent) will find it very easy.

It is out for every operating system of note.

Seriously, go give it a try.

What else is needed

Conversion - minimal, well documented, mostly automated steps from ESRI - TileMill/FOSS

Samples like crazy - more or less emulate the ESRI samples, complete with documentation

Posted via email from The Pragmatic Geographer

Introducing (belatedly) nx_spatial

It's been more or less done a while, but here is finally a blog post about it.

nx_spatial is a collection of addon functions for the networkx python graph library. What can you do with it?

1. Load GIS formats into networkx graphs (where you can do all sorts of crazy analytics on them)
2. Perform upstream and downstream traces with stopping points.
3. Set sources and find/repair edges that don't have the correct to/from nodes.

Example from the wiki:

>>> import nx_spatial as ns 
>>> net = ns.read_shp('/shapes/lines.shp') 
>>> net.edges() [[(1.0, 1.0), (2.0, 2.0)], [(2.0, 2.0), (3.0, 3.0)], [(0.9, 0.9), (4.0, 2.0)]] 
>>> net.nodes() [(1.0, 1.0), (2.0, 2.0), (3.0, 3.0), (0.9, 0.9), (4.0, 2.0)] 
>>> source = (2.0, 2.0) 
>>> ns.setdirection(net, source) 
>>> net.edges() [[(2.0, 2.0), (1.0, 1.0)], [(2.0, 2.0), (3.0, 3.0)], [(0.9, 0.9), (4.0, 2.0)]]

Available on pypi or bitbucket. Eventually I want to integrate it with networkx trunk (loading shapefiles is already in 1.4).

Posted via email from The Pragmatic Geographer

WhereCampPDX 2010: Some of the Sessions

It's going to take me some time to fully decompress and write this up, as a good portion of what I saw in Portland was new to me in some form. It is easy enough to read about the hacker community and fiddle with the cutting edge stuff like GeoCouch: it is quite another to actually meet the people and see what they are cooking up.

QTDesigner (Holly Glaser), GeoCouch/GeoJson (Max Ogden)

I hopped between sessions on this one, so I missed about half of Max's talk. The QT talk was small and had David Turner walking Holly through making a survey plugin.

The latter session was showed off an interop tool for converting shapefiles into GeoCouch and why you might want to use that datastore over say PostGIS. The larger point here, which was only briefly mentioned and which I'd like to examine more closely at a later date, is the assertion that simplicity in this sphere breeds adoption. I don't know I necessarily agree.

Domination of the Shapefile (James Fee)

I don't recall the real title, but the alt title was "Why I was Wrong in my last Keynote". It turns out Spatialite, though it was an improvement over the traditional method of bulk data delivery (masses of shapefiles), didn't have enough going for it to actually see mainstream adoption. There just isn't enough wrong with shapefiles to bother switching for data providers. Worse is simply better, at least in this case.

Still, there are problems in data interchange that need to be addressed, the big ones being (1) styling, (2) attribute relationships. I honestly don't see too big a problem with the shapefile being multiple files, since people have a tendency to zip up such things anyway and you probably want more than one shapefile at a time.

This was an interesting discussion and probably merits its own post, the examples were particularly illuminating (FGDC, GML, Gulf Spill map styling).

Keynote (Nathaniel Kelso)

Nathaniel works for the Washington Post making beautiful maps. He showed off examples - the accuracy of the coordinates for WhereCamp, Top Secret America, people estimation at the Obama inauguration - around the theme of "filling in" the map. Of an estimated 2.3 million habitated places we have good maps for a small fraction of them.

Multimodal Trip Planner (David Turner)

This talk is an odd intersection of network topology, real-time scheduling, dynamic mobile resources, and human behavior. When people say they only want to travel on bike lanes, does that mean they'll go five miles out of their way to do it? What if they want to bike only downhill by cleverly leveraging Portland's excellent mass transit system?

One thing I didn't bring up, and wish I had, is that this is an app that essentially lets people plan their travel around a relatively fixed mass transit schedule. What happens when everyone is using such information all the time? Would it be possible to start dynamically modifying the transit system itself to best integrate with user demands?

Geodata for the Masses (various)

I caught part of this talk late after briefly stopping in on the OSM editing and civic apps sessions. By the time I got there it appeared to have devolved into a talk on DIY data collection by kite and balloon.

Posted via email from The Pragmatic Geographer

Google Wave

Yesterday at their I/O conference, Google demonstrated a new communication/collaboration tool that appears to be intended to kill email as the standard form of personal web communication. My first feeling is: Good! Have you ever tried to use email for long, branching communications? The flat, threaded view is trash, and a tree view is not much better. I would love to see this get mainstream attention because it might not only kill email, but also kill off less collaboration tools I personally find less than enjoyable to use. Sharepoint, for instance, I have found to be clunky and have poor integration with email. That isn't really the fault of Sharepoint - email was never meant to be an issue tracker or dynamically update websites. Nor would you want it to, it doesn't have the tools. One of the first things they show off with Wave is how easy it is to do just that (particularly bugtracking). Wave is also free, open source (in some fashion or another), and federated. That last bit means your Waves don't necessarily need to live or interact with Google at all - just like internal email servers you can whip up a Wave server and apparently quite easily. The motivation for Google doesn't immediately occur to me. Best, moderately educated guess is that it might have something to do with HTML5 - the next version of HTML currently supported by roughly zero browsers yet - will be easier to index and search than Flash/Flex/Silverlight. Is Wave supposed to be HTML5's killer app, driving the browsers to pick up support for it faster? As a side note, there appears to be a conspicuous number of references to the television show Firefly in the above presentation and in the project. In the show, the standard communication term used was 'wave', the poll robot they demonstrated included an option for Serenity as the Best Movie Ever, etc. This isn't too wild for them. Keep in mind that Google Earth was apparently inspired by the "Earth" program described in Snowcrash.

The i9/11 and the iPATRIOT Act

Excellent, excellent talk on the future of the internet with some very smart people. Jonathan Zittrain: Author of "The Future of the Internet and How to Stop It" (available free here). Lawrence Lessig: Lawyer and author of dozens of books about free culture, recently having switched from talking about copyright specifically to a more broad range of IT and government corruption issues. Vin Cerf: He basically invented the internet with two other guys. No, really. Currently he is Google's chief internet evangelical (that is his real job title). The whole video is good, but the thing that stands out the most is a comment made by Lessig - the PATRIOT Act wasn't written really quickly in the wake of 9/11, it was sitting in a desk drawer waiting for the opportunity to be passed by a frantic legislature. He asked Richard Clarke, former chief counterterrorism advisor to numerous presidents, if a version of the PATRIOT Act for the internet was waiting for some kind of major internet-based attack which could be compared to 9/11 in terms of disruption and financial loss. Clarke said that, without question, there was. The heart of the problem is our national infrastructure is based on a system that our representatives are largely entirely ignorant of. Here is Senator Ted Stevens demonstrating such ignorance. Here is John McCain admitted to not knowing how to use a computer, much less the internet. None of this would be a problem, except these are the kind of people that can (at least attempt) to legislate internet activity in the country. And when some major incident occurs, which is inevitable due to the open nature of the internet, they may kill the largest and most important communication/content/distribution instrument ever conceived by humankind.

The viable solution to internet piracy

The sad fact of the matter is piracy as it is currently defined cannot be stopped without enormous breaches of privacy. Absent the restriction of tools to even create programs that can effortlessly defeat protection methods, you can't stop people from copying digital information. Mostly because the mere act of being on the internet means you are required to copy things just to view/listen to them. This is a sample of some of the points of a paper I recently read at Cato called The Future of Copyright. My personal favorite part:

When American troops liberated the city of Luxembourg in 1944, they made a strange capture: a machine capable of recording sound on magnetic tapes. Shortly after the war, this German military invention made its appearance in private homes. Tape recorders integrated listening and reproduction in one device, but as separate functions. That’s no longer the case with digital technology. Today, to use digital information is to copy it.

Computers operate by copying. They couldn’t care less whether the physical distance between original and copy is measured in micrometers or in miles; both work equally well for them. Copyright law, on the other hand, must somehow draw a line between use and distribution. That means putting an imaginary grid over the chaotic myriad of network nodes, delineating clusters of devices that can be attributed to individuals or households.

Every person reading this article is actually copying it illegally. You can't help but do it. As you surf the internet your computer is constantly caching (saving locally) data that you are viewing. Anyone who has ever used the internet for anything is almost certainly a pirate. I'll let you all off with a warning this time. Humor aside, the problem is one that the music industry found itself in a long while ago with the invention and mass adoption of radio. John Philip Sousa was convinced anything but live shows would completely destroy music. The solution, after much complaining, was to just license the distributor. Now of course the distributor and consumer and creator is anyone and everyone on the internet. The compromise is an idea that is not new and is even talked about by both sides of the issue with increasing interest: mass public electronic media licensing. If everyone on the internet is a pirate, license everyone. Everyone who wants to buy in at least. You've solved 99% of your piracy problems by facing the reality of a system, the internet, that requires copying. You also don't even have to host the files themselves, users are more than willing to do so. The idea has merit but the problem is the pie to be divided. We need an organization as trusted as the Nielsen statistics are for downloaded media content to correctly award creators and their labels/studios/whatever. The only alternatives are to entire restrict piracy with the kind of locking down that would destroy the internet and privacy in general (people could still just rip stuff from friends/renters) OR to see an attempt to destroy internet and privacy while the major content holders die a slow and painful (for everyone involved) death at the hands of unrelenting technological innovation. Edit: Hahaha. Bonus quote from a Guardian article linked from the Cato piece:

"For somebody who has spent 30 years in the music industry, you instinctively know this stuff is going on. But when you actually sit looking at your computer and see a number that says 95% of people are copying music at home, you suddenly go, 'Bloody hell'," he said.

Turns out you could nuke the internet from orbit and the current copyright model is doomed. Note: Copying this work is no longer actually illegal in any sense, since I have licensed everything with Creative Commons.

The benefits of being a late bloomer

This isn't a post about an embarrassing teenage angst. It is actually a story about natural resources, the industrial revolution, and programming for Windows. Ars Technica, one of the better tech sites on the internet, recently did a series on the lead up to the most recent malaise we've seen with Windows programming. As I read the piece, I kept noticing parallels between Microsoft's difficulties and the early innovators in the industrial revolution. Despite the fact that you probably know nothing about the 19th century about Britain besides the fact the nascent United States kicked its ass off the better parts of North America, it is actually pretty relevant. Rule Britannia Britain ran the 19th century. It controlled all of the sea lanes, dominated world trade/finance, and ran nations which were ten times its landmass and population by occasionally machinegunning down crazy-brave hordes of natives who were attacking them with spears. It got this way because it not only was the first to recognize and reap the massive benefits from new industrial machinery, but also because it ruthlessly closed itself off in terms of technology to retain their advantage. A lot of the first major industries that started in America were based on stealing the technology anyway - not completely unlike piracy in China at the moment. In the late 19th century this initial advantage actually started to hurt them in places. A "second" industrial revolution started when better techniques for creating stuff like steel were developed. Nations a little behind the times compared to Britain could buy better stuff fresh without the perceived cost of replacing older equipment. A newly united Germany bears mention here. Relevance to Windows development Programming in Windows is a bunch of old equipment that needs replacing according to the Ars series (they are not at all alone in this suggestion). Old 16bit windows stuff will still work in the new 64 bit systems, but the odd things you have to account for to maintain this backwards compatibility makes programming new stuff for it a tremendous pain. Relatively late comers (in terms of popularity) like Apple and the more user friendly Linux distributions like Ubuntu can and do start from scratch in order to build better tools and not incur the kind of fallout that would happen if MS did the same. Apple pulled this of by creating a virtual machine for their old apps. Desktop Linux distributions don't really have to worry about this because their user base really isn't so large as to cause mass customer loss at breaking compatibility. The apparently sexier place to do development is for the web, which is OS neutral and can increasingly do stuff that was once limited only to the desktop. That stuff that still needs to be done on a local machine - heavy audio/image/video processing - have a lot of good tools on alternatives to Windows. The only exception here seems to be in the gaming industry, which in general steadfastly refuses to develop for multiple operating systems because, from what I understand, the open tools for rendering graphics (OpenGL) are more difficult to use than Microsoft's DirectX.

In a decade or two, anything less will be like being illiterate

I casually linked a 12 year old who is doing a lot of development on a major Javascript framework in the previous post. Here he is giving a talk at Google on jQuery. The tools will only get easier and the next generation will have no problem picking it all up with their sponge-like brains. There are four consequences of this that I can immediately think of:

• More open source development. It looks great on a resume, there is a possibility of making a big name for yourself (publish something in academic journals, giving talks at Google, etc.) the satisfaction derived from contributing, and naturally you, like everyone else, get the benefits of the improvements.
• Possibly a citizenry that can more easily identify logical fallacies. Perhaps nothing is more important for a democracy besides free speech. As with the open source stuff, people should demand more transparency from their elected officials, and even if they don't get it though official channels, they will always find ways to fulfill this need.
• A greater appreciation for mathematics. Hopefully higher level math taught earlier, when the kids can absorb basically any information.
• More rational discussion of basic scientific facts about the world around us. Less of this tragic, mortifying stupidity.

Hybrid firms and Proprietary Software Licensing

This kind of thing is why I wouldn't want to open my own GIS oriented business with proprietary software.

ESRI have told us that as an educational charity we are no longer allowed to have an educational discount for using their software and, not only that, our license codes will cease to work at the end of this month. So, we have 3 weeks, with the Easter holidays in the middle, to extricate ourselves and our ongoing projects from ArcGIS and into something else or find the many thousands of pounds to buy the full licenses for all of our staff.

This is my general blog, so for those that don't know, ESRI is the "big dog" proprietary GIS software firm. At least in the United States, it is relatively accurate to think of them as the Microsoft of the computer mapping world (though little can really compare with the OS market share Microsoft commands). The problem here is not limited to ESRI. The idea of time-duration licenses for any good that is tempered by special discounts for educational institutions, charities, etc, is one fraught with difficulties. This is especially as you cross cultural/institutional lines (as is the case with the above example). You can see something similar with annual income taxes, the problems inherent there need really no detailed mention. The risk associated with a change in the licensing regime - specifically in discount - will always result in some firms that are blindsided with an outrageous (and possibly unintentional) cost increase. The risk associated with this happening - which includes not merely the higher licensing fee but also the cost to assess whether it is worth paying for - combined with often already high costs resulting from interoperability issues and current fees. Fair warning or not, a switch being forced by a change in fees is going to cost money. Basically: Cost of Being Locked In + Licensing Fees + Risk of Fee Changes You avoid all of that with open source software, though there is the question of maturity. QGIS, the likely contender for the heart of the user otherwise locked into ArcMap, has come a long way but my short experience with it wasn't entirely pleasant in terms of stability. For other situations, open source software is often equal or superior - Microsoft SQL Server has only just caught up with Postgres for spatial querying. It all depends on the given software. The costs of starting a business is already high. My hope is by the time I can start a GIS consulting firm there will be a simple, stable open source GIS platform/viewer. I don't need it to do a lot really, just display data and allow for easy editing. The spatial analysis can be done though some scripting with Python/SQL, and dissemination though the web with the future version of MapServer/OpenLayers. It sounds simple, and would be perfect for what I am doing right now, but who knows what GIS will be in the future. At the end of the day it is all about what the client wants.

Links for Today

Happy generic Hallmark created holiday everyone. Here is a list of interesting things.

• The brilliant anti-corruption lawyer and author Lawrence Lessig on Obama. I apologize if at any time my rather strong support for Barack Obama has offended anyone who reads this. If you at all on the fence about this guy or just don't really know a lot about him, this is a good thing to see.
• The Reuters news agency releases its semantic web tool. Put in a document of anything and it will spit out metadata, tags, and other information. Very handy for searches. If you could have this accept spatial data you could whip up a generic search tool like what appears on Google Earth.
• Firefox 3 beta 3 is out. Faster and more stable than Firefox 2, which means several orders of magnitude better than IE7.
• Miro. I don't care if I mentioned this in a previous links post, it is still awesome. Free shows and music videos (TED talks, PBS Frontline, The Onion).
• The Escapist; funny game reviews. Funny even if you never played them or only have a passing interest in video games.
• This comic is almost as misogynistic as the comment I left on my mother's blog.

Favorite nerd stuff of 2007

Yes, everyone seems to be doing something like this. And probably got it finished a week ago. So sue me.

• StumbleUpon: I first heard about it as an up-and-coming search-like engine in Wired. Effectively, it is a psuedo search engine that relies on user generated interests - not passively as Google does with link backs, but with actual intentional voting. There is something similar on the open source front (Wikia). Link

• xkcd: How did I miss this gem before? Link.

• DS R4: Play movies, listen to music, emulate classic games, and back up your whole DS game library for a little more than the cost of a single DS game. No mincing words, you want one of these if you own a DS. If you plan on buying one you'd be a fool not to get an R4 (or an M3, which I've been told does basically the same thing). Link.

• Mozilla Firefox 3 beta - It is still beta but amazingly stable. Automatically restarts with the tabs you had open when you closed it, prettier search bar, and way better multi tab support (no slowdowns or crashes with the 987,345 tabs I tend to have open). Bad news: perhaps it is just me, but the bookmarking isn't as good and more difficult to manage. Still really nice for beta.

• A finished Methods section draft: It is a good feeling having this thing 1/2 to 2/3rds done. Lets see how viciously my thesis chair, Bob Hickey, tears it apart. I think I might post it and have the 0 people that visit the site mock it.

• Ubuntu/QGIS: Though not ready to take over the huge ESRI-house I work for or actually threatening Windows desktop domination, the speed and good looks of the new open source GIS platforms that can be created for zero cost (and without having to reformat your computer - use use VMWare Player to create a virtual machine).

• Death of DRM music: For those that don't know, DRM is the reason you can't move around music you buy off iTunes to whatever you want as many times as you like. All of the major music content holders have realized what people on the internet have been telling them for years - only legitimate customers are hurt by DRM. Pirates - a term with already includes probably everyone younger than 20 - simply hack it or rip CDs their friends get during parties. Video will be next as it is easily as trivial to defeat.

• @GoogleTalks: It will be a supreme challenge for Google to keep its relatively good public image even as it grows more massive and run by more suits than engineers. They have given the internet many, many good things. One that is less commonly mentioned is this series of talks which include all of the major candidates, many famous/influential authors, and some other folks. Link.

• Portal: Game of the year out of nowhere. Witty dialog, excellent ending, and interesting mechanics.

Pydev and Python

I am not a software developer and I have no wish to be. However, I am young and being able to customize stuff in GIS is a very marketable skill. Python is an excellent language for this. It's been adopted as the primary scripting language of the industry leader (ESRI), it's popular, easy to learn, easy to maintain, and open source. The only downside is it can't access everything ESRI has implemented in their API - for instance, I am having to basically rewrite the Utility Network tracing tools so I can do some analysis on SRP's water system - but ESRI may not be the industry leader forever. In terms of simple viewers/online mapping it is getting mighty competition from NASA's WorldWind, Google's offering, and an increasingly good package of independently created but easily interoperable open source solutions. Any way it goes however, Python seems like it would help for scripting (unlike the alternative in VBA). JavaScript would be useful too, but I learned Python first and thus I've been spoiled rotten by a language that isn't jam-packed with issues (it is also two big scoops of ugly). Getting to the point, PyDev is an addon for Eclipse that gives you a superb platform for writing stuff in Python. I've been messing with it for barely an hour and it by all appearances is sleek, organized, and fast. It is probably overkill for my one-off scripting needs, but I'm loving the tabbed views, on-the-fly error checks (that are not the annoying crap you get in VBA), and the Outline view. It also does a bunch of other languages as well.