Cleanse Feature Class Names in Python

Today I ran across a problem with what should be a simple solution, but took me the better part of the morning to figure out…primarily because I was not familiar enough with regular expression matching. ArcGIS is pretty picky about feature class names, whether a feature class in a geodatabase or a shapefile. Either way, the feature class name must start with a letter and the rest of the characters can only be either letters, numbers or underscores.

My script is pulling names as part of selections from an external source. The names can and do break all of the aforementioned rules. Hence, I had a need for a name scrubber. Although it took more time than it should have, the end result is a two line function I have shared as a Gist.

Hopefully this saves you a little time and heartache! As always, have fun!

Update: Now I feel really stupid. Fifteen minutes after figuring this out, a colleague pointed out the arcpy.ValidateTableName function. Not only does it clean up invalid characters, it also checks existing tables for conflicting names. Again, learn from my labor and frustration, but if nothing else, have fun!

Esri-Leaflet, Lightweight Open Source Esri Web Map Control

Recently I stumbled across a blog posting detailing recent open source JavaScript projects Esri has been working on. Although all of the projects are interesting in their own right, the one really catching my attention is the Esri-Leaflet project. Esri-Leaflet offers a lightweight alternative to the much more robust Esri JavaScript API.

Esri-Leaflet is nowhere near as robust nor as functional, and this is advantageous. The Esri JavaScript API excels at building robust web mapping applications. However, were I envision Esri-Leaflet being useful is for building web applications with a map.

What exactly does the last statement mean? If you want to build a full frame web mapping application with controls built into the map, the Esri JavaScript API still is the way to go. However, if you want to integrate a map into a larger web application, Esri-Leaflet is a very viable alternative.

This is how I discovered Esri-Leaflet. Recently I have been trying to build a web application using a combination of Ember.js and Bootstrap. Obviously Bootstrap is being used for the layout and Ember is being used to construct the architecture and much of the business logic.

Generic Map ApplicationThis web application needs to have a map control integrated into it. The map control needs to include a basemap and a dynamic layer. The dynamic layer needs to reference a feature service hosted on ArcGIS Online. Further the map needs to center on the current feature for the page and only display this feature when viewed. The image to the right is a Balsamic mockup of what I envision the page looking like.

The Esri JavaScript API, for this purpose, is complete overkill. Esri-Leaflet includes the capability to do everything I need.

The next step is actually taking my idea from concept to something closer to reality. If and when I figure out pieces to this puzzle, I will post updates. First, I am going to tackle how to include a map control in an Ember view.

Here are the references I am using thus far to try and figure this out.

CodeSchool JavaScript Road Trip

Ask my mother. She will be the first to tell you. I am stubborn. A few years back she told me, “We raised you to be independent, but you really took the concept and ran with it.” My mother has a lot more tact than I do.

On the good side, pure stubbornness and persistence is how I learned ArcGIS and later Python scripting. By nature I am curious and like to be able to do things and know how they work. If there is  a better way to do it, a way to build a better donculator, I want to be able to figure out how to build the donculator. As of late, this has meant a desire to learn a lot more about web technologies, specifically the language of web applications, JavaScript. Continue reading

Installing GeoDjango on Ubuntu 12.04

Although not very good at it, I have been toying with what I can do with Django. Right now it admittedly is not much. It is rather pathetic, really. Still, being a GIS guy, when stumbling across GeoDjango, I was curious. It took some rough hacking to get something working, but here is what I have come up with so far.

Add the UbuntuGIS PPA

I am adding the stable PPA repository, adding it using the follwoing command.

sudo add-apt-repository ppa:ubuntugis/ppa

Install software

Get going and install a heck of a lot of software. There are going to be dependencies galore, so you may as well get comfortable.

sudo apt-get install python-gdal python-mapnik2 python-pyproj postgresql-9.1 postgis pgadmin3 libspatialite3 spatialite-gui

Although these packages may not look much different than what you would normally get, by adding the UbuntuGIS source, the packages are more current versions of the same software. Specifically and most notable, PostGIS in Ubuntu 12.04 is version 1.5, but by using this repository, you are getting 2.0.1.

Also, you may notice we are adding two database options for working with Django, PostgreSQL with PostGIS and SpatialLite, the spatial extension for Sqlite. It is entirely up to you (obviously), but my thought is to have both to see what GeoDjango can do. Best of luck and please let me know your mileage if you decide to try this.

1-6-2014 3-36-56 PM

ArcGIS for Server Credentials Self-Service

I am jumping the gun a little on this one, but there is a good reason. Off and on, when time allows, I have been putting together a solution to a common issue when an ArcGIS for Server site is set up to use the local user store. When the local user store is used, only the administrator can change any of the user’s credentials. This means individual users cannot change their own password.

The solution involves a geoprocessing and web component. The solution includes a Python geoprocessing script tool to be published to the ArcGIS for Server site. The web component is little more than an html form posting to the rest endpoint made available by publishing the script.

I have the first part working, the Python script. This is the part where my experience is. I Python. The web form has slowed me down a smidge though. Although slogging away at it, I still have some work to do before the form validation and posting is all working correctly, the way I want it to.

This is why I am telling you about this before it is fully baked. If you have the expertise to help out, this project is already publicly available as a GitHub repo. Please feel free to fork it and help out with  finishing the validate_change_pass.js file. I welcome any assistance and expertise to get this completed.


Purge Ubuntu 12.04 Desktop

Frequently I use Ubuntu 12.04 Desktop as a virtual machine for working with a completely virgin Python development environment. Right after initial installation, I typically remove a lot of the things Cannocial tosses on there to make it a really nifty and cute desktop environment for people wanting this type of thing. This morning on my new work laptop I did not have the virtual machine I typically use, so I set up a new one and finally figured out a little faster way to get the initial first few things set up, installing Chrome and uninstalling the office and messenger applications.

Installing Chrome

Thank you to NixCraft for this one.

  1. Download the Chrome deb file from
  2. Install Chrome from the command line
    $ sudo dpkg -i google-chrome-stable_current_amd64.deb
  3. The install will ultimately fail. Resolve the dependencies using this command.
    $ sudo apt-get -f install
  4. Open Chrome and lock it to the launcher

Take Out The Trash

Next is getting rid of all the stuff I do not want on the system. This is pretty easy using apt-get.

sudo apt-get purge "gwibber." ".firefox." "indicator-messages"

Yes, there are likely other packages I could toss out, but this at least gets rid of the stuff I do not immediately want. If you know of more, please feel free to comment below and help this list out.

Upgrade All Python PIP Packages

The discussion could go on for days on why I should be using virtualenv. On my Mac, I am not…much anyway. Get over it. Given this reality, I have a number of packages installed using the Python Package Index (pip) and want a command to keep them current. Digging around the interweb, there appears to be no built in way to accomplish this. Fortunately, there is somebody who knows a thing or 12 about the *nix command line and posted a great little snippet for accomplishing the task as a response on StackOverflow.

pip freeze --local | grep -v '^\-e' | cut -d = -f 1  | xargs pip install -U

Setup OSX for Python PIP

Getting Scrapy installed on OSX proved to be a pain. One of the requirements for Scrapy is LXML, requiring a C compiler. Even with XCode installed, I was missing one thing. For Python Package Index  (pip) packages to successfully compile when installing, one more thing needs to be installed, the command line tool for XCode. This can be installed using the command:

xcode-select --install

Once installed, installing Scrapy requires little more than using the command:
pip install scrapy


New Year?

Happy new year?…hmmmm…sure. The new year, the time it falls during the year, has always been a somewhat depressing time of year. New year, happy…not so much.

Full disclosure: All of my life I have been a summer sports guy.

In college I was a runner. Since college, I have been a paddlesport athlete. More recently I have been trying to teach myself to be a somewhat better mountain biker on a singlespeed hardtail 29er. I continue to train for all three. True, my training is not really all that serious anymore, but enough (in the words of new daddy of twins Nathan Davis) to not get fat. Continue reading

Automatically Find and Log All Broken Layers in Multiple Maps

Last week while teaching a Python class a student asked about a logical workflow, finding all maps with broken layers under a parent directory and reporting the results. Ideally this could be set up as a scheduled task and the results either written to a log file or better yet, emailed to somebody.

This appeared a straightforward opportunity to use the arcpy.da.Walk method with the optional ‘Map’ filter to return all maps. In my testing though, no matter what I tried, I could not get this method to return any maps.

Fortunately, maps are not located in a geodatabase and as a result, Python’s os.walk method can be used instead. Since all map documents end with the mxd suffix, the results of os.walk can be filtered using the Python string.endswith method. Once all maps have been located under the parent directory, the arcpy.mapping.ListBrokenDataSources can be used to locate broken data sources.

The way the script is designed, if there are no broken map layers, no output will be created. However, if broken map layers exist, an output log file will be created in the parent directory listing all the maps and their respective broken layers. This is accomplished using the Python logging module. Although arguably overkill for such a simple task, if you are interested in modifying this script to support sending an email, it would only require taking advantage of logging.handlers.SMTPHandler (link) to send an email via an available smtp server.

Once I got the entire thing working, I uploaded it to a GitHub repository (link). From here you can download the script or fork it and improve it if you are versed in Python.