Python

A collection of 56 posts

Python

Copy PySpark Schemas to Clipboard

Frequently, I have a need to both explicitly define input data schemas for processing using PySpark and simultaneously include the schema in Sphinx documentation. Usually I can get the schema for loading without too much difficulty by outputting df.schema in a Jupyter Notebook, and copy-paste into my loading method.

Python

Install ArcGIS Python API on Apple Silicon

NOTE: This has been resolved by the ArcGIS Python API team. Now, you can install the ArcGIS Python API without having to use the Rosetta Emulator. Apple's new ARM-based processors, the M1 and M2, "Apple Silicon," chips, offer impressive performance. Unfortunately, when it comes to the

Python

ArcPy Python Logging Handler

Update 01Nov2022 - After Pat's comment, I decided to create a discrete repo, ArcPy-Logger, to make it easier to use.  In the process, I also updated the Gist based on a few tweaks I made, and added brief documentation as well. Especially with long-running or even scheduled tasks,

Python

Learn to Code

"I hate doing things twice!" I was in Okinawa, Japan on a US military installation. At the time I was a software instructor, and I was watching another contractor doing some maintenance on one of the computers literally writing a batch script faster than I could type. It

arcpy

ArcGIS Pro Ensure Minimum Version in Python

Functionality is continually added to ArcGIS Pro. When creating Python scripts taking advantage of new functionality, it is helpful to be able to check the version to either use an alternative method, or at least provide an error message informing the end user of needing to upgrade. Fortunately, ArcPy provides

Python

ArcPy Describe <dot> Properties

A tremendous amount of information is available through arcpy.Describe. Until ArcGIS Pro 2.8 though, figuring out what properties were available for the object you had just described required digging into the documentation pretty deep. Now, it is much easier to access these properties introspectively as a dictionary. Since

Python

Python Dictionary as Object

Update 16Aug2022: When I shared this on LinkedIn, Ian Broad commented to let me know about a great Python project doing exactly this, Bunch. As is frequently the case with Python, the solution is a pip or conda install away! It's something simple enough, I just want to

Python

Conda and ArcGIS Pro 2.7

For those of us who regularly work with Python and ArcGIS, especially for data science, life just got a lot easier with ArcGIS Pro 2.7. Now, arcpy can be installed just like any other Conda package. This means ArcGIS Pro now can coexist on the same machine and play nicely with Conda.

Python

Cloning ArcGIS Conda Environments

ArcGIS Pro, when initially released, included an incredibly powerful capability, the included Python environment is a Conda environment. This unlocks the extremely powerful and vast universe of installable packages available through Conda. Further, using Conda environments enables ensuring a project can be successfully moved to another machine, and successfully run

Python

Create `arcgis` Conda Environment with a Batch File

Although first and foremost a Geographer, after transitioning to the GeoAI Business Development Team at Esri, now I spend most of my time in Jupyter Lab. Although not altogther difficult, if following best practices, when installing other packages and modifying the primary Conda environment shipped with ArcGIS Pro, arcgispro-py3, you

Python

Automatically Convert to Esri Spatially Enabled Dataframe

I frequnetly need a Spatially Enabled Dataframe from a variety of sources, and these sources frequently change. Guessing there is a very good chance I am not the only one doing this - here is my solution to this as a Gist you are more than welcome to use.

Python

Relative Module Imports in an ArcGIS Python Toolbox

The idea of ArcGIS Python toolboxes really appeals to me. I really like them. As is frequently the case in life, the devil is in the details. Python Toolboxes have plenty of interesting personality quirks. Among these is relative module imports when creating Python Toolboxes for ArcGIS Pro. Here is

Python

Purge Scratch GeoDatabase

In a lot of the scripts I write, especially data preparation pipelines, I use the scratch Geodatabase - a lot. At the end of these pipelines, especially when testing and re-running the processes multiple times, I like to clean out this workspace to ensure my results are the new results,

arcpy

Use ArcPy to Match Spatial Reference

Frequently when performing data preparation or analysis in GIS the spatial reference or projections need to be modified to match. Also, much of the time the data needs to be modified to match one specific feature class. The difficulty lies in frequently also needing to select a transformation if the

Python

Creating a SpatialDataFrame from a Web GIS Item

The relatively recent addition of the SpatialDataFrame object to the ArcGIS Python API was a gigantic leap forward. The SpatialDataFrame marries the common data structure used for data munging and data science, the Pandas DataFrame [https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html], with ArcGIS in a familiar

Python

Keeping the ArcGIS Python API with ArcGIS Pro Current

If you are running the ArcGIS Python API in a virgin Conda environment, this very likely does not apply to you. If you are running the ArcGIS Python API in the Python environment installed with ArcGIS Pro, as of today 05Mar2018, you are stuck at version 1.2.5 while

Pandas

Filter NotNull Geometries in an ArcGIS SpatialDataFrame

When working with Pandas DataFrames it is fairly common to filter out records with null values using the notnull Pandas function. This however, does not work with the geometry column created when loading data into a SpatialDataFrame. It still has a value, albeit an invalid value, but Pandas still sees

Python

PyCharm Debugging - Stop on Unhandled Exceptions

One of the most useful features of PyCharm is the robust debugging capabilities, especially the ability to set it to stop during debugging when encountering an unhandled exception. Enabling this, I frequently have to remember how to enable it on my machine. It is not altogether difficult - just not

ArcGIS

ArcGIS Python Reproject Geometry Object List

Frequently I have a large list of geometry objects I need to get reprojected for analysis. A while back I discovered the reprojectAs method of the arcpy.Geometry [http://desktop.arcgis.com/en/arcmap/latest/analyze/arcpy-classes/geometry.htm] object. While useful, I still needed a way to handle a

Conda

ArcGIS Pro + Conda + Jupyter Notebook + ArcGIS Python API

ArcGIS Pro, like it's predecessor ArcMap, installs Python tightly integrated with the software. ArcGIS Pro 1.4 installs Python 3.5 complete with the Conda package management system [https://conda.io/docs/index.html]. The environment ArcGIS Pro uses is actually even a Conda environment named arcgispro-py3. This

Python

ArcGIS Chunk Data for Analysis Using Python

Frequently I have to process very large datasets so large my computer with a quad core i7 with 20GB of RAM cannot handle it. The solution is to chunk the analysis. While there are multiple ways to accomplish this, since ArcGIS Geoprocessing tools honor selections, the analysis can be chunked

Pandas

ArcGIS to Pandas Data Frame v2.0

One of the most popular posts on my blog discusses loading data from an ArcGIS table, either spatial (a Feature Class) or even nonspatial, to a Pandas DataFrame [http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html] to take advantage of the powerful data analysis tools available in Python

Python

Displaying Matplotlib Graphs Inline in Jupyter Notebook

* Updated to include how to modify the config file so this is the default behavior. Recently I have begun to try, at least at a cursory level, to begin to understand some of the world of machine learning. Since I already know Python, doing this in Python seemed a good

Python

ArcGIS + Python + Jupyter Notebook = Win!

One of my favorite tools when working with Python and exploring how things work is Jupyter Notebook [http://jupyter.org/]. Jupyter Notebook however, is not included with the ArcGIS installation of Python. Until the 1.3 release of ArcGIS Pro, there really was no practical way to install packages to

Python

Getting Started with Conda and ArcGIS Pro 1.3

Python is an incredible tool for streamlining workflows and doing analysis with ArcGIS. However, until now it was frustratingly difficult, or practically impossible to take advantage of the plethora of packages available to streamline workflows, and make life much easier for scripting. With ArcGIS Pro 1.3 comes the ability