ArcGIS Branched Versioning - Start to Finish

ArcGIS Branched Versioning - Start to Finish

When doing ArcGIS Enterprise installs and upgrades, it is extremely common to set up an Enterprise Geodatabase at the same time with Branch Versioning. Although not everybody needs their data versioned, it is extremely common when clients are setting up ArcGIS Enterprise. Typically the entire motivation for implementing ArcGIS Enterprise in the first place is to manage canonical authoritative geographic data such as utilities or land records. Versioning with archiving (part of branch versioning), enables multiple people to edit at the same time, an approval process for these edits with a history of changes.

For these reasons, as part of an ArcGIS Enterprise installation, it is quite common to set up Branch Versioning. The steps themselves are not hard. What makes it confusing is that the workflow jumps between the database, the geodatabase, ArcGIS Pro and ArcGIS Enterprise. Miss one detail in one of those places, and the whole thing quietly stops working three steps later.

So, here is the entire path in one place. We start with an empty database and end with an editable feature service, named versions, and a view-only feature service that does not hold the editing service hostage over schema locks.

First, though, it is worth talking about why branch versioning matters at all.

Why Branch Versioning

Traditional versioning has been around for a long time. It works. I have used it plenty. It was designed in a different era, though: desktop clients connecting directly to the database, editing through a geodatabase connection.

Under the hood, every edit goes into delta tables, the adds (A) and deletes (D) tables. Every edit operation creates a new state, and a state tree grows behind the scenes. Then somebody has to schedule a compress so performance does not slowly fall apart.

Branch versioning was built for a different world, one where editing happens through services and multiple clients (desktop, web and mobile):

Traditional Versioning              Branch Versioning

ArcMap / ArcGIS Pro                 Pro / Field Maps / Web Apps
   |                                   |
   | direct database connection        | feature service
   v                                   v
Base Table + A Table + D Table      ArcGIS Enterprise
   |                                   |
   +-- state tree                      v
   +-- compress                     Base Table (with branch + history fields)

There are a few reasons this matters.

Editing goes through services. Branch versioned data is edited through feature services. ArcGIS Pro, Field Maps, Map Viewer, Experience Builder apps and custom applications all edit the same way, through the same service. One editing pattern serves everybody.

There are no delta tables and no compress. Every insert, update and delete is written as a new row in the base table, with a few system fields keeping track of which branch it belongs to, when it happened, and whether it is a delete. Nobody has to babysit a state tree. This means significantly less database maintenance.

History is built in. Since every edit is a row with a timestamp, the history of Default is just there. You do not have to enable archiving separately to answer, "What did this parcel look like last March?"

The version model is simpler. Every named version is a child of Default. There is one level, with no version trees five generations deep where nobody remembers what QA_Final_v2_REAL is for. You reconcile and post against Default, and that is it.

Newer capabilities depend on it. The service-based Utility Network, Parcel Fabric and Trace Network all use branch versioning for multiuser editing. Esri's investment in versioning is going into the branch model.

There are tradeoffs. Edits have to go through services, so ArcGIS Enterprise becomes part of the editing architecture. Geodatabase replication is not part of the branch versioning story, and the single-level hierarchy will not fit every legacy workflow. For most organizations moving toward web and field editing, though, branch versioning is the model that fits how people actually work now.

Step 1: Create the Enterprise Geodatabase

Everything starts with the Create Enterprise Geodatabase tool in ArcGIS Pro (Data Management Tools > Geodatabase Administration). You need:

  • the database platform and instance
  • the database name (the tool creates it for SQL Server and PostgreSQL)
  • database administrator credentials
  • the geodatabase administrator, sde
  • the authorization (keycodes) file from your ArcGIS Server install

When it finishes, you have an empty geodatabase owned by sde.

Do not load your data as sde. Yes, I am shouting. This sounds ridiculously obvious, but I still see it. The geodatabase administrator administers the geodatabase. It should not also own the parcels.

Step 2: Create the Data Owner

Connect as the database administrator and run Create Database User. Create the user who will own the data. I usually call it something boring like gisowner.

This user matters more than it might seem. Branch versioning requires the data to be registered by the user who owns it, and the published service will connect to the database as this user. Pretty much everything downstream depends on getting this right.

Step 3: Register the Database with ArcGIS Enterprise

Branch versioned data is edited through services, so ArcGIS Server needs to be able to reach it. In ArcGIS Pro, sign in to your portal. Then use Share > Add Data Store, or right-click the connection (the connection as the data owner) and choose Add To Data Store. Pick the federated server, and use the data owner's branch versioned connection.

If validation fails, it is almost never ArcGIS. In my experience, it is nearly always a missing database client on the ArcGIS Server machine, or a firewall blocking the database port. Hence, if you are having difficulties, ensure the client libraries are installed on the ArcGIS Server instance, and the necessary ports are opened.

Step 4: Load Data as the Data Owner

Create a database connection file in ArcGIS Pro authenticating as gisowner, and load your data through that connection. The geoprocessing tools Feature Class to Geodatabase, Export Features, or Append into an existing schema all work.

Before going further, do three things:

  1. Add Global IDs. This is required. Branch versioning will not register a dataset without them.
  2. Enable Editor Tracking. It is not strictly required, but once you are sorting out conflicts between versions, you will very much want to know who changed what, and when. You will thank yourself later.
  3. Get the schema right now. Fields, domains, subtypes and relationship classes are all easier to change before services are pointing at the data. We will come back to this in Step 8.

Step 5: Enable Branch Versioning

This takes two steps, and the first one is the one people miss.

First, set the connection's versioning type. In the Catalog pane, right-click the data owner connection, choose Geodatabase Connection Properties, and set Versioning Type to Branch. If you skip this, Register As Versioned happily registers your data with traditional versioning, and you will not figure it out until publishing fails to give you version management.

Second, register the data. Run Register As Versioned on the feature dataset or feature classes, still connected as the data owner.

Behind the scenes, the geodatabase adds a handful of fields to each table:

GDB_ARCHIVE_OID
GDB_BRANCH_ID      -- which version (0 = Default)
GDB_FROM_DATE      -- when the row was written
GDB_IS_DELETE      -- is this row a delete
GDB_DELETED_AT
GDB_DELETED_BY

Keep these in mind. We will need them again when building the view.

Step 6: Publish the Editable Feature Layer

Add the data to a map from the data owner connection, then go to Share > Web Layer > Publish Web Layer:

  • Data: Reference registered data
  • Feature capability: Create, Update, Delete (and Sync, if field crews need offline)
  • Version Management: enabled

The Version Management capability is what turns on the Version Management Service. Without it, there are no named versions, and everybody is editing Default directly. Sometimes that is fine. Usually, it is not what you actually want.

Share the layer with the groups who will be editing, and you have a working branch versioned service.

Step 7: Editing, Versions and Keeping Things Tidy

Editing Default

Out of the box, anybody editing the feature layer is editing Default. Save the edit, and everybody sees it immediately. For quick, low-risk changes, this is perfectly reasonable.

Working in a Named Version

For bigger work, such as a new subdivision, a design or a QA/QC pass, create a named version so the work stays isolated until it is ready.

Default
   |
   +-- create --> jmccune.Subdivision_Phase2
                     |
                     +-- edit, edit, edit
                     |
                     +-- reconcile  (pull Default's changes in, find conflicts)
                     |
                     +-- post       (push my changes into Default)
                     |
                     +-- delete     (done, clean up)

In ArcGIS Pro:

  1. Create. Open Manage Versions on the service. Create a new version and give it an access level:
    • Private: only you can see and edit it.
    • Protected: everybody can see it; only you can edit it.
    • Public: everybody can see and edit it.
  2. Change Version to the new version and edit away. Nothing you do shows up in Default yet.
  3. Reconcile. This pulls in everything posted to Default since you started and flags conflicts. Detect conflicts by object or by attribute, and decide whether the edit version or the target version wins by default. Then review the conflicts in the Conflicts view.
  4. Post. This pushes your reconciled edits into Default.
  5. Delete the version once you are done with it.
Protecting Default

Out of the box, Default is Public. Anybody with edit access to the service can edit Default directly and post their versions to it. For a small team doing quick fixes, that may be fine. For most organizations, though, Default is the authoritative data. It is what the view-only service shows, what the public app displays, and what feeds reports. It really should not change just because somebody accidentally saved an edit while still connected to Default.

The fix is simple: set Default's access to Protected.

Default = Public                      Default = Protected

Editor --> edits Default directly     Editor --> named version --> edits
Editor --> posts to Default                        |
                                                   v
anybody with edit access can change   Version Administrator
the authoritative data                  |  reconcile, review, post
                                        v
                                      Default

With Default protected:

  • Everybody can still see Default. Readers, the view-only service and anybody browsing the data see exactly what they did before.
  • Only version administrators can edit Default or post to it. By default, that means the service owner and portal administrators. You can give others the ability through a custom role with the version management privilege.
  • Everybody else works in named versions. Editors create their versions, edit, and reconcile as often as they like. When the work is ready, a version administrator reviews it and posts it.

This makes the named version workflow the only way changes reach the authoritative data. Edits get reviewed, and the edits you care about are much less likely to show up unexpectedly in the view-only service.

To set it in ArcGIS Pro, connect to the service as a version administrator, open Manage Versions, select Default, and change its access to Protected.

A couple of things to plan for:

  • Somebody has to post. Protecting Default creates a review step, which means somebody is responsible for it. If nobody owns posting, finished versions pile up and Default goes stale. Decide who your version administrators are before you flip the switch.
  • Field and web editors need a named version. Editors who used to edit Default directly in Field Maps or a web app now need to work in a named version. Check that each app your editors use supports this before protecting Default, or you will find out from a field crew who cannot save edits.
  • Do it early. Protecting Default on day one is a non-event. Protecting it six months in means retraining everybody who has gotten used to editing Default directly.

Version Management

A few things I have learned to do from the start:

  • Decide who your version administrators are. If Default is protected, they are the only people who can post. Make sure there are enough of them, and that posting is part of somebody's job.
  • Give somebody the Manage Versions privilege. Use a custom role, so someone other than the data owner can clean up abandoned versions.
  • Keep versions short-lived. The longer a version lives, the more Default drifts away from it, and the uglier the reconcile gets.
  • Delete what is done. A version list full of finished and forgotten versions is how you end up with QA_Final_v2_REAL again.
  • Use a naming convention. Something like owner.project_purpose means anybody looking at the version list can tell what each version is.
  • Keep an eye on history. History lives in the base table and grows over time. Look at Trim Archive History if retention becomes a concern.

Step 8: A View-Only Service Without the Schema Lock Headache

The Problem

Here is the scenario. The editors need an editable service. The public app, or the broad internal audience, needs a read-only service. So, two services get published from the same feature class.

It works. I have done it. Then, a few months later, somebody needs to add a field.

Every service referencing a feature class holds a shared schema lock on it. To change the schema, every service pointing at that feature class has to be stopped. With two services on one feature class, a simple field add turns into coordinating outages for both the editors and the public, at the same time.

The Better Way

Publish the view-only service from a database view instead of the feature class.

                 gisowner.parcels  (branch versioned)
                    |                       |
                    |                       | SQL view
                    v                       v
          Editing Feature Service     gisowner.parcels_public_vw
          (CRUD + Version Mgmt)              |
                                             v
                                   View-Only Feature Service
                                        (Query only)

The editing service owns the feature class. The view-only service references something else entirely.

Create the View

Connect as the data owner and run Create Database View.

There is one catch. Remember those GDB_ fields? A branch versioned table holds every row ever written: history, edits in named versions, and deletes. If the view is just SELECT * FROM parcels, you get duplicates and deleted features. The view has to filter down to the current state of Default:

SELECT p.OBJECTID, p.PARCEL_ID, p.ZONING, p.SHAPE
FROM gisowner.parcels p
WHERE p.GDB_BRANCH_ID = 0
  AND p.GDB_IS_DELETE = 0
  AND p.GDB_FROM_DATE = (
        SELECT MAX(p2.GDB_FROM_DATE)
        FROM gisowner.parcels p2
        WHERE p2.OBJECTID = p.OBJECTID
          AND p2.GDB_BRANCH_ID = 0
  )

This returns the most recent row for each feature in Default, and drops anything whose most recent row is a delete. As a bonus, you choose exactly which fields the public gets to see. Owner names and internal notes simply are not in the view.

Put an index on OBJECTID, GDB_BRANCH_ID and GDB_FROM_DATE, and test the view with realistic data volumes before going live.

Register the View with the Geodatabase

Run Register with Geodatabase on the view so ArcGIS knows the ObjectID field, geometry type and spatial reference. Once it is registered, ArcGIS Pro treats the view like a read-only feature class.

Publish the View-Only Service

Add the registered view to a new map. Share it as a web layer referencing registered data, with Query only. Leave Create, Update, Delete and Version Management off.

It Is Also a Better Use of the Server

Schema locks are not the only reason to split editing and viewing into separate services. How ArcGIS Server runs them matters too.

ArcGIS Server gives each service one of two instance types:

Dedicated Instances                    Shared Instance Pool

Editing Service                        View Service A ---+
  |                                    View Service B ---+
  +-- ArcSOC (min)                     View Service C ---+--> ArcSOC
  +-- ArcSOC                           View Service D ---+    ArcSOC
  +-- ArcSOC (max)                     View Service E ---+    ArcSOC
                                                              ArcSOC
  reserved for this one service          (pool sized to the machine,
  whether anybody is using it or not      used by whichever service
                                          needs it at the moment)

Dedicated instances belong to one service. Each one is an ArcSOC process running on the server whether anybody is using it or not, and it uses memory the whole time. You set a minimum and maximum per service. Multiply those across every service on the machine, and memory quickly becomes the limiting factor, well before the CPU is anywhere near busy.

Shared instances come from one pool used by all the services assigned to it. A request comes in and grabs an available instance. When it finishes, the instance goes back to the pool for the next request, which may be for a different service entirely. The pool is sized to the machine, usually somewhere around the number of cores, not to the number of services.

So, what does this have to do with branch versioning?

The editing service has to be dedicated. A service with Version Management keeps track of which version each editor is working in, and it does not work in the shared pool. That is fine. Editing is the workload that deserves reserved resources. Size the minimum and maximum instances to the number of editors you actually have. That is usually a handful of instances, not dozens.

The view-only service does not. It references a registered view, has Query only, and has no Version Management. That makes it a good candidate for the shared pool. Most read-only layers in an organization are used occasionally: a reference layer in a few web maps, a layer behind a dashboard somebody opens on Monday mornings. Reserving dedicated ArcSOCs for each of those means paying for memory that sits idle most of the day.

Compare what typically happens with two services published from the same branch versioned feature class:

Same feature class, two services        Feature class + view

Editing Service    (dedicated)          Editing Service  (dedicated)
  min 1 - max 4                           min 1 - max 4

"View" Service     (dedicated)          View Service     (shared pool)
  min 1 - max 4                           no reserved ArcSOCs
  + Version Management carried along
    because it's branch versioned data

In the first case, the view service is a second copy of the editing workload. It is the same branch versioned data, with the same instance type requirements, holding its own reserved processes and its own schema lock. In the second case, the view service costs essentially nothing until somebody uses it.

There is one exception worth calling out. If the view-only service is genuinely heavily used, such as a public-facing app taking a steady stream of requests, give it its own dedicated instances anyway. You are still better off. The editing service and the view service now scale independently: you size one for a dozen editors and the other for the public, and a spike in public traffic does not take instances away from the editors. That is only possible because they are separate services pointing at separate objects.

To summarize:

  • Editing service: dedicated, sized for editors.
  • View-only service: shared pool by default, dedicated only if traffic justifies it.
  • Both: configured independently, because they are independent.

Why This Works

  • Schema changes are simpler. The editing service is the only service directly on the feature class, so a schema change needs only the editing service stopped.
  • Posted edits show up immediately. The view reads Default, so there is nothing to replicate or sync.
  • The public sees only what you choose. The view defines exactly which fields are exposed.
  • Server resources go where the work is. The editing service gets dedicated instances sized for editors. The view-only service can use the shared pool instead of holding memory it rarely needs.
  • Each service scales on its own. Editors and viewers get separate services with their own security, sharing and scaling.

There Is One Catch

A view still depends on the table under it. Depending on the database platform, dropping or altering a column the view references can be blocked, or can break the view. In those cases, the view has to be dropped and recreated, which means briefly stopping the view-only service. Adding new fields that are not in the view usually does not matter. Also, the view only ever shows Default. Edits in named versions stay invisible until they are posted, which, for a public service, is usually exactly what you want.

My Default Pattern

For new multiuser editing in an enterprise geodatabase, this is now where I start:

DBA       --> Create Enterprise Geodatabase
DBA       --> Create Database User (gisowner)
gisowner  --> Load data, Global IDs, Editor Tracking
gisowner  --> Connection = Branch, Register As Versioned
Publisher --> Register data store (gisowner connection)
Publisher --> Editing service (CRUD + Version Management)  --> dedicated instances
gisowner  --> Database view of current Default, register with geodatabase
Publisher --> View-only service from the view (Query only)  --> shared pool

The principles behind it are pretty simple:

  • The geodatabase administrator administers. The data owner owns the data.
  • Branch versioning, because editing happens through services now.
  • Protect Default. Changes reach the authoritative data through a reviewed post.
  • Editors get named versions for anything bigger than a quick fix.
  • Clean up versions when the work is done.
  • One feature class, one service. Everybody else gets a view.
  • Reserve server resources for editing. Share them for viewing.

None of this is particularly complicated once it is all laid out in order. It is just spread across enough different places that it is really easy to miss a step. Hopefully, this saves you a little time figuring it out.

References