Parsing NAV Object Dependencies

No Source Control

Introduction

I recently had the privilege of being interviewed by Linda Rosencrance from MSDynamicsWorld.com about how my employer has adopted Dynamics NAV and my experiences with customization and development. The topic of source control came up, which I thought would make an interesting two-part series for my blog. My coworker has already started a great series on our custom source control solution, so rather than rehash his excellent overview, I want to focus specifically on one of the challenges we faced: parsing and sorting nav object dependencies. This series will cover the following:

Part 1 – Parsing NAV object dependencies (this post)
Part 2 – Sorting NAV object dependencies

Background

When our team first began working with NAV a few years ago (NAV 2009 R2), our first reaction was “What do you mean it has no source control?” (well ok, that was probably the second reaction, behind the occasional bouts of profanity due to the lack of undo functionality in the code editor…). We were very surprised to hear that most developers worked in a shared database and the closest thing to source control was using a locking mechanism, which prevented anyone else from working on the given object while one person had write access. There are times when this can be advantageous (for example, locking binary objects that cannot be merged), but being forced to use this model exclusively does not work well in practice for a system under constant change (at least in my experience). When you factor in that more than a couple of developers on our team are actively working on the system, this just makes a locking approach even less practical. Still, we gave it a shot for a while, but ultimately, this was a no-go for us.

We quickly realized that our workflow often required many developers to be working on the same objects simultaneously, which meant each developer needed to work in an isolated database. Working locally with an isolated database means you need to address merging changes from other developers. It needs to relatively quick and easy. Picard DVCS This lead us to create a custom tool to act as a bridge between the Dynamics NAV database housing the objects and our distributed version control system (Kiln, a tool built on top of Mercurial), which works well with text files. While it is true that NAV 2015 is offering a way to create Deltas using PowerShell and some handy cmdlets, this was not an option for us a few years ago (and we are still using NAV 2013 R2 at the moment). Also, with our current custom solution, we get the added benefit that the changesets (Mercurial’s equivalent to Deltas) are fully integrated with Mercurial, giving us a full history of all changes (with commit messages), branching abilities, relatively easy merging and hooks into our bug tracking software, FogBugz.

The main focus of this post is to talk about a challenge we faced, which was the need to parse NAV objects for dependencies. I would like to cover why we need to do this and how we did it.

Why Do We Need to Parse Object Dependencies?

The first question is “Why do we need to parse NAV object dependencies?”. To answer this question, we need a little background information on how our source control tool actually works. Lets use a hypothetical example:

Object Dependencies

Suppose a programmer, we’ll call him Skip List, makes a change to codeunit 50000, table 50005 and page 50010. We’ll also say that page 50010 references codeunit 50000, which in turn references table 50005. When Skip exports his changes, he doesn’t want to export each object individually, but rather wants to export all three functionally related objects at once (our tool exports each object as individual text files, but does it in a single export command). When Skip commits his change to the repository, it groups all three files, their Deltas, the commit message and some other metadata into a single changeset. Skip then pushes his changes to the remote repository for the other developers to pull down his changes whenever they are ready.

Meanwhile, Beatrice Tree (her colleagues call her B-Tree) has arrived for work a little later in the day (she had a dentist appointment and then got caught in a crazy traffic jam that involved a fight between a squirrel and a goat… long story). The first thing she does before beginning her work is to get any new changes from the remote repository to her local repository and import them into her database. She issues a pull command to mercurial, gets the changes Skip had pushed earlier, updates her local repository with his changes (merging if required) and then does an import into her local NAV database. Most of this is handled directly from our custom tool. Now since the three objects will be imported and compiled as individual files, it is important that they are handled in the proper order. If page 50010 is compiled before codeunit 50000 or codeunit 50000 is compiled before table 50005, we’ll get a compile error (depending on the nature of the changes of course).

To ensure that the objects are compiled in the proper order, the first step is to figure out which objects in the list of text files being imported depend on each other. This is why we need to parse the files and figure out the dependencies. You can imagine this gets much more complicated when you start pulling in multiple changesets at once with more than one developer changing the same object.

How Do We Parse Object Dependencies?

In my previous post, I talked about finite state machines and how my preferred implementation was the State Pattern when working with Object Oriented languages. Since we wrote our tool using C#, this allowed me to do just that.

There is quite a bit of code behind the entire parsing strategy and how it ties into our tool, so I’m going to attempt to do a shallow overview to give an idea of how everything works at a basic level. Hopefully this is enough to get you started if you are interested in doing something similar.

Parser Strategy Overview

First it makes sense to show the processes surrounding the parsing:

ImportParseSortCompile

Before parsing, we have finished step 1, which gives us a list of all files (and by extension, all objects) that have been imported into the database. It should be noted that the same objects may have been imported multiple times if they appeared in multiple changesets (either multiple commits by the same author and/or changes done by another developer). Step 3 will be the topic of my next post. For this post we are concerned with step 2.

The parser we built doesn’t go to the extreme level of building abstract syntax trees or interpreting and transforming the code etc. It is very basic in that it only attempts to pull out references to other objects and builds a dependency graph. As it is building the graph, it is only interested in objects that are part of the current list of objects that were imported into the database, which will need to be compiled.

Unfortunately, we were not able to find a formal grammar definition of the C/AL language, so we used a little trial and error in order to come up with a fairly generic (for all object types), but reliable structure for the text file layout for NAV objects. We broke the objects down into the following sections:

NavObjectStates

These sections each represent a state in our state machine. Each of these states will perform some similar functionality. The state machine follows this general flow:

ParseStateMachine

Read a line of text
Parse the text
Determine if the current line has important information for the given state (e.g. find out what the current object is, locate references to other objects etc.)
When a certain condition is met, set the state machine into another state (or you may reach the end of the file(s))

Parser State Classes

In order to create this state machine, we’ll need to create the following classes (it may be helpful to look at text file export of a NAV object to follow along):

ParserState abstract base class
- Contains some common functionality, properties and place holders for functionality that either must be implemented in the derived state classes or optionally overridden (in C#, this means abstract methods and virtual methods respectively).
ObjectHeaderState class
- Pulls out the current NAV object from the list of imported objects to become part of the dependency graph (we use a regex for this to grab the object type and the object id).
- Sample line: “OBJECT Codeunit 50000 My Codeunit of Wonders”
- Sets the state to ObjectPropertiesState class when the string “OBJECT-PROPERTIES” is read (case-insensitive string check).
ObjectPropertiesState class
- Nothing notable to pull out (at least in our implementation).
- Waits for the regular properties to begin (via the string “PROPERTIES”) to set the state to PropertiesState class.
PropertiesState class
- Looks for source table references (e.g. “TableNo=27;” or “SourceTable=Table27;”) and adds them to the dependency graph if they are included on the list of imported objects.
- Looks for the string “=VAR” to prepare for any variable declarations on the upcoming lines (inside of actions, triggers etc.).
- Parses variables, adding objects to the dependency graph if they are included on the list of imported objects.
- Looks for the string “CODE” when we are not looking for variables, to set the state to GlobalVariablesState class.
- NOTE: Just realized we are not handling “RunObject” in action definitions! The side benefit of writing a blog.
GlobalVariablesState class
- Looks for the string “=VAR” to prepare for any variable declarations on the upcoming lines (basically all of the global variables).
- Parses variables, adding objects to the dependency graph if they are included on the list of imported objects.
- Looks for the string “PROCEDURE” to set the state to ProceduresState class (note: this is the Functions section mentioned in the NavObjectStates diagram).
ProceduresState class
- Needs to deal with nested levels (to know when we are inside a procedure’s variable declarations, inside the procedure code or completely outside of a procedure).
- Looks for the string “VAR” to prepare for any variable declarations on the upcoming lines (inside of actions, triggers etc.).
- Parses variables, adding objects to the dependency graph if they are included on the list of imported objects.
- If we are at a nested level of 0 (outside of any procedures), and we hit the string “BEGIN”, set the state to CommentBlockState class.
CommentBlockState class
- We’re not currently doing anything with comments, but you could potentially use it for something.

Here is a sample of the ParserState class. I have elided some of the code that isn’t relevant to get the gist of it. We aren’t focusing on implementation details, or this will turn into a huge multi-part series (maybe if there is enough interest in this, I will consider a full-fledged tutorial).

	public abstract class ParserState
	{
	private readonly Parser _parser;

	protected ParserState(Parser parser)
	{
	_parser = parser;
	InitRegex();
	}

	public Parser Parser
	{
	get { return _parser; }
	}

	protected Regex VariableRegex { get; private set; }
	protected int ObjectTypeIndex { get; set; }

	protected int ObjectIdIndex
	{
	get;
	set;
	}

	// required by all state classes to handle specific
	// parsing strategy
	public abstract void ReadLine(string line);

	public virtual void Reset()
	{
	// related to nested levels of code,
	// tracking if we are ready for variable declarations etc.
	}

	protected void ParseVariable(string line)
	{
	// variable parsing logic here based on the regex below
	}

	private void InitRegex()
	{
	// Matching against variablename@variableid : type objectid;
	// e.g. BinContent@1000 : Record 7302;
	// VariableRegex is set here
	}
	}

view raw ParserState.cs hosted with ❤ by GitHub

Here is a sample of the class used to parse the object header section, which is at the top of every text file. It is represented by the ObjectHeaderState class.

	public class ObjectHeaderState : ParserState
	{
	private Regex _regex;
	private int _objectTypeIndex;
	private int _objectIdIndex;

	public ObjectHeaderState(Parser parser)
	: base(parser)
	{
	InitRegex();
	}

	private void InitRegex()
	{
	// regex used to pull out object type and id
	}

	public override void ReadLine(string line)
	{
	var match = _regex.Match(line);
	if (match.Success)
	{
	// add object to dependency graph if it
	// is on the list of imported objects
	}
	// If we hit OBJECT-PROPERTIES move to next
	// state (object properties), otherwise, just ignore
	// this line.
	else if (line.IndexOf("object-properties", StringComparison.OrdinalIgnoreCase) >= 0)
	{
	Parser.SetState(Parser.ObjectPropertiesState);
	}
	}
	}

view raw ObjectHeaderState.cs hosted with ❤ by GitHub

I will show one more state class, but then we must move on to the Parser class itself. Here is the GlobalVariablesState class.

	public class GlobalVariablesState : ParserState
	{
	private bool _readyForVariableDeclarations;

	public GlobalVariablesState(Parser parser)
	: base(parser)
	{
	}

	public override void ReadLine(string line)
	{
	if (line.StartsWith("procedure ", StringComparison.OrdinalIgnoreCase))
	{
	Parser.SetState(Parser.ProceduresState);
	}
	else if (line.IndexOf("=var", StringComparison.OrdinalIgnoreCase) > 0)
	{
	_readyForVariableDeclarations = true;
	}
	else if (_readyForVariableDeclarations)
	{
	// Uses ParseVariables from ParserState base class ParseVariable(line);
	}
	}

	public override void Reset()
	{
	_readyForVariableDeclarations = false;
	}
	}

view raw GlobalVariableState.cs hosted with ❤ by GitHub

Parser Class (Context Class)

With all of these state classes, we need to have a single driver that contains them and makes use of them (the Context class in the State Pattern). For our parsing code, this is the actual Parser class. The implementation of this class that I have chosen looks like this:

	public class Parser
	{
	// Will cover in next blog post likely
	private readonly ObjectDependencyManager _objectDependencyManager;

	// All available states
	private readonly ObjectHeaderState _objectHeaderState;
	private readonly ObjectPropertiesState _objectPropertiesState;
	private readonly PropertiesState _propertiesState;
	private readonly GlobalVariablesState _globalVariablesState;
	private readonly ProceduresState _proceduresState;
	private readonly CommentBlockState _commentBlockState;

	// tracks current state
	private ParserState _currentState;

	// Will cover in next blog post
	private NavObjectNode _currentObject;

	public Parser(ObjectDependencyManager objectDependencyManager)
	{
	_objectDependencyManager = objectDependencyManager;
	_objectHeaderState = new ObjectHeaderState(this);
	_objectPropertiesState = new ObjectPropertiesState(this);
	_propertiesState = new PropertiesState(this);
	_globalVariablesState = new GlobalVariablesState(this);
	_propertiesState = new PropertiesState(this);
	_proceduresState = new ProceduresState(this);
	_commentBlockState = new CommentBlockState(this);

	// start in the object header state
	_currentState = ObjectHeaderState;
	}

	// One property to expose each state type...
	// I will show first, but remaining are elided
	public ObjectHeaderState ObjectHeaderState
	{
	get { return _objectHeaderState; }
	}

	// Will cover in next blog post likely
	public ObjectDependencyManager ObjectDependencyManager
	{
	get { return _objectDependencyManager; }
	}

	// Will cover in next blog post
	public NavObjectNode CurrentObject
	{
	get { return _currentObject; }
	set
	{
	_currentObject = value;
	_objectDependencyManager.AddNavObjectToCompile(value);
	}
	}

	// the main driver...parse a file
	// and the state machine cogs begin to turn
	public void ParseFile(string file)
	{
	SetState(ObjectHeaderState);
	using (var rdr = new StreamReader(file))
	{
	while (!rdr.EndOfStream)
	{
	var readLine = rdr.ReadLine();
	if (readLine != null) _currentState.ReadLine(readLine.Trim());
	}
	}
	}

	// Called from within the states to change to another state
	public void SetState(ParserState newState)
	{
	_currentState = newState;
	_currentState.Reset();
	}
	}

view raw Parser.cs hosted with ❤ by GitHub

There is not really a lot going on in this class, even though it may initially appear so. It essentially does the following:

picard-riker-worf-hat Holds a reference to each available state.

Passes itself into each state’s constructor so they can call SetState and add objects to the dependency graph (via the CurrentObject property and the ObjectDependencyManager property, which will be covered in the next blog post when we talk about sorting the dependencies).

Reads each line using the current state’s implementation of ReadLine (polymorphism, yay!). As each state reads a line, it will perform the actions we talked about earlier in the post and change the Parser’s _currentState variable automatically as we move through the file. This makes the Parser class seem like it wears many hats, but the calling class is exposed to a simple API.

That wraps up part 1. In Part 2 – Sorting NAV object dependencies, we will cover sorting and also touch on how the Parser class is created and used.

Parsing NAV Object Dependencies – Part 1

Introduction

Background