Esri ASCII Grid Format

Session 1

The first session goal was to load an OS Terrain 50 data file into a terrain object.  The existing structure of the terrain object is a simple C# dictionary using a 2D point as a key and the height data as the value.  This was redesigned to allow for multiple tiles and to allow different types of loaders to populate the data, i.e. Perlin noise, GML, ASCII grid etc.

The new structure contains an overarching terrain object.  The terrain object can contain 0..* tiles.  The tile contains 1..1 tile content object. The tile content object is an abstract class that contains 0..* tileData objects and a virtual method, LoadContent. 

First design of the tile loader package

There are two items of note.  The first is that the tile is given the tile content object so that I can use lazy loading, i.e. the tile can exist without any data.  This way I can create all the tiles I require and position them before loading the data of the tile. The second point is the the tile object can be oblivious to where its data came from, i.e. file loaded, auto generated etc.  This is a first draft of the design and I expect to change as things develop.

ESRI ASCII Grid Format

I have decided to start with the simplest format for the first data file which is the ASCII grid format.  The details of the format can be found here but here are the main points:

The format has a header block containing the following items:

ncols - The number of columns in the data
nrows - The number of rows in the data
xllcorner - The X offset from the lower left corner
yllcorner - The Y offset from the lower left corner
cellsize - The cell size in metres
NODATA_value - Value that indicates no data recorded.

After the header block is the main data starting in the upper left corner.

To read the data in I am using the C# stream reader to read the file in line by line.  I have created a new object that inherits from TileContent called ASCIITileContent and implements the LoadContent method.  The load content method creates a background worker thread to read the file and load into the terrainData object.  For this I had to create a new event that tells the tile when its content is loaded which is raised by the background worker thread upon completion.  I added the new event to the TileContent class as I am planning to use the same technique for the rest of the classes.

I have also modified the Perlin algorithm to fit into the new design.  I have created a new class, PerlinTileContent, which inherits from TileContent and generates a random terrain.  This is then saved in the terrainData and an event is raised by the background worker.

For the tile loading I have created a new package in the project called tile loader that contains all the tile classes so that I can keep the data and UI loosely coupled.  

Overall, the first session went very well.  The data can now be read in successfully using the background worker threads and the class design seems to work well.  I think I may tweak it tomorrow to allow more information to be passed back and forth (mainly to report on progress whilst loading the file) but for now I am happy that I have met the objectives of first session.  Reading the ASCII files was very simple job but I may have to think about using a 3rd party zip library so that I don't have to keep unzipping the data files from the master zip but that may be a task for phase 2.  Tomorrow, I am hoping to link it up to the WinForms UI and look at how the UI can be improved.

Podcasts

I have had a daily commute of around an hour to work each day for a number of years now.  When I started this commute, a lot of this time felt like wasted time as I listened to the same songs on the radio or just daydreamed.  However, a few years ago I was given an IPod for my birthday and discovered pod casts.

Initially I listened to everything and downloaded any pod cast of interest to me and my interests of software, comedy, books and running.  The list below represents the pod casts I currently subscribe to or have found to be the most entertaining and thought provoking.

.NET Rocks - .NET Development and all subjects around the Microsoft tech stack.

The Infinite Monkey Cage - Science and comedy chat

Outriders - Web news show

Desert Island Discs - Always entertaining

Click - Technology news show by the BBC

TED Radio - Lectures on Technology, Engineering and Design

Anything by Richard Herring the Podfather of comedy

OS OpenData

I recently came across the Ordnance Surveys OpenData products and decided to look into how I could use some of the data in some of my existing projects.  In particular I have a personal project that I have been developing that uses an A* path finding algorithm with a heuristic that avoids steep inclines and declines, picking the most efficient route through a 3D terrain.

At the current time I have been using a Perlin noise algorithm to randomly generate terrain.  The weakness of this algorithm for generating the terrain is that, without some particular exceptions, it cannot produce certain terrain effects such as cliffs, inlets and sharp changes in terrain.  There are some techniques that can be employed to produce these effects after the Perlin noise has run but again they take some time to get a realistic looking terrain.  Using real terrain would be a great test for the algorithm.



Perlin noise generated terrain

A*algorithm avoiding the water and raised terrain

The OpenData products come in a number of formats and are continually updated to give information from postcodes and boundaries to digital maps.  The product that caught my eye was the OS Terrain 50 which gives height data at 1:50 scale across the whole of the UK. 

OS Terrain 50 is available as:

• 50 metre grid in ASCII grid and GML 3.2.1
• 10 metre contours in Esri® shapefile or GML 3.2.1

ASCII grid is a simple file format that specifies height data by storing tile information in a header block then a table of data in the body containing the payload. Geography Markup Language (GML) is an XML grammar to express geographical features. Esri shapefile is a geospatial vector data format for specifying shapes as points, lines and polygons that can be used to represent rivers, water wells and lakes.

My goal is to use one (or possibly all) of these file formats to produce some real terrain data to test my A* algorithm.  To do this I will use my existing test bed application written using WinForms and C# and modify it to read in the OS Terrain 50 data.  The existing test bed is a very simple application that produces a terrain object which is rendered in 2D and is colour coded to show different heights in different colours.  The terrain object is then used by the A* algorithm to calculate the most efficient route through the terrain.

The main tasks are:

• A GML/ASCII/Shapefile reader will be developed.  There are a number of readers available but I am planning to write my own so that I can learn more about the file format.
• The application will be redesigned so that a new layer is added that can feed the main application with either a terrain object that was produced by the Perlin noise algorithm or a terrain object built from the OS terrain 50.  In this way the A* algorithm will not be affected by the type of terrain data loaded.
• The main user interface will be changed to display multiple tiles;
• The application will be changed to use threads to load the tiles in the background.
• The colour code will be updated to produce a better quality display.

I am planning to tackle these tasks in hour long sessions over a period of a week and see where I am after the week has finished.  The main risk to completion are finding the time in a week for an hour of coding and half an hour of blogging. 

Creating the ASP.net Membership Tables In SQL Server

Because I always forget and I rarely have to create the asp.net membership tables (usually backup from exisitng database) I decided to put the command in a blog post. So the command:

aspnet_regsql -S -d -E -A mr 

-S and -d are obvious command line options. -E authenticates using the Windows credentials of the currently logged-on user. - A adds support for one or more ASP.NET application services. Service identifiers can be specified together or separately. The following identifiers are used for ASP.NET application services: m - Membership r - Role Manager

Microsoft Virtual Academy

I've been looking at the Microsoft Azure offerings lately and was really looking for a route in to start looking at what it could offer when I had a discussion with a colleague who mentioned the Microsoft Virtual Academy. The resources on their training academy are superb and I soon found a few courses and added them to my training plan.

I followed the Azure lesson and was blown away by the features that Azure had.  From my perspective, using the platform as a test bed would be fantastic but combined with MS Sync Framework and the ability to sync with on site instances of SQL server would be incredible.  I have started working on s tech demo to explore Azure further and to demonstrate to colleagues how we could use it in our day to day working.

Back to 1997

Every now and again in my job I am asked to slip back in time and look at some legacy code that is now not playing ball.  Today was one of those days as I was given the task to investigate some ancient VBA that sat behind MS project 97.

The most enjoyable aspect of my current role is the variety.  Yesterday I was looking at performance tweaks to a bunch of T-SQL stored procedures that were taking an age to run (more on that another day) and today it was some C#, VBA and excel formulas.

The problem that was occurring with the VBA was in the pay rates object of the project file.  Basically, the VBA was interrogating the object model and dumping selected elements out to excel.  The first problem I encountered was that the project file was linked to a project server that provided some custom elements to the project and enabled access to a lot of back-end data on costs, resources etc. Once I had the project file linked to the server it was a simple case of run the macro that performed the export and see where it was blowing up.

After a few minutes into the export the debugger stopped in a function that was iterating through a table of costs and was complaining about "invalid value for argument".  A quick look in the watch windows was showing that the pay rates table object contained a collection of pay rates.  The count of the collection showed 27 objects and under closer investigation I could see that the debugger had stopped whilst processing the 26th pay rate. Looking at the 26th object a bit closer it was obvious that it was corrupt and didn't contain any information , as did the 27th object in the list.  I flipped back to the project from the VBA editor to look at the table and saw that the table only contained 25 objects.  Strange.

A quick Google on the pay rates object found that the table in project can only hold 25 rows but there was no explanation of how I was getting 27 out of the count.  I switched the count and index loop for a for each loop to see if there was a bug in the count but again it attempted to iterate through 27 objects before blowing up on the 26th.  The quick fix was to put a limiter on the count if it exceeded 25 and stop it processing the last objects and indeed this sorted the problem but I couldn't really explain why it was happening.  Further trawls through the forums and blogs found nothing so I was forced to go back to the customer with the quick fix and three guesses on what might be happening.

My first guess was that there was a bug in the collection that was just not returning the correct count of objects in the collection. My second guess was that there might be some cleanup issues where the users had deleted some rows and then overwritten them and the collection had a bug which was not cleaning up the dead rows correctly.  This would explain why they were full of junk.  My third guess was that there was some other MS project setting that I was just failing to spot that was somehow hiding items in the collection or extending the collection to store extra detail.

I presented my conclusion and quick fix to the customer along with the recommendation that they first look at upgrading the version of project so that at least there will be more support. I also suggested that a more thorough review of the code at some point would be beneficial to ensure that there were no other strange things happening that were going unnoticed.

Overall it was a nice little side problem to get my teeth into and although it still nags me that I didn't find out why it was happening the customer was satisfied with the result.  Hopefully I'll be able to give an update soon on what the root cause was.

New Wheels

I picked up some new wheels a few weeks back and have delayed the review until I had at least given them a good run in.  For a few years I have always maintained that Asics were the best for me and my running gait as the fit was always really great and the quality of the shoe was always high.  So with all the hype about bare foot running I decided to replace my primary shoes with a new barefoot model to try for a few weeks and see how I got on.

I generally keep two pairs of shoes, a primary pair for work where I do most of my running and a secondary pair for home that are generally trail shoes for out in the forest.  After a bit of research and having decided not to blow the bank on what was really an experiment I decided on the New Balance 110 MT barefoot trail shoe.  The shoes selling points were the weight, price and the fact that it combined trail potential with barefoot running.

The shoes are really light compared to my last pair of Asics and very snugly fitted ( I had to size up).  I haven't weighed them but they are noticeably lighter and feel more like socks than shoes.  They retail at around £40 in the UK so they were a lot cheaper than rival models and are the only model that I had seen in the price band that looked like true trail shoes.  Although I didn't want to admit it I also was swayed by the aesthetics as they look pretty cool too.

I have now completed around 100 miles in the shoes and I've been really impressed.  My first venture into barefoot has seen a slight change in gait with more emphasis on the mid foot striking rather than on the heel. This, combined with the reduced weight, has seen some noticeable speed increases in my times.  I have gone from around 8 1/2 minutes  to around 8 minutes a mile for my usual run of around 6 miles.  I've had no blisters or any other irritations in my knees or ankles so far and overall I have been really impressed.  I've been out to 9 miles with them and to be honest I'm not sure I am ready to take them on the longer runs so I can only really give them a full recommendation based on my 6-8 mile distance runs.

The other selling point I failed to mention is that they are waterproof but with all the hot weather lately I haven't had the chance to test this which is pretty unbelievable living in Britain.