When I started programming, my focus was on writing code. But I learned that when you need to perform a series of actions, it’s best to drive that using data, which is interpreted by code. The data is easier to understand, and to change. It’s easier to show to business users if that is appropriate. And it’s easier to have code that acts in a robust and resilient manner with that data.

Example One

When I worked in Shield Life, I wrote a program which allowed external users to do terminal emulation onto the Wang VS minicomputer over a dial-up connection. The program had to do a series of actions to connect to the VS:

• Interact with the modem and dial the appropriate phone number
• Interact with the X.25 PAD if appropriate (this used by users around the country to avoid having to make long-distance phone calls)
• Log in to the VS (we did not want to show the login screen to these users – instead we logged them in automatically)

Rather than coding all of this, I used DATA statements in BASIC. Here are some examples:

DATA “RETRIES”, 3, 3

DATA “SEND”, “AD~”

DATA “WAIT”, OK”, 6

DATA “DELAY”, 2

DATA “SEND”, “ADTP <number>~”

DATA “WAIT”, “CONNECT”, 60

DATA “IF”, “X.25”

etc.

This means:

• Set the number of retries for the subsequent commands. This means that if the response which is required in a WAIT command is not received, that the preceding SEND command will be repeated up to 3 times. The statement also acts as a marker within the script. The second number indicates that the processing should return here up to 3 times if the smaller retry loops don’t get the desired result
• Send AT followed by a carriage return to make sure that the modem is responding
• Wait up to 6 seconds for a response of OK
• Delay for 2 seconds
• Send the dial command followed by the appropriate phone number and a carriage return to the modem
• Wait up to 60 second for the modem to respond with a CONNECT message
• The “IF” statement starts a block of commands which will only be used if the computer is configured to connect via the X.25 packet switching network

Having all of this in data, rather than code, meant that it was easier to tweak the commands to respond to changing needs.  Examples might be:

• The data could be changed to work with different modems with different command sets.
• Delays could be added between each character which is sent to suit systems which could not respond to data which is sent at full speed

Notes:

• The data could have been held in a file rather than in DATA statements but by putting them inside the compiled program code we made it more difficult for people to see it and potentially tamper with it. A good solution, in fact, would be to check for the existence of such a file and then use it in preference to the DATA statements if one exists. This would allow you to troubleshoot and adapt when working on a client machine
• On-site troubleshooting would be facilitated by having a secret series of keystrokes which would bring up a window which would show the commands which are being processed along with the data which is being sent and received

Example Two

When I worked in Lotus Development, we had a Lotus Notes application which  we used to receive and manage queries from customer helpdesks around EMEA. We wanted to add some workflows to this application so that it could send notifications, reminders, escalate  cases, etc. This was an early version of Lotus Notes where this couldn’t be done in the application itself (this was before scheduled agents) and would have to be done using the C API.

I could have designed the workflow and got some C developers to code it. Instead, I designed a workflow engine where you could define the workflow using documents in a Lotus Notes database, and then that workflow could run against our tracking application. But, of course, it could also be used to create workflows for ANY Lotus Notes application (and we had a huge number of them!)

So it was a bit like Microsoft Flow (Power Automate) but about 25 years earlier!

The first document in a set would define the schedule so you could have a workflow which would run every hour, day, week, or whatever was required.

The second document would define the set of documents which would be processed. This would normally be a specific “view” in a specific Lotus Notes database.

The remaining documents in the set would define the actions. Each action could have a formula which would be run against the documents – the action would only be executed if the formula returned true (this would like an “if”). And then  the action itself could do things like:

• Sending an email
• Updating the document
• Adding a new (response) document
• Copying or moving the document
• Deleting the document

Basically I looked at the API documentation and I created an action for anything which as if it might be useful for creating a workflow.

There was also a flag on the action so you could tell the engine to skip on to the next document in the data set without processing any more actions for this document.

So I could have got the developers to hard-code a workflow for me, but instead I got them to develop a data-driven engine which could be used to create any workflow without the need to write something as complex and error-prone as C code.

When I started reading computer programs, they were written in BASIC for personal computers such as the Apple II and the Commodore PET. They appeared in magazines or books.

When you wrote a program in BASIC, your code had complete control over the CPU from the time you entered the “RUN” command. Nothing could happen that was not under your control (if you ignore bugs!)

If you had an INPUT command, then your program would pause until the user typed something and pressed the Enter key. Then your program would resume on the next line.

For greater control, you would use the INKEY$ function so that you could see and process each character as it was typed.

So your code was “procedural”: it would do something, and then it would do something else, all under your complete control.

When Visual Basic came along, things changed. Suddenly you had a series of control on a screen, and each one of them had code attached to events.

This idea of having to respond to events as they happen in an unpredictable order is good. It means that, as a developer, you need to think about your user doing things in a sequence other than the one you were anticipating.

In his book “Getting Things Done, David Allen talks about something he learned in the martial arts which is being in a state of alertness and readiness which he refers to a “mind like water” (which he has apparently trademarked, believe it or not!)

I think this is a good image for programming – your code should be alert and ready to respond to whatever happens next. Your code should “let go”, and be ready to kick into action again then the user (or something else) completes an action. You should not presume what will happen next, and neither should your code.

Example One

I remember way back when Basic (probably Turbo Basic) introduced ON KEY and ON COM statements so that you could get code to run (a subroutine specifically) when  a specific key was pressed or when something arrived on the serial port.  This was exciting because you could have code that ran on an event automatically without you having to check for it specifically. So the user could press F1 for help (for example) or F10 to quit at any point within the lifetime of the program (while those keys were enabled).

Example Two

When writing a program in Visual Basic (back in the day) you would sometimes write a long-running piece of code. You might be processing every row in a database or a file, for example. If you wanted to update the User Interface (e.g. showing a progress bar) then you would need to call the DoEvents function periodically to yield so that Windows gets a chance to  do this.

VB.NET has a Application.DoEvents method for a similar purpose.

DoEvents is a reminder that your code is not the only important code on the computer and your code should be collaborative. Blocking is bad, yielding is good.

Example Three

I was working with  a developer way back on a program which transmitted data between two mini-computers, a development machine and a production machine. Sending a file involves a series of steps: receiving the instruction to send the file; sending the header; sending the data blocks; sending the trailer; updating the log.

I explained to the developer that the program on the development machine might be sending a file and fetching a file at the same time. So the program would need to check each block as it was received to see what it was and what to do with it.

Because she was thinking in a procedural way, she wrote the code to do all of the steps one after the other.

I had to help her to refactor it so that it had a central tight loop, and that loop would have to respond to its circumstances each time. It might have received some data from its peer; it might have clearance to transmit a new block; it might have received a new command from a user; or it might be able to execute the next command from its queue.

This flexibility was needed. When the program finished receiving a file, for example,  it would move it to the designated location, potentially replacing an existing file. If that file was in use, then the move would fail, and would have to be retried. If the code was determined to perform that operation right away, and was unable to do anything else while waiting to do so, then the exchange of files would be halted for minutes or even hours!

So a central loop with an awareness of what was happening with various requests, and what was waiting in the queue, was essential to the correct operation of that system.

Example Four

When you write code for Arduino (a “sketch”), you get to write a “setup” function and a “loop” function. The former is called once, and the latter is called repeatedly.

Writing code in Processing or p5.js works in the same way, except that the function which is called repeatedly is called “draw”.

When you are writing a loop or draw function, you should do what you need to do right now and then finish, confident in the knowledge that you will be called again in a moment. And by yielding in this manner, you give the CPU a change to do other housekeeping tasks.

When the loop restarts, your code must be able to resume and do the next thing.

When I wanted to add graphics to my COVID program for the MXCHIP,  I downloaded a library which included a sample program which created an animation using graphics commands.

I was shocked to see that the “loop” function included a “while (1)” statement. You can see the program here.

I was surprised to see that the programmer did not respect the need to yield at the end of each loop. I refactored it to do this and it worked just as well.

Example Five

When writing my COVID program, I had to decide what I would do in the setup function, and what I would do in the loop.

At one stage I was doing the HTTP GET functions to get the up-to-date COVID statistics from the REST API. But I moved these to the loop and I used a variable called “init” to track where I was in the step for initialisation. When init is 1, I get the data for Ireland. When it is 2 I get the data for the USA. I do one thing each time loop is called and I increment the init variable. When I have completed the initialisation, I set init to 0 and I can respond to button presses during each subsequent pass through loop.

The great thing about this is that if the user presses the two buttons simultaneously, I just set init back to 1 and the fetching of data starts all over again. So not only is being flexible good, but it had an immediate benefits when I wanted to do something new.

Conclusion

It is important to get away from procedural thinking, to write code which is fluid and flexible, to avoid thinking that you have completed control over the CPU, the network, etc., and to avoid making assumptions about what is going to happen next.

As Bruce Lee said:

Empty your mind. Be formless, shapeless, like water.

Test-driven development (TDD) is very popular at the moment. The basic idea is to code a series of test all of which fail, and then address the failures.

My approach is also incremental, but works in the opposite way. I like to get something working, and then add something else and get that working, and keep adding layers until the code does what is required.

It’s basically a cycle of write, test, fix, write, test, etc.

But the idea is that you are adding layers of functionality, and building on success.

Some of the additions are so simple that they SHOULD work fine, but I just do them and verify that they do.

The opposite approach would be to write a large amount of code and then test and debug it. But if we did that, the bug could be ANYWHERE!

When I started programming we were writing batch programs, and we had limited time on terminals. So we would write the code at our desks, and then get access to a terminal to enter it. Then we would run it (or get an Operator to run it for us) and we would take the output (report) back to our desks to debug.

That approach make no sense these days now that we have IDEs, debuggers, etc.

I should point out that I don’t think anyone else refers to this as “layering”, but for me the word conveys the idea of building layers of functionality which are demonstrably working.

Example

The COVID program holds 14 statistics for 3 counties plus the same for the world. It displays these statistics on six screens which the user can scroll though.  But as I built it up, at one stage it was able to hold and display three statistics for one country.  And before that it was able to display the raw results from a HTTP request. And before that, it was just able to display that it had managed to make a Wi-Fi connection to the router.

Conclusion

It is either arrogant or foolhardy to think that you can write substantial amounts of bug-free code. The best approach is to write small incremental pieces of functionality, proving that it is working correctly each step along the way

Year ago, I read a book on time management (or procrastination) called “Eat That Frog!”.

In the book the author, Brian Tracy, proposes that you should always tackle the most challenging task on your To Do list first.

Now I have not applied that principle in my life, but I have certainly applied it in my programming.

The first thing you should tackle when writing a program is always the part about which you are most unsure, or the part which the most risky.

It’s amazing how often I have asked a programmer “how did you get on with X” and they reply that they haven’t got to it yet because they decided to start with Y. And Y is often something which involves no risk at all (like settings, configuration, admin screens, or even a splash screen!)

Example

When I was working on the COVID program recently, I started with the things with which I was least familiar: the HTTP Client class first and then the JSON library. Once I got these working, then I knew that everything else would fall into place quite easily.

Conclusion

When you start a new program, always start with the piece which scares you most. Don’t waste days doing the easy stuff before discovering that there really is a problem with that scary piece.

You often hear stories of programmers who work best in the evening or at night, or programmers who “pulled an all-nighter”.

I take a different approach.

When I am struggling with a piece of code (a bug I cannot track down, for example) I call it a day and sleep on it. I am still amazing at how often I find that I will awaken with a solution.

They say that the following happened in dreams:

• Dmitry Mendeleyev came up with the periodic table of elements
• Dr. James Watson came up with the double helix structure for DNA .

Now I have never done anything as earth-shattering, but here are two recent examples from me.

Example One

I wrote a COVID-19 app recently and I was struggling with the HTTP Client class which I was using to interact with the API to get the data.  The documentation for the class was pretty light and I could not find any sample code. The problem I was having was specifically in relation to the callback function.

I was having so many problems that I decided to try the Websocket Client class instead. I was struggling with this too.

But when I woke up the next day I was sure that I could get the HTTP Client to work without using a callback and that is exactly what I did and it worked for me.

Example Two

My daughter was doing a project for her university course which involved designing the hardware and software for a networked automatic hand sanitiser dispenser. I wrote some pseudocode for the software which monitored the proximity sensors and the liquid levels.

The code had a check for thresholds on the liquid levels and could send alerts as these thresholds were passed.

I realised that there might be an issue where (due to slight fluctuations in the readings) the liquid level might seem to cross back and forth over a threshold and this would trigger multiple alerts.

Since this was just pseudocode for a project which would never by built, I decided not to worry about it.

But when I woke up the next day I realised that a simple tweak would fix this.

So now when I get a new reading, I check it see if it is greater then the previous reading. If so, I check to see if the increase is about a configurable threshold (min_increase). If it is not greater then min_increase, then I just set the new reading to be the same as the previous reading. Job done!

Conclusion

If you are struggling with a design or coding problem then try sleeping on the problem.

The first two principles for good programming were Simplicity and Elegance.

But there are not independent. The aim must be to write code which is as simple and elegant as possible, while getting the job done.

Shibumi is a concept I came across in the book of the same name, written by Rodney Whitaker in 1979 under the pen name of Trevanian. I mentioned this book (which is my all-time favourite) in my old blog.

It is a Japanese word with a number of meanings, including “elegant simplicity”. Here is a passage from the book:

Shibumi has to do with great refinement underlying commonplace appearances. It is a statement so correct that it does not have to be bold, so poignant it does not have to be pretty, so true it does not have to be real. Shibumi is understanding, rather than knowledge. Eloquent silence. In demeanor, it is modesty without pudency. In art, where the spirit of shibumi takes the form of sabi, it is elegant simplicity, articulate brevity

When Nicholai Hel (the hero of the book) heard these words, he decided to dedicate his life to achieving Shibumi.

I’m afraid I have not dedicated my life to Shibumi, but I do look for Shibumi when I am programming.

In programming, Shibumi is thinking about a problem until a simple elegant solution is uncovered, and then coding accordingly. Whenever your design or code seems complex or convoluted, you need to do some more thinking.

Programming is an art, not a science. Some people might baulk at the idea that it’s an art, but it is certainly a craft.

Bad programmers write bad code, great programmer write elegant code.

I have read a lot of code in my time. And some of it was so badly written that it was scary.

Conversely, it is a pleasure to read code that is so well-written that it is elegant.

Writing a program involves making dozens (or hundreds) of decisions. It involves solving lots and lots of puzzles.  Solving puzzles is one of the great joys of programming. There always multiple ways to tackle each puzzle. And each decision is an opportunity to move towards elegance, or towards chaos.

I don’t think everyone can be a good programmer. I have worked with people who wrote chaotic code. This code was a nightmare to read, and a nightmare to maintain/enhance.

Some people have the gift of finding the elegance, and some do not. That’s what separates bad programmers, from good programmers, from great programmers.

In my opinion, a great programmer would not be happy just to get a program working. They would not be happy until the program is elegant.

I think it’s acceptable to write a small piece of inelegant code while you’re trying to get something working, or to understand it fully. But once you have that clear picture, you should always rewrite/refactor for elegance.

A simple example of the decisions which a programmer needs to make is when functionality should be delivered inline, when it should be moved out to a separate function/method, and when a new Class is needed.

Elegant code should be easy to read and understand. Badly written code can be a nightmare to understand!

First Example

I have a book called “Java Examples in a Nutshell” (written by David Flanagan) which I cannot throw out (even though it is dated – my edition was written in 1997) because the code is so elegant. Each of the programs is as simple as possible, while illustrating the technique in question. Almost all of them have a “main” method and most of them process simple argument so that they can be called from the command line.  One, for example, is called “SimpleProxyServer” and it illustrates a simple web proxy which has one thread for passing on outgoing content, and another for passing on incoming traffic.

Second Example

This is a link to some Java code which implements the Asteroids game as a Java Applet (it was written a long time ago!)

It is a very faithful recreation of the original game and I think the code (although there is quite a lot of it) is very nicely written and very easy to read.

The main look contains the following:

updateShip();
updatePhotons();
updateUfo();
updateMissle();
updateAsteroids();
updateExplosions();

Makes sense right?!

Aside from the main Class, there is only one other one: “AsteroidsSprite”. This is used to manage all of the “vector graphics”-style objects.

The Class has a method called “isColliding” which takes one argument which is another AsteroidsSprite. So you can easily ask any AsteroidsSprite if it is colliding with any other AsteroidsSprite. Simple as that.

And there is a method called “explode” which takes an AsteroidsSprite as an argument. It creates a set of AsteroidsSprites, one for each line in the original object. And then those  AsteroidsSprite are added to an array called “explosions” and those lines move and fade as they would have in the original game.

In the hands of a less capable programmer than Mike Hall this could have been a real mess. But it’s not. It is elegant.

Conclusion

The difference between a mediocre programmer and a great programmer is that the latter can make a series of decision which allow him/her to write elegant code.

KISS (or “Keep it Simple Stupid”) is a principle which was created in the 1960s apparently.

Certainly it was something we talked about when I started programming.

It is as true now as it was then.

Apparently Einstein said that ““Everything should be made as simple as possible, but no simpler.” He was a smart guy!

The trick in programming is to resist using something because it’s clever, or because it’s new, or because it’s trendy.

When something is simple, it easier to get right, it’s easier for others to understand, and it’s easier to maintain and enhance.

I have two stories to illustrate this: one new, and one very old.

First Story

The old one was when I was working in Shield Life and going to Trinity College in the evenings.

On my course I learned how you could create files where each record had a pointer to the next record (and optionally a pointer back to the previous record). So when you found the first record in a set, you would easily navigate to the others. And you could have another chain of free records which would could use when you needed to create a new record.

So we had a requirement for an “Outstanding Premium File” (OPF) which would store all of the premiums which made up an individual Direct Debit for the bank. Then if the Direct Debit was declined, you could work out which policies had not been paid so that you could take appropriate actions. I decided that this was a perfect use case for a file with record pointers.

I developed the program and it worked perfectly.

The problem was that, after I left the company, the team decided to re-develop the program using a more traditional approach which everyone could understand and maintain.

So I had ignored the KISS principle and I had developed something that other programmers did not like. I had been seduced by something that seemed to be very clever.

Story Two

The second story is recent. We were developing a solution using SharePoint Online, SPFx, Azure, etc. and the architect and developers decided to use a Term Set in SharePoint to hold data which could have been held in a lookup list.

I could not understand the purpose of using a Term Set for this. Term Sets have an important role to play in some circumstances. But to me it looked like complete overkill for a lookup field.

And it turned out that it introduced a number of unnecessary complexities:

• We needed to have separate term sets for test and production. Using lists, it would have been as simple as having two sites and independent lists in each.
• We had to add values to lists using PowerShell and these scripts were made more complex and more troublesome because they had to populate the term sets
• The client wanted to export the data to Excel. We used the native export at first. But the term set columns were weird when exported to Excel. In the end, we did a custom export.

Conclusion

Einstein was right: it’s best to keep everything as simple as possible, but no simpler

I have been developing software for a long time. And over those years, I have come up with an approach to programming and some basic principles.

These are now part of my “gut feeling” when I’m developing something new. But in the coming posts I am going to try to isolate, examine and document them.

I’m calling this series “Peter’s programming principles”. There are a couple of things to say about this:

• I could not resist the alliteration!
• I’m not sure when exactly “programming” became “software development”, when “programmers” became “software engineers”, and when “programs” became applications/apps. But I think the old names were just as good.

The principles are all high-level. They are about an approach. If I talk about a specific technology then it will only be to explain the principle.

The principles I am going to describe are:

• Simplicity
• Elegance
• Shibumi
• Sleep on it
• Difficult things first
• Layering
• Event-Driven
• Data-Driven