Golocron – Software Development With Go

Common

The ultimate goal of choosing an appropriate structure is to keep things nice and tidy which simplifies interactions with a codebase.

This includes:

• providing required files for a project
• keeping the number of elements at the root level at a reasonable minimum
• keeping the root (and any other) level clean and up to date
• grouping non-code files by their purpose
• refraining from creating unnecessary hierarchy of folders
• having a single entry point for routine tasks
• storing credentials somewhere else

Let's talk about these in more detail.

Provide Files Required for a Project

A project should contain files that describe its purpose, provide required information for the programming language and tools, and simplify its maintenance.

The following files are considered as required:

The files listed above are a bare minimum, but required for a modern project. You do this once, but if it's done right, it helps you all the way further.

One thing to note is that even with no code at all, a repository is likely to contain about 6 files. This is why the next suggestion is important.

Keep the Number of Root Elements at Minimum

The number of folders and files at the root level should be as minimal as it's possible and makes sense.

It's important because it allows you to stay focused on the task you intend to perform, and not be distracted by the need of wading through the files, scripts and configs.

To continue our clothes analogy, it's like with a wardrobe - in the morning it's better to take a nicely folded thing from a shelf. But this convenience costs you the responsibility. To get it, you have to fold clothes neatly after the use, and it's also important to carefully choose what to put into the wardrobe.

While this suggestion sounds vague, and it is, it can help you. Each time when you're about to add something at the root level, by default question the need. Usually, things are set at the root level at the beginning, and then added here rarely. When you justify the presence of something at the top level, think twice about a better place for a file/folder, or if it's needed at all.

Keep Any Level Clean and Up to Date

Similarly to the root level, any level of the file tree in a project should be clean, with reasonably minimal number of files, free from garbage, and up to date.

This is important because:

• long file listings make navigation harder
• poor organisation complicates maintenance of the mental model of a project in the developer's head
• out-of-date files take up extra mental and physical space
• accidentally committed files are misleading and confusing.

When a new developer joins a project, how much time do they need to get a firm understanding of what's going on in a project? Does the structure help and guide your colleagues through the project? Does it reduce the effort to keep the important details in their heads? Or do they have to stop for a moment each time when stumbled upon a file which no one knows about, whether it's current or not?

A well-organised and maintained project itself helps people in their everyday jobs. Each time when adding a new file, think twice what else can be added in it further. When creating a directory, consider different scenarios a few months in advance. You don't want to end up with a mess one year after, and tidying up a larger and live project is much more of an effort than keeping it clean everyday. If the structure is supported on a daily basis, it will pay off in long-term.

While it's important to have a neat structure, sometimes it's easy to get too serious about a hierarchy. This is what the next suggestion is about.

Don't Create Artificial Hierarchy

Keep the directory tree in a project as flat as it possible and makes sense. In other words, don't create artificial hierarchy just for the sake of hiding something.

This applies to both, files containing actual code (and we will discuss it in Unit 2), and other files like scripts, configurations, documentation, etc.

A general piece of advice here is that any directory that contains only a single or multiple directories, or a single file, should be questioned for its purpose and existence. Does it belong in here, or somewhere else? Sometimes it might not be clear at the beginning, and this is why it's important to think about evolution of a project some time further.

Consider a situation with scripts. One might have a root-level directory called scripts, and the directory contains several files providing tools for CI/CD process, maybe tests, some pre- and post-install steps. That's a good example.

On the opposite, consider the same number of files but located under separate directories like scripts/build/build.sh, scripts/install/install.sh, and then scripts/ci.sh that sources those files. Don't do this.

Here's another example. Say we have an http directory. From here, we've got two options. If we know for sure that there will be no other networking stuff in the project any time soon (means a couple of years), then it makes sense to keep it simply as http. On the other hand, if there is a non-zero chance of implementing things for smtp and dns, then it does make sense to put it under net. Later, everyone will be happier with the structure because it reflects the nature of things and supports the purpose of the files.

At this point, it becomes clear that grouping for files and folders should be done based on its purpose. Even for non-code files.

Group Non-Code Files by Their Purpose

Group and keep maintenance and routine files together by their purpose, in one or a few directories.

This helps in keeping the root level free from clatter, as well as reduces the time required to find a file when looking at the listing. You simply know where all your scripts reside, so no time and energy is wasted. When a new developer joins the project, they need minimal time to get an idea of what is where. Everyone is a bit happier.

As it was shown previously, group all scripts in the scripts directory. If there is plenty of them, and they're specific to the project itself (like startup scripts, init system scripts, cron files, etc), then it might make sense to keep the CI/CD stuff separately. Stay lean though, having ci/scripts and scripts is not a good sign. What else do you have for CI but scripts? If that's a couple of yaml files, let them stay under the same ci directory (see the previous point).

Nowadays many projects use Docker. If a project uses only one Dockerfile, and maybe one file for Docker Compose, then it's fine to keep them at the root. If you have multiple Docker files, and also have some scripts for building containers, then keep them grouped in a directory.

At this point, a valid concern may arise: if all those tools are in directories, it's less convenient to type commands in the terminal. This is what the next recommendation helps you with.

Provide the Makefile

Provide the Makefile for a project to automate and simplify routine and maintenance tasks.

The make tool is available on almost any platform one can imagine, even on Windows without WSL. A minimal knowledge of the syntax of a Makefile is enough to produce a handy set of targets that simplify everyday workflows. The tool have existed for more than 44 years now, and it's likely to continue existing for another 40 years. And in the course of its life, the syntax hasn't changed very much. So it makes a lot of sense investing in learning it once, and benefit from it for the rest of your career.

Just to give you an idea how it can be helpful, consider several, rather basic, examples below.

1. Building a binary.

Without make and Makefile:

GOOS=linux GOARCH=amd64 go build -ldflags "-X main.revision=`git rev-parse --short HEAD`" -o bin/my-app ./cmd/my-app

With make:

make my-app

2. Building a container and running an app in it.

Without make:

docker-compose build --no-cache my-app
docker-compose up -d my-app

With make:

make run-my-app

And so forth. While being basic, these examples make a huge difference in everyday experience.

To make your and colleagues' lives easier, define all routine tasks as targets in the Makefile. It can be then used not only by developers directly, but also in your CI/CD scenarios. This makes the configurations clean, straightforward, and easy to understand and maintain.

It's important to give targets in the Makefile meaningful names. A meaningful name reflects the purpose and meaning of the operation it describes. When this is done well, instead of cryptic bash incantations your config for CI may look something like the one below. Pretty nice, isn't it?

# skipped

script:
  - make test
  - make cover
  - make build
  - make deploy

# skipped

We're almost done with common suggestions. Last in the list, but not the least in importance, is a piece of advice about storing credentials.

Store Credentials Somewhere Else

Under no circumstance should you store any sensitive data in a repository, even if it's a private one. Just don't do that.

Don't add ssh keys, files with passwords, private ssl keys, API or any other credentials. It's alright if a repository contains those for development purpose. Anything else is a taboo. Good sleep at night is worth more than deceptive convenience of having production credentials committed to a repository.

What can be used instead? A private bucket in a block storage that contains an encrypted file with credentials might do a good job. A specialised service from your cloud provider of choice might help too. A special service running in your infrastructure that is responsible for storing and providing other services with sensitive data may be even a better option. But once again, do not store any credentials in a repository along with code.

All these suggestions are supposed to help and support a team working with a project throughout its lifecycle. Considered and implemented carefully, they form a strong foundation for the project's success.