In this post, I want to share with you @puya/fh, a small cli utility we developed at our company Puyasoft in nodejs that we think is helpful in keeping a folder in sync with another folder by copying only changed files.

TL;DR

Jump right to Too Long, Didn't Read section.

Introduction

The main usage of @puya/fh is helping in syncing content of a folder based on its differences with another folder, no matter where they are located (same machine or different machines).

The tool does not perform the sync job itself when the two folders are located in separate machines. It provides features though that helps in creating a changeset out of the changes.

The next step (transferring and extracting the changeset onto target folder) is easy - can be performed manually or by any automation tool or script.

Story

Traditional/Legacy Application Update

Troubles of manual application update by bringing dev changes to production is a similar memory.

No matter how attentively we track the files we change during development, we find ourselves end up creating a zip, transfer it to production and extract it there.

We always asked ourselves at the end of the day, "Couldn't be there a tool using which we could copy/transfer only the changed files, not the whole local publish folder?"

@puya/fh is a tool that facilitates this job.

Modern way: Docker/Containerization

Docker is the modern automated way of bringing a feature to production. It has proved itself as a mandatory non-removable ingredient in ci/cd applications.

The benefits of docker are non-disputable. The only drawback is, even a tiny change results in a large impact, requiring the containers being restarted and waste of time/resource/traffic/space.

If there could be a tool by which we could copy only changed files into a running container and update the app inside that, there wouldn't be a need to restart the container.

Are we hearing blasphemy or is this a joke? Not only are containers immutable and no one is ever allowed to change them, an update may change dependencies or configuration.

Well. If we are careful about what we do in certain situations, we can cross red lines safely.

I believe, the end result is as rewarding as it worth it to give it a try.

The end result is gaining a huge optimization in fixing bugs or even releasing application updates: less time, space, traffic, latency, response time, and faster release.

If the new update does not require infrastructural change, like dependency change (installing new packages), we may be able to bypass docker and put our files directly in the container.

Target Audience

  • Any developer in any technology, php, java, .net, nodejs, python, etc.
  • DevOps engineers
  • Support teams

@puya/fh is not limited to tech environments or software production. It can be used by anyone who intends to make a folder in sync with another folder.

How does it work?

The way @puya/fh works is simple. It navigates a folder, checks its files and sub-folders recursively and generates a final .json file.

It writes names of the files/sub-folders into the generated .json, together with their hash in a nested parent/child hierarchy.

The hash of a file is generated based on its md5.

For sub-folders, the hash is generated based on the hash of its files/sub-folders.

It is this hash that will be later used to compare folders together.

Features

@puya/fh is able to ...

  • Generate a json for a folder
  • compare two folders (based on their paths or json)
  • produce a report based on the differences between two folders
  • create a batch file (to copy changes manually)
  • copy the changes directly
  • create a zip file for changes

Installation 

npm i @puya/fh -g

How to use

Generate .json for a folder

Example 1: current directory

> fh hash

Example 2: another directory

> fh hash -d /path/to/my/dir

Note that, it doesn't matter where current directory is. @puya/fh produces the same result for the generated json.

By default @puya/fh uses a list of excluded folders and files such as node_modules, .git, etc. to prevent copying not-necessary items or folder/files that should be ignored based on our discretion. These lists can be customized. This is explained a little further.

Compare folders

The main job is comparing two folders. This is done through diff and apply commands.

  • diff is used to report changes or generate a batch file to copy changes
  • apply is used to copy changes directly to another folder or create a zip archive for them.

The way comparison is performed is the same between these two commands.

Compare by paths 

> fh diff -f /path/to/source -t /path/to/target

Here, source folder is the folder containing new changes that we intend to copy to target folder.

Compare by path and json

As it was said, source and target folders are not necessarily required to be located physically on the same machine.

If target is located on another machine, we can generate its json on the target machine, bring the json to source machine and perform comparison.

> fh diff -f /path/to/source -t /path/to/target.json

Compare by json

We can perform comparison solely based on json files.

> fh diff -f /path/to/source.json -t /path/to/target.json

diff: report or generate batch

diff does not copy anything. It is used to report the changes or create a batch file for copying them. The behavior is specified through -k argument.

  • report: report changes to console (default)
  • cmd: generate a windows .bat file.
  • bash: generate a linux .sh file.

apply: copy changes or create a zip archive for them

There are 3 usecases that can happen for apply command.

  • Copy changes directly to target folder This is only applicable if the two folders are located on the same machine. 
> fh apply -f /path/to/source -t /path/to/target
  • Copy changes to a temp directory on source machine We can then refer to the temp directory and do whatever we want to that. 
> fh apply -f /path/to/source -t /path/to/target -rt /path/to/temp
  • Create a zip archive Using -c or --compress argument, we can create a zip archive instead of copying the changes. 
> fh apply -f /path/to/source -t /path/to/target -c

It is evident that, no matter what usecase we are using, the apply command should have access to source folder to read changed files to either copy them or create an archive.

source/target path accommodation

When using diff and apply commands, since source and target folders can reside in different locations, we may need to accommodate paths, so that the copy process or the created batch file works correctly.

This is done through -rf and -rt arguments for adapting source and target paths respectively.

> fh apply -f source -t target -rf /path/to/source -rt /path/to/target

Specifying Exclude/Include folder/files

By default, @puya/fh uses a list of excluded files and folders as below:

Excluded folders:

  • node_modules
  • .git
  • tests
  • __tests__
  • packages
  • wwwroot
  • coverage
  • .vscode
  • .idea
  • build
  • publish
  • .vs

Excluded files:

  • thumbs.db
  • package.json
  • packages.config
  • .env
  • .gitignore
  • .ds_store
  • *.log
  • *.test.js
  • *.spec.js
  • *.bak
  • *.tmp
  • sync.bat
  • sync.sh

These lists can be customized through the following arguments:

  • -ed or --exclude-dirs: specify excluded directories
  • -id or --include-dirs: specify included directories
  • -ef or --exclude-files: specify excluded files
  • -if or --include-files: specify included files

The list should be provided as a comma separated list.

> fh -ed ".git,node_modules"

If the value starts with comma, the list is added to the default list:

> fh -ed ",my-excluded-dir1,my-excluded-dir2"

Bringing changes to production

Traditional way

  1. Run @puya/fh on production where target folder is located.
  2. Bring target's json to the source machine.
  3. Compare target json with source folder and create a changeset archive.
  4. Transfer the changeset to production.
  5. Extract it there at the root of target folder.

Modern way: docker/container

step 1: run @puya/fh inside the container (assuming @puya/fh is already installed in the container)

docker exec my_container fh -d /app

step 2: copy app.json to the host

docker cp my_container:/app.json ./app.json

step 3: compare app.json with the new version of the app and create a changeset archive

fh apply -f ./app -t app.json -c -o changeset.zip

step 4: copy changeset into the container:

docker cp changeset.zip my_container:/app

step 5: extract changeset inside the container (assume unzip is installed already inside container)

docker exec my_container unzip /app/changeset.zip -d .

These commands can be put in a pipeline script and are activated conditionally based on our decision.

Now, when we have a small mild change that we know does not require say, dependency change, we can directly update our app inside our running container without restarting it.

Too Long, Didn't Read

@puya/fh is a cli tool by which we can copy new changes from a source folder to a target folder or create a zip archive out of the changes, so that we apply (transfer/extract) the changes to target manually.

The latter is useful when the two folders are not located on the same machine.

Usecase 1: copy changes from source to target 

fh apply -f source -t target

Usecase 2: create archive out of changes

step 1: create a json for target folder

fh -d /target

step 2: bring target.json to source machine.
step 3: create a zip changeset based on changes between source and target

fh apply -f /source -t target.json -c -o changeset.zip

step 4: transfer changes.zip to target machine.
step 5: extract changes.zip in target folder.

step 2, 4 and 5 can be done manually or using any tool/command.