Node whatnow?

Node Sentinel File Watcher (NSFW). It’s a file watching module built for Node.js. I built NSFW to overcome an obstacle we were encountering while developing GitKraken, and I released it as an ongoing open-source project. It’s actually entirely safe for work. Ideal for it, even.

NSFW is a native Node module, which means it is developed and written to run natively on an operating system without an interpreter (JavaScript uses an interpreter, whereas a language like C++ compiles to machine code).

NSFW has a file watching implementation for Linux, MacOS, and Windows, which are wrapped and bound for the node runtime to utilize. Thanks to its native implementation, NSFW is able to watch recursive directories on all operating systems in a very performant way.

What problem does NSFW solve?

At this time, Node has pretty poor support for file watching. Every operating system has a different capacity to watch directories. Here’s what some celebs have been saying on Twitter:

Linux can’t perform recursive directory watching. MacOS doesn’t provide the filename when a file was changed.

Add to these the fact that there are numerous unsolved issues for Node’s file watcher and that Node does not have a consistent experience across operating systems.

How does NSFW solve the problem?

NSFW solves the poor file watching experience on Node by utilizing 3 low-level file watching utilities written in C++ and targeted for the Linux, MacOS, and Windows operating systems.

NSFW does most of its work on a separate thread, giving it big performance gains over the built-in Node file system (FS) watcher API. NSFW queues file events on a separate thread and batches file change dispatches into a single callback. That callback can be throttled internally to prevent spamming the JavaScript/C++ bridge.

This means NSFW doesn’t slow down JavaScript applications, even when they’re under the load of large FS operations.

Linux, MacOS and Windows each ship with their own file watching APIs. Since NSFW’s watch utility is targeted specifically at each of those APIs, it means the experience of using the module is consistent across all three OSs. NSFW fills in the gaps for each API so they’re all consistently feature-complete:

• Linux: The Inotify file watching system does not perform recursive directory watching, so NSFW builds and maintains a recursive watch tree for you.
• MacOS: FSEvents is known to produce inconsistent file events (the file event bitmask becomes corrupted if events occur too quickly), so NSFW stats and disambiguates file change events for you.
• Windows: supports all targeted needs out of the box. (RECORD SCRATCH) That’s right, Windows has the best native support. I said it.

Why this was an important problem for the Axosoft dev team to solve

GitKraken is currently the primary consumer of NSFW. A good Git client should not ship with a refresh button because it should automatically know when things change.

NSFW is essential to the smooth, cross-platform experience of GitKraken, as it helps the app respond quickly and accurately to changes in a repository it is viewing. NSFW is a quiet and humble, no-frills background process. It doesn’t make waves. It doesn’t talk through movies, chew loudly or snore, but its transparency is its strength; if you notice it, it’s more than likely because something isn’t working. So cheers to watching files silently and effectively – something made easy by the NSFW module.

Development hurdles

Oh yes.

With no experience in any of the file watching APIs, little experience in the Native Abstractions for Node (NAN) API, and no clear understanding of how to model the interactions Node makes with each file watching utility, the 0.x.x NSFW releases were, umm, troubled.

On Windows, I tried using C++/CLI to use the fancier file watching module provided by the .NET library. While this made it quick to get up and running, it inevitably backfired, because some users were unable to run GitKraken due to missing library dependencies. Sigh.

I also encountered strange errors related to having two distinct garbage collectors running concurrently (one for Node and one for the managed file watcher). When I scrapped the .NET implementation, I achieved much stronger stability for the Windows operating system.

On Linux and MacOS, issues revolved around unfamiliarity with the file watching API and native constructs. The overall unfamiliarity combined with the different strategies each operating system incorporated, ended up turning the entire project into a spaghetti turd. [REDACTED: DEEMED NSFW BY THE AXOSOFT DEPARTMENT OF PROPAGANDA]

Finally, large file change operations slowed down every operating system. At the time, those messages had no throttling behavior. The Node application would get absolutely hammered by file system change notifications and slow to a crawl. We suffered the consequences of that in GitKraken whenever we walked through our bootstrap process, which involves a lot of file manipulation.

After fiddling around with the project through the 0.x.x months (aka The Dark Times), I learned a lot about how each underlying file watcher API works, including their caveats, demands, and broken bits.

After spending a week putting together a complete system diagram, I scrapped the entire 0.x.x project and rebuilt the project to handle the differences of each operating system in a planned way. I also did away with the C++/CLI interface and opted for ReadDirectoryChangesW in Windows.

A couple of lessons learned

1. Predicting a system’s architecture without first dirtying your hands with the core features, means you’ll probably end up throwing away your project, dirtying your hands with the core features, and starting the project over. As a new developer, be prepared to throw away your prototypes.
2. I’m not sure if C++/CLI and Node should ever be a thing. Ever.