ImpVis technical / maintenance documentation

Written by Rob King, October 2020.

This document hopes to summarise the procedures for various technical operations that might need to be done in order to maintain the Vue-Components library and the ImpVis Command Line Interface Tool (CLI).

Prerequisites

The vast majority of the ImpVis code base is written in JavaScript as this is the only language that is natively understood by every browser in existence. Furthermore, the new Vue-Components library and by extension most new visualisations developed since Summer 2020 will utilise the popular Vue JS framework. This is a powerful yet lightweight JavaScript library that makes it easier to create interactions between the various elements in a visualisation.

From this point onwards this document will assume that you have a good understanding of both the core concepts in JavaScript and Vue JS. If you need to brush up on your knowledge it is recommended to check out the ImpVis YouTube channel where there are a number of introductory videos available.

In addition, this document assumes that you are comfortable with HTML and CSS code however these will not be discussed as much in this document.

Browser & OS Support

In the ImpVis project we aim to ensure that all the visualisations function correctly in all modern browsers both mobile and desktop. In practice this means that means new features added to the library should be tested in various browsers in particular:

Chrome
Firefox
Safari
MS Edge

The main exception to this support is Internet Explorer which we currently do not support nor do we have any intention of supporting in the future due to its lack of support for modern JS features (see section on ES6 below) and modern CSS. If you encounter anyone who is unable to view visualisations as a result of their usage of Internet Explorer it is strongly recommended that you encourage them with all earnest to upgrade to a more modern browser. Writing visualisations that would be natively compatible with Internet Explorer is simply too much of a time investment and too much of a sacrifice of modern features to cater to a rapidly shrinking userbase (~ 1% as of 2020).

For offline programs such as the CLI, we aim to support all major operating systems (Windows, MacOS and Linux). As these tools are written in JavaScript and executed using Node JS, this should hopefully come for free however certain 3rd party libraries may function differently on different platforms.

ES6 JavaScript

If you have previous experience with JavaScript or if you happen to look at any older documentation for various JavaScript libraries you most likely will see variables declared using the var keyword whilst if you look over code written in the ImpVis repositories you will see variables declared using the let/const keywords. This is a result of the ImpVis project utilising features present in the ES6 JavaScript standard (sometimes called ECMAScript 2015) which makes writing robust and simple code much easier than in pre ES6 JavaScript.

Most of the features available in ES6 are well documented on the MDN website here: https://developer.mozilla.org/en-US/docs/Web/JavaScript/A_re-introduction_to_JavaScript

Here is a brief summary of some of the features available:

let & const keywords available for declaring variables. Unlike the old var keyword which defines variables in the function scope, these keywords are block scoped. What this means in practice is that variables declared within if statements or for loops cannot be accessed from outside the corresponding control block. This is in contrast to the var keyword which is function scoped meaning that these variables may be accessed anywhere from inside a function even outside of the for/if block they were declared inside of. As this result was often undesirable and could lead to variables leaking to the global scope patterns such as the Immediately invoking function expression (IIFE) are often seen in older non ES6 JS code. Modern ES6 block scoping features give the programmer much more control and mean patterns such as this are not as important. Many other programming languages like Python also use block scoped variables making the transition to this style easier.
Arrow functions () => {} Importantly these are NOT a replacement for the old function() {} syntax from pre-ES6. Instead arrow functions make it much easier to write inline functions to pass as a parameter to functions e.g Array.prototype.map(). Arrow functions differ from function () {} in that the this keyword is said to be lexically-scoped. This means that the value of this inside an arrow function is set equal to the value of this in the parent context, whereas using function sets this from the context from which the function is executed. What this means in practice is that say we have the code: > let some_arr = [3,5,6,8] > this.multiplication_factor = 2 > some_arr.map( (x) => x * this.multiplication_factor ) In the above case, this is set equal to this that exists outside of the function that is run (which if in a browser typically refers to the window object). In contrast if we wrote: > let some_arr = [3,5,6,8] > this.multiplication_factor = 2 > some_arr.map( function (x) => {return x * this.multiplication_factor;}) Then now the this keyword would be set equal to an Array object, indeed it would refer to the some_array variable we defined at the start. This is because map is a method on the some_array object. Sometimes this behaviour (such as writing methods for a custom object) is desired, thus you should not use arrow functions blindly instead reserving them for situations where they are best utilised. An example of a good situation is when writing callback functions or simple mapping functions. The bind method can be used on old style function objects to manually set the value of the this keyword, so the lexical scoping nature of arrow functions can be recreated even in old style classes.
Format Strings Similar to Python 3.8, ES6 JS allows for proper inline format strings. What this means is that using the ` symbol on your keyboard you can write inline JS directly inside of a string, for example: `Hello,${ //JS CODE HERE } `. This is a nifty trick that prevents you from having to write too much string concatenation.

NodeJS & NPM

For many years, the only way one could execute JS code was directly in the console of a web browser. NodeJS was created to allow developers to execute JS code without the need for a web browser to be running and thus letting you use JS to write offline programs similar to how you would with Python. Indeed NodeJS also provides several additional Node specific functions that allow you to do operations such as file IO that core JS does not include.

NodeJS also comes with the powerful Node Package Manager or NPM which intelligently handles all 3rd party JS packages and their dependencies for you in your project. NPM is able to download 3rd party packages from its online repository located at https://npmjs.com and this is where both the CLI and the Vue-Components packages for ImpVis are located.

Any directory on your system is able to act as a NPM project so long as it contains a package.json file. This is a JSON file which stores a variety of information about your project, most crucially the details of the 3rd party packages that your project depends on which will be stored under the dependencies & dev dependencies fields. Packages listed under either of these fields will both be installed automatically when npm install is run in the directory for the first time. For a discussion of the differences between dev and non-dev dependencies, see: here.

One important thing when using NPM in conjunction with Git is to ensure that the node_modules folder that is created automatically on a package install is not committed. This is because this folder contains OS specific binaries and files that may not work properly on someone else's machine. Furthermore this folder is often rather large (>100MB) and thus committing and pushing this to the master repository will have the unintended side effect of slowing down Git for everyone who is using the repository (plus GitHub gets mad if you have too many large files). The easiest way to ensure that node_modules is not committed is to include a .gitignore file in the root of your project (next to package.json).

In addition to the automatic node_modules folder being created when you run npm install, you will also notice another file package-lock.json being created. This file is typically rather long and stores the full details of ANY package that is installed in your project even including ones that you have not explicitly listed in your dependencies field in package.json. As such this file often includes the details of the dependencies of your dependencies as well as THEIR dependencies and so on. The point of this file is that it is intelligently able to remember what packages are installed as well as what version was installed such that no more packages are installed than what needs to be and that every package that is installed should just work. Unlike the node_modules folder, you are encouraged to commit this file to GitHub.

Most of the other fields in package.json refer to details that will be used by NPM if you actually go on to publish your project to the NPM repository, in particular this is needed for the CLI and Vue-Components projects. For most visualisation developers, they should not need to touch package.json too much however as simply installing any additional dependencies using npm install <package-name> will automatically add the relevant package to package.json.

Publishing and Updating Packages

Once you have made changes to either the Vue-Components repository or the CLI it may be necessary to update the package on NPM such that users of the ImpVis project can use your brand new changes. Luckily the process of updating an NPM package is relatively simple once you have done a few simple one time setup steps.

Packages should be versioned using standard semantic versioning tags. Minor updates should increment the last digit (a.k.a 1.0.0 -> 1.0.1) whilst more medium updates should increment the middle digit (a.k.a 1.0.1 -> 1.1.0). Finally the leading digit is for MAJOR version updates, typically those that could be considered non backwards compatible, for example 1.1.0 -> 2.0.0. Note that by default packages installed using npm install will automatically update using npm update up to the latest version of the current major version! So this means that it will update in the first two cases but not the last case if a version 2.0 is available. Thus if you do decide to publish a major update you should prompt users to manually bump their major package version. The easiest way for them to do this is to uninstall the current version of the package they have: npm uninstall <package-name> and then reinstall with npm install. Make sure to warn developers that updates to major versions may break their current visualisation and require fixes (this is where handy documentation comes in handy).

Once you have published a new update to an NPM package, most of the ImpVis tools created should prompt their developers with the relevant steps needed to update the package. Failing that any visualisation developer can request a full update check of all packages in a given project by running npm update.

Webpack & Rollup - A general introduction to web bundlers

The problem of dependency management is a tricky one when it comes to web development. Often a given visualisation may depend on many different external libraries, separate script files, images and CSS documents. Not only that but inside of your visualisation, various files may need to be loaded in a certain order. Web bundlers intelligently tackle this problem by scanning through your JavaScript files as well as other file types such as images and breaks your code down into so called chunks. These chunks can then be loaded at the appropriate time and in a specific order in a manner automatically determined by the bundler. This ensures that you never again have to worry about making sure images, CSS files and multiple JS files are loaded in the correct order.

Modern bundlers such as Webpack & Rollup also have support for a number of other cool features such as lazy-loading and tree-shaking. Lazy loading allows the developer to load some assets after the initial page load and only when they are needed. This can help to make the initial page load faster by cutting out unneeded code and only loading certain functionality when it is explicitly needed. Tree shaking on the other hand refers to how a bundler can scan through your code as well as external 3rd party libraries and only load the parts of the libraries that your code actually uses. This is pretty nifty and can dramatically increase the load time of your visualisation however the main downside is that typically libraries need to be written in the newer ESM format to use this (more on that later).

Note that for offline NodeJS scripts such as the CLI, we do not need to use a bundler as there is no performance enhancement to be gained from lazy-loading or tree-shaking as the files are already present on your hard drive, nor do we need to worry about dependency order as Node will handle that for us (more on that later).

How we use Rollup & Webpack

In the current ImpVis development setup we use both Webpack & Rollup bundlers for working on different projects, playing to the strengths of each. For the creation of the Vue-Components library we utilise the Rollup bundler which is a very flexible and powerful bundler optimized for creating libraries such as this. Rollup has very good support for creating libraries whilst webpack is preferred for the actual creation of visualisations using the libraries that we develop. For a full discussion of why this is the case see:

To execute rollup and build the Vue-Components library for production on your machine you can simply run npx rollup -c after npm installing in the project. This will create a series of files located in a folder called dist (short for distribution), in particular it will extract all CSS and SCSS code in the library and place these into a single *.css file, transpiling the SCSS code to CSS as needed and then also create a pair of minified *.js files in two different module formats (explained below). If you are planning on updating the NPM package for the Vue-Components library it is crucial that you run rollup before you publish the new package version as otherwise the library will not be updated! One handy thing that you can do during development is to run npx rollup -cw. This command will build the library as before but now the additional -w flag will watch for changes to the source code as you tweak automatically. Pretty nifty!

Rollup Maintenance & Tweaks

Inspecting the Vue-Components repository on GitHub you will encounter a file called rollup.config.js in the root of the repository. This is a configuration file which determines how the rollup build of the Vue-Components library should function. This file determines a few important fields that rollup needs in order to function, I will attempt to briefly summarise how the current fields function but for more detail I would recommend the official rollup documentation: https://rollupjs.org/guide/en/

Import statements

At the top of the rollup file you will see a bunch of import statements for various plugins for rollup. This is because rollup on its own is not a very advanced piece of software and can in fact only handle *.js files and thus cannot deal with *.vue files used in the components library nor can it deal with any other assets such as image files. Thus at the top of the file we import the plugins that are needed to load the relevant files that we need. Each plugin will need to be installed and is listed as a dev dependency in the package.json file. For the Vue-Components library the plugins used are:

Rollup-plugin-vue: Allows us to load *.vue files with template, script and style tags inside of them
Rollup-plugin-includepaths: By default rollup can only deal with absolute paths from the root of the project. However many developers prefer to use relative paths to access files in the same directory. This plugin allows for this
Rollup-plugin-scss: Allows us to use SCSS for styling. This plugin also compiles all the CSS in both separate *.css files and in inline style blocks in *.vue files into a single impvis-*.css file as well as a minified variant
@rollup/plugin-image: Allows us to load PNG, SVG, JPG and GIF images. These images are converted into a form of binary data called Base64 and embedded as a string directly inside of the output JS files. In this form, the images are around 25% larger in size than they take in their regular sized form and thus it is recommended to use a few images as possible in the final library.
- If you have many larger images or files that you would like to include in a impvis library please consider copying the files to the output directory and having the visualisation developer manually importing them into their visualisation for use
@rollup/plugin-commonjs: Allows us to import 3rd party libraries written in the CommonJS module format. This is explained more in a later section but this typically includes the majority of most 3rd party libraries.
@rollup/plugin-node-resolve: Allows the use of the standard node module resolution system for importing 3rd party modules. Don’t worry too much about how this one works, just know that you probably need this plugin for pretty much every project you use.
@rbnlffl/rollup-plugin-eslint: This is a fork of the official rollup-plugin-eslint plugin which was not working correctly as of July 2020, but may be fixed now. This plugin runs eslint before the rollup build which scans the code and checks for any warnings or code style errors.
@rollup/plugin-alias: Allows for aliases to certain directories to be setup. In particular we alias the src/ directory to the @/ sign as this is an alias that webpack does by default.
Rollup-plugin-terser: This plugin minifies the outputted *.js files from rollup by removing unneeded whitespace characters.
@rollup/plugin-json: Allows for JSON files to be imported.

Entry

The entry line sets the first file that rollup will execute. In ImpVis this file is typically named main.js and is present inside of the src directory. This file has the responsibility of importing all components that should be included in the library and then registering them as global Vue Components

CommonJS, UMD, ESM - An attempt to deconfuse the confusing state of JS modules in 2020

Throughout this guide there have been a few references to JS modules and the different standards that currently exist such as CommonJS. A module in JS is no different to modules in other languages in that it typically represents a singular file which can contain multiple variables, functions and class definitions. In a language like Python there is standardized way for variables and definitions from other files to be accessed for example using import statements. In JS this is not so simple as the various use cases of the language (browser + NodeJS) both have different requirements for how an import system should work.

For those of you who have prior experience working with the ins and outs of the im

Comment by Caroline Clewley - the original Google doc was truncated here...