Conversion to Scalability

Welcome. You're here either because you want to build a scalable static site, or you don't.

Locate a copy of your existing non-scalable static site. Its layout might look something like this (mine did):

 1[pastly@home:~/src/my-first-website]$ tree .
 2.
 3├── blog
 4│   ├── 1st-post.html
 5│   ├── first-post.html
 6│   ├── hello-world.html
 7│   ├── post-1.html
 8│   ├── test.html
 9├── css
10│   ├── exploit-tor-browser-users.css
11│   └── style.ccs
12│   └── style.css
13├── index.html
14└── js
15    └── deliver-ads.js
16
174 directories, 14 files

In this post I am moving the contents of my blog into my computer's nix store. For more information on Nix, see the first first post, Introduction to Nix.

I created a directory and starting hacking away. Here's where I've landed, with false starts pointless details elided.

1[pastly@home:~/src/wsss]$ tree
2.
3├── do-dir.nix
4├── do-dir.sh
5├── do-file.nix
6├── do-file.sh
7├── repo.nix
8└── source-static-site -> [...]

source-static-site is a symlink to ~/src/my-first-website for you.

repo.nix is this, and defines the root of a "nix package repository" built upon the main nixpkgs repository that comes with NixOS.

 1let
 2  nixpkgs = import <nixpkgs> {};
 3  allPkgs = nixpkgs // pkgs;
 4  callPackage = path: overrides:
 5    let f = import path;
 6    in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs) // overrides);
 7  pkgs = with nixpkgs; {
 8    post01 = callPackage ./do-dir.nix { dir_name = ./source-static-site; };
 9  };
10in pkgs

I build the "post01" package with nix-build repo.nix -A post01. This kicks off a chain of recursion that stores each individual file and directory in my nix store.

do-file.nix and do-file.sh are here, where the former is basically just boiler plate to call the second, and the second is boiler plate for calling cp.

1{ pkgs, file_name }:
2derivation {
3  inherit file_name;
4  inherit (pkgs) coreutils;
5  name = builtins.concatStringsSep "-" [(builtins.baseNameOf file_name) "compiled"];
6  builder = "${pkgs.bash}/bin/bash";
7  args = [ ./do-file.sh ];
8  system = builtins.currentSystem;
9}
1$coreutils/bin/cp $file_name $out

Beautiful, and in need of no explanation.

do-dir.nix and do-dir.sh are here and slightly more complex than their file counterparts.

 1{ pkgs, lib, callPackage, dir_name }:
 2let
 3  contents = builtins.readDir dir_name;
 4  files = map (x: callPackage ./do-file.nix { file_name = dir_name + "/${x}"; }) (builtins.attrNames (lib.attrsets.filterAttrs (n: v: v == "regular") contents));
 5  dirs =  map (x: callPackage ./do-dir.nix  { dir_name  = dir_name + "/${x}"; }) (builtins.attrNames (lib.attrsets.filterAttrs (n: v: v == "directory") contents));
 6in 
 7  derivation {
 8    inherit dir_name;
 9    inherit files dirs;
10    inherit (pkgs) coreutils;
11    name = builtins.concatStringsSep "-" [(builtins.baseNameOf dir_name) "compiled"];
12    builder = "${pkgs.bash}/bin/bash";
13    args = [ ./do-dir.sh ];
14    system = builtins.currentSystem;
15  }
1echo "FILES -------------------------------------------------" >> $out
2for f in $files; do
3	echo $f >> $out
4done
5
6echo "DIRS -------------------------------------------------" >> $out
7for f in $dirs; do
8	echo $f >> $out
9done

The output of a directory in the nix store is a text file that lists all its file and directory children. The key to recursion is the callPackage calls that are made once for every sub file and sub directory of the current one.

After a nix-build repo.nix -A post01, you can now cat result and get something like this:

[pastly@home:~/src/wsss]$ cat result
FILES -------------------------------------------------
/nix/store/8qzmb9k8im2hli8xfzi8skxf1ps389dw-index.html-compiled
DIRS -------------------------------------------------
/nix/store/l3m9m96nck2wmm8nh1w5308g0pnb3p85-blog-compiled
/nix/store/9d3a196hx42c2gsf2qnv867sj4ymmh2k-css-compiled
/nix/store/dilifja1h3v6ppwxb4dlc0nm310hir4q-js-compiled

And you can further drill down. E.g.

1[pastly@home:~/src/wsss]$ cat /nix/store/dilifja1h3v6ppwxb4dlc0nm310hir4q-js-compiled
2FILES -------------------------------------------------
3/nix/store/zyvb8qcmlqn4vsf8q7bi4p2mbybv6phn-deliver-ads.js-compiled
4DIRS -------------------------------------------------
5
6[pastly@home:~/src/wsss]$ cat /nix/store/zyvb8qcmlqn4vsf8q7bi4p2mbybv6phn-deliver-ads.js-compiled
7[... the contents of deliver-ads.js ...]

This is the stopping point for today. Next time we'll involve Rust in some way.

The WSSS source code is on github: https://github.com/pastly/wsss. Code as of the original version of this post is on this branch: post01.

Matt Traudt (pastly)

Tech, Pets, and Vettes


2023-10-09