@clacke My theory was that as it hadn't enough memory it had to swap more (as is usually the case). I can't say how it is on your system. It was just an assumption. So I suggested that you could try increase the size of tmpfs so it wouldn't need to swap as much (as it's faster).
It pages to swap when ram is not not needed or insufficient.. Like it can happen when compiling... I can't say if gcc, haskell or Nix works differently here.
Perhaps the swappiness is tuned high on NixOS?