Skip to content

Instantly share code, notes, and snippets.

@CMCDragonkai
Last active March 29, 2019 11:03
Show Gist options
  • Select an option

  • Save CMCDragonkai/dcc1b538352624ea690d695ee1ff1528 to your computer and use it in GitHub Desktop.

Select an option

Save CMCDragonkai/dcc1b538352624ea690d695ee1ff1528 to your computer and use it in GitHub Desktop.
Developing with Nix (Nodejs, Haskell, Python, PHP, Emscripten) #nix #nixpkgs #nixos #python

Developing with Nix

The nixpkgs community has many subcommunities focused on particular language infrastructure. The most well developed infrastructures would be C/C++, Haskell and Python. Other languages also have support but they tend to have less people activity and people involved.

Basically each language tends to have their own way of setting up a nix shell environment to develop with.

The first problem is often to do with IDE support. For example, spacemacs/emacs can be considered and IDE. IDEs often need integration into external language specific support tools to extend the IDE's capability. This I believe should be done outside the nix shell, and in the nix user profile configuration. More on this later. This is quite important if you're working on multiple languages and complex applications.

Now within the package that you're developing, you must choose to figure out whether this package is an "application" or a "library". If it's an application, you need a default.nix and a shell.nix. But if it's a library you only need a shell.nix.

Think of the "default.nix" as an out-of-nixpkgs nix expression that a nix user can use to install the application. In fact such a "default.nix" could even be contributed to the nixpkgs tree, if needed to. This can also occur with a library, but generally you would put that in nixpkgs, and not keep around a default.nix for that library. Another way to think about it is that private libraries and private applications should use a default.nix, while open source packages should have their default.nix already inside the nixpkgs tree, and there's no need to keep around the default.nix in the project repository, that just results in the need for synchronisation of state.

Think of the "shell.nix" as only creating the development environment for the development of the package. It's nix code would not be contributed to the nixpkgs tree.

One of the main issues is not all language specific packages are available inside the nixpkgs package set. For some languages, the coverage is near 100% for example the haskell package set. For others, it's much lower.

This is often because there's not yet an automation on the conversion of a language specific package set to nixpkgs. Which means individual contributors must package each and every new version of every package. Over time they should all become more automatic.

This means sometimes you'll need to bring in an extra package. There are many ways to achieve this.

You write out a local nix expression that derives the package. You could use one of the conversion tools to autoderive it. You could use the language specific packager inside the nix-shell.

Out of all of these, it is my experience that the shell.nix should be using the language specific packager, while the default.nix should be using one of the conversion tools if available, and if not, a manually written nix expression.

The reason is simple, a language specific packager will allow you to get up to speed and to productivity faster, while exposing various tooling support for that language. For example for nodejs, using npm allows you to monkey patch the dependencies easily, while you can't do that if you use node2nix.

However once everything is done and ready to go, you should be using the full power of the nix tooling when writing the default.nix.

However some languages make this difficult, for example python, since its pip system does not install packages locally. This is why you need to pair it up with virtualenv, so it creates a virtual environment for that python and then you can run pip on the requirements.txt and continue developing. However you can mix and match this with pip and virtualenv. So for example, you can get gdal from nixpkgs, while also specifying pip and downloading the other dependencies via pip. Pip will actually know about the gdal already installed from nixpkgs, so as long as that's compatible, it should be fine. Note that doing this may require you to use compatible version specifiers, because nixpkgs may not be carrying the version of that package you want. However some language infrastructures encourage and is well developed for shell environments, so it is possible to skip the language package management tool and just use that nix tool, but this workflow is spotty (but it might be better for when you're installing dependencies with non-language dependencies). Take care to understand this: https://caremad.io/posts/2013/07/setup-vs-requirement/

Some languages make this easy, such as npm, because it already installs all the packages locally.

So this is how I do it in Python:

create shell.nix with python, pip, setuptools and virtualenv (and also any system dependencies that certain dependencies need, this will require your hard work unless there's a tool that automates some of the derivations)
nix-shell (enter the shell monad)
virtualenv .venv
make sure .venv is ignored in .gitignore
. .venv/bin/activate
pip install whatever
pip install -r requirements.txt
deactivate (or straight to exit is also ok, since you are in a nix-shell)
exit

Another really important factor is whether your shell.nix is reproducible. If you don't specify the content hash of the nixpkgs package set you're using, the shell.nix is not truly reproducible, at least it's not a complete specification. It's an unapplied function, that is still applied to whatever nixpkgs you have in the environment. This is also the key difference between a project specific default.nix and the default.nix you contribute to nixpkgs, in the one contributed to nixpkgs, that would be an unapplied function, using the package set for harmonious operation. While the one exposed by your repository should be content addressed and totally specified and pinned to a nixpkgs content hash. For shell.nix this, this might depend on what you care about. For ease of use, leaving unspecified and dependent on your OS is ok, but this might not be good when working in teams, by making sure your shell.nix pins a nixpkgs content hash, everybody on the team is working on the same package set.

It turns out that python infrastructure on nixpkgs, automatically sets up a development environment. This is called develop mode. This means you can start using pip right away, which means you do not need an extra virtualenv. See this: https://www.johbo.com/2016/using-pip-inside-of-nix-shell.html

This means it should be easy to mix-and-match python dependencies brought from nixpkgs, and python dependencies brought in from pip.

Note that this doesn't necessarily exist for other languages.

For PHP, the extension support hasn't been fully figured out. While composer packages would be installed similar to npm in nodejs.

For Nodejs, packages should just be installed via npm, when developing a library, but use node2nix when deploying an application. Development can use npm or node2nix, just remember, this means an extra overhead when developing, while making it harder to monkey patch.

Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix.

@danbst
Copy link

danbst commented Oct 22, 2017

I've read only the Python section. You may want to add

unset SOURCE_DATE_EPOCH

to shellhook, because otherwise pip can fail with Zip error (NixOS/nixpkgs@6f1551d)

@CMCDragonkai
Copy link
Author

@danbst Doesn't that link say to set SOURCE_DATE_EPOCH to an actual date somewhere?

Also I haven't finished this gist.

@CMCDragonkai
Copy link
Author

@danbst You are right. You do need unset SOURCE_DATE_EPOCH in the shellHook for Python packages.

@vakili
Copy link

vakili commented Mar 29, 2019

Blogpost url has changed to https://matrix.ai/blog/developing-with-nix/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment