The nixpkgs community has many subcommunities focused on particular language infrastructure. The most well developed infrastructures would be C/C++, Haskell and Python. Other languages also have support but they tend to have less people activity and people involved.

Basically each language tends to have their own way of setting up a nix shell environment to develop with.

The first problem is often to do with IDE support. For example, spacemacs/emacs can be considered and IDE. IDEs often need integration into external language specific support tools to extend the IDE's capability. This I believe should be done outside the nix shell, and in the nix user profile configuration. More on this later. This is quite important if you're working on multiple languages and complex applications.

Now within the package that you're developing, you must choose to figure out whether this package is an "application" or a "library". If it's an application, you need a default.nix and a shell.nix. But if it's a library you only need a shell.nix.

Think of the "default.nix" as an out-of-nixpkgs nix expression that a nix user can use to install the application. In fact such a "default.nix" could even be contributed to the nixpkgs tree, if needed to. This can also occur with a library, but generally you would put that in nixpkgs, and not keep around a default.nix for that library. Another way to think about it is that private libraries and private applications should use a default.nix, while open source packages should have their default.nix already inside the nixpkgs tree, and there's no need to keep around the default.nix in the project repository, that just results in the need for synchronisation of state.

Think of the "shell.nix" as only creating the development environment for the development of the package. It's nix code would not be contributed to the nixpkgs tree.

One of the main issues is not all language specific packages are available inside the nixpkgs package set. For some languages, the coverage is near 100% for example the haskell package set. For others, it's much lower.

This is often because there's not yet an automation on the conversion of a language specific package set to nixpkgs. Which means individual contributors must package each and every new version of every package. Over time they should all become more automatic.

This means sometimes you'll need to bring in an extra package. There are many ways to achieve this.

You write out a local nix expression that derives the package. You could use one of the conversion tools to autoderive it. You could use the language specific packager inside the nix-shell.

Out of all of these, it is my experience that the shell.nix should be using the language specific packager, while the default.nix should be using one of the conversion tools if available, and if not, a manually written nix expression.

The reason is simple, a language specific packager will allow you to get up to speed and to productivity faster, while exposing various tooling support for that language. For example for nodejs, using npm allows you to monkey patch the dependencies easily, while you can't do that if you use node2nix.

However once everything is done and ready to go, you should be using the full power of the nix tooling when writing the default.nix.

However some languages make this difficult, for example python, since its pip system does not install packages locally. This is why you need to pair it up with virtualenv, so it creates a virtual environment for that python and then you can run pip on the requirements.txt and continue developing. However you can mix and match this with pip and virtualenv. So for example, you can get gdal from nixpkgs, while also specifying pip and downloading the other dependencies via pip. Pip will actually know about the gdal already installed from nixpkgs, so as long as that's compatible, it should be fine. Note that doing this may require you to use compatible version specifiers, because nixpkgs may not be carrying the version of that package you want. However some language infrastructures encourage and is well developed for shell environments, so it is possible to skip the language package management tool and just use that nix tool, but this workflow is spotty (but it might be better for when you're installing dependencies with non-language dependencies). Take care to understand this: https://caremad.io/posts/2013/07/setup-vs-requirement/

Some languages make this easy, such as npm, because it already installs all the packages locally.

So this is how I do it in Python:

create shell.nix with python, pip, setuptools and virtualenv (and also any system dependencies that certain dependencies need, this will require your hard work unless there's a tool that automates some of the derivations)
nix-shell (enter the shell monad)
virtualenv .venv
make sure .venv is ignored in .gitignore
. .venv/bin/activate
pip install whatever
pip install -r requirements.txt
deactivate (or straight to exit is also ok, since you are in a nix-shell)
exit

Another really important factor is whether your shell.nix is reproducible. If you don't specify the content hash of the nixpkgs package set you're using, the shell.nix is not truly reproducible, at least it's not a complete specification. It's an unapplied function, that is still applied to whatever nixpkgs you have in the environment. This is also the key difference between a project specific default.nix and the default.nix you contribute to nixpkgs, in the one contributed to nixpkgs, that would be an unapplied function, using the package set for harmonious operation. While the one exposed by your repository should be content addressed and totally specified and pinned to a nixpkgs content hash. For shell.nix this, this might depend on what you care about. For ease of use, leaving unspecified and dependent on your OS is ok, but this might not be good when working in teams, by making sure your shell.nix pins a nixpkgs content hash, everybody on the team is working on the same package set.

It turns out that python infrastructure on nixpkgs, automatically sets up a development environment. This is called develop mode. This means you can start using pip right away, which means you do not need an extra virtualenv. See this: https://www.johbo.com/2016/using-pip-inside-of-nix-shell.html

This means it should be easy to mix-and-match python dependencies brought from nixpkgs, and python dependencies brought in from pip.

Note that this doesn't necessarily exist for other languages.

For PHP, the extension support hasn't been fully figured out. While composer packages would be installed similar to npm in nodejs.

For Nodejs, packages should just be installed via npm, when developing a library, but use node2nix when deploying an application. Development can use npm or node2nix, just remember, this means an extra overhead when developing, while making it harder to monkey patch.

Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix. To expand on this, if you have networked services, it is easier to isolate it from the perspective from just using nix-shell, if you instead use unix domain sockets and put them into project specific hidden folders. For example ./.mysql for the mysql directory and ./.mysql/mysql.sock for the mysql socket. Then set the necessary environment variables to make sure all mysql commands will be using it.

To run a command directly when launching into shell.nix. You should be using shellHook. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the shellHook. This is also where you can set environment variables inside here.

This is how you setup a local mysql:

rm -rf .mysql && mkdir .mysql
mysqld --datadir="$(pwd)/.mysql" --initialize-insecure
mysqld --datadir="$(pwd)/.mysql" --socket="$(pwd)/.mysql/mysql.sock" --skip-networking &
mysql --socket='./.mysql/mysql.sock'

Now all services connect over the unix domain socket. Each has different ways of doing this. Now if only HTTP services could also do this, now local development would be super easy. Well browsers need to support unix domain socket access, while curl and like do support it, and you won't need the container for isolating networks, or network namespace at all.

Although this is only available on unix systems, windows will have to suck it.

Note that SSH can forward over unix domain sockets without the need of socat.

Deploying an unfree no source package that uses the GUI can be pretty complicated due to all the QT dependencies.

We take the example of https://code-industry.net/free-pdf-editor/.

First we download the correct architecture for our system. Extract it and check the insides. You can try ldd on the executable however it won't work, because the loader is set incorrectly. So we need to first set the path to the loader. By using patchelf. Find an appropriate interpreter: ls /nix/store/*-glibc-*/lib/ld-linux*so. And then set it on the binary: patchelf --set-interpreter /nix/store/68sa3m89shpfaqq1b9xp5p1360vqhwx6-glibc-2.25/lib/ld-linux-x86-64.so.2 ./masterpdfeditor4. Now you can use ldd to check all the shared objectis it is relying on.

At this point there is domain knowledge required to find out which nix dependencies correspond to the shared libraries required. But some common ones are like pthread come from the glibc package. One of the most common will probably be the C libraries, which can come from stdenv.cc.cc. Then qt libraries is pretty complicated as that can come in a set. Finally alot of the x11 libraries will come from xorg.

All these will need to be encoded into the executable using patchelf again, but this time via the RPATH setting.

Furthermore stripping may need to be disabled, as it can break the executable.

Beyond dynamic libraries, the executable may call external resources, these may be supplied via a relative path. In the tar.gz we have, we find folders that are relative to the executable, so when you call the executable, it will be looking for these resources relative to its position, this means we may need to copy these directories to the bin folder of the out directory of the build nix package. Finding ways to rewrite the paths may be possible with a textual program, but not really with a binary program.

We also want a sha256 hash of the executable, so we can use nix-prefetch-url on the link.

Finally you need to write the default.nix file. At first I tried to create one similar to an out-of-tree default.nix and just use nix-shell and nix-build to try to build it. In order to detect further paths that is not being opened try using strace like strace -e open ./result/... | grep ENOENT. Which will show you open attempts that fail, now not all attempts are bad, because of the way RPATH works, it may try to open libraries in every RPATH and include path available. So you usually are looking for paths that it may try to open relative to the binary location.

In the default.nix that you're writing, you primarily only need to override the installPhase, since this is already a prebuilt executable, so there's not much other phases you need to change here.

In the dev workflow, this default.nix is going to be different from the default.nix for an out-of-tree package, since now we need to be harmonious with the rest of the tree, and we need to add our package to the pkgs/top-level/all-packages.nix. Most importantly after you add this to your development version of nixpkgs (remember not the /nix/nixpkgs), you can just test the installation by doing nix-env -f ./dev/nixpkgs/default.nix -i masterpdfeditor, and this will try to install it from the dev tree of nixpkgs.

Now everything should go fine, UNLESS there are side-effectful things you still haven't encapsulated. Examples include GL libraries which usually come from LD_LIBRARY_PATH, as these may be supplied by your graphics card. So these will usually not be specified inside the default.nix inside the nixpkgs, unless you really need to, usually as a backupo graphics library that uses the mesa drivers.

But other things can happen to like QT plugins that may be conflicting.

Anyway the lack of purity by default is what makes this a difficult process.

Note that for choosing between different architectures, a nice trick is to use attribute keys with architecture targets, and choosing them using stdenv.system. For example:

    url = {
      "i686-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_i386.tar.gz";
      "x86_64-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_qt5.amd64.tar.gz";
    }."${stdenv.system}";