Skip to content

Instantly share code, notes, and snippets.

@CMCDragonkai

CMCDragonkai/developing_with_nix.md

Last active March 29, 2019 11:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save CMCDragonkai/dcc1b538352624ea690d695ee1ff1528 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save CMCDragonkai/dcc1b538352624ea690d695ee1ff1528 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. CMCDragonkai revised this gist Aug 17, 2018. 1 changed file with 9 additions and 1 deletion.
    10 changes: 9 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,12 @@
    Developing with Nix
    ===================

    This has been moved to the Matrix AI blog: https://matrix.ai/2018/03/24/developing-with-nix/
    This has been moved to the Matrix AI blog: https://matrix.ai/2018/03/24/developing-with-nix/

    Note that if you just to compile a simple C program, and you don't want to fiddle with environments:

    ```sh
    nix-shell --packages stdenv
    ```

    That's enough to enter an environment with `stdenv`.
  2. CMCDragonkai revised this gist Mar 24, 2018. 1 changed file with 1 addition and 300 deletions.
    301 changes: 1 addition & 300 deletions developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,303 +1,4 @@
    Developing with Nix
    ===================

    The nixpkgs community has many subcommunities focused on particular language infrastructure.
    The most well developed infrastructures would be C/C++, Haskell and Python. Other languages also
    have support but they tend to have less people activity and people involved.

    Basically each language tends to have their own way of setting up a nix shell environment to develop with.

    The first problem is often to do with IDE support. For example, spacemacs/emacs can be considered and IDE.
    IDEs often need integration into external language specific support tools to extend the IDE's capability.
    This I believe should be done outside the nix shell, and in the nix user profile configuration. More on this later.
    This is quite important if you're working on multiple languages and complex applications.

    Now within the package that you're developing, you must choose to figure out whether this package is an "application"
    or a "library". If it's an application, you need a default.nix and a shell.nix. But if it's a library you only need a shell.nix.

    Think of the "default.nix" as an out-of-nixpkgs nix expression that a nix user can use to install the application.
    In fact such a "default.nix" could even be contributed to the nixpkgs tree, if needed to. This can also occur with a library,
    but generally you would put that in nixpkgs, and not keep around a default.nix for that library. Another way to think about it
    is that private libraries and private applications should use a default.nix, while open source packages should have their default.nix
    already inside the nixpkgs tree, and there's no need to keep around the default.nix in the project repository, that just results
    in the need for synchronisation of state.

    Think of the "shell.nix" as only creating the development environment for the development of the package. It's nix code
    would not be contributed to the nixpkgs tree.

    One of the main issues is not all language specific packages are available inside the nixpkgs package set.
    For some languages, the coverage is near 100% for example the haskell package set. For others, it's much lower.

    This is often because there's not yet an automation on the conversion of a language specific package set to nixpkgs.
    Which means individual contributors must package each and every new version of every package. Over time they should
    all become more automatic.

    This means sometimes you'll need to bring in an extra package. There are many ways to achieve this.

    You write out a local nix expression that derives the package.
    You could use one of the conversion tools to autoderive it.
    You could use the language specific packager inside the nix-shell.

    Out of all of these, it is my experience that the shell.nix should be using the language specific packager, while
    the default.nix should be using one of the conversion tools if available, and if not, a manually written nix expression.

    The reason is simple, a language specific packager will allow you to get up to speed and to productivity faster, while
    exposing various tooling support for that language. For example for nodejs, using npm allows you to monkey patch the dependencies
    easily, while you can't do that if you use node2nix.

    However once everything is done and ready to go, you should be using the full power of the nix tooling when writing the default.nix.

    However some languages make this difficult, for example python, since its pip system does not install packages locally.
    This is why you need to pair it up with virtualenv, so it creates a virtual environment for that python and then you can
    run pip on the requirements.txt and continue developing. However you can mix and match this with pip and virtualenv. So for example, you can get gdal from nixpkgs, while also specifying pip and downloading the other dependencies via pip. Pip will actually know about the gdal already installed from nixpkgs, so as long as that's compatible, it should be fine. Note that doing this may require you to use compatible version specifiers, because nixpkgs may not be carrying the version of that package you want. However some language infrastructures encourage and is well developed for shell environments, so it is possible to skip the language package management tool and just use that nix tool, but this workflow is spotty (but it might be better for when you're installing dependencies with non-language dependencies). Take care to understand this: https://caremad.io/posts/2013/07/setup-vs-requirement/

    Some languages make this easy, such as npm, because it already installs all the packages locally.

    So this is how I do it in Python:

    ```
    create shell.nix with python, pip, setuptools and virtualenv (and also any system dependencies that certain dependencies need, this will require your hard work unless there's a tool that automates some of the derivations)
    nix-shell (enter the shell monad)
    virtualenv .venv
    make sure .venv is ignored in .gitignore
    . .venv/bin/activate
    pip install whatever
    pip install -r requirements.txt
    deactivate (or straight to exit is also ok, since you are in a nix-shell)
    exit
    ```

    Another really important factor is whether your shell.nix is reproducible. If you don't specify the content hash of the nixpkgs package set you're using, the shell.nix is not truly reproducible, at least it's not a complete specification. It's an unapplied function, that is still applied to whatever nixpkgs you have in the environment. This is also the key difference between a project specific default.nix and the default.nix you contribute to nixpkgs, in the one contributed to nixpkgs, that would be an unapplied function, using the package set for harmonious operation. While the one exposed by your repository should be content addressed and totally specified and pinned to a nixpkgs content hash. For shell.nix this, this might depend on what you care about. For ease of use, leaving unspecified and dependent on your OS is ok, but this might not be good when working in teams, by making sure your shell.nix pins a nixpkgs content hash, everybody on the team is working on the same package set.

    It turns out that python infrastructure on nixpkgs, automatically sets up a development environment. This is called develop mode. This means you can start using pip right away, which means you do not need an extra virtualenv. See this: https://www.johbo.com/2016/using-pip-inside-of-nix-shell.html

    This means it should be easy to mix-and-match python dependencies brought from nixpkgs, and python dependencies brought in from pip.

    Note that this doesn't necessarily exist for other languages.

    For PHP, the extension support hasn't been fully figured out. While composer packages would be installed similar to npm in nodejs.

    For Nodejs, packages should just be installed via npm, when developing a library, but use node2nix when deploying an application. Development can use npm or node2nix, just remember, this means an extra overhead when developing, while making it harder to monkey patch.

    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix. To expand on this, if you have networked services, it is easier to isolate it from the perspective from just using nix-shell, if you instead use unix domain sockets and put them into project specific hidden folders. For example `./.mysql` for the mysql directory and `./.mysql/mysql.sock` for the mysql socket. Then set the necessary environment variables to make sure all mysql commands will be using it.

    To run a command directly when launching into shell.nix. You should be using `shellHook`. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the `shellHook`. This is also where you can set environment variables inside here.

    This is how you setup a local mysql:

    ```sh
    rm -rf .mysql && mkdir .mysql
    mysqld --datadir="$(pwd)/.mysql" --initialize-insecure
    mysqld --datadir="$(pwd)/.mysql" --socket="$(pwd)/.mysql/mysql.sock" --skip-networking &
    mysql --socket='./.mysql/mysql.sock'
    ```

    Now all services connect over the unix domain socket. Each has different ways of doing this. Now if only HTTP services could also do this, now local development would be super easy. Well browsers need to support unix domain socket access, while curl and like do support it, and you won't need the container for isolating networks, or network namespace at all.

    Although this is only available on unix systems, windows will have to suck it.

    Note that SSH can forward over unix domain sockets without the need of socat.

    Deploying an unfree no source package that uses the GUI can be pretty complicated due to all the QT dependencies.

    We take the example of https://code-industry.net/free-pdf-editor/.

    First we download the correct architecture for our system. Extract it and check the insides. You can try ldd on the executable however it won't work, because the loader is set incorrectly. So we need to first set the path to the loader. By using `patchelf`. Find an appropriate interpreter: `ls /nix/store/*-glibc-*/lib/ld-linux*so`. And then set it on the binary: `patchelf --set-interpreter /nix/store/68sa3m89shpfaqq1b9xp5p1360vqhwx6-glibc-2.25/lib/ld-linux-x86-64.so.2 ./masterpdfeditor4`. Now you can use `ldd` to check all the shared objectis it is relying on.

    At this point there is domain knowledge required to find out which nix dependencies correspond to the shared libraries required. But some common ones are like pthread come from the glibc package. One of the most common will probably be the C libraries, which can come from stdenv.cc.cc. Then qt libraries is pretty complicated as that can come in a set. Finally alot of the x11 libraries will come from xorg.

    All these will need to be encoded into the executable using `patchelf` again, but this time via the RPATH setting.

    Furthermore stripping may need to be disabled, as it can break the executable.

    Beyond dynamic libraries, the executable may call external resources, these may be supplied via a relative path. In the tar.gz we have, we find folders that are relative to the executable, so when you call the executable, it will be looking for these resources relative to its position, this means we may need to copy these directories to the bin folder of the out directory of the build nix package. Finding ways to rewrite the paths may be possible with a textual program, but not really with a binary program.

    We also want a sha256 hash of the executable, so we can use `nix-prefetch-url` on the link.

    Finally you need to write the default.nix file. At first I tried to create one similar to an out-of-tree default.nix and just use nix-shell and nix-build to try to build it. In order to detect further paths that is not being opened try using strace like `strace -e open ./result/... | grep ENOENT`. Which will show you open attempts that fail, now not all attempts are bad, because of the way RPATH works, it may try to open libraries in every RPATH and include path available. So you usually are looking for paths that it may try to open relative to the binary location.

    In the default.nix that you're writing, you primarily only need to override the `installPhase`, since this is already a prebuilt executable, so there's not much other phases you need to change here.

    In the dev workflow, this default.nix is going to be different from the default.nix for an out-of-tree package, since now we need to be harmonious with the rest of the tree, and we need to add our package to the pkgs/top-level/all-packages.nix. Most importantly after you add this to your development version of nixpkgs (remember not the /nix/nixpkgs), you can just test the installation by doing `nix-env -f ./dev/nixpkgs/default.nix -i masterpdfeditor`, and this will try to install it from the dev tree of nixpkgs.

    Now everything should go fine, UNLESS there are side-effectful things you still haven't encapsulated. Examples include GL libraries which usually come from `LD_LIBRARY_PATH`, as these may be supplied by your graphics card. So these will usually not be specified inside the default.nix inside the nixpkgs, unless you really need to, usually as a backupo graphics library that uses the mesa drivers.

    But other things can happen to like QT plugins that may be conflicting.

    Anyway the lack of purity by default is what makes this a difficult process.

    Note that for choosing between different architectures, a nice trick is to use attribute keys with architecture targets, and choosing them using `stdenv.system`. For example:

    ```
    url = {
    "i686-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_i386.tar.gz";
    "x86_64-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_qt5.amd64.tar.gz";
    }."${stdenv.system}";
    ```

    To wipe out a nix-build result without leaving the data in /nix/store, use `nix-store --delete --ignore-liveness $(readlink ./result) && rm ./result`.

    If you just delete the result symlink, then the only way to get rid of it is to use `nix-store --gc`. You can also show all the dead paths to be deleted by using `nix-store --print-dead`.

    Python
    ------

    I found the best way to develop in Python with nix-shell.

    You need a shell.nix:

    ```
    { pkgs ? import <nixpkgs> {} }:
    with pkgs;
    python35Packages.buildPythonApplication {
    name = "myawesomepythonapp";
    buildInputs = [ zlib ] ++ with python35Packages; [ numpy requests2 ];
    }
    ```

    Note the difference between a python application and a python library. An application's main consumption isn't meant to be a library form. Whereas a package is meant to be submitted to pip and be consumed like a library.

    Now what happens is that we have a shell environment with system `zlib` available, while `numpy` and `requests2` come from Nix. These are all available within a python extended `stdenv.mkDerivation` environment. So you can do everything that `mkDerivation` does, but you also have all the nice package hooks of python, and if you use this for `nix-build` it will run some functions relating to `setup.py` that will test things and build wheels... etc.

    Next you need a `setup.py`, and just fill it out normally. Optionally you can have a `requirements.txt`. In fact you should always have this with the corresponding hashes. Think of the `requirements.txt` as the a package lock. Which locks the dependencies and the hashes you expect. This makes the end result work across non-nix environments.

    Now you can use `pip`. There's a trick to make pip behave like npm or composer. You need to use the `--prefix` option. This is the correct option, `--root` isn't and neither is `--target`. You can now do things like `pip install --prefix ./pip_packages -r requirements.txt`. Or you can specify dependencies imperatively. Pip will know about the `$PYTHONPATH` which is already filled with your nix installed python packages, and any specified package imperatively or via `requirements.txt` that are compatible with the nix installed python packages will not be further installed. Any non-compatible or missing dependencies will now be installed into the `./pip_packages` folder. However if it detects a conflict between a nix dependency and your own, after installing the dependency into `./pip_packages` it will try to uninstall the nix installed dependency. This will simply fail because nix dependencies are immutable. And this is a good thing. It forces you to make sure that either all of your application dependencies can be satisfied by your nix shell.nix specification or that it will give an error. However... you can force pip to ignore and make sure to install whatever version you want, by using the `--ignore-installed` option (shortcut `-I`). This will force install and not bother trying to uninstall things. You shouldn't really need to do this.

    Finally now that you have all the dependencies available, either via the nix or via the `./pip_packages`. You can now set this as part of your `$PYTHONPATH`. It's important to be able to set this as a prepend. So any packages found there will take precedence over nix installed dependencies. It's also what makes it possible to use `pip freeze` and `pip uninstall` that will first prefer listing or uninstalling the `./pip_packages`.

    But the way to do this is a bit more complicated. You need to do this:

    ```
    export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"
    ```

    However I haven't found a way to get the name of the `python3.5` automatically. At any case this would alway be fixed to your `python35Packages` set, and you will just have to change the name accordingly.

    Now you can use pip and everything else naturally! Remember to add `pip_packages` to your `.gitignore`. Also remember to delete the cache if you have problems `~/.cache/pip`.

    Here's a full shell.nix:

    ```
    { pkgs ? import <nixpkgs> {} }:
    with pkgs;
    python35Packages.buildPythonApplication {
    name = "illuminate-magsurv";
    buildInputs = (with python35Packages; [
    numpy
    (matplotlib.override { enableQt = true; })
    gdal
    scikitimage
    tensorflowWithCuda
    ]);
    shellHook = ''
    echo 'Entering Illuminate Magsurv Environmnet'
    set -v
    # extra packages can be installed here
    unset SOURCE_DATE_EPOCH
    export PIP_PREFIX="$(pwd)/pip_packages"
    python_path=(
    "$(pwd)/pip_packages/lib/python3.6/site-packages"
    "$PYTHONPATH"
    )
    # use double single quotes to escape bash quoting
    IFS=: eval 'python_path="''${python_path[*]}"'
    export PYTHONPATH="$python_path"
    export MPLBACKEND='Qt4Agg'
    set +v
    '';
    }
    ```

    My motto: Implicit imperative is bad. Implicit or explicit declarative is good. Explicit imperative is ok.


    ---

    If you are using the nixos pinning, use shell.nix pinning on the current hash of your nixpkgs:

    ```
    git -C /nix/nixpkgs rev-parse HEAD
    ```

    This gives you the long form hash:

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/8fefa85f5cc95a99c78776fdfb971deff3592cab.tar.gz) {}
    }:
    ```

    ---

    Emscripten:

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/00e56fbbee06088bf3bf82169032f5f5778588b7.tar.gz) {}
    }:
    with pkgs;
    stdenv.mkDerivation {
    name = "js-virtualgit";
    buildInputs = [
    emscripten
    cmakeCurses
    pkgconfig
    python2
    nodejs
    nodePackages.node2nix
    flow
    ];
    shellHook = ''
    EMSCRIPTEN_ROOT_PATH='${emscripten}/share/emscripten'
    EMSCRIPTEN='${emscripten}/share/emscripten'
    '';
    }
    ```

    What's left in Emscripten is package hooks for each emscripten dependency so you can use emscripten versions of zlib. Currently the pkg-config isn't done properly.

    ---

    Node

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/00e56fbbee06088bf3bf82169032f5f5778588b7.tar.gz) {}
    }:
    with pkgs;
    stdenv.mkDerivation {
    name = "js-virtualgit";
    buildInputs = [ nodejs-8_x nodePackages_6_x.node-gyp blas flow ];
    }
    ```

    The above will make sure to get nodejs 8, node-gyp, blas and flow. The node-gyp and blas is what is necessary to perform `npm install nblas`. So that's how you get npm packages with C++ bindings working.

    ---

    Haskell + Stack:

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/8b1cf100cd8badad6e1b6d4650b904b88aa870db.tar.gz) {}
    }:
    with pkgs;
    haskell.lib.buildStackProject {
    name = "graphql-demo";
    buildInputs = [];
    shellHook = ''
    echo 'Entering GraphQL Demo Environment'
    set -v
    alias stack="\stack --nix"
    set +v
    '';
    }
    ```

    ---

    Using aliases inside your shellHook is pretty great and easy. One thing you have to watch out for is that aliases are not inherited by shell scripts. So if you run a shell script while inside your nix-shell environment, that will not use the alias call. So it's actually better to be using environment variables as a switch for whatever your command requires. On the other hand, if you use environment variables, you have to be careful about environment variable conflict or leakage. It kind of sucks. But the really only foolproof method requires you to write the correct nix expression for the commands you expect to run, but this is quite verbose. So you just have to do whatever you can to make it all work.
    This has been moved to the Matrix AI blog: https://matrix.ai/2018/03/24/developing-with-nix/
  3. CMCDragonkai revised this gist Mar 20, 2018. 1 changed file with 11 additions and 2 deletions.
    13 changes: 11 additions & 2 deletions developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -193,8 +193,17 @@ Here's a full shell.nix:
    echo 'Entering Illuminate Magsurv Environmnet'
    set -v
    alias pip="PIP_PREFIX='$(pwd)/pip_packages' \pip"
    export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"
    # extra packages can be installed here
    unset SOURCE_DATE_EPOCH
    export PIP_PREFIX="$(pwd)/pip_packages"
    python_path=(
    "$(pwd)/pip_packages/lib/python3.6/site-packages"
    "$PYTHONPATH"
    )
    # use double single quotes to escape bash quoting
    IFS=: eval 'python_path="''${python_path[*]}"'
    export PYTHONPATH="$python_path"
    export MPLBACKEND='Qt4Agg'
    set +v
    '';
  4. CMCDragonkai revised this gist Feb 21, 2018. 1 changed file with 23 additions and 0 deletions.
    23 changes: 23 additions & 0 deletions developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -268,4 +268,27 @@ The above will make sure to get nodejs 8, node-gyp, blas and flow. The node-gyp

    ---

    Haskell + Stack:

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/8b1cf100cd8badad6e1b6d4650b904b88aa870db.tar.gz) {}
    }:
    with pkgs;
    haskell.lib.buildStackProject {
    name = "graphql-demo";
    buildInputs = [];
    shellHook = ''
    echo 'Entering GraphQL Demo Environment'
    set -v
    alias stack="\stack --nix"
    set +v
    '';
    }
    ```

    ---

    Using aliases inside your shellHook is pretty great and easy. One thing you have to watch out for is that aliases are not inherited by shell scripts. So if you run a shell script while inside your nix-shell environment, that will not use the alias call. So it's actually better to be using environment variables as a switch for whatever your command requires. On the other hand, if you use environment variables, you have to be careful about environment variable conflict or leakage. It kind of sucks. But the really only foolproof method requires you to write the correct nix expression for the commands you expect to run, but this is quite verbose. So you just have to do whatever you can to make it all work.
  5. CMCDragonkai revised this gist Feb 21, 2018. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -264,4 +264,8 @@ Node
    }
    ```

    The above will make sure to get nodejs 8, node-gyp, blas and flow. The node-gyp and blas is what is necessary to perform `npm install nblas`. So that's how you get npm packages with C++ bindings working.
    The above will make sure to get nodejs 8, node-gyp, blas and flow. The node-gyp and blas is what is necessary to perform `npm install nblas`. So that's how you get npm packages with C++ bindings working.

    ---

    Using aliases inside your shellHook is pretty great and easy. One thing you have to watch out for is that aliases are not inherited by shell scripts. So if you run a shell script while inside your nix-shell environment, that will not use the alias call. So it's actually better to be using environment variables as a switch for whatever your command requires. On the other hand, if you use environment variables, you have to be careful about environment variable conflict or leakage. It kind of sucks. But the really only foolproof method requires you to write the correct nix expression for the commands you expect to run, but this is quite verbose. So you just have to do whatever you can to make it all work.
  6. CMCDragonkai revised this gist Dec 16, 2017. 1 changed file with 18 additions and 1 deletion.
    19 changes: 18 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -247,4 +247,21 @@ Emscripten:
    }
    ```

    What's left in Emscripten is package hooks for each emscripten dependency so you can use emscripten versions of zlib. Currently the pkg-config isn't done properly.
    What's left in Emscripten is package hooks for each emscripten dependency so you can use emscripten versions of zlib. Currently the pkg-config isn't done properly.

    ---

    Node

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/00e56fbbee06088bf3bf82169032f5f5778588b7.tar.gz) {}
    }:
    with pkgs;
    stdenv.mkDerivation {
    name = "js-virtualgit";
    buildInputs = [ nodejs-8_x nodePackages_6_x.node-gyp blas flow ];
    }
    ```

    The above will make sure to get nodejs 8, node-gyp, blas and flow. The node-gyp and blas is what is necessary to perform `npm install nblas`. So that's how you get npm packages with C++ bindings working.
  7. CMCDragonkai revised this gist Nov 11, 2017. 1 changed file with 30 additions and 1 deletion.
    31 changes: 30 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -218,4 +218,33 @@ This gives you the long form hash:
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/8fefa85f5cc95a99c78776fdfb971deff3592cab.tar.gz) {}
    }:
    ```
    ```

    ---

    Emscripten:

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/00e56fbbee06088bf3bf82169032f5f5778588b7.tar.gz) {}
    }:
    with pkgs;
    stdenv.mkDerivation {
    name = "js-virtualgit";
    buildInputs = [
    emscripten
    cmakeCurses
    pkgconfig
    python2
    nodejs
    nodePackages.node2nix
    flow
    ];
    shellHook = ''
    EMSCRIPTEN_ROOT_PATH='${emscripten}/share/emscripten'
    EMSCRIPTEN='${emscripten}/share/emscripten'
    '';
    }
    ```

    What's left in Emscripten is package hooks for each emscripten dependency so you can use emscripten versions of zlib. Currently the pkg-config isn't done properly.
  8. CMCDragonkai revised this gist Oct 30, 2017. 1 changed file with 18 additions and 1 deletion.
    19 changes: 18 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -201,4 +201,21 @@ Here's a full shell.nix:
    }
    ```

    My motto: Implicit imperative is bad. Implicit or explicit declarative is good. Explicit imperative is ok.
    My motto: Implicit imperative is bad. Implicit or explicit declarative is good. Explicit imperative is ok.


    ---

    If you are using the nixos pinning, use shell.nix pinning on the current hash of your nixpkgs:

    ```
    git -C /nix/nixpkgs rev-parse HEAD
    ```

    This gives you the long form hash:

    ```
    {
    pkgs ? import (fetchTarball https://github.com/NixOS/nixpkgs-channels/archive/8fefa85f5cc95a99c78776fdfb971deff3592cab.tar.gz) {}
    }:
    ```
  9. CMCDragonkai revised this gist Sep 20, 2017. 1 changed file with 0 additions and 1 deletion.
    1 change: 0 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -193,7 +193,6 @@ Here's a full shell.nix:
    echo 'Entering Illuminate Magsurv Environmnet'
    set -v
    mkdir --parent pip_packages
    alias pip="PIP_PREFIX='$(pwd)/pip_packages' \pip"
    export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"
  10. CMCDragonkai revised this gist Sep 20, 2017. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -193,7 +193,8 @@ Here's a full shell.nix:
    echo 'Entering Illuminate Magsurv Environmnet'
    set -v
    alias pip="PIP_PREFIX=pip_packages \pip"
    mkdir --parent pip_packages
    alias pip="PIP_PREFIX='$(pwd)/pip_packages' \pip"
    export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"
    set +v
  11. CMCDragonkai revised this gist Sep 20, 2017. 1 changed file with 26 additions and 0 deletions.
    26 changes: 26 additions & 0 deletions developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -175,4 +175,30 @@ However I haven't found a way to get the name of the `python3.5` automatically.

    Now you can use pip and everything else naturally! Remember to add `pip_packages` to your `.gitignore`. Also remember to delete the cache if you have problems `~/.cache/pip`.

    Here's a full shell.nix:

    ```
    { pkgs ? import <nixpkgs> {} }:
    with pkgs;
    python35Packages.buildPythonApplication {
    name = "illuminate-magsurv";
    buildInputs = (with python35Packages; [
    numpy
    (matplotlib.override { enableQt = true; })
    gdal
    scikitimage
    tensorflowWithCuda
    ]);
    shellHook = ''
    echo 'Entering Illuminate Magsurv Environmnet'
    set -v
    alias pip="PIP_PREFIX=pip_packages \pip"
    export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"
    set +v
    '';
    }
    ```

    My motto: Implicit imperative is bad. Implicit or explicit declarative is good. Explicit imperative is ok.
  12. CMCDragonkai revised this gist Sep 20, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -173,6 +173,6 @@ export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"

    However I haven't found a way to get the name of the `python3.5` automatically. At any case this would alway be fixed to your `python35Packages` set, and you will just have to change the name accordingly.

    Now you can use pip and everything else naturally! Remember to add `pip_packages` to your `.gitignore`.
    Now you can use pip and everything else naturally! Remember to add `pip_packages` to your `.gitignore`. Also remember to delete the cache if you have problems `~/.cache/pip`.

    My motto: Implicit imperative is bad. Implicit or explicit declarative is good. Explicit imperative is ok.
  13. CMCDragonkai revised this gist Sep 20, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -173,6 +173,6 @@ export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"

    However I haven't found a way to get the name of the `python3.5` automatically. At any case this would alway be fixed to your `python35Packages` set, and you will just have to change the name accordingly.

    Now you can use pip and everything else naturally!
    Now you can use pip and everything else naturally! Remember to add `pip_packages` to your `.gitignore`.

    My motto: Implicit imperative is bad. Implicit or explicit declarative is good. Explicit imperative is ok.
  14. CMCDragonkai revised this gist Sep 20, 2017. No changes.
  15. CMCDragonkai revised this gist Sep 20, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -151,7 +151,7 @@ You need a shell.nix:
    with pkgs;
    python35Packages.buildPythonApplication {
    name = "myawesomepythonapp";
    buildInputs = [ zlib ] ++ with python35Packages; [ numpy requests2];
    buildInputs = [ zlib ] ++ with python35Packages; [ numpy requests2 ];
    }
    ```

  16. CMCDragonkai revised this gist Sep 20, 2017. 1 changed file with 39 additions and 1 deletion.
    40 changes: 39 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -137,4 +137,42 @@ Note that for choosing between different architectures, a nice trick is to use a

    To wipe out a nix-build result without leaving the data in /nix/store, use `nix-store --delete --ignore-liveness $(readlink ./result) && rm ./result`.

    If you just delete the result symlink, then the only way to get rid of it is to use `nix-store --gc`. You can also show all the dead paths to be deleted by using `nix-store --print-dead`.
    If you just delete the result symlink, then the only way to get rid of it is to use `nix-store --gc`. You can also show all the dead paths to be deleted by using `nix-store --print-dead`.

    Python
    ------

    I found the best way to develop in Python with nix-shell.

    You need a shell.nix:

    ```
    { pkgs ? import <nixpkgs> {} }:
    with pkgs;
    python35Packages.buildPythonApplication {
    name = "myawesomepythonapp";
    buildInputs = [ zlib ] ++ with python35Packages; [ numpy requests2];
    }
    ```

    Note the difference between a python application and a python library. An application's main consumption isn't meant to be a library form. Whereas a package is meant to be submitted to pip and be consumed like a library.

    Now what happens is that we have a shell environment with system `zlib` available, while `numpy` and `requests2` come from Nix. These are all available within a python extended `stdenv.mkDerivation` environment. So you can do everything that `mkDerivation` does, but you also have all the nice package hooks of python, and if you use this for `nix-build` it will run some functions relating to `setup.py` that will test things and build wheels... etc.

    Next you need a `setup.py`, and just fill it out normally. Optionally you can have a `requirements.txt`. In fact you should always have this with the corresponding hashes. Think of the `requirements.txt` as the a package lock. Which locks the dependencies and the hashes you expect. This makes the end result work across non-nix environments.

    Now you can use `pip`. There's a trick to make pip behave like npm or composer. You need to use the `--prefix` option. This is the correct option, `--root` isn't and neither is `--target`. You can now do things like `pip install --prefix ./pip_packages -r requirements.txt`. Or you can specify dependencies imperatively. Pip will know about the `$PYTHONPATH` which is already filled with your nix installed python packages, and any specified package imperatively or via `requirements.txt` that are compatible with the nix installed python packages will not be further installed. Any non-compatible or missing dependencies will now be installed into the `./pip_packages` folder. However if it detects a conflict between a nix dependency and your own, after installing the dependency into `./pip_packages` it will try to uninstall the nix installed dependency. This will simply fail because nix dependencies are immutable. And this is a good thing. It forces you to make sure that either all of your application dependencies can be satisfied by your nix shell.nix specification or that it will give an error. However... you can force pip to ignore and make sure to install whatever version you want, by using the `--ignore-installed` option (shortcut `-I`). This will force install and not bother trying to uninstall things. You shouldn't really need to do this.

    Finally now that you have all the dependencies available, either via the nix or via the `./pip_packages`. You can now set this as part of your `$PYTHONPATH`. It's important to be able to set this as a prepend. So any packages found there will take precedence over nix installed dependencies. It's also what makes it possible to use `pip freeze` and `pip uninstall` that will first prefer listing or uninstalling the `./pip_packages`.

    But the way to do this is a bit more complicated. You need to do this:

    ```
    export PYTHONPATH="$(pwd)/pip_packages/lib/python3.5/site-packages:$PYTHONPATH"
    ```

    However I haven't found a way to get the name of the `python3.5` automatically. At any case this would alway be fixed to your `python35Packages` set, and you will just have to change the name accordingly.

    Now you can use pip and everything else naturally!

    My motto: Implicit imperative is bad. Implicit or explicit declarative is good. Explicit imperative is ok.
  17. CMCDragonkai revised this gist Aug 12, 2017. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -133,4 +133,8 @@ Note that for choosing between different architectures, a nice trick is to use a
    "i686-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_i386.tar.gz";
    "x86_64-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_qt5.amd64.tar.gz";
    }."${stdenv.system}";
    ```
    ```

    To wipe out a nix-build result without leaving the data in /nix/store, use `nix-store --delete --ignore-liveness $(readlink ./result) && rm ./result`.

    If you just delete the result symlink, then the only way to get rid of it is to use `nix-store --gc`. You can also show all the dead paths to be deleted by using `nix-store --print-dead`.
  18. CMCDragonkai revised this gist Aug 1, 2017. 1 changed file with 10 additions and 1 deletion.
    11 changes: 10 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -124,4 +124,13 @@ Now everything should go fine, UNLESS there are side-effectful things you still

    But other things can happen to like QT plugins that may be conflicting.

    Anyway the lack of purity by default is what makes this a difficult process.
    Anyway the lack of purity by default is what makes this a difficult process.

    Note that for choosing between different architectures, a nice trick is to use attribute keys with architecture targets, and choosing them using `stdenv.system`. For example:

    ```
    url = {
    "i686-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_i386.tar.gz";
    "x86_64-linux" = "http://get.code-industry.net/public/master-pdf-editor-${version}_qt5.amd64.tar.gz";
    }."${stdenv.system}";
    ```
  19. CMCDragonkai revised this gist Jul 31, 2017. 1 changed file with 29 additions and 1 deletion.
    30 changes: 29 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -96,4 +96,32 @@ Now all services connect over the unix domain socket. Each has different ways of

    Although this is only available on unix systems, windows will have to suck it.

    Note that SSH can forward over unix domain sockets without the need of socat.
    Note that SSH can forward over unix domain sockets without the need of socat.

    Deploying an unfree no source package that uses the GUI can be pretty complicated due to all the QT dependencies.

    We take the example of https://code-industry.net/free-pdf-editor/.

    First we download the correct architecture for our system. Extract it and check the insides. You can try ldd on the executable however it won't work, because the loader is set incorrectly. So we need to first set the path to the loader. By using `patchelf`. Find an appropriate interpreter: `ls /nix/store/*-glibc-*/lib/ld-linux*so`. And then set it on the binary: `patchelf --set-interpreter /nix/store/68sa3m89shpfaqq1b9xp5p1360vqhwx6-glibc-2.25/lib/ld-linux-x86-64.so.2 ./masterpdfeditor4`. Now you can use `ldd` to check all the shared objectis it is relying on.

    At this point there is domain knowledge required to find out which nix dependencies correspond to the shared libraries required. But some common ones are like pthread come from the glibc package. One of the most common will probably be the C libraries, which can come from stdenv.cc.cc. Then qt libraries is pretty complicated as that can come in a set. Finally alot of the x11 libraries will come from xorg.

    All these will need to be encoded into the executable using `patchelf` again, but this time via the RPATH setting.

    Furthermore stripping may need to be disabled, as it can break the executable.

    Beyond dynamic libraries, the executable may call external resources, these may be supplied via a relative path. In the tar.gz we have, we find folders that are relative to the executable, so when you call the executable, it will be looking for these resources relative to its position, this means we may need to copy these directories to the bin folder of the out directory of the build nix package. Finding ways to rewrite the paths may be possible with a textual program, but not really with a binary program.

    We also want a sha256 hash of the executable, so we can use `nix-prefetch-url` on the link.

    Finally you need to write the default.nix file. At first I tried to create one similar to an out-of-tree default.nix and just use nix-shell and nix-build to try to build it. In order to detect further paths that is not being opened try using strace like `strace -e open ./result/... | grep ENOENT`. Which will show you open attempts that fail, now not all attempts are bad, because of the way RPATH works, it may try to open libraries in every RPATH and include path available. So you usually are looking for paths that it may try to open relative to the binary location.

    In the default.nix that you're writing, you primarily only need to override the `installPhase`, since this is already a prebuilt executable, so there's not much other phases you need to change here.

    In the dev workflow, this default.nix is going to be different from the default.nix for an out-of-tree package, since now we need to be harmonious with the rest of the tree, and we need to add our package to the pkgs/top-level/all-packages.nix. Most importantly after you add this to your development version of nixpkgs (remember not the /nix/nixpkgs), you can just test the installation by doing `nix-env -f ./dev/nixpkgs/default.nix -i masterpdfeditor`, and this will try to install it from the dev tree of nixpkgs.

    Now everything should go fine, UNLESS there are side-effectful things you still haven't encapsulated. Examples include GL libraries which usually come from `LD_LIBRARY_PATH`, as these may be supplied by your graphics card. So these will usually not be specified inside the default.nix inside the nixpkgs, unless you really need to, usually as a backupo graphics library that uses the mesa drivers.

    But other things can happen to like QT plugins that may be conflicting.

    Anyway the lack of purity by default is what makes this a difficult process.
  20. CMCDragonkai revised this gist Jul 19, 2017. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -94,4 +94,6 @@ mysql --socket='./.mysql/mysql.sock'

    Now all services connect over the unix domain socket. Each has different ways of doing this. Now if only HTTP services could also do this, now local development would be super easy. Well browsers need to support unix domain socket access, while curl and like do support it, and you won't need the container for isolating networks, or network namespace at all.

    Although this is only available on unix systems, windows will have to suck it.
    Although this is only available on unix systems, windows will have to suck it.

    Note that SSH can forward over unix domain sockets without the need of socat.
  21. CMCDragonkai revised this gist Jul 19, 2017. 1 changed file with 14 additions and 1 deletion.
    15 changes: 14 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -81,4 +81,17 @@ For Nodejs, packages should just be installed via npm, when developing a library

    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix. To expand on this, if you have networked services, it is easier to isolate it from the perspective from just using nix-shell, if you instead use unix domain sockets and put them into project specific hidden folders. For example `./.mysql` for the mysql directory and `./.mysql/mysql.sock` for the mysql socket. Then set the necessary environment variables to make sure all mysql commands will be using it.

    To run a command directly when launching into shell.nix. You should be using `shellHook`. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the `shellHook`. This is also where you can set environment variables inside here.
    To run a command directly when launching into shell.nix. You should be using `shellHook`. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the `shellHook`. This is also where you can set environment variables inside here.

    This is how you setup a local mysql:

    ```sh
    rm -rf .mysql && mkdir .mysql
    mysqld --datadir="$(pwd)/.mysql" --initialize-insecure
    mysqld --datadir="$(pwd)/.mysql" --socket="$(pwd)/.mysql/mysql.sock" --skip-networking &
    mysql --socket='./.mysql/mysql.sock'
    ```

    Now all services connect over the unix domain socket. Each has different ways of doing this. Now if only HTTP services could also do this, now local development would be super easy. Well browsers need to support unix domain socket access, while curl and like do support it, and you won't need the container for isolating networks, or network namespace at all.

    Although this is only available on unix systems, windows will have to suck it.
  22. CMCDragonkai revised this gist Jul 17, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -81,4 +81,4 @@ For Nodejs, packages should just be installed via npm, when developing a library

    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix. To expand on this, if you have networked services, it is easier to isolate it from the perspective from just using nix-shell, if you instead use unix domain sockets and put them into project specific hidden folders. For example `./.mysql` for the mysql directory and `./.mysql/mysql.sock` for the mysql socket. Then set the necessary environment variables to make sure all mysql commands will be using it.

    To run a command directly when launching into shell.nix. You should be using `shellHook`. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the `shellHook`.
    To run a command directly when launching into shell.nix. You should be using `shellHook`. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the `shellHook`. This is also where you can set environment variables inside here.
  23. CMCDragonkai revised this gist Jul 17, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -79,6 +79,6 @@ For PHP, the extension support hasn't been fully figured out. While composer pac

    For Nodejs, packages should just be installed via npm, when developing a library, but use node2nix when deploying an application. Development can use npm or node2nix, just remember, this means an extra overhead when developing, while making it harder to monkey patch.

    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix.
    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix. To expand on this, if you have networked services, it is easier to isolate it from the perspective from just using nix-shell, if you instead use unix domain sockets and put them into project specific hidden folders. For example `./.mysql` for the mysql directory and `./.mysql/mysql.sock` for the mysql socket. Then set the necessary environment variables to make sure all mysql commands will be using it.

    To run a command directly when launching into shell.nix. You should be using `shellHook`. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the `shellHook`.
  24. CMCDragonkai revised this gist Jul 17, 2017. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -79,4 +79,6 @@ For PHP, the extension support hasn't been fully figured out. While composer pac

    For Nodejs, packages should just be installed via npm, when developing a library, but use node2nix when deploying an application. Development can use npm or node2nix, just remember, this means an extra overhead when developing, while making it harder to monkey patch.

    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix.
    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix.

    To run a command directly when launching into shell.nix. You should be using `shellHook`. Make sure these commands are idempotent. They will be running in the launched bash shell, so that means the environment will already be built. These commands will be special only to nix-shell, nix-build will not execute the `shellHook`.
  25. CMCDragonkai revised this gist Jul 12, 2017. 1 changed file with 13 additions and 1 deletion.
    14 changes: 13 additions & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -67,4 +67,16 @@ deactivate (or straight to exit is also ok, since you are in a nix-shell)
    exit
    ```

    Another really important factor is whether your shell.nix is reproducible. If you don't specify the content hash of the nixpkgs package set you're using, the shell.nix is not truly reproducible, at least it's not a complete specification. It's an unapplied function, that is still applied to whatever nixpkgs you have in the environment. This is also the key difference between a project specific default.nix and the default.nix you contribute to nixpkgs, in the one contributed to nixpkgs, that would be an unapplied function, using the package set for harmonious operation. While the one exposed by your repository should be content addressed and totally specified and pinned to a nixpkgs content hash. For shell.nix this, this might depend on what you care about. For ease of use, leaving unspecified and dependent on your OS is ok, but this might not be good when working in teams, by making sure your shell.nix pins a nixpkgs content hash, everybody on the team is working on the same package set.
    Another really important factor is whether your shell.nix is reproducible. If you don't specify the content hash of the nixpkgs package set you're using, the shell.nix is not truly reproducible, at least it's not a complete specification. It's an unapplied function, that is still applied to whatever nixpkgs you have in the environment. This is also the key difference between a project specific default.nix and the default.nix you contribute to nixpkgs, in the one contributed to nixpkgs, that would be an unapplied function, using the package set for harmonious operation. While the one exposed by your repository should be content addressed and totally specified and pinned to a nixpkgs content hash. For shell.nix this, this might depend on what you care about. For ease of use, leaving unspecified and dependent on your OS is ok, but this might not be good when working in teams, by making sure your shell.nix pins a nixpkgs content hash, everybody on the team is working on the same package set.

    It turns out that python infrastructure on nixpkgs, automatically sets up a development environment. This is called develop mode. This means you can start using pip right away, which means you do not need an extra virtualenv. See this: https://www.johbo.com/2016/using-pip-inside-of-nix-shell.html

    This means it should be easy to mix-and-match python dependencies brought from nixpkgs, and python dependencies brought in from pip.

    Note that this doesn't necessarily exist for other languages.

    For PHP, the extension support hasn't been fully figured out. While composer packages would be installed similar to npm in nodejs.

    For Nodejs, packages should just be installed via npm, when developing a library, but use node2nix when deploying an application. Development can use npm or node2nix, just remember, this means an extra overhead when developing, while making it harder to monkey patch.

    Nix-shell is just really about filesystem path dependencies, not really OS virtualisation. It's language generic virtualenv and dotenv combined with a language generic package manager. But what if you want more virtualisation? What if you need to test distributed services, each with port allocations, DNS, databases, firewalls... etc? At this point more infrastructure is needed to scale the development environment. I need to investigate the usage of nixos containers and docker containers, and how they can be launched from a single shell.nix.
  26. CMCDragonkai revised this gist Jul 9, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -56,7 +56,7 @@ Some languages make this easy, such as npm, because it already installs all the
    So this is how I do it in Python:

    ```
    create shell.nix with python, pip and virtualenv (and also any system dependencies that certain dependencies need, this will require your hard work unless there's a tool that automates some of the derivations)
    create shell.nix with python, pip, setuptools and virtualenv (and also any system dependencies that certain dependencies need, this will require your hard work unless there's a tool that automates some of the derivations)
    nix-shell (enter the shell monad)
    virtualenv .venv
    make sure .venv is ignored in .gitignore
  27. CMCDragonkai revised this gist Jul 9, 2017. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -58,9 +58,9 @@ So this is how I do it in Python:
    ```
    create shell.nix with python, pip and virtualenv (and also any system dependencies that certain dependencies need, this will require your hard work unless there's a tool that automates some of the derivations)
    nix-shell (enter the shell monad)
    virtualenv venv
    make sure venv is ignored in .gitignore
    . venv/bin/activate
    virtualenv .venv
    make sure .venv is ignored in .gitignore
    . .venv/bin/activate
    pip install whatever
    pip install -r requirements.txt
    deactivate (or straight to exit is also ok, since you are in a nix-shell)
  28. CMCDragonkai revised this gist Jul 9, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -49,7 +49,7 @@ However once everything is done and ready to go, you should be using the full po

    However some languages make this difficult, for example python, since its pip system does not install packages locally.
    This is why you need to pair it up with virtualenv, so it creates a virtual environment for that python and then you can
    run pip on the requirements.txt and continue developing. However you can mix and match this with pip and virtualenv. So for example, you can get gdal from nixpkgs, while also specifying pip and downloading the other dependencies via pip. Pip will actually know about the gdal already installed from nixpkgs, so as long as that's compatible, it should be fine. Note that doing this may require you to use compatible version specifiers, because nixpkgs may not be carrying the version of that package you want. However some language infrastructures encourage and is well developed for shell environments, so it is possible to skip the language package management tool and just use that nix tool, but this workflow is spotty (but it might be better for when you're installing dependencies with non-language dependencies).
    run pip on the requirements.txt and continue developing. However you can mix and match this with pip and virtualenv. So for example, you can get gdal from nixpkgs, while also specifying pip and downloading the other dependencies via pip. Pip will actually know about the gdal already installed from nixpkgs, so as long as that's compatible, it should be fine. Note that doing this may require you to use compatible version specifiers, because nixpkgs may not be carrying the version of that package you want. However some language infrastructures encourage and is well developed for shell environments, so it is possible to skip the language package management tool and just use that nix tool, but this workflow is spotty (but it might be better for when you're installing dependencies with non-language dependencies). Take care to understand this: https://caremad.io/posts/2013/07/setup-vs-requirement/

    Some languages make this easy, such as npm, because it already installs all the packages locally.

  29. CMCDragonkai revised this gist Jul 9, 2017. 1 changed file with 14 additions and 0 deletions.
    14 changes: 14 additions & 0 deletions developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -53,4 +53,18 @@ run pip on the requirements.txt and continue developing. However you can mix and

    Some languages make this easy, such as npm, because it already installs all the packages locally.

    So this is how I do it in Python:

    ```
    create shell.nix with python, pip and virtualenv (and also any system dependencies that certain dependencies need, this will require your hard work unless there's a tool that automates some of the derivations)
    nix-shell (enter the shell monad)
    virtualenv venv
    make sure venv is ignored in .gitignore
    . venv/bin/activate
    pip install whatever
    pip install -r requirements.txt
    deactivate (or straight to exit is also ok, since you are in a nix-shell)
    exit
    ```

    Another really important factor is whether your shell.nix is reproducible. If you don't specify the content hash of the nixpkgs package set you're using, the shell.nix is not truly reproducible, at least it's not a complete specification. It's an unapplied function, that is still applied to whatever nixpkgs you have in the environment. This is also the key difference between a project specific default.nix and the default.nix you contribute to nixpkgs, in the one contributed to nixpkgs, that would be an unapplied function, using the package set for harmonious operation. While the one exposed by your repository should be content addressed and totally specified and pinned to a nixpkgs content hash. For shell.nix this, this might depend on what you care about. For ease of use, leaving unspecified and dependent on your OS is ok, but this might not be good when working in teams, by making sure your shell.nix pins a nixpkgs content hash, everybody on the team is working on the same package set.
  30. CMCDragonkai revised this gist Jul 9, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion developing_with_nix.md
    View file
    Original file line number Diff line number Diff line change
    @@ -49,7 +49,7 @@ However once everything is done and ready to go, you should be using the full po

    However some languages make this difficult, for example python, since its pip system does not install packages locally.
    This is why you need to pair it up with virtualenv, so it creates a virtual environment for that python and then you can
    run pip on the requirements.txt and continue developing. However you can mix and match this with pip and virtualenv. So for example, you can get gdal from nixpkgs, while also specifying pip and downloading the other dependencies via pip. Pip will actually know about the gdal already installed from nixpkgs, so as long as that's compatible, it should be fine. Note that doing this may require you to use compatible version specifiers, because nixpkgs may not be carrying the version of that package you want.
    run pip on the requirements.txt and continue developing. However you can mix and match this with pip and virtualenv. So for example, you can get gdal from nixpkgs, while also specifying pip and downloading the other dependencies via pip. Pip will actually know about the gdal already installed from nixpkgs, so as long as that's compatible, it should be fine. Note that doing this may require you to use compatible version specifiers, because nixpkgs may not be carrying the version of that package you want. However some language infrastructures encourage and is well developed for shell environments, so it is possible to skip the language package management tool and just use that nix tool, but this workflow is spotty (but it might be better for when you're installing dependencies with non-language dependencies).

    Some languages make this easy, such as npm, because it already installs all the packages locally.