.. title: Essential pkgsrc - The Missing Mini Handbook .. slug: essential-pkgsrc-the-missing-mini-handbook .. date: 2017-03-25 19:29:16 UTC .. updated: 2018-05-16 20:30:54 UTC .. tags: macos, netbsd, pkgsrc, python .. category: .. link: .. description: .. type: text pkgsrc is a cross operating system package manager. It supports -- among many others -- NetBSD, Minix, SmartOS, Linux, and macOS. I like it because of this portability. It also has the additional, and I would say the best, benefit of being installed in the home directory and run completely without needing root access. I also like that I don't have to depend on binary packages built by someone else, say Joyent, although there's absolutely nothing wrong with it. Finally, it provides a large number of different packages. I have never encountered a package that I needed but was not available. In short pkgsrc is a portable, featureful, and flexible package manager. What's not to like? pkgsrc can sometimes be a little behind native package managers, such as MacPorts on macOS, but it catches up quickly. For my use case -- getting access to multiple versions of Python -- it works well enough if I closely follow its trunk branch. There's generally good and detailed documentation available for pkgsrc but an introductory guide that pulled in some essential starter information was lacking. This guide fills that void by making it easy to get started with pkgsrc and learn about some of its core concepts. Thus, I dubbed it the mini handbook or the missing starter handbook. .. TEASER_END: Read more I wrote this guide by running pkgsrc on macOS. The instructions should work on Linux as well but may need minor modifications. I wrote a similar guide for Linux (`pkgsrc on Linux - Quickstart Guide `_) a while ago. This guide is an updated and more general version of it. .. contents:: Install ------- You need ``git`` to clone the pkgsrc repo from GitHub. I prefer a shallow clone to save disk space. Newer ``git`` versions can pull and push shallow clones just fine. I also prefer *trunk* branch since it has the latest goodies available. If you prefer stable branches then substitue *trunk* below with a quarterly release such as *pkgsrc_2016Q4*. :: $ git clone https://github.com/NetBSD/pkgsrc.git -b trunk --single-branch ~/pkgsrc I believe you may need build tools such as a compiler. Make sure you have these tools installed just in case. Unlike the official install I prefer an *unprivileged* install because it gets installed in my home directory without affecting the rest of the system. As `The pkgsrc guide `_ puts it, In unprivileged mode in contrast to privileged one all programs are installed under one particular user and cannot utilise privileged operations (packages don't create special users and all special file permissions like setuid are ignored). If you're running on Linux (e.g. Ubuntu), run these steps first. Reason is an error message described on pkgsrc wiki (`Shell's echo command is not BSD-compatible `_). :: $ export SH=/bin/bash Then run the bootstrap. :: $ cd ~/pkgsrc/bootstrap $ ./bootstrap --unprivileged Environment Variables --------------------- Insert these lines in ``~/.profile`` (macOS or Ubuntu) if they're not already present. Set the PATH variable. I prefer to give priority to pkgsrc-installed packages. :: export PATH PATH="${HOME}/pkg/sbin:${HOME}/pkg/bin:${PATH}" Set the MANPATH variable to access man pages that come with packages installed with pkgsrc. :: export MANPATH MANPATH="${HOME}/pkg/man:${MANPATH}" `NetBSD wiki `_ recommends these settings for Unicode to work. :: export LC_CTYPE LC_CTYPE="en_US.UTF-8" export LC_COLLATE LC_COLLATE="C" export LC_TIME LC_TIME="C" export LC_NUMERIC LC_NUMERIC="C" export LC_MONETARY LC_MONETARY="C" export LC_MESSAGES LC_MESSAGES="en_US.UTF-8" export LC_ALL LC_ALL="" Don't forget to source the file. :: $ source ~/.profile # macOS or Ubuntu mk.conf ------- The bootstrap process creates *~/pkg/etc/mk.conf*. `mk.conf `_ is a configuration file used when building packages. Familiarize yourself with it as many problems you will face as a novice will be solved by editing it. You are encouraged to keep the pkgsrc directory clean by moving all work directories and `distfiles `_ in separate paths, such as *~/pkg/usr/work* and *~/pkg/usr/distfiles* respectively. Create these directories. :: $ mkdir -p ~/pkg/usr/{work,distfiles} `Add these two lines `_ to *mk.conf* before the last line which most likely will be *.endif # end pkgsrc settings*. :: WRKOBJDIR=${HOME}/pkg/usr/work DISTDIR=${HOME}/pkg/usr/distfiles Python ------ Since the pkgsrc install in this guide is custom and *unprivileged* you have to build packages from source. If you prefer not to do that then you must follow the official instructions to install pkgsrc. You can then use ``pkgin`` to manage packages. Python 3.6 is the latest version available at the time of writing. It's available in *trunk* branch of pkgsrc. Edit *~/pkg/etc/mk.conf* and insert the following line before the last line which most likely will be *.endif # end pkgsrc settings*. This configures the build system to always use Python 3.6 as the default version. :: PYTHON_VERSION_DEFAULT=36 To install packages from source navigate to the directory of the package and run ``bmake``. Here we install Python 3.6. :: $ cd ~/pkgsrc/lang/python36 $ bmake install Confirm ``python3.6`` was installed and your *PATH* is setup correctly. :: $ which python3.6 ~/pkg/bin/python3.6 Installing ``pip`` is a little trickier. There's *~/pkgsrc/devel/py-pip* but it installs only one version of ``pip`` depending on how you set up *PYTHON_VERSION_DEFAULT* above. In our case it will install ``pip3.6``. We'll discuss how to install multiple ``pip`` versions in the next section titled *pkg_alternatives*. :: $ cd ~/pkgsrc/devel/py-pip $ bmake install Confirm ``pip3.6`` was installed and your *PATH* is setup correctly. :: $ which pip3.6 ~/pkg/bin/pip3.6 Last thing we need is ``virtualenv-3.6``. :: $ cd ~/pkgsrc/devel/py-virtualenv $ bmake install Confirm ``virtualenv-3.6`` was installed and your *PATH* is setup correctly. :: $ which virtualenv-3.6 ~/pkg/bin/virtualenv-3.6 We have all the things we need to create a virtualenv and start working with Python 3.6. :: $ python3.6 Python 3.6.0 (default, Mar 25 2017, 00:15:25) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import pip >>> import virtualenv >>> ^D pkg_alternatives ---------------- Just like `Debian `_ pkgsrc has an `alternatives system `_. But what is an "alternatives system"? It is a framework that allows multiple packages providing similar functionality to be installed concurrently (by removing files with common names), and then using a utility to set up those common names with symlinks to the preferred program. In other words, we can install, for example, ``pip2.7`` and ``pip3.6`` and set ``pip`` to point (symlink) to ``pip3.6``. We can still use ``pip2.7`` and ``pip3.6`` independent of each other but when we use ``pip`` it will run ``pip3.6``. So let's try this and install Python 2.7 and ``pip2.7``. :: $ cd ~/pkgsrc/lang/python27 $ bmake install When I initially tried to install an alternative ``pip`` I ran into the same problem as reported here: `Can't have "pip" packages for python 2.7 and 3.4 installed at the same time `_. The solution provided by Mark Davies was Don't install pip and use the ALTERNATES mechanism to create it. See the similar changes I did to the py-docutils and py-sphinx packages last night. I asked the pkgsrc-users mailing list on how to do it. `Thanks to Adam `_ for pointing me to the solution. The trick is to temporarily override *PYTHON_VERSION_DEFAULT* when you run ``bmake install`` like so. :: $ cd ~/pkgsrc/devel/py-pip $ bmake install PYTHON_VERSION_DEFAULT=27 => Bootstrap dependency digest>=20010302: found digest-20160304 ===> Checking for vulnerabilities in py27-pip-9.0.1 ===> Installing binary package of py27-pip-9.0.1 pkg_add: no pkg found for '~/pkg/usr/work/devel/py-pip/work/work/.packages/py27-pip-9.0.1.tgz', sorry. pkg_add: 1 package addition failed *** Error code 1 Stop. bmake[2]: stopped in ~/pkgsrc/devel/py-pip *** Error code 1 Stop. bmake[1]: stopped in ~/pkgsrc/devel/py-pip *** Error code 1 Stop. bmake: stopped in ~/pkgsrc/devel/py-pip Uh oh. Where did this error come from? From what I understand we need to clean the work directories from when we built ``pip3.6``. Otherwise the install process looks for a pre-built package to install. In this case it finds *py36-pip* but not *py27-pip* and fails. Fair warning, this is just my very flawed understanding. I can fix the symptom but can't explain the cause. The fix is to clean all work directories and run ``bmake install`` again. :: $ rm -rf ~/pkg/usr/work/* $ cd ~/pkgsrc/devel/py-pip $ bmake install PYTHON_VERSION_DEFAULT=27 A better way to avoid the problem in the first place is to always run ``bmake install clean clean-depends`` which will build and install the package and clean the work directories for the package and its dependencies. The down side is that the next time you build the package it will re-do all the work. Similar to how we override *PYTHON_VERSION_DEFAULT* with *py27* we can use other Python versions like *py35* and *py34*. Let's look at what versions of ``pip`` we have installed. :: $ ls -l ~/pkg/bin/pip?.? ~/pkg/bin/pip2.7* ~/pkg/bin/pip3.6* To run say ``pip3.6`` when you run ``~/pkg/bin/pip`` you need to use the ``pkg_alternatives`` tool. Install it first. :: $ cd ~/pkgsrc/pkgtools/pkg_alternatives $ bmake install Let's list all packages that provide alternatives. :: $ pkg_alternatives list pkg_alternatives: looking for alternatives in `~/pkg/pkgdb' py27-pip-9.0.1 py36-pip-9.0.1 py36-virtualenv-15.1.0 python27-2.7.13nb1 python36-3.6.0nb2 Let's ensure ``pip`` points to ``pip3.6``. :: $ pkg_alternatives manual py36-pip pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/pip' Similarly we can use ``python3.6`` as default Python for other binaries. :: $ pkg_alternatives manual python36 pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/2to3' pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/pydoc3' pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/python' pkg_alternatives: modifying configuration from `~/pkg/etc/pkg_alternatives~/pkg/bin/python3' As always read the man page of ``pkg_alternatives`` for more information. :: $ man -S 8 pkg_alternatives Vulnerabilities --------------- pkgsrc makes it simple to `check for vulnerabilities `_ in installed packages. Fetch the latest list of known vulnerabilities. :: $ pkg_admin -K ~/pkg/pkgdb fetch-pkg-vulnerabilities Audit all installed packages for known vulnerabilities. :: $ pkg_admin -v audit No vulnerabilities found Sometimes packages will not build because there are known vulnerabilities in them or their dependencies. For example, :: $ cd ~/pkgsrc/devel/ncurses $ bmake install => Bootstrap dependency digest>=20010302: found digest-20160304 ===> Checking for vulnerabilities in ncurses-6.1 Package ncurses-6.1 has a null-pointer-dereference vulnerability, see https://nvd.nist.gov/vuln/detail/CVE-2018-10754 ERROR: Define ALLOW_VULNERABLE_PACKAGES in mk.conf or IGNORE_URL in pkg_install.conf(5) if this package is absolutely essential. *** Error code 1 Stop. bmake: stopped in /home/codeghar/pkgsrc/devel/ncurses Sometimes you have no choice but to install the package despite the vulnerability. I prefer to ignore specific CVEs in *~/pkg/etc/pkg_install.conf* file rather than ignore *all* CVEs in *~/pkg/etc/mk.conf* file. In this example, the file would look like, :: $ more ~/pkg/etc/pkg_install.conf IGNORE_URL=https://nvd.nist.gov/vuln/detail/CVE-2018-10754 Now you can build the package successfully. Upgrade Single Package ---------------------- Follow these steps when you want to update a single package, say Python 3.6. :: $ cd ~/pkgsrc/lang/python36 $ bmake update clean clean-depends Upgrade All Packages -------------------- ``pkg_rolling-replace`` removes packages before installing them in a safer way. Read more on `how to upgrade packages `_. :: $ cd ~/pkgsrc/pkgtools/pkg_rolling-replace $ bmake install clean clean-depends Add *PKGSRCDIR* variable to *~/pkg/etc/mk.conf* just before the last line which most likely will be *.endif # end pkgsrc settings*. :: PKGSRCDIR=${HOME}/pkgsrc Pull any upstream changes. :: $ cd ~/pkgsrc $ git pull List available updates. Note the use of ``-n`` flag. :: $ pkg_rolling-replace -u -n -v Apply available updates. Note the absence of ``-n`` flag. :: $ pkg_rolling-replace -u -v List Installed Packages ----------------------- Get a list of all installed packages with ``pkg_info``. The ``-u`` flag does not list any dependencies. I prefer this format over using ``-a`` flag. :: $ pkg_info -u Read ``pkg_info`` man page for more information. :: $ man -S 1 pkg_info Delete Package -------------- Use ``pkg_delete`` to remove a package. :: $ pkg_delete py27-pip You can uninstall a package and all its dependencies that are not needed by any other package with the ``-R`` flag. :: $ pkg_delete -R py27-pip