PKGSRC

When you receive a vanilla Accelerator from Joyent, all the software available for you to use lives under /opt/local on the file system, and has been built using pkgsrc. Pkgsrc is the centralized package management system originally developed for the NetBSD operating system, but currently available for multiple UNIX-like operating systems. It is one of the methods available to you for installing open-source / freeware software packages on your Accelerator.

pkgsrc is the official packaging system for Accelerators. All Accelerators deployed as of March 22, 2008 are built using pkgsrc, not Blastwave. Previous Accelerators were built using Blastwave Packages.

Joyent runs building and testing systems, builds binary packages using pkgsrc for all software that are available at a binary package repository, and uses these binary packages to create Accelerator templates, to clone for deployment to customer. This means that when you need something that's not installled by default on an Accelerator, you do not have to resort to building from source (although you can, of course), but can simply use our binary package repository and/or employ pkgsrc to build your own.

Where things live

Some things worth remembering:

/opt/local - This is where all software is installed (binaries, libraries, configuration files, supporting files, examples, documentation etc.)
/opt/local/etc - Configuration files for software installed.
/opt/local/share/examples - Example configuration files. When you add a new package, sample configuration files get copied here, and relevant files are then cloned under /opt/local/etc. If you make any changes to the files under /opt/local/etc, they are no longer sync'ed when you remove/re-add the package (i.e. you never lose any customization made to /opt/local/etc). On the other hand, if you break anything under /opt/local/etc, you can look under /opt/local/share/examples for the stock files.
/var/db/pkg* - Two directories there, which hold the essential package database. If you lose these two, you'll no longer be able to use any pkgsrc package management tools, as the notion of what's installed and which package depends on which, will be lost.

There is one notable exception to the example/config files rule outlined above. While most config files are provided by pkgsrc, Joyent provides completely different configuration files for some software that comes pre-installed on Accelerators (e.g. Apache and MySQL). These are not currently pre-stored under /opt/local/share/examples, as we are currently working out the best way to reconcile this with the default config files being installed by pkgsrc.

Releases

Joyent follows pkgsrc's native quarterly release scheme ('2009Q1', '2009Q1' etc.), and generally rebuilds the repository for each new release published. The way the pkgsrc community updates packages works as follows:

All changes are applied to the current (trunk) CVS branch of the pkgsrc meta tree. Joyent does not use the HEAD branch in general.
Security patches are back-ported to the last quarterly release made. Joyent generally attempts to keep its current binary package build-up in sync with the current pkgsrc branch, but updates are being made in the order of weeks rather than days.
When the (more or less) quarterly interval comes up, the current branch is frozen and becomes the new quarterly release. For instance, 2009Q2 was released in early July, and 2009Q3 is to be expected some time in October. When a new release is made, Joyent starts rebuilding their package repository, and may cut a new Accelerator template based on the updated package set.

The important part is that only the current release is getting any security patches. Older releases are never updated any more, which is why staying on top of the latest release is crucial for keeping a secure setup. Joyent may though backport some changes to the previous binary package sets, if really needed.

While you do not have to strictly adhere to the release cycle when updating packages, very much often deep system libraries (e.g. openssl, db4 or libxml2) are bumped when launching a new release, so you may be forced to update half of your software world by pulling in more recent versions of other packages, to match what they were build against. Hence depending on the exact situation and your needs, you should choose between updating particular packages, and updating your entire Accelerator to a new release.

How you do tell which release your Accelerator is populated with? On templates made with 2007Q4 and 2008Q2, you'll find the PKG_PATH variable set under /root/.profile, on Facebook Accelerators (2007Q3) under /root/.bash_profile. This means that when logged in as root, you can just run 'echo $PKG_PATH' to find out. On templates made with 2009Q1 and on, you'll find the PKG_PATH variable defined in /opt/local/etc/pkg_install.conf instead.

Historically, we used 2007Q3 for Facebook Accelerators, 2007Q4 for Accelerators deployed on and after March 22, 2008, followed with 2008Q2 in November 2008, and 2009Q1 in summer 2009.

Package management

The Joyent binary repository for Accelerators lives at:

http://pkgsrc.joyent.com/RELEASE/

Where RELEASE is to be replaced with the pkgsrc release that your Accelerator is currently populated with (see above).

If your Accelerator runs 2009Q1 at least, we strongly recommend using pkgin to add/remove/update installed packages. Pkgin is an apt/yum like tool that was recently coded by a 3rd party programmer for use with pkgsrc.

The default pkgsrc tools to manage packages are pkg_add, pkg_delete, pkg_info (and a few others), lumped into the pkg_install package. They work fine for package addition/removal, but are not generally suited well for package updates, especially when dependencies are involved.

Also, there is a experimental Dapper RSS feed to track down 2009Q1 packages as they are being added/updated:

http://www.dapper.net/services/joyent-pkgsrc-2009Q1

Updating to a newer release

By updating to a newer release, we mean that you update every installed package on your Accelerator to a more recent version that's available under the new release. This is a complex operation and there are various tools that can help you. Our favorite tested tool at this point is pkg_chk.

It makes sense to have a plan before you start updating your entire Accelerator though. The update typically takes a few minutes, and software may become unavailable during the process, so it's a good measure to think of it as 'server maintenance' or 'downtime' even, and plan accordingly. Can you claim a maintenance and shut off all services? Good. If not, letting your users/visitors know at least will work - just in case.

These are the upgrade paths that we tested and documented:

Building from pkgsrc yourself

Any pkgsrc-based Accelerator already has pkgsrc bootstrapped, and the building tools installed. In order to build your own packages (e.g. those that exist in pkgsrc, but are not built by Joyent for whatever reason), you only need to download a pkgsrc meta tree, and start building - pkgsrc will pick up existing packages and will not try to build them again.

You may decide to bootstrap your very own pkgsrc setup (i.e. ignore what's already installed), maybe to create a separate installation under a new prefix, or to get pkgsrc going on a pre-pkgsrc (Blastwave) Accelerator. Note that there is normally no need for this, as our Accelerators already come bootstrapped and ready to use as far as pkgsrc is concerned.

In both cases, see this comprehensive page on using your own pkgsrc.

Reference

pkgsrc documentation

Deprecated pages on this wiki (no longer up-to-date):