Fixing broken port dependencies based on missing files

So I cunningly removed ‘orphaned packages’ in an effort to reclaim much needed space on my system root, only to find that a certain daemon (CouchPotato) failed to run. There could have other consequences of this ‘mindless sweep’, but I had yet to encounter those.

To remedy this, I did:

1
# pkg check -d
  • to list missing dependencies (if any) by file-path.

Then,

1
# find /usr/ports -name pkg-plist -depth 3 -exec grep <name_of_missing_file> {} \+
  • revealing the primary port affected by missing dependencies. It was only then a matter of rebuilding both the primary port and its dependencies with:
1
# portmaster -fR <primary port>

Update to Latest FreeBSD Release (9.2)

For my records, here is an outline of the steps I took to upgrade FreeBSD from 9.1 to 9.2.

# freebsd-update fetch

# freebsd-update install

(and reboot if any changes were made)

Now, fetch the upgrades that are included in the 9.2-RELEASE. The differences in configuration files will need to be merged manually.

# freebsd-update upgrade -r 9.2-RELEASE

# freebsd-update install

Reboot with the new kernel before the non-kernel components receive updates:

# shutdown -r now

After rebooting, install the new user-land components, and upgrade 3rd party packages:

# freebsd-update install

# pkg upgrade

To delete the old (no longer used) system libraries:

# freebsd-update install

Finally, reboot into 9.2-RELEASE

# shutdown -r now

Address Merged Ports

My most recent # portmaster -Da revealed a package conflict: py-setuptools was merged with py-distribute.

From FreshPorts:

2013-03-05
Affects: users of devel/py-setuptools (i.e you)
Author: rm@FreeBSD.org
Reason: devel/py-setuptools was replaced with devel/py-distribute. py-setuptools port will be removed shortly.

To fix this:
# portmaster -o devel/py-distribute devel/py-setuptools

From man 8 portmaster:

-o new port dir in /usr/ports installed port:replace the installed port with a port from a different origin

CouchPotato on FreeBSD

For CouchPotato in FreeBSD, firstly acquaint yourself with my previous blog entry on Sabnzbd+Sickbeard on FreeBSD, and then perform the following additional steps:

At Step 3, fetch CouchPotato’s source code:

# cd /usr/local && git clone git://github.com/RuudBurger/CouchPotato.git couchpotato

Also, it is advisable to change the permission and ownership of the CouchPotato folder:

# chown root:_sabnzbd /usr/local/couchpotato

# chmod g+w /usr/local/couchpotato

At Step 4, add the following to /etc/rc.conf:

1
2
couchpotato_enable="YES"

.. and create /usr/local/etc/rc.d/couchpotato containing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
#!/bin/sh
#
# PROVIDE: couchpotato
# REQUIRE: DAEMON sabnzbd
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# couchpotato_enable (bool): Set to NO by default.
# Set it to YES to enable it.
# couchpotato_user: The user account Couch Potato daemon runs as what
# you want it to be. It uses '_sabnzbd' user by
# default. Do not sets it as empty or it will run
# as root.
# couchpotato_dir: Directory where Couch Potato lives.
# Default: /usr/local/couchpotato
# couchpotato_chdir: Change to this directory before running Couch Potato.
# Default is same as couchpotato_dir.
# couchpotato_pid: The name of the pidfile to create.
# Default is couchpotato.pid in couchpotato_dir.
PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
. /etc/rc.subr
name="couchpotato"
rcvar=${name}_enable
load_rc_config ${name}
: ${couchpotato_enable:="NO"}
: ${couchpotato_user:="_sabnzbd"}
: ${couchpotato_dir:="/usr/local/couchpotato"}
: ${couchpotato_chdir:="${couchpotato_dir}"}
: ${couchpotato_pid:="${couchpotato_dir}/couchpotato.pid"}
WGET="/usr/local/bin/wget" # You need wget for this script to safely shutdown Couch Potato.
HOST="127.0.0.1" # Set Couch Potato address here.
PORT="5000" # Set Couch Potato port here.
SBUSR="" # Set Couch Potato username (if you use one) here.
SBPWD="" # Set Couch Potato password (if you use one) here.
status_cmd="${name}_status"
stop_cmd="${name}_stop"
command="/usr/sbin/daemon"
command_args="-f -p ${couchpotato_pid} python ${couchpotato_dir}/CouchPotato.py ${couchpotato_flags} --quiet"
# Check for wget and refuse to start without it.
if [ ! -x "${WGET}" ]; then
warn "Couchpotato not started: You need wget to safely shut down Couch Potato."
exit 1
fi
# Ensure user is root when running this script.
if [ `id -u` != "0" ]; then
echo "Oops, you should be root before running this!"
exit 1
fi
verify_couchpotato_pid() {
# Make sure the pid corresponds to the Couch Potato process.
pid=`cat ${couchpotato_pid} 2>/dev/null`
ps -p ${pid} | grep -q "python ${couchpotato_dir}/CouchPotato.py"
return $?
}
# Try to stop Couch Potato cleanly by calling shutdown over http.
couchpotato_stop() {
echo "Stopping $name"
verify_couchpotato_pid
${WGET} -O - -q --user=${SBUSR} --password=${SBPWD} "http://${HOST}:${PORT}/home/shutdown/?pid=${pid}" >/dev/null
if [ -n "${pid}" ]; then
wait_for_pids ${pid}
echo "Stopped"
fi
}
couchpotato_status() {
verify_couchpotato_pid && echo "$name is running as ${pid}" || echo "$name is not running"
}
run_rc_command "$1"

NTFS-Write in FreeBSD

NTFS-write in FreeBSD is possible with NTFS-3G from ports. From tuxera.com:

NTFS-3G is a stable, full-featured, read-write NTFS driver for Linux, Android, Mac OS X, FreeBSD, NetBSD, OpenSolaris, QNX, Haiku, and other operating systems. It provides safe handling of the Windows XP, Windows Server 2003, Windows 2000, Windows Vista, Windows Server 2008 and Windows 7 NTFS file systems.

To install NTFS-3G, do:

# portmaster sysutils/fusefs-ntfs

Next, create a symbolic link from the newly-installed /usr/local/bin/ntfs-3g into /usr/sbin/:

# ln -s $(which ntfs-3g) /usr/sbin/mount_ntfs-3g

Modify your /etc/rc.conf to contain:

1
2
fusefs_enable="YES"

Provided you know on which hard-disk (based on make/model/capacity of the device) your NTFS partition resides, you can figure out it’s exact device node in FreeBSD:

# egrep 'ad[0-9]|cd[0-9]|da[0-9]' /var/run/dmesg.boot

ad0: 35303MB  at ata0-master
ad2: 57240MB  at ata1-master
ad4: 286168MB  at ata2-master

My NTFS partition resides on a 300GB WD Velociraptor drive, ad4. To confirm that /dev/ad4 does indeed contain the NTFS partition, do:

# gpart show -p ad4

=>        34  586072064    ad4 GPT  (286G)
          34  586072064  ad4p1 ntfs (286G)

To mount the partition manually, do:

# ntfs-3g /dev/ad4s1 /mnt/windows7

Finally, add the appropriate line to /etc/fstab to have the NTFS partition mounted at the next system boot. For instance, my /etc/fstab contains:

/dev/ad4s1              /mnt/windows7            ntfs-3g rw,late         0       0

Parse /usr/ports/UPDATING for Relevant Issues

Tired of sifting through /usr/ports/UPDATING for notes that affect only those packages installed on your system?

Tired of grepping for issues that may have arisen from/to certain dates?

pkg_updating(1) analyses /usr/ports/UPDATING and does the heavy lifting for you.

To obtain the relevant entries from, say, two weeks ago, do:

# /usr/sbin/pkg_updating -d $(/bin/date -v-2w +%Y%m%d)

Rebuild FreeBSD Ports

To rebuild a specific FreeBSD port and the ports that it depends on, for example, with pydf, do:

# portmaster -fR sysutils/pydf

To rebuild all install FreeBSD ports, it’s important to firstly read /usr/ports/UPDATING for any update advisories.

Then, to list ports with issued warnings, do:

# portaudit -a

Once you are confident that the advisories for installed packages are negligible, issue:

# DISABLE_VULNERABILITIES=yes portmaster -R -a -f -B -C -d -G

From portmaster(8):

-R skips ports updated on the previous run
-a checks all ports and updates when needed
-f forces build
-B prevents portmaster from creating a backup package
-G retains custom build config options
-C prevents # make clean from being executed before the build
-d always clean distfiles

DISABLE_VULNERABILITIES=yes prevents portmaster from aborting when an ‘unsafe’ port is encountered.

Import FreeBSD Root ZPool to Other Environments

While testing various memory disk filesystems in FreeBSD, I intelligently made changes to /etc/fstab that pointed to non-existent mountpoints (zroot/usr/home did not exist until later in the boot process - and the system refused to boot into FreeBSD due to mounting errors.)

To remedy this, I booted into my Ubuntu Oneiric installation, which fortunately supported native ZFS (follow this guide). Using the following commands, I was able to mount my root partition zpool - zroot - to make the necessary adjustments to /etc/fstab.

In Ubuntu, as root, list the available zpool imports with:
# zpool import

This will reveal all existing zpools found on all disk drives. Check to see if the zpool can be properly imported by reading the state, status, and action descriptions of the zpool.

Continuing on, create a temporary mountpoint for the zpool:
# mkdir -p /mnt/zpool_temp

Then import the zpool:
# zpool import -N -R /mnt/zpool_temp zroot

To confirm this worked, do:
# zfs list

Also, note the dataset’s original mountpoint value with:
# zfs get mountpoint zroot

Then:
# zfs set mountpoint=/mnt/zpool_temp zroot
# zfs mount zroot

This will temporarily mount the root zpool R/W at /mnt/zpool_temp, allowing you to make the necessary changes.

After you are done making changes to the mounted dataset, while making sure that it is no longer in use:
# zfs unmount zroot

Then, reset the mountpoint to it’s original value:
# zfs set mountpoint=/ zroot

Now reboot the machine without exporting the zpool - this is very important.

Your FreeBSD should now boot properly.

Move Chromium's Cache to MFS Memory Disk on FreeBSD

It is possible to use FreeBSD’s MFS Memory Disk to speed up access times of cached web objects.

From md(4):

The md driver provides support for four kinds of memory backed virtual disks:

swap:
Backing store is allocated from buffer memory. Pages get pushed out to the swap when the system is under memory pressure, otherwise they stay in the operating memory. Using swap backing is generally preferable over malloc backing.

Chromium’s cache is at $HOME/.cache/chromium, and so, my /etc/fstab contains:

1
2
3
# 256M mfs for google chrome
/dev/md0 /home/ootput/.cache/chromium mfs rw,-s256m,-wootput:ootput,noauto 2 0

From mount_mfs(8):

-s256m specifies a memory reservation of 256 M;

-wootput:ootput hands ownership of the mountpoint from root to the user; and

noauto prevents the system from failing to boot properly if the mountpoint does not (yet) exist.

The following additions to /etc/rc.conf were also made:

1
2
3
4
mdconfig_md0="-t swap -s 256m"
mdconfig_md0_owner="ootput:ootput"
mdconfig_md0_perms="1777"

which lists arguments to mdconfig(8) for device md0.

At mininum the -t type must be specified and either a -s size for malloc or swap backed md(4) devices or a -f file for vnode backed md(4) devices.

The redundancies found in fstab and rc.conf allows automatic mounting at boot, or manual mounting after boot.

To test the changes made, as root:

# /etc/rc.d/mdconfig restart

On to the $USER/bin/pack_chrome executable script that will both mount the md device, and/or sync contents from the cache at RAM to a backup cache on the HDD:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#!/usr/local/bin/bash
CACHE="$HOME/.cache/chromium"
BACKUP="$CACHE.bak"
for DIR in $CACHE $BACKUP; do
if [ ! -d "${DIR}" ]; then
mkdir -p ${DIR}
fi
done
if test -z "$(/sbin/mount | grep -F "$CACHE" )"; then
sudo /sbin/mount "$CACHE"
fi
if test -f "$CACHE/.synced"; then
rsync --exclude '.synced' --exclude '.snap' --delete -avrPx
"$CACHE/" "$BACKUP"
else
rsync --delete -avrPx "$BACKUP/" "$CACHE"
touch "$CACHE/.synced"
fi

The script requires that the $USER is in sudoers(5), and is able to use the /sbin/mount command.

Finally, after $ crontab -e:

1
2
*/30 * * * * $HOME/bin/pack_chrome >/dev/null 2>&1

.. and the script gets called every half-hour by cron to automate the syncing process. This also requires that $USER is specifically allowed in /var/cron/allow.

Upgrading to FreeBSD 8-Stable

As a reminder to myself:

# pkg_add -r fastest_cvsup
# fastest_cvsup -c all

In /root/stable-supfile, have:

1
2
3
4
5
6
7
8
*default host=cvsup.au.freebsd.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=RELENG_8
*default delete use-rel-suffix
*default compress
src-all

# csup /root/stable-supfile

# cd /usr/src
# rm -r -f /usr/obj

Customize /etc/make.conf (read notes). Mine contains:

1
2
3
4
5
6
7
8
9
10
11
CPUTYPE?= core2
NO_LPR= YES
NO_NIS= YES
CUPS_OVERWRITE_BASE= YES
NO_INET6= YES
WITH_NVIDIA_GL= YES
WITH_NVIDIA= YES
WITHOUT_NOUVEAU= YES

# script /var/tmp/buildworld-$(date "+%Y%m%d%H%M.%S")

# make -jX buildworld

( where X == number of CPUs + 1 )

# exit
# script /var/tmp/buildkernel-$(date "+%Y%m%d%H%M.%S")

# make buildkernel KODIR=/boot/testing KERNCONF=CUSTOM64

( where CUSTOM64 is documented here )

# make installkernel KODIR=/boot/testing KERNCONF=CUSTOM64
# exit
# nextboot -k testing
# shutdown -r now

# cd /boot
# rm -r -f OLD
# mv kernel OLD
# mv testing kernel

# adjkerntz -i
# rm -rf /etc.old && cp -Rp /etc /etc.old
# mergemaster -p
# cd /usr/src
# make installworld
# mergemaster -iU
# make delete-old
# shutdown -r now

NB: NO_NIS in /etc/make.conf requires the following changes in /etc/nsswitch.conf:

1
2
3
group: files
passwd: files

If you’re feeling game, you can try the following commands to save yourself some keystrokes and delays:

# csup /root/stable-supfile && rm -r -f /usr/obj && cd /usr/src && make -j5 buildworld && make buildkernel KODIR=/boot/testing KERNCONF=CUSTOM64 && make installkernel KODIR=/boot/testing KERNCONF=CUSTOM64 && cd /boot && rm -r -f OLD && mv kernel OLD && mv testing kernel && reboot

Here, you can revert back to the old kernel if you experience boot up problems. Otherwise, continue with:

# adjkerntz -i && rm -rf /etc.old && cp -Rp /etc /etc.old && mergemaster -p && cd /usr/src && make installworld && mergemaster -iU && make delete-old && reboot