Show Source

Debian

Upgrade With Low Disk Space

The official recommendation is to grab additional storage and temporarily put /var/cache/apt/archives there. Something like the following may also help; the goal here is to upgrade the packages that are already marked as manually installed so that other packages do not get so marked. This may reduce the disk requirements for a subsequent upgrade.

apt-mark showmanual > amsm
apt-get --just-print upgrade \
  | awk '/^Conf/{ print $2 }' \
  | (while read pkg; do grep -e "^$pkg\$" amsm; done) \
  | (while read pkg; do sudo apt-get -y install $pkg; sudo apt-get clean; done)

Ubuntu

Live CD Persistence By Default

Sometimes we want to boot systems persistently by default. Yes, yes, we’re weird. Grab the ISO; for the purposes of this exercise, we used lubuntu-15.10-desktop-amd64.iso. Extract the boot menu configuration:

osirrox -indev lubuntu-15.10-desktop-amd64.iso -cdi /isolinux -extract_single ./txt.cfg txt.cfg

Apply this diff to remove the install option and append persistent:

--- txt.cfg.orig    2015-10-21 12:39:29.000000000 -0400
+++ txt.cfg 2016-02-03 04:37:12.442310613 -0500
@@ -2,11 +2,7 @@
 label live
   menu label ^Try Lubuntu without installing
   kernel /casper/vmlinuz.efi
-  append  file=/cdrom/preseed/lubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash ---
-label live-install
-  menu label ^Install Lubuntu
-  kernel /casper/vmlinuz.efi
-  append  file=/cdrom/preseed/lubuntu.seed boot=casper only-ubiquity initrd=/casper/initrd.lz quiet splash ---
+  append  file=/cdrom/preseed/lubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash persistent ---
 label check
   menu label ^Check disc for defects
   kernel /casper/vmlinuz.efi

Repack the ISO image:

xorriso -indev lubuntu-15.10-desktop-amd64.iso -outdev custom.iso -boot_image isolinux keep -cdi /isolinux -cpr txt.cfg .

At boot, now, the livecd will scan for (among other options) a filesystem whose label is casper-rw; you can make such a thing by running, within the live system, for example:

mkfs.ext4 -L casper-rw /dev/sda1

File Transfer

Help, all I have is a shell

Pipe the output of this to your shell. This is exceptionally not fast

hexdump -e '"echo -e '\''" 120/1 "Y%03o" "'Z\'' >> xetc.tgz\n"' xetc.tgz \
  | sed 's;Y;\\0;g;s/Z/\\c/;s/\\0 *\\0/\\c/;1s/>/ /'

SLIP

Ah, the Serial Line Internet Protocol. Here’s a worked example. On the host:

slattach -L -l -n -s 38400 -p slip /dev/tty_dgrp_a2_6
ifconfig sl0 172.29.8.1 pointopoint 172.29.8.2
echo 1 > /proc/sys/net/ipv4/conf/sl0/forwarding
iptables ...

On the guest:

slattach -L -d -m -p slip -s 38400 /dev/ttySC1 &
ifconfig sl0 172.29.8.2 pointopoint 172.29.8.1

ZFS

Large File Deletion

ZFS would occasionally stall a machine if you ask it to delete a large file from a deduplicated data set as it would have engage in huge transactions involving the DDT. Now that https://github.com/zfsonlinux/zfs/issues/3725 is fixed, this should almost certainly not matter; the code here is for historical reference only.

You can hold its hand and slow the process down, which should eliminate IO stalls:

for i in `seq $(($(stat -f %z $FILE)/1024/1024)) -1 1`; do  \
        echo $i; \
        truncate -s $((i*1024*1024)) $FILE; \
        sync; \
        sleep 5; \
done; \
rm $FILE

UNIX Shell

Software Watchdog With runit

runit / daemontools / s6 and friends can be used as a kind of watchdog to detect the absence of some event. Use sv restart ... or the equivalent to tickle the watchdog and delay the event, perhaps in another supervised script!

  • ./run creates a new process group:

    #!/bin/sh
    exec setsid ./run2
    
  • ./run2 traps SIGTERM to exit and waits for a timeout before invoking the watchdog response:

    #!/bin/sh
    onTerm() { trap '' TERM; kill -TERM 0; exit; }
    trap onTerm TERM
    
    sleep $(cat ./watchdog-period) &
    wait
    trap '' TERM
    exec ./watchdog-fire
    

    Where ./watchdog-period contains the interval in seconds during which the watchdog must be reset, and ./watchdog-fire is the script to run when it’s all gone south.

Line-based Parallel Mapping in Shell

“I was nerd-sniped” is really the only defense I have for this particular section’s existence. It’s something of a response to http://catern.com/posts/pipes.html, which goes into a fair bit of detail and to moderate lengths to work around not having message boundaries respected by UNIX pipes. That approach didn’t sit well with me, aesthetically, and I guess I was wanting to get to know my shell better.

Anyway, given a shell pipeline

source | mapper | sink

if mapper‘s action is dependent only on each line, but takes a while, you might wish to have multiple mappers running in parallel. For various reasons, existing tools like GNU parallel and xargs are not sufficient (as of this writing). Thus, I present fanout.sh which uses zsh coproc-esses and the zsh/zselect to manage a flock of workers.

Use as in this self-test:

diff <(./fanout.sh 3 cat < /usr/share/dict/words | sort) <(sort /usr/share/dict/words)

If we pass a slow worker, we can see the right thing happening as the pipes first fill to their internal buffer sizes and then things start blocking:

$ ./fanout.sh 3 ./scat.sh < /usr/share/dict/words
A
Iroquoian's
Sutton's
A's
Iroquois
Suva
AA's