The Rubyists

How to Build a Namecoin or Bitcoin Miner in 98 Easy Steps

If you don’t know what namecoin or bitcoin mining is, this article isn’t for you. See Bitcoin Home for that intro.

Part 1 (Steps 1-6): The Hardware

For all of these steps this I highly recommend using local sources. Feel free to look online but everyone who knows anything about mining has already picked the cherries. Even well-stocked stalwarts like newegg are out of all the cards that can handle mining 24/7 at good rates without overheating. Find a Ryan. Ryan is an encyclopedia of hardware who works at Fry’s Electronics in Arlington, Texas. I don’t mean find just him, but find someone who knows their stuff. 10 Minutes with a Ryan and you get a great rig for a great price. The hardest to find cards are available in the bargain bin and Ryan knows where they are. He knows which of the newest cards can be clocked highest without overheating. He knows all the OEM parts they only keep in the Cage and never show pricing for otherwise (RAM for $12, Sata Drive for $20). There’s nothing (other than the mighty Radeon) more valuable for rigging than a Ryan.

1. See this Mining Comparison and make your own determination on which ATI Radeon to get.

   Important factors in choosing a Radeon include mhash/s, mhash/$, and mhash/J(oule). This is going to be the majority of your purchase price in most miners, so don’t scrimp. If you can find a 5830 these are said to be the cheapest mhash/$, but the 6870 (Asus brand has been our best) is easy to find and will do ~300mhash and can be found for ~$200.
   Such a card can pay for itself mining 24/7 for about 4-6 days. You can find the gems/gold (5870/5970/6990) from time to time in small stores (Altex in Texas had a few in each store), grab them when you can.

2. Pick your Case.

   This step is where I’ve made the most mistakes. A great Name Brand means nothing if the cards won’t fit. Sometimes the bottom cards (in a 2-way setup) will hit the PSU, sometimes the top one will hit the CPU or some mondo fan they’ve included to look cool. What you need is space. Space for the 2 (or 3, or 4) gigantic radeons (seriously, go look at one) to fit in the thing without running in to anything important. It also has to have a lot of airflow. Sure you can watercool or whatever, but we want cheap and low-maintenance. Put a fan in every place available, a good one (coolermaster 120mm for $9 does great here). Our current favorite case for rigging is the Antec DF-85. It’s around $150 but has a ton of space, no internals in the way, a well-located side-port for a fan to exhaust video card heat, and great airflow with the 7 included fans. It should have a ‘backside’ behind the motherboard where you can run all the cables out of the way so they don’t interfere with airflow.

3. Motherboard Time.

   Space. That’s all you care about. Look at the motherboard, personally (this is where Ryan helps again) and make sure there’s not a sata controller under where your massive Radeon needs to extend. Make sure it’s not going to block your Power and Reset cables (I’m looking at you, GigaByte). You may have to make a mistake here, yet another reason to go Local. Make sure they have a return policy in case you got it wrong. MSI 790x-g55 is our latest buy, fitting two Radeon gigantors with decent spacing. We have not yet tried a 3 or 4 way setup, as the PSU price and mobo price jump substantially, and the cooling seems to be hitting a threshold already.

4. Processor and RAM

   Who cares? If you’re using it for a miner, buy the cheapest (OEM, make Ryan bring out his little special price sheet) crap they have in stock, it’ll never be used. If you are using it for other than a miner, make your own decisions, just know that you don’t be able to do anything graphics-wise while it’s mining. We have had great luck throwing a Radeon in a mail or dns or db server and having the GPU mine while the regular server processes are running, mining on the GPU seems to not affect the rest of the system much at all. If you can find a cheap CPU internal, self-contained liquid cooler like the Antec Kuhler, it could lower the overall temperature of the internals, which could significantly lower the temps on your Radeons. Ymmv, the jury is still out on cost vs. temperature decrease on that one. In the most recent rig Radeon temps are 4C lower with the Kuhler than with a stock CPU cooler.

5. PSU

   The power supply has to be strong enough to run your cards at max power. I don’t want to begin to think about the math here, but a Coolermaster 800W Gold and Raidmax 850 have been able to run every rig we have (all dual cards, from Radeon 5850-6970). Did I mention a Ryan? You really want something that draws as little power as possible, as your electric bill is the only recurring expense of mining.

6. Extras

   As stated earlier, fill in all the fan spaces (exhaust from the side port, if possible). Unless you’re using this for something other than mining you don’t need anything else, save your money.

Part 2 (Steps 7-16) Build the Rig

Yes, you can get someone else to do this. Maybe your Ryan does it on the side. Maybe you know the MacGuyver of hardware. One way or another, someone has to build it, here’s how to do it yourself. Grab scissors or a box knife, a #2 phillips, and a needlenose (optional, do you have tiny fingers?).

1. Open the case

   Take it out of the box, and open both sides. take all those crazy cables running from the front panel and route them to behind the motherboard. If you didn’t get a case where you can do this, go back to Part 1, section 2. Get all the cardboard crap out of it and clear plastic crap off off everything. Put all the hardware that came in the little bag or box somewhere easily accessible.

2. Put the hard drive in it.

   Most cases of this class are fairly tool-less here. You may need to put some doo-dads on the side of the drive and slide it in or (in the DF-85) just find out how to open the front panel and slide it in.

3. Put the motherboard faceplate in the case.

   If you forget this step you will really be hating yourself later. It takes 4 seconds but it’s important enough to make its’ own step. It’s the metal rectangle with all the holes for the PS2, USB, network, and other connections are. Just snap it into the case, removing the one that came with the case first, if there is one.

4. Put the Processor and Ram in the Motherboard

   Do this before you mount the motherboard to the case, yes. You should only have to take the motherboard out of the anti-static bag and put it right back down on that grey foam underneath. This way when you apply pressure to the RAM it’ll be softly backed. You also get the advantage of full lighting without the case in the way. Shouldn’t be anything to think about here, the processor only fits one way and if you followed Part 1, Step 4, you only got one stick of cheap ass ram, just put it in a slot. If you’re using the stock CPU fan/heatsink, mount it now, as well. It’s a pita, involving hooking one end of the lever/hook system then hooking the other and pulling down a torsion lever. If you didn’t use a stock fan/heatsink the aftermarket cooler should come with its own instructions. Mount it now only if it makes sense (does not for most liquid coolers).

5. Mount the Motherboard.

   Line up as many of those brass mounting open-threaded bolt/screw thingies as you can with holes in the motherboard. 6 is the minimum, more is better, and you can’t get too many. Generally unless you get an E-ATX board you’re stuck with 6, live with it. Line them up, squeeze your board under the metal springy things on the backplane (you did Step 3 of this Part, right?), and put in the motherboard screws to secure it (motherboard screws come with the motherboard or case, and are usually labeled as such). If they aren’t labeled, find the hexagonal screws with coarse threads. Do not over tighten, it only has to be snug.

6. Mount the PSU

   This is 4 screws (sometimes thumb screws) that only fit one direction. Mount it snug but don’t over tighten. Don’t put any modular power rails (cables) in place yet, and run all the cables out the backside as in Step 1 of Part 2, if possible. .

7. Hook up all the cables that do stuff.

   You probably only need Power Switch, Reset SW, and enough USB to install, but it’s nice to also have all your USB, the Power LED, and any doo dads you want to use (audio?). Make sure none of the optional stuff gets in the way of your Radeons, or ditch them. Hook your drive’s SATA cable up at this time, but not the power, yet.

8. Hook up all the cables that power stuff.

   Those thick cables you ran through the back in Step 6, Part 2, now should find their homes. There will be an 8 prong female to plug your 8 prong male (or two 4 prongs on the same cable) and one 20something prong really thick one that has only one obvious female large enough to take it. Run these behind the motherboard if possible to the closest opening to the female and insert them until the click. All of those fans will need rails run to them, as well, of that big 4 prong type that’s a pain in the neck to get secure. Fiddle with them until they’re all plugged in. Now you can route the SATA power to your single lonely hard drive.

9. Put those Radeons in.

   If all has gone well, you have room to fit the two Radeons into the two PCI-E slots on the motherboard. They should have a gap of more than 1/2 between them, if not you better have a great side-port fan exhausting the heat. They click and have 2 screws on the PCI bracket holding each secure. Each needs a 12 prong (usually dual 6-prong) rail run from the PSU to it after it’s mounted. Make sure you hold the back side of the card secure while you attach the power rails, for those cards which have the power lead facing the outside of the case.

10. Tidy and close it up.

    Take any cable ties the hardware came with and make sure you keep all the cables away from the airflow. Behind the motherboard is always best, but some will have to be exposed, just tie them out of the way. Now put the case side panels back on.

Part 3 (Steps 17-29) Install the Operating System.

This is a step-by-step of how we build archlinux manually. A build could be scripted with the aif arch install framework, as well. This in no way supercedes or replaces the Official Arch Linux Install Guide nor the Beginner’s Guide. Please refer to those documents for build setups outside the scope of this simple miner.

1. Download Arch Linux

   Download 2010.05 from the Arch Linux Download Site and follow the instructions to burn it to a cheap USB stick. You can get 32 bit or 64, we’re using the 64bit version, instructions should be the same.

2. Boot Arch Linux

   Put the USB stick you wrote the archlinux installer on in a USB slot, power on the computer, and watch for the ‘Hit _ For the Boot Menu’, where _ is some key which lets you modify the device to boot from. Choose your USB stick and continue booting. When you get to the screen that says “Arch Linux Live ISO”, you’re there.

3. Log In and Get it on the network.

   Log in with ‘root’, it will have no password. For a simple wired connection, this can be done with

   dhcpcd eth0

   If you have a wireless network,

   /arch/setup
   
   Now choose 1) Select Source
   Then 2) net NET (FTP/HTTP)
   Hit <OK> when it talks about loading modules
   Now 1) Setup Network
   Choose your Wireless Card (probably wlan0)
   Fill in your wireless network information.
   Say <Yes> to "Do you want to use DHCP?"

   You should see “The network is configured.”

   If you have another machine on that network and want to use it for the install, you can exit the installer now and complete step 20.

   Hit <OK>
   Then 3) Return to Main Menu
   Then 8) Exit install

   Otherwise skip to Step 21.

4. Start SSHD And Log In

   Installing via ssh over the network has many advantages, the primary one being you can look at this guide split screen and copy/paste to the ssh session. To start ssh

   /etc/rc.d/sshd start
   echo "sshd : ALL : allow" >> /etc/hosts.allow
   passwd
   (set a password)

   Now ssh from your regular computer to the IP of the new miner. You can find its current IP with

   ip a l

5. Start (or continue) the Arch Linux Installer

   You should be at the Main Menu of the arch linux installer. If you ssh’d in, this can be started with

   /arch/setup
   
   Choose 1) Select Source
   Choose net NET
   hit <OK>

6. Choose a Mirror

   Choose a mirror which is close to you geographically, if possible. Don’t use a throttled one. http://mirror.rit.edu is good in the U.S. Now “Return to the Main Menu”

7. Set Clock

   Select your region and timezone, then set time and date. Always choose UTC for the Clock Configuration. If the time is off after you select UTC, choose the ntp option to set the time and date using ntp. Return to the Main Menu

8. Prepare Hard Drive

   You got the cheapest POS hard drive you could find, right? Just choose 1) Auto-Prepare, choose the hard drive (usually /dev/sda), and let it use all the defaults it presents you for partition sizes. When it asks for Filesystem selection, we use ext4 but you’re free to use any you like. Let it COMPLETELY ERASE! the drive you chose. You are absolutely sure. If for some reason this fails you can manually partition or choose the individual block devices yourself. You only need / and Swap. Back to the Main Menu when it says “Paritions successfully created”

9. Select Packages

   Choose both base and base-devel, then when the big screen with all the packages comes up find ‘netcfg’ and choose it, then hit OK

10. Install Packages

    Hit OK and this runs on its own. Watch and wait til it’s complete.

11. Configure System

    Choose this, say Yes to using network settings from the installer in rc.conf and resolv.conf, then choose an editor. If you don’t know any of them, choose nano.

    /etc/rc.conf: Change the HOSTNAME to something you wish to name this.  
    Now find the line that says #NETWORKS=(main) and remove the # from the beginning of that line
    change DAEMONS=, replacing 'network' with 'net-profiles' and adding 'sshd' (no quotes on any of them)
    Save the file
    
    Skip /etc/fstab and /etc/mkinitcpio.conf
    /etc/modprobe.d/modprobe.conf: add a line that reads 'blacklist radeon' (without the quotes).  Save the file
    
    Skip til /etc/hosts.allow: add 'sshd : ALL : allow' to it and save it.
    
    Skip to Root-Password and set one.  Make sure you get it right, it won't give you much warning if you get it wrong.
    Choose Done, it will do a little scroll dance after a few seconds then return to the main menu.

12. Install Bootloader

    Choose Grub, then hit OK to open an editor for the grub configuration file. Find the first line that says kernel /vmlinuz26 root=… and add ‘nomodeset’ to the end of it (no quotes). Save the file, then choose the correct device to install the bootloader on, likely /dev/sda. You should see “GRUB was successfully installed”.

13. Exit install and reboot.

    Once you exit the installer, type reboot -f and hit enter to reboot.

Part 4 (steps 30-61) Install the miner software.

1. Boot the new Arch Linux and Log In.

   It should boot on its own, all the way to a log in prompt. Log in as root with the password you set in the “Configure System” step.

2. Set up the network.

   cd /etc/network.d
   cp examples/ethernet-dhcp ./main
   netcfg main

   If you aren’t using ethernet-dhcp, or have to choose a different interface from eth0, you can edit the file and change any parameters you need. In /etc/network.d/examples are other types of connectioned clearly named. Copy any one of them you need to /etc/network.d as ‘main’ and edit to taste, if the above did not work for your connection.

3. Create a normal user account

   This is the user that will run the mining and you can also use it to administrate the system. If you want separate admin account from the miner account you can create as many as you like here.

   useradd miner

   Follow the prompts, accepting defaults if you don’t want admin privileges for the user, adding the group ‘wheel’ (no quotes) as an Additional group when prompted for Additional groups.

   New account will be created as follows:
   
   ---------------------------------------
   Login name.......:  miner
   UID..............:  [ Next available ]
   Initial group....:  users
   Additional groups:  wheel
   Home directory...:  /home/miner
   Shell............:  /bin/bash
   Expiry date......:  [ Never ]

   If yours looks like that, hit ENTER to create the account. You will have to set a password, you don’t have to fill in the other information, you can just enter through it all, leaving them blank.

4. Install some software

   pacman -Syu

   This will make sure everything on your system is current. Pay attention to anything it tells you to do while it’s upgrading. You will likely have to

   pacman-db-upgrade

   Before it will allow you to use it

   pacman -S rsync git openssh yajl sudo tmux

   Once this installs, you can start openssh with /etc/rc.d/sshd start and log in remotely to continue the steps, if you prefer.

5. Configure Sudo

   If you don’t know how to use vi

   EDITOR=nano visudo

   Otherwise, just visudo. Uncomment (take the # out from in front of) the line that reads

   # %wheel ALL=(ALL) ALL

   So it should now be

   %wheel ALL=(ALL) ALL

6. Change to your regular user and install Yaourt.

   If you’re going over ssh, ssh to the new miner as your normal user, not root. If you’re local, logout as root and back in as the normal user (with wheel privileges) you created earlier.

   cd /tmp/
   wget http://aur.archlinux.org/packages/package-query/package-query.tar.gz
   wget http://aur.archlinux.org/packages/yaourt/yaourt.tar.gz
   tar zxvf package-query.tar.gz
   cd package-query
   makepkg -si (say Yes to all the prompts about installing, then watch the scroll WHEE).
   cd ..
   tar zxvf yaourt.tar.gz
   cd yaourt
   makepkg -si

7. Install the ATI Drivers

   All of these commands are to be done as your regular user, as well. Each will ask to install a lot of dependencies then prompt you to build, then to install. Say Y/yes to the questions about installing and watch the scroll while it downloads and builds the packages. Say N/no to the ‘edit the PKGBUILD’ and ‘edit the .install’ questions. Any time yaourt asks for a password it expects the password of the regular user you created, not root’s password.

   yaourt -S catalyst-utils
   yaourt -S kernel26-headers catalyst-hook
   yaourt -S amdstream

   Amdstream will take quite a while, maybe grab a smoke or eat some twizzlers here.

8. Install X.

   yaourt -S xorg-server xorg-apps xorg-appres xorg-xinit xautolock xlockmore xorg-fonts xorg-xhost xorg-xauth ratpoison xterm

   Choose the default (all) when it asks about xorg-xfonts, the proceed with the installation.

9. Create a .Xauthority

   Put the following in a script (i name it newkey.sh and put it in ~/bin)

   mkdir ~/bin
   vim ~/bin/newkey.sh

   Here’s the contents of that file

   #!/bin/sh
   # create new .Xauthority file
   # try some security
   auth=$HOME/.Xauthority
   cp /dev/null $auth
   # generate a nice long random key
   if [ "$1" = "-md5" ]; then
     # use a random noise source and get a strong checksum of it.
     # this is probably a stronger random number than the other method.
     key=`pstat -pfS | md5`
   else
     # quick and dirty.  can probably be recreated if time can be guessed.
   #  key=`perl -e 'srand; printf int(rand(100000000000000000))'`
     key=`perl -e 'srand; printf int(rand(10000000000000))'`
     # use $key$key to make sure even length.
     key=$key$key
   fi
   # add to auth file. 
   xauth add unix:0 . $key
   # was xauth add ${HOST}/unix:0 . $key
   xauth add ${HOST}:0 . $key

   Once you have this created, issue the following

   chmod 755 ~/bin/newkey.sh
   ~/bin/newkey.sh

10. Configure X

    First find out where your cards are.

    lspci|grep VGA

    Make a note of that first number. In the following I have three

    04:00.0 VGA compatible controller: ATI Technologies Inc Cayman PRO [AMD Radeon 6900 Series]
    05:00.0 VGA compatible controller: ATI Technologies Inc Cayman PRO [AMD Radeon 6900 Series]
    0a:00.0 VGA compatible controller: ATI Technologies Inc Cayman PRO [AMD Radeon 6900 Series]

    The number you want is the PCI Bus ID. Here it’s 4, 5, and 10 (0a in hex is 10 to us humans)

    Now use the aticonfig tool to create a basic configuration which we’ll modify.

    sudo aticonfig --initial=dual-head

    We have to edit what it created, use an editor to open /etc/X11/xorg.conf

    Find each section like

    Section "Device"
      Identifier  "aticonfig-Device[0]-0"
      Driver      "fglrx"
      BusID       "PCI:4:0:0"
    EndSection

    Set that BusID to be the number you saw in lspci, there should be one Device section for each card. (Device0, Device1, etc) Save the file.

11. Make an .xinitrc and start x

    Let’s make sure it will run, first we create a simple xinitrc.

    cat 'setxkbmap -option terminate:ctrl_alt_bksp
    xautolock -corners +-+- -locker "xlock -mode blank" &
    xterm &
    exec ratpoison' > .xinitrc

    And then start it

    xinit -- -auth ~/.Xauthority

    To stop, hit ctrl-alt-bksp (backspace).

12. Install poclbm and start mining.

    We’re using poclbm because it’s given us the least hassle and good numbers. Your mileage may vary.

    yaourt -S python2 poclbm-git
    
    poclbm 

Runit for Ruby (And Everything Else)

The Runit system is a process supervisor and set of utilities surrounding supervising processes.

It can be used as a sysvinit replacement or run side-by-side with many init systems to maintain supervised process heirarchies.

We began using runit as a replacement for daemontools in November 2001, because runit was OSI licensed at the time (daemontools was not) and for the extensive documentation offerred for runit utilities (including manpages).

Fully implementing runit, as a replacement for init/PID 1 or as a standalone supervisor, is covered in general at the runit page above and in detail for Arch Linux.

Some things we especially like about this system are:

Service View

Runit offers a concise but complete view of process status, including the optional log service.

bougyman@jimmy:~$ sudo sv s /service/*
run: /service/callcenter: (pid 2870) 5266009s
run: /service/cron: (pid 3769) 7700115s
run: /service/fxc: (pid 3714) 7700118s
run: /service/getty-1: (pid 3713) 7700118s
run: /service/getty-2: (pid 3730) 7700117s
run: /service/getty-3: (pid 3716) 7700118s
run: /service/getty-4: (pid 3690) 7700118s
run: /service/getty-5: (pid 3691) 7700118s
run: /service/helpdesk: (pid 3715) 7700118s
run: /service/lighttpd: (pid 27321) 5208602s; run: log: (pid 3724) 7700117s
run: /service/openntpd: (pid 3747) 7700116s; run: log: (pid 3741) 7700117s
run: /service/postgres: (pid 3732) 7700117s
run: /service/redmine: (pid 3729) 7700117s
run: /service/socklog-klog: (pid 3746) 7700116s; run: log: (pid 3733) 7700117s
run: /service/socklog-unix: (pid 3758) 7700116s; run: log: (pid 3756) 7700116s
run: /service/ssh: (pid 3757) 7700116s; run: log: (pid 3731) 7700117s  

As you can see, all services live in one place, defined by $SVDIR or /service by default, which runsvdir manages.
The services in /service/* are symlinks to directories (usually in /etc/sv/) which must contain one executable file, named ‘run’.

The run executable should exec the process in the foreground with stderr redirected to stdout.

Runit avoids unecessary complexity by separating the roles into small, dietlibc-friendly utilities, written in C without library dependencies.
The above listing represents many individual services, each with one instance of the per-process supervisor runsv running.

Example Service (sshd)

/etc/sv/sshd/run:

#!/bin/sh
exec /usr/sbin/sshd -De 2>&1

If the directory log/ exists, it will be treated as a log service.

runsv will create a pipe and redirect standard out of the service (and its optional finish script) to log/run.

Log Service (for sshd)

/etc/sv/sshd/log/run:

#!/bin/sh
exec svlogd -t /var/log/sshd/

Svlogd Logging

Runit includes a superb lightweight logging system in svlogd.

When you supervise a process, the optional log service’s stdout is directed to svlogd (or any other logger which accepts stdin).

Svlogd offers granular control of how to log that output, including rotation on many metrics (without stopping the process it’s logging), post-processing of logs, networked logging (both standard syslog style and offers its own network option), notifications, filtering, and more.

Config file for a log service

/var/log/sshd/config:

s100000
n10
N5
t86400
u192.168.6.20
pSSHD_LOG
-bougyman
!logwatcher

The above would set the max size of a log file to 100000 bytes, keep 10 files maximum, 5 files minimum (if the disk were full it would delete 6-10 until it had room), rotate every 86400 seconds (even if the size limit were not exceeded), network log to 192.168.6.20, prepend each log message with SSHD_LOG, not log messages matching the pattern “bougyman”, and upon rotation run each logfile through sh -c 'logwatcher'.

Not very descriptive, but well documented (in man svlogd).

Process respawning and control

If a process stops, it will be started again immediately without intervention or outside process monitoring.
runsv avoids a failed or faulty process hammering resources by delaying 1 second before starting a process back up if it’s failing instantly.

When a process stops, if a file named ‘finish’ exists and is executable, finish will be run with two arguments, the exit code and exit status of run.

Example finish script

/service/sshd/finish:

#!/bin/sh
code=$1
status=$2
hostname=$(hostname)
if [ $status -ne 0 ];then
  echo "SSHD FAILED on $hostname: $code, $status"|mail admins@domain.com
else
  echo "SSHD Stopped on $hostname: $code, $status"|mail auditing@domain.com
fi

sv usage examples

sv is used to get and change status of a particular service. This is your swiss army knife of process control.

Get status

$ sv s sshd
run: sshd: (pid 4040) 6435s; run: log: (pid 4028) 6435s

Send the TERM signal, which will restart a process. This works because runsv will always keep a process started unless you tell it to stay down.

$ sv t sshd

Generally there will be no output from such commands, use -v to get some output (examples from here on out will use -v)

$ sv -v t sshd
ok: run: sshd: (pid 26764) 0s

Stop a service and keep it down (until the next boot or you specifically tell it to come back up)

$ sv -v d sshd
ok: down: sshd: 1s, normally up

Send USR1 to a service

$ sv -v 1 sshd
ok: down: sshd: 56s, normally up

Remove a service (stop it and make it not start back up, even on boot)

$ rm /service/sshd

Of course there will be no sv output from the above.

Removing a symlink in /service (or your $SVDIR if not /service) will send the equivalent of an sv d(own) to runsv, stopping the process and its log service, then stopping runsv.

Flexible dependency system

The sv program (with the check subcommand) also allows for any dependency tree you can dream of (and script).

Basic dependency example

/service/lighttpd/run:

#!/bin/sh -e
sv -w7 check postgresql
exec 2>&1 \
  lighttpd -f /etc/lighttpd/lighttpd.conf -D

This would wait 7 seconds for the postgresql service to be running, exiting with an error if that timout is reached.
runsv will then run this script again. Lighttpd will never be executed unless sv check exits without an error (postgresql is up).

Dependency hierarchies (Per-User service trees)

Building on the above dependency checking, we can go one step further and give a user control over their own $SVDIR, /home/username/service here.
This gives non-root users the same ability to guarantee services are running, supervised, properly logged, and all the advantages of the main $SVDIR.

User-level supervision example (with dependency checking)

/service/callcenter/run:

#!/bin/sh -e
sv check postgresql couchdb memcached
exec 2>&1 \
  sudo -H -u callcenter runsvdir -P /home/callcenter/service 'log:.........................................................................................................'

Now any service directory the callcenter user symlinks in ~/service/ will get its own runsv process, running as callcenter. None of the callcenter user’s processes will be started (because runsvdir will not run) until postgresql, couchdb, and memcached are up and running successfully.

What’s with all the dots?

The log:………… is a construct runsvdir uses to log to proctitle (seen in ps aux). Each dot represents 15 seconds of time, with any stderr/out being flushed to proctitle for each dot. Thus if you saw

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      4001  0.0  0.0    184    28 ?        Ss   08:00   0:00 runsvdir -P /service log: ed with loglevel notice?Could not load host key: /etc/ssh/ssh_host_ecdsa_key??...

You’d know sshd had a problem, but it hasn’t had that problem in 45 seconds.

Example Ruby Service for a User

/home/callcenter/service/fs2ws/run:

#!/bin/sh -e
envdir=$PWD/env
root=$(<env/TCC_Root)
echo Starting from $root
cd $root
if [ -s $HOME/.rvm/scripts/rvm ]; then
  source $HOME/.rvm/scripts/rvm
  echo "Using RVM $(which rvm)"
  rvm use 1.9.2 2>&1
fi
exec 2>&1 \
  chpst -e $envdir ./bin/fs2ws 2>&1

Finally, a hint of Ruby!
This script includes a check to see if rvm is installed and if so uses its 1.9.2 version.
The -e to chpst specifies an environment directory, which is a way to set environment variables, but chpst can do much more.

Standalone environments

chpst allows multiple controls around processes we run, including memory capping, user privileges, nice levels, lock files, open files, data segments, cores, stdin/out, and all environment variables.
It is the swiss army knife for manipulating what a running process can do.
This command encapsulates all of the functionality of the daemontools commands setuidgid, envuidgid, envdir (used above), softlimit, and setlock. If called as one of those commands, it will behave as that utility.
chpst adds the behavior of pgrphack and fghack, as well as some behavior not available in daemontools such as nice level.

That’s a lot of manipulation in one tool, but most of them are rarely used.
The -e/envdir option is the one we take advantage of most, essentially using it as a replacement for options/config files.
A directory of files where an enviroment variable will be created for each file, with the value set to the contents of that file, may appear cumbersome at first glance.
In practice, however, we find that changing one or two options is the most likely workflow.
With the envdir setup, this becomes

echo 'sofia/gateway/default/%s' > ~/service/fs2ws/env/TCC_ProxyServerFormatString

Not so cumbersome for that! It may indeed take longer to set these variables up the first time, but maintenance is not so bad.
One advantage of having this data in environment variables is the ability to query /proc/PID/env/ for the options your ruby got on startup.
The second (more important) one we found is that we make fewer options.
That’s right, it makes us think about each option we add, knowing it will be another file in env/. So we don’t have as many options. Good Thing.

Now our (ruby) process has access to all of these in ENV["TCC_*"], how to use them?.
See the Tiny Call Center default envdir and options.rb for one way to glue them., but rolling your own should be trivial (just look at ENV in a chpst -e ./env irb).

Parallel startup

Services all start concurrently, drasticly reducing time spent changing runlevels (including the initial boot).

How Fast (Benchmarks!)

The only fair benchmark available for runlevel switching, because of the differences in what arbitrary runlevels mean in various systems, was initial boot time.
The results of the initial boot time are of course influenced by POST time, which is listed for each of the below reference systems. Each of these systems is a default Arch Linux install with the Arch Linux default (sysvinit), runit-run-git, or initscripts-systemd installed, running getty’s, a syslogger, ssh, and cron (no X).
Because POST has varied on the systems, the initial boot times and reboot times are an average of 10 runs of each action. A system is considered ‘booted’ when a user is able to log in to a getty.

Acer 1830t Small Form Factor Notebook

680U i7 1.47Ghz processor
4G Ram
640G 7200RPM Sata HD
Intel Graphics on board
Post time 5-7 seconds.

Toshiba Qosmio Portable Workstation

740QM i7 processor
8G Ram
Corsair Performance III SSD
Nvidia GTX460m 1.5G Discreet Graphics
Post time 2-4 seconds.

The main time difference we saw in the SystemD startup was the 2-3 seconds dbus itself takes to start, giving runit its advantage. If you started dbus in runit stage 1 we expect these times would be nearly identical.

Infinite runlevels

You are not limited to 0-6, with only current and previous as reserved.

Runlevels become (unlimited amount of) directories of services (in /etc/runit/runsvdir) which can be switched to quickly and simply.

# Switch to the services described in `/etc/runit/runsvdir/primary`
$ runsvchdir primary

# Switch to the services described in `/etc/runit/runsvdir/failover`
$ runsvchdir failover

No mysterious backgrounding

Processes run in the foreground logging to stdout/stderr.
This requirement for well-behaved processes can be a blessing but some may see it as a curse.
Processes that are not well behaved can be supported through a once service, if you absolutely cannot do without them.
We use this as a bit of a litmus test for potential new services. When they can not run in a well-behaved manner that’s a high-priority Con, one which takes extraordinary Pros or absolutely necessary business-case to overcome.

Ebeats Competition Leaderboard scj Killing It

scj is claiming victory early in the Ebeats Competition, as his lead is (his words) insurmountable now. He’s implemented ebeats in everything from prolog to postscript, and he spits on the idea of anyone catching his couple-dozen implementation lead.

We’re happy to reward scj for his efforts, but would love to see him sweat a bit with some competition, especially in light of all the easy languages still available. None of the BASICs, the pre-sql dbs, not even a pure shell (zsh/ksh lovers?).

Keep in mind once this competition is over the next will begin. Perhaps it will be golfing, perhaps something more (or less) interesting. Keep on the lookout; one thing is certain, the bounty will raise.

Ebeats Competition Leaderboard Update

Fun first day of competition to watch, two submitters come to the fore, scj (5) from freenode and paulitex (4) on github, obvious pizza appreciators.

Show us your beats, earn a pizza

We’re getting the ball rolling on ebeats implementations by offering $77.77USD for the individual or team who contributes the most implementations of ebeats by April 15, 2011.

Payment will be made via any method the winner chooses, save carrier pidgeon, we’re fresh out.

See Ebeats Home and Ebeats Implementations for what-the-heck-is-an-ebeat, and to see which languages have already been implemented.

Freeswitcher 0.6.8 Released. Welcome jShaf and swk

It’s been a long time since a blog post, we’ll claim DayJob for that.

This release is a culmination of quite a few changes, including:

• mod_callcenter support with the call_center command
• conference support with the conference command
• more apps: pre_answer, riny_ready, uuid_transfer, sched_hangup
• Improved Inbound listener with reconnect support and better logging

jShaf and swk have jumped in and been helping on FSR and projects surrounding it, we welcome their contributions.

TinyCDR now public

A small internal project we’d used for raw storage of FreeSWITCH CDR data has now been resurrected.

This iteration adds the logging of the full XML posted by FreeSWITCH into a couchdb data source, with a link to the id of the couch record stored in the postgres (or other Sequel-supported database) record.

We’ll be adding reporting over the next week or so, time for ideas is now. Check our github project for more info.

Quick ruby one liner to reboot phones attached to freeswitch

Needed this to reboot all my extensions from 1000-1009 every night. Adjust as needed.

irb(main):039:0> (Nokogiri(%x{fs_cli -x 'sofia xmlstatus profile internal'})/:registration).select{|x| (x/"sip-auth-user").text.to_i < 1010 }.map { |n| (n/"call-id").text }.each { |p| reboot(p) }
Sending reboot to 3c2671257674-tlf1pz01hu9g
Sending reboot to 3c26700e5573-7zw9k9793uer@snom320-0004132CC8C8

Freeswitcher 0.4.4 Released, and welcome Harry

This is a fairly minor bugfix (thanks diegoviela) to fix the Playback app. The 0.4.4 should be available via gem.

Mostly i’m sorry this blog has been so idle, we’ve been working hard on FSR, FXC, and related components, but much of it on commercial or proprietary projects. In the coming months we’ll be using this knowledge to release applications which enhance the functionality of open source telecom.

Thanks to all the new contributors, and welcome to the core team Harry Vandberg. He’s pushed development along while we played end user for a few months, we’re happy to have him.

Rack::Utils and CGI escape and unescape performance boost

As performance boosts are about speed, we’ll start with the benchmarks. Here is a run of spec/bench.rb, from the url_escape source tree.

Escape

Unescape

URLEscape provides these two methods as a C extension, suitable for use on ruby 1.8.6-8 and 1.9.1+; tested on linux, XP, and Vista.

The jruby version uses the java stdlib’s java.net.URLEncoder and URLDecoder. We only see a 200-700% increase with this change, and would like to improve on those numbers.

Why?

Josh Susser initially noticed the ability to overload #escape and #unescape while testing a client application. At the same time, we had just come across a bottleneck when regression testing FXC (a web app which serves configuration information to the FreeSWITCH softswitch) where requests were being delayed in our rack middleware, which parses the POST data sent by FreeSWITCH and routes requests to the ramaze application for processing. The delay was noticeable under loads of only 50 req/second; where rack became the bottleneck, not ramaze, the db, or any other factor. Adding the above library (on linux, with ruby 1.9.1) removed the delay in rack, pushing the work back to to the web app (or database) where it’s free to be as slow as it must. Optimally we’d like to perform at a speed equal to the database, making it the final bottleneck in a dynamic application.

Installation and Usage

To use URLEscape standalone

Install with one of the following methods:

• gem install url_escape
• get the tarball from RubyForge
• get the source from GitHub and rake install in the source top-level.

Then simply require “url_escape” and you have access to URLEscape.escape(string) and URLEscape.unescape(string)

To use URLEscape’s escape/unescape in place of CGI or Rack::Utils versions

gem install rack_fast_escape

or

gem install cgi_fast_escape

This will install url_escape if it’s not already installed, as well.

You can optionally install rack_fast_escape or cgi_fast_escape from rubyforge’s tarball or the github source (rake install as with url_escape). If you use tarball or source rake install, you will have to manually install url_escape first.

Once installed, simply use

require "rack_fast_escape"

to replace Rack::Utils version, or

require "cgi_fast_escape"

to replace the CGI version.

What else?

The ability of large posts to slow down a web application cannot be removed by just speeding up the POST parser. In order to alleviate the risk of such large POSTs being used to deny a service, firewall or web server throttling or limiting is a more reliable protection to enable. Here are a few examples:

• Lighttpd: http://lighttpd.net * Offers mod_evasive which limits connections per ip, as well as the ability to limit the data rate per connection.

• Nginx: http://nginx.org * Flexible limiting system, per vhost, per user, per connection.

• Netfilter/QoS (linux): http://l7-filter.sourceforge.net/ * Allow classifiation of HTTP packets so iptables/tc or whatever utility you’d like can have the info it needs about the HTTP protocol to make limiting/dropping/queueing decisions

• Others: Apache, Squid, Litespeed, many others will have various methods of limiting size and frequency of requests.

Side Note

When speccing these libraries, a few implementation differences came to light which we’ll highlight here.

1. Rack::Utils and CGI both throw errors on a mixed ASCII and Unicode string in ruby 1.9.1 and above
2. Java’s URLDecoder and URLEncoder do not escape or unescape mixed ASCII/Unicode properly.

URLEscape (the C version) handles these cases properly, though we don’t expect you’d see them much in proper requests.

Thanks

To Evan Phoenix, Josh Susser, Trey Dempsey, Jayson Vaughn, Michael Fellinger, Kevin Berry, and all the other contributors of ideas and support who made this product a reality.

License

Nothing to fear, it’s MIT