COSI Linux Build

From CSLabsWiki

Jump to: navigation, search

This page summarizes how to set up the COSI Lab Build, and describes what procedures are in place to maintain it.

The Lab Build is based off of Ubuntu 9.10 ("Karmic Koala"). It employs a `metapackage' repository (currently hosted in COSI by web2) to keep track of all packages that should be installed. This way, when a user requests that a new package be installed, it is easy to update all of the machines - just add the requested package to the metapackage, which all lab build computers will update from overnight. Please see the `metapackage' section for more information.

DEPRECATED:
The Lab Build is currently based off of Ubuntu 9.04 (Jaunty Jackalope). It employs a `metapackage' repository hosted in COSI to keep track of all packages that should be installed. This way, when a user requests that a new package be installed, it is easy to update all of the machines - just add the requested package to the metapackage, which all lab build computers will update from overnight. The metapackage will be discussed in more detail later on.

Contents

Project Members

Current

Past

Requests

Enter requests for the Lab Build here. If the request is available through apt, it will be added to the metapackage and the package will show up the next day if the machine is left running overnight. If the request is not available as a package, it will be added the next time the lab build is cloned out. Please leave your name when you make a request. Alternatively, you can email the maintainer.

Pending

Please list requests here.

  • Consider using the lab DNS servers (128.153.145.{3,4}) and Clarkson as the fail safe.--Matt 00:25, 3 March 2010 (EST)
  • SSH locked down.--Matt 00:25, 3 March 2010 (EST)
  • Run iptables to block everything coming into the systems--Matt 00:25, 3 March 2010 (EST)
  • Post the static IPs that are going to be used on the wiki. Many of us (that should know) don't know what IPs OIT gave us to use for static IPs on the 144 subnet. If you want *.cslabs.clarkson.edu hostnames, I can provide those. Also, you will need static IPs for the internal network since DHCP is no longer present there. Contact me when you need those.--Matt 00:25, 3 March 2010 (EST)
  • Have the search field in /etc/resolv.conf be search cslabs.clarkson.edu clarkson.edu. This allows for shortened urls. Ex. Type docs in the url and you get the wiki.--Matt 00:25, 3 March 2010 (EST)
  • Add spice client to both cosi and cosi-vr systems (see SPICE section)
  • Add package mp32ogg

Finished

Note: items in this category have been added to the metapackage, which the build updates from at 4:00 AM. So, packages listed here might not appear right away.

  • jEdit, apcalc, ant, antlr added 2009-10-31 (Sam Payson)
  • Java runtime in Firefox --Matt 09:12, 3 February 2010 (EST)
  • Adblock plus in firefox on csguest account --Matt 16:57, 8 February 2010 (EST)
  • Keep history until user logs closes firefox on csguest account --Matt 16:57, 8 February 2010 (EST)
  • Install chromium and add extensions such that browsing experience is the same as Firefox. -- Todd
  • Remove all bittorrent clients. We had some problems last year with people downloading illegal items on lab PCs and removing the clients helps to discourage usage.--Matt 00:25, 3 March 2010 (EST)
  • Enable headphone jack sense.

Next Version

Here is a list of features, etc. that I'm going to add to the next lab build. Please add feature requests to the `Requests' section.

  • update the metapackage
    • (create .debs for?) extra programs: VMWare player (not open source, shenanigans, etc)
    • extra packages:
  • BUG: cleardesktop - directories created are of the correct date, but that looks odd if the script runs after midnight. Maybe just dump everything in a directory and only remove files older than one week?
  • add kiosckar support (non cosi-vr machines can use spice over the internal network to connect to VMs running on the cosi-vr machines)
    • SPICE for Windows XP VM
    • KVM GPL Windows XP Block driver (update) for Windows XP VM
  • Lots of fun network config
    • set static IPs for each machine using list of MACs
    • set up a clonezilla server (drbl) for ease of cloning
    • configure iptables to block all incoming ports (except for specific allowances, such as ssh and http/https)
    • configure SSH to only accept incoming messages on the 128.153.144 subnet - if off-campus go through isengard or polairs
    • /etc/resolv.conf hax (See Matt's requests)
  • Store master list of scripts on web2, have each machine wget the latest version of each before running them

Lab Build Configuration

Installation

WARNING: This guide is partially deprecated. I will remove this warning when all is well once again.

Here are the steps to setting up the lab build. I will assume that the metapackage is already set up and hosted in a repository.

  • Boot a single machine using the standard Ubuntu liveCD. Install to hard drive, creating a 100M /boot partition (I used ext2), 1G of swap, and allocate the rest of the disk as / (I used ext4).
  • Create user csadmin with a secure password; the root and csguest users will be set up later. Complete any post-installation administrative tasks as required by the installer, such as setting the timezone. Be sure to specify that csadmin does NOT log in automatically without entering the password. Set the hostname as cosi-<machine number>, like they're labeled. Get to an installed system in ready state at the desktop.
  • Modify the file /etc/apt/sources.list. This is apt's repository configuration file. It is to be changed such that apt downloads packages from Mirror. Perform the following steps:
  • Run 
    sudo apt-get update
    to update apt's list of installable packages. You can check for errors in sources.list by looking over the output.
  • Run 
    sudo apt-get -y upgrade
    to install the latest version of all default packages. Make sure everything is upgraded to the latest version before continuing.
  • Run 
    sudo apt-get -y install cosi-metapackage
    to install all packages specified by the metapackage. This will take a while! Sun Java forces you to accept a license agreement before it's installed, so you'll have to hang around at least until that point. After that, the process should be automatic, so don't bother hanging around waiting for it. The metapackage will also perform limited configuration steps, such as creating the script directory /etc/cosi-scripts/ (see Metapackage->postinst).

Environment Setup

  • Install programs that can't be handled by the metapackage.
    • Install Alloy Analyzer. Go to http://alloy.mit.edu/alloy4/ and download the latest .jar file for Alloy Analyzer. Store this .jar file in /usr/bin/ and add a launcher to the gnome applications list that executes
      java -jar /usr/bin/alloy4.jar
    • Install yices. Go to http://yices.csl.sri.com/download.shtml and download yices. Copy the directory to /usr/bin and create a symlink from the executable (.../yices-directory/bin/yices to /usr/bin. Now yices can be invoked from any command line!
    • Install piVC. Go to http://theory.stanford.edu/~arbrad/pivc/download/index.html and download the tarfile. Extract it and, following the directions on the webpage, run ./configure to generate a binary. Copy over the yices binary and then move the whole thing to /usr/bin. As with yices, make a nice convenient symlink (and also add a nice convenient menu entry).
  • Make configuration changes that can't be performed by postinst
    • The root user can be enabled from System->Administration->Users and Groups. Create standard unprivileged user csguest (without sudo privileges) and set their account to automatically log in from System->Administration->Login Window.
    • Set up metapackage_update.sh and cleardesktop.sh. Follow the instructions that accompany each script in the `Scripts' section.
    • Set up subversion's .config file so that passwords aren't stored: 
[auth]
store-passwords = no
store-auth-creds = no
    • Remove the irritating console beep with an even more irritating sudo command 
      echo blacklist pcspkr | sudo tee -a /etc/modprobe.d/blacklist.conf > /dev/null
      or if you're not insane root, 
      echo blacklist pcspkr >> /etc/modprobe.d/blacklist.conf
    • Set Firefox homepage to http://www.google.com and set it to ask the user to clear their session data when Firefox is closed. Add convenient bookmark for http://print.cslabs.clarkson.edu.
    • Disable all sounds from System->Preferences->Sound. Also remove GNOME login sound from Startup Programs.
    • If ssh is to be left open, install package denyhosts and modify /etc/denyhosts to your satisfaction.
    • Install the Chromium browser. It can be downloaded as a .deb from http://www.google.com/chrome. Add extensions to mimic Firefox browsing experience.
    • Finally, set up the desktop (widgets, stuff in the taskbars, etc) as you see fit.
  • Create the final clonezilla image
    • Be sure to remove the file /etc/udev/rules.d/70-persistent-net.rules before making the image.

SPICE, the Lab Build, and you

SPICE allows for operation of a remote VM over VDI. This means that a remote VM can be started and used as if it were running on the local system. It includes drivers to be installed on the VM that make the experience as smooth as possible - it can run fullscreen or in a window (like qemu does) with minimal lag/slowdown/choppiness. This section will cover setup of the SPICE client in the lab build.

Installation

The SPICE client can be installed from Adam J. Lincoln's ppa: 

add-apt-repository ppa:adamjlincoln/spicec
apt-get update
apt-get install libqcairo2 libqpixman-1-0
apt-get install spicec

On some systems, the dependencies aren't resolved properly. Check that you don't get any of the following errors:

  • If you get the following error during install: 
    spicec: Depends: libavutil49 but it is not going to be installed
    then install package libavutil49. Notice however that installing this will remove quite few packages that depend on libavcodec52, it should be possible to find replacement packages if needed.
  • If you get the runtime error 
    spicec: error while loading shared libraries: libcelt051.so.0: cannot open shared object file: No such file or directory
    install package libcelt0.

If something goes terribly awry, the SPICE client can be compiled from source. However, the server cannot be built on a 32-bit platform! Why? See the last entry in the SPICE FAQ for an explanation: http://www.spice-space.org/faq.html. The server is not required on the standard lab build, as non-vr-cosi machines are not capable of running a VM (well) anyway.

Configuration

TBD

Scripts

This section will contain scripts useful to the Lab Build.

WARNING: These scripts are also partially deprecated. I'll put the most recent ones up when they become stable, and at that point remove this warning.

Notes

  • Unless stated otherwise, all scripts reside in /etc/cosi-scripts/.

Desktop Cleaning

This script moves all files in /home/csguest/Desktop/ to /home/csguest/cleandesktop/[date]. Only files from the last 7 days are kept.

Setup

To install this, just install the metapackage (or, more specifically, install the package `cleardesktop' present in the COSI respository. Then, add it to csguest's crontab:

  • Create file /etc/cron.allow and add the line csguest followed by a carriage return.
  • As csguest, invoke command `crontab -e'. Add the line 
0 4 * * * /etc/cosi-scripts/cleardesktop.sh

to csguest's crontab, then save and exit. This example will run the script at 4:00 AM daily.

  • Check the permissions. As root or with sudo, run: 
chown csguest.csguest /etc/cosi-scripts/cleardesktop.sh
chmod +x /etc/cosi-scripts/cleardesktop.sh
  • That's it. If you want to test that the script is executing properly, set it to run every minute.
cleardesktop.sh 

#!/bin/bash

# should have used regex
# but my sed-fu is weak, awk
# method is bogus

# deletes any directory that's older than the 7th oldest directory present (in /home/csguest/cleandesktop)
removedirs()
{
        mkdir /home/csguest/cleandesktop_tmp/
        DIRS=`ls /home/csguest/cleandesktop/ -xr -w 10000 | awk '{print $1 " "  $2 " " $3 " " $4 " " $5 " " $6 " " $7}'`
        for dir in $DIRS
        do
                mv /home/csguest/cleandesktop/$dir /home/csguest/cleandesktop_tmp/
        done

        rm -rf /home/csguest/cleandesktop/*

        for dir in $DIRS
        do
                mv /home/csguest/cleandesktop_tmp/$dir /home/csguest/cleandesktop/
        done

        rmdir /home/csguest/cleandesktop_tmp
}

# makes the backup directory containing all files on the Desktop
move_desktop_files()
{
        DATE=`date +%Y-%m-%d`
        mkdir /home/csguest/cleandesktop/$DATE
        mv /home/csguest/Desktop/* /home/csguest/cleandesktop/$DATE/
}

# make cleandesktop dir
# for if it does not exist
# script fails completely

if [ ! -d /home/csguest/cleandesktop ]; then
        mkdir /home/csguest/cleandesktop
fi

# ask the user if they want to stop the script from running
export DISPLAY=:0.0
zenity --question --timeout 60 --title "Danger, Will Robinson!" --text "The Desktop cleaning script is about to run. When it does, all files on the desktop will be moved to /home/csguest/cleandesktop/[today\'s date]. \n\n If you want to STOP the script and leave files on the desktop, hit \'OK\'. Otherwise, hit \'Cancel\' to dismiss this message."

if [ $? == 0 ]; then
        # somebody's still using the machine
        DATE=`date +%Y-%m-%d`
        # not strictly necessary to even create this directory, but it will keep things neat, so
        # there will always be a week's worth of directories even if some are empty
        # (that is, if this computer is not shut off)
        mkdir /home/csguest/cleandesktop/$DATE
        removedirs
        exit
fi

# nobody home, perform the desktop cleaning
move_desktop_files
removedirs

exit

Automatic Updates

This simple script updates all installed packages, and reinstalls the metapackage so as to pull in any new packages that may have been added. Add this to root's crontab, and make sure it's executable.

metapackage_upgrade.sh 

#!/bin/bash

# this script installs new packages added to the metapackge, and upgrades system packages
# purging the metapackage doesn't remove anything but the metapackage!
# ...it's necessary to do this in order for the postinst script to run at every update

/usr/bin/apt-get -y update
/usr/bin/apt-get -y purge cosi-metapackage-karmic
/usr/bin/apt-get -y --force-yes install cosi-metapackage-karmic
/usr/bin/apt-get -y upgrade

exit

Network Setup

words words words

net_setup.sh

This sets static IPs based upon a table of known MAC addresses. It also sets the hostname. 

#!/bin/bash

my_mac=`ifconfig | grep eth0 | awk '{print $5}'`
echo "detected mac $my_mac"

my_hostname=`grep $my_mac mac_addresses.csv | awk -F, '{print $1}'`
echo "my hostname should be set to $my_hostname"


my_eth0_ip=`grep ^$my_hostname mac_addresses.csv | awk -F, '{print $4}'`
my_eth1_ip=`grep ^$my_hostname mac_addresses.csv | awk -F, '{print $5}'`
my_eth0_mac=`grep ^$my_hostname mac_addresses.csv | awk -F, '{print $2}'`
my_eth1_mac=`grep ^$my_hostname mac_addresses.csv | awk -F, '{print $3}'`


cat > /etc/hosts <<EOF
127.0.0.1       localhost
127.0.1.1       $my_hostname
EOF

cat > /etc/hostname <<EOF
$my_hostname
EOF

cat > /etc/network/interfaces <<EOF
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address $my_eth0_ip
gateway 128.153.144.1
netmask 255.255.254.0
nameserver 128.153.0.254
search clarkson.edu

auto eth1
iface eth1 inet static
address $my_eth1_ip
gateway 192.168.44.1
netmask 255.255.254.0

EOF

cat > /etc/udev/rules.d/70-persistent-net.rules <<EOF

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$my_eth0_mac", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="$my_eth1_mac", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

EOF
hostname_setup.sh

This is a subset of the functionality provided by net_setup.sh. It uses DHCP to acquire IPs and sets only the hostname (based on the MAC address). 

#!/bin/bash

macaddrs="/root/bin/mac_addresses.csv"

my_mac=`/sbin/ifconfig -a | grep eth0 | awk '{print $5}'`
echo "detected mac $my_mac"

my_hostname=`grep $my_mac $macaddrs | awk -F, '{print $1}'`
echo "my hostname should be set to $my_hostname"

cat > /etc/hosts <<EOF
127.0.0.1       localhost
127.0.1.1       $my_hostname
EOF

cat > /etc/hostname <<EOF
$my_hostname
EOF

exit
mac_addresses.csv
cosi-01,00:11:25:f6:15:22,00:14:6c:2e:49:e0,128.153.144.141,10.0.0.51
cosi-02,00:11:25:f6:5c:13,00:0f:b5:fd:a4:00,128.153.144.142,10.0.0.52
cosi-03,00:11:25:f6:52:95,00:0f:b5:fe:4b:e5,128.153.144.143,10.0.0.53
cosi-04,00:11:25:f6:5d:c9,00:0f:b5:fb:77:da,128.153.144.144,10.0.0.54
cosi-05,00:11:25:f6:70:7a,00:0f:b5:8f:9d:67,128.153.144.145,10.0.0.55
cosi-06,00:11:25:f6:5d:7c,00:0f:b5:fb:9f:81,128.153.144.146,10.0.0.56
cosi-07,00:11:25:f6:70:8c,00:0f:b5:f8:75:db,128.153.144.147,10.0.0.57
cosi-08,00:11:25:f6:67:9c,00:0f:b5:47:a7:d5,128.153.144.148,10.0.0.58
cosi-09,00:11:25:f6:2f:c1,00:14:6c:2e:47:df,128.153.144.149,10.0.0.59
cosi-10,00:11:25:f6:5b:2e,00:14:6c:2e:49:e0,128.153.144.150,10.0.0.60
vr-cosi-01,00:1a:a0:a9:62:88,00:18:4d:f0:0f:a4,128.153.144.151,10.0.0.61
vr-cosi-02,00:1a:a0:a9:f4:f1,00:18:4d:f0:0e:43,128.153.144.152,10.0.0.62
vr-cosi-03,00:1a:a0:a9:f2:81,00:18:4d:f0:19:07,128.153.144.153,10.0.0.63
vr-cosi-04,00:1a:a0:a9:f5:32,00:18:4d:f0:0a:4a,128.153.144.154,10.0.0.64
vr-cosi-05,00:1a:a0:a9:f5:0a,00:18:4d:f0:11:17,128.153.144.155,10.0.0.65
vr-cosi-06,00:1a:a0:a9:60:e2,00:18:4d:f0:10:f5,128.153.144.156,10.0.0.66

Postinst

This is the postinst for the cosi-metapackage. All it does is check against a file to see if post-installation tasks have been performed yet. It is used when setting up the metapackage. 

#!/bin/bash

# script directory
D="/etc/cosi-scripts/"

# file to check against
F="/etc/cosi-scripts/postinst_check"

# check if these exist
if [ ! -d "$D" ]; then
        mkdir $D
fi

if [ ! -e "$F" ]; then
        touch $F
fi

# change default jre to sun java
S=`/bin/cat $F | grep sun-java6-jdk`
if [ "$S" != "sun-java6-jdk" ]; then
        update-java-alternatives -s java-6-sun
        echo sun-java6-jdk >> $F
fi


# NuSMV requires this symlink
S=`/bin/cat $F | grep nusmv`
if [ "$S" != "nusmv" ]; then
        ln -s /usr/lib/libexpat.so /usr/lib/libexpat.so.0
        echo nusmv >> $F
fi

exit

The Metapackage

Overview

The metapackage is nothing but a Debian package (that is, it's installable with apt-get) that itself installs other packages. The idea is that, rather than expend lots of bandwidth and cause premature wear on hard drives by re-cloning every time packages are added to the lab build, each machine can keep itself up to date by reinstalling the metapackage (which itself can be easily updated to contain the new packages).

Benefits

  • Cloning takes time and puts strain on the hardware if performed frequently. Using a metapackage, each machine will only perform the minimum amount of work necessary to stay updated, and need not be taken out of commission while cloning occurs.
  • The metapackage itself is a convenient list of the software composing the lab build.

Drawbacks

  • The machines will still have to be cloned if there are non-trivial configuration changes.
  • Some software is not distributed as a Debian package and cannot be included in the metapackage. Such software must be handled on an individual basis and cloned out if a .deb cannot be created.

The Current Metapackage

Right now, the metapackage is hosted on web2: http://lab-build.cslabs.clarkson.edu/apt/. The directory is: /var/lab_build-www/apt.

Setup

There are two main components of the metapackage system. First is the metapackage itself, which will be created with dpkg. Second is the repository, which will be created and managed with reprepro.

Creating a Metapackage

The general idea is to create a directory structure that can be understood by dpkg, then to write the control (and postinst) files to describe the metapackage. Finally, dpkg will be used to create a .deb file.

  • Create the following directory structure: 
    ../cosi-metapackage-<version>/DEBIAN
  • Inside the DEBIAN directory, create file control. This textfile will describe the metapackage. A sample control file is given below: 
Package: cosi-metapackage
Version: <version number>
Section: main
Priority: optional
Architecture: i386
Depends: list, of, packages, to, install, here
Conflicts: packages, to, be, removed
Replaces: packages, to, be, removed
Maintainer: <name>
Description: Text describing the package
  • To add packages to the metapackage, list them under `Depends'. To ensure that certain packages are not installed, or to remove them, add the package name to BOTH `Conflicts' and `Replaces'.
  • Create file (postinst) in the DEBIAN directory. This file will be executed as a script after all of the packages in `Depends' have been installed. It will be used to configure packages that might not be ready-to-run right after being installed. If any packages need this extra step, add the relevant bash commands to this script. A copy of the current postinst can be found in the scripts section.
  • Return to the directory two levels above DEBIAN. If you don't, dpkg will still work if you give it the relative pathname to directory cosi-metapackage-<version>/.
  • Run 
    dpkg --build cosi-metapackage-<version>/ cosi-metapackage-<version>.deb
    which will create in the working directory the .deb file to added to the repository.

Creating a Repository

  • Create a directory for the repository to use. Inside it, create directory conf. Inside that, create file distributions.
  • Fill out the distributions file with information defining the repository. A sample (the current distributions file) is given below: 
Origin: <maintainer name>
Label: COSI Lab Build repository
Suite: stable
Codename: karmic
Version: 9.10
Architectures: i386
Components: main
Description: Text description
  • The `Components' field must match the metapackage .deb's control file, otherwise you won't be able to add it to the repository.
  • From the base repository directory, run 
    sudo reprepro includedeb <version> /path/to/cosi-metapackage-<version>.deb
  • If all goes well, you can now point the sources.list file to the base repository directory, and be able to install the metapackage using apt.
  • If a package must be removed from the repository, run
    sudo reprepro remove <version> <packagename>
Retrieved from "http://docs.cslabs.clarkson.edu/wiki/COSI_Linux_Build"
Personal tools