Old Mirror

From CSLabsWiki
Revision as of 11:42, 4 April 2008 by Petermcv (talk | contribs) (Mirrored Software)
Hostname: mirror.clarkson.edu
LDAP Support: No
Development Status: Stable
Status: Running


Mirror is an open source projects mirror that mirrors different Linux distributions (see below) and the Linux kernel itself. It can be accessed through a web interface or by anonymous FTP.

Mirrored Software

Rsync Timetable

This is the table for when Rsyncs start. Only two rsyncs may start at once so that we don't overload the people we are mirroring from

Time Rsync 1 Rsync 2
01:00 Fedora Jailtime.org
02:00 Kernel.org Archlinux
03:00 CentOS Ubuntu
04:00 Debian KDE

Scripts

Mirror is maintained by many small scripts, this page documents them all. The following are located in the directory: /etc/mirrorscripts

rsyncVars

This scriptlet defines all the variables used in the scripts that actually do the mirroring.

<source lang="bash">

    1. Master Include File for the Mirror Rsync Scripts
    2. DISTRO MUST BE DEFINED BEFORE SOURCING THIS FILE!
    1. Define program paths

TOUCH=/usr/bin/touch DATE=/bin/date RSYNC=/usr/bin/rsync CHGRP=/bin/chgrp RM=/bin/rm MKDIR=/bin/mkdir

    1. Define File Paths

LOGFILE=/var/log/mirrorlogs/${DISTRO}-$(${DATE} +%m-%d-%y) LOCKFILE=/var/lock/${DISTRO}.lck DEST=/aoedisk/${DISTRO}

    1. Rsync Options

RSYNCOPTS='-avrH --partial --stats --delay-updates --delete --delete-after'

</source>

template

This is the master template for all the new mirror scripts (meaning that Archlinux's and Kernel.org's don't follow this exactly). Just plugin URL and DISTRO and you are good to go.

<source lang="bash">#!/bin/bash -

  1. !/bin/bash -
    1. Template Mirror Script for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=

    1. Define the URL which we rsync from

URL=

    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then

       ## No one else is running, put down a lockfile so no one interupts us
       ${TOUCH} ${LOCKFILE}
       ## Make sure DEST exists
       if [ ! -d ${DEST} ]; then
               ${MKDIR} ${DEST}
       fi
       ## Begin the Rsync, send all output to the logfiles
       ${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
       ## Change the group ownership of the log files to Mirrorlogs
       ${CHGRP} mirrorlogs ${LOGFILE}
       ## Remove the lockfile, we are done
       ${RM} ${LOCKFILE}
       ## Send a good exit code
       exit 0

else

       ## Someone else was running. Not really an error so we exit with a successful error code
       exit 0

fi

</source>

archlinux

This is Archlinux's rsync script. It was created before the prototype, and is a little different than others.

<source lang="bash">#!/bin/bash -

    1. Define which distro we are mirroring

DISTRO=archlinux

    1. Load all the variables from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile to make sure no one interrupts us ${TOUCH} ${LOCKFILE} ## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} distro.ibiblio.org::distros/archlinux/ ${DEST} > ${LOGFILE} ## Change the group ownership of the log files to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE} ## Remove the lockfile, we are done ${RM} ${LOCKFILE} ## Send a good exit code exit 0 else ## Someone else was running. Not really an error so we exit with a successful error code exit 0 fi </source>

linux

This is the script for Kernel.org. It is notable because we actually rsync two different modules at once.

<source lang="bash">#!/bin/bash -

    1. Define which distro we are mirroring

DISTRO=linux

    1. Load all the variables from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile to make sure no one interrupts us ${TOUCH} ${LOCKFILE}

## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/linux/ ${DEST} > ${LOGFILE} ${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/software/ /aoedisk/software >> ${LOGFILE}

## Change the group ownership of the logs to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE}

## We are done, remove the lockfile ${RM} ${LOCKFILE}

## Send a clean exit code exit 0 else ## Another copy of the script was running, not really an error on our part, so send a successful error code exit 0 fi </source>

ubuntu

This is ubuntu's mirrorscript. First one based off the template, but different because we rsync another module to a different dest.

<source lang="bash">

  1. !/bin/bash -
    1. Ubuntu Mirror Script for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=ubuntu

    1. Define the URL which we rsync from

URL='mirrors.rit.edu::ubuntu'

    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile so no one interupts us ${TOUCH} ${LOCKFILE}

## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}

## Rsync up the ISOs ${RSYNC} ${RSYNCOPTS} mirrors.rit.edu::ubuntu-releases /aoedisk/isos/ubuntu >> ${LOGFILE}

## Change the group ownership of the log files to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE}

## Remove the lockfile, we are done ${RM} ${LOCKFILE}

## Send a good exit code exit 0 else ## Someone else was running. Not really an error so we exit with a successful error code exit 0 fi </source>

fedora

Fedora's Mirrorscript. Again two rsyncs back to back, and a mess of excludes to refine what we get to save on disk space

<source lang='bash'>

  1. !/bin/bash -
    1. Fedora Mirrorscript for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=fedora

    1. Define the URL which we rsync from
  1. URL=
    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile so no one interupts us ${TOUCH} ${LOCKFILE}

## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" mirrors.kernel.org::fedora/releases ${DEST} > ${LOGFILE}

## Second RSync for the Updates repo ${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*SRPMS*" --exclude "*debug*" mirrors.kernel.org::fedora/updates ${DEST} >> ${LOGFILE}

## Change the group ownership of the log files to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE}

## Remove the lockfile, we are done ${RM} ${LOCKFILE}

## Send a good exit code exit 0 else ## Someone else was running. Not really an error so we exit with a successful error code exit 0 fi </source>

debian

Debian's mirrorscript. Excludes a plenty for Arch's we don't use

<source lang='bash'>

  1. !/bin/bash -
    1. Debian Mirror Script for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=debian

    1. Define the URL which we rsync from

URL=mirrors.kernel.org::debian

    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then

       ## No one else is running, put down a lockfile so no one interupts us
       ${TOUCH} ${LOCKFILE}
       ## Begin the Rsync, send all output to the logfiles
       ${RSYNC} ${RSYNCOPTS} --exclude binary-alpha/ --exclude *_alpha.deb --exclude binary-arm/ --exclude *_arm.deb --exclude binary-hppa/ --exclude *_hppa.deb --exclude binary-hurd-i386/ --exclude *_hurd-i386.deb --exclude binary-m68k/ --exclude *_m68k.deb --exclude binary-mips/ --exclude *_mips.deb --exclude binary-mipsel/ --exclude *_mipsel.deb --exclude binary-powerpc/ --exclude *_powerpc.deb --exclude binary-s390/ --exclude *_s390.deb --exclude binary-sparc/ --exclude *_sparc.deb ${URL} ${DEST} > ${LOGFILE}
       ${RSYNC} ${RSYNCOPTS} mirrors.kernel.org::debian-cd ${DEST} >> ${LOGFILE}
       ## Change the group ownership of the log files to Mirrorlogs
       ${CHGRP} mirrorlogs ${LOGFILE}
       ## Remove the lockfile, we are done
       ${RM} ${LOCKFILE}
       ## Send a good exit code
       exit 0

else

       ## Someone else was running. Not really an error so we exit with a successful error code
       exit 0

fi

</source>

CentOS

This is CentOS's mirrorscript and it was based off of Fedora's <source lang="bash">

  1. !/bin/bash -
    1. CentOS Mirrorscript for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=centos

    1. Define the URL which we rsync from
  1. URL=
    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then

       ## No one else is running, put down a lockfile so no one interupts us
       ${TOUCH} ${LOCKFILE}
       ## Make sure the dest directory exists!
       if [ ! -d ${DEST} ]; then
               ${MKDIR} ${DEST}
       fi
       ## Begin the Rsync, send all output to the logfiles
       ${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" --exclude "*SRPMS*" mirrors.rit.edu::centos ${DEST} > ${LOGFILE}
       ## Change the group ownership of the log files to Mirrorlogs
       ${CHGRP} mirrorlogs ${LOGFILE}
       ## Remove the lockfile, we are done
       ${RM} ${LOCKFILE}
       ## Send a good exit code
       exit 0

else

       ## Someone else was running. Not really an error so we exit with a successful error code
       exit 0

fi </source>

Jailtime.org

This is the mirrorscript for Jailtime.org. I think this is the first one that didn't need massive editing from the Template..

<source lang="bash">

  1. !/bin/bash -
    1. Jailtime.org Mirror Script for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO='jailtime.org'

    1. Define the URL which we rsync from

URL='jailtime.org::jailtime'

    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then

       ## No one else is running, put down a lockfile so no one interupts us
       ${TOUCH} ${LOCKFILE}
       ## Begin the Rsync, send all output to the logfiles
       ${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
       ## Change the group ownership of the log files to Mirrorlogs
       ${CHGRP} mirrorlogs ${LOGFILE}
       ## Remove the lockfile, we are done
       ${RM} ${LOCKFILE}
       ## Send a good exit code
       exit 0

else

       ## Someone else was running. Not really an error so we exit with a successful error code
       exit 0

fi </source>

mirrorLogRotate

This is located in /etc/cron.monthly and rotates the logs in /var/log/mirrorlogs

<source lang="bash">

  1. !/bin/bash -
    1. Log Rotation Script for mirror.clarkson.edu's rsync logs
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2


    1. Set month number to LAST month

(( month = $(/bin/date +%m) - 1 ))

    1. Account for January being 1 and December being 12. There is no month
    2. 0, at least as returned by /bin/date

if [ ${month} -eq 0 ] then

       month=12

fi

    1. Hack the month setting to give it back two digits

case "$month" in

       "1") month=01;;
       "2") month=02;;
       "3") month=03;;
       "4") month=04;;
       "5") month=05;;
       "6") month=06;;
       "7") month=07;;
       "8") month=08;;
       "9") month=09;;

esac

    1. Set the year to the current year unless the month is 12 (December)
    2. in that case set it to the PREVIOUS year

if [ ${month} -eq 12 ] then

       (( year = $(/bin/date +%y) - 1 ))

else

       year=$(/bin/date +%y)

fi

    1. Set the mirror log directory

logDir=/var/log/mirrorlogs

for distro in archlinux fedora ubuntu debian linux centos do

       ## Archive the logs from last month, then delete them (NOT the Archive)
       /bin/tar -czvf /tmp/${distro}-${month}-${year}.tgz ${logDir}/${distro}-${month}-*-${year} && /bin/rm -v ${logDir}/${distro}-${month}-*-${year}
       ## Move the log Tarball back to the logdir
       /bin/mv /tmp/${distro}-${month}-${year}.tgz ${logDir}

done

exit 0


</source>

watcherMailer

An oddly named script that watches for certain conditions to occur on mirror then emails an alert to the Mirror Admin. Kind of like a home brewed Nagios. Obviously you need to define the variables first. Runs as a cronjob every 10 minutes. It is currently living in Chris Peterman's home directory seeing as it doesn't need root privs to run. <source lang="bash">

  1. !/bin/bash
  2. Watcher Script Script for mirror
  3. Watches for a set critia and then emails the DESTEMAIL if said conditions are met
  4. Designed to be run out of a cronjob
  5. Copyright 2008 Chris Peterman <kyral@azuredreams.us>
  6. Licensed under the GPLv2

MAILFROM= MAILSERV= MAILUSER= PASSWORD= DESTEMAIL=

    1. We are going to do this with a function. 1st argument is the subject line, second is the body

sendEmail() {

       /usr/bin/sendEmail -f ${MAILFROM} -t ${DESTEMAIL} -u ${1} -m ${2} -s ${MAILSERV} -xu ${MAILUSER} -xp ${PASSWORD}

}

    1. Start test conditions
    1. Test to make sure that aoedisk is MOUNTED

if [ $(/bin/mount | /bin/grep aoedisk &> /dev/null; echo $?) -ne 0 ] then

       sendEmail "ALERT: aoedisk" "aoedisk is umounted on mirror!"

fi

    1. Test to make sure various services are running

for i in apache2 proftpd rsyncd sshd do

       if [ ! -e /var/run/${i}.pid ]
       then
               sendEmail "ALERT: ${i}" "${i} isn't running on mirror!"
       fi

done

exit 0 </source>