Difference between revisions of "Old Mirror"

From CSLabsWiki
(→‎watcherMailer)
Line 316: Line 316:
   
 
=== watcherMailer ===
  
=== watcherMailer ===
−
An oddly named script that watches for certain conditions to occur on mirror then emails an alert to the Mirror Admin. Kind of like a home brewed Nagios. Obviously you need to define the variables first. Runs as a cronjob every 10 minutes.
+
An oddly named script that watches for certain conditions to occur on mirror then emails an alert to the Mirror Admin. Kind of like a home brewed Nagios. Obviously you need to define the variables first. Runs as a cronjob every 10 minutes. It is currently living in Chris Peterman's home directory seeing as it doesn't need root privs to run.
 
<source lang="bash">
  
<source lang="bash">
 
#!/bin/bash
  
#!/bin/bash

Revision as of 16:41, 28 January 2008

Emblem-important.png Please help improve this article or section by expanding it.
Further information might be found on the talk page. (This article has been tagged since: October 3, 2007).
Hostname: mirror.clarkson.edu
Status: Running


Mirror is an open source projects mirror that mirrors different Linux distributions (see below) and the Linux kernel itself. It can be accessed through a web interface or by anonymous FTP.

Mirrored Software

  • ArchLinux
  • Fedora (7,8, x86, no SRPMS)
  • Kernel.org
  • Ubuntu
  • Debian (i386, ia64, amd64)

Scripts

Mirror is maintained by many small scripts, this page documents them all. The following are located in the directory: /etc/mirrorscripts

rsyncVars

This scriptlet defines all the variables used in the scripts that actually do the mirroring.

<source lang="bash">## Master Include File for the Mirror Rsync Scripts

    1. DISTRO MUST BE DEFINED BEFORE SOURCING THIS FILE!
    1. Define program paths

TOUCH=/usr/bin/touch DATE=/bin/date RSYNC=/usr/bin/rsync CHGRP=/bin/chgrp RM=/bin/rm

    1. Define File Paths

LOGFILE=/var/log/mirrorlogs/${DISTRO}-$(${DATE} +%m-%d-%y) LOCKFILE=/var/lock/${DISTRO}.lck DEST=/aoedisk/${DISTRO}

    1. Rsync Options

RSYNCOPTS='-avrH --partial --stats --delay-updates --delete --delete-after' </source>

template

This is the master template for all the new mirror scripts (meaning that Archlinux's and Kernel.org's don't follow this exactly). Just plugin URL and DISTRO and you are good to go.

<source lang="bash">#!/bin/bash -

    1. Template Mirror Script for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=

    1. Define the URL which we rsync from

URL=

    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then 

       ## No one else is running, put down a lockfile so no one interupts us
       ${TOUCH} ${LOCKFILE}

       ## Begin the Rsync, send all output to the logfiles
       ${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}

       ## Change the group ownership of the log files to Mirrorlogs
       ${CHGRP} mirrorlogs ${LOGFILE}

       ## Remove the lockfile, we are done
       ${RM} ${LOCKFILE}

       ## Send a good exit code
       exit 0

else 

       ## Someone else was running. Not really an error so we exit with a successful error code
       exit 0

fi </source>

archlinux

This is Archlinux's rsync script. It was created before the prototype, and is a little different than others.

<source lang="bash">#!/bin/bash -

    1. Define which distro we are mirroring

DISTRO=archlinux

    1. Load all the variables from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile to make sure no one interrupts us ${TOUCH} ${LOCKFILE} ## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} distro.ibiblio.org::distros/archlinux/ ${DEST} > ${LOGFILE} ## Change the group ownership of the log files to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE} ## Remove the lockfile, we are done ${RM} ${LOCKFILE} ## Send a good exit code exit 0 else ## Someone else was running. Not really an error so we exit with a successful error code exit 0 fi </source>

linux

This is the script for Kernel.org. It is notable because we actually rsync two different modules at once.

<source lang="bash">#!/bin/bash -

    1. Define which distro we are mirroring

DISTRO=linux

    1. Load all the variables from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile to make sure no one interrupts us ${TOUCH} ${LOCKFILE}

## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/linux/ ${DEST} > ${LOGFILE} ${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/software/ /aoedisk/software >> ${LOGFILE}

## Change the group ownership of the logs to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE}

## We are done, remove the lockfile ${RM} ${LOCKFILE}

## Send a clean exit code exit 0 else ## Another copy of the script was running, not really an error on our part, so send a successful error code exit 0 fi </source>

ubuntu

This is ubuntu's mirrorscript. First one based off the template, but different because we rsync another module to a different dest.

<source lang="bash">

  1. !/bin/bash -
    1. Ubuntu Mirror Script for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=ubuntu

    1. Define the URL which we rsync from

URL='mirrors.rit.edu::ubuntu'

    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile so no one interupts us ${TOUCH} ${LOCKFILE}

## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}

## Rsync up the ISOs ${RSYNC} ${RSYNCOPTS} mirrors.rit.edu::ubuntu-releases /aoedisk/isos/ubuntu >> ${LOGFILE}

## Change the group ownership of the log files to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE}

## Remove the lockfile, we are done ${RM} ${LOCKFILE}

## Send a good exit code exit 0 else ## Someone else was running. Not really an error so we exit with a successful error code exit 0 fi </source>

fedora

Fedora's Mirrorscript. Again two rsyncs back to back, and a mess of excludes to refine what we get to save on disk space

<source lang='bash'>

  1. !/bin/bash -
    1. Fedora Mirrorscript for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=fedora

    1. Define the URL which we rsync from
  2. URL=
    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile so no one interupts us ${TOUCH} ${LOCKFILE}

## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" mirrors.kernel.org::fedora/releases ${DEST} > ${LOGFILE}

## Second RSync for the Updates repo ${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*SRPMS*" --exclude "*debug*" mirrors.kernel.org::fedora/updates ${DEST} >> ${LOGFILE}

## Change the group ownership of the log files to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE}

## Remove the lockfile, we are done ${RM} ${LOCKFILE}

## Send a good exit code exit 0 else ## Someone else was running. Not really an error so we exit with a successful error code exit 0 fi </source>

debian

Debian's mirrorscript. Excludes a plenty for Arch's we don't use

<source lang='bash'>

  1. !/bin/bash -
    1. Debian Mirror Script for mirror.clarkson.edu
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Define which distro we are mirroring here

DISTRO=debian

    1. Define the URL which we rsync from

URL=mirrors.usc.edu::debian

    1. Load all the variable from the master include file

source /etc/mirrorscripts/rsyncVars

    1. Make sure another instance of this script isn't running right now

if [ ! -e ${LOCKFILE} ] then ## No one else is running, put down a lockfile so no one interupts us ${TOUCH} ${LOCKFILE}

## Begin the Rsync, send all output to the logfiles ${RSYNC} ${RSYNCOPTS} --exclude binary-alpha/ --exclude *_alpha.deb --exclude binary-arm/ --exclude *_arm.deb --exclude binary-hppa/ --exclude *_hppa.deb --exclude binary-hurd-i386/ --exclude *_hurd-i386.deb --exclude binary-m68k/ --exclude *_m68k.deb --exclude binary-mips/ --exclude *_mips.deb --exclude binary-mipsel/ --exclude *_mipsel.deb --exclude binary-powerpc/ --exclude *_powerpc.deb --exclude binary-s390/ --exclude *_s390.deb --exclude binary-sparc/ --exclude *_sparc.deb ${URL} ${DEST} > ${LOGFILE}

## Change the group ownership of the log files to Mirrorlogs ${CHGRP} mirrorlogs ${LOGFILE}

## Remove the lockfile, we are done ${RM} ${LOCKFILE}

## Send a good exit code exit 0 else ## Someone else was running. Not really an error so we exit with a successful error code exit 0 fi </source>

mirrorLogRotate

This is located in /etc/cron.monthly and rotates the logs in /var/log/mirrorlogs

<source lang="bash">

  1. !/bin/bash -
    1. Log Rotation Script for mirror.clarkson.edu's rsync logs
    2. Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
    3. Licensed under the GPLv2
    1. Set month number to LAST month

(( month = $(/bin/date +%m) - 1 ))

    1. Account for January being 1 and December being 12. There is no month
    2. 0, at least as returned by /bin/date

if [ ${month} -eq 0 ] then month=12 fi

    1. Set the year

year=$(/bin/date +%y)

    1. Set the mirror log directory

logDir=/var/log/mirrorlogs

for distro in archlinux fedora ubuntu centos gentoo debian do ## Archive the logs from last month, then delete them (NOT the Archive) /bin/tar -czvf /tmp/${distro}-${month}-${year}.tgz ${logDir}/${distro}-${month}-*-${year} && /bin/rm -v ${logDir}/${distro}-${month}-*-${year}

## Move the log Tarball back to the logdir /bin/mv /tmp/${distro}-${month}-${year}.tgz ${logDir} done

exit 0 </source>

watcherMailer

An oddly named script that watches for certain conditions to occur on mirror then emails an alert to the Mirror Admin. Kind of like a home brewed Nagios. Obviously you need to define the variables first. Runs as a cronjob every 10 minutes. It is currently living in Chris Peterman's home directory seeing as it doesn't need root privs to run. <source lang="bash">

  1. !/bin/bash
  2. Watcher Script Script for mirror
  3. Watches for a set critia and then emails the DESTEMAIL if said conditions are met
  4. Designed to be run out of a cronjob
  5. Copyright 2008 Chris Peterman <kyral@azuredreams.us>
  6. Licensed under the GPLv2

MAILFROM= MAILSERV= MAILUSER= PASSWORD= DESTEMAIL=

    1. We are going to do this with a function. 1st argument is the subject line, second is the body

sendEmail() { 

       /usr/bin/sendEmail -f ${MAILFROM} -t ${DESTEMAIL} -u ${1} -m ${2} -s ${MAILSERV} -xu ${MAILUSER} -xp ${PASSWORD}

}

    1. Start test conditions
    1. Test to make sure that aoedisk is MOUNTED

if [ $(/bin/mount | /bin/grep aoedisk &> /dev/null; echo $?) -ne 0 ] then 

       sendEmail "ALERT: aoedisk" "aoedisk is umounted on mirror!"

fi

    1. Test to make sure various services are running

for i in apache2 proftpd rsyncd sshd do 

       if [ ! -e /var/run/${i}.pid ]
       then
               sendEmail "ALERT: ${i}" "${i} isn't running on mirror!"
       fi

done

exit 0 </source>

Retrieved from "http://docs.cslabs.clarkson.edu/mediawiki/index.php?title=Old_Mirror&oldid=980"