Difference between revisions of "Old Mirror"

From CSLabsWiki
m (moved to running)
m (move to be machine)
Line 1: Line 1:
{{VM
+
{{Machine
|hostname=[http://mirror.clarkson.edu mirror.clarkson.edu], [http://kernelmirror.clarkson.edu kernelmirror.clarkson.edu], [http://mirror.cslabs.clarkson.edu mirror.cslabs.clarkson.edu], & [http://kernelmirror.cslabs.clarkson.edu kernelmirror.cslabs.clarkson.edu]
+
|hostname=mirror.clarkson.edu mirror.cslabs.clarkson.edu mirror.int.cslabs.clarkson.edu mirror.sr.cslabs.clarkson.edu
  +
|interface1={{Network Interface |name=Clarkson Network |mac=00:1B:21:28:C8:4E |ip=128.153.145.19}}
|ldap=No
 
  +
|interface2={{Network Interface |name=Server Room Network |mac=00:E0:81:B5:88:85 |ip=10.0.1.}}
|development_status=Stable
 
  +
|interface3={{Network Interface |name=Internal Network |mac=00:E0:81:B5:88:84 |ip=10.0.0.17}}
|status=Running
 
|maintainer=[[User:petermcv|Chris Peterman]]
+
|maintainer=[[User:mccarrms|Matt McCarrell]]
|dom0=hydrogen
 
 
}}
 
}}
  +
'''Mirror''' is an open source projects mirror that mirrors different Linux distributions and the Linux kernel itself. It can be accessed through via http or ftp.
   
  +
Mirror is the 3U server located in the [[Unisys Rack]] below [[Xen1]].
'''Mirror''' is an open source projects mirror that mirrors different Linux distributions (see below) and the Linux kernel itself. It can be accessed through a web interface or by anonymous FTP.
 
   
  +
Mirror runs CentOS 5.
==Mirrored Software==
 
*[http://www.archlinux.org ArchLinux]
 
*[http://www.fedoraproject.org Fedora] (7 and up, x86, no SRPMS)
 
*[http://www.kernel.org Kernel.org]
 
*[http://www.ubuntu.com Ubuntu]
 
*[http://www.debian.org Debian] (i386, ia64, amd64)
 
*[http://www.centos.org CentOS] (5 and up, x86, no SRPMS)
 
*[http://jailtime.org Jailtime.org]
 
   
==Rsync Timetable==
+
==Setup Process==
  +
*[[Mirror Setup Process]]
This is the table for when Rsyncs start. Only two rsyncs may ''start'' at once so that we don't overload the people we are mirroring from
 
{|
 
!Time !! Rsync 1 !! Rsync 2
 
|-
 
|'''01:00''' || Fedora || Jailtime.org
 
|-
 
|'''02:00''' || Kernel.org || Archlinux
 
|-
 
|'''03:00''' || CentOS || Ubuntu
 
|-
 
|'''04:00''' || Debian ||
 
|}
 
   
  +
==Technical Specifications==
==Scripts==
 
Mirror is maintained by many small scripts, this page documents them all. The following are located in the directory: ''/etc/mirrorscripts''
 
   
  +
[[Category:Documentation]]
=== rsyncVars ===
 
  +
[[Category:Infrastructure]]
This scriptlet defines all the variables used in the scripts that actually do the mirroring.
 
 
<source lang="bash">
 
## Master Include File for the Mirror Rsync Scripts
 
## DISTRO MUST BE DEFINED BEFORE SOURCING THIS FILE!
 
 
## Define program paths
 
TOUCH=/usr/bin/touch
 
DATE=/bin/date
 
RSYNC=/usr/bin/rsync
 
CHGRP=/bin/chgrp
 
RM=/bin/rm
 
MKDIR=/bin/mkdir
 
 
## Define File Paths
 
LOGFILE=/var/log/mirrorlogs/${DISTRO}-$(${DATE} +%m-%d-%y).log
 
LOCKFILE=/var/lock/${DISTRO}.lck
 
DEST=/aoedisk/${DISTRO}
 
 
## Rsync Options
 
RSYNCOPTS='-avrH --partial --stats --delay-updates --delete --delete-after'
 
 
</source>
 
 
=== template ===
 
This is the master template for all the new mirror scripts (meaning that Archlinux's and Kernel.org's don't follow this exactly). Just plugin URL and DISTRO and you are good to go.
 
 
<source lang="bash">#!/bin/bash -
 
#!/bin/bash -
 
## Template Mirror Script for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=
 
 
## Define the URL which we rsync from
 
URL=
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Make sure DEST exists
 
if [ ! -d ${DEST} ]; then
 
${MKDIR} ${DEST}
 
fi
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
 
</source>
 
 
=== archlinux ===
 
This is Archlinux's rsync script. It was created before the prototype, and is a little different than others.
 
 
<source lang="bash">#!/bin/bash -
 
 
## Define which distro we are mirroring
 
DISTRO=archlinux
 
 
## Load all the variables from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile to make sure no one interrupts us
 
${TOUCH} ${LOCKFILE}
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} distro.ibiblio.org::distros/archlinux/ ${DEST} > ${LOGFILE}
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== linux ===
 
This is the script for Kernel.org. It is notable because we actually rsync two different modules at once.
 
 
<source lang="bash">#!/bin/bash -
 
 
## Define which distro we are mirroring
 
DISTRO=linux
 
 
## Load all the variables from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile to make sure no one interrupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/linux/ ${DEST} > ${LOGFILE}
 
${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/software/ /aoedisk/software >> ${LOGFILE}
 
 
## Change the group ownership of the logs to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## We are done, remove the lockfile
 
${RM} ${LOCKFILE}
 
 
## Send a clean exit code
 
exit 0
 
else
 
## Another copy of the script was running, not really an error on our part, so send a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== ubuntu ===
 
This is ubuntu's mirrorscript. First one based off the template, but different because we rsync another module to a different dest.
 
 
<source lang="bash">
 
#!/bin/bash -
 
## Ubuntu Mirror Script for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=ubuntu
 
 
## Define the URL which we rsync from
 
URL='mirrors.rit.edu::ubuntu'
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
 
 
## Rsync up the ISOs
 
${RSYNC} ${RSYNCOPTS} mirrors.rit.edu::ubuntu-releases /aoedisk/isos/ubuntu >> ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== fedora ===
 
Fedora's Mirrorscript. Again two rsyncs back to back, and a mess of excludes to refine what we get to save on disk space
 
 
<source lang='bash'>
 
#!/bin/bash -
 
## Fedora Mirrorscript for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=fedora
 
 
## Define the URL which we rsync from
 
#URL=
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" mirrors.kernel.org::fedora/releases ${DEST} > ${LOGFILE}
 
 
## Second RSync for the Updates repo
 
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*SRPMS*" --exclude "*debug*" mirrors.kernel.org::fedora/updates ${DEST} >> ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== debian ===
 
Debian's mirrorscript. Excludes a plenty for Arch's we don't use
 
 
<source lang='bash'>
 
#!/bin/bash -
 
## Debian Mirror Script for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=debian
 
 
## Define the URL which we rsync from
 
URL=mirrors.kernel.org::debian
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} --exclude binary-alpha/ --exclude *_alpha.deb --exclude binary-arm/ --exclude *_arm.deb --exclude binary-hppa/ --exclude *_hppa.deb --exclude binary-hurd-i386/ --exclude *_hurd-i386.deb --exclude binary-m68k/ --exclude *_m68k.deb --exclude binary-mips/ --exclude *_mips.deb --exclude binary-mipsel/ --exclude *_mipsel.deb --exclude binary-powerpc/ --exclude *_powerpc.deb --exclude binary-s390/ --exclude *_s390.deb --exclude binary-sparc/ --exclude *_sparc.deb ${URL} ${DEST} > ${LOGFILE}
 
 
${RSYNC} ${RSYNCOPTS} mirrors.kernel.org::debian-cd ${DEST} >> ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
 
</source>
 
 
=== CentOS ===
 
This is CentOS's mirrorscript and it was based off of Fedora's
 
<source lang="bash">
 
#!/bin/bash -
 
## CentOS Mirrorscript for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=centos
 
 
## Define the URL which we rsync from
 
#URL=
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Make sure the dest directory exists!
 
if [ ! -d ${DEST} ]; then
 
${MKDIR} ${DEST}
 
fi
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" --exclude "*SRPMS*" mirrors.rit.edu::centos ${DEST} > ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== Jailtime.org ===
 
This is the mirrorscript for Jailtime.org. I think this is the first one that didn't need massive editing from the Template..
 
 
<source lang="bash">
 
#!/bin/bash -
 
## Jailtime.org Mirror Script for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO='jailtime.org'
 
 
## Define the URL which we rsync from
 
URL='jailtime.org::jailtime'
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
 
 
=== watcherMailer ===
 
An oddly named script that watches for certain conditions to occur on mirror then emails an alert to the Mirror Admin. Kind of like a home brewed Nagios. Obviously you need to define the variables first. Runs as a cronjob every 10 minutes. It is currently living in Chris Peterman's home directory seeing as it doesn't need root privs to run.
 
<source lang="bash">
 
#!/bin/bash
 
# Watcher Script Script for mirror
 
# Watches for a set critia and then emails the DESTEMAIL if said conditions are met
 
# Designed to be run out of a cronjob
 
# Copyright 2008 Chris Peterman <kyral@azuredreams.us>
 
# Licensed under the GPLv2
 
 
MAILFROM=
 
MAILSERV=
 
MAILUSER=
 
PASSWORD=
 
DESTEMAIL=
 
 
## We are going to do this with a function. 1st argument is the subject line, second is the body
 
sendEmail() {
 
/usr/bin/sendEmail -f ${MAILFROM} -t ${DESTEMAIL} -u ${1} -m ${2} -s ${MAILSERV} -xu ${MAILUSER} -xp ${PASSWORD}
 
}
 
 
## Start test conditions
 
 
## Test to make sure that aoedisk is MOUNTED
 
if [ $(/bin/mount | /bin/grep aoedisk &> /dev/null; echo $?) -ne 0 ]
 
then
 
sendEmail "ALERT: aoedisk" "aoedisk is umounted on mirror!"
 
fi
 
 
## Test to make sure various services are running
 
for i in apache2 proftpd rsyncd sshd
 
do
 
if [ ! -e /var/run/${i}.pid ]
 
then
 
sendEmail "ALERT: ${i}" "${i} isn't running on mirror!"
 
fi
 
done
 
 
exit 0
 
</source>
 
 
=== Logrotation ===
 
Rotation of logs in ''/var/log/'' is handled by the logrotate program. Ubuntu provides most of the scripts in ''/etc/logrotate.d'' but there are two that are either heavily modified or completely new.
 
 
==== apache2 ====
 
Apache's logrotation file. Rotates ''/var/log/apache2/access.log'' and ''/var/log/apache2/error.log'' on a daily basis and keeps the past two weeks worth of logs. It uses BZip2 for better compression than Gzip and names them with a datestamp
 
 
<source>
 
/var/log/apache2/*.log {
 
daily
 
missingok
 
rotate 14
 
compress
 
compresscmd bzip2
 
compressoptions "-f"
 
uncompresscmd bunzip2
 
dateext
 
notifempty
 
create 640 root adm
 
sharedscripts
 
postrotate
 
if [ -f /var/run/apache2.pid ]; then
 
/etc/init.d/apache2 reload > /dev/null
 
fi
 
endscript
 
}
 
</source>
 
 
==== rsyncs ====
 
The logrotation file for ''/var/log/mirrorlogs''. Rotates on a monthly basis and keeps the past 5 months worth of logs
 
 
<source>
 
/var/log/mirrorlogs/*.log {
 
monthly
 
missingok
 
rotate 5
 
compress
 
compresscmd bzip2
 
compressoptions "-f"
 
uncompresscmd bunzip2
 
dateext
 
notifempty
 
create 640 root mirrorlogs
 
sharedscripts
 
}
 
</source>
 

Revision as of 15:30, 4 April 2009

Hostname: mirror.clarkson.edu mirror.cslabs.clarkson.edu mirror.int.cslabs.clarkson.edu mirror.sr.cslabs.clarkson.edu
NIC 1: Clarkson Network
MAC: 00:1B:21:28:C8:4E
IP: 128.153.145.19
NIC 2: Server Room Network
MAC: 00:E0:81:B5:88:85
IP: 10.0.1.
NIC 3: Internal Network
MAC: 00:E0:81:B5:88:84
IP: 10.0.0.17

Mirror is an open source projects mirror that mirrors different Linux distributions and the Linux kernel itself. It can be accessed through via http or ftp.

Mirror is the 3U server located in the Unisys Rack below Xen1.

Mirror runs CentOS 5.

Setup Process

Technical Specifications