Difference between revisions of "Old Mirror"

From CSLabsWiki
(Rsync Timetable: Added KDE's timeslot)
 
(55 intermediate revisions by 10 users not shown)
Line 1: Line 1:
  +
{{archived}}
__NOTOC__
 
{{Expand
 
|date=October 3, 2007
 
}}
 
{{VM
 
|hostname=[http://mirror.clarkson.edu mirror.clarkson.edu]
 
|ldap=No
 
|development_status=Stable
 
|status=Running
 
|maintainer=[[User:petermcv|Chris Peterman]]
 
|dom0=[[hydrogen]]
 
}}
 
   
'''Mirror''' is an open source projects mirror that mirrors different Linux distributions (see below) and the Linux kernel itself. It can be accessed through a web interface or by anonymous FTP.
+
'''Mirror''' is COSI's mirror for FOSS projects. It hosts an array of GNU/Linux distributions and other software. It can be accessed through via [http://mirror.clarkson.edu/ http] or rsync.
   
  +
This page is for the old mirror which was retired in April 2016. The next iteration of Mirror is [[Mirror]]
==Mirrored Software==
 
*[http://www.archlinux.org ArchLinux]
 
*[http://www.fedoraproject.org Fedora] (7,8, x86, no SRPMS)
 
*[http://www.kernel.org Kernel.org]
 
*[http://www.ubuntu.com Ubuntu]
 
*[http://www.debian.org Debian] (i386, ia64, amd64)
 
*[http://jailtime.org Jailtime.org]
 
   
==Rsync Timetable==
+
==Specs==
  +
CPU: Intel Core 2 Duo E7200 2.53GHz
This is the table for when Rsyncs start. Only two rsyncs may ''start'' at once so that we don't overload the people we are mirroring from
 
{|
 
!Time !! Rsync 1 !! Rsync 2
 
|-
 
|'''01:00''' || Fedora || Jailtime.org
 
|-
 
|'''02:00''' || Kernel.org || Archlinux
 
|-
 
|'''03:00''' || CentOS || Ubuntu
 
|-
 
|'''04:00''' || Debian || KDE
 
|}
 
   
  +
RAM: 6GB
==Scripts==
 
Mirror is maintained by many small scripts, this page documents them all. The following are located in the directory: ''/etc/mirrorscripts''
 
   
  +
Storage: 8TB
=== rsyncVars ===
 
This scriptlet defines all the variables used in the scripts that actually do the mirroring.
 
   
  +
==Setup Process==
<source lang="bash">
 
  +
*[[Mirror Setup Process]]
## Master Include File for the Mirror Rsync Scripts
 
## DISTRO MUST BE DEFINED BEFORE SOURCING THIS FILE!
 
   
  +
==Requests==
## Define program paths
 
TOUCH=/usr/bin/touch
 
DATE=/bin/date
 
RSYNC=/usr/bin/rsync
 
CHGRP=/bin/chgrp
 
RM=/bin/rm
 
MKDIR=/bin/mkdir
 
   
  +
*Replace the ugly web page with something automated. (The progress of this can be seen at the [https://github.com/cosi-lab/Mirror2016 Mirror2016] repository on Github.)
## Define File Paths
 
  +
*Mirror http://www.parrotsec.org/.
LOGFILE=/var/log/mirrorlogs/${DISTRO}-$(${DATE} +%m-%d-%y)
 
  +
*Mirror NetBSD
LOCKFILE=/var/lock/${DISTRO}.lck
 
DEST=/aoedisk/${DISTRO}
 
   
  +
==Stuff we already mirror==
## Rsync Options
 
  +
* Arch
RSYNCOPTS='-avrH --partial --stats --delay-updates --delete --delete-after'
 
  +
* Backtrack
  +
* CentOS (Official Mirror)
  +
* Debian
  +
* Fedora (Official Mirror)
  +
* Gentoo
  +
* IPFire (Official Mirror)
  +
* Ubuntu (Official Mirror)
  +
* Serenity (Main Mirror)
  +
* OpenBSD
  +
* openSUSE
  +
* pfsense
  +
* Knoppix
  +
* Mint
  +
* Sabayon (Official Mirror)
  +
* Slackware
  +
* Gnu Project
  +
* Slitaz
  +
* Unity Linux (Official Mirror)
  +
* RPMforge
  +
* Linux kernel
  +
* LibreOffice
  +
* Isabelle (Official Mirror)
   
  +
==Stuff that we should mirror==
</source>
 
  +
* FreeBSD
 
  +
* Raspbian
=== template ===
 
  +
* CRUX
This is the master template for all the new mirror scripts (meaning that Archlinux's and Kernel.org's don't follow this exactly). Just plugin URL and DISTRO and you are good to go.
 
  +
* Trisquel
 
  +
* FRUX
<source lang="bash">#!/bin/bash -
 
  +
* Guix
#!/bin/bash -
 
  +
* Dragora
## Template Mirror Script for mirror.clarkson.edu
 
  +
* Parabola
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
  +
* Nix
## Licensed under the GPLv2
 
  +
* NixOS
 
  +
* Tails
## Define which distro we are mirroring here
 
  +
* Kali
DISTRO=
 
  +
* Clonezilla
 
  +
* LibreBoot
## Define the URL which we rsync from
 
URL=
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Make sure DEST exists
 
if [ ! -d ${DEST} ]; then
 
${MKDIR} ${DEST}
 
fi
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
 
</source>
 
 
=== archlinux ===
 
This is Archlinux's rsync script. It was created before the prototype, and is a little different than others.
 
 
<source lang="bash">#!/bin/bash -
 
 
## Define which distro we are mirroring
 
DISTRO=archlinux
 
 
## Load all the variables from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile to make sure no one interrupts us
 
${TOUCH} ${LOCKFILE}
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} distro.ibiblio.org::distros/archlinux/ ${DEST} > ${LOGFILE}
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== linux ===
 
This is the script for Kernel.org. It is notable because we actually rsync two different modules at once.
 
 
<source lang="bash">#!/bin/bash -
 
 
## Define which distro we are mirroring
 
DISTRO=linux
 
 
## Load all the variables from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile to make sure no one interrupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/linux/ ${DEST} > ${LOGFILE}
 
${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/software/ /aoedisk/software >> ${LOGFILE}
 
 
## Change the group ownership of the logs to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## We are done, remove the lockfile
 
${RM} ${LOCKFILE}
 
 
## Send a clean exit code
 
exit 0
 
else
 
## Another copy of the script was running, not really an error on our part, so send a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== ubuntu ===
 
This is ubuntu's mirrorscript. First one based off the template, but different because we rsync another module to a different dest.
 
 
<source lang="bash">
 
#!/bin/bash -
 
## Ubuntu Mirror Script for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=ubuntu
 
 
## Define the URL which we rsync from
 
URL='mirrors.rit.edu::ubuntu'
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
 
 
## Rsync up the ISOs
 
${RSYNC} ${RSYNCOPTS} mirrors.rit.edu::ubuntu-releases /aoedisk/isos/ubuntu >> ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== fedora ===
 
Fedora's Mirrorscript. Again two rsyncs back to back, and a mess of excludes to refine what we get to save on disk space
 
 
<source lang='bash'>
 
#!/bin/bash -
 
## Fedora Mirrorscript for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=fedora
 
 
## Define the URL which we rsync from
 
#URL=
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" mirrors.kernel.org::fedora/releases ${DEST} > ${LOGFILE}
 
 
## Second RSync for the Updates repo
 
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*SRPMS*" --exclude "*debug*" mirrors.kernel.org::fedora/updates ${DEST} >> ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== debian ===
 
Debian's mirrorscript. Excludes a plenty for Arch's we don't use
 
 
<source lang='bash'>
 
#!/bin/bash -
 
## Debian Mirror Script for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=debian
 
 
## Define the URL which we rsync from
 
URL=mirrors.kernel.org::debian
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} --exclude binary-alpha/ --exclude *_alpha.deb --exclude binary-arm/ --exclude *_arm.deb --exclude binary-hppa/ --exclude *_hppa.deb --exclude binary-hurd-i386/ --exclude *_hurd-i386.deb --exclude binary-m68k/ --exclude *_m68k.deb --exclude binary-mips/ --exclude *_mips.deb --exclude binary-mipsel/ --exclude *_mipsel.deb --exclude binary-powerpc/ --exclude *_powerpc.deb --exclude binary-s390/ --exclude *_s390.deb --exclude binary-sparc/ --exclude *_sparc.deb ${URL} ${DEST} > ${LOGFILE}
 
 
${RSYNC} ${RSYNCOPTS} mirrors.kernel.org::debian-cd ${DEST} >> ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
 
</source>
 
 
=== CentOS ===
 
This is CentOS's mirrorscript and it was based off of Fedora's
 
<source lang="bash">
 
#!/bin/bash -
 
## CentOS Mirrorscript for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO=centos
 
 
## Define the URL which we rsync from
 
#URL=
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Make sure the dest directory exists!
 
if [ ! -d ${DEST} ]; then
 
${MKDIR} ${DEST}
 
fi
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" --exclude "*SRPMS*" mirrors.rit.edu::centos ${DEST} > ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== Jailtime.org ===
 
This is the mirrorscript for Jailtime.org. I think this is the first one that didn't need massive editing from the Template..
 
 
<source lang="bash">
 
#!/bin/bash -
 
## Jailtime.org Mirror Script for mirror.clarkson.edu
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
## Define which distro we are mirroring here
 
DISTRO='jailtime.org'
 
 
## Define the URL which we rsync from
 
URL='jailtime.org::jailtime'
 
 
## Load all the variable from the master include file
 
source /etc/mirrorscripts/rsyncVars
 
 
## Make sure another instance of this script isn't running right now
 
if [ ! -e ${LOCKFILE} ]
 
then
 
## No one else is running, put down a lockfile so no one interupts us
 
${TOUCH} ${LOCKFILE}
 
 
## Begin the Rsync, send all output to the logfiles
 
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
 
 
## Change the group ownership of the log files to Mirrorlogs
 
${CHGRP} mirrorlogs ${LOGFILE}
 
 
## Remove the lockfile, we are done
 
${RM} ${LOCKFILE}
 
 
## Send a good exit code
 
exit 0
 
else
 
## Someone else was running. Not really an error so we exit with a successful error code
 
exit 0
 
fi
 
</source>
 
 
=== mirrorLogRotate ===
 
This is located in ''/etc/cron.monthly'' and rotates the logs in ''/var/log/mirrorlogs''
 
 
<source lang="bash">
 
#!/bin/bash -
 
## Log Rotation Script for mirror.clarkson.edu's rsync logs
 
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
 
## Licensed under the GPLv2
 
 
 
## Set month number to LAST month
 
(( month = $(/bin/date +%m) - 1 ))
 
 
## Account for January being 1 and December being 12. There is no month
 
## 0, at least as returned by /bin/date
 
if [ ${month} -eq 0 ]
 
then
 
month=12
 
fi
 
 
## Hack the month setting to give it back two digits
 
case "$month" in
 
"1") month=01;;
 
"2") month=02;;
 
"3") month=03;;
 
"4") month=04;;
 
"5") month=05;;
 
"6") month=06;;
 
"7") month=07;;
 
"8") month=08;;
 
"9") month=09;;
 
esac
 
 
## Set the year to the current year unless the month is 12 (December)
 
## in that case set it to the PREVIOUS year
 
if [ ${month} -eq 12 ]
 
then
 
(( year = $(/bin/date +%y) - 1 ))
 
else
 
year=$(/bin/date +%y)
 
fi
 
 
## Set the mirror log directory
 
logDir=/var/log/mirrorlogs
 
 
for distro in archlinux fedora ubuntu debian linux centos
 
do
 
## Archive the logs from last month, then delete them (NOT the Archive)
 
/bin/tar -czvf /tmp/${distro}-${month}-${year}.tgz ${logDir}/${distro}-${month}-*-${year} && /bin/rm -v ${logDir}/${distro}-${month}-*-${year}
 
 
## Move the log Tarball back to the logdir
 
/bin/mv /tmp/${distro}-${month}-${year}.tgz ${logDir}
 
done
 
 
exit 0
 
 
 
</source>
 
 
=== watcherMailer ===
 
An oddly named script that watches for certain conditions to occur on mirror then emails an alert to the Mirror Admin. Kind of like a home brewed Nagios. Obviously you need to define the variables first. Runs as a cronjob every 10 minutes. It is currently living in Chris Peterman's home directory seeing as it doesn't need root privs to run.
 
<source lang="bash">
 
#!/bin/bash
 
# Watcher Script Script for mirror
 
# Watches for a set critia and then emails the DESTEMAIL if said conditions are met
 
# Designed to be run out of a cronjob
 
# Copyright 2008 Chris Peterman <kyral@azuredreams.us>
 
# Licensed under the GPLv2
 
 
MAILFROM=
 
MAILSERV=
 
MAILUSER=
 
PASSWORD=
 
DESTEMAIL=
 
 
## We are going to do this with a function. 1st argument is the subject line, second is the body
 
sendEmail() {
 
/usr/bin/sendEmail -f ${MAILFROM} -t ${DESTEMAIL} -u ${1} -m ${2} -s ${MAILSERV} -xu ${MAILUSER} -xp ${PASSWORD}
 
}
 
 
## Start test conditions
 
 
## Test to make sure that aoedisk is MOUNTED
 
if [ $(/bin/mount | /bin/grep aoedisk &> /dev/null; echo $?) -ne 0 ]
 
then
 
sendEmail "ALERT: aoedisk" "aoedisk is umounted on mirror!"
 
fi
 
 
## Test to make sure various services are running
 
for i in apache2 proftpd rsyncd sshd
 
do
 
if [ ! -e /var/run/${i}.pid ]
 
then
 
sendEmail "ALERT: ${i}" "${i} isn't running on mirror!"
 
fi
 
done
 
 
exit 0
 
</source>
 

Latest revision as of 16:11, 29 April 2016


Mirror is COSI's mirror for FOSS projects. It hosts an array of GNU/Linux distributions and other software. It can be accessed through via http or rsync.

This page is for the old mirror which was retired in April 2016. The next iteration of Mirror is Mirror

Specs

CPU: Intel Core 2 Duo E7200 2.53GHz

RAM: 6GB

Storage: 8TB

Setup Process

Requests

Stuff we already mirror

  • Arch
  • Backtrack
  • CentOS (Official Mirror)
  • Debian
  • Fedora (Official Mirror)
  • Gentoo
  • IPFire (Official Mirror)
  • Ubuntu (Official Mirror)
  • Serenity (Main Mirror)
  • OpenBSD
  • openSUSE
  • pfsense
  • Knoppix
  • Mint
  • Sabayon (Official Mirror)
  • Slackware
  • Gnu Project
  • Slitaz
  • Unity Linux (Official Mirror)
  • RPMforge
  • Linux kernel
  • LibreOffice
  • Isabelle (Official Mirror)

Stuff that we should mirror

  • FreeBSD
  • Raspbian
  • CRUX
  • Trisquel
  • FRUX
  • Guix
  • Dragora
  • Parabola
  • Nix
  • NixOS
  • Tails
  • Kali
  • Clonezilla
  • LibreBoot