Difference between revisions of "Old Mirror"

From CSLabsWiki
(→‎Mirrored Software)
 
(54 intermediate revisions by 10 users not shown)
Line 1: Line 1:
  +
{{archived}}
−
__NOTOC__
  
−
{{Expand
  
−
|date=October 3, 2007
  
−
}}
  
−
{{VM
  
−
|hostname=[http://mirror.clarkson.edu mirror.clarkson.edu]
  
−
|ldap=No
  
−
|development_status=Stable
  
−
|status=Running
  
−
|maintainer=[[User:petermcv|Chris Peterman]]
  
−
|dom0=[[hydrogen]]
  
−
}}
  
   
−
'''Mirror''' is an open source projects mirror that mirrors different Linux distributions (see below) and the Linux kernel itself. It can be accessed through a web interface or by anonymous FTP.
 +
'''Mirror''' is COSI's mirror for FOSS projects. It hosts an array of GNU/Linux distributions and other software. It can be accessed through via [http://mirror.clarkson.edu/ http] or rsync.
   
  +
This page is for the old mirror which was retired in April 2016. The next iteration of Mirror is [[Mirror]]
−
==Mirrored Software==
  
−
*[http://www.archlinux.org ArchLinux]
  
−
*[http://www.fedoraproject.org Fedora] (7,8, x86, no SRPMS)
  
−
*[http://www.kernel.org Kernel.org]
  
−
*[http://www.ubuntu.com Ubuntu]
  
−
*[http://www.debian.org Debian] (i386, ia64, amd64)
  
−
*[http://jailtime.org Jailtime.org]
  
−
*[http://www.kde.org KDE]
  
   
−
==Rsync Timetable==
 +
==Specs==
  +
CPU: Intel Core 2 Duo E7200 2.53GHz
−
This is the table for when Rsyncs start. Only two rsyncs may ''start'' at once so that we don't overload the people we are mirroring from
  
−
{|
  
−
!Time !! Rsync 1 !! Rsync 2
 
−
|-
  
−
|'''01:00''' || Fedora || Jailtime.org
 
−
|-
  
−
|'''02:00''' || Kernel.org || Archlinux
  
−
|-
  
−
|'''03:00''' || CentOS || Ubuntu
  
−
|-
  
−
|'''04:00''' || Debian || KDE
  
−
|}
  
   
  +
RAM: 6GB
−
==Scripts==
  
−
Mirror is maintained by many small scripts, this page documents them all. The following are located in the directory: ''/etc/mirrorscripts''
  
   
  +
Storage: 8TB
−
=== rsyncVars ===
  
−
This scriptlet defines all the variables used in the scripts that actually do the mirroring.
  
   
  +
==Setup Process==
−
<source lang="bash">
  
  +
*[[Mirror Setup Process]]
−
## Master Include File for the Mirror Rsync Scripts
  
−
## DISTRO MUST BE DEFINED BEFORE SOURCING THIS FILE!
  
   
  +
==Requests==
−
## Define program paths
  
−
TOUCH=/usr/bin/touch
  
−
DATE=/bin/date
  
−
RSYNC=/usr/bin/rsync
  
−
CHGRP=/bin/chgrp
  
−
RM=/bin/rm
  
−
MKDIR=/bin/mkdir
  
   
  +
*Replace the ugly web page with something automated. (The progress of this can be seen at the [https://github.com/cosi-lab/Mirror2016 Mirror2016] repository on Github.)
−
## Define File Paths
  
  +
*Mirror http://www.parrotsec.org/.
−
LOGFILE=/var/log/mirrorlogs/${DISTRO}-$(${DATE} +%m-%d-%y)
  
  +
*Mirror NetBSD
−
LOCKFILE=/var/lock/${DISTRO}.lck
  
−
DEST=/aoedisk/${DISTRO}
  
   
  +
==Stuff we already mirror==
−
## Rsync Options
  
  +
* Arch
−
RSYNCOPTS='-avrH --partial --stats --delay-updates --delete --delete-after'
  
  +
* Backtrack
  +
* CentOS (Official Mirror)
  +
* Debian
  +
* Fedora (Official Mirror)
  +
* Gentoo
  +
* IPFire (Official Mirror)
  +
* Ubuntu (Official Mirror)
  +
* Serenity (Main Mirror)
  +
* OpenBSD
  +
* openSUSE
  +
* pfsense
  +
* Knoppix
  +
* Mint
  +
* Sabayon (Official Mirror)
  +
* Slackware
  +
* Gnu Project
  +
* Slitaz
  +
* Unity Linux (Official Mirror)
  +
* RPMforge
  +
* Linux kernel
  +
* LibreOffice
  +
* Isabelle (Official Mirror)
   
  +
==Stuff that we should mirror==
−
</source>
  
  +
* FreeBSD
−  
  +
* Raspbian
−
=== template ===
  
  +
* CRUX
−
This is the master template for all the new mirror scripts (meaning that Archlinux's and Kernel.org's don't follow this exactly). Just plugin URL and DISTRO and you are good to go.
  
  +
* Trisquel
−  
  +
* FRUX
−
<source lang="bash">#!/bin/bash -
  
  +
* Guix
−
#!/bin/bash -
  
  +
* Dragora
−
## Template Mirror Script for mirror.clarkson.edu
  
  +
* Parabola
−
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
  
  +
* Nix
−
## Licensed under the GPLv2
  
  +
* NixOS
−  
  +
* Tails
−
## Define which distro we are mirroring here
  
  +
* Kali
−
DISTRO=
  
  +
* Clonezilla
−  
  +
* LibreBoot
−
## Define the URL which we rsync from
  
−
URL=
  
−  
−
## Load all the variable from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile so no one interupts us
  
−
${TOUCH} ${LOCKFILE}
  
−  
−
## Make sure DEST exists
  
−
if [ ! -d ${DEST} ]; then
  
−
${MKDIR} ${DEST}
  
−
fi
  
−  
−
## Begin the Rsync, send all output to the logfiles
  
−
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
  
−  
−
## Change the group ownership of the log files to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−  
−
## Remove the lockfile, we are done
  
−
${RM} ${LOCKFILE}
  
−  
−
## Send a good exit code
  
−
exit 0
  
−
else
  
−
## Someone else was running. Not really an error so we exit with a successful error code
  
−
exit 0
  
−
fi
  
−  
−
</source>
  
−  
−
=== archlinux ===
  
−
This is Archlinux's rsync script. It was created before the prototype, and is a little different than others.
  
−  
−
<source lang="bash">#!/bin/bash -
  
−  
−
## Define which distro we are mirroring
  
−
DISTRO=archlinux
  
−  
−
## Load all the variables from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile to make sure no one interrupts us
  
−
${TOUCH} ${LOCKFILE}
  
−
## Begin the Rsync, send all output to the logfiles
  
−
${RSYNC} ${RSYNCOPTS} distro.ibiblio.org::distros/archlinux/ ${DEST} > ${LOGFILE}
  
−
## Change the group ownership of the log files to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−
## Remove the lockfile, we are done
  
−
${RM} ${LOCKFILE}
  
−
## Send a good exit code
  
−
exit 0
  
−
else
  
−
## Someone else was running. Not really an error so we exit with a successful error code
  
−
exit 0
  
−
fi
  
−
</source>
  
−  
−
=== linux ===
  
−
This is the script for Kernel.org. It is notable because we actually rsync two different modules at once.
  
−  
−
<source lang="bash">#!/bin/bash -
  
−  
−
## Define which distro we are mirroring
  
−
DISTRO=linux
  
−  
−
## Load all the variables from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile to make sure no one interrupts us
  
−
${TOUCH} ${LOCKFILE}
  
−
 
−
## Begin the Rsync, send all output to the logfiles
 
−
${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/linux/ ${DEST} > ${LOGFILE}
  
−
${RSYNC} ${RSYNCOPTS} ftp.uofo.lkams.kernel.org::ftp/pub/software/ /aoedisk/software >> ${LOGFILE}
  
−
 
−
## Change the group ownership of the logs to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−  
−
## We are done, remove the lockfile
  
−
${RM} ${LOCKFILE}
  
−  
−
## Send a clean exit code
  
−
exit 0
  
−
else
  
−
## Another copy of the script was running, not really an error on our part, so send a successful error code
  
−
exit 0
  
−
fi
  
−
</source>
  
−  
−
=== ubuntu ===
  
−
This is ubuntu's mirrorscript. First one based off the template, but different because we rsync another module to a different dest.
  
−  
−
<source lang="bash">
  
−
#!/bin/bash -
  
−
## Ubuntu Mirror Script for mirror.clarkson.edu
  
−
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
  
−
## Licensed under the GPLv2
  
−  
−
## Define which distro we are mirroring here
  
−
DISTRO=ubuntu
  
−  
−
## Define the URL which we rsync from
  
−
URL='mirrors.rit.edu::ubuntu'
  
−  
−
## Load all the variable from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile so no one interupts us
  
−
${TOUCH} ${LOCKFILE}
  
−  
−
## Begin the Rsync, send all output to the logfiles
  
−
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
  
−  
−
## Rsync up the ISOs
  
−
${RSYNC} ${RSYNCOPTS} mirrors.rit.edu::ubuntu-releases /aoedisk/isos/ubuntu >> ${LOGFILE}
  
−  
−
## Change the group ownership of the log files to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−  
−
## Remove the lockfile, we are done
  
−
${RM} ${LOCKFILE}
  
−  
−
## Send a good exit code
  
−
exit 0
  
−
else
  
−
## Someone else was running. Not really an error so we exit with a successful error code
  
−
exit 0
  
−
fi
  
−
</source>
  
−  
−
=== fedora ===
  
−
Fedora's Mirrorscript. Again two rsyncs back to back, and a mess of excludes to refine what we get to save on disk space
  
−  
−
<source lang='bash'>
  
−
#!/bin/bash -
  
−
## Fedora Mirrorscript for mirror.clarkson.edu
  
−
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
  
−
## Licensed under the GPLv2
  
−  
−
## Define which distro we are mirroring here
  
−
DISTRO=fedora
  
−  
−
## Define the URL which we rsync from
  
−
#URL=
  
−  
−
## Load all the variable from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile so no one interupts us
  
−
${TOUCH} ${LOCKFILE}
  
−  
−
## Begin the Rsync, send all output to the logfiles
  
−
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" mirrors.kernel.org::fedora/releases ${DEST} > ${LOGFILE}
  
−  
−
## Second RSync for the Updates repo
  
−
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*SRPMS*" --exclude "*debug*" mirrors.kernel.org::fedora/updates ${DEST} >> ${LOGFILE}
  
−  
−
## Change the group ownership of the log files to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−  
−
## Remove the lockfile, we are done
  
−
${RM} ${LOCKFILE}
  
−  
−
## Send a good exit code
  
−
exit 0
  
−
else
  
−
## Someone else was running. Not really an error so we exit with a successful error code
  
−
exit 0
  
−
fi
  
−
</source>
  
−  
−
=== debian ===
  
−
Debian's mirrorscript. Excludes a plenty for Arch's we don't use
  
−  
−
<source lang='bash'>
  
−
#!/bin/bash -
  
−
## Debian Mirror Script for mirror.clarkson.edu
  
−
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
  
−
## Licensed under the GPLv2
  
−  
−
## Define which distro we are mirroring here
  
−
DISTRO=debian
  
−  
−
## Define the URL which we rsync from
  
−
URL=mirrors.kernel.org::debian
  
−  
−
## Load all the variable from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile so no one interupts us
  
−
${TOUCH} ${LOCKFILE}
  
−  
−
## Begin the Rsync, send all output to the logfiles
  
−
${RSYNC} ${RSYNCOPTS} --exclude binary-alpha/ --exclude *_alpha.deb --exclude binary-arm/ --exclude *_arm.deb --exclude binary-hppa/ --exclude *_hppa.deb --exclude binary-hurd-i386/ --exclude *_hurd-i386.deb --exclude binary-m68k/ --exclude *_m68k.deb --exclude binary-mips/ --exclude *_mips.deb --exclude binary-mipsel/ --exclude *_mipsel.deb --exclude binary-powerpc/ --exclude *_powerpc.deb --exclude binary-s390/ --exclude *_s390.deb --exclude binary-sparc/ --exclude *_sparc.deb ${URL} ${DEST} > ${LOGFILE}
  
−  
−
${RSYNC} ${RSYNCOPTS} mirrors.kernel.org::debian-cd ${DEST} >> ${LOGFILE}
  
−  
−
## Change the group ownership of the log files to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−  
−
## Remove the lockfile, we are done
  
−
${RM} ${LOCKFILE}
  
−  
−
## Send a good exit code
  
−
exit 0
  
−
else
  
−
## Someone else was running. Not really an error so we exit with a successful error code
  
−
exit 0
  
−
fi
  
−  
−
</source>
  
−  
−
=== CentOS ===
  
−
This is CentOS's mirrorscript and it was based off of Fedora's
  
−
<source lang="bash">
  
−
#!/bin/bash -
  
−
## CentOS Mirrorscript for mirror.clarkson.edu
  
−
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
  
−
## Licensed under the GPLv2
  
−  
−
## Define which distro we are mirroring here
  
−
DISTRO=centos
  
−  
−
## Define the URL which we rsync from
  
−
#URL=
  
−  
−
## Load all the variable from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile so no one interupts us
  
−
${TOUCH} ${LOCKFILE}
  
−  
−
## Make sure the dest directory exists!
  
−
if [ ! -d ${DEST} ]; then
  
−
${MKDIR} ${DEST}
  
−
fi
  
−  
−
## Begin the Rsync, send all output to the logfiles
  
−
${RSYNC} ${RSYNCOPTS} --exclude "**ppc**" --exclude "*test*" --exclude "**x86_64**" --exclude "**source**" --exclude "*debug*" --exclude "*SRPMS*" mirrors.rit.edu::centos ${DEST} > ${LOGFILE}
  
−  
−
## Change the group ownership of the log files to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−  
−
## Remove the lockfile, we are done
  
−
${RM} ${LOCKFILE}
  
−  
−
## Send a good exit code
  
−
exit 0
  
−
else
  
−
## Someone else was running. Not really an error so we exit with a successful error code
  
−
exit 0
  
−
fi
  
−
</source>
  
−  
−
=== Jailtime.org ===
  
−
This is the mirrorscript for Jailtime.org. I think this is the first one that didn't need massive editing from the Template..
  
−  
−
<source lang="bash">
  
−
#!/bin/bash -
  
−
## Jailtime.org Mirror Script for mirror.clarkson.edu
  
−
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
  
−
## Licensed under the GPLv2
  
−  
−
## Define which distro we are mirroring here
  
−
DISTRO='jailtime.org'
  
−  
−
## Define the URL which we rsync from
  
−
URL='jailtime.org::jailtime'
  
−  
−
## Load all the variable from the master include file
  
−
source /etc/mirrorscripts/rsyncVars
  
−  
−
## Make sure another instance of this script isn't running right now
  
−
if [ ! -e ${LOCKFILE} ]
  
−
then
  
−
## No one else is running, put down a lockfile so no one interupts us
  
−
${TOUCH} ${LOCKFILE}
  
−  
−
## Begin the Rsync, send all output to the logfiles
  
−
${RSYNC} ${RSYNCOPTS} ${URL} ${DEST} > ${LOGFILE}
  
−  
−
## Change the group ownership of the log files to Mirrorlogs
  
−
${CHGRP} mirrorlogs ${LOGFILE}
  
−  
−
## Remove the lockfile, we are done
  
−
${RM} ${LOCKFILE}
  
−  
−
## Send a good exit code
  
−
exit 0
  
−
else
  
−
## Someone else was running. Not really an error so we exit with a successful error code
  
−
exit 0
  
−
fi
  
−
</source>
  
−  
−
=== mirrorLogRotate ===
  
−
This is located in ''/etc/cron.monthly'' and rotates the logs in ''/var/log/mirrorlogs''
  
−  
−
<source lang="bash">
  
−
#!/bin/bash -
  
−
## Log Rotation Script for mirror.clarkson.edu's rsync logs
  
−
## Copyright 2007 Chris Peterman <petermcv@clarkson.edu>
  
−
## Licensed under the GPLv2
  
−  
−  
−
## Set month number to LAST month
  
−
(( month = $(/bin/date +%m) - 1 ))
  
−  
−
## Account for January being 1 and December being 12. There is no month
 
−
## 0, at least as returned by /bin/date
  
−
if [ ${month} -eq 0 ]
  
−
then
  
−
month=12
  
−
fi
  
−  
−
## Hack the month setting to give it back two digits
  
−
case "$month" in
  
−
"1") month=01;;
  
−
"2") month=02;;
  
−
"3") month=03;;
  
−
"4") month=04;;
  
−
"5") month=05;;
  
−
"6") month=06;;
  
−
"7") month=07;;
  
−
"8") month=08;;
  
−
"9") month=09;;
  
−
esac
  
−  
−
## Set the year to the current year unless the month is 12 (December)
  
−
## in that case set it to the PREVIOUS year
  
−
if [ ${month} -eq 12 ]
  
−
then
  
−
(( year = $(/bin/date +%y) - 1 ))
  
−
else
  
−
year=$(/bin/date +%y)
  
−
fi
  
−  
−
## Set the mirror log directory
  
−
logDir=/var/log/mirrorlogs
  
−  
−
for distro in archlinux fedora ubuntu debian linux centos
  
−
do
  
−
## Archive the logs from last month, then delete them (NOT the Archive)
  
−
/bin/tar -czvf /tmp/${distro}-${month}-${year}.tgz ${logDir}/${distro}-${month}-*-${year} && /bin/rm -v ${logDir}/${distro}-${month}-*-${year}
  
−  
−
## Move the log Tarball back to the logdir
  
−
/bin/mv /tmp/${distro}-${month}-${year}.tgz ${logDir}
  
−
done
  
−  
−
exit 0
  
−  
−  
−
</source>
  
−  
−
=== watcherMailer ===
  
−
An oddly named script that watches for certain conditions to occur on mirror then emails an alert to the Mirror Admin. Kind of like a home brewed Nagios. Obviously you need to define the variables first. Runs as a cronjob every 10 minutes. It is currently living in Chris Peterman's home directory seeing as it doesn't need root privs to run.
  
−
<source lang="bash">
  
−
#!/bin/bash
  
−
# Watcher Script Script for mirror
  
−
# Watches for a set critia and then emails the DESTEMAIL if said conditions are met
  
−
# Designed to be run out of a cronjob
  
−
# Copyright 2008 Chris Peterman <kyral@azuredreams.us>
  
−
# Licensed under the GPLv2
  
−  
−
MAILFROM=
  
−
MAILSERV=
  
−
MAILUSER=
  
−
PASSWORD=
  
−
DESTEMAIL=
  
−  
−
## We are going to do this with a function. 1st argument is the subject line, second is the body
  
−
sendEmail() {
  
−
/usr/bin/sendEmail -f ${MAILFROM} -t ${DESTEMAIL} -u ${1} -m ${2} -s ${MAILSERV} -xu ${MAILUSER} -xp ${PASSWORD}
  
−
}
  
−  
−
## Start test conditions
  
−  
−
## Test to make sure that aoedisk is MOUNTED
  
−
if [ $(/bin/mount | /bin/grep aoedisk &> /dev/null; echo $?) -ne 0 ]
  
−
then
  
−
sendEmail "ALERT: aoedisk" "aoedisk is umounted on mirror!"
  
−
fi
  
−  
−
## Test to make sure various services are running
  
−
for i in apache2 proftpd rsyncd sshd
  
−
do
  
−
if [ ! -e /var/run/${i}.pid ]
  
−
then
  
−
sendEmail "ALERT: ${i}" "${i} isn't running on mirror!"
  
−
fi
  
−
done
  
−  
−
exit 0
  
−
</source>
  

Latest revision as of 15:11, 29 April 2016

Emblem-important.png This is an archived article or section.
This page is a legacy practice, project, or tutorial, and the information may not be up to date. See the talk page for more information about the page's status.


Mirror is COSI's mirror for FOSS projects. It hosts an array of GNU/Linux distributions and other software. It can be accessed through via http or rsync.

This page is for the old mirror which was retired in April 2016. The next iteration of Mirror is Mirror

Specs

CPU: Intel Core 2 Duo E7200 2.53GHz

RAM: 6GB

Storage: 8TB

Setup Process

Requests

Stuff we already mirror

  • Arch
  • Backtrack
  • CentOS (Official Mirror)
  • Debian
  • Fedora (Official Mirror)
  • Gentoo
  • IPFire (Official Mirror)
  • Ubuntu (Official Mirror)
  • Serenity (Main Mirror)
  • OpenBSD
  • openSUSE
  • pfsense
  • Knoppix
  • Mint
  • Sabayon (Official Mirror)
  • Slackware
  • Gnu Project
  • Slitaz
  • Unity Linux (Official Mirror)
  • RPMforge
  • Linux kernel
  • LibreOffice
  • Isabelle (Official Mirror)

Stuff that we should mirror

  • FreeBSD
  • Raspbian
  • CRUX
  • Trisquel
  • FRUX
  • Guix
  • Dragora
  • Parabola
  • Nix
  • NixOS
  • Tails
  • Kali
  • Clonezilla
  • LibreBoot
Retrieved from "http://docs.cslabs.clarkson.edu/mediawiki/index.php?title=Old_Mirror&oldid=7759"