Mirror Wrangling

1.  Mirror Wrangling

Preparing and testing mirrors

1. New mirrors should file a file bug under the "Mirrors" product, "New Server" component. If a request comes in via email, ask them to file the request on Bugzilla.

This will help us test out the mirror using our Gentoo Automated Rsync Distfiles checker script, or GARD for short.

2. Ask for the following information on the bug, if not provided already:

• Official mirror name (preferably not your hostname, but the name of your organization)
• Mirror type: distfiles/releases, gentoo-portage. NOTE: gentoo-portage is always rsync only. Some mirrors may have rsync for distfiles as well. Do not be confused by the difference!
• URLs: full protocol and path for release/distfile mirrors, hostname or IP for gentoo-portage mirrors
• Hardware specifications
• Bandwidth specifications (transfer speed, eg 100Mbit, and any limits you may have)
• Location
• Intended concurrent users (for each protocol, mirror type)
• Admin contact name and email address (This address should be registered in bugzilla so we can CC you to any bug reports)

3. Modify the summary to have the type of mirror and the city as well as country if possible.

4. Build a testing string for GARD. The ; : and | are important, make sure they're where they're supposed to be:

due:YYYY/MM/DD;dist:http://URL|ftp://URL|rsync://URL (distfiles);portage:FQDN or IP

Here are some string samples:

(Everything mirror: distfiles + gentoo-portage)
due:2009/01/01;dist:http://mirror.gentoo.org/gentoo/|ftp://mirror.gentoo.org/gentoo/|rsync://mirror.gentoo.org/gentoo/;portage:mirror.gentoo.org

(gentoo-portage mirror only)
due:2009/01/01;dist:||;portage:mirror.gentoo.org

(http distfiles mirror only)
due:2009/01/01;dist:http://mirror.gentoo.org/gentoo/||;portage:

5. Inform the mirror of testing policies, tell them the due date (when testing will be completed) and give them the following links to make sure their mirror(s) have been setup correctly:

• Distfiles: http://www.gentoo.org/doc/en/source_mirrors.xml
• gentoo-portage: http://www.gentoo.org/doc/en/rsync.xml

Our policy right now for testing duration is as follows:

• New distfiles/gentoo-portage mirrors: 2 weeks from date of infra accepting the bug
• Returning distfiles/gentoo-portage mirrors: Same hardware/IP - 1 week. Different hardware/IP - 2 weeks
• Problematic mirrors: 1 week to fix the issue, else they're removed from rotation unless we're provided with valid reasons by the admins.
• Rsync mirrors in any continental rotation on DNS with issues should be removed immediately from those until the issue is resolved. For example, rsync.namerica.gentoo.org, rsync.europe.gentoo.org. You can leave the mirror in its country's rotation.

6. Put the testing string in the Status whiteboard of the bug and mark the bug as IN_PROGRESS. These steps are important for GARD to start checking the mirror.

7. Review the GARD output for issues daily.

8. Report problems back to the mirror ASAP.

9. If there are no problems, try to give a (good) report at the halfway point of the period.

10. Give a final good at the end of the testing period, if everything is good.

Now the mirror is ready to be added officially.

2.  Adding Mirrors

For distfiles mirrors

1. Add the mirror to CVS gentoo/xml/htdocs/main/en/mirrors3.xml (check the format of the doc, follow the same).

2. Send the distfiles password email to the new mirror admins, CC mirror-admin@gentoo.org (mirror admins have the password, it is not public).

Hi,
We're glad to have you as our newest source mirror, welcome aboard!

Use this information to sync with our private mirror every 4 hours from
Midnight local time. This is for mirror admins only, please do not share
this information.

----------------------------------------
rsync module: masterdistfiles-{eu,us}.gentoo.org::gentoo
username: $insert_here
password: $insert_here

To make this scriptable, we recommend placing the password in a file and
then using the --password-file option with rsync.  The file must not be
accessible by others (i.e. chmod 640) or rsync will complain.  So, as an
example, placing the password into '~/distfile_passwd', you would use an
rsync command like the following:

rsync -rlptDq --delete -H --password-file=~/distfile_passwd \
$username@masterdistfiles-{eu,us}..gentoo.org::gentoo /your/local/path/goes/here

Please note that we have received several reports of problems when using
the -z (compress) option of rsync.  Thus, please DO NOT use this option
when syncing against this mirror.
---------------------------------------

Please also make sure you are subscribed to gentoo-mirrors@gentoo.org as
we often send out administrative notices and policy changes via this
list.

Thanks for helping Gentoo!

Cheers,
$name

3. Add the mirror/admin-email to the infra wiki.

4. Put in the following comment and close the bug:

Thanks for helping out Gentoo. I have added your mirror to our list at http://www.gentoo.org/main/en/mirrors2.xml. Please start syncing from masterdistfiles.gentoo.org as the email I have sent to you instructs. We would like you to subscribe to the gentoo-mirrors Mailing List by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send out notifications of updates, releases, and changes in policy to the mirrors. Please also monitor http://mirrorstats.gentoo.org as this application is used to keep track of the up-to-date nature of our mirrors. Thanks for the hardware and bandwidth!

For gentoo-portage mirrors

1. Add the IP to osprey.gentoo.org, at /home/gmirror/rsync-hosts-allow/hosts.allow.master. Follow the format, re-use existing numbers for a country before adding new ones.

$ ssh osprey.gentoo.org
$ cd /home/gmirror/rsync-hosts-allow
$ ${EDITOR} hosts.allow.master
$ git commit hosts.allow.master

2. To add a new country, just make sure it isn't already there, then create one.

3. Add the IP into the country (and if needed, continent) DNS rotation(s) in UltraDNS. For example, a new mirror in the US would be rsync(number).us.gentoo.org, so you will add the new IP:

rsync(number).us.gentoo.org - TTL - 86400

(US/Country rotation)
rsync.us.gentoo.org - TTL - 1800

(Continental rotation)
rsync.namerica.gentoo.org - TTL - 1800

4. Add the mirror to CVS gentoo/xml/htdocs/main/en/mirrors-rsync-data.xml (check the format of the doc, follow the same).

5. Add the mirror/admin-email to the infra wiki.

6. Put in the following comment (modify as needed) and close the bug:

Thanks for helping out Gentoo. I have added the IP you specified in the bug to our access database, please start syncing to masterportage.gentoo.org per our rsync document at http://www.gentoo.org/doc/en/rsync.xml. We would like you to subscribe to the gentoo-mirrors Mailing List by sending an email to gentoo-mirrors+subscribe@lists.gentoo.org, as we send our notifications of updates, releases, and changes in policy to the mirrors. Once again, thanks for the hardware and bandwidth! Please also monitor http://mirrorstats.gentoo.org/rsync as this application is used to keep track of the up-to-date nature of our mirrors. You are mirror rsync(number).$country.gentoo.org in the rsync.$country.gentoo.org rotation and have been added to the rsync.$continent.gentoo.org rotation as well.

3.  Removing Mirrors

For distfiles mirrors

1. Remove the mirror from CVS gentoo/xml/htdocs/main/en/mirrors3.xml (check the format of the doc, follow the same).

2. Remove the mirror/admin-email from the infra wiki.

3. Put in the following comment and close the bug:

Removed $server_name from mirrors listings. Thanks for the hardware and bandwidth donation, your services helped many Gentoo Users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)

For gentoo-portage mirrors

1. Remove the IP from osprey.gentoo.org, at /home/gmirror/rsync-hosts-allow/hosts.allow.master. Leave the existing rsyncX.$CC comment so that it is known that it should be reused.

$ ssh osprey.gentoo.org
$ cd /home/gmirror/rsync-hosts-allow
$ ${EDITOR} hosts.allow.master
$ git commit hosts.allow.master

2. Remove the IP from the country DNS rotation(s) in UltraDNS.

3. Remove the mirror from CVS gentoo/xml/htdocs/main/en/mirrors-rsync-data.xml (check the format of the doc, follow the same).

4. Put in the following comment (modify as needed) and close the bug:

Removed $server_name from ACLs, removed from DNS. Thanks for the hardware and bandwidth donation, your services helped many Gentoo Users. (You have no more obligation to monitor the gentoo-mirrors mailing list.)

The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-2.5 license. The Gentoo Name and Logo Usage Guidelines apply.