Difference between revisions of "Admin reference"

From ProgClub
Jump to: navigation, search
(Fully qualified hostname)
(Configure /etc/skel)
Line 126: Line 126:
  export EDITOR=/usr/bin/vim
  export EDITOR=/usr/bin/vim
  script_path=~/session/`date +%Y-%m-%d-%H%M%S`
  script_path=~/.session/`date +%Y-%m-%d-%H%M%S`
  script -f -q -a -t $script_path.script 2> $script_path.timing && exit
  script -f -q -a -t $script_path.script 2> $script_path.timing && exit

Revision as of 01:57, 7 February 2012

__NOTITLE__ This page is a reference for ProgClub system administrators. For information about ProgClub domains, see Domains. For information about member services, see Services. See Machines for information about hosts on the ProgClub network. See Projects for current projects or check out our Forums to get in touch.

Administrative reference

If you're administering ProgClub assets, please document your actions on the wiki. See the relevant pages:

See Future machine names for the kind of stuff we're on about.

See, or update, Network administration for work that needs to be done.

Code of conduct

As a ProgClub administrator you have a lot of power. You have the capability to destroy ProgClub's files and configuration, to access all of ProgClub's databases, to pretend to be other users, and to access other users' private data. We expect you won't abuse your power. Specifically, under no circumstances should you:

  • Delete or modify in inappropriate ways ProgClub files or members' files
  • Read other users' email or private files
  • Copy data from any of ProgClub's administrative databases into your own system
  • Read or copy any data from members' databases that are not yours
  • Pretend to be another member
  • Publicly disclose member details, such as name or email address, without their permission

All of your systems administration activity should be documented, and you shouldn't be doing anything evil. If you're not sure what qualifies as evil, if you have any doubt at all, please ask.

Although ProgClub doesn't make any guarantees, members should be able to feel as though their privacy is respected at ProgClub, and they should be able to feel confident that administrators aren't spying on them by reading their email or their other data.

Where are the keys?

Your administrator login on charity and your member account in LDAP will give you sudo privs on all of the ProgClub machines, both administrative and user machines. Some tools require special purpose logins, and for those you will find login details in the /home/jj5/login_* files which you can access from any of the machines.


For login information:

  • See /home/jj5/login_ldap for the LDAP login.
  • See /home/jj5/login_data for the database logins.
  • See /home/jj5/login_mail for the email login.
  • See /home/jj5/login_list for the email lists logins.


The process for documenting network administration activity is:

Note: don't use screen while administering systems, because it frags PuTTY's scroll buffer, and you need that in order to copy-and-paste your session log.

  • Update the network administration page with a note about what you did:

It's not a bad idea to link to the relevant section(s) of the admin page(s) from the network administration 'done' note, but that's not strictly necessary.

  • Let everyone know what new features are available:

Administrative and user machines

As explained on the Machines page ProgClub separates its machines into two groups: administrative machines and user machines. There is only one administrative machine, charity. There are two user machines, hope and honesty. If you're doing sysadmin work on the user machines, make sure you duplicate your work and create an identical system configuration on each machine. Both user machines should have the same configuration. And, yes, this means that you have to duplicate your documentation too. If you're doing some experimental configuration I'd suggest doing it on honesty first, and then once you've got everything figured out and stable duplicating onto hope. You can use hope first though if it suits you. Try not to let too much time pass with the systems in different configurations, generally you should do your sysadmin work on both systems immediately one after the other.

Administrator logins

The way logins work for administrators (as opposed to normal users) is that you have two logins. One login is for the administrative server (currently there is only one of these, charity), and the other login is your Kerberos/LDAP login that gets you access to the user machines. You can use different passwords if you want, but the usernames (and UIDs) will be the same. All administrators are members of the 'sudo' group on charity and in LDAP. This means that you can use your sudo privileges on either administrative machines or user machines.


Etckeeper is now installed on charity (and hope and honesty), which means the correct procedure after making edits to files in /etc is to issue:

$ sudo etckeeper commit "my reasoning here"

after you're done with your changes. There's an auto commit every day, and an autocommit whenever you apt-get install something. You can manually commit your changes as above. To see the commit log for a particular file:

$ sudo bzr log /etc/passwd

To revert an unwanted or bad change, work out which revision you want to revert to (see log above) and run:

$ sudo bzr revert --revision <commit number> <file>

For example, to restore the /etc/passwd file to the state it was in in revision 3,

$ sudo bzr revert --revision 3 /etc/passwd

To check for uncommited changes, run (From inside /etc):

$ sudo bzr status

For example:

$ sudo bzr status

If you see no output, there are no uncommited changes.

Setting up an Ubuntu server

When configuring a new server consider the following checklist:

Make sure the en_AU.UTF-8 locale is generated and configured

# locale-gen en_AU.UTF-8
# update-locale LANG=en_AU.UTF-8

Configure /etc/skel

# cd /etc/skel
# vim .bashrc


# vim .profile

At the end:

export EDITOR=/usr/bin/vim

script_path=~/.session/`date +%Y-%m-%d-%H%M%S`
script -f -q -a -t $script_path.script 2> $script_path.timing && exit
# vim .vimrc
set tabstop=2
set shiftwidth=2
set expandtab
# mkdir session
# chmod 700 session/
# cp -R session /root/

Create John's account

Create an account for jj5:

# adduser jj5

Make John feel powerful:

# adduser jj5 sudo

Let John read the log files:

# adduser jj5 adm

Now login as jj5 to continue with configuration.

Authorise keys

Vim tabs

See Vim: configuring spaces instead of tabs.

$ vim ~/.vimrc
set tabstop=2
set shiftwidth=2
set expandtab
$ sudo cp ~/.vimrc /root/

Configure SSH client

Check for an existing ~/.ssh/id_rsa file. If there isn't one:

$ ssh-keygen -t rsa

Then configure the SSH client:

$ vim ~/.ssh/config
Host *
  ServerAliveInterval 240

See Configuring SSH key login for details about how to authorise your new key on particular servers.

Configure vim as your editor

$ vim ~/.profile

And somewhere near the end add:

export EDITOR=/usr/bin/vim

You can do that for your own user and for root. (Or for root you might just want to source /home/jj5/.profile).

Then run:

# update-alternatives --config editor

Enable console colours

$ cd ~
$ vim .bashrc

And uncomment the line:


You might like to do that for root too.

Script your sessions

$ cd ~
$ mkdir session
$ chmod 700 session/
$ vim .profile

And at the end add these two lines:

script_path=~/session/`date +%Y-%m-%d-%H%M%S`
script -f -a -q -t $script_path.script 2> $script_path.timing && exit

You can set that up for root too if you like. If root's .profile is sourcing /home/jj5/.profile then make sure that the /root/session directory is created and has its permissions set at 700.

Also, make sure the script command is the very last command in your .profile. If root is sourcing /home/jj5/.profile make sure . /home/jj5/.profile is the very last line of root's profile.

Disable root SSH password authentication

# vim /etc/ssh/sshd_config
PermitRootLogin without-password

Don't require sudo password

Only do this on development servers. Run

$ sudo visudo

Then change the line:

%sudo ALL=(ALL) ALL


#%sudo ALL=(ALL) ALL

jj5-bin in root path

# vim /root/.profile

Somewhere near the end add:

if [ -d "/home/jj5/bin" ] ; then

Then use sudo -i rather than sudo -s for a sudo login shell and root's path will have my scripts in it.

Make sure the /home/jj5/bin directory exists and that it's permissions are 755.


# apt-get install etckeeper


# cd /etc/network/if-pre-up.d/
# vim iptables
/sbin/iptables -F
/sbin/iptables-restore < /etc/iptables.up.rules
# chmod +x iptables
# vim /etc/iptables.up.rules
#  Allow all loopback (lo0) traffic
-A INPUT -i lo -j ACCEPT
# Drop all traffic to 127/8 that does not use lo0
-A INPUT ! -i lo -d -j REJECT
#  Accept all established inbound connections
#  Allow all outbound traffic
#  Allow SSH connections
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT
# Allow SMTP connections from anywhere
#-A INPUT -p tcp --dport 25 -j ACCEPT
# Allow SSMTP connections from anywhere
#-A INPUT -p tcp -m tcp --dport 465 -j ACCEPT
# Allow POP3 connections from anywhere
#-A INPUT -p tcp --dport 110 -j ACCEPT
# Allow POP3S connections from anywhere
#-A INPUT -p tcp --dport 995 -j ACCEPT
# Allow IMAP connections from anywhere
#-A INPUT -p tcp --dport 143 -j ACCEPT
# Allow IMAPS connections from anywhere
#-A INPUT -p tcp --dport 993 -j ACCEPT
# Allow HTTP connections from anywhere
#-A INPUT -p tcp --dport 80 -j ACCEPT
# Allow HTTPS connections from anywhere
#-A INPUT -p tcp --dport 443 -j ACCEPT
# Allow DNS traffic
#-A INPUT -p udp -m udp --dport 53 -j ACCEPT
#-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
# Allow Samba traffic from anywhere
#-A INPUT -p udp -m udp --dport 137 -j ACCEPT
#-A INPUT -p udp -m udp --dport 138 -j ACCEPT
#-A INPUT -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT
#-A INPUT -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT
# Allow svnserve
#-A INPUT -p tcp --dport 3690 -j ACCEPT
# Accept anything from sixsigma
# Accept anything from hope
# Accept anything from honesty
# Accept anything from charity
# Accept anything from courtesy
# Accept anything from modesty
# Accept anything from devotion
# Accept anything from respect
# Accept anything from trust
# Accept anything from humility
# Accept anything from courage
# Accept anything from empathy
# Allow MySQL connections from John's house
#-A INPUT -s -p tcp -m tcp --dport 3306 -j ACCEPT
# Allow MySQL connections from localhost
#-A INPUT -s -p tcp -m tcp --dport 3306 -j ACCEPT
# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
#-A INPUT -j LOG --log-prefix "iptables debug: " --log-level 7
# Reject all other inbound - default deny unless explicitly allowed policy

Fully qualified mailname

Check the /etc/mailname file and confirm it's the fully qualified domain name, e.g. charity.progclub.org.

Fully qualified hostname

# hostname -f
# cd /etc/network/if-pre-up.d/
# cp iptables hosts
# vim hosts
# cat hosts
cp /etc/hosts.master /etc/hosts
# cp hosts ../if-up.d/
# vim /etc/hosts.master
# cat /etc/hosts.master
#localhost       hostname.localdomain localhost.localdomain localhost
#sixsigma    sixsigma.blackbrick.com sixsigma   sixsigma.blackbrick.com sixsigma-public
#hope    hope.progclub.net hope  hope.progclub.net hope-public
#honesty    honesty.progclub.net honesty  honesty.progclub.net honesty-public
#charity    charity.progclub.org charity  charity.progclub.org charity-public
#courtesy   courtesy.blackbrick.com courtesy   courtesy.blackbrick.com courtesy-public
#modesty    modesty.blackbrick.com modesty  modesty.blackbrick.com modesty-public
#devotion   devotion.blackbrick.com devotion   devotion.blackbrick.com devotion-public
#respect    respect.blackbrick.com respect     respect.blackbrick.com respect-public
#trust    trust.blackbrick.com trust     trust.blackbrick.com trust-public
#humility    humility.blackbrick.com humility     humility.blackbrick.com humility-public
#courage    courage.blackbrick.com courage     courage.blackbrick.com courage-public
#empathy    empathy.jj5.net empathy     empathy.jj5.net empathy-public
# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
# hostname -f


# apt-get install fail2ban
# vim /etc/fail2ban/jail.local

Something like this:

ignoreip =
bantime  = 3600
maxretry = 3
destemail = admin@blackbrick.com
banaction = iptables-multiport
mta = sendmail
action = %(action_mwl)s
enabled = true
enabled = false
enabled   = false
enabled = true
enabled = true
enabled = false
enabled = true
enabled = true
enabled  = false
enabled  = false
enabled  = false
enabled  = true
enabled  = false
enabled  = true
enabled  = false
enabled  = false
enabled  = false

Configure mail and forwarding

# apt-get install bsd-mailx postfix
Package configuration

 ┌────────────────────────┤ Postfix Configuration ├────────────────────────┐
 │                                                                         │
 │ Please select the mail server configuration type that best meets your
 │ needs.
 │                                                                         ▒
 │  No configuration:                                                      ▒
 │   Should be chosen to leave the current configuration unchanged.        ▒
 │  Internet site:                                                         ▒
 │   Mail is sent and received directly using SMTP.                        ▒
 │  Internet with smarthost:                                               ▒
 │   Mail is received directly using SMTP or by running a utility such     ▒
 │   as fetchmail. Outgoing mail is sent using a smarthost.                ▒
 │  Satellite system:                                                      ▒
 │   All mail is sent to another machine, called a 'smarthost', for        ▒
 │ delivery.                                                               ▒
 │  Local only:
 │   The only delivered mail is the mail for local users. There is no
 │ network.
 │                                 <Ok>
 │                                                                         │

Package configuration

                   ┌──────┤ Postfix Configuration ├───────┐
                   │ General type of mail configuration:  │
                   │                                      │
                   │       No configuration               │
                   │       Internet Site                  │
                   │       Internet with smarthost        │
                   │     * Satellite system               │
                   │       Local only                     │
                   │                                      │
                   │                                      │
                   │       <Ok>           <Cancel>        │
                   │                                      │
Package configuration

┌─────────────────────────┤ Postfix Configuration ├─────────────────────────┐
│ The "mail name" is the domain name used to "qualify" _ALL_ mail           │
│ addresses without a domain name. This includes mail to and from <root>:   │
│ please do not make your machine send out mail from root@example.org       │
│ unless root@example.org has told you to.                                  │
│                                                                           │
│ This name will also be used by other programs. It should be the single,   │
│ fully qualified domain name (FQDN).                                       │
│                                                                           │
│ Thus, if a mail address on the local host is foo@example.org, the         │
│ correct value for this option would be example.org.                       │
│                                                                           │
│ System mail name:                                                         │
│                                                                           │
│ example.progclub.net_____________________________________________________ │
│                                                                           │
│                    <Ok>                        <Cancel>                   │
│                                                                           │
Package configuration

┌─────────────────────────┤ Postfix Configuration ├─────────────────────────┐
│ Please specify a domain, host, host:port, [address] or [address]:port.    │
│ Use the form [destination] to turn off MX lookups. Leave this blank for   │
│ no relay host.                                                            │
│                                                                           │
│ Do not specify more than one host.                                        │
│                                                                           │
│ The relayhost parameter specifies the default host to send mail to when   │
│ no entry is matched in the optional transport(5) table. When no relay     │
│ host is given, mail is routed directly to the destination.                │
│                                                                           │
│ SMTP relay host (blank for none):                                         │
│                                                                           │
│ mail.progclub.org________________________________________________________ │
│                                                                           │
│                    <Ok>                        <Cancel>                   │
│                                                                           │
# vim /etc/aliases
# cat /etc/aliases
# See man 5 aliases for format
postmaster:    jj5
root: jj5
# newaliases

Unattended upgrades

# apt-get install unattended-upgrades
# vim /etc/apt/apt.conf.d/50unattended-upgrades
# cat /etc/apt/apt.conf.d/50unattended-upgrades
// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
        "Ubuntu lucid-security";
        "Ubuntu lucid-updates";

// List of packages to not update
Unattended-Upgrade::Package-Blacklist {
//      "vim";
//      "libc6";
//      "libc6-dev";
//      "libc6-i686";

// Send email to this address for problems or packages upgrades 
// If empty or unset then no email is sent, make sure that you
// have a working mail setup on your system. The package 'mailx'
// must be installed or anything that provides /usr/bin/mail.
Unattended-Upgrade::Mail "admin@progclub.org";

// Do automatic removal of new unused dependencies after the upgrade
// (equivalent to apt-get autoremove)
Unattended-Upgrade::Remove-Unused-Dependencies "true";

// Automatically reboot *WITHOUT CONFIRMATION* if a
// the file /var/run/reboot-required is found after the upgrade
Unattended-Upgrade::Automatic-Reboot "true";

// Use apt bandwidth limit feature, this example limits the download
// speed to 70kb/sec
//Acquire::http::Dl-Limit "70";
# vim /etc/apt/apt.conf.d/10periodic
# cat /etc/apt/apt.conf.d/10periodic
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";

MySQL character set and collation

If you've installed MySQL:

# vim /etc/mysql/my.cnf

In the [mysqld] section:



# apt-get install logwatch
# cd /var/cache
# mkdir logwatch
# cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/
# vim /etc/logwatch/conf/logwatch.conf
MailTo = admin@blackbrick.com
Detail = High

Allocating user IDs

If you need to allocate a user id for a member, use their member number from the Members page. Member numbers start at 1000 and increment from there.

If you need to allocate a user id for a system user, such as a user that a process will run as, check for UIDs starting from 500 and use the next available one. If you create a user on one machine with a particular UID you should create the same user with the same UID on all the other machines too.

Adding a new user

Updating members database

At the moment member information is recorded in Members. A member should have a Wiki account to start with (create one for them if they haven't already created their own), then an entry in the Members page (where they are allocated a member number), then the "User info" on their user page should be filled out (leave fields as "Not disclosed" unless you have their permission to publish their details). After the Members page has been updated you will have:

  • {username}: the Unix/Kerberos username of the new member
  • {member number}: the member's User ID
  • {group}: whether the user has 'sudo' membership or not
    • there are two groups at ProgClub: 'sudo' and 'user'
      • everyone is in 'user' (gidNumber == 500)
      • administrators are in 'sudo' (gidNumber == 27)

And that's enough information to create their account as detailed below.

Kerberos administration

To create a new user in Kerberos:

  • SSH to charity.progclub.org
  • Login
  • Run kadmin addprinc:
$ sudo kadmin -p {your username} -q "addprinc {username}"
    • Note: use sudo so that the log file can be written
    • {your username} is your ProgClub username
    • {username} is the ProgClub username of the user your are adding

If the user is an administrator, add them to /etc/krb5kdc/kadm5.acl. E.g., at the end of the file,

username@PROGCLUB.ORG *

replacing 'username' as appropriate. You need to restart the Kerberos administration server after this change:

$ sudo /etc/init.d/krb5-admin-server restart

Creating home directory or administrative login

  • SSH to charity.progclub.org
  • Login

If you are creating an administrative account, run:

$ sudo adduser --uid {member number} --gid 27 {username}

If you are creating a normal user account, run:

$ sudo adduser --uid {member number} --gid 500 --disabled-password {username}

(This creates the user's home directory)

LDAP administration

To create a user in LDAP:

  • Open the LDAP administration tool
  • Login: cn=admin,dc=progclub,dc=org (/home/jj5/login_ldap for password)
  • Navigate to: ou=users
  • Click 'Create new entry here'
  • Choose 'Default'
  • Container should be: ou=users,dc=progclub,dc=org
  • ObjectClass: inetOrgPerson
  • Select the RDN attribute: Username (uid)
  • Enter 'cn', this is the Common Name, i.e. the user's full name, or their username
  • Enter 'sn', this is the Surname, i.e. the user's last name, if it's disclosed, else username
  • Enter 'displayName', i.e. the user's full name, if it's disclosed, else username
  • Enter 'Email', i.e. username@progclub.org if they want an email account, otherwise the user's email address, if it's disclosed, else blank
  • Enter 'givenName', i.e. the user's first name, if it's disclosed, else blank
  • Enter 'Username', this is their Posix username, all lowercase
  • Click 'Create Object'
  • Review the entry and click 'Commit'
  • Scroll down to 'objectClass' and click 'add value'
  • Select 'posixAccount'
  • Enter the 'gidNumber': 27 for admin, 500 for user
  • Enter the 'homeDirectory': /home/{username}, e.g. /home/jj5
  • Enter the 'uidNumber': this is the member number allocated on the web-site in Members
  • Click 'Add ObjectClass and Attributes'
  • Review the entry and click 'Update Object'
  • Scroll down to 'objectClass' and click 'add value'
  • Select 'shadowAccount'
  • Click 'Add new ObjectClass'
  • Click 'Add new attribute'
  • Select 'loginShell'
  • Enter "/bin/bash"
  • Click 'Update Object'
  • Review changes and click 'Update Object'
  • Navigate to ou=groups
  • Select 'user'
  • Select the "Generic: Posix Group" template
  • Select the new user (all users should be selected)
  • Click 'Update Object'
  • Review changes and click 'Update Object'

If the user is an administrator:

  • Navigate to ou=groups
  • Select 'sudo'
  • Select the "Generic: Posix Group" template
  • Select the new user (all administrative users should be selected)
  • Click 'Update Object'
  • Review changes and click 'Update Object'

Email account

Login to email administration. The username is postmaster@progclub.org, and the password is in /home/jj5/login_mail.

  • Click Virtual List => Add Mailbox
    • Username: the member's username
    • @progclub.org (it will default to progclub.co, which is wrong, so be careful with this)
    • Password: make a random one up and email it to the new member, they can change it themselves
    • Name: if the member gave you permission to use their name, then their name. Otherwise their username.
    • Active: checked
    • Send welcome email: checked
  • Click 'Add mailbox'.

Once you've done that, login to the user email administration facility with the member's username and the password you generated.

  • Click 'Change your forward'.
    • To: the member's non-ProgClub email address
    • Select: Deliver to the local mailbox
  • Click 'Edit alias'.

Subscribe member to ProgClub announcement

Login to announcement administration. The password is in /home/jj5/login_list.

  • Click 'Membership Management...'
  • Click 'Mass Subscription'
  • Subscribe these users now or invite them?: Subscribe
  • Send welcome messages to new subscribees?: No
  • Send notifications of new subscriptions to the list owner?: Yes
  • Enter the ProgClub email address of the new member
  • Click 'Submit Your Changes'

Tell the member about their new account

Now send an email to the member at their new email address (username@progclub.org) letting them know the details of their new account. You should include:

Subscribe member to mailing lists

You should already have subscribed them to the announcement list. You might also like to subscribe their new ProgClub email address to any or all of the following: