Admin reference

From ProgClub
Jump to: navigation, search

This page is a reference for ProgClub system administrators. For information about ProgClub domains, see Domains. For information about member services, see Services. See Machines for information about hosts on the ProgClub network. See Projects for current projects or check out our Forums to get in touch.

Administrative reference

The migration to Network v2 is now in full swing!

If you're administering ProgClub assets, please document your actions on the wiki. See the relevant pages:

See Future machine names for the kind of stuff we're on about.

See, or update, Network administration for work that needs to be done.

Denying hosts with UFW

So the new systems integrity and strength have been deployed as part of the Network v2 migration. They are mostly managed via AWS and Salt. If you're watching the web-logs and see a brute force attempt against WordPress etc. then you can issue a "temporary" ban for the offending IP address (it's "temporary" because you should remove it after a day or two once the problem has gone away). The actual firewall for our services is with AWS and its security groups, so we leverage this fact to simplify our UFW rules, for example, by using a default allow incoming policy.

root@integrity:/# ufw default allow incoming
root@integrity:/# ufw default allow outgoing
root@integrity:/# ufw reset
root@integrity:/# ufw default allow incoming
root@integrity:/# ufw default allow outgoing
root@integrity:/# ufw deny from 127.206.172.17
root@integrity:/# ufw status
root@integrity:/# ufw enable
root@integrity:/# ufw status

Deleting ufw rules

  1. ufw status numbered
  2. ufw delete 123

Wiki maintenance

If you need to update the wiki sidebar see MediaWiki:Sidebar.

Code of conduct

As a ProgClub administrator you have a lot of power. You have the capability to destroy ProgClub's files and configuration, to access all of ProgClub's databases, to pretend to be other users, and to access other users' private data. We expect you won't abuse your power. Specifically, under no circumstances should you:

  • Delete or modify in inappropriate ways ProgClub files or members' files
  • Read other users' email or private files
  • Copy data from any of ProgClub's administrative databases into your own system
  • Read or copy any data from members' databases that are not yours
  • Pretend to be another member
  • Publicly disclose member details, such as name or email address, without their permission

All of your systems administration activity should be documented, and you shouldn't be doing anything evil. If you're not sure what qualifies as evil, if you have any doubt at all, please ask.

Although ProgClub doesn't make any guarantees, members should be able to feel as though their privacy is respected at ProgClub, and they should be able to feel confident that administrators aren't spying on them by reading their email or their other data.

Where are the keys?

Your administrator login on charity and your member account in LDAP will give you sudo privs on all of the ProgClub machines, both administrative and user machines. Some tools require special purpose logins, and for those you will find login details in the /home/jj5/login_* files which you can access from any of the machines.

Tools

For login information:

  • See /home/jj5/login_ldap for the LDAP login.
  • See /home/jj5/login_data for the database logins.
  • See /home/jj5/login_mail for the email login.
  • See /home/jj5/login_list for the email lists logins.

Documentation

The process for documenting network administration activity is:

Note: don't use screen while administering systems, because it frags PuTTY's scroll buffer, and you need that in order to copy-and-paste your session log.

  • Update the network administration page with a note about what you did:

It's not a bad idea to link to the relevant section(s) of the admin page(s) from the network administration 'done' note, but that's not strictly necessary.

  • Let everyone know what new features are available:

Administrative and user machines

As explained on the Machines page ProgClub separates its machines into two groups: administrative machines and user machines. There is only one administrative machine, charity. There are two user machines, hope and honesty. If you're doing sysadmin work on the user machines, make sure you duplicate your work and create an identical system configuration on each machine. Both user machines should have the same configuration. And, yes, this means that you have to duplicate your documentation too. If you're doing some experimental configuration I'd suggest doing it on honesty first, and then once you've got everything figured out and stable duplicating onto hope. You can use hope first though if it suits you. Try not to let too much time pass with the systems in different configurations, generally you should do your sysadmin work on both systems immediately one after the other.

Administrator logins

The way logins work for administrators (as opposed to normal users) is that you have two logins. One login is for the administrative server (currently there is only one of these, charity), and the other login is your Kerberos/LDAP login that gets you access to the user machines. You can use different passwords if you want, but the usernames (and UIDs) will be the same. All administrators are members of the 'sudo' group on charity and in LDAP. This means that you can use your sudo privileges on either administrative machines or user machines.

Etckeeper

Etckeeper is now installed on charity (and hope and honesty), which means the correct procedure after making edits to files in /etc is to issue:

$ sudo etckeeper commit "my reasoning here"

after you're done with your changes. There's an auto commit every day, and an autocommit whenever you apt-get install something. You can manually commit your changes as above. To see the commit log for a particular file:

$ sudo bzr log /etc/passwd

To revert an unwanted or bad change, work out which revision you want to revert to (see log above) and run:

$ sudo bzr revert --revision <commit number> <file>

For example, to restore the /etc/passwd file to the state it was in in revision 3,

$ sudo bzr revert --revision 3 /etc/passwd

To check for uncommited changes, run (From inside /etc):

$ sudo bzr status

For example:

$ sudo bzr status
modified:
  crontab

If you see no output, there are no uncommited changes.

Mailman

ProgClub uses the Mailman software to manage its mailing lists. From time to time some spam makes it through the mail filters onto the list. Such spam then really needs to be removed otherwise it will become a part of the HTML list archives published on our site.

To remove spam from a mailing list, ssh to charity and then:

$ sudo -s
# cd /var/lib/mailman/archives/private
# vim list.mbox/list.mbox

Then from vim delete the offending messages. When you're done:

# cd ../..
# bin/arch --wipe list
# chown -R list:www-data archives/private/list

And then you're back in business.

Setting up an Ubuntu server

When configuring a new server consider the following checklist:

SSH in as root

Your virtual server will be configured as an SSH system accessible by root. SSH to your new system as root and then change the password:

passwd

Base-level upgrade

Make sure all existing packages are up-to-date:

apt-get update
apt-get dist-upgrade
reboot

Make sure vim is installed

apt-get install vim

Make sure the en_AU.UTF-8 locale is generated and configured

Run the following commands as root:

locale-gen en_AU.UTF-8
update-locale LANG=en_AU.UTF-8

Alternatively you might try the following (untested):

apt-get install language-pack-en

Or try:

dpkg-reconfigure locales

Configure the timezone

I know UTC is best-practice, but Sydney time is just too convenient for me...

# dpkg-reconfigure tzdata

Then select Australia / Sydney.

Configure cron

# vim /etc/crontab

And set the times like this:

1 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
1 0    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
1 0    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
1 0    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

So all the standard scripts will run at 1 minute past midnight in the Australia/Sydney timezone.

Configure NTP

Type the following to see the current date:

date -R

Then install NTP utilities:

apt-get install ntp

Type the following to see the servers you are sync'ing with:

ntpq -p

Print the date again and see if it updated:

date -R

Configure /etc/skel

Run the following as root:

cd /etc/skel
mkdir .session
chmod 700 .session/
mkdir .ssh
chmod 700 .ssh/
cat > .ssh/config <<EOF
Host *
  ServerAliveInterval 240
EOF
cat > .vimrc <<EOF
set tabstop=2
set shiftwidth=2
set expandtab
syntax on
EOF
cp .vimrc /root/
cd /root
sed -e 's/#force/force/' .bashrc > .bashrc.new
mv .bashrc.new .bashrc

See Vim: configuring spaces instead of tabs.

Create John's account

Create an account for jj5:

adduser --gecos ,,, jj5

Make John feel powerful:

adduser jj5 sudo
adduser jj5 adm

Now logout of root and login as jj5 to continue with configuration.

Authorise keys

Before logging in as jj5, from an existing system that has access (e.g. sixsigma), copy in the authorised keys:

scp ~/.ssh/authorized_keys jj5@example.com:.ssh/

Then SSH to the new machine as jj5 and run:

sudo cp -R ~/.ssh /root/

Configure SSH client

Check for an existing ~/.ssh/id_rsa file. If there isn't one:

$ ssh-keygen -t rsa

See Configuring SSH key login for details about how to authorise your new key on particular servers.

Install sudo

apt-get install sudo

Configure vim as your editor

Run:

sudo update-alternatives --config editor

Choose vim.basic from the list.

Disable root SSH password authentication

sed -i -e 's/PermitRootLogin yes/PermitRootLogin without-password/' /etc/ssh/sshd_config

Create backup account

sudo adduser --disabled-password --gecos ,,, --uid 999 jjbackup

Install rsync

sudo apt-get install rsync

Don't require sudo password

Only do this on development servers. Run

sudo visudo

Then change the line:

%sudo ALL=(ALL) ALL

To:

#%sudo ALL=(ALL) ALL
%sudo ALL=(ALL) NOPASSWD: ALL

Then add backup permissions:

# Cmnd alias specification
Cmnd_Alias RSYNCBACKUP = /usr/bin/rsync --server --sender -* . *
Cmnd_Alias RSYNCRESTORE = /usr/bin/rsync --server -* . /var/restore
Cmnd_Alias JJ5BIN = /home/jj5/bin/*

# User privilege specification
root  ALL=(ALL:ALL) ALL
jjbackup ALL=NOPASSWD: RSYNCBACKUP
jjbackup ALL=NOPASSWD: RSYNCRESTORE
jjbackup ALL=NOPASSWD: JJ5BIN

Configure time server

# apt-get install ntpdate
# vim /etc/cron.daily/ntptime
#!/bin/sh
#https://help.ubuntu.com/community/UbuntuTime
ntpdate ntp.ubuntu.com > /dev/null
# chmod +x /etc/cron.daily/ntptime

jj5-bin integration

Make sure svn is installed:

sudo apt-get install subversion

Make sure any existing ~/bin directory is removed:

[ -d ~/bin -a ! -d ~/bin/.svn ] && mv ~/bin ~/bin-$HOSTNAME-orig

Checkout jj5-bin:

svn co https://svn.jj5.net/svn/public/jjrepo/jj5-bin/trunk/bin ~/bin && \
chmod +x ~/bin/*

Apply jj5-bin for all users:

sudo /home/jj5/bin/apply-standard-profile.sh

Configure subversion servers

sudo vim /etc/subversion/servers

For jj5.net servers:

[groups]
jj5 = *.jj5.net
blackbrick = *.blackbrick.com
progclub = *.progclub.org

[jj5]
store-plaintext-passwords = yes

[blackbrick]
store-plaintext-passwords = yes

[progclub]
store-plaintext-passwords = yes

For Blackbrick servers:

[groups]
jj5 = *.jj5.net
blackbrick = *.blackbrick.com
progclub = *.progclub.org

[jj5]
store-plaintext-passwords = no

[blackbrick]
store-plaintext-passwords = no

[progclub]
store-plaintext-passwords = yes

For ProgClub servers:

[groups]
jj5 = *.jj5.net
blackbrick = *.blackbrick.com
progclub = *.progclub.org

[jj5]
store-plaintext-passwords = no

[blackbrick]
store-plaintext-passwords = no

[progclub]
store-plaintext-passwords = yes

Create Blackbrick accounts

If this is a Blackbrick server:

sudo adduser --disabled-password --gecos ,,, sanguinev
sudo adduser --disabled-password --gecos ,,, tasaio
sudo adduser --disabled-password --gecos ,,, adriano

ProgClub servers use Kerberos.

Install Etckeeper

sudo apt-get install etckeeper

Configure Etckeeper email notifications

The old way

note Configuring etckeeper email commit notifications
sudo -s
apt-get install bzr
cd /srv
mkdir bzr
cd bzr
bzr branch lp:bzr-hookless-email
cd /etc/cron.d
echo "*/5 * * * * root /srv/bzr/bzr-hookless-email/bzr_hookless_email.py" \
 "-e log@blackbrick.com -r /etc" > /etc/cron.d/etckeeper-commit-notification
#vim etckeeper-commit-notification
exit

Note: don't forget to change the email address from log@blackbrick.com for jj5.net and ProgClub servers.

If you want to update the hookless email script:

sudo -s
cd /srv/bzr/bzr-hookless-email/
bzr pull

Also, you might need to patch this bug.

The new way

# apt-get install git-email

Edit /etc/.git/hooks/post-commit

#!/bin/bash
# Install this file as /etc/.git/hooks/post-commit
# Dependencies: etckeeper git-email
git send-email --from etckeeper --to root --confirm=never --quiet @{1}

Then:

# chmod +x /etc/.git/hooks/post-commit

Configure IPTables

sudo -s
cd /etc/network/if-pre-up.d/
cat > iptables <<EOF
#!/bin/sh
/sbin/iptables -F
/sbin/iptables-restore < /etc/iptables.up.rules
EOF
chmod +x iptables
cat > /etc/iptables.up.rules <<EOF
*filter
#  Allow all loopback (lo0) traffic
-A INPUT -i lo -j ACCEPT
# Drop all traffic to 127/8 that does not use lo0
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT
#  Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
#  Allow all outbound traffic
-A OUTPUT -j ACCEPT
#  Allow SSH connections
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT
# Allow SMTP connections from anywhere
#-A INPUT -p tcp --dport 25 -j ACCEPT
# Allow SSMTP connections from anywhere
#-A INPUT -p tcp -m tcp --dport 465 -j ACCEPT
# Allow POP3 connections from anywhere
#-A INPUT -p tcp --dport 110 -j ACCEPT
# Allow POP3S connections from anywhere
#-A INPUT -p tcp --dport 995 -j ACCEPT
# Allow IMAP connections from anywhere
#-A INPUT -p tcp --dport 143 -j ACCEPT
# Allow IMAPS connections from anywhere
#-A INPUT -p tcp --dport 993 -j ACCEPT
# Allow HTTP connections from anywhere
#-A INPUT -p tcp --dport 80 -j ACCEPT
# Allow HTTPS connections from anywhere
#-A INPUT -p tcp --dport 443 -j ACCEPT
# Allow DNS traffic
#-A INPUT -p udp -m udp --dport 53 -j ACCEPT
#-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
# Allow Samba traffic from anywhere
#-A INPUT -p udp -m udp --dport 137 -j ACCEPT
#-A INPUT -p udp -m udp --dport 138 -j ACCEPT
#-A INPUT -m state --state NEW -m tcp -p tcp --dport 139 -j ACCEPT
#-A INPUT -m state --state NEW -m tcp -p tcp --dport 445 -j ACCEPT
# Allow svnserve
#-A INPUT -p tcp --dport 3690 -j ACCEPT
# Accept anything from sixsigma
#-A INPUT -s 67.207.139.99 -j ACCEPT
#-A INPUT -s 10.183.97.22 -j ACCEPT
# Accept anything from hope
#-A INPUT -s 67.207.130.204 -j ACCEPT
#-A INPUT -s 10.183.97.27 -j ACCEPT
# Accept anything from honesty
#-A INPUT -s 67.207.129.103 -j ACCEPT
#-A INPUT -s 10.183.97.45 -j ACCEPT
# Accept anything from charity
#-A INPUT -s 67.207.128.184 -j ACCEPT
#-A INPUT -s 10.183.97.44 -j ACCEPT
# Accept anything from courtesy
#-A INPUT -s 67.207.128.37 -j ACCEPT
#-A INPUT -s 10.183.96.229 -j ACCEPT
# Accept anything from modesty
#-A INPUT -s 67.207.128.220 -j ACCEPT
#-A INPUT -s 10.183.96.34 -j ACCEPT
# Accept anything from devotion
#-A INPUT -s 67.207.130.61 -j ACCEPT
#-A INPUT -s 10.183.96.214 -j ACCEPT
# Accept anything from respect
#-A INPUT -s 50.57.50.13 -j ACCEPT
#-A INPUT -s 10.183.96.10 -j ACCEPT
# Accept anything from trust
#-A INPUT -s 50.57.50.15 -j ACCEPT
#-A INPUT -s 10.183.96.21 -j ACCEPT
# Accept anything from humility
#-A INPUT -s 50.57.50.41 -j ACCEPT
#-A INPUT -s 10.183.96.22 -j ACCEPT
# Accept anything from courage
#-A INPUT -s 50.57.50.42 -j ACCEPT
#-A INPUT -s 10.183.96.23 -j ACCEPT
# Accept anything from empathy
#-A INPUT -s 50.57.50.43 -j ACCEPT
#-A INPUT -s 10.183.96.24 -j ACCEPT
# Allow MySQL connections from John's house
#-A INPUT -s 120.150.174.187/32 -p tcp -m tcp --dport 3306 -j ACCEPT
# Allow MySQL connections from localhost
#-A INPUT -s 127.0.0.1/32 -p tcp -m tcp --dport 3306 -j ACCEPT
# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
# log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
#-A INPUT -j LOG --log-prefix "iptables debug: " --log-level 7
# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT
COMMIT
EOF
vim /etc/iptables.up.rules

Copy in the SSH deny rules that are applicable to the new server. These start on the second line after the *filter line.

#empathy:
# Reject SSH connections from sixsigma
-A INPUT -s 10.183.97.22/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.139.99/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from respect
-A INPUT -s 10.183.96.10/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.13/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from trust
-A INPUT -s 10.183.96.21/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.15/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from humility
-A INPUT -s 10.183.96.22/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.41/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courage
-A INPUT -s 10.183.96.23/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.42/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courtesy
-A INPUT -s 10.183.96.229/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.37/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from modesty
-A INPUT -s 10.183.96.34/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.220/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#sixsigma:
# Reject SSH connections from respect
-A INPUT -s 10.183.96.10/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.13/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from trust
-A INPUT -s 10.183.96.21/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.15/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from humility
-A INPUT -s 10.183.96.22/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.41/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courage
-A INPUT -s 10.183.96.23/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.42/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courtesy
-A INPUT -s 10.183.96.229/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.37/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from modesty
-A INPUT -s 10.183.96.34/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.220/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#respect:
# Reject SSH connections from trust
-A INPUT -s 10.183.96.21/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.15/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from humility
-A INPUT -s 10.183.96.22/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.41/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courage
-A INPUT -s 10.183.96.23/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.42/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courtesy
-A INPUT -s 10.183.96.229/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.37/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from modesty
-A INPUT -s 10.183.96.34/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.220/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#trust:
# Reject SSH connections from humility
-A INPUT -s 10.183.96.22/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.41/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courage
-A INPUT -s 10.183.96.23/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.42/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courtesy
-A INPUT -s 10.183.96.229/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.37/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from modesty
-A INPUT -s 10.183.96.34/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.220/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#humility:
# Reject SSH connections from courage
-A INPUT -s 10.183.96.23/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 50.57.50.42/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from courtesy
-A INPUT -s 10.183.96.229/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.37/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from modesty
-A INPUT -s 10.183.96.34/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.220/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#courage:
# Reject SSH connections from courtesy
-A INPUT -s 10.183.96.229/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.37/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from modesty
-A INPUT -s 10.183.96.34/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.220/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#courtesy:
# Reject SSH connections from modesty
-A INPUT -s 10.183.96.34/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.220/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#modesty:
# Reject SSH connections from devotion
-A INPUT -s 10.183.96.214/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.61/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#devotion:
# Reject SSH connections from charity
-A INPUT -s 10.183.97.44/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.128.184/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#charity:
# Reject SSH connections from hope
-A INPUT -s 10.183.97.27/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.130.204/32 -p tcp -m tcp --dport 22 -j REJECT
# Reject SSH connections from honesty
-A INPUT -s 10.183.97.45/32 -p tcp -m tcp --dport 22 -j REJECT
-A INPUT -s 67.207.129.103/32 -p tcp -m tcp --dport 22 -j REJECT
#hope:
#honesty:

Then exit:

exit

Fully qualified mailname

Check the /etc/mailname file and confirm it's the fully qualified domain name, e.g. charity.progclub.org.

sudo vim /etc/mailname

Fully qualified hostname

Get the FQDN:

hostname -f

E.g.:

honesty

If the FQDN is not fully qualified:

sudo -s
cd /etc/network/if-pre-up.d/
cp iptables hosts
cat > hosts <<EOF
#!/bin/sh
cp /etc/hosts.master /etc/hosts
EOF
cp hosts ../if-up.d/
cat > /etc/hosts.master <<EOF
#localhost
127.0.0.1       hostname.localdomain localhost.localdomain localhost
#sixsigma
10.183.97.22    sixsigma.blackbrick.com sixsigma
67.207.139.99   sixsigma.blackbrick.com sixsigma-public
#hope
10.183.97.27    hope.progclub.net hope
67.207.130.204  hope.progclub.net hope-public
#honesty
10.183.97.45    honesty.progclub.net honesty
67.207.129.103  honesty.progclub.net honesty-public
#charity
10.183.97.44    charity.progclub.org charity
67.207.128.184  charity.progclub.org charity-public
#courtesy
10.183.96.229   courtesy.blackbrick.com courtesy
67.207.128.37   courtesy.blackbrick.com courtesy-public
#modesty
10.183.96.34    modesty.blackbrick.com modesty
67.207.128.220  modesty.blackbrick.com modesty-public
#devotion
10.183.96.214   devotion.blackbrick.com devotion
67.207.130.61   devotion.blackbrick.com devotion-public
#respect
10.183.96.10    respect.blackbrick.com respect
50.57.50.13     respect.blackbrick.com respect-public
#trust
10.183.96.21    trust.blackbrick.com trust
50.57.50.15     trust.blackbrick.com trust-public
#humility
10.183.96.22    humility.blackbrick.com humility
50.57.50.41     humility.blackbrick.com humility-public
#courage
10.183.96.23    courage.blackbrick.com courage
50.57.50.42     courage.blackbrick.com courage-public
#sonydadc
10.177.23.37    sonydadc.blackbrick.com sonydadc
198.61.177.216  sonydadc.blackbrick.com sonydadc-public
#empathy
10.183.96.24    empathy.jj5.net empathy
50.57.50.43     empathy.jj5.net empathy-public
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
EOF
vim /etc/hosts.master

While in vim change the 127.0.0.1 address to the machine name, not 'hostname.localname' as in the template. Rather use something like 'charity.localdomain'. You can run hostname -f again to check that everything worked.

cp /etc/hosts.master /etc/hosts
exit
hostname -f

E.g.

charity.progclub.org

Install Fail2ban

sudo apt-get install fail2ban
sudo -s
cat > /etc/fail2ban/jail.local <<EOF
[DEFAULT]
ignoreip = 127.0.0.1
bantime  = 3600
maxretry = 3
destemail = admin@blackbrick.com
banaction = iptables-multiport
mta = sendmail
action = %(action_mwl)s
[ssh]
enabled = true
[pam-generic]
enabled = true
[xinetd-fail]
enabled   = false
[ssh-ddos]
enabled = true
[apache]
enabled = true
[apache-multiport]
enabled = false
[apache-noscript]
enabled = true
[apache-overflows]
enabled = true
[vsftpd]
enabled  = false
[proftpd]
enabled  = false
[wuftpd]
enabled  = false
[postfix]
enabled  = true
[couriersmtp]
enabled  = false
[courierauth]
enabled  = true
[sasl]
enabled  = false
[named-refused-udp]
enabled  = false
[named-refused-tcp]
enabled  = false
EOF
exit

Then open in vim to adjust email address or other settings if necessary:

sudo vim /etc/fail2ban/jail.local

Configure mail and forwarding

sudo apt-get install bsd-mailx postfix
Package configuration

 ┌────────────────────────┤ Postfix Configuration ├────────────────────────┐
 │                                                                         │
 │ Please select the mail server configuration type that best meets your
 │ needs.
 │                                                                         ▒
 │  No configuration:                                                      ▒
 │   Should be chosen to leave the current configuration unchanged.        ▒
 │  Internet site:                                                         ▒
 │   Mail is sent and received directly using SMTP.                        ▒
 │  Internet with smarthost:                                               ▒
 │   Mail is received directly using SMTP or by running a utility such     ▒
 │   as fetchmail. Outgoing mail is sent using a smarthost.                ▒
 │  Satellite system:                                                      ▒
 │   All mail is sent to another machine, called a 'smarthost', for        ▒
 │ delivery.                                                               ▒
 │  Local only:
 │   The only delivered mail is the mail for local users. There is no
 │ network.
 │
 │
 │                                 <Ok>
 │                                                                         │
 └─────────────────────────────────────────────────────────────────────────┘

Package configuration



                   ┌──────┤ Postfix Configuration ├───────┐
                   │ General type of mail configuration:  │
                   │                                      │
                   │       No configuration               │
                   │       Internet Site                  │
                   │       Internet with smarthost        │
                   │     * Satellite system               │
                   │       Local only                     │
                   │                                      │
                   │                                      │
                   │       <Ok>           <Cancel>        │
                   │                                      │
                   └──────────────────────────────────────┘
Package configuration

┌─────────────────────────┤ Postfix Configuration ├─────────────────────────┐
│ The "mail name" is the domain name used to "qualify" _ALL_ mail           │
│ addresses without a domain name. This includes mail to and from <root>:   │
│ please do not make your machine send out mail from root@example.org       │
│ unless root@example.org has told you to.                                  │
│                                                                           │
│ This name will also be used by other programs. It should be the single,   │
│ fully qualified domain name (FQDN).                                       │
│                                                                           │
│ Thus, if a mail address on the local host is foo@example.org, the         │
│ correct value for this option would be example.org.                       │
│                                                                           │
│ System mail name:                                                         │
│                                                                           │
│ example.progclub.net_____________________________________________________ │
│                                                                           │
│                    <Ok>                        <Cancel>                   │
│                                                                           │
└───────────────────────────────────────────────────────────────────────────┘
Package configuration


┌─────────────────────────┤ Postfix Configuration ├─────────────────────────┐
│ Please specify a domain, host, host:port, [address] or [address]:port.    │
│ Use the form [destination] to turn off MX lookups. Leave this blank for   │
│ no relay host.                                                            │
│                                                                           │
│ Do not specify more than one host.                                        │
│                                                                           │
│ The relayhost parameter specifies the default host to send mail to when   │
│ no entry is matched in the optional transport(5) table. When no relay     │
│ host is given, mail is routed directly to the destination.                │
│                                                                           │
│ SMTP relay host (blank for none):                                         │
│                                                                           │
│ mail.progclub.org________________________________________________________ │
│                                                                           │
│                    <Ok>                        <Cancel>                   │
│                                                                           │
└───────────────────────────────────────────────────────────────────────────┘
sudo -s
cat > /etc/aliases <<EOF
# See man 5 aliases for format
postmaster: jj5
root: jj5
EOF
newaliases
exit

Install unattended upgrades

sudo apt-get install unattended-upgrades
sudo -s
cat > /etc/apt/apt.conf.d/50unattended-upgrades <<EOF
// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
        "Ubuntu lucid-security";
        "Ubuntu lucid-updates";
}; 

// List of packages to not update
Unattended-Upgrade::Package-Blacklist {
//      "vim";
//      "libc6";
//      "libc6-dev";
//      "libc6-i686";
};

// Send email to this address for problems or packages upgrades 
// If empty or unset then no email is sent, make sure that you
// have a working mail setup on your system. The package 'mailx'
// must be installed or anything that provides /usr/bin/mail.
Unattended-Upgrade::Mail "admin@blackbrick.com";

// Do automatic removal of new unused dependencies after the upgrade
// (equivalent to apt-get autoremove)
Unattended-Upgrade::Remove-Unused-Dependencies "true";

// Automatically reboot *WITHOUT CONFIRMATION* if a
// the file /var/run/reboot-required is found after the upgrade
Unattended-Upgrade::Automatic-Reboot "true";

// Use apt bandwidth limit feature, this example limits the download
// speed to 70kb/sec
//Acquire::http::Dl-Limit "70";
EOF
cat > /etc/apt/apt.conf.d/10periodic <<EOF
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
EOF
exit

Then if you need to change the email address for upgrade reports:

sudo vim /etc/apt/apt.conf.d/50unattended-upgrades

Install Logwatch

sudo apt-get install logwatch
sudo -s
cd /var/cache
mkdir logwatch
cd /etc/logwatch/conf/
cp /usr/share/logwatch/default.conf/logwatch.conf .
sed -i -e 's/MailTo = root/MailTo = admin@blackbrick.com/' logwatch.conf
sed -i -e 's/Detail = Low/Detail = High/' logwatch.conf
sed -i -e 's/Format = text/Format = html/' logwatch.conf
exit

Install bc

Install the bc utility used by some jj5-bin scripts:

sudo apt-get install bc

Install GNU awk

Install GNU awk used by some jj5-bin scripts:

sudo apt-get install gawk

Disable/limit swap

vim /etc/sysctl.d/local.conf
vm.swappiness=1

Or to disable entirely (apparently not optimal):

vm.swappiness=0

Setup complete: reboot

And now you're done. Just reboot the box for good measure.

sudo reboot && exit

If you like now you can setup another server.

Allocating user IDs

If you need to allocate a user id for a member, use their member number from the Members page. Member numbers start at 1000 and increment from there.

If you need to allocate a user id for a system user, such as a user that a process will run as, check for UIDs starting from 500 and use the next available one. If you create a user on one machine with a particular UID you should create the same user with the same UID on all the other machines too.

Adding a new user

Generating passwords

While setting up a new user account you will need to generate passwords for the user. To generate passwords use the Unconfusable Password Generator.

Updating members database

At the moment member information is recorded in Members. A member should have a Wiki account to start with (create one for them if they haven't already created their own), then an entry in the Members page (where they are allocated a member number), then the "User info" on their user page should be filled out (leave fields as "Not disclosed" unless you have their permission to publish their details). After the Members page has been updated you will have:

  • {username}: the Unix/Kerberos username of the new member
  • {member number}: the member's User ID
  • {group}: whether the user has 'sudo' membership or not
    • there are two groups at ProgClub: 'sudo' and 'user'
      • everyone is in 'user' (gidNumber == 500)
      • administrators are in 'sudo' (gidNumber == 27)

And that's enough information to create their account as detailed below.

Kerberos administration

To create a new user in Kerberos:

  • SSH to charity.progclub.org
  • Login
  • Run kadmin addprinc:
$ sudo kadmin -p {your username} -q "addprinc {username}"
    • Note: use sudo so that the log file can be written
    • {your username} is your ProgClub username
    • {username} is the ProgClub username of the user your are adding

If the user is an administrator, add them to /etc/krb5kdc/kadm5.acl. E.g., at the end of the file,

username@PROGCLUB.ORG *

replacing 'username' as appropriate. You need to restart the Kerberos administration server after this change:

$ sudo /etc/init.d/krb5-admin-server restart

Changing a user's password

To reset a user's password:

$ sudo kadmin -p {your username} -q "change_password -pw {new password} {username}"

Creating home directory or administrative login

  • SSH to charity.progclub.org
  • Login

If you are creating an administrative account, run:

$ sudo adduser --uid {member number} --gid 27 {username}

If you are creating a normal user account, run:

$ sudo adduser --uid {member number} --gid 500 --disabled-password {username}

(This creates the user's home directory)

LDAP administration

To create a user in LDAP:

  • Open the LDAP administration tool
  • Login: cn=admin,dc=progclub,dc=org (/home/jj5/login_ldap for password)
  • Navigate to: ou=users
  • Click 'Create new entry here'
  • Choose 'Default'
  • Container should be: ou=users,dc=progclub,dc=org
  • ObjectClass: inetOrgPerson
  • Select the RDN attribute: Username (uid)
  • Enter 'cn', this is the Common Name, i.e. the user's full name, or their username
  • Enter 'sn', this is the Surname, i.e. the user's last name, if it's disclosed, else username
  • Enter 'displayName', i.e. the user's full name, if it's disclosed, else username
  • Enter 'Email', i.e. username@progclub.org if they want an email account, otherwise the user's email address, if it's disclosed, else blank
  • Enter 'givenName', i.e. the user's first name, if it's disclosed, else blank
  • Enter 'Username', this is their Posix username, all lowercase
  • Click 'Create Object'
  • Review the entry and click 'Commit'
  • Scroll down to 'objectClass' and click 'add value'
  • Select 'posixAccount'
  • Enter the 'gidNumber': 27 for admin, 500 for user
  • Enter the 'homeDirectory': /home/{username}, e.g. /home/jj5
  • Enter the 'uidNumber': this is the member number allocated on the web-site in Members
  • Click 'Add ObjectClass and Attributes'
  • Review the entry and click 'Update Object'
  • Scroll down to 'objectClass' and click 'add value'
  • Select 'shadowAccount'
  • Click 'Add new ObjectClass'
  • Click 'Add new attribute'
  • Select 'loginShell'
  • Enter "/bin/bash"
  • Click 'Update Object'
  • Review changes and click 'Update Object'
  • Navigate to ou=groups
  • Select 'user'
  • Select the "Generic: Posix Group" template
  • Select the new user (all users should be selected)
  • Click 'Update Object'
  • Review changes and click 'Update Object'

If the user is an administrator:

  • Navigate to ou=groups
  • Select 'sudo'
  • Select the "Generic: Posix Group" template
  • Select the new user (all administrative users should be selected)
  • Click 'Update Object'
  • Review changes and click 'Update Object'

Email account

Login to email administration. The username is postmaster@progclub.org, and the password is in /home/jj5/login_mail.

  • Click Virtual List => Add Mailbox
    • Username: the member's username
    • @progclub.org (it will default to progclub.co, which is wrong, so be careful with this)
    • Password: make a random one up and email it to the new member, they can change it themselves
    • Name: if the member gave you permission to use their name, then their name. Otherwise their username.
    • Active: checked
    • Send welcome email: checked
  • Click 'Add mailbox'.

Once you've done that, login to the user email administration facility with the member's username and the password you generated.

  • Click 'Change your forward'.
    • To: the member's non-ProgClub email address
    • Select: Deliver to the local mailbox
  • Click 'Edit alias'.

Subscribe member to ProgClub announcement

Login to announcement administration. The password is in /home/jj5/login_list.

  • Click 'Membership Management...'
  • Click 'Mass Subscription'
  • Subscribe these users now or invite them?: Subscribe
  • Send welcome messages to new subscribees?: No
  • Send notifications of new subscriptions to the list owner?: Yes
  • Enter the ProgClub email address of the new member
  • Click 'Submit Your Changes'

Tell the member about their new account

Now send an email to the member at their new email address (username@progclub.org) letting them know the details of their new account. You should include:

Subscribe member to mailing lists

You should already have subscribed them to the announcement list. You might also like to subscribe their new ProgClub email address to any or all of the following: