SystemAdministration/AccessInstruction

SSH Requirements

We require that all users autheticate with SSH public keys, and that to prevent stupid config errors from becoming massive security breachs, require that users have to pass through a central point to successfully autheticate and access our cluster. As of writing 'boron', public name of 'staff.soylentnews.org' is the primary access usage; all members of staff can access this (access permissions set for firefighter). SSH keys are stored in LDAP, and currently require a sysop to change, edit, or enroll new keys. If you need an account, ask any sysop to create one.

If you feel you need access to a specific machine, you should discuss the matter with your teamleader. If they second your request, they can e-mail the staff mailing list and request access for you then it's only a matter of the sysops team to approve or reject the request, we try and practice least amount of access necessary in an attempt to keep things relatively secure.

With that out of the way, accessing the staff box is extremely straight-forward. Assuming you have a key and and an account ...

mcasadevall@tranquility:~/src/charybdis$ ssh staff.soylentnews.org
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-60-virtual x86_64)

Last login: Sun Mar 23 20:52:43 2014
mcasadevall@boron:~$ 

Of course, this isn't that useful in and of itself. To get further through the network, you need to either use Kerberos, or SSH proxing. As Kerberos is useful for other things, we'll cover that first

Kerberos

Getting Your Ticket

Kerberos acts as a single authority for authetication services on the network. Normally, Kerberos would generate a ticket on login, but that doesn't work correctly when using SSH public key authetication. When you got your user account, you should have received a kerberos password. With that in hand, autheticating is easy, just run kinit

mcasadevall@boron:~$ kinit
Password for mcasadevall@LI694-22: 
mcasadevall@boron:~$ 

Kerberos passwords can be changed with the kpasswd command.

Tickets and You

Kerberos works on the concept of tickets, which allows means you are autheticated to access Kerberos services. At the moment, SSH is the only kerberos installed service, but we can use this infrastructure for both user and host authetication (i.e., proving a host is who they say they are for accessing LDAP).

At any time, you can check what tickets you can have with the klist command. The krbtgt ticket means you're known to Kerberos, and kerberos will issue sub-tickets as long as this ticket is valid.

mcasadevall@boron:~$ klist
Ticket cache: FILE:/tmp/krb5cc_2500
Default principal: mcasadevall@LI694-22

Valid starting    Expires           Service principal
23/03/2014 22:47  24/03/2014 08:47  krbtgt/LI694-22@LI694-22
	renew until 24/03/2014 22:47

Tickets remain valid for 10 hours after being issued, and can be renewed with the kinit -r command up to 7 days. We may change this in the future to longer limits, but this is good enough for now.

SSH and Tickets

Once you have a ticket issued, you can access all machines allowed by your group. For example:

mcasadevall@helium:~$ ssh lithium

Last login: Sun Mar 23 16:50:06 2014 from boron.li694-22
mcasadevall@lithium:~$ ssh hydrogen

Last login: Sun Mar 23 17:37:44 2014 from boron.li694-22
mcasadevall@hydrogen:~$ 

Tickets follow you from one system to another via the GSSAPI used by SSH. In other words, once you've autheticated to kerberos on one system, you're autheticated across the network. If you SSH to a new machine, you'll automatically be issued a new ticket on that machine, allowing you to traverse the network at will; this allows you to do node-to-node scps and such without having to worry about setting up keys. Quick, easy and transparent.

When Kerberos Breaks

Sometimes when the krb5 configuration is changed on a node, tickets become invalid. Kerberos doesn't automatically reissue tickets, so if you get Permission Denied messages for things you know you can access, the easy solution is to delete all the tickets, and re-kinit. This is done with the kdestroy command

mcasadevall@boron:~$ kdestroy 
mcasadevall@boron:~$ klist
klist: No credentials cache found (ticket cache FILE:/tmp/krb5cc_2500)
mcasadevall@boron:~$ 

If you have any issues at all, try this before asking a sysop to look into the kerberos config.

SSH Proxying

While Kerberos is nice, its not perfect. For instance, uploading external files means uploading to boron, then punting them around the network with scp. Welcome to the wonderful world of SSH proxying.

SSH Proxying allows SSH pass-through through a single point, and can be used for NAT and such. Enabling it requires adding the following to your local .ssh/config file

Host staff.soylentnews.org
	User your_username

Host *.li694-22
	User your_username
	ProxyCommand ssh staff.soylentnews.org nc -q0  %h %p

Once this is in place, you can now do stuff like this (tranquility is my local machine in this example)

mcasadevall@tranquility:~$ ssh helium.li694-22
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.2.0-60-virtual x86_64)

Last login: Sun Mar 23 22:52:27 2014 from boron.li694-22
mcasadevall@helium:~$

SCP and SFTP also support SSH proxying, and it can be used for port forwarding from your machine to remote nodes; in effect this allows you to treat any node in the data center as though it were directly accessible.

All nodes support Kerberos services, so you can kinit from anywhere on the network, and then enjoy Kerberos access in addition to SSH proxying.

PuTTY

If you are running Windows and PuTTY, you can use SSH proxying by following the instructions in this guide. You will need to replace any references to "instance.pmpta.wmflabs" with "<machine>.li694-22" and any references to "bastion.wmflabs.org" with "staff.soylentnews.org".