EmergencyProcedures

SystemAdministration - parent

What you need to do anything useful

You need ssh access to the machines affected, with the ability to get root privileges. Without this, there is no way to do anything useful to fix any problem with the system

You must ssh to the appropriate machine, typically to a user account, and then sudo su - to root.

(This ability must be obtained in advance, such information cannot for obvious reasons be placed on a public wiki)

Slash basic description

The slash system consists of three separate componants:

• A Mysql database server - The database engine which holds most of the data such as articles, users, etc.
• The Apache web server - which handles the web interface
• The slashd daemon, which acts as a timed batch processor for various slash events and tasks.

In addition on the main server, an http caching program called varnish is running, which also must be functional, and running.

• Varnish - an http cache.

For slash to work, all three (or four) componants must be running.

To tell if they are running, use the command 'pstree' which will show, in condensed form, the tasks that are running.

Here is what the result of the pstree command looks like on the slashcott server:

[root@slashcode init.d]# pstree
init─┬─auditd───{auditd}
     ├─httpd───10*[httpd]
     ├─master─┬─pickup
     │        └─qmgr
     ├─6*[mingetty]
     ├─mysqld_safe───mysqld───37*[{mysqld}]
     ├─ntpd
     ├─rsyslogd───3*[{rsyslogd}]
     ├─sshd─┬─3*[sshd───sshd───bash───sudo───su───bash]
     │      └─sshd───sshd───bash───sudo───su───bash───pstree
     ├─su───slashd───2*[slashd]
     └─udevd───2*[udevd]
[root@slashcode init.d]# 

If you get 'garbage' as the output of the pstree command, use this form instead:

[root@slashcode init.d]# pstree -A

We can see that apache is running, it shows up as httpd:

     ├─httpd───10*[httpd]

We can see that mysql is running, it shows up like this:

     ├─mysqld_safe───mysqld───37*[{mysqld}]

And we can see that slashd is running:

     |-su---slashd---2*[slashd]

This one is ever so slightly easy to miss, because it is running as a child of the su command, so is not first on the list.

On the main server, Varnish must also be running:

|─varnishd───varnishd───20*[{varnishd}]

The exact numbers (10*[httpd] etc) are not important.

This is what a properly running system should look like, on slashcott. On the main site, the database is on one machine, which must have mysql running, and and the other machine must have apache (httpd) running, varnish running, and slashd running.

Important note - at the present time, you cannot fix either system by a reboot. Apache MUST be started by hand

Primary site down

Log in to main slash machine, obtain root.

Check process list:

root@soylent-www:/etc/varnish# pstree -A

Make sure that slashd, apache, and varnish are running, as described above.