EmergencyProcedures - Revision history

The Mighty Buzzard: /* If memcached is not running */

2019-01-10T16:11:55Z

If memcached is not running

← Older revision		Revision as of 16:11, 10 January 2019
Line 163:		Line 163:

	=== If memcached is not running ===		=== If memcached is not running ===
			=== memcached is not currently used. ignore this section ===

	Restart it:		Restart it:
Line 170:		Line 171:

	Confirm it is running after a few seconds by looking at the process list again. (pstree)		Confirm it is running after a few seconds by looking at the process list again. (pstree)


	=== If the database server is not running ===		=== If the database server is not running ===

The Mighty Buzzard: We're not currently using memcached

2016-05-27T15:54:40Z

We're not currently using memcached

← Older revision		Revision as of 15:54, 27 May 2016
Line 23:		Line 23:
	* The slashd daemon, which acts as a timed batch processor for various slash events and tasks.		* The slashd daemon, which acts as a timed batch processor for various slash events and tasks.

	In addition on the main server, an http caching program called varnish is running, which also must be functional, and running, and a cache program for the database, memcached.		In addition on the main server, an http caching program called varnish is running, which also must be functional, and running, and a cache program for the database, <strike>memcached</strike>.

	* Varnish - an http cache.		* Varnish - an http cache.
	* memcached - A database cache		* memcached - A database cache

	For slash to work, all three components must be running, plus varnish and memcached if they are in use.		For slash to work, all three components must be running, plus varnish <strike>and memcached</strike> if they are in use.

	To tell if they are running, use the command 'pstree' which will show, in condensed form, the tasks that are running.		To tell if they are running, use the command 'pstree' which will show, in condensed form, the tasks that are running.
Line 85:		Line 85:
	</pre>		</pre>

	As well as memcached:		<strike>As well as memcached:
	<pre>		<pre>
	├─memcached───5*[{memcached}]		├─memcached───5*[{memcached}]
	</pre>		</pre></strike>

	On the database machine for the main site, we would see that mysql is running,		On the database machine for the main site, we would see that mysql is running,
Line 99:		Line 99:
	The exact numbers (10*[httpd] etc) are not important.		The exact numbers (10*[httpd] etc) are not important.

	This is what a properly running system should look like, on the main site. The database is on one machine, which must have mysql running, and and the other machine must have apache (httpd) running, varnish running, memcached running, and slashd running.		This is what a properly running system should look like, on the main site. The database is on one machine, which must have mysql running, and and the other machine must have apache (httpd) running, varnish running, <strike>memcached running</strike>, and slashd running.

	'''Important note - at the present time, you cannot fix either system by a reboot. Apache MUST be started by hand'''		'''Important note - at the present time, you cannot fix either system by a reboot. Apache MUST be started by hand'''

137.69.117.58: corrected mis-spellings

2014-04-01T13:01:46Z

corrected mis-spellings

← Older revision		Revision as of 13:01, 1 April 2014
Line 17:		Line 17:
	==Slash basic description==		==Slash basic description==

	The slash system consists of three separate ~~componants~~:		The slash system consists of three separate components:

	* A Mysql database server - The database engine which holds most of the data such as articles, users, etc.		* A Mysql database server - The database engine which holds most of the data such as articles, users, etc.
Line 28:		Line 28:
	* memcached - A database cache		* memcached - A database cache

	For slash to work, all three ~~componants~~ must be running, plus varnish and memcached if they are in use.		For slash to work, all three components must be running, plus varnish and memcached if they are in use.

	To tell if they are running, use the command 'pstree' which will show, in condensed form, the tasks that are running.		To tell if they are running, use the command 'pstree' which will show, in condensed form, the tasks that are running.
Line 143:		Line 143:
	Confirm it is running after a few seconds by looking at the process list again. (pstree)		Confirm it is running after a few seconds by looking at the process list again. (pstree)

	If you get an error message about apache being unable to ~~conect~~ to the ~~dastabase~~, you will need to ssh into the database machine and restart the database, then come back and restart apache. '''Apache cannot start unless it can make a database connection.'''		If you get an error message about apache being unable to connect to the database, you will need to ssh into the database machine and restart the database, then come back and restart apache. '''Apache cannot start unless it can make a database connection.'''

	=== If only slashd is not running ===		=== If only slashd is not running ===

Martyb: child of the "sudo" command, not "su"

2014-03-24T11:43:46Z

child of the "sudo" command, not "su"

← Older revision		Revision as of 11:43, 24 March 2014
Line 78:		Line 78:
	</pre>		</pre>

	This one is ever so slightly easy to miss, because it is running as a child of the su command, so is not first on the list.		This one is ever so slightly easy to miss, because it is running as a child of the sudo command, so is not first on the list.

	On the main server, Varnish must also be running:		On the main server, Varnish must also be running:

Martyb: Fixed detail line for slashd running; old entry was not in pstree output

2014-03-24T11:42:28Z

Fixed detail line for slashd running; old entry was not in pstree output

← Older revision		Revision as of 11:42, 24 March 2014
Line 75:		Line 75:
	And we can see that slashd is running:		And we can see that slashd is running:
	<pre>		<pre>
	\|-su---slashd---2*[slashd]		\|-sudo---slashd---3*[slashd]
	</pre>		</pre>

AudioGuy at 00:48, 20 March 2014

2014-03-20T00:48:06Z

← Older revision		Revision as of 00:48, 20 March 2014
Line 234:		Line 234:
	# /etc/rc.d/init.d/slash stop		# /etc/rc.d/init.d/slash stop
	# /usr/local/apache/bin/apachectl stop		# /usr/local/apache/bin/apachectl stop
	~~# /etc/rc.d/init.d/varnish stop~~
	# /etc/rc.d/init.d/mysqld stop		# /etc/rc.d/init.d/mysqld stop
	</pre>		</pre>
Line 241:		Line 240:
	<pre>		<pre>
	# /etc/rc.d/init.d/mysqld start		# /etc/rc.d/init.d/mysqld start
	~~# /etc/rc.d/init.d/varnish start~~
	# /usr/local/apache/bin/apachectl start		# /usr/local/apache/bin/apachectl start
	# /etc/rc.d/init.d/slash start		# /etc/rc.d/init.d/slash start

AudioGuy at 22:11, 18 March 2014

2014-03-18T22:11:35Z

← Older revision		Revision as of 22:11, 18 March 2014
Line 23:		Line 23:
	* The slashd daemon, which acts as a timed batch processor for various slash events and tasks.		* The slashd daemon, which acts as a timed batch processor for various slash events and tasks.

	In addition on the main server, an http caching program called varnish is running, which also must be functional, and running.		In addition on the main server, an http caching program called varnish is running, which also must be functional, and running, and a cache program for the database, memcached.

	* Varnish - an http cache.		* Varnish - an http cache.
			* memcached - A database cache

	For slash to work, all three ~~(or four)~~ componants must be running.		For slash to work, all three componants must be running, plus varnish and memcached if they are in use.

	To tell if they are running, use the command 'pstree' which will show, in condensed form, the tasks that are running.		To tell if they are running, use the command 'pstree' which will show, in condensed form, the tasks that are running.

	Here is what the result of the pstree command looks like on the ~~slashcott~~ server:		Here is what the result of the pstree command looks like on the main server:

	<pre>		<pre>
	~~[root@slashcode init.d]~~# pstree		# pstree
	~~init─┬─auditd───{auditd}~~		init─┬─atd
	~~├─httpd───10~~*[httpd]		├─cron
			├─dbus-daemon
			├─httpd─┬─10*[httpd]
			│ └─sh
			├─landscape-clien─┬─landscape-broke───2*[{landscape-broke}]
			│ ├─landscape-manag
			│ └─landscape-monit
			├─linode-longview
			├─login───bash
	├─master─┬─pickup		├─master─┬─pickup
	│ ~~└─qmgr~~		│ ├─qmgr
	~~├─6~~*[~~mingetty~~]		│ └─tlsmgr
	~~├─mysqld_safe───mysqld───37~~*[~~{mysqld}~~]		├─memcached───5*[{memcached}]
			├─nginx───4*[nginx]
	├─ntpd		├─ntpd
	├─rsyslogd───3*[{rsyslogd}]		├─rsyslogd───3*[{rsyslogd}]
	~~├─sshd─┬─3~~*[~~sshd───sshd───bash───sudo───su───bash~~]		├─slapd───4*[{slapd}]
	~~│ └─sshd───sshd───bash───sudo───su───bash───pstree~~		├─sshd───sshd───sshd───bash───sudo───su───bash───pstree
	~~├─su───slashd───2~~*[slashd]		├─sudo───slashd───3*[slashd]
	~~└─udevd───2~~*[udevd]		├─udevd───2*[udevd]
	[~~root@slashcode init.d~~]#		├─upstart-socket-
			├─upstart-udev-br
			└─varnishd───varnishd───20*[{varnishd}]
	</pre>		</pre>

Line 59:		Line 71:
	<pre>		<pre>
	├─httpd───10*[httpd]		├─httpd───10*[httpd]
	~~</pre>~~

	~~We can see that mysql is running, it shows up like this:~~
	~~<pre>~~
	~~├─mysqld_safe───mysqld───37*[{mysqld}]~~
	</pre>		</pre>

Line 77:		Line 84:
	\|─varnishd───varnishd───20*[{varnishd}]		\|─varnishd───varnishd───20*[{varnishd}]
	</pre>		</pre>

			As well as memcached:
			<pre>
			├─memcached───5*[{memcached}]
			</pre>

			On the database machine for the main site, we would see that mysql is running,
			it shows up like this:
			<pre>
			├─mysqld_safe───mysqld───37*[{mysqld}]
			</pre>


	The exact numbers (10*[httpd] etc) are not important.		The exact numbers (10*[httpd] etc) are not important.

	This is what a properly running system should look like, on ~~slashcott. On~~ the main site~~, the~~ database is on one machine, which must have mysql running, and and the other machine must have apache (httpd) running, varnish running, and slashd running.		This is what a properly running system should look like, on the main site. The database is on one machine, which must have mysql running, and and the other machine must have apache (httpd) running, varnish running, memcached running, and slashd running.

	'''Important note - at the present time, you cannot fix either system by a reboot. Apache MUST be started by hand'''		'''Important note - at the present time, you cannot fix either system by a reboot. Apache MUST be started by hand'''
Line 142:		Line 161:

	You may see some errors when starting slashd, at the present time this is 'normal'. Provided it starts, and seems to operate correctly, there is little that can be done about this until some code fixes are done.		You may see some errors when starting slashd, at the present time this is 'normal'. Provided it starts, and seems to operate correctly, there is little that can be done about this until some code fixes are done.

			=== If memcached is not running ===

			Restart it:
			<pre>
			# /etc/init.d/memcached start
			</pre>

			Confirm it is running after a few seconds by looking at the process list again. (pstree)


	=== If the database server is not running ===		=== If the database server is not running ===
Line 177:		Line 206:
	# /srv/soylentnews.org/apache/bin/apachectl stop		# /srv/soylentnews.org/apache/bin/apachectl stop
	# /etc/init.d/varnish stop		# /etc/init.d/varnish stop
			# /etc/init.d/memcached stop
	# /etc/init.d/mysqld stop		# /etc/init.d/mysqld stop
	</pre>		</pre>
Line 183:		Line 213:
	<pre>		<pre>
	# /etc/init.d/mysqld start		# /etc/init.d/mysqld start
			# /etc/init.d/memcached start
	# /etc/init.d/varnish start		# /etc/init.d/varnish start
	# /srv/soylentnews.org/apache/bin/apachectl start		# /srv/soylentnews.org/apache/bin/apachectl start

AudioGuy at 21:49, 18 March 2014

2014-03-18T21:49:26Z

← Older revision		Revision as of 21:49, 18 March 2014
Line 202:		Line 202:
	<pre>		<pre>
	# /etc/rc.d/init.d/slash stop		# /etc/rc.d/init.d/slash stop
	# /~~srv~~/~~soylentnews.org~~/apache/bin/apachectl stop		# /usr/local/apache/bin/apachectl stop
	# /etc/rc.d/init.d/varnish stop		# /etc/rc.d/init.d/varnish stop
	# /etc/rc.d/init.d/mysqld stop		# /etc/rc.d/init.d/mysqld stop
Line 211:		Line 211:
	# /etc/rc.d/init.d/mysqld start		# /etc/rc.d/init.d/mysqld start
	# /etc/rc.d/init.d/varnish start		# /etc/rc.d/init.d/varnish start
	# /~~srv~~/~~soylentnews.org~~/apache/bin/apachectl start		# /usr/local/apache/bin/apachectl start
	# /etc/rc.d/init.d/slash start		# /etc/rc.d/init.d/slash start
	</pre>		</pre>

AudioGuy at 21:46, 18 March 2014

2014-03-18T21:46:06Z

← Older revision		Revision as of 21:46, 18 March 2014
Line 5:		Line 5:
	In ALL emergency situations, the most qualified person to fix them should be sought out. These instructions are here to cover the hopefully VERY RARE case where that is not possible.		In ALL emergency situations, the most qualified person to fix them should be sought out. These instructions are here to cover the hopefully VERY RARE case where that is not possible.

	They are deliberately written at a very basic level, probably more basic than anyone with access to the servers would actually need. Howver, it is ~~till~~ useful to have a sort of checklist. The times when services are down tend to be stressful, and humans sometimes do not operate at their best when stressed. Plus, who knows? Maybe someday someone will have to walk the janitor through this...		They are deliberately written at a very basic level, probably more basic than anyone with access to the servers would actually need. Howver, it is still useful to have a sort of checklist. The times when services are down tend to be stressful, and humans sometimes do not operate at their best when stressed. Plus, who knows? Maybe someday someone will have to walk the janitor through this...

	==What you need to do anything useful==		==What you need to do anything useful==

AudioGuy at 21:45, 18 March 2014

2014-03-18T21:45:22Z

← Older revision		Revision as of 21:45, 18 March 2014
Line 93:		Line 93:
	</pre>		</pre>

	You should see ~~tese~~ in process list, among ~~others~~:		You should see these in process list, among other unrelated processes:
	<pre>		<pre>
	├─httpd───10*[httpd]		├─httpd───10*[httpd]
Line 153:		Line 153:

	Confirm it is running after a few seconds by looking at the process list again. (pstree)		Confirm it is running after a few seconds by looking at the process list again. (pstree)
			<pre>
			├─mysqld_safe───mysqld───37*[{mysqld}]
			</pre>

	If it will not start, you will need additional help.		If it will not start, you will need additional help.