IncidentLog - Revision history

Xlefay: /* 00:34 GMT */

2014-03-21T05:01:51Z

00:34 GMT

← Older revision		Revision as of 05:01, 21 March 2014
Line 23:		Line 23:
	I would like to thank both FunPika and MechanicJay for helping out.		I would like to thank both FunPika and MechanicJay for helping out.

	~Incident Commander Xlefay		- Incident Commander Xlefay

	==March 13,2014==		==March 13,2014==

Xlefay: /* 00:34 GMT */

2014-03-21T05:00:58Z

00:34 GMT

← Older revision		Revision as of 05:00, 21 March 2014
Line 15:		Line 15:
	I'm glad to announce it's resolved, for now. We've also came up with some ideas MechanicJay & I going to look into to prevent this from happening in the future:		I'm glad to announce it's resolved, for now. We've also came up with some ideas MechanicJay & I going to look into to prevent this from happening in the future:

	-'''[https://en.wikipedia.org/wiki/Cgroups cgroups]''': in short, control groups that will allow us to limit resources per user, e.g. we'll have the wiki and all other things running under their own user and it won't be able to eat up all the servers memory or spawn more processes then we define.		- '''[https://en.wikipedia.org/wiki/Cgroups cgroups]''': in short, control groups that will allow us to limit resources per user, e.g. we'll have the wiki and all other things running under their own user and it won't be able to eat up all the servers memory or spawn more processes then we define.

	- httpd/mod_ruid2: this will force the webserver to start processes under a predefined user. For instance, all wiki things would be started under the "wiki" user, now combine that with cgroups, and we've got ourselves an initial solution for what happened earlier (e.g. the entire server and all services suffering because of one Wiki instance).		- httpd/mod_ruid2: this will force the webserver to start processes under a predefined user. For instance, all wiki things would be started under the "wiki" user, now combine that with cgroups, and we've got ourselves an initial solution for what happened earlier (e.g. the entire server and all services suffering because of one Wiki instance).

Xlefay: /* 24:34 GMT */

2014-03-21T05:00:30Z

24:34 GMT

← Older revision		Revision as of 05:00, 21 March 2014
Line 3:		Line 3:

	==March 20,2014==		==March 20,2014==
	=== 24:34 GMT ===		=== 00:34 GMT ===

	~~The service machine had a headache (Not to worry, we fed it aspirin.)~~		It seems our Wiki made a bit of a mess, it started a PHP script at every page load, a script that idled and lurked around and caused the server's memory to fill up after a while and as a result, the MySQL server got killed, causing the Wiki and Mail server to stop functioning properly and caused the Spamassassin to be killed as well.

	It seems our Wiki made a bit of a mess, it started a PHP script at every page load, a script that idled and lurked around and caused the server's memory to fill up after a while and as a result, the MySQL server got killed, causing the Wiki and Mail server to stop functioning properly and caused the Spamassassin ~~(our spam killer, as the name may have given away)~~ to be killed as well. ~~(the kernel definitely wasn't messing around, best not to feed it too much!)~~

	Note: due to the MySQL server going down, most mail functions, incl. the webmail didn't work anymore.		Note: due to the MySQL server going down, most mail functions, incl. the webmail didn't work anymore.

	When we got alerted (by mrcoolbp) that the wiki wasn't working, I started looking in to why. It soon became apparent there were 251 PHP processes active without a good reason, after killing those, the servers memory usage went down to 250 - 400 (can't remember precisely) but, as people loaded the wiki after restarting the database, PHP processes got started again, so FunPika & I started to investigate, he later found there was a bug in MediaWiki ~~(the software, we use for the wiki, you guessed it already, didn't you?!)~~.		When we got alerted (by mrcoolbp) that the wiki wasn't working, I started looking in to why. It soon became apparent there were 251 PHP processes active without a good reason, after killing those, the servers memory usage went down to 250 - 400 (can't remember precisely) but, as people loaded the wiki after restarting the database, PHP processes got started again, so FunPika & I started to investigate, he later found there was a bug in MediaWiki.

	After changing a configuration setting in the Wiki and killing all the remaining PHP processes, the problem went away. Naturally, the issue still exists, it's just not a problem at this moment.		After changing a configuration setting in the Wiki and killing all the remaining PHP processes, the problem went away. Naturally, the issue still exists, it's just not a problem at this moment.

Mrcoolbp at 04:55, 21 March 2014

2014-03-21T04:55:44Z

← Older revision		Revision as of 04:55, 21 March 2014
Line 3:		Line 3:

	==March 20,2014==		==March 20,2014==
	=== 20:~~30 EST~~ ===		=== 24:34 GMT ===

	The service machine had a headache (Not to worry, we fed it aspirin.)		The service machine had a headache (Not to worry, we fed it aspirin.)
Line 17:		Line 17:
	I'm glad to announce it's resolved, for now. We've also came up with some ideas MechanicJay & I going to look into to prevent this from happening in the future:		I'm glad to announce it's resolved, for now. We've also came up with some ideas MechanicJay & I going to look into to prevent this from happening in the future:

	- cgroups[1]: in short, control groups that will allow us to limit resources per user, e.g. we'll have the wiki and all other things running under their own user and it won't be able to eat up all the servers memory or spawn more processes then we define.		-'''[https://en.wikipedia.org/wiki/Cgroups cgroups]''': in short, control groups that will allow us to limit resources per user, e.g. we'll have the wiki and all other things running under their own user and it won't be able to eat up all the servers memory or spawn more processes then we define.

	- httpd/mod_ruid2: this will force the webserver to start processes under a predefined user. For instance, all wiki things would be started under the "wiki" user, now combine that with cgroups, and we've got ourselves an initial solution for what happened earlier (e.g. the entire server and all services suffering because of one Wiki instance).		- httpd/mod_ruid2: this will force the webserver to start processes under a predefined user. For instance, all wiki things would be started under the "wiki" user, now combine that with cgroups, and we've got ourselves an initial solution for what happened earlier (e.g. the entire server and all services suffering because of one Wiki instance).

Mrcoolbp at 04:48, 21 March 2014

2014-03-21T04:48:34Z

← Older revision		Revision as of 04:48, 21 March 2014
Line 1:		Line 1:
	This is a log of incidents that occurred and how they were remedied.		This is a log of incidents that occurred and how they were remedied.


			==March 20,2014==
			=== 20:30 EST ===

			The service machine had a headache (Not to worry, we fed it aspirin.)

			It seems our Wiki made a bit of a mess, it started a PHP script at every page load, a script that idled and lurked around and caused the server's memory to fill up after a while and as a result, the MySQL server got killed, causing the Wiki and Mail server to stop functioning properly and caused the Spamassassin (our spam killer, as the name may have given away) to be killed as well. (the kernel definitely wasn't messing around, best not to feed it too much!)

			Note: due to the MySQL server going down, most mail functions, incl. the webmail didn't work anymore.

			When we got alerted (by mrcoolbp) that the wiki wasn't working, I started looking in to why. It soon became apparent there were 251 PHP processes active without a good reason, after killing those, the servers memory usage went down to 250 - 400 (can't remember precisely) but, as people loaded the wiki after restarting the database, PHP processes got started again, so FunPika & I started to investigate, he later found there was a bug in MediaWiki (the software, we use for the wiki, you guessed it already, didn't you?!).

			After changing a configuration setting in the Wiki and killing all the remaining PHP processes, the problem went away. Naturally, the issue still exists, it's just not a problem at this moment.

			I'm glad to announce it's resolved, for now. We've also came up with some ideas MechanicJay & I going to look into to prevent this from happening in the future:

			- cgroups[1]: in short, control groups that will allow us to limit resources per user, e.g. we'll have the wiki and all other things running under their own user and it won't be able to eat up all the servers memory or spawn more processes then we define.

			- httpd/mod_ruid2: this will force the webserver to start processes under a predefined user. For instance, all wiki things would be started under the "wiki" user, now combine that with cgroups, and we've got ourselves an initial solution for what happened earlier (e.g. the entire server and all services suffering because of one Wiki instance).

			It should be noted that as of now, the solution isn't implemented yet, currently, the only thing that's changed is the Wiki settings preventing it from starting processes.

			I would like to thank both FunPika and MechanicJay for helping out.

			~Incident Commander Xlefay

	==March 13,2014==		==March 13,2014==

Mrcoolbp: /* ==March 16,2014 */

2014-03-17T04:58:17Z

==March 16,2014

← Older revision		Revision as of 04:58, 17 March 2014
Line 95:		Line 95:


	== ==March 16,2014==		==March 16,2014==
	=== ~ 20:30 EST ===		=== ~ 20:30 EST ===
			[[User:Mrcoolbp\|mrcoolbp]] ([[User talk:Mrcoolbp\|talk]]) 04:58, 17 March 2014 (UTC)
	Today the story queue ran dry and no stories were posted for a few hours. The situation was remedied by paulej72 who stepped in and edited some stories, and was later joined by others.		Today the story queue ran dry and no stories were posted for a few hours. The situation was remedied by paulej72 who stepped in and edited some stories, and was later joined by others.

Mrcoolbp: /* Lack of Commander Incidents */

2014-03-17T04:55:18Z

Lack of Commander Incidents

← Older revision		Revision as of 04:55, 17 March 2014
Line 95:		Line 95:


	== ~~Lack of Commander Incidents~~ ==		== ==March 16,2014==
			=== ~ 20:30 EST ===

			Today the story queue ran dry and no stories were posted for a few hours. The situation was remedied by paulej72 who stepped in and edited some stories, and was later joined by others.

			No commander was presiding at the time when the incident occurred.

			Below is a excerpt from IRC:

	<pre>Sat 2014-03-16:		<pre>Sat 2014-03-16:

FatPhil at 01:52, 17 March 2014

2014-03-17T01:52:26Z

@@ Line 111: / Line 111: @@
 :23 <@FatPhil> .deop
 :43 -!- You have been marked as being away
 </pre>
 [[Category:Development]]

FatPhil: Not having someone in command is a risk whilst we're not ticking over without worry

2014-03-16T22:44:55Z

Not having someone in command is a risk whilst we're not ticking over without worry

← Older revision		Revision as of 22:44, 16 March 2014
Line 93:		Line 93:

	[[User:Mrcoolbp\|mrcoolbp]] ([[User talk:Mrcoolbp\|talk]]) 23:24, 11 March 2014 (UTC)		[[User:Mrcoolbp\|mrcoolbp]] ([[User talk:Mrcoolbp\|talk]]) 23:24, 11 March 2014 (UTC)


			== Lack of Commander Incidents ==

			<pre>Sat 2014-03-16:
			17:59 <@janrinok> Sorry guys, but I've got to go look after my other half.
			17:59 <@janrinok> .deop
			17:59 -!- mode/#staff [-o janrinok] by what-if-i
			18:04 -!- kobach [~nope@Soylent/Staff/IRC/kobach] has left #staff [nope]
			18:05 -!- janrinok [~janrinok@Soylent/Staff/Editor/janrinok] has quit [Quit: leaving]
			18:41 <+FatPhil> dammit
			18:41 <+FatPhil> .op
			</pre>

			<pre>Sun 2014-03-17:
			00:18 <@FatPhil> "21:52 * FatPhil has itchy feet" That was 2 and a half hours ago - I'm cracking open a beer now
			00:23 <@FatPhil> .deop
			00:43 -!- You have been marked as being away
			</pre>

	[[Category:Development]]		[[Category:Development]]

FunPika: added Category:Development using HotCat

2014-03-15T15:33:52Z

added Category:Development using HotCat

← Older revision		Revision as of 15:33, 15 March 2014
Line 93:		Line 93:

	[[User:Mrcoolbp\|mrcoolbp]] ([[User talk:Mrcoolbp\|talk]]) 23:24, 11 March 2014 (UTC)		[[User:Mrcoolbp\|mrcoolbp]] ([[User talk:Mrcoolbp\|talk]]) 23:24, 11 March 2014 (UTC)

			[[Category:Development]]