Reply
Thread Tools
Posts: 35 | Thanked: 504 times | Joined on Jan 2013 @ Germany
#1
Here's the promised complete post mortem of everything that happened.

What was planned:
Do a backup of our VMs and a software upgrade to all machines on a Saturday. I estimated about 2 hours for that, xes corrected me into 6.

Turns out it went into 10 days of frantic firefighting colliding with day jobs, family and giving talks at conferences.

What happend:

Saturday, 19.11.2016
10:00 - start updates and backups on blade-a
14:30 - backups and updates complete on blade-a, reboot confirmed successful
14:31 - uptime induced filesystem check after 1347 days
15:00 - start of backups on blade-b
17:12 - filesystem check complete, blade-a up and running
17:30 - first systems on blade-a confirmed up and working
18:30 - software upgrade on stage and mail complete
20:15 - backups of blade-b finished and copied onto blade-a backup space
20:16 - start of updates on blade-b
21:00 - updates on blade-b complete, reboot
21:01 - blade-b stuck in boot with corrupt bios image in flash
23:30 - all available remote recovery options tried, none working
23:40 - decision to go for Plan B, boot talk.maemo.org on blade-a, redirect everything else to talk.m.o
23:45 - blade-b turned off through IPMI
23:53 - talk.m.o available again

Monday, 21.11.2016
16:00 - Datacenter visit, trying to boot blade-b with attached USB key for BIOS recovery
18:00 - No recovery possible, Board hangs with "A9" after attaching USB devices, decision to swap board. Unable to swap board directly because Hardware has to be powered off and removed from the rack, there are APC powerstrips in the way.

Tuesday, 22.11.2016
22:30 - Starting backup of stage.m.o

Wednesday, 23.11.2016
21:30 - stage.m.o available again

Saturday, 26.11.2016
16:00 - Again, Datacenter visit, Security Guard doesn't show up until 16:45
16:45 - Swapping CPU and Memory to a spare board in the chassis
17:30 - Powering up spare board, removing udev.d rules, thereby accidentally swapping network interfaces of blade-b
19:30 - Upgrade of both blades to latest kernel and XEN version with security patches
21:40 - Correction of interface udev rules to match interface names to physical interfaces
22:20 - Every thing considered to be fully operational, applying finishing touches

Sunday, 27.11.2016
00:20 - maemo.org infrastructure declared operational

Monday, 28.11.2016
05:22 - blade-b kernel: NETDEV WATCHDOG: eth1 (igb): transmit queue 3 timed out
Network interfaces of blade-b started to reset due to bogon emissions
Affected systems: www, wiki, garage, builder, vcs
19:20 - reload of igb kernel modules fixed the condition, everything working again.

Tuesday, 29.11.2016
05:22 - blade-b has the same error as on monday
15:20 - fixed by reload of igb kernel module
15:40 - reboot of blade-b, disabled APSM

Total time spent staring at screens and in the colocation: 60+ hours

Time without serious hiccups: 1234 days (which was the uptime of both blade-a and blade-b)

So if you want to thank your admins, here are our wishlists:

xes - (prefers giftcards from amazon.it to xes.maemo (at) gmail.com) https://www.amazon.it/dp/B005VG4G3U
falk - https://www.amazon.de/gp/registry/wishlist/168D3W6163KG

Best,

xes & falk
 

The Following 48 Users Say Thank You to fstern For This Useful Post:
peterleinchen's Avatar
Posts: 4,117 | Thanked: 8,901 times | Joined on Aug 2010 @ Ruhrgebiet, Germany
#2
Thanks.

Do you have any special order of wishes?
Or is there a possibility to pay only part of a high-prized item so few ants can make an elefant?

Furthermore I propose that council/board should think about spending some funds to those guys.
__________________
SIM-Switcher, automated SIM switching with a Double (Dual) SIM adapter
--
Thank you all for voting me into the Community Council 2014-2016!

Please consider your membership / supporting Maemo e.V. and help to spread this by following/copying this link to your TMO signature:
[MC eV] Maemo Community eV membership application, http://talk.maemo.org/showthread.php?t=94257

editsignature, http://talk.maemo.org/profile.php?do=editsignature
 

The Following 15 Users Say Thank You to peterleinchen For This Useful Post:
Posts: 1,288 | Thanked: 4,316 times | Joined on Oct 2014
#3
Thanks!
This is a great example of what you guys are doing behind-the-scene.
Hope your Xmas wishes comes true
 

The Following 4 Users Say Thank You to nieldk For This Useful Post:
Fellfrosch's Avatar
Posts: 1,092 | Thanked: 4,995 times | Joined on Dec 2009 @ beautiful cave
#4
Originally Posted by peterleinchen View Post
Thanks.
Furthermore I propose that council/board should think about spending some funds to those guys.
Yes that would be a nice idea. I don't like amazon and have deleted my account there some time ago. I'm not willing to reopen it for you two guys. But of course i'm grateful for your efforts, and would make a separate donation for you, if I can just use the bank account of maemo e.V..

Thanx guys!
 

The Following 4 Users Say Thank You to Fellfrosch For This Useful Post:
Posts: 35 | Thanked: 504 times | Joined on Jan 2013 @ Germany
#5
Originally Posted by peterleinchen View Post
Thanks.

Do you have any special order of wishes?
Or is there a possibility to pay only part of a high-prized item so few ants can make an elefant?
I'd be very happy about the bamboo tablet. Sadly there is no ants way, but this is only a wishlist

Best,

Falk
__________________
--
We reject kings, presidents and voting.
We believe in rough consensus and running code.
- David Clark
 

The Following 4 Users Say Thank You to fstern For This Useful Post:
peterleinchen's Avatar
Posts: 4,117 | Thanked: 8,901 times | Joined on Aug 2010 @ Ruhrgebiet, Germany
#6
BUMP! bump.
__________________
SIM-Switcher, automated SIM switching with a Double (Dual) SIM adapter
--
Thank you all for voting me into the Community Council 2014-2016!

Please consider your membership / supporting Maemo e.V. and help to spread this by following/copying this link to your TMO signature:
[MC eV] Maemo Community eV membership application, http://talk.maemo.org/showthread.php?t=94257

editsignature, http://talk.maemo.org/profile.php?do=editsignature
 

The Following 4 Users Say Thank You to peterleinchen For This Useful Post:
Posts: 1,808 | Thanked: 4,272 times | Joined on Feb 2011 @ Germany
#7
Originally Posted by fstern View Post
xes - (prefers giftcards from amazon.it to xes.maemo (at) gmail.com) https://www.amazon.it/dp/B005VG4G3U
Done

Originally Posted by fstern View Post
falk - https://www.amazon.de/gp/registry/wishlist/168D3W6163KG
Done
Edit: didn't see that about the tablet, and I picked a safe (cheap) choice.
Will consider top-up, but no promises here.

Thanks a lot guys.
 

The Following 8 Users Say Thank You to reinob For This Useful Post:
peterleinchen's Avatar
Posts: 4,117 | Thanked: 8,901 times | Joined on Aug 2010 @ Ruhrgebiet, Germany
#8
@reinob
Thanks, generous as ever!

What I do not get is the lack of supporters here in this thread. For some devs we easily crossed the 1k border (and never heard again). So I do not get why people reluct to say thanks with a little gift.
Those guys are working since years behind the scenes in the background but without their support we would have only 404.
Of course they are all doing it on a voluntary base but this example shows how much effort/time they put into it!


@all for those who cannot understand italian, you can use your own amazon site (just follow gift card / via e-mail) please follow below link (please do not use own amazon site as this will clutter the gifted money to different domains)
https://www.amazon.de/gp/switch-lang...&language=enGB
on this site you may choose appropriate language

@fstern: would make sense you create/post an e-mail address, too?
You can enter any value into the gift card ...


--
I would like to thank here also our long-time, unfortunately not anymore, repo maintainer merlin1991 and the CSSU/FPTF devs. ( as well as Maemo e.V. board )
__________________
SIM-Switcher, automated SIM switching with a Double (Dual) SIM adapter
--
Thank you all for voting me into the Community Council 2014-2016!

Please consider your membership / supporting Maemo e.V. and help to spread this by following/copying this link to your TMO signature:
[MC eV] Maemo Community eV membership application, http://talk.maemo.org/showthread.php?t=94257

editsignature, http://talk.maemo.org/profile.php?do=editsignature

Last edited by peterleinchen; 2016-12-05 at 13:08.
 

The Following 7 Users Say Thank You to peterleinchen For This Useful Post:
Fellfrosch's Avatar
Posts: 1,092 | Thanked: 4,995 times | Joined on Dec 2009 @ beautiful cave
#9
@reinob
You are in the Community council and the question still stands: Any way to use the Maemo e.V. account for saying thank you?
 

The Following 4 Users Say Thank You to Fellfrosch For This Useful Post:
Community Council | Posts: 4,920 | Thanked: 12,867 times | Joined on May 2012 @ Southerrn Finland
#10
Originally Posted by Fellfrosch View Post
@reinob
You are in the Community council and the question still stands: Any way to use the Maemo e.V. account for saying thank you?
It is a bit troublesome for tax reasons to do it directly. You can donate to Maemo Community and give directions how you would like to have your donation used.
I urge you however to make donations directly to @xes & @warfare.
 

The Following 5 Users Say Thank You to juiceme For This Useful Post:
Reply

Thread Tools

 
Forum Jump


All times are GMT. The time now is 13:24.