ebutong.com

Wednesday, February 23, 2005

On Monday, at 10.14 pm UTC, the Wikimedia server cluster experienced a total power failure, taking down Wikipedia, Wikinews, and all other Wikimedia websites.

The servers are housed in a colocation facility (colo) in Tampa, Florida, USA. They occupy two racks, with each rack receiving electricity from two independent supplies. However, both supplies have circuit-breakers in them, and both opened at the same time, leading to a total power failure. All computers immediately went down. It’s normal for fire safety regulations to prohibit uninterruptible power supplies in colos, with the colo providing its own UPS and generator instead. The circuit breakers were on the computer side of this emergency power system, so none of the computers continued to receive power to survive the breaker trip or shut down safely.

The actual reason why the circuit breakers tripped is currently unknown.

When power was restored, it was discovered that most of the MySQL databases that store the data which makes up Wikipedia et al had been corrupted. The main database and the four slaves had all damaged the data on their hard disk drives beyond the ability of the auto-correction to repair. Only one copy survived safely, on a machine that is used for report generation and maintenance tasks, which remained 31 hours backlogged while catching up after an unusually heavy update load during the previous week.

Volunteer Wikimedia engineers worked through the night rebuilding the databases from the sole good copy onto the other servers. The Wikipedia database is over 180Gb in size, making the copying process last 1.5 – 2 hours for every server it was performed upon.

Regular back-ups of the database of Wikipedia projects are maintained – the encyclopedia in its entirety was not at risk. The last database download was made on February 9; all edits since then could only have been laboriously rebuilt from logs and recovered from the damaged database requiring much more time and effort.

Limited read-only service was established late Tuesday afternoon, with editing becoming possible 24 hours after the power failure. Final repairs continue now, as well as upgrades to prevent similar issues in the future. Server-intensive features, such as categories and ‘watchlists’ that display recent changes to selected articles to registered users, remain disabled to ease the load on the recovering systems.

The process which led to the damage originated with the operating system, disk controllers, or hard drives failing to flush the data correctly.

If the power to a database server is cut mid-write, the database may be corrupted and unreadable, however the operating system, hardware, and software are designed to make this very unlikely. In a previous incident in 2004 power was also lost to a server but the database was undamaged.

To avoid such damage, each database server saves a copy of an edit to be applied to the database on a separate storage system before making the actual update to the database itself. This so-called ‘write-ahead logging’ should ensure that in the event of a system crash, the database can be rebuilt from a ‘last-good’ state by replaying the edits saved in the log.

Earlier this year popular blogging site LiveJournal suffered a similar power failure when another customer at their colocation facility pressed an Emergency Power Off button, intended for use only by firefighters. The company suffered database corruption similar to that seen at Wikimedia.

LiveJournal are now fitting UPS to their servers to ensure that they have time to shut down safely in the event of a power failure. Wikimedia was said to be investigating the possibility of fitting similar equipment at the time of this failure.

Several pundits have suggested that the use of another database, such as the proprietary database Oracle or the free PostgreSQL, would have avoided the database corruption seen at the server cluster. A post-mortem of the incident show the failure was in the operating system, or the hardware, or some combination of the two. LiveJournal, which also uses MySQL, reported similar database corruption after their power cut.

The Wikimedia foundation only allows the use of free software on its systems, and future versions of the Mediawiki software will support the PostgreSQL database.

Users are reminded that during times of system failure or excessive demand, they can still search Wikipedia using Google. The articles may be viewed using Google’s cache.

Friday, August 24, 2007

Former National Basketball Association (NBA) player Eddie Griffin died on August 17, 2007, at age 25 due to injuries sustained in a car crash, the Harris County medical examiner’s office confirmed on Tuesday.

The former Minnesota Timberwolves forward, who was waived in March for violating the League’s substance abuse program, ignored a railroad warning, drove his SUV through a barrier, and collided with a moving train at about 1:30 a.m., according to Houston Police. His vehicle caught fire and was soon engulfed in flames.

No identification was found and the body was badly burned. For that reason, dental records were used to identify him. Griffin, who played college Basketball at Seton Hall University, played for the Houston Rockets from 2001–2003, and the Minnesota Timberwolves from 2004–2007. The five-year veteran had been battling alcoholism since leaving Seton Hall. He is survived by his three-year-old daughter Amaree.

Wednesday, March 27, 2013

The Director of the National Museum of Australia, Andrew Sayers, has resigned his position effective July 1 in a move that came as a surprise to his colleagues. Sayers cited distance issues as his wife is currently working full time in Melbourne.

Sayers is quoted in a statement as saying, “I leave the museum confident that the reputation of the Museum as the home of our national treasures is one of which we can all be proud. […] Professionally, I have enjoyed making a contribution to the Museum, yet, as many couples have discovered a ‘commuter relationship’ is not ideal.”

Sayers was contracted for five years, and was only into his third year in the post. Prior to his position at the National Museum, he spent ten years in the same role at the National Portrait Gallery of Australia. He also spent thirteen years working as as a curator and assistant director at the National Gallery of Australia. He began his museum career at Art Gallery of New South Wales and Newcastle Region Art Gallery. Following his resignation, Sayers will retire to live in Melbourne with his wife.

Saturday, February 16, 2008

There were 5 games played in the National Hockey League on February 15, 2008.

Tuesday, May 31, 2005

One thousand residents of the Defense Department-managed Armed Forces Retirement Home in Washington, D.C. filed a class-action lawsuit on May 24, asserting that the cut-backs in medical and dental services imposed by Secretary of Defense Donald Rumsfeld are illegal. The operating budget for the home was reduced from $63 million in 2004 to $58 million for 2005. The residents cite cuts in on-site X-ray, electrocardiogram, physical and dental services, and the closing of the home’s main clinic and an on-site pharmacy.

Chief Financial Officer Steve McManus responded that the changes not only save money but also achieved improved efficiencies. “We’re really trying to improve the benefits to our residents,” he said.

Most of the home’s costs are paid for by a trust fund and monthly fees paid by residents. By law, the Armed Forces Retirement Homes are required to fund, “on-site primary care, medical care and a continuum of long-term care services.”

Tuesday, February 9, 2010

Dr. Conrad Murray has been charged with involuntary manslaughter in connection to the death of Michael Jackson, who Murray was doctor to. Dr. Murray was with the singer when he died on June 25 last year.

The doctor is alleged to have acted “unlawfully and without malice“. The prosecution’s complaint said that when Dr. Murray gave Michael Jackson a powerful sedative to attempt to assist Jackson with his sleeping, he acted “without the caution and circumspection required”. During rehearsals for a series of return concerts planned in London, England last year, the entertainer had employed the doctor to be his personal physician for the spring. If convicted of the charges, Dr. Murray could be imprisoned for up to four years.

According to Ed Chernoff, Dr. Murray’s lawyer, he will not plead guilty. Speaking on Monday, before the charge was filed, Chernoff stated: “We’ll make bail, we’ll plead not guilty and we’ll fight like hell.” The coroner’s office in Los Angeles, California, US reported that the death of Michael Jackson was a homicide, with the cause being an overdose of sedatives.