Exchange 2010: Recover from a Full Log Drive

Download PDF Version

Download PDF Version.
View in Google Docs

1.0                Introduction

Despite best efforts, at some time or another, in an Exchange Admin’s lifetime, your system will run out of log space, resulting in the dismounting of databases and thus users not being able to get to email.
*** To get users back online, you only need to complete sections 2.0 to 5.0. ***

My Exchange system has two mailbox servers (EXCH03 & EXCH04) where the database and transactional log files reside. There are 10 user mailbox databases, MBDB01 – MBDB10. The two mailbox server act in an active / passive cluster and are laid out thus:

Mailbox Database

 

EXCH03

 

 

EXCH04

 

 

MBDB01

 

 

Active

 

 

Passive

 

 

MBDB02

 

 

Active

 

 

Passive

 

 

MBDB03

 

 

Active

 

 

Passive

 

 

MBDB04

 

 

Active

 

 

Passive

 

 

MBDB05

 

 

Active

 

 

Passive

 

 

MBDB06

 

 

Passive

 

 

Active

 

 

MBDB07

 

 

Passive

 

 

Active

 

 

MBDB08

 

 

Passive

 

 

Active

 

 

MBDB09

 

 

Passive

 

 

Active

 

 

MBDB10

 

 

Passive

 

 

Active

 

 

The server hosting the active database sends transaction logs to the passive database server in a process known as “Log Shipping”.

 

IMPORTANT: When performing these documented tasks, make sure the cmdlets are carried out on the correct server. Unless stated otherwise, this is always the mailbox server hosting the active copy.

1.1                 Disclaimer:

As this is a public document, unfortunately, I have to mention this. You cannot hold me responsible for any negative impact on your system; loss of data, downtime or the like. It is your responsibility to confirm any information contained herein is correct, complete and interpreted correctly.  This document should only be used as a suggestive outline. Nothing should be taken as fact. If you are not sure of anything, ask Google, Microsoft or test on a non-production system first.
2.0                Move committed log files

Step

 

Process

 

 

1.                     

Connect to EXCH03

 

 

2.                     

Launch Exchange Management Shell

 

 

3.                     

Navigate to R:\ and create a folder “MBDB01”

 

 

4.                     

Cd to L:\MBDB01

 

 

5.                     

Find the “chk” file by run the command:

 

 

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.02
Copyright (C) Microsoft Corporation. All Rights Reserved.

 

Initiating FILE DUMP mode…
      Checkpoint file: E02.chk

 

      LastFullBackupCheckpoint: (0x0,0,0)
      Checkpoint: (0x86D85,8,16)
      FullBackup: (0x857DC,459,14F)
      FullBackup time: 12/07/2013 18:04:01
      IncBackup: (0x84AD1,34F,1AD)
      IncBackup time: 12/06/2013 18:03:31
      Signature: Create time:05/06/2012 21:30:55 Rand:5193468 Computer:
      Env (CircLog,Session,Opentbl,VerPage,Cursors,LogBufs,LogFile,Buffers)
 (    off,   1027,  51350,  16384,  51350,   2048,   2048, 318492)

 

Operation completed successfully in 0.31 seconds.

 

7.                     

Make a note of the value in red. E.g. 86D85

 

 

8.                     

In Windows Explorer, navigate to L:\MBDB01\

 

 

9.                     

List the files in descending date order.

 

 

10.                  

Look for the file ending in 86D85.log. (E0200086D85.log)

 

 

11.                  

Select the next oldest file and all older *.log files

 

 

12.                  

Move these files to R:\MBDB01\

 

 

13.                  

Repeat these steps for all failed databases.

 

EXCH03 = MBDB01 to MBDB05
EXCH04 = MBDB06 to MBDB10

3.0                Enable Circular Logging

When Circular logging is enabled, once the log is committed to the database, Exchange removed the log file, regardless of whether a backup has completed or not. This will clear down the logs on the passive copy.
Step

 

Process

 

 

14.                  

Run the following cmdlet:

 

1 [PS] C:\>    Get-MailboxDatabase -Identity MBDB01 | Set-MailboxDatabase -CircularloggingEnabled $true

 

15.                  

Repeat for all failed databases.

4.0                Suspend Database Copies

To stop Exchange copying logs from the active to passive database server, suspend the copy process.
Step

 

Process

 

 

16.                  

Run the following cmdlet:

 

C:\>    Suspend-MailboxDatabaseCopy –Identity MBDB01\*03 –Confirm:$false

 

17.                  

Repeat for all failed databases on their respective “active” server.

5.0                Mount the Database

Once circular logging is enabled, mount the database.
Step

 

Process

 

 

18.                  

Run the following cmdlet:

 

2 [PS] C:\>    Mount-Database –Identity MBDB01

 

19.                  

Repeat for all failed databases on their respective “active” server.

 

 

6.0                Enable Passive Copies

6.1                 Resume Database Copies

Initially, we will attempts to resume the database copy. Success is dependent on several factors, but mainly, how long the database has not replicated for.
Step

 

Process

 

 

20.                  

Run the following cmdlet:

 

1 [PS] C:\>    Resume-MailboxDatabaseCopy MBDB01\EXCH04

 

21.                  

Repeat for all failed databases on their respective “active” server.

 

 

22.                  

Check the status:

 

2 [PS] C:\>    Get-MailboxDatabaseCopyStatus –Server EXCH04 | ft Name,Status -AutoSize
Where EXCH04 is the passive node for the given database.

 

23.                  

If the Status column shows “FailedAndSuspended” proceed to Section 6.2.

 

 

 

The copy will have failed between Exchange mailbox servers. The database on the passive side will have to be copied across. A process known as “Reseeding”.
Step

 

Process

 

 

24.                  

Run the following cmdlet:

 

3 [PS] C:\>    Update-MailboxDatabaseCopy -Identity MBDB01\EXCH04 -DeleteExistingFiles
The database will “reseed” depending on how long the mailbox was down for, the reseed may take a considerable amount of time.

 

25.                  

Repeat for all failed databases on their respective “active” server.

 

It is possible to open multiple EMS windows and run the same command

 

26.                  

Run the following cmdlet to confirm the copy progress:

 

4 [PS] C:\>    Get-MailboxDatabaseCopyStatus –Server EXCH04 | ft Name,Status -AutoSize

7.0                Balance out Databases

If the databases are mounted on one side or not correctly, a script can be executed to automatically balance the Active / Passive layout according to the Database’s Activation Preference.

 

It is important this is done as the databases are backed up from the passive copy and the backup job is configured to point to the server that normally hosts the passive copy.

 

This can be run from either mailbox database server.
Step

 

Process

 

 

27.                  

To check the current balance, run the following script:

 

1 [PS] C:\>    Get-MailboxDatabase | ft name, server, activationpreference –AutoSize
In the server Column, mailboxes 01 to 05 should be listed on EXCH03 and 06 to 10 should be listed on EXCH04.

 

28.                  

To balance mailbox databases correctly, run the following cmdlet:

 

2 [PS] C:\>    cd $exscripts
3 [PS] C:\>    .\RedistributeActiveDatabases.ps1 -DagName EXCHDAG01 –BalanceDBsByActivationPreference –Confirm:$False

 

29.                  

Rerun the following cmdlet to confirm they are balanced correctly:

 

4 [PS] C:\>    Get-MailboxDatabase | ft name, server, activationpreference –AutoSize

8.0                Disable Circular Logging

Once the log drive used space has dropped back down to a reasonable size, disable Circular logging. A backup should help flush logs.
Step

 

Process

 

 

30.                  

To list all databases where circular logging is currently enabled:

 

1 [PS] C:\>    Get-MailboxDatabase | ?{$_.CircularLoggingEnabled -eq $true} | ft Name,CircularLoggingEnabled -AutoSize

 

31.                  

To turn off Circular Logging on an individual database:

 

2 [PS] C:\>    Get-mailboxdatabase -Identity MBDB01 | Set-MailboxDatabase -CircularloggingEnabled $false

 

32.                  

To turn off Circular Logging for all databases where it is enabled:

 

3 [PS] C:\>    Get-MailboxDatabase | ?{$_.CircularLoggingEnabled -eq $true} | Set-MailboxDatabase -CircularloggingEnabled $false

 

33.                  

Check the change has been applied.

 

 

 https://googledrive.com/host/0B7bCCNRA_s7CSTlMRldIeEJjMUU/Exchange2010-OutOfLogDriveSpace(blog).pdf

Please feel free to leave a comment...