Removing duplicate indexes

SQL Server has a horrible habit of letting bad habits move forward (for the sake of backward compatibility [e.g. autoshrink]). And, I do understand *some* of the reasons for allowing some of these "features/options" to move forward. However, there are a few that frustrate me and I've talked about this one quite often.

SQL Server lets you create completely redundant and totally duplicate indexes.

Yes, you can even create the same index 999 times! Try it:

SET NOCOUNT ON
go

CREATE TABLE TestTable
(
     col1 int  identity
)
go

DECLARE @IndexID smallint,
        @ExecStr nvarchar(500)
SELECT @IndexID = 1
WHILE @IndexID <= 999
BEGIN
  SELECT @ExecStr = 'CREATE INDEX [Test' + right('00' + convert(varchar(5), @IndexID), 3) + '] ON TestTable (col1)'
  EXEC(@ExecStr)
  SELECT @IndexID = @IndexID + 1
END
go

SELECT count(*)
FROM sys.indexes
WHERE object_id = object_id('TestTable')
go

--DROP TABLE TestTable
go

For somewhat obvious reasons having 999 duplicate indexes does seem completely useless... so, why does SQL Server allow it at all? And, what's the backward compatibility argument? Well, I've written about that before on our SQL Server Magazine Q&A blog (Kimberly & Paul - Questions Answered): Why SQL Server Lets You Create Redundant Indexes.

Having said that - and, for those of you that are only checking YOUR code, you don't use index hints and/or you don't care about backward compat - then, you can use my proc to help you find duplicate indexes (and drop them)! It's all based on my "sp_helpindex rewrites" stored procedures but I had to do some "tweaking" to get the procedures to understand that the order of the columns in the INCLUDE list are irrelevant (in terms of order). So, you can't use the sps as written. They give you the exact (and correct) structure - and, that's exactly how SQL Server stores them. For example, the following two indexes have a different structure. However, they are completely redundant and you do NOT need both.

create index test1 on member(lastname) include (firstname, middleinitial)
create index test2 on member(lastname) include (middleinitial, firstname)
go

Here's what you need:

First, my modified sp_ that exposes all of the columns in all levels of the index. This one does NOT replace the one needed by sp_SQLskills_SQL2008_helpindex. You need both. This version is used to reorder the INCLUDEd columns so that the "finddupes" proc can remove indexes that are different solely because of the order of the columns in the INCLUDE clause (which is irrelevant for navigation/index usage).

(1) Install this one first: 20110715_sp_SQLskills_ExposeColsInIndexLevels_INCLUDE_UNORDERED.sql (6.95 kb)

Second, my modifed sp_helpindex that is used SOLELY by finddupes. Basically, it's just modified to use the UNORDERED version of "exposing the columns" in the nonclustered indexes. Again, this does NOT replace the other sp - you need both.

(2) Install this one second: 20110715_sp_SQLskills_SQL2008_finddupes_helpindex.sql (10.95 kb)

Finally, here's the "finddupes" procedure.

(3) Install this one last: 20110720_sp_SQLskills_SQL2008_finddupes.sql (5.30 kb)

NOTES: At some point in the not-too-distant future I'll modify this for the following:

1) One set of code that will work for 2005/2008

2) I'll combine the two versions of the "expose columns in index levels" so that there's only one.

3) I'll make this work for indexed views

Finally, how can you use this:

To find the duplicates for just one table - you can use one-part or two-part naming:

USE AdventureWorks2008
go

EXEC sp_SQLskills_SQL2008_finddupes 'Production.Document'
go

OR... to find all duplicates in the entire database:

USE AdventureWorks2008
go

EXEC sp_SQLskills_SQL2008_finddupes
go

OR... if you're really motivated, you can run it for all databases - but, before you do - see Aaron Bertrand's comment to this post for his replacement for sp_msforeachdb and use that instead!!

sp_msforeachdb 'USE ?; exec sp_SQLskills_SQL2008_finddupes'
go

Have fun and let me know how many duplicates you find? I found one in AdventureWorks2008 and a few in Northwind. I suspect you'll find a few! This isn't something that will find tons of indexes (to drop) but if there are even a couple of dupes (especially on large tables) you'll have savings in database modification statements, logging, caching, maintenance, storage (and therefore backups, etc.) and this is why dupes are HORRIBLE! I do wish that SQL Server had an option to prevent their being created! Maybe someday?!

Enjoy,
kt