Login or Sign Up to become a member!
LessThanDot Sit Logo

LessThanDot

Data Management

Less Than Dot is a community of passionate IT professionals and enthusiasts dedicated to sharing technical knowledge, experience, and assistance. Inside you will find reference materials, interesting technical discussions, and expert tips and commentary. Once you register for an account you will have immediate access to the forums and all past articles and commentaries.

LTD Social Sitings

Lessthandot twitter Lessthandot Linkedin Lessthandot friendfeed Lessthandot facebook Lessthandot rss

Note: Watch for social icons on posts by your favorite authors to follow their postings on these and other social sites.

Your profile

    Search

    XML Feeds

    Google Ads

    « Backup file contents with SSIS - Defining the Design24 Hours of PASS: WIT Edition - Awesome Women I'd Love to Hear Talk SQL »
    comments

    Recently I had a request to DELETE all the data except the last 30 days of data from a log table that is 80 GB in size.  It sounded like a simple request on the surface.  However, I couldn’t shut down the web service that logs to that table every 5 minutes.  So I had to get the delete done between inserts.  So, my only choice was to run the delete in small batches.  That’s when I stumbled upon this nice little trick.  New to 2005 and 2008 you can place a number after your GO statement and it will execute the batch that many times.  So, I was able delete records from my log table 10000 at a time without causing issues for the Web Service.

    For Example:

    1. DECLARE @Timestamp datetime
    2. DELETE TOP (10000) from  Log With (tablockx, holdlock)
    3. WHERE Timestamp < @Timestamp
    4. GO 500

    About the Author

    Paul is a SR DBA and one of the Co-Founders of LessThanDot. Paul has over 12 years experience in RDBMS for Financial and Manufacturing companies and experienced with AS400, DB2, Oracle, Sybase and SQL Server. Currently living in the Newport, Rhode Island area with his wife and 5 children Paul dedicates his spare time to his family and coaching Little League and serving as the President of Middletown Little League Board of Directors.
    Social SitingsTwitterLinkedInLTD RSS Feed
    1244 views
    Instapaper

    6 comments

    Comment from: AaronBertrand [Member] Email
    AaronBertrand Paul, this is good for something you really want to do 500 times. With a DELETE, what if there are only 100,000 rows to delete? You're going to execute the query 490 times for no reason... and if a scan is chosen to locate the rows to delete, this could be a pretty expensive no-op. I would rather do something like this (tailoring the TOP and % values depending on the size of the table and my tolerance for log pain):

    http://twitpic.com/3rql7o


    It's messier code, I admit, and I think the [GO int] is a fantastic and little-known gem, but I think a filtered delete is not a very practical use case.

    Also, just as an aside, you didn't give @Timestamp a value. So actually you are going to execute this delete 500 times and never affect a single row...
    01/20/11 @ 07:59
    Comment from: Naomi Nosonovsky [Member]
    Naomi Nosonovsky If you put the above in a loop with
    if @@ROWCOUNT = 0
    BREAK

    can you still use this trick?
    01/20/11 @ 10:15
    Comment from: wqw [Visitor]
    wqw Or you could do it the other way


    SELECT *
    INTO dbo.tmp_log
    FROM dbo.log
    WHERE timestamp >= @Timestamp

    DROP TABLE dbo.log

    exec sp_rename 'dbo.tmp_log', 'dbo.log'



    ... if there are no FKs to log table. You'll have to reindex it too.
    01/20/11 @ 10:39
    Comment from: SQLDenis [Member] Email
    SQLDenis I usually do it this way


    That way it only loops until the table is empty


    I had to run something like it against a big table...just be careful with enough log space...sometime it is faster to insert into another table what you want to keep, truncate this table and then insert the rows you want..if what you want to keep is much smaller

    CREATE TABLE #test (id INT, somevalue VARCHAR(200) )
    GO

    --INSERT 2506 rows
    INSERT #test
    SELECT number, NEWID()
    FROM master..spt_values

    WHILE @@rowcount > 0
    BEGIN
    DELETE TOP (500) #test
    END


    ---------------------
    (500 row(s) affected)
    (500 row(s) affected)
    (500 row(s) affected)
    (500 row(s) affected)
    (500 row(s) affected)
    (6 row(s) affected)
    (0 row(s) affected)
    01/20/11 @ 10:48
    Comment from: Erik [Member] Email
    Erik Log space is definitely a hidden gotcha when doing huge deletes (truncates, too).

    For some colossally-sized delete operation, if you don't want your log file to grow really large, batched deletes with interspersed log backups are the only way to prevent this.

    If your database is in simple recovery mode, you don't even need the log backups, but you still need the batched deletes to keep the total log size small.

    Using batched deletes is going to be faster, too. But truncate is fastest... but locks the table for the entire operation which also may not be right.

    I agree with Denis to use WHILE @@Rowcount > 0 DELETE ...
    01/20/11 @ 15:51
    Comment from: ptheriault [Member] Email
    ptheriault Wow, I touched off a firestorm here. I know the code isn't 100% correct, I was really only trying to show the GO [int] statment. It was something I only just learned and I thought I would share it. WGW, your approach doesn't always work if the table needs to accessed by other processes in between your batch deletes.
    01/21/11 @ 05:40

    Leave a comment


    Your email address will not be revealed on this site.

    Your URL will be displayed.
    (Line breaks become <br />)
    (Name, email & website)
    (Allow users to contact you through a message form (your email will not be revealed.)