I’m going to go over some methods to import data from text files into SQL Server today. The particular file I went out and grabbed is comma delimited and with a few qualifiers in it. It is a typical file you may get and a request made to import or just for your own administrative tasks.

Below is the location of field layout and file that I grabbed off the net to play with. This is just a text file comma separated of zip codes. I will attach the file as well to this blog.

http://spatialnews.geocomm.com/newsletter/2000/jan/zipcodes.html

Field 1 – State Fips Code
Field 2 – 5-digit Zipcode
Field 3 – State Abbreviation
Field 4 – Zipcode Name
Field 5 – Longitude in Decimal Degrees (West is assumed, no minus sign)
Field 6 – Latitude in Decimal Degrees (North is assumed, no plus sign)
Field 7 – 1990 Population (100%)
Field 8 – Allocation Factor (decimal portion of state within zipcode)

Example of file

Import Wizard

First and very manual technique is the import wizard. This is great for ad-hoc and just to slam it in tasks.

In SSMS right click the database you want to import into. Scroll to Tasks and select Import Data…

For the data source we want out zips.txt file. Browse for it and select it. You should notice the wizard tries to fill in the blanks for you. One key thing here with this file I picked is there are “ “ qualifiers. So we need to make sure we add “ into the text qualifier field. The wizard will not do this for you.

Go through the remaining pages to view everything. No further changes should be needed though

Hit next after checking the pages out and select your destination. This in our case will be DBA.dbo.zips.

Following the destination step, go into the edit mappings section to ensure we look good on the types and counts.

Hit next and then finish. Once completed you will see the count of rows transferred and the success or failure rate

Import wizard completed and you have the data!

bcp utility

Method two is bcp with a format file http://msdn.microsoft.com/en-us/library/ms162802.aspx

This is probably going to win for speed on most occasions but is limited to the formatting of the file being imported. For this file it actually works well with a small format file to show the contents and mappings to SQL Server.

To create a format file all we really need is the type and the count of columns for the most basic files. In our case the qualifier makes it a bit difficult but there is a trick to ignoring them. The trick is to basically throw a field into the format file that will reference it but basically ignore it in the import process.

Given that our format file in this case would appear like this

9.0
9
1       SQLCHAR       0       0       """         0     dummy1             ""
2       SQLCHAR       0       50      "",""      1     Field1             ""
3       SQLCHAR       0       50      "",""      2     Field2             ""
4       SQLCHAR       0       50      "",""      3     Field3             ""
5       SQLCHAR       0       50      "","        4     Field4             ""
6       SQLCHAR       0       50      ","          5     Field5             ""
7       SQLCHAR       0       50      ","          6     Field6             ""
8       SQLCHAR       0       50      ","          7     Field7             ""
9       SQLCHAR       0       50      "n"         8     Field8             ""

The bcp call would be as follows

C:Program FilesMicrosoft SQL Server90ToolsBinn>bcp DBA..zips in “C:zips.txt” -f “c:zip_format_file.txt” -S LKFW0133 -T

Given a successful run you should see this in command prompt after executing the statement

Starting copy...
1000 rows sent to SQL Server. Total sent: 1000
1000 rows sent to SQL Server. Total sent: 2000
1000 rows sent to SQL Server. Total sent: 3000
1000 rows sent to SQL Server. Total sent: 4000
1000 rows sent to SQL Server. Total sent: 5000
1000 rows sent to SQL Server. Total sent: 6000
1000 rows sent to SQL Server. Total sent: 7000
1000 rows sent to SQL Server. Total sent: 8000
1000 rows sent to SQL Server. Total sent: 9000
1000 rows sent to SQL Server. Total sent: 10000
1000 rows sent to SQL Server. Total sent: 11000
1000 rows sent to SQL Server. Total sent: 12000
1000 rows sent to SQL Server. Total sent: 13000
1000 rows sent to SQL Server. Total sent: 14000
1000 rows sent to SQL Server. Total sent: 15000
1000 rows sent to SQL Server. Total sent: 16000
1000 rows sent to SQL Server. Total sent: 17000
1000 rows sent to SQL Server. Total sent: 18000
1000 rows sent to SQL Server. Total sent: 19000
1000 rows sent to SQL Server. Total sent: 20000
1000 rows sent to SQL Server. Total sent: 21000
1000 rows sent to SQL Server. Total sent: 22000
1000 rows sent to SQL Server. Total sent: 23000
1000 rows sent to SQL Server. Total sent: 24000
1000 rows sent to SQL Server. Total sent: 25000
1000 rows sent to SQL Server. Total sent: 26000
1000 rows sent to SQL Server. Total sent: 27000
1000 rows sent to SQL Server. Total sent: 28000
1000 rows sent to SQL Server. Total sent: 29000

bcp import completed!

BULK INSERT

Next, we have BULK INSERT given the same format file from bcp

CREATE TABLE zips (
   Col1 nvarchar(50),
   Col2 nvarchar(50),
   Col3 nvarchar(50),
   Col4 nvarchar(50),
   Col5 nvarchar(50),
   Col6 nvarchar(50),
   Col7 nvarchar(50),
   Col8 nvarchar(50)
   );
GO
INSERT INTO zips
    SELECT *
      FROM  OPENROWSET(BULK  'C:Documents and SettingstkruegerMy Documentsblogcenzuszipcodeszips.txt',
      FORMATFILE='C:Documents and SettingstkruegerMy Documentsblogzip_format_file.txt'     
      ) as t1 ;
GO

That was simple enough given the work on the format file that we already did. Bulk insert isn’t as fast as bcp but gives you some freedom from within TSQL and SSMS to add functionality to the import.

SSIS

Next is my favorite playground in SSIS

We can do many methods in SSIS to get data from point A, to point B. I’ll show you data flow task and the SSIS version of BULK INSERT

First create a new integrated services project.

Create a new flat file connection by right clicking the connection managers area. This will be used in both methods

Bulk insert

You can use format file here as well which is beneficial to moving methods around. This essentially is calling the same processes with format file usage. Drag over a bulk insert task and double click it to go into the editor.

Fill in the information starting with connection. This will populate much as the wizard did.

Example of format file usage

Or specify your own details

Execute this and again, we have some data

Data Flow method

Bring over a data flow task and double click it to go into the data flow tab.

Bring over a flat file source and SQL Server destination. Edit the flat file source to use the connection manager “The file” we already created. Connect the two once they are there

Double click the SQL Server Destination task to open the editor. Enter in the connection manager information and select the table to import into.

Go into the mappings and connect the dots per say

Typical issue of type conversions is Unicode to non-unicode.

We fix this with a Data conversion or explicit conversion in the editor. Data conversion tasks are usually the route I take. Drag over a data conversation task and place it between the connection from the flat file source to the SQL Server destination.

New look in the mappings

And after execution…

SqlBulkCopy Method

Sense we’re in the SSIS package we can use that awesome “script task” to show SlqBulkCopy. Not only fast but also handy for those really “unique” file formats we receive so often

Bring over a script task into the control flow

Double click the task and go to the script page. Click the Design script to open up the code behind

Go ahead and put this code into the task.

Imports System
Imports System.Data
Imports System.Math
Imports System.Xml
Imports System.Data.SqlClient
Imports System.Data.OleDb
Imports Microsoft.SqlServer.Dts.Runtime

Public Class ScriptMain


    Public Sub Main()
        Dim conn As New SqlConnection("Data Source=LKFW0133;Initial Catalog=DBA;Integrated Security=SSPI")

        Using bulk As New SqlBulkCopy(conn.ConnectionString)
            bulk.BatchSize = 1000
            bulk.NotifyAfter = 1000
            bulk.DestinationTableName = "zips"
            AddHandler bulk.SqlRowsCopied, AddressOf OnSqlRowsCopied
            bulk.WriteToServer(LoadupDTFromTxt)
        End Using

        Dts.TaskResult = Dts.Results.Success
    End Sub

    Private Sub OnSqlRowsCopied(ByVal sender As Object, _
        ByVal args As SqlRowsCopiedEventArgs)
        Console.WriteLine("Copied {0} so far...", args.RowsCopied)
    End Sub


    Private Function LoadupDTFromTxt() As DataTable
        Dim cn As New OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=C:;Extended Properties=""Text;HDR=No;FMT=Delimited""")
        Dim da As New OleDbDataAdapter()
        Dim ds As New DataSet
        Dim cd As New OleDbCommand("SELECT * FROM C:zips.txt", cn)
        cn.Open()
        da.SelectCommand = cd
        ds.Clear()
        da.Fill(ds, "zips")
        Return ds.Tables(0)
        cn.Close()
    End Function
End Class

Then execute the script task

Again the same results as previous methods but with a new look.

All of these methods have a place in each unique situation. Performance wise in my experience, bcp wins on speed typically. This is not always a method you can use though which leads us to other resources SQL Server and services provide us.

Of course none of this would be completely finished unless we added some statistics on runtimes along with these methods. I went ahead and created 3 addition zips.txt files to go with the original. In these files we have the following

zips.txt – 29,471 rows at around 1.8MB
zips_halfmill.txt – 500,991 rows at around 31.4MB
zips_million.txt – 1,001,981 rows at around 62.8MB
zips_5mill.txt – 5,009,901 rows and around 314.3MB

I ran these each through all the methods. My results are below. Mind you, the important thing to understand is that I write my blogs/articles off my eprsonal test lab. In no way do I utilize monster servers that would be more suited for benchmarking each. All hardware is created not so equal and results will be varying given that variable. Take these results but keep in mind that resources will make them move up and down the chart. Memory, I/O and CPU is a big factor in speed.

Tests were complete by running each process 5 times. All resources cleared on each execution.
Shown in AVG of milliseconds between the types. Import Wizard was not tested as this is basically a Data Flow Task behind the scenes and can be seen (minus the slow user clicking things) from the Data Flow Task in SSIS results

Hope this helps as a good reference in your own imports.

Related Posts