12/6/2014

HowtoReadRawFilesinSSIS

JoinSimpleTalk

Signin
Search...

HowtoReadRawFilesinSSIS
01May2007
bySimonSabin

Ifyou'veeverusedSQLServer2005IntegrationServices(SSIS),youwillhavecome
acrossrawfiles.MicrosoftintroducedthemwithSSISasamechanismforstoringdataon
thelocalfilesystem.Thedesignofrawfilesenablesdatatobewrittenveryquicklybut,
unfortunately,toachievethisperformanceMicrosofthasusedaproprietarybinarystorage
format.Thismeansthatyoucan'topenthesefilesandviewtheircontentsusinganormal
editor.What'smorefrustratingisthatMicrosofthasyettosupplyatoolforreadingthese
files.
Togetaroundthislimitation,Idevelopedmyowncustom"RawFileReader"tool,which
thisarticlebrieflydescribes.

Whyrawfiles?
Oneofthebigbenefitsofworkingwithrawfilesistheeasewithwhichyoucanaddthem
toyourpackages.Therawfiledestinationrequiresyoutospecifyonlythenameoffileto
useandthecolumnsyouwanttostore.Conversely,ifyouuseatextfiledestination,then
youhavetobuildthestructureofthefileandensureitmakesyourdataflow.Ifyouusean
OLEDBdestination,thenyouhavetohaveadatabaseandalsoatablecreatedtostore
thedata.
Theireaseofconfigurationmakesrawfilesidealforstoringerrorsinyourdataflows.Inthe
pastI'vebeenguiltyofconfiguringcertaincomponentsofmypackagestoignorefailures,
ontheassumptionthatit"can'terror".Ofcourse,thisisn'tidealandamuchbetter
approachistocaptureallerrorsandhandlethemappropriately.Thisiswhererawfilesare
veryuseful.Allyouneedtodoisdirectyourerrorflowstoarawfiledestination,specify
andfilenameandselectthecolumnsyouwanttostore.Iferrorsdooccur,thentheywillbe
storedintherawfiles.
So,you'restoringanyerrorsinarawfile,andnowyouwanttoreviewthembutyoucan't
becauseoftheproprietaryformatused.Thisistheexactlythesrotfoscenarioforwhich
myrawfilereaderisdesigned.

UsingtheRawFileReader
TheRawFileReaderisdesignedtomakeitverysimpletoreadthecontentsorrawfiles.
Afewofthekeyfeaturesareasfollows(afulllistcanbeseenattheabovelink):
ReadanySSISRawfile
Resultsdisplayedinagrid
Columnsautomaticallyresized
Orderofcolumnscanbechanged
DatacanbecopiedfromtheresultsgridusingCTRL+C

https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

1/8

12/6/2014

HowtoReadRawFilesinSSIS

DatacanbecopiedfromtheresultsgridusingCTRL+C
Columnheaderscanbeincludedorexcludedfromthecopieddata
Notification(andoptiontoreload)whenthecurrentlyloadedrawfilehaschanged
i.e.packageisrerun

Openingarawfile
Thesimplestoperationistoopenarawfile(File|Open).Thisprocesswill,bydefault,
readtherawfileconfigurationandthenimmediatelyreadthedatafromtherawfile.The
configurationisdisplayedinalistatthetopoftheformandthedataisdisplayedinagrid
atthebottomoftheform(youcanchangethisdefaultbehavioursothatthedataisnot
automaticallyloadedseelater).

Oncethedatafromyourrawfilehasbeenloadedyoucansortthedataandcopyitinto
anotherapplication.Oncearawfilehassuccessfullybeenread,thefilenamewillappearin
theRecentFilessectionoftheFilemenu.Thelast10filesopenedwillbedisplayedhere.

Sortingandcopyingdata
Dataonthegridcanbesortedandcopiedusingstandard"excelstyle"functionality.To
sortdata,simplyclickonthecolumnheadingofthecolumnyouwishtosort.Clickingona
columnheadingforthesecondtimewillreversetheorderofthesorting.
Youcanselectacellfromthegrid(ormultiplecells,bydraggingthecursorand/orusing
theCTRLkey)andcopytotheclipboardusingCRL+C,fromwheretheycanbepastedinto
otherapplications.Whenthedataiscopiedtotheclipboardtheareacopiedwillbethe
maximumarearequiredtoincludealltheselectedcells,howevercellsnotselectedwillbe
blank.Intheexamplebelow,wecopyonlytheobjectnameandtheobjecttype:
https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

2/8

12/6/2014

HowtoReadRawFilesinSSIS

WhenthecopieddataispastedintoExcelyoucanseethat,bydefault,columnheadings
areincludedbutthedataintheunselectedcellsarenotincluded(youcanchangethis
defaultbehaviourseetheOptionssection,later).

Openingtherawfilesassociatedwithapackage
Openingeveryrawfileindividuallymaybetimeconsuming.Therefore,thetoolprovides
theoptionofopeningeveryrawfileassociatedwithapackage.Simply,openthepackage
configuration,identifytherawfiledestinationsandopenthefilesthatarespecifiedfor
thesedestinations.Atpresentthefilenamefortherawfiledestinationsmustbeconfigured
directly,notviaexpressionsorvariables.
Whenyouopenthepackage,alloftherawfilesforthepackagewillbeopened,andcan
beorganizedusingtheoptionsintheWindowmenu.Aswithopeningrawfiles,recently
openedpackagesareincludedintheRecentProjectslistontheFilemenu.

Workingwithrawfileswhilstdebuggingpackages
TheRawFileReaderwillautomaticallyreloadarawfilewhenitdetectsthefilehas
changed.Thisisaconfigurableoptionandsocanbeturnedoffifrequired.Ifyouturnthis
optionoffyoucanmanuallyreloadthefilebyusingtheReadFilecommandintheTools
menu.

Readinglargefiles
Rawfilesareoftenlarge.IfyouareusinglargefilesIrecommendthatyoudisablethe
optionthatenablestherawfiledatatobeloadedassoonasthefileisopened(seethe
Optionssection).Oncetheconfigurationhasbeenread,youchoosetoloadonlythe
requiredcolumns,bycheckinganduncheckingthecolumnsintheRawfilecolumnspane
atthetopofthewindow.Thiscandrasticallyreducethetimeittakestoloadafile.Ifyou
wanttoselect/deselectmorethanonecolumnatatime,selectthemusingtheSHIFTor
CTRLkeysandthenclickononeofthecheckboxes.Alltheselecteditemswillthenbe
changedtothestateofthecheckboxselected.
Onceyouhaveselectedthecolumnsyouwant,therawfilecanbereadbyusingtheRead
FilecommandintheToolsmenu.
Options
TheconfigurableoptionscanbeaccessedbytheOptionsmenuintheToolsmenu.The
copyoptionsconfigureswhenheadersareincludedwithcopiedcells.
Option

Description

Alwaysinclude
headers

Thecolumnheaders,forthecellscopied,arealwaysincluded,
irrespectiveofthenumberordistributionofcellsselected.

Neverincludeheaders Headersareneverincluded
Columnheadersareonlyincludedifthecellsselectedarein
https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

3/8

12/6/2014

HowtoReadRawFilesinSSIS

Autoincludeheaders morethanonecolumn.Thisenablesyoutocopydatafroma
singlecolumnwithoutcolumnheadersbeingadded.
Otheroptionsareasfollows:
Option

Description

Reloadfilewhenit
changes

Thisenablestheapplicationtoreloadthedataforarawfile
automaticallywhentheapplicationdetectstherawfilehas
changed

Ifselectedthedatafromarawfileisreadimmediatelythata
Loadfilewhenopened fileisopened.Ifdeselectedthenonlytherawfileconfiguration
isreadwhenthefileisopened.
Thereisnostandardnamingforrawfileextensions,thisallows
youtoconfiguretheextensionthattheOpenFiledialoguses
todeterminearawfile.Multipleextensionscanbespecifiedby
separatingthembysemicolons""

Rawfileextensions

Futureenhancements
Iamlookingtodeveloptheapplicationfurtherandwouldappreciatecommentsonwhat
featuresyouwouldliketoseeinfutureversions.Someofthosecurrentlyonmylistare
1. Changingtherawfilecolumnspanetodisplaythemetadataforeachcolumni.e.
datatypes,sizesetc,thiswillallowsortingofcolumns
2. Rownumbers
3. Extracolumnheadersdetails
4. Pagingrows
5. Monitoringafolderforrawfiles
Thisarticlehasbeenviewed57743times.

Authorprofile:SimonSabin
SimonSabinisaindependentconsultantspecialisingindatabase
architecture.HehasbeenworkingwithSQLServerforalmost10years,was
awardedasaMicrosoftMVPinApril2006andisanactivememberofUKSQL
ServerUsergroup.YoucanreadhisramblingsonSQLServeronhisblog,
http://sqlblogcasts.com/blogs/simons.
SearchforotherarticlesbySimonSabin

Ratethisarticle:Avgrating:
Poor

OK

HaveYourSay

fromatotalof53votes.
Good

Great

https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

Mustread

4/8

12/6/2014

HowtoReadRawFilesinSSIS

HaveYourSay
Doyouhaveanopiniononthisarticle?Thenaddyourcommentbelow:

Youmustbeloggedintoposttothisforum
Clickheretologin.
Subject:
Postedby:
Postedon:
Message:

HIiSimonSabin
AshwinKumar.H(notsignedin)
Tuesday,June5,2007at6:40AM
Yourarticleisnice.....canuplsguidemetotransferdatafrom
multipleexcelfilestooledbdestinationinssis.....itcanbeinanyway
bythedesignerorbycode.....

Subject:
Postedby:
Postedon:
Message:

Misleadingsubject
Dr.Blue(notsignedin)
Thursday,June7,2007at12:41AM
Giventhetitelofthisarticle,Iexpectedanansweronthequestion
"HowtoReadRawFilesinSSIS".Itturnsout,thatyoudescribehow
toreadrawfilesinyourtool,"whichthisarticlebrieflydescribes."
Nevertheless,agoodtoolIguess,howeverIkeeppreferingstoring
errorsinarelationalenvironmenttomakesomeanalysiseasier.I
guessyouwillnotimplementanySQLdialecttoquerytherawdata
withinyourtool?!

Subject:
Postedby:
Postedon:
Message:

FromJayaaSathiyanarayan
JayaaJayaaSathiyanarayan(notsignedin)
Thursday,June7,2007at3:12AM
HiSimonSabin,
Happytoreadyourintelligentarticle,whichwouldbeverymuch
helpfultome.Mywishestoyouforyourfuturearticles.
Iwouldliketohaveafavourfromyou,Pleasecouldyoupleasehelp
mewiththexmldocumentimportandexportinsqlserver2005using
theManagementstudio.Sincethesqlxmlconfigurationsettingsisbit
confusing.Doyouneedaspecialxmldriverfileforcreatinginodbc
thedatasourceandthencallinthetaskswizardinthemanagement
studioOristhereanysolution.Iknowthereisasolutioninusingvb
scriptandwriteXSDandxmlschemadocument,butIamtryingan
easywayouttoimportusingthetaskswizard.HopeIamnot
disturbingamidsturbusyschedule.Youcandropanemailtome
jayaseeli@hotmail.com
Thankingyou
Regards
JayaaSathiyanarayan

Subject:
Postedby:
Postedon:
Message:

RAWfilelayout
SteveWilliams(notsignedin)
Thursday,June7,2007at3:41PM
Goodinitiativetodevelopyourowntoola"Converttodatabase"
optionwouldprobablybeusefuli.e.createatablewiththesame
layoutastherawfileandthenyoucoulduseitinotherreporting
Doyouhavealinkyoucanpublishtotheformatoftherawfileitself
forotherswhomaywanttoreadthedatabutnotbereliantonathird

https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

5/8

12/6/2014

HowtoReadRawFilesinSSIS

partytool?
Subject:
Postedby:
Postedon:
Message:

RAWfilelayout
SimonSabin(notsignedin)
Monday,August13,2007at10:06AM
Idon'tdoanythingclever.IjustuseSSIStoreadthefile.ThatwayI
don'tneedknowthestructureoftheRAWfile

Subject:
Postedby:
Postedon:
Message:

SSIS
Anonymous(notsignedin)
Wednesday,December5,2007at7:51AM
himichael,
Pleasehelpmeoutinimplementingthelogicstatedbelowwhile
usingSSISpackage.
Supposeaflatfilecontains1000recordswithaheaderandatrailer
records.AndSSISafterreading500recordssuccessfullyinto
database,itfails.NowIwanttheSSIStoreadfromthe501threcord
whenitisstartedagaininsteadofstartingfromthefirst.
Pleasereply,thanksinadvance.

Subject:
Postedby:
Postedon:
Message:

SSIS
Stais(viewprofile)
Wednesday,May14,2008at9:12AM
Cananyonehelpmeindevelopingasimpletask.
Ihavefewfilesinmyimportdirectoryihavetoappenddateendof
theeachfileandarchivetheminanotherFolder.Sotodothiswhich
tasksihavetouse.Ifiusescripttaskwhatcouldbethevbscriptfor
this.
Thanksinadvance.

Subject:
Postedby:
Postedon:
Message:

FilecreatedbySSISandhavingaproblemwiththetool..
Sanjeev(viewprofile)
Monday,June16,2008at5:43PM
Tooldisplaystheconfigurationbutnodatagridasyoumentioned
here.Thereisnoerrormessagedisplayed.
Pleasehelp,asIneedtocontacttheinputfilecreatorwiththe
errorneousrows.
Thanks...

Subject:
Postedby:
Postedon:
Message:

ReadRaw
Anonymous(notsignedin)
Sunday,June22,2008at1:43PM
needoneplayerforreadthisformatonmypcanyonecanhelpme?
RAWRAWRAWAPFFFFFFFFFF

Subject:
Postedby:
Postedon:
Message:

ReadRaw
Anonymous(notsignedin)
Sunday,June22,2008at1:49PM
needoneplayerforreadthisformatonmypcanyonecanhelpme?
RAWRAWRAWAPFFFFFFFFFF

Subject:
Postedby:
Postedon:

SSIS:RAWFilepauseon1000rows
mit_2807(viewprofile)
Tuesday,October14,2008at2:44AM

https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

6/8

12/6/2014

Postedon:
Message:

HowtoReadRawFilesinSSIS

Tuesday,October14,2008at2:44AM
HiSimon,

Thisismithere.Wellihavetriedoutonethingearlierthatwas
RAWFiletoTextFileconversionusingSSIS2005.
Secondly,ItriedsomewhatsimillartoRAWReader.
(Readingrawfileloadingitinrecordsettofilldatagridview.)
Now,Iwanttohaveapausewhileexecutingthepackagefor1000
recordsandifMsgboxyesispressednext1000recordsstartingfrom
1001shouldbeloadedintodatagridview.
Canyoupleasehelpme?
Subject:
Postedby:
Postedon:
Message:

hiSimon...!!!
mit_2807(viewprofile)
Tuesday,October14,2008at9:17AM
Well,
Iamgladtohavethesolutiontomyquerrytodayitselfbutstillifeel
notreachedtosatisfactionlevel.
B'cozididpaging(pausewith1000recordsinrecordset)hereafter
Packageexecutionifeelyoubetterunderstandwhatihavedone.
Stilliwishitshouldhappenduringexecutionitselfactuallyithas
performanceissueattachedtoit.Beforeloadingwholepackagei
shouldbedoing1000rowsload.
Canyoupleaseanswertomyquerry?
Thankingyou.Pleaseasapineedtoprepareutilitywhichmightbe
helpfultoourclientnmightbesomanyofmyfriendsoutthere.

Subject:
Postedby:
Postedon:
Message:

Howtoopenarawfile
Arifa(viewprofile)
Wednesday,July14,2010at5:55AM
HiSimon...
Thisiswhatustatedintheabovepost,"Thesimplestoperationisto
openarawfile(File|Open)."butstillicoundn'tbeabletoopenthe
rawfile.
Pleasehelpmebylittlemoreexplanatory.

About
Sitemap
Becomeanauthor
Newsletters
Contactus
Help
Privacypolicy

Termsandconditions

20052014RedGateSoftware

https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

7/8

12/6/2014

https://www.simpletalk.com/sql/ssis/howtoreadrawfilesinssis/

HowtoReadRawFilesinSSIS

8/8