About Me


Articles

Fast Way to Read a File

Did you ever try to read a potentially large text file into an array using kornshell 93?

The conservative approach

The obvious way would be to read the file line by line in a while loop like this

typeset -i i=0
 
while read line
do
    lines[i++]="$line"
done <inputfile

The kornshell approach

But there is also a faster way to read all lines in a single command

IFS=$'\n\n'
set -A lines $(<inputfile)

Setting IFS is necessary to split the file on line boundaries, not word boundaries. Writing the newline character twice preserves empty lines.

How fast is it?

To compare the execution time of the two approaches, I created a textfile with one million lines adding up to about 60 MB. With this file as input I ran both versions on a x86 server running Solaris.

While the first version took over 9 seconds to complete, the other one was done in less than 3 seconds. The second version does not only give you a 300 percent speed boost, the code is also more terse and shell-like.

Paul Herger

SCSA Zertifikat

 
knowhow/kornshell/readfilearray.txt · Last modified: 2011/11/07 17:25 by Paul Herger
© Paul Herger