Tools‎ > ‎Ubuntu/Linux server‎ > ‎

Loop over list of files

Ubuntu / Linux bash shell

Shell loop through files - Examples

Run a command on each file

do something (echo) with all .txt files
for f in *.txt;  do echo ${f}; done;

Example: Search string ‘ABC’ in all .txt files
for f in *.txt ; do echo --${f}-- ; grep "ABC" ${f} ; done

same loop over files, but using a pipe (reading from standard input), and a while-loop
ls *.txt | while read f; do echo ${f}; done;


do something with a small set of files 
for f in file1.txt file2.txt file3.txt; do echo ${f}; done;
file1.txt
file2.txt
file3.txt

same, but separately defined list of files
filelist="file1.txt file2.txt file3.txt"
for f in ${filelist}; do echo ${f}; done;


reading list of files from file 'filelist.txt' (for larger number of files)
ls *.csv > filelist.txt    # define your list of files
for f in `cat filelist.txt`; do echo ${f}; done;
if a line may include spaces better use a while loop:
cat filelist.txt | while read LINE; do echo "${LINE}"; done


loop over filenames, but remove file extension, see → basename string split
for f in Data/*.txt; do FILENAME=${f%%.*};  echo ${FILENAME};  done;
Data/fileA .txt
Data/fileB .txt

loop over filenames, but remove file extension and path prefix
for f in Data/*.txt ; do FILENAME=`basename ${f%%.*}`; echo ${FILENAME}; done
Data/ fileA .txt
Data/ fileB .txt


Exclude samples that are already processed
process input files Data/*.fastq only if result-files Result/*.txt does not exist
for f in Data/*.fastq; do \
  SAMPLE=`basename ${f%%.*}`; \
  if [ ! -f Results/${SAMPLE}.txt ]; then \
     echo "processing sample ${SAMPLE}"; \
     # do something \
  fi; \
done


# do something 10 times
for N in {1..10}; do \
   echo ${N}; \
done

# double loop over a series of numbers and letters
for N in {1..5}; do
  for S in {A..C}; do
    echo ${N} ${S};
  done;
done;
1 A
1 B
1 C
2 A
2 B
2 C
°°°

general use of:  if then else  (-f  check if normal file)
if [ -f /path/to/${SAMPLE} ];
then
   # do something if file exist
else
   # do something if file does not exist
fi


Copy selected subset of files
# copy all files in filelist.txt to newDir/  
ls *.csv > filelist.txt  # define your list of files (edit text file)
mkdir newDir             # create a new directory
for f in `cat filelist.txt`; do echo copying ${f}; cp ${f} newDir/; done;
# add path if files are not located in working directory
for f in `cat filelist.txt`; do echo copying ${f}; cp path/to/files/${f} newDir/; done;
see command → find