How to Read a File Line by Line in Bash [4 Easy Methods]

As a developer or system administrator, knowing how to read files line by line in Bash is an important skill that can come in handy in a variety of situations. Whether you’re working with log files, configuration files, or any other kind of text file, being able to read them line by line can help you extract the information you need and perform various tasks. In this article, I’ll cover the four methods of reading files line by line in Bash and best practices for handling files and optimizing your scripts.

How to Read a File Line by Line in Bash

There are multiple ways to read a file line by line in Bash, including while and for loop, sed, and awk commands. These four methods allow you to iterate through the file and perform actions like printing lines or manipulating text.

1. Use a While Loop

The while loop is a simple and effective way to read a file line by line in Bash. Here’s how to use this method:

1. Launch the Terminal window and create a while loop structure using the “while” keyword.

while read line; do
<code to be executed>
done

2. This sets up a loop that will continue as long as more input is read.

3. You can include any code you want to execute within the loop for each file line. In this case, the code is $line, which will execute the contents of the variable line.

while read line; do
$line
done

4. Your code in the Terminal should look like this:

5. To specify the file you want to read a file line by line in Bash, use the < symbol followed by the file name. In this example, the file is named “file.txt“.

while read line; do
echo $line
done < file.txt

6. When the script runs, the while loop will read a file line by line in Bash until it reaches the end of the file.

7. The read command within the while loop can also be used to read input from other sources, such as user input. Instead of reading from a file, you can replace file.txt with another input source, like a variable or a command that produces output.

while read line; do
echo $line
done < $input_variable

8. Here, $input_variable represents the variable containing the input or command that produces output.

2. Use a For Loop

The for loop is another efficient way to read a file line by line in Bash. Using the cat command within the for loop, you can iterate over each line of the file, executing code or performing operations on each line individually. Here’s how to use it: 

1. In the Linux command prompt, navigate to the directory using the cd command containing the file you want to read. Then, use the ls command to list the files in the directory.

2. Next, use the cat command with the for loop to read a file line by line in Bash:

for line in $(cat file.txt); do echo $line done

3. Once this command is executed, you’ll see the following output:

4. You can also use the cat command within the for loop to concatenate multiple files into a single stream of data. To do this, simply replace “file.txt” with the names of the files you want to concatenate, separated by spaces. For instance, if you want to read two files, file1.txt and file2.txt, use the following command:

for line in $(cat file1.txt file2.txt); do echo $line; done

5. This will concatenate the contents of file1.txt and file2.txt and read them line by line using the for loop.

3. Use the Sed Command

The sed command is a powerful text editor that can be used to manipulate text in a file. Here’s how to use it to read a file line by line in Bash:

1. In the Terminal, use the cd command to navigate to the directory containing the file you want to read. Then, execute the following command to read a file line by line in Bash:

sed -n 'p' file.txt

2. This command tells sed to print the contents of each line in the file to the Terminal interface.

3. You can also use the sed command in combination with other Bash commands to perform complex operations. For example, to find and replace text in a file, execute the following code:

sed 's/old_text/new_text/g' file.txt

4. This command will search for all occurrences of old_text in the file and replace them with new_text.

4. Use the Awk Command

The awk command is a versatile and powerful tool for text processing in Bash. Here’s how to use it to read a file line by line in Bash:

1. Head to the Terminal and navigate to the directory containing the file you want to read. Then, execute the following command to read a file line by line in Bash:

awk '{ print }' file.txt

2. Now, the awk command will print the contents of each line in the file to the Terminal, and you’ll get the following output:

4. Moreover, you can use the awk command to perform various text-processing tasks, such as filtering and manipulating text. For example, to print only the first field of each line in a file, run the following command:

awk '{ print $1 }' file.txt

5. This command will print only the first field (i.e., the first word) of each line in the file.

3 Common Issues When Reading Files Line by Line in Bash

Reading files line by line in Bash can sometimes be tricky. But don’t worry, understanding how to fix these problems can make working with any file a breeze. Here are the top three things that can go wrong and some easy steps you can follow to sort them out:

1. Improper File Permissions

If the file permissions are not set correctly, you might be unable to read the file line by line. To troubleshoot this issue, check the file permissions using the ls -l command. Then, you might need to use the chmod command to change the file permissions so that you can have the necessary access. For example, if you are getting a “Permission Denied” error, use the chmod +r filename command to add read permission to the file.

2. Blank Lines or Extra Spaces

The file being read might sometimes contain blank lines or extra spaces at the beginning or end of each line. This can cause issues when trying to process the file line by line. To troubleshoot this issue, try using the sed command to remove any extra spaces or blank lines. For example, the command sed -e 's/^[ \t]*//' sample.txt | cat -A can remove leading spaces from each line in the file. The sed command should have removed all leading spaces or tabs from each line in the sample.txt file, and the cat -A command will show you that result.

3. Reading Too Large File

If the file being read is very large, it can slow down the script or cause it to crash. To troubleshoot this issue, you should consider using a method that reads the file in chunks or a command, like a head command, to read only the first few lines of the file. For example, the command head -n 5 filename can be used to read only the first 5 lines of the filename.

3 Best Practices to Read Files Line by Line in Bash

To read files line by line in Bash, there are some best practices to remember to get the most out of it. Here are the top three best practices that you should follow:

• 👀 Always check for errors: It’s important to check for errors and handle them appropriately when reading or writing files. For example, if a file doesn’t exist or can’t be opened, your script should handle the error technically. To check for errors, you can use the if statement in Bash and the test command to check if a file exists before attempting to read it. 
• 📂 Use absolute file paths: When specifying file paths in your scripts, it’s a good idea to use them instead of relative paths. This can help avoid confusion and ensure your script always accesses the correct file. To specify an absolute path, start the path with a forward slash (/) to indicate the root directory. You can also use commands like realpath or readlink to get the absolute file path, use it to perform your file-related operations.
• ❌ Avoid using wildcard characters: When specifying file names in your scripts, it’s best to avoid using wildcard characters such as * and ?. These characters can cause unexpected behavior if the file name contains special characters. Instead, use a specific file name or path to ensure that you’re accessing the correct file. 

To Sum Up

Knowing how to read files line by line in Bash is an important skill for efficient data processing, automation, error handling, and system analysis. And in this article, I’ve provided the four easy methods to do so. It includes using the while loop, for loop, sed and awk commands. Moreover, you’ve learned about the three best practices for handling files in Bash.

If you want to continue your learning journey with other Bash-related topics, check out my other detailed articles on debugging code with set x, manipulating or replacing strings in a text file, and improving code readability with the echo command. Each piece is tailored to help you grasp the concepts at your own pace. Remember, learning is a journey, not a race. So take your time, practice consistently, and most importantly, enjoy the process of mastering Linux!

Frequently Asked Questions