Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

10 Working with Files: Basics in Reading and Writing

10.1Lesson goals

10.2Overview

Science and engineering deals with data, lots of data. This data will often come in the form of a file. Python has many tools available to read, write, and append data to and from files. In this lesson we will focus on some basic, built-in functionality that Python has when interacting with files.

10.3Working with files

The basic functionality of opening a file for reading or writing is done with the built-in open() function. The open() function takes in one required positional argument and many optional keyword arguments:

open(file, mode = "r", buffering = -1, encoding = None, errors = None, newline = None, closefd = True, opener = None)

The file positional argument specifies the path to the file to be opened. Of the optional keyword arguments, the mode argument is the one you will use the most. The mode specifies how open() should treat the file that is being opened. There are several modes, and you can read up on all the modes in the documentation, but the most common are r for read, w for write, a for append. By default, the mode is set to r to simply read the file. Below is a summary of the behavior, and we will try each out in the following examples.

ModeBehavior
wCreates file or clears the file contents, opens it for writing.
rOpens the file only for reading, writing is not allowed.
aOpens the file for writing, appends content to the end of the file

As a reminder, you can check out all the arguments and modes of open() with the help() function:

help(open)

A file path is a string that contains a listing of the folders and subfolders to where a file is located followed by the name of a file. For example, we can generate the file path of a file called tutorial_test.txt with the following commands:

# generate a file path with current directory
directory = "./"
filename = "tutorial_test.txt"
file_path = directory + filename
print(file_path)
./tutorial_test.txt

The directory ./ is shorthand for the current directory that the Python shell is operating in. If you are unsure what directory you are in, we can use the built-in os library to navigate our computer’s file system. For example, let’s look at our current directory (a.k.a., a folder!):

# generate a file path using os library
import os

# get current directory
directory = os.getcwd()

# file name
filename = "tutorial_test.txt"

# combine to get full path
file_path = directory + "/" + filename
print(file_path)
/srv/docs_sci_python/pages/python-fundamentals/tutorial_test.txt

The os.getcwd() command gives us our current working directory. We can also get the absolute file path for a file in Python using the built-in os.path module:

import os

# generate a file path with absolute path using os.path.abspath()
directory = os.path.abspath(".")    # get absolute path from the current directory
filename = "tutorial_test.txt"
file_path = directory + "/" + filename
print(file_path)
/srv/docs_sci_python/pages/python-fundamentals/tutorial_test.txt

Let us first try writing to a file. The following code block will open the file called tutorial_test.txt in the current directory:

# write data to a file called "tutorial_test.txt" in current directory
import os

# Create file path
current_directory = os.getcwd()
file_name = "tutorial_test.txt"
file_path = current_directory + "/" + file_name
print(file_path)

# Writing data to file
file = open(file_path, "w")
file.write("hello world!")
file.close()
/srv/docs_sci_python/pages/python-fundamentals/tutorial_test.txt

Writing to a file typically involves three basic steps. First, we open the file using open() with the file name and w as input arguments. Next, we write to the file using the .write() method (i.e., an object specific function). We then close the file using the .close() method. Since tutorial_test.txt did not already exist it was also created. If you use a text editor like TextEdit on macOS or Notepad on Windows to open the file “tutorial_test.txt”, you should see “hello world!” printed out.

We can also use the ./ characters before the file name to write the file to the current directory:

# write data to a file called "tutorial_test.txt" in current directory
file = open("./tutorial_test.txt", "w")
file.write("hello world!")
file.close()

10.5Reading a file

Now instead of using a text editor to open a file, let us use Python to read the contents of a file:

# read data from "./tutorial_test.txt"
file = open("./tutorial_test.txt", "r")
print(file.read())
file.close()
hello world!

When a file is opened in r mode (i.e., read mode), you can use the f.read() function to read the entire contents of the file. In this case, “hello world!” was printed out since that was already stored in our previous write step.

10.6Appending to a file

We can also append a file by using the a mode option in open(). This allows us to add to an existing file without completely writing over it. Appending adds the new content at the end of the file. Below creates a fresh copy of tutorial_test.txt first and then appends to it:

# start with fresh copy of file, write, and close
file = open("./tutorial_test.txt", "w")
file.write("hello world!")
file.close()

# append to file and close
file = open("./tutorial_test.txt", "a")
file.write("goodbye world!")
file.close()

# read file and close
file = open("./tutorial_test.txt", "r")
print(file.read())
file.close()
hello world!goodbye world!

OOPS! We need a line return! Appending adds right to the end of the file so if we want it more human readable we should add a new line (\n) character too:

# start with fresh copy of file, write, and close
file = open("./tutorial_test.txt", "w")
file.write("hello world!")
file.close()

# append to file and close
file = open("./tutorial_test.txt", "a")
file.write("\n")
file.write("goodbye world!") # notice the double .write() commands...valid!
file.close()

# read file and close
file = open("./tutorial_test.txt", "r")
print(file.read())
file.close()
hello world!
goodbye world!

Notice that we can perform sequential .write() commands without closing the file. This way we can add multiple lines of text / data before closing the file.

Using open(), .write(), and then .close() to edit files does work, but it does require keeping track of when a file is opened and closed. The recommended way to work on files is to utilize the with keyword to edit a file through the with open() as file: statement followed by a code block that contains either the .write() or .read() methods.

# write data to a file called "tutorial_test.txt" in current directory & close
with open("./tutorial_test.txt", "w") as file:
    file.write("hello world!")

# reopen, append, and close
with open("./tutorial_test.txt", "a") as file:
    file.write("\n")
    file.write("goodbye world!")

# read file and close
with open("./tutorial_test.txt", "r") as file:
    print(file.read())
hello world!
goodbye world!

Notice that the code block above does not have the command file.close() anywhere. That is because file.close() gets automatically run at the end of the with code block. Using the with open() as file: pattern is useful for containing all the logic around writing or reading to a file in one code block. We will be using this pattern for the rest of the lesson.

10.8CSV files

So far we have shown a few simple ways to open, write, and read strings of data to files. In science and engineering applications, strings of data are often stored in an “array-style” format consisting of rows and columns. In order to be able separate out these data strings from one another, a “delimiter character” is often placed between these strings. Commonly used delimiters include commas (,), semicolons (;), quotes (“ ” or ‘ ’), pipes ( | ), slashes ( / or \ ), and tabs.

In particular, comma delimiters are a very popular choice in separating tabulated data. This has led to the creation of the comma-separated values (CSV) file format. Let’s go over various ways we can create CSV files using both built-in functions and external libraries.

This route builds off what we already know about using .write() to write out our data for each row with a comma between each entry:

In this example we use the random.gauss() function from the random library to generate a set of 20 random numbers. These random numbers and a increment number are initially stored in the variable data (see the first for loop). We bound round() around our gauss() call to limit the number of sig figs to 3 after the decimal point.

The first with open() code block writes the data to tutorial_test.csv. We start by writing a “header” row (file.write("entry,data")) for readability purposes and then we use a for loop to write each increment-number pairing. Notice that we have to cast all the values as str objects when writing to the file and use the + character to chain the string parts together. We then use the second with open() code block to read the data and display it to the shell.

While the process is a bit cumbersome, it is very doable!

The built-in csv library is a simple, but powerful library designed to read, write, and modify CSV files. The library has two primary functions when working with CSV files: csv.reader() and csv.writer(). Let us first demonstrate how to write to a CSV file using the `csv.writer() function using our example from above:

import csv
from random import gauss

data = []

# generate array of random gaussian distribution (mean 10, std 3) in format of
# [ increment, random_float ]
for i in range(20):
    data.append([i, round(gauss(mu=10, sigma=3), 3)])

print("data:")
print(data)
print("~~~~~~~~~~~~~")

# write data to "./tutorial_test.csv" in current directory
with open("./tutorial_test.csv", "w", newline="") as file:
    # create writer object to do the writing
    writer = csv.writer(file, delimiter=",")

    # write header row
    writer.writerow(["entry", "data"])

    # write data rows
    for i in data:
        writer.writerow(i)

# read file
print("file contents:")
with open("./tutorial_test.csv", "r") as file:
    print(file.read())
data:
[[0, 12.413], [1, 15.445], [2, 8.688], [3, 10.63], [4, 11.29], [5, 15.763], [6, 9.847], [7, 10.658], [8, 14.495], [9, 12.234], [10, 8.346], [11, 14.25], [12, 9.698], [13, 11.084], [14, 12.859], [15, 7.385], [16, 13.484], [17, 12.502], [18, 9.789], [19, 9.135]]
~~~~~~~~~~~~~
file contents:
entry,data
0,12.413
1,15.445
2,8.688
3,10.63
4,11.29
5,15.763
6,9.847
7,10.658
8,14.495
9,12.234
10,8.346
11,14.25
12,9.698
13,11.084
14,12.859
15,7.385
16,13.484
17,12.502
18,9.789
19,9.135

Notice our familiar with open() as file: pattern, but now include the optional argument newline = "". This optional argument ensures that no additional line returns are placed after writing a comma delimited row of data and is recommended when working with CSV files.

Next, we create a writer object that is instantiated by calling csv.writer(). The function csv.writer() takes in a file object as a required first argument, and allows for many optional formatting options. In our example, we specifically specify that delimiter = ",", which is technically redundant since the CSV file format uses , as its delimiter. We have added this for educational purposes in order to show how an optional format can be included. After creating the writer object, we call the writer.writerow() method each time we want to write to the file.

10.8.3Reading CSV files with the csv library

Now that we have written a file with csv.writer(), we can try to read it with the csv.reader() function:

data = []

with open("./tutorial_test.csv", "r", newline="") as file:
    reader = csv.reader(file, delimiter=",")
    for i in reader:
        data.append(i)
        
print("Data from file:", data)
Data from file: [['entry', 'data'], ['0', '12.413'], ['1', '15.445'], ['2', '8.688'], ['3', '10.63'], ['4', '11.29'], ['5', '15.763'], ['6', '9.847'], ['7', '10.658'], ['8', '14.495'], ['9', '12.234'], ['10', '8.346'], ['11', '14.25'], ['12', '9.698'], ['13', '11.084'], ['14', '12.859'], ['15', '7.385'], ['16', '13.484'], ['17', '12.502'], ['18', '9.789'], ['19', '9.135']]

Similar to before, we start with an empty list called data and use the with open() as file: pattern to open our file for reading (again adding the additional newline = "" to the open() function). The csv.reader() function is called to create a reader object that reads the file. We then iterate through the reader object to loop through the rows in the file, and we append each row to the data list.

We can see from the print() command that data is a nested list (i.e., a list of lists) made up of strings. Our current implementation of csv.reader() casts all data as str objects, which may not be desirable since we are expecting our values to be float objects. There are a few different ways we can fix this. The modified code below shows how a simple post-processing step after reading the CSV file can remove the header row and casts the rest of the entries as float objects:

data = []

# read data
with open("./tutorial_test.csv", "r", newline="") as f:
    reader = csv.reader(f, delimiter=",")
    for i in reader:
        data.append(i)

print("data:")
print(data)
print("\n")

# remove the header row and store it separately
header = data.pop(0)

# use list comprehension to cast values at floats
data = [[float(i[0]), float(i[1])] for i in data]

# print results
print("Header data:", header)
print("Data from file:", data)
data:
[['entry', 'data'], ['0', '12.413'], ['1', '15.445'], ['2', '8.688'], ['3', '10.63'], ['4', '11.29'], ['5', '15.763'], ['6', '9.847'], ['7', '10.658'], ['8', '14.495'], ['9', '12.234'], ['10', '8.346'], ['11', '14.25'], ['12', '9.698'], ['13', '11.084'], ['14', '12.859'], ['15', '7.385'], ['16', '13.484'], ['17', '12.502'], ['18', '9.789'], ['19', '9.135']]


Header data: ['entry', 'data']
Data from file: [[0.0, 12.413], [1.0, 15.445], [2.0, 8.688], [3.0, 10.63], [4.0, 11.29], [5.0, 15.763], [6.0, 9.847], [7.0, 10.658], [8.0, 14.495], [9.0, 12.234], [10.0, 8.346], [11.0, 14.25], [12.0, 9.698], [13.0, 11.084], [14.0, 12.859], [15.0, 7.385], [16.0, 13.484], [17.0, 12.502], [18.0, 9.789], [19.0, 9.135]]

After reading our CSV file, we “pop” out the first element with the .pop() built-in method for lists. The command .pop(0) will pop out the passed index (i.e., 0) and remove it from the list. Then we use list comprehension to loop through the data, cast the first and second elements of each sub-list as a float object, and reassign the whole thing back to data.

Another solution is to change when writer objects generate quotes for data (i.e., making them str objects). According to the csv documentation, we can pass the formatting argument quoting = csv.QUOTE_NONNUMERIC to our csv.writer() and csv.reader() calls to add quotes to all data EXCEPT numerical-based data, which instead will be represented as float objects. This modification is done in the code below:

data = []

# generate array of random gaussian distribution (mean 10, std 3) in format of
# [ increment, random_float ]
for i in range(20):
    data.append([i, round(gauss(mu=10, sigma=3), 3)])

# write data to "./tutorial_test.csv" in current directory
with open("./tutorial_test.csv", "w", newline="") as file:
    # create writer object to do the writing
    writer = csv.writer(file, delimiter=",", quoting=csv.QUOTE_NONNUMERIC)

    # write header row
    writer.writerow(["entry", "data"])

    # write data rows
    for i in data:
        writer.writerow(i)

# read file
with open("./tutorial_test.csv", "r", newline="") as file:
    reader = csv.reader(file, delimiter=",", quoting=csv.QUOTE_NONNUMERIC)
    for i in reader:
        data.append(i)
        
print("Data from file:", data)
Data from file: [[0, 9.413], [1, 8.162], [2, 8.205], [3, 19.74], [4, 5.394], [5, 12.11], [6, 10.376], [7, 7.948], [8, 10.151], [9, 11.979], [10, 13.843], [11, 4.175], [12, 3.462], [13, 10.276], [14, 15.098], [15, 11.278], [16, 8.536], [17, 11.434], [18, 3.19], [19, 7.306], ['entry', 'data'], [0.0, 9.413], [1.0, 8.162], [2.0, 8.205], [3.0, 19.74], [4.0, 5.394], [5.0, 12.11], [6.0, 10.376], [7.0, 7.948], [8.0, 10.151], [9.0, 11.979], [10.0, 13.843], [11.0, 4.175], [12.0, 3.462], [13.0, 10.276], [14.0, 15.098], [15.0, 11.278], [16.0, 8.536], [17.0, 11.434], [18.0, 3.19], [19.0, 7.306]]

Yet another solution to reading data from a file utilizes the NumPy library’s function numpy.loadtxt(). This function will to read a text file, skip its header row(s), and cast the entries as float objects. Even though we have briefly discussed loading the NumPy library earlier in this lesson and will hold off discussing this library in much more detail in a later lesson, let us use the numpy.loadtxt() function from the numpy library to load our CSV file:

from numpy import loadtxt

data = loadtxt("./tutorial_test.csv", delimiter=",", skiprows=1)

print("Data from file:", data)
Data from file: [[ 0.     9.413]
 [ 1.     8.162]
 [ 2.     8.205]
 [ 3.    19.74 ]
 [ 4.     5.394]
 [ 5.    12.11 ]
 [ 6.    10.376]
 [ 7.     7.948]
 [ 8.    10.151]
 [ 9.    11.979]
 [10.    13.843]
 [11.     4.175]
 [12.     3.462]
 [13.    10.276]
 [14.    15.098]
 [15.    11.278]
 [16.     8.536]
 [17.    11.434]
 [18.     3.19 ]
 [19.     7.306]]

This call is remarkably short compared to the other examples. The only required argument for numpy.loadtxt() is the path to the file that will be read. The numpy.loadtxt() function is used to load any type of text file, not just CSV files, so the default delimiter is a space character " ". The delimiter="," arguments sets the single comma character as the delimiter in our CSV file, and the skiprows=1 argument skips over the first row that contains our column headers of "Timestamp", "Data".

10.9Final thoughts

This lesson focused on the “age-old” problem of reading and writing data. Science and engineering applications often require you collect / store data and then read it later for processing. We first learned how to read and write files using the built-in open() function and the .write(), .read(), and .close() methods associated with file objects. We then learned how to use the with open() as file: pattern to automatically handle closing a file when we are done. Finally, we discussed the ever popular CSV file format and different ways to create and read CSV files. Happy data processing!

Part 1: Python Fundamentals
Understanding Errors
Part 2: Scientific Python
Object-Oriented Programming: Objects, Classes, Attributes, Methods