13 NumPy

13.1Lesson overview¶

NumPy is the go-to library to use when working with scientific data. With NumPy, we get access to numerous numerical method tools that we can use on our data. NumPy, at its core, handles data using arrays (think matrices in math). These arrays are described with a “shape” parameter (i.e., how many dimensions the array takes up and the size in each dimension), a “data type” parameter (i.e., what type of data can be stored in the array), and the values that reside in the array. If you have a MATLAB background, is may sound similar to how data is stored in this program. In fact, NumPy is practically equivalent to MATLAB’s numerical method functionality. In this lesson, we will create NumPy arrays, checkout the different NumPy data types, learn how arrays can be referenced and copied, and explore a few of the many methods and mathematical operations that NumPy provides.

13.2Installing the NumPy library¶

NumPy is not part of the standard Python library. However many library repositories have it. The steps below go over how to install NumPy to your Python virtual environment using Miniforge:

1. Open up Miniforge.

2. Activate your Python environment.

3. Type conda install -c conda-forge numpy in the command line interface.

4. Follow the onscreen commands to install the library.

5. Restart VSCode or JupyterLab if they are currently opened once the library is installed.

13.3Importing NumPy¶

In order to use NumPy, we first need to import the numpy library into Python:

import numpy as np

The as keyword creates an alias for numpy as np. This is very common when using NumPy as it cuts down on keystrokes. We will use the np shorthand in the code blocks and external links throughout this lesson.

13.4Creating a NumPy array¶

Array creation is often done by passing a sequence-based object (e.g., a list) into numpy.array():

a = np.array([1, 2, 3])
print("type a:", type(a))
print("a:", a)

type a: <class 'numpy.ndarray'>
a: [1 2 3]

The np.array([1, 2, 3]) call creates a numpy.ndarray object, which is core array data class when using NumPy. Almost all NumPy calculations or operations will be done using NumPy arrays.

We can check the number of dimensions an array takes up and the size of each dimension using the .ndim and the .shape attributes, respectively:

print("a.ndim:", a.ndim)
print("a.shape:", a.shape)

a.ndim: 1
a.shape: (3,)

In this example, a.ndim returns the value 1, meaning a is a one-dimensional array. Similarly, a.shape returns (3,), meaning this array has one dimension that is three elements long. The output notation for a.shape looks odd at first glance because this attribute returns a tuple that reports the lengths of each dimension. Since there is only one dimension in a, it returns the tuple object (3,). If an array has more than one dimension, you would see a sequence of numbers in the tuple.

13.5Two-dimensional arrays¶

Two-dimensional and higher dimensional arrays are very common in data science applications. They can be used to effectively store data files and simulate matrices in math. We can create multi-dimensional arrays in NumPy quickly by including more list objects in our np.array() call:

a = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
print("a.shape:", a.shape)
print("a:")
print(a)

a.shape: (3, 4)
a:
[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]

The above code block creates a $3 \times 4$ two-dimensional array and its dimensionality is verified with the a.shape attribute call. Therefore, each list entry acts as a “row” for the array if we want to think of the array as a matrix unit in math.

13.6Indexing¶

NumPy has two common ways to index arrays: (1) using Python’s sequence-based indexing schema and (2) using MATLAB’s indexing schema. Let’s go over how to utilize both schemas since you will probably encounter both when looking at code.

13.6.1Using Python’s sequence-based indexing schema¶

We can index and extract elements from numpy.ndarray objects following Python’s schema for sequences (e.g. lists) and nested structures. The code block demonstrates this using our 2D array from earlier:

print("a:")
print(a)

# Recall first row, first column position, index: 0,0
print("First row, first column:", a[0][0])

# Recall third row, first column, index: 1,0
print("Third row, first column:", a[2][0])

# Recall entire second row
print("Second row:", a[1])

a:
[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]
First row, first column: 1
Third row, first column: 9
Second row: [5 6 7 8]

This schema is easy to implement but is difficult to use when more elaborate indexing setups are needed (e.g., multiple rows or columns).

13.6.2Using MATLAB’s indexing schema¶

We can also use MATLAB’s indexing schema, which follows the format of a(row,col), but we need to use brackets ([ ]) instead. So the NumPy equivalent follows the general format of a[row,col]. We can couple this with the : character to get multiple rows and columns:

print("a:")
print(a)

# Recall the last three columns
print("Last three columns:")
print(a[:,1:4])

a:
[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]
Last three columns:
[[ 2  3  4]
 [ 6  7  8]
 [10 11 12]]

The row argument’s of just the : character tells the Python interpreter to look over all rows. The column argument has 1:4, which tells the interpreter to extract between Column 1 (inclusive) and Column 4 (exclusive). As you can guess, you can set up elaborate indexing calls using this schema:

print("a:")
print(a)

# Recall the last two columns and rows
print("Last two columns and rows:")
print(a[1:3,2:4])

# Recall second and fourth columns
print("Second and fourth columns:")
print(a[:,(1,3)])

a:
[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]
Last two columns and rows:
[[ 7  8]
 [11 12]]
Second and fourth columns:
[[ 2  4]
 [ 6  8]
 [10 12]]

In general, this is the preferred route since it allows for more ways to extract out elements from an array.

13.7Data types stored in NumPy arrays¶

Earlier in this guide we covered the basic numeric types that are built into the standard library of Python: int (integers), float (floating point numbers), and complex (complex numbers). NumPy arrays (i.e., the numpy.ndarray class) can store these built-in data types as well as many more types, giving you more control over the precision of the numbers in your array. The .dtype attribute returns the data type of the objects being stored into a NumPy array:

a = np.array([1, 2, 3])
print("`a` data type:", type(a))
print("Data types stored in `a`:", a.dtype)

`a` data type: <class 'numpy.ndarray'>
Data types stored in `a`: int64

Here, the shell returns that our NumPy array is storing int64, which is shorthand for the numpy.int64 data type. This means that each entry is a 64-bit level integer. If not specified, the data type for numpy.array() is inferred from the elements that make up the list. Our data type in the above array np.int64 is because we passed a list entirely composed of integers to numpy.array(). If we reinitialize the array with a float object to our list, we should see a different data type for the array:

a = np.array([1, 2.2, 3])
print(a.dtype)

float64

Now, because our list includes a float, the inferred data type for the entire array is float64 (again, shorthand for numpy.float64), which means a 64-bit floating point number (a.k.a., double precision floating point). There are many more data types available in NumPy. NumPy’s scalars webpage has a good listing of all available data types. The command numpy.sctypeDict also returns a dict with all data types possible.

np.sctypeDict

Many of these data types should be familiar to people who have used C-style languages as NumPy uses the C programming language for many low-level numerical operations. Therefore, we would call NumPy as a C programming language “wrapper” for programming in Python.

Different NumPy data types will have different levels of precision and range. For instance, numpy.int8 (shorthand name is called byte) can represent an 8-bit integer between -128 to 127 (8-bit = 2⁸ = 256 distinct values), while numpy.int16 (also called short) can represent a 16-bit integer between -32,768 to 32,767 (16-bit = 2¹⁶ = 65,536 distinct values). Since the increase from numpy.int8 to numpy.int16 denotes the increased number of bits that are used to represent a number, more bits mean bigger or more precise numbers. The list of available standard C data types returned by numpy.sctypeDict can differ depending on your computer architecture (e.g., if your computer’s operating system is 32-bit or 64-bit).

13.7.1Array data type preservation¶

Earlier we showed that NumPy tries to pick the best data type for an array during initialization. However, you can experience some odd behavior if you try to mix data types to an existing array. For example, let’s create an all int array and then change one value to a float:

a = np.array([1, 2 ,3])
print(a.dtype)         # our numpy data type is inferred to be int64

a[0] = 11.5            # let's now replace an int with a float
print("a array:", a)

int64

a array: [11  2  3]

We can see that the float 11.5 became 11 since it was casted into a numpy.int64 data array. You should never assume the data type based on the NumPy function alone.

There are two common ways to fix this issue. The first way is to define the array’s data type during initialization. This is done using the dtype argument in numpy.array(). For example, if we know that at some point an array will need to contain float objects, we should set the array’s data type to some level of floating point precision during initialization:

# create array knowing that floats will be later used
a = np.array([1, 2 ,3], dtype=np.float64)
print(a.dtype)

# add the float
a[0] = 11.5
print("a array:", a)

float64
a array: [11.5  2.   3. ]

The other route is to create a copy of the starting array with the new data type using .astype() method and then modify the array. This assumes you know the data type you want to cast into:

a = np.array([1, 2, 3, 4], dtype=np.int64)
print("a before creating the copy:")
print(a)
print("\n")

b = a.astype(np.float64)
b[2] = 9.53

print("a after creating the copy:")
print(a)
print(a.dtype)
print("\n")

print("b after creating the copy:")
print(b)
print(b.dtype)

a before creating the copy:

[1 2 3 4]


a after creating the copy:
[1 2 3 4]
int64


b after creating the copy:
[1.   2.   9.53 4.  ]
float64

13.7.2Side note: NumPy arrays are for numbers only!...Usually!¶

NumPy arrays usually contain number-based objects (e.g., int, float). They normally cannot contain str objects. This seems obvious at first, but keep this in mind when we want to import data files as NumPy objects as we may need to strip metadata entries and header rows!

NumPy can handle arrays of various data types with Structured arrays. We won’t cover it here, but can be useful for data management needs.

13.8Other ways to create NumPy arrays¶

There are a number of ways to create NumPy arrays. Here are a few common routes:

13.8.1The array() function¶

As we have previously shown, the numpy.array() function is a quick way to make arrays. We simply include our sequence-based object (e.g., simple list or a nested list structure) and define a data type using the dtype argument:

a = np.array([[2.2, -3.3, 1.2], [5.3, -15.0, -72.5]], dtype=np.float64)

print(a)

[[  2.2  -3.3   1.2]
 [  5.3 -15.  -72.5]]

13.8.2The ones() function¶

Use numpy.ones() function allows us to create an array filled with ones:

# Create a three-dimensional array of all ones with shape (2, 3, 5)
a = np.ones((2, 3, 5))

print(a)

[[[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]

 [[1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]
  [1. 1. 1. 1. 1.]]]

Very useful when you need to quickly create a basic array structure that will have its values later replaced.

13.8.3The zeros() function¶

The numpy.zeroes() function is very similar to numpy.ones() except it fills the array with zeros:

# Create a two-dimensional array of all zeros with shape (3, 5)
a = np.zeros((3, 5))

print(a)

[[0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0.]]

Again, this function (as well as the other functions that will be listed below) allow you to quickly create basic arrays.

13.8.4The full() function¶

The numpy.full() function is similar to numpy.ones() and numpy.zeros() but allows you to choose the fill value:

# Create a two-dimensional, rectangular 4 x 6 array filled with 42
a = np.full((4, 6), fill_value=42)

print(a)

[[42 42 42 42 42 42]
 [42 42 42 42 42 42]
 [42 42 42 42 42 42]
 [42 42 42 42 42 42]]

13.8.5The eye() function¶

Use numpy.eye() to create an array with only the main diagonal value (or an offset diagonal) having value of 1. This style of array is often used in matrix theory.

# Create a two-dimensional, square 6 x 6 array with 1 on diagonal
a = np.eye(6)
print("a:")
print(a)
print("\n")

# Create a two-dimensional, square 6 x 6 array with 1 on the diagonal just
# right of center
b = np.eye(6, k=1)
print("b:")
print(b)
print("\n")

# Create a two-dimensional, square 4 x 6 array with 1 on diagonal
# N = number of rows, M = number of columns
c = np.eye(N=4,M=6)
print("c:")
print(c)

a:
[[1. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0.]
 [0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 1.]]


b:
[[0. 1. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0.]
 [0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 1.]
 [0. 0. 0. 0. 0. 0.]]


c:
[[1. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 1. 0. 0.]]

13.9Keeping memory in mind¶

So why not use numpy.float64 all the time since it can use 64 bits to represent very large integers and floats? The thing to keep in mind when creating NumPy arrays is the memory footprint of the data types. A modern desktop or laptop (circa 2026) has multiple gigabytes of random access memory (RAM; often colloquially called “computer memory”) available to store actively worked on data at one time. The amount of memory allocated for a NumPy array is dependent on both the total number of “elements” (i.e, the number of discrete data values) in the array and the data type of the array.

NumPy provides two useful array attributes to determine the total memory size for an array. The .size attribute reports the number of elements present in an array and the .itemsize attribute reports the number of bytes used for each element (set by the data type). Therefore, the total memory allocated for a NumPy array is array.size * array.itemsize.

The code block below demonstrates these concepts by calculating the amount of memory for a three element array made up of 64-bit floating point values:

a = np.array([1.234, 2.12, -2343.3])
print("data type:", a.dtype)
print("size:", a.size)
print("element size:", a.itemsize)
print("number of bytes:", a.size * a.itemsize)

data type: float64
size: 3
element size: 8
number of bytes: 24

In this example, a.size returns 3 as the number of elements in a and a.itemsize shows that each element makes up 8 bytes of RAM. This checks out, as np.float64 takes up 64 bits and each byte is made of 8 bits, so 8 bytes equals 64 bits. By multiplying the memory requirements of the data type by the number of elements we find that the memory footprint of the array is 24 bytes. While 24 bytes is not a big deal for a typical desktop (as of 2026) with 16 GB (i.e., 16,000,000,000 bytes) of RAM, but you can imagine as we increase the dimensions of our array the memory footprint can drastically increase. Let us run through the memory footprints of some data types for an array of shape (100, 100, 100):

a = np.zeros((100, 100, 100), dtype=bool)
print("number of bytes:", a.size * a.itemsize)

number of bytes: 1000000

A bool array of shape (100, 100, 100) takes up 1,000,000 bytes, or 1 MB of RAM.

a = np.zeros((100, 100, 100), dtype=np.int16)
print("number of bytes:", a.size * a.itemsize)

number of bytes: 2000000

A numpy.int16 array of shape (100, 100, 100) takes up 2,000,000 bytes, or 2 MB of RAM.

a = np.zeros((100, 100, 100), dtype=np.float32)
print("number of bytes:", a.size * a.itemsize)

number of bytes: 4000000

A numpy.float32 array of shape (100, 100, 100) takes up 4,000,000 bytes, or 4 MB of RAM.

a = np.zeros((100, 100, 100), dtype=np.float64)
print("number of bytes:", a.size * a.itemsize)

number of bytes: 8000000

A numpy.float64 array of shape (100, 100, 100) takes up 8,000,000 bytes, or 8 MB of RAM. NumPy arrays with the exact same shape can take different amount of memory depending on their data type. We should try to match the data type of our array to the actual type of data we are working with. For instance, if we are working on a True and False table, we should use the bool data type instead of numpy.float64. If we are working with floating point numbers, we can use numpy.float64. We do need to be mindful about the size and data type of the NumPy arrays we create. For example, if we create an 10000 x 10000 x 10000 element array containing zeros represented in the numpy.float64 format we find the memory allocation to be:

a = np.zeros((10000, 10000, 10000), dtype=np.float64)
print("number of bytes:", a.size * a.itemsize)

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
Cell In[24], line 1
----> 1 a = np.zeros((10000, 10000, 10000), dtype=np.float64)
      2 print("number of bytes:", a.size * a.itemsize)

MemoryError: Unable to allocate 7.28 TiB for an array with shape (10000, 10000, 10000) and data type float64

Unless you are on a massive supercomputer, you should have gotten a MemoryError exception when we tried to allocate multiple terabytes of memory (1 TB = 1000 GB = 1,000,000,000,000 bytes!) to create the NumPy array. This demonstrates how increasing the number of elements in an array can cause memory allocations problems.

13.10Initializing arrays¶

Recall that Python allows you to initialize an object with explicitly assigning it a value. Python in these cases uses a default value based on the data type. One way to initialize an NumPy array without knowing the actual value is create an array with unrealistic values and then substitute this values later. This is also useful for useful for debugging purposes. A common method is to fill an array with -1 values:

data = np.full(5, fill_value=-1, dtype=np.float64)
print(data)

[-1. -1. -1. -1. -1.]

Another route to initialize an array is to use the None data type. Recall that None is a special data class that is not equivalent to is not equivalent to 0, False, or an empty string. To use None in an NumPy array we first create an array using the full() function and set fill_value=None and dtype to our desired data type. We can later change these values without an incident! The code block below shows how to do this for an array that will store numpy.float64 objects:

# Create an array of known size but add None objects as placeholders
data = np.full(5, fill_value=None, dtype=np.float64)

print("array after initialization:")
print(data)
print(data.dtype)
print("\n")

# Now substitute the None objects
for i in range(data.size):
    data[i] = i

print("array after substitution:")
print(data)
print(data.dtype)

array after initialization:
[nan nan nan nan nan]
float64


array after substitution:
[0. 1. 2. 3. 4.]
float64

Using None in array initialization is very common and you will often see this if you frequently use the NumPy library.

13.10.1Example: An array of random trigonometry¶

A previous example showed how to take the cosine of a random number using the math and random libraries. Building off this example, create a two-dimensional array using NumPy that stores five random values between $0 - 2\pi$ as well as their cosine. Have the first column in each row contain the random value and the second column in each row contain the cosine of that value.

Solution:

In order to solve the problem, we utilize both the NumPy numpy.ones() function and the .shape attribute. See the code block below for details. In short, the numpy.ones() function is first used to create a 2D NumPy array called a that has five columns and two rows. The ones are used as an initializer and will soon be replaced. The for loop contains the code that creates the random value and the cosine of that value. The a.shape[0] command pulls the row size of a into range(). While we could have hard coded the for command with i in range(5), this would limit the use of the code block if we ever changed the size of a.

# Libraries
from random import random
from math import cos, pi
import numpy as np

# Create array with ones
a = np.ones((5,2))

# Populate with values
for i in range(a.shape[0]):
    a[i,0] = random() * (2 * pi)
    a[i,1] = cos(a[i,0])

# Display results
print(a)

[[ 6.21616621  0.99775506]
 [ 3.42284027 -0.9607099 ]
 [ 5.28694297  0.54346045]
 [ 5.41180326  0.64376959]
 [ 0.28679785  0.95915462]]

13.11Reshaping arrays¶

We can also change the “shape” (i.e., the size and / or dimension ) a NumPy array. For example, let us create a one dimensional (1D) array and reshape into having two dimensions (2D):

# 1D array with elements from 0 up to 12
a = np.arange(0, 12, dtype=np.float64)
print("a array:", a, sep="\n")

# Reshape to 4 x 3 array
a = a.reshape(4, 3)
print("a array reshaped:", a, sep="\n")

a array:
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11.]
a array reshaped:
[[ 0.  1.  2.]
 [ 3.  4.  5.]
 [ 6.  7.  8.]
 [ 9. 10. 11.]]

This example first uses the numpy.arange() function, which is similar to the Python standard library built-in range() function, to create a 1D NumPy array. This call for numpy.arange() includes a start point (0), an end point (12), and the data type (numpy.float64). There are a few other optional arguments that can be passed when creating an array. Then we reshape the 1D array to a 4 x 3 array using the .reshape() method which is part of any NumPy array object. This returns an array with the passed in shape, in this case (4,3).

The step argument allows you to also use non-integer steps, which range() could not do:

# Create a 1D array with floating point start, stop, and steps

b = np.arange(1.2, 14.7, step=0.5)
print(b)

[ 1.2  1.7  2.2  2.7  3.2  3.7  4.2  4.7  5.2  5.7  6.2  6.7  7.2  7.7
  8.2  8.7  9.2  9.7 10.2 10.7 11.2 11.7 12.2 12.7 13.2 13.7 14.2]

So now we also have a useful means to create arrays with uniform step size!

13.12Array copying¶

Creating new NumPy arrays based on existing arrays can lead to unexpected results if you are not careful. For example, let’s look at the code block below:

a = np.arange(0, 12, dtype=np.float64) # 1D array with elements from 0 up to 12
print("a array start:", a, sep="\n")

b = a.reshape(2, 6)                    # Reshape to 2 x 6 array
b[0,0] = 23

print("a array end:", a, sep="\n")
print("b array end:", b, sep="\n")

a array start:
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11.]
a array end:
[23.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11.]
b array end:
[[23.  1.  2.  3.  4.  5.]
 [ 6.  7.  8.  9. 10. 11.]]

At first glance this is odd! Turns out that NumPy does not create a new array if it is based on a previous array. Rather they are linked! To create a new array we need to run the .copy() method first and then modify the contents of the copy. This we place the new array in a different memory location so we no longer have this linking issue:

a = np.arange(0, 12, dtype=np.float64) # 1D array with elements from 0 up to 12
print("a array start:", a, sep="\n")

b = a.copy()                           # Create the copy
b = b.reshape(2, 6)                    # Reshape to 2 x 6 array
b[0,0] = 23

print("a array end:", a, sep="\n")
print("b array end:", b, sep="\n")

a array start:
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11.]
a array end:
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11.]
b array end:
[[23.  1.  2.  3.  4.  5.]
 [ 6.  7.  8.  9. 10. 11.]]

Forgetting to use the .copy() method is very common programming bug! Always check your starting and copied arrays to see if they are decoupled.

13.13Analyzing array metrics¶

Besides .reshape(), NumPy arrays have many other useful methods. NumPy provides numerous “helper” methods to assess an array. These include .max() that reports the largest element in an array, .min() that reports the smallest element in an array, .mean() that reports the average value of the array, .std() that reports the standard deviation value of the array, and .sum() that reports the sum of the array. You can use the optional axis argument into these methods to allow you to assess the metric along a particular direction of an array. The code block below demonstrates how to use these five methods:

a = np.arange(0, 12, dtype=np.float64).reshape(4, 3)
print("a array", a, sep="\n")

print("Max element:", a.max(), sep="\n")
print("Row with max element:", a.max(axis=0), sep="\n")

print("Min element:", a.min(), sep="\n")
print("Column with min element:", a.min(axis=1), sep="\n")

print("Average value or array:", a.mean(), sep="\n")

print("Standard deviation of array:", round(a.std(),1), sep="\n")

print("Total sum of array:", a.sum())

a array
[[ 0.  1.  2.]
 [ 3.  4.  5.]
 [ 6.  7.  8.]
 [ 9. 10. 11.]]
Max element:
11.0
Row with max element:
[ 9. 10. 11.]
Min element:
0.0
Column with min element:
[0. 3. 6. 9.]
Average value or array:
5.5
Standard deviation of array:
3.5
Total sum of array: 66.0

You can even apply them to specific rows and columns by using the MATLAB-based indexing schema from earlier!

print("a array", a, sep="\n")

print("Average value of second column:", a[:,1].mean(), sep="\n")

a array
[[ 0.  1.  2.]
 [ 3.  4.  5.]
 [ 6.  7.  8.]
 [ 9. 10. 11.]]
Average value of second column:
5.5

13.13.1Example: Analyzing data¶

The following file is an optical transmission dataset from a blue foil. The first column in the data file is the wavelength of light measured (units: nm) and the second column is the detected light intensity (units: counts). The data is tab-delimited and the first row of data is a header row that contains information about each column.

Load this data file into Python and determine the average wavelength, the maximum wavelength, the minimum wavelength, and the standard deviation of wavelengths used in the measurement. Report all values with two digits of precision past the decimal point. See the tip below about importing the file into Python.

Solution:

The importing of data can be tricky because you need to correctly link the file path and also ensure that Python is not trying to read any header rows. As seen in the code below, the arguments delimiter="\t" and skiprows=1 are needed in order to read the file.

From here it is reasonably straightforward. We first separate the data file into two separate arrays for ease of use, and then use the .mean(), .max(), .min(), and std() methods to get the average wavelength, maximum wavelength, minimum wavelength, and standard deviation of wavelengths, respectively. Comments have been added to the code for readability purposes.

# Load data
spectrum_data = np.loadtxt("./static/example-data/blue_foil_transmission_spectrum.txt",
                           delimiter="\t",
                           skiprows=1)

# Separate out data into two separate 1D arrays
wavelength = spectrum_data[:,0]
intensity = spectrum_data[:,1]

# Outputting values
print(f"The average wavelength is {wavelength.mean():.2f} nm.")
print(f"The maximum wavelength is {wavelength.max():.2f} nm.")
print(f"The minimum wavelength is {wavelength.min():.2f} nm.")
print(f"The standard deviation in the wavelengths is {wavelength.std():.2f} nm.")

The average wavelength is 653.60 nm.
The maximum wavelength is 1102.21 nm.
The minimum wavelength is 186.72 nm.
The standard deviation in the wavelengths is 264.66 nm.

13.14Mathematical operations¶

One key aspect of the NumPy library is its ability to perform mathematical operations on arrays. NumPy provides numerous scalar, elementwise, matrix, and linear algebra mathematical operations. Let’s go over a few examples for each of these operation classes.

13.14.1Scalar operations¶

Scalar-based operations are mathematical operations between a single number (e.g., $2$, $3.5$, $125.35$) and an array. In scalar-based operations, the scalar is operated upon each element in the array separately. The example below demonstrates these type of operations with a $3 \times 3$ array:

a = np.array([[1, 2, 3], [5, 6, 7], [8, 9, 10]], dtype=np.float64)

print("a:")
print(a)
print("\n")

print("a + 5:")
print(a + 5)
print("\n")

print("a - 4:")
print(a - 4)
print("\n")

print("a * 2:")
print(a * 2)
print("\n")

print("a / 3:")
print(a / 3)
print("\n")

a:
[[ 1.  2.  3.]
 [ 5.  6.  7.]
 [ 8.  9. 10.]]


a + 5:
[[ 6.  7.  8.]
 [10. 11. 12.]
 [13. 14. 15.]]


a - 4:
[[-3. -2. -1.]
 [ 1.  2.  3.]
 [ 4.  5.  6.]]


a * 2:
[[ 2.  4.  6.]
 [10. 12. 14.]
 [16. 18. 20.]]


a / 3:
[[0.33333333 0.66666667 1.        ]
 [1.66666667 2.         2.33333333]
 [2.66666667 3.         3.33333333]]

Be aware that if you are reassigning elements in an array based on mathematical operations, the result will be casted to match the array’s data type:

a = np.array((5, 11))
print("a[0] = a[0] / 2")
a[0] = a[0] / 2
print("array a")
print(a)
print("data type for elements in a:", a.dtype)

a[0] = a[0] / 2
array a
[ 2 11]
data type for elements in a: int64

We can see that a[0] = a[0] / 2 results in a[0] becoming 2 instead of 2.5. That’s because the data type of the values in a is numpy.int64, which was inferred from the list (5,11). So assigning a[0] is similar to casting the result as an integer (i.e., the equivalent command: int(5 / 2)). This can be avoided by declaring our array to be of type numpy.float64 if we are expecting the array to contain floats in the future.

13.14.2Elementwise operations¶

NumPy also performs elementwise operations (i.e., mathematical operations between corresponding elements in two different arrays) using the same mathematical operators used in scalar operations. The example below shows elementwise operations on two $3 \times 3$ two-dimensional arrays:

a = np.array([[1, 2, 3], [5, 6, 7], [8, 9, 10]], dtype=np.float64)
b = 2 * np.ones((3, 3), dtype=np.float64)
print("a")
print(a)
print("\n")

print("b")
print(b)
print("\n")

print("a + b:")
print(a + b)
print("\n")

print("a * b:")
print(a * b)
print("\n")

print("a / b:")
print(a / b)

a
[[ 1.  2.  3.]
 [ 5.  6.  7.]
 [ 8.  9. 10.]]


b
[[2. 2. 2.]
 [2. 2. 2.]
 [2. 2. 2.]]


a + b:
[[ 3.  4.  5.]
 [ 7.  8.  9.]
 [10. 11. 12.]]


a * b:
[[ 2.  4.  6.]
 [10. 12. 14.]
 [16. 18. 20.]]


a / b:
[[0.5 1.  1.5]
 [2.5 3.  3.5]
 [4.  4.5 5. ]]

It is important to note that these operations are elementwise and are not the traditional matrix mathematical operations that one sees in linear algebra courses (e.g., matrix multiplication, dot product, etc.). NumPy has these operations as well, and we will cover them shortly!

NumPy can also attempt elementwise operations on certain arrays that have different dimensions. For example, the following code block below shows an elementwise operation between a $3 \times 3$ array and a $1 \times 3$ array:

a = np.array([[1, 2, 3], [5, 6, 7], [8, 9, 10]], dtype=np.float64)
b = 3 * np.array([1, 2, 3], dtype=np.float64)

print("a")
print(a)

print("b")
print(b)

print("a * b:")
print(a * b)

a
[[ 1.  2.  3.]
 [ 5.  6.  7.]
 [ 8.  9. 10.]]
b
[3. 6. 9.]
a * b:
[[ 3. 12. 27.]
 [15. 36. 63.]
 [24. 54. 90.]]

As seen above, NumPy appears to have “looped” the b array three times over in order to complete the elementwise operations needed. NumPy calls this broadcasting and is very convenient as it prevents the need to completely reshape arrays for simple elementwise operations. However, there are limits to what NumPy can do for broadcasting and a failed broadcasting attempt results in a ValueError: operands could not be broadcast together exception.

13.14.3Matrix operations¶

NumPy also provides the matrix mathematical operations you would expect to use when dealing with matrices. For example, matrix multiplication is handled using the numpy.matmul() function. Below shows an example of matrix multiplication between two, $2 \times 2$ matrices:

a = np.array([[2, 3], [3, 4]])
b = np.array([[6, 7], [8, 9]])

print("Matrix multiplication of a & b:")
print(np.matmul(a, b))

Matrix multiplication of a & b:
[[36 41]
 [50 57]]

All normal matrix multiplication rules (e.g., proper matching of dimensions of the arrays) apply. As a shorthand to numpy.matmul(), one can also use the @ operator as a shorthand to numpy.matmul():

a = np.array([[2, 3],[5, 6]], dtype=np.float64)
b = np.array([[1, 4],[2, 1]], dtype=np.float64)

print("a @ b")
print(a @ b)

a @ b
[[ 8. 11.]
 [17. 26.]]

We can also take the dot product between two vectors (i.e., two one-dimensional arrays) using the numpy.dot() function:

a = np.array([2.2, 3.4, 10.3])
b = np.array([1.4, 0.2, 15.7])

print("Dot product of a and b:")
print(np.dot(a, b))

Dot product of a and b:
165.47

NumPy also provides a .dot() method to calculate the dot product as well:

a = np.array([2, 3], dtype=np.float64)
b = np.array([5, 6], dtype=np.float64)

print("Dot product of a and b using method route:")
print(a.dot(b))

Dot product of a and b using method route:
28.0

The np.dot() function is also extended to multidimensional arrays and will calculate different values depending on the shape of the arrays being operated on. It is highly recommended to read the documentation for details.

NumPy provides a few different ways to transpose an array. One can use the numpy.transpose() function, the NumPy .transpose() array method, or the NumPy .T array attribute to transpose a data array. The code block below demonstrates all three methods:

a = np.array([[5, 6], [7, 8], [9, 10]])

print("a:")
print(a)
print("\n")

print("Using transpose():")
print(np.transpose(a))
print("\n")

print("Using .transpose():")
print(a.transpose())
print("\n")

print("Using .T:")
print(a.T)

a:

[[ 5  6]
 [ 7  8]
 [ 9 10]]


Using transpose():
[[ 5  7  9]
 [ 6  8 10]]


Using .transpose():
[[ 5  7  9]
 [ 6  8 10]]


Using .T:
[[ 5  7  9]
 [ 6  8 10]]

All three routes give the same correct answer. Both np.transpose() and .transpose() allow for an additional axis argument to alter how the transpose operation works. The transpose of an array is a very important operation not only for linear algebra using matrices but also when plotting data. We will revisit transposing and plotting in an upcoming lesson about plotting using Python.

13.14.4Linear algebra operations¶

In addition to providing many important scalar, elementwise, and matrix operations, NumPy also has an extensive list of linear algebra operations. While these operations are more advanced than the scope of this document, you can explore these operations in NumPy’s linear algebra documentation page.

13.15Importing data with NumPy¶

As we have shown in an earlier lesson, NumPy has a powerful data importer function called numpy.loadtxt(). We can load data directly into a NumPy array, which is very convenient as we will often using NumPy to process datasets. The code block below goes over how to load the file tutorial_test.csv into an NumPy array:

data = np.loadtxt("./static/example-data/tutorial_test.csv", delimiter=",", skiprows=1)
print(data)

[[ 0.     9.015]
 [ 1.     9.172]
 [ 2.     4.982]
 [ 3.    16.713]
 [ 4.    16.632]
 [ 5.     7.481]
 [ 6.     9.703]
 [ 7.    10.342]
 [ 8.     7.583]
 [ 9.    11.178]
 [10.     8.913]
 [11.    11.436]
 [12.     7.247]
 [13.     3.156]
 [14.    12.616]
 [15.     7.194]
 [16.     8.059]
 [17.     6.536]
 [18.    10.56 ]
 [19.    10.876]]

The delimiter="," argument tells Python that each value in row of text is separated by a , character and the skiprows=1 allows us to skip the header row in the data file. Overall, a very easy process to quickly load data.

13.16Exporting data with NumPy¶

NumPy can also write data to files. The function numpy.savetxt() is a great, general purpose writing command. To see how this works, let’s first create some data:

from random import random
from math import cos
import csv

# Initialize numpy array to store values (Note the `-1.0` to force floats #s)
data = np.full((6,3), fill_value=-1.0)

# Populate array
for idx in range(len(data)):
    data[idx,0] = idx
    data[idx,1] = random() * 10
    data[idx,2] = cos(data[idx,1])

print(data)

[[ 0.          8.260544   -0.39545425]
 [ 1.          7.75916626  0.09467338]
 [ 2.          4.58233269 -0.12968996]
 [ 3.          0.17550669  0.98463819]
 [ 4.          4.85360088  0.14074306]
 [ 5.          1.94297364 -0.36364454]]

Here, we have created a $6 \times 3$ two-dimensional NumPy array that stores three columns of data: an index value, a random number that is multiplied by $10$, and the cosine of that random number. Let’s now save the data to a CSV file:

output_file = "./output.csv"

# Write data to a CSV file
with open(output_file, "w") as file:
    np.savetxt(file, data, delimiter=",")

# Create list to re-read the data
data_return = []

# Read file
print("file contents:")
with open(output_file, "r") as file:
    print(file.read())

file contents:
0.000000000000000000e+00,8.260544000700146938e+00,-3.954542538916032690e-01
1.000000000000000000e+00,7.759166255417739855e+00,9.467337805362498193e-02
2.000000000000000000e+00,4.582332688091408812e+00,-1.296899597071989041e-01
3.000000000000000000e+00,1.755066897030910233e-01,9.846381937015521446e-01
4.000000000000000000e+00,4.853600882873271516e+00,1.407430571065204916e-01
5.000000000000000000e+00,1.942973641972793875e+00,-3.636445436863542935e-01

The code block above uses the with open() as file: statement from earlier combined with the newly introduced numpy.savetxt() function. We also issue the delimiter="," argument in numpy.savetxt() since out output file will be a CSV file. Before issuing this command we first created a variable called output_file that defines the file path of the file we will write to. The second half the code block is not required but does prove that we can then open this newly written file and display its contents.

There are many other optional arguments that make this function useful. For example we can specify a header row using the header argument:

output_file = "./output.csv"

header_row = "index,random,cosine value"

# Writing data to CSV file
with open(output_file, "w") as file:
    np.savetxt(file, data, delimiter=",", header=header_row)

# Create list to restore data
data_return = []

# Read file
print("file contents:")
with open(output_file, "r") as file:
    print(file.read())

file contents:
# index,random,cosine value
0.000000000000000000e+00,8.260544000700146938e+00,-3.954542538916032690e-01
1.000000000000000000e+00,7.759166255417739855e+00,9.467337805362498193e-02
2.000000000000000000e+00,4.582332688091408812e+00,-1.296899597071989041e-01
3.000000000000000000e+00,1.755066897030910233e-01,9.846381937015521446e-01
4.000000000000000000e+00,4.853600882873271516e+00,1.407430571065204916e-01
5.000000000000000000e+00,1.942973641972793875e+00,-3.636445436863542935e-01

Adding a header is great for readiblity, but we can see that NumPy added a # at the start of the header row. This is because the header row is considered a comment in NumPy. We can include the argument comments="" to remove this character:

output_file = "./output.csv"

header_row = "index,random,cosine value"

# Writing data to CSV file
with open(output_file, "w") as file:
    np.savetxt(file, data, delimiter=",", header=header_row, comments="")

# Create list to restore data
data_return = []

# Read file
print("file contents:")
with open(output_file, "r") as file:
    print(file.read())

file contents:

index,random,cosine value
0.000000000000000000e+00,8.260544000700146938e+00,-3.954542538916032690e-01
1.000000000000000000e+00,7.759166255417739855e+00,9.467337805362498193e-02
2.000000000000000000e+00,4.582332688091408812e+00,-1.296899597071989041e-01
3.000000000000000000e+00,1.755066897030910233e-01,9.846381937015521446e-01
4.000000000000000000e+00,4.853600882873271516e+00,1.407430571065204916e-01
5.000000000000000000e+00,1.942973641972793875e+00,-3.636445436863542935e-01

We can even format the values sent to the file. If we want to format everything in scientific notation with 4 digits past the decimal point we would add the argument fmt="%.4e" to numpy.savetxt(), which is shown below:

output_file = "./output.csv"

header_row = "index,random,cosine value"

# Writing data to CSV file
with open(output_file, "w") as file:
    np.savetxt(file,
               data,
               delimiter=",",
               header=header_row,
               comments="",
               fmt="%.4e")

# Read file
print("file contents:")
with open(output_file, "r") as file:
    print(file.read())

file contents:
index,random,cosine value
0.0000e+00,8.2605e+00,-3.9545e-01
1.0000e+00,7.7592e+00,9.4673e-02
2.0000e+00,4.5823e+00,-1.2969e-01
3.0000e+00,1.7551e-01,9.8464e-01
4.0000e+00,4.8536e+00,1.4074e-01
5.0000e+00,1.9430e+00,-3.6364e-01

We can even format individual columns by passing a list into the fmt argument:

output_file = "./output.csv"

header_row = "index,random,cosine value"
fmt_table = ["%d", "%.4e", "%.3f"]

# Writing data to CSV file
with open(output_file, "w") as file:
    np.savetxt(file,
               data,
               delimiter=",",
               header=header_row,
               comments="",
               fmt=fmt_table)

# Read file
print("file contents:")
with open(output_file, "r") as file:
    print(file.read())

file contents:
index,random,cosine value
0,8.2605e+00,-0.395
1,7.7592e+00,0.095
2,4.5823e+00,-0.130
3,1.7551e-01,0.985
4,4.8536e+00,0.141
5,1.9430e+00,-0.364

The fmt_table variable in the code block above formats the first column values to be integers ("%d"), the second column values to be in scientific notation with four values displayed after the decimal point ("%.4e"), and the third column values to be floating point values with three values displayed after the decimal point ("%.3f").

One downside with numpy.savetxt() is that is limited to only one header row. We can get around this limitation by also including our two previously discussed writing methods: (1) Using the .write() method and (2) using the csv library. The code block below demonstrates how both writing methods can be combined with numpy.savetxt() to create a well documented data file:

output_file = "./output.csv"

header_row = "index,random,cosine value"
fmt_table = ["%d", "%.4e", "%.3f"]

# Write method 1: Using the write()
with open(output_file, "w") as file:
   # write header rows
   file.write("Hello!")
   file.write("\n")
   file.write("This is a file!")
   file.write("\n")

# Write method 2: Using the csv library
with open(output_file, "a", newline="") as file:
    # create writer object to do the writing
    writer = csv.writer(file)
    writer.writerow(["UMN", "CEMS"])
    writer.writerow([1, 2, 3])

# Write method 3: Using np.savetxt()
with open(output_file, "a") as file:
    np.savetxt(file,
               data,
               delimiter=",",
               header=header_row,
               comments="",
               fmt=fmt_table)

# Read file
print("file contents:")
with open(output_file, "r") as file:
    print(file.read())

file contents:
Hello!
This is a file!
UMN,CEMS
1,2,3
index,random,cosine value
0,8.2605e+00,-0.395
1,7.7592e+00,0.095
2,4.5823e+00,-0.130
3,1.7551e-01,0.985
4,4.8536e+00,0.141
5,1.9430e+00,-0.364

Overall, numpy.savetxt() is a very powerful function as it helps us build organized and standardized output files.

13.17Final thoughts¶

The NumPy library is an extremely powerful and popular Python library. For scientific Python applications, NumPy is often the go-to library of choice and is utilized in many other libraries. The NumPy libary is large and has many comannds available to create and modify data arrays. We highly recommended that you spend some time further exploring this versatile library.