Lecture 2

The objectives of this lecture are:

  1. Indexing, slicing and iterating over NumPy arrays.
  2. "Shallow" and "deep" copies of arrays.
  3. Manipulating array shape -- reshaping.

Much of the lecture is based on this tutorial: http://wiki.scipy.org/Tentative_NumPy_Tutorial

Indexing, Slicing and Iterating

As with the list object and other collections in Python, 1D arrays can be indexed, sliced, and iterated over. The same indexing paradigm is used, the first entry has index 0 and last entry index len-1,

In [2]:
from numpy import *

A = arange(10)

# the 3rd entry in the array
print(A)
A[0]
[0 1 2 3 4 5 6 7 8 9]
Out[2]:
0
In [3]:
# slicing the array into a sub-array from the 3rd element up to (but not including) the
# 6th element
A[2:5]
Out[3]:
array([2, 3, 4])
In [4]:
# slicing the array into a sub-array from the 1st to 7th element, incrementing by 2 elements
A[0:6:2]
Out[4]:
array([0, 2, 4])
In [5]:
# slicing the array in reverse
A[6:0:-1]
Out[5]:
array([6, 5, 4, 3, 2, 1])

Omitting one or both of the index values results in NumPy inferring to start/end at the beginning or end of the array. This is quite convenient in that you do not need to explicitly know the length of the array.

In [6]:
A[:6] # NumPy infers A[0:6]
Out[6]:
array([0, 1, 2, 3, 4, 5])
In [7]:
A[2:] # NumPy infers A[2:10]
Out[7]:
array([2, 3, 4, 5, 6, 7, 8, 9])
In [8]:
A[::2] # NumPy infers A[0:10:2]
Out[8]:
array([0, 2, 4, 6, 8])

The syntax for indexing into multidimensional arrays is not equivalent to indexing into nested listed or other nested sequences. These indexing operations require a tuple separated by commas of the index value in each axis:

In [11]:
B = arange(15).reshape(3,5)

B
Out[11]:
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
In [12]:
B[(1,4)] # index into array to find the value of the 2nd row 5th column
B[1,4] # equivalent but more compact syntax
Out[12]:
9
In [13]:
B[:, 1] # slice corresponding to each row, second column of B
Out[13]:
array([ 1,  6, 11])
In [14]:
B[1:3,:] # each column in the second and third row of B
Out[14]:
array([[ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])

Thus all of the previous syntax may be used in multidimensional array, but must be specified for each axis. When the slice contains index expressions for fewer indices than are present in the array, NumPy infers that the the missing indices are complete slices of the remaining axes:

In [16]:
B[1] # the second row, NumPy infers B[1,:]
B[-1] # an equivalent expression since the second row is the last row
Out[16]:
array([10, 11, 12, 13, 14])

There are many ways to iterate over multidimensional arrays, but the best way is through elementwise operations as we have already learned. Explicit iteration using for loops should be avoided due to performance issues,

In [17]:
import time

A = linspace(0., 1., 1000).reshape(10,100)

# square every element in an array in different ways

start = time.clock()
A = A**2 # create a new array and assign values of A**2, then reassign A to new array, fast but not ideal
print(time.clock() - start, 'sec')

start = time.clock()
A **= 2 # performs augmented assignment, fastest approach
print(time.clock() - start, 'sec')

# use nested for loops to iterate over each element and reassign, slowest
start = time.clock()
(m, n) = A.shape
for i in range(m):
    for j in range(n):
        A[i,j] **= 2
    
print(time.clock() - start, 'sec')
0.00019199999999996997 sec
0.00015300000000006975 sec
0.0010149999999999881 sec

The difference between using augmented assignment versus not is non-negligible, but small. The difference between using NumPy arithmetic operations and standard Python for loops, however, is an order of magnitude. As the size of the array increases this difference in efficiency will increase substantially; try increasing the array size to 10000...

The reason for this is that most NumPy code is written in a low-level language, typically C, and not Python. The overhead of the interpreter is quite substantial, thus using a high-level language such as Python to "manage" low-level code pays significant dividends!

Copies and Views

One of the most significant differences between other high-level computing languages (MATLAB, Octave, SciLab, etc) and Python (using NumPy) is that Python+NumPy allows greater control over whether or not an arithmetic operation creates a new array or uses an existing one. This is extremely important when dealing with very large data-sets which motivates many scientists and engineers to use NumPy, but there is additional complexity in writing programs using NumPy because of this.

This attribute of NumPy is almost always a source of confusion for beginners. There are three different possible scenerios that involve copying arrays:

No Copy at All

Normal assignments do not result in the creation of a new array! Instead a new variable is created that references the same array object as the first,

In [ ]:
A = arange(12)

B = A            # no new object is created

B is A           # A and B are two variables that reference the same ndarray object
In [ ]:
B[3] = -1. # changes the value of the 4 element of B, what will happen to A[3]?

print(B[3], A[3])
In [ ]:
A = ones((2, 3)) # what happens to the array referenced by B?

print(A)
print(8*'-')
print(B)

As long as you are aware of this behaviour, it will not cause too much confusion. In fact, we see in the next section that being able to reference the same array in different ways is extremely convenient!

Views or Shallow Copy

In the previous case, creating a new reference to the same data has some limitations. One of the main limitations is this method does not allow us to view the same array data in a different array shape or view a sub-array of an array.

In [ ]:
A = arange(12)

B = A            # no new object is created, A and B refer to the same array object

B.shape = (3,4) # this reassigns the shape of the array that A also references

print (B is A)
print(8*'-')
print(A)
print(8*'-')
print(B)

Thankfully, we may "view" array data in different ways using the view() method. The view method creates a new array object that looks at the same data and many array methods return arrays that are views of the same array data.

In [ ]:
C = A.view() # create a *new* array that references the same data

C is A
In [ ]:
C.base is A # the `base` of a NumPy array is the array which originally refered to the data
In [ ]:
C.shape = (2, 6) # A's shape doesn't change, they are not the same object anymore

print(A.shape)
In [ ]:
C[0,4] = 1234 # A's data changes!

A

Now we have the basis to truly understand the results of many of the array manipulations we have already been using. For example, slicing an array returns a view of that array,

In [ ]:
S = A[:, 1:3] # create a view of a sub-array of A, the second through fourth element of all rows of A
S[:] = 10 # assign the value 10 to all elements in the view

print(S)
print(8*'-')
print(A)

Using the reshape() method returns a view of the array data with a different shape,

In [ ]:
D = A.reshape(2,6) # the shape of A is not affected

print(A.shape)
print(D.shape)

One frequently used view of an array has its own special method, flatten(), which returns a 1D view of the array,

In [ ]:
E = A.flatten()

print(A)
print(8*'-')
print(E)

Deep Copy

If you want a true copy of an array that creates a new object and data, this is called a "deep" copy and can be accomplished with the copy() method of the source array,

In [ ]:
F = A.copy() # a new array object with new data (with the same values) is created
F is A
In [ ]:
F.base is A # even though the values are equal, these are completely different arrays
In [ ]:
F[0, 0] = -1 # has no affect on the values of A

print(A)
print(8*'-')
print(F)

Shape Manipulation

At this point, it should be clear that the shape of an array is a tuple with the number of elements along each axis. Let's finish-up this lecture with methods for changing the shape of an array, as opposed to creating views of the array with different shapes.

In [ ]:
A = floor(10*random.random((3,4)))
print(A, A.shape)

The shape of an array can be changed by changing the value of its shape object,

In [ ]:
A.shape = (2, 6) # flatten a multidimensional array

print(A, A.shape)

Almost all other methods of an array that manipulate its shape return views,

In [ ]:
print(A.ravel())
print(A.transpose())
print(A.flatten())

print(A.shape) # the shape is unaffected

The only remaining method that does not return a view is resize, which can be used to both modify the shape of the array (no view) and/or resize the array data,

In [ ]:
A.resize((3, 4)) # no view is returned, this is equivalent to `A.shape = (3, 4)`
print(A)
In [ ]:
A.resize((4, 4)) # the array data is resized *in-place* (if possible) and additional array entries are initialized to 0
print(A)
print(8*'-')
A.resize((4, 1)) # the array data is resized *in-place* (if possible) and additional array entries are initialized to 0
print(A)