Iterators and Generators

Learn how iterators and generators work in Python, understand the iterator protocol, and generate values efficiently using lazy evaluation.

An iterator is an object that returns one value at a time from a collection, while a generator is a special type of iterator created using the yield keyword. They provide a memory-efficient way to process data without loading everything into memory at once.

Learning Objectives

After completing this lesson, you will be able to:

• Understand iterables, iterators, and generators.
• Create iterators using iter() and next().
• Build custom iterators.
• Create generators using yield.
• Differentiate between yield and return.
• Create generator expressions.
• Compare iterators and generators.
• Identify real-world use cases of generators.

What is an Iterator?

An iterator is an object that returns one element at a time from a collection. It remembers its current position and produces the next value only when requested.

Python uses iterators internally whenever you iterate over a collection using a for loop.

Iterator Protocol

An iterator implements the following special methods:

• __iter__() – Returns the iterator object.
• __next__() – Returns the next element.

When no more elements are available, __next__() raises a StopIteration exception.

Creating an Iterator

Use the iter() function to create an iterator from an iterable.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(iterator)

Output

<list_iterator object at 0x...>

Retrieving Values

Use the next() function to retrieve values from an iterator.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))
print(next(iterator))

Output

10
20
30

StopIteration

Once all elements are consumed, calling next() again raises a StopIteration exception.

numbers = [10]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))

Practice

Exercise 1

Predict the output.

numbers = [100, 200, 300]

it = iter(numbers)

print(next(it))
print(next(it))

Solution

100
200

The iterator returns one element at a time and remembers its current position.

Exercise 2

Predict the output.

name = "Python"

it = iter(name)

print(next(it))
print(next(it))
print(next(it))

Solution

P
y
t

Strings are iterable objects, so they can be converted into iterators using iter().

Exercise 3

What exception will be raised by the following code?

numbers = [1]

it = iter(numbers)

print(next(it))
print(next(it))

Solution

A StopIteration exception is raised because the iterator has no more elements to return.

Creating a Custom Iterator

You can create your own iterator by implementing the __iter__() and __next__() methods.

• __iter__() returns the iterator object.
• __next__() returns the next value.
• When all values are consumed, __next__() raises a StopIteration exception.

Example

class Counter:

    def __init__(self, limit):
        self.current = 1
        self.limit = limit

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.limit:
            raise StopIteration

        value = self.current
        self.current += 1
        return value


counter = Counter(5)

for number in counter:
    print(number)

Output

1
2
3
4
5

How It Works

1. The Counter object is created.
2. The for loop calls __iter__() to obtain the iterator.
3. The loop repeatedly calls __next__().
4. Each call returns the next value.
5. When the limit is reached, StopIteration is raised, ending the loop.

Practice

Exercise 1

Predict the output.

counter = Counter(3)

for value in counter:
    print(value)

Solution

1
2
3

The iterator returns values from 1 to 3 and then raises StopIteration.

Exercise 2

What happens if raise StopIteration is removed from the __next__() method?

Solution

The iterator will never indicate that it has finished, causing the loop to continue indefinitely or resulting in incorrect behavior.

Exercise 3

Which two special methods must every custom iterator implement?

Solution

Every custom iterator must implement:

• __iter__()
• __next__()

What is a Generator?

A generator is a special type of iterator created using a function that contains the yield keyword. Unlike a normal function that returns all values at once, a generator produces one value at a time and automatically remembers its execution state.

Generators are easier to write than custom iterators because Python automatically implements the iterator protocol for you.

Creating a Generator

A function becomes a generator as soon as it contains a yield statement.

def numbers():
    yield 1
    yield 2
    yield 3


gen = numbers()

print(gen)

Output

<generator object numbers at 0x...>

Notice that calling the function does not execute it immediately. Instead, it returns a generator object.

Using next() with a Generator

The next() function starts the generator and retrieves one value at a time.

def numbers():
    print("Starting")

    yield 1
    yield 2
    yield 3

    print("Ending")


gen = numbers()

print(next(gen))
print(next(gen))
print(next(gen))

Output

Starting
1
2
3

The "Ending" message is not printed because the generator pauses after the third yield. It executes the remaining statements only when resumed again.

Using a Generator with a for Loop

Generators can be directly used in a for loop.

def numbers():
    yield 1
    yield 2
    yield 3


for number in numbers():
    print(number)

Output

1
2
3

The for loop automatically calls next() until the generator raises StopIteration.

Practice

Exercise 1

Predict the output.

def demo():
    yield 10
    yield 20

g = demo()

print(next(g))
print(next(g))

Solution

10
20

Each call to next() returns the next value produced by the generator.

Exercise 2

Predict the output.

def greet():
    print("Hello")
    yield 1

g = greet()

print("Generator Created")

Solution

Generator Created

Calling a generator function does not execute its body immediately. It simply creates a generator object. The "Hello" message is printed only when the generator starts executing (for example, by calling next(g) or iterating over it).

Exercise 3

What is the output?

def values():
    yield "A"
    yield "B"

for value in values():
    print(value)

Solution

A
B

The for loop automatically retrieves values from the generator until it is exhausted.

Understanding yield

The yield keyword is used to produce a value from a generator. Unlike return, which terminates a function, yield pauses the function and preserves its current state. The next time the generator is resumed, execution continues from the statement immediately after the previous yield.

yield vs return

return	yield
Terminates the function	Pauses the function
Returns a single value	Produces one value at a time
Function cannot resume	Function resumes from the same point
Used in normal functions	Used in generator functions

Execution Flow

def greet():

    print("Step 1")
    yield "Hello"

    print("Step 2")
    yield "World"

    print("Step 3")


g = greet()

print(next(g))
print(next(g))
next(g)

Output

Step 1
Hello
Step 2
World
Step 3
Traceback (most recent call last):
...
StopIteration

Notice that the function resumes exactly where it paused after each yield.

State Preservation

One of the biggest advantages of generators is that they automatically preserve the values of local variables.

def counter():

    count = 1

    while count <= 3:
        yield count
        count += 1


g = counter()

print(next(g))
print(next(g))
print(next(g))

Output

1
2
3

The variable count is not reinitialized each time. Its value is preserved between successive calls to next().

Multiple yield Statements

A generator can contain multiple yield statements.

def colors():

    yield "Red"
    yield "Green"
    yield "Blue"


for color in colors():
    print(color)

Output

Red
Green
Blue

Each yield produces one value before the generator pauses.

Practice

Exercise 1

Predict the output.

def demo():

    print("A")
    yield 10

    print("B")
    yield 20

    print("C")


g = demo()

print(next(g))
print(next(g))

Solution

Output

A
10
B
20

The generator pauses after each yield. Since the generator is not resumed again, "C" is not printed.

Exercise 2

Predict the output.

def test():

    x = 5
    yield x

    x += 5
    yield x


g = test()

print(next(g))
print(next(g))

Solution

Output

5
10

The value of x is preserved between the two yield statements.

Exercise 3

What is the main difference between return and yield?

Solution

• return terminates the function and returns a value.
• yield pauses the function, returns a value, preserves its state, and resumes execution when requested again.

Generator Expressions

A generator expression provides a concise way to create generators. It is similar to a list comprehension but uses parentheses () instead of square brackets [].

Generator expressions generate values only when required, making them memory efficient.

Syntax

(expression for item in iterable)

Example

squares = (x * x for x in range(5))

for square in squares:
    print(square)

Output

0
1
4
9
16

Generator Expression vs List Comprehension

# List Comprehension
numbers = [x * x for x in range(5)]

# Generator Expression
numbers = (x * x for x in range(5))

• A list comprehension stores all values in memory.
• A generator expression generates values one at a time.

Practice

Exercise 1

Predict the output.

gen = (x + 1 for x in range(3))

print(next(gen))
print(next(gen))

Solution

1
2

Each call to next() computes the next value in the sequence.

Exercise 2

Which symbol is used to create a generator expression?

Solution

Generator expressions use parentheses (), whereas list comprehensions use square brackets [].

---

Infinite Generators

Generators can produce infinite sequences because values are generated only when requested.

Example

def even_numbers():

    number = 0

    while True:
        yield number
        number += 2


gen = even_numbers()

for _ in range(5):
    print(next(gen))

Output

0
2
4
6
8

Practice

Exercise 1

Predict the output.

def numbers():

    n = 1

    while True:
        yield n
        n += 1


gen = numbers()

print(next(gen))
print(next(gen))
print(next(gen))

Solution

1
2
3

The generator can continue producing values indefinitely.

---

Fibonacci Generator

Generators are commonly used to generate mathematical sequences.

Example

def fibonacci(limit):

    a, b = 0, 1

    while a <= limit:
        yield a
        a, b = b, a + b


for number in fibonacci(20):
    print(number)

Output

0
1
1
2
3
5
8
13
21

Practice

Exercise 1

What is the first value produced by the generator?

Solution

The first value is 0, because the generator yields a before updating its value.

---

Memory Efficiency

One of the biggest advantages of generators is memory efficiency.

List Example

numbers = [x for x in range(1000000)]

The above statement creates one million values in memory.

Generator Example

numbers = (x for x in range(1000000))

The generator creates only one value at a time, significantly reducing memory usage.

When to Use Generators

Use generators when:

• Working with large datasets.
• Reading large files.
• Processing streaming data.
• Producing values on demand.
• Creating infinite sequences.

Practice

Exercise 1

Which consumes less memory?

[x for x in range(1000000)]

or

(x for x in range(1000000))

Solution

The generator expression consumes significantly less memory because values are generated only when needed.

---

Iterator vs Generator

Feature	Iterator	Generator
Created Using	Class	Function
Implements	__iter__() and __next__()	yield
State Management	Manual	Automatic
Code Size	More	Less
Lazy Evaluation	Yes	Yes
Memory Efficient	Yes	Yes
Best Use Case	Custom iteration logic	Sequential data generation

Remember: Every generator is an iterator, but not every iterator is a generator.

---

Iterable vs Iterator vs Generator

                Iterable
       (list, tuple, set, dict)
                   │
             iter(iterable)
                   │
                   ▼
               Iterator
       (__iter__ + __next__)
                   ▲
                   │
         Generator Object
        (Created using yield)

• Iterable → An object that can produce an iterator.
• Iterator → Produces one value at a time.
• Generator → A special iterator created using the yield keyword.

---

Real-World Applications

Generators are commonly used for:

• Reading large files line by line.
• Processing large datasets.
• Streaming data from APIs.
• Log processing.
• Data pipelines.
• Machine learning workflows.
• Infinite sequences.

Example: Reading a File

def read_file(filename):

    with open(filename, "r") as file:
        for line in file:
            yield line.strip()


for line in read_file("data.txt"):
    print(line)

Instead of loading the entire file into memory, one line is processed at a time.

---

Key Takeaways

• An iterable is an object that can produce an iterator.
• An iterator returns one value at a time using next().
• A generator is a simpler way to create an iterator using yield.
• The yield keyword pauses execution and preserves the function’s state.
• Generator expressions provide a concise syntax for creating generators.
• Generators are ideal for processing large datasets because they use lazy evaluation.
• Every generator is an iterator, but not every iterator is a generator.

---

Check Your Understanding

Question 1

What is the purpose of the iter() function?

Solution

The iter() function converts an iterable into an iterator.

Question 2

Which special methods make an object an iterator?

Solution

__iter__() and __next__()

Question 3

What is the purpose of the yield keyword?

Solution

The yield keyword pauses a generator, returns a value, preserves its state, and resumes execution from the same point when requested again.

Question 4

What is the difference between yield and return?

Solution

• return terminates the function.
• yield pauses the function and allows it to continue later.

Question 5

What is a generator expression?

Solution

A generator expression is a concise way to create a generator using parentheses ().

Question 6

Why are generators memory efficient?

Solution

Generators create values only when they are requested instead of storing all values in memory.

Question 7

Can generators be used in a for loop?

Solution

Yes. A generator is an iterator and can be directly used in a for loop.

Question 8

True or False: Every iterator is a generator.

Solution

False. Every generator is an iterator, but not every iterator is a generator.

Question 9

Name two real-world use cases of generators.

Solution

Examples include:

• Reading large files
• Processing large datasets
• Streaming API data
• Log processing
• Infinite sequences

---

Practice & Exercises

To reinforce what you’ve learned in this section (Iterators, Custom Iterators, Generators, and Generator Expressions), practice with these interactive notebooks:

Follow-Along Practice

Practice creating iterators, implementing custom iterator classes, writing generators with yield, and creating generator expressions.

💻 VS Code | 🚀 Colab | 📥 Download

Practice Exercises

Test your skills with exercises on custom range iterators, cubes generators, odd numbers generator expressions, and infinite powers of three generators.

💻 VS Code | 🚀 Colab | 📥 Download