04-Advanced Python Concepts

Learn advanced Python concepts including functions as objects, higher-order functions, closures, decorators, iterators, generators, and context managers.

This chapter introduces advanced Python concepts that make programs more modular, reusable, and memory efficient. These concepts are widely used in modern Python applications, web frameworks, data processing, and AI libraries.

Since functions are objects, they can be assigned to variables, passed as arguments, returned from other functions, and stored in collections. These capabilities form the foundation for higher-order functions, decorators, closures, iterators and generators.

Topics Covered

In this module, you’ll learn:

1. Functions are Objects
2. Higher-Order Functions
3. Closures
4. Decorators
5. Iterators
6. Generators
7. Context Managers

Try Yourself: 💻 VS Code | 🚀 Colab | 📥 Download
Verify Solutions: 💻 VS Code | 🚀 Colab | 📥 Download

---

Back to Top ↑

Functions are Objects

Like integers, strings, lists, and dictionaries, functions are also objects. Therefore, a function can:

• Be assigned to a variable.
• Be passed as an argument.
• Be returned from another function.
• Be stored in a collection.

Languages that support these capabilities are said to support first-class functions.

Assigning a Function

def greet():
    return "Hello"

message = greet

print(message())

Output ?

Show Output

Hello

Notice the difference:

message = greet      # Function object
message = greet()    # Function call

Storing Functions

def add(a, b):
    return a + b


def subtract(a, b):
    return a - b


operations = [add, subtract]

print(operations[0](10, 5))
print(operations[1](10, 5))

Output ?

Show Output

15
5

Exercise 2

What is the difference between the following statements?

f = greet

f = greet()

Solution

• f = greet assigns the function object.
• f = greet() calls the function and stores its return value.

---

Back to Top ↑

Higher-Order Functions

A higher-order function is a function that:

• Accepts one or more functions as arguments.
• Returns a function.

Since functions are objects, they can be passed to and returned from other functions.

Passing Functions as Arguments

def greet():
    print("Hello")


def execute(func):
    func()


execute(greet)

Output ?

Show Output

Hello

Returning Functions

def outer():

    def inner():
        print("Hello")

    return inner


message = outer()

message()

Output ?

Show Output

Hello

Example

def add(a, b):
    return a + b


def multiply(a, b):
    return a * b


def calculate(operation, a, b):
    return operation(a, b)


print(calculate(add, 10, 5))
print(calculate(multiply, 10, 5))

Output ?

Show Output

15
50

Exercise 2

Predict the output.

def outer():

    def inner():
        return "Hello"

    return inner


func = outer()

print(func())

Output ?

Show Output

Hello

Solution

Hello

Exercise 3

When is a function called a higher-order function?

Solution

A function is called a higher-order function if it:

• Accepts one or more functions as arguments.
• Returns a function.

Higher-order functions form the foundation for closures and decorators.

Back to Top ↑

Closures

Sometimes we want a function to remember information from previous function calls.

A normal function cannot do this because its local variables are destroyed when the function finishes executing.

Example: Normal Function

def counter():

    count = 0

    count += 1

    return count


print(counter())
print(counter())
print(counter())

Output ?

Show Output

1
1
1

Each time counter() is called, the local variable count is created again and initialized to 0. Therefore, the function cannot remember its previous state.

To preserve the state between function calls, we can use a closure.

What is a Closure?

A closure is an inner function that remembers the variables of its enclosing function even after the enclosing function has finished executing.

A closure is created when:

• A function is defined inside another function.
• The inner function uses variables from the outer function.
• The inner function is returned.

Example

def counter():

    count = 0

    def increment():
        nonlocal count

        count += 1
        return count

    return increment


c = counter()

print(c())
print(c())
print(c())

Output ?

Show Output

1
2
3

Here, the variable count is preserved even after counter() has finished executing. Each call to increment() updates the same variable instead of creating a new one.

The nonlocal keyword allows the inner function to modify a variable defined in the enclosing function.

Exercise 2

Predict the output.

def counter():

    count = 10

    def increment():
        nonlocal count
        count += 5
        return count

    return increment


c = counter()

print(c())
print(c())

Output ?

Show Output

15
20

Solution

15
20

The variable count is preserved inside the closure and updated on each function call.

Exercise 3

Why do we use closures?

Solution

Closures allow a function to remember and preserve variables from its enclosing function even after the enclosing function has finished executing.

Closures are widely used for state preservation and form the foundation of decorators, where the wrapper function remembers the original function passed to the decorator.

Back to Top ↑

Decorators

A decorator is a function that extends or modifies the behavior of another function without changing its original code.

A decorator is a higher-order function because it:

• Accepts a function as an argument.
• Returns another function.

Creating a Decorator

def logger(func):

    def wrapper():
        print("Before function")

        func()

        print("After function")

    return wrapper


def greet():
    print("Hello")


greet = logger(greet)

greet()

Output ?

Show Output

Before function
Hello
After function

Instead of modifying greet(), the decorator returns a new function with additional behavior.

Using the @ Syntax

Python provides the @ syntax as a convenient way to apply decorators.

def logger(func):

    def wrapper():
        print("Before function")

        func()

        print("After function")

    return wrapper


@logger
def greet():
    print("Hello")


greet()

Output ?

Show Output

Before function
Hello
After function

The above code is equivalent to:

def greet():
    print("Hello")


greet = logger(greet)

Exercise 2

Which statement is equivalent to the following code?

@logger
def greet():
    print("Hello")

Solution

def greet():
    print("Hello")


greet = logger(greet)

The @ syntax is a shorthand for applying a decorator.

Decorating Functions with Parameters

The previous decorator works only for functions that do not accept any arguments.

@logger
def greet():
    print("Hello")

Suppose we decorate a function that accepts parameters.

@logger
def add(a, b):
    return a + b

Our decorator is:

def logger(func):

    def wrapper():
        print("Function Started")

        result = func()

        print("Function Completed")

        return result

    return wrapper

When we call:

add(10, 20)

Output ?

Show Output

Error executing: logger.<locals>.wrapper() takes 0 positional arguments but 2 were given

Python actually executes:

wrapper(10, 20)

Output ?

Show Output

Error executing: name 'wrapper' is not defined

Since wrapper() does not accept any arguments, Python raises an error.

TypeError: wrapper() takes 0 positional arguments but 2 were given

One solution is to make the wrapper accept the same parameters.

def logger(func):

    def wrapper(a, b):
        print("Function Started")

        result = func(a, b)

        print("Function Completed")

        return result

    return wrapper

This works only for functions having exactly two parameters.

To make the decorator work with any function, Python provides argument packing.

def logger(func):

    def wrapper(*args, **kwargs):
        print("Function Started")

        result = func(*args, **kwargs)

        print("Function Completed")

        return result

    return wrapper

Here,

• *args collects all positional arguments.
• **kwargs collects all keyword arguments.
• func(*args, **kwargs) forwards all arguments to the original function.

Now the decorator can be applied to functions with any number of arguments.

@logger
def multiply(a, b):
    return a * b


@logger
def greet(name):
    print(f"Hello {name}")


print(multiply(10, 5))
greet("Alice")

Output ?

Show Output

Function Started
Function Completed
50
Function Started
Hello Alice
Function Completed

The wrapper() function is defined inside another function and remembers the original func even after the outer function has finished executing. This behavior was possible because of closure.

---

An iterator is an object that returns one value at a time from a collection, while a generator is a special type of iterator created using the yield keyword. They provide a memory-efficient way to process data without loading everything into memory at once.

Back to Top ↑

Learning Objectives

After completing this lesson, you will be able to:

• Understand iterables, iterators, and generators.
• Create iterators using iter() and next().
• Build custom iterators.
• Create generators using yield.
• Differentiate between yield and return.
• Create generator expressions.
• Compare iterators and generators.
• Identify real-world use cases of generators.

Back to Top ↑

What is an Iterator?

An iterator is an object that returns one element at a time from a collection. It remembers its current position and produces the next value only when requested.

Python uses iterators internally whenever you iterate over a collection using a for loop.

Iterator Protocol

An iterator implements the following special methods:

• __iter__() – Returns the iterator object.
• __next__() – Returns the next element.

When no more elements are available, __next__() raises a StopIteration exception.

Creating an Iterator

Use the iter() function to create an iterator from an iterable.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(iterator)

Output ?

Show Output

<list_iterator object at 0x...>

Retrieving Values

Use the next() function to retrieve values from an iterator.

numbers = [10, 20, 30]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))
print(next(iterator))

Output ?

Show Output

10
20
30

StopIteration

Once all elements are consumed, calling next() again raises a StopIteration exception.

numbers = [10]

iterator = iter(numbers)

print(next(iterator))
print(next(iterator))

Output ?

Show Output

10
Error executing: 

Exercise 2

Predict the output.

name = "Python"

it = iter(name)

print(next(it))
print(next(it))
print(next(it))

Output ?

Show Output

P
y
t

Solution

P
y
t

Strings are iterable objects, so they can be converted into iterators using iter().

Exercise 3

What exception will be raised by the following code?

numbers = [1]

it = iter(numbers)

print(next(it))
print(next(it))

Output ?

Show Output

1
Error executing: 

Solution

A StopIteration exception is raised because the iterator has no more elements to return.

Back to Top ↑

Creating a Custom Iterator

You can create your own iterator by implementing the __iter__() and __next__() methods.

• __iter__() returns the iterator object.
• __next__() returns the next value.
• When all values are consumed, __next__() raises a StopIteration exception.

Example

class Counter:

    def __init__(self, limit):
        self.current = 1
        self.limit = limit

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.limit:
            raise StopIteration

        value = self.current
        self.current += 1
        return value


counter = Counter(5)

for number in counter:
    print(number)

Output ?

Show Output

1
2
3
4
5

How It Works

1. The Counter object is created.
2. The for loop calls __iter__() to obtain the iterator.
3. The loop repeatedly calls __next__().
4. Each call returns the next value.
5. When the limit is reached, StopIteration is raised, ending the loop.

Exercise 2

What happens if raise StopIteration is removed from the __next__() method?

Solution

The iterator will never indicate that it has finished, causing the loop to continue indefinitely or resulting in incorrect behavior.

Exercise 3

Which two special methods must every custom iterator implement?

Solution

Every custom iterator must implement:

• __iter__()
• __next__()

Back to Top ↑

What is a Generator?

A generator is a special type of iterator created using a function that contains the yield keyword. Unlike a normal function that returns all values at once, a generator produces one value at a time and automatically remembers its execution state.

Generators are easier to write than custom iterators because Python automatically implements the iterator protocol for you.

Creating a Generator

A function becomes a generator as soon as it contains a yield statement.

def numbers():
    yield 1
    yield 2
    yield 3


gen = numbers()

print(gen)

Output ?

Show Output

<generator object numbers at 0x...>

Notice that calling the function does not execute it immediately. Instead, it returns a generator object.

Using next() with a Generator

The next() function starts the generator and retrieves one value at a time.

def numbers():
    print("Starting")

    yield 1
    yield 2
    yield 3

    print("Ending")


gen = numbers()

print(next(gen))
print(next(gen))
print(next(gen))

Output ?

Show Output

Starting
1
2
3

The "Ending" message is not printed because the generator pauses after the third yield. It executes the remaining statements only when resumed again.

Using a Generator with a for Loop

Generators can be directly used in a for loop.

def numbers():
    yield 1
    yield 2
    yield 3


for number in numbers():
    print(number)

Output ?

Show Output

1
2
3

The for loop automatically calls next() until the generator raises StopIteration.

Exercise 2

Predict the output.

def greet():
    print("Hello")
    yield 1

g = greet()

print("Generator Created")

Output ?

Show Output

Generator Created

Solution

Generator Created

Calling a generator function does not execute its body immediately. It simply creates a generator object. The "Hello" message is printed only when the generator starts executing (for example, by calling next(g) or iterating over it).

Exercise 3

What is the output?

def values():
    yield "A"
    yield "B"

for value in values():
    print(value)

Output ?

Show Output

A
B

Solution

A
B

The for loop automatically retrieves values from the generator until it is exhausted.

Back to Top ↑

Understanding yield

The yield keyword is used to produce a value from a generator. Unlike return, which terminates a function, yield pauses the function and preserves its current state. The next time the generator is resumed, execution continues from the statement immediately after the previous yield.

yield vs return

return	yield
Terminates the function	Pauses the function
Returns a single value	Produces one value at a time
Function cannot resume	Function resumes from the same point
Used in normal functions	Used in generator functions

Execution Flow

def greet():

    print("Step 1")
    yield "Hello"

    print("Step 2")
    yield "World"

    print("Step 3")


g = greet()

print(next(g))
print(next(g))
next(g)

Output ?

Show Output

Step 1
Hello
Step 2
World
Step 3
Traceback (most recent call last):
...
StopIteration

Notice that the function resumes exactly where it paused after each yield.

State Preservation

One of the biggest advantages of generators is that they automatically preserve the values of local variables.

def counter():

    count = 1

    while count <= 3:
        yield count
        count += 1


g = counter()

print(next(g))
print(next(g))
print(next(g))

Output ?

Show Output

1
2
3

The variable count is not reinitialized each time. Its value is preserved between successive calls to next().

Multiple yield Statements

A generator can contain multiple yield statements.

def colors():

    yield "Red"
    yield "Green"
    yield "Blue"


for color in colors():
    print(color)

Output ?

Show Output

Red
Green
Blue

Each yield produces one value before the generator pauses.

Exercise 2

Predict the output.

def test():

    x = 5
    yield x

    x += 5
    yield x


g = test()

print(next(g))
print(next(g))

Solution

Output ?

Show Output

5
10

The value of x is preserved between the two yield statements.

Exercise 3

What is the main difference between return and yield?

Solution

• return terminates the function and returns a value.
• yield pauses the function, returns a value, preserves its state, and resumes execution when requested again.

Back to Top ↑

Generator Expressions

A generator expression provides a concise way to create generators. It is similar to a list comprehension but uses parentheses () instead of square brackets [].

Generator expressions generate values only when required, making them memory efficient.

Syntax

(expression for item in iterable)

Output ?

Show Output

Error executing: name 'iterable' is not defined

Example

squares = (x * x for x in range(5))

for square in squares:
    print(square)

Output ?

Show Output

0
1
4
9
16

Generator Expression vs List Comprehension

# List Comprehension
numbers = [x * x for x in range(5)]

# Generator Expression
numbers = (x * x for x in range(5))

• A list comprehension stores all values in memory.
• A generator expression generates values one at a time.

Exercise 2

Which symbol is used to create a generator expression?

Solution

Generator expressions use parentheses (), whereas list comprehensions use square brackets [].

---

Back to Top ↑

Infinite Generators

Generators can produce infinite sequences because values are generated only when requested.

Example

def even_numbers():

    number = 0

    while True:
        yield number
        number += 2


gen = even_numbers()

for _ in range(5):
    print(next(gen))

Output ?

Show Output

0
2
4
6
8

---

Back to Top ↑

Fibonacci Generator

Generators are commonly used to generate mathematical sequences.

Example

def fibonacci(limit):

    a, b = 0, 1

    while a <= limit:
        yield a
        a, b = b, a + b


for number in fibonacci(20):
    print(number)

Output ?

Show Output

0
1
1
2
3
5
8
13
21

---

Back to Top ↑

Memory Efficiency

One of the biggest advantages of generators is memory efficiency.

List Example

numbers = [x for x in range(1000000)]

The above statement creates one million values in memory.

Generator Example

numbers = (x for x in range(1000000))

The generator creates only one value at a time, significantly reducing memory usage.

When to Use Generators

Use generators when:

• Working with large datasets.
• Reading large files.
• Processing streaming data.
• Producing values on demand.
• Creating infinite sequences.

---

Back to Top ↑

Iterator vs Generator

Feature	Iterator	Generator
Created Using	Class	Function
Implements	__iter__() and __next__()	yield
State Management	Manual	Automatic
Code Size	More	Less
Lazy Evaluation	Yes	Yes
Memory Efficient	Yes	Yes
Best Use Case	Custom iteration logic	Sequential data generation

Remember: Every generator is an iterator, but not every iterator is a generator.

---

Back to Top ↑

Iterable vs Iterator vs Generator

                Iterable
       (list, tuple, set, dict)
                   │
             iter(iterable)
                   │
                   ▼
               Iterator
       (__iter__ + __next__)
                   ▲
                   │
         Generator Object
        (Created using yield)

• Iterable → An object that can produce an iterator.
• Iterator → Produces one value at a time.
• Generator → A special iterator created using the yield keyword.

---

Back to Top ↑

Real-World Applications

Generators are commonly used for:

• Reading large files line by line.
• Processing large datasets.
• Streaming data from APIs.
• Log processing.
• Data pipelines.
• Machine learning workflows.
• Infinite sequences.

Example: Reading a File

def read_file(filename):

    with open(filename, "r") as file:
        for line in file:
            yield line.strip()


for line in read_file("data.txt"):
    print(line)

Output ?

Show Output

Error executing: [Errno 2] No such file or directory: 'data.txt'

Instead of loading the entire file into memory, one line is processed at a time.

---

Back to Top ↑

Key Takeaways

• An iterable is an object that can produce an iterator.
• An iterator returns one value at a time using next().
• A generator is a simpler way to create an iterator using yield.
• The yield keyword pauses execution and preserves the function’s state.
• Generator expressions provide a concise syntax for creating generators.
• Generators are ideal for processing large datasets because they use lazy evaluation.
• Every generator is an iterator, but not every iterator is a generator.

---

Back to Top ↑

Check Your Understanding

Question 1

What is the purpose of the iter() function?

Solution

The iter() function converts an iterable into an iterator.

Question 2

Which special methods make an object an iterator?

Solution

__iter__() and __next__()

Question 3

What is the purpose of the yield keyword?

Solution

The yield keyword pauses a generator, returns a value, preserves its state, and resumes execution from the same point when requested again.

Question 4

What is the difference between yield and return?

Solution

• return terminates the function.
• yield pauses the function and allows it to continue later.

Question 5

What is a generator expression?

Solution

A generator expression is a concise way to create a generator using parentheses ().

Question 6

Why are generators memory efficient?

Solution

Generators create values only when they are requested instead of storing all values in memory.

Question 7

Can generators be used in a for loop?

Solution

Yes. A generator is an iterator and can be directly used in a for loop.

Question 8

True or False: Every iterator is a generator.

Solution

False. Every generator is an iterator, but not every iterator is a generator.

Question 9

Name two real-world use cases of generators.

Solution

Examples include:

• Reading large files
• Processing large datasets
• Streaming API data
• Log processing
• Infinite sequences

---

Back to Top ↑

Context Managers

Whenever we open external resources such as files or database connections, they should be closed properly. Python provides the with statement to handle this automatically.

Without a Context Manager

file = open("sample.txt", "r")
content = file.read()
print(content)
file.close()

Output ?

Show Output

Error executing: [Errno 2] No such file or directory: 'sample.txt'

If an exception occurs before close(), the file may remain open.

Using a Context Manager

with open("sample.txt", "r") as file:
    content = file.read()
    print(content)

Output ?

Show Output

Error executing: [Errno 2] No such file or directory: 'sample.txt'

The file is automatically closed after leaving the with block.

Custom Context Manager

A context manager implements two special methods:

• __enter__()
• __exit__()

class Database:
    def __enter__(self):
        print("Connection Opened")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("Connection Closed")

with Database():
    print("Executing Queries")

Show Output

Connection Opened
Executing Queries
Connection Closed

Exercise 1

Open a file using the with statement and display its contents.

Sample Input

# Assume file sample.txt exists

Expected Output

Displays the file contents.

Solution

with open("sample.txt", "r") as file:
    print(file.read())

Exercise 2

Create a context manager that prints "Start" on entry and "End" on exit.

Sample Input

with Demo():
    print("Inside")

Expected Output

Start
Inside
End

Solution

class Demo:
    def __enter__(self):
        print("Start")
        return self

    def __exit__(self, exc_type, exc_value, traceback):
        print("End")

with Demo():
    print("Inside")

---