加载中

Prior to beginning tutoring sessions, I ask new students to fill out a brief self-assessment where they rate their understanding of various Python concepts. Some topics ("control flow with if/else" or "defining and using functions") are understood by a majority of students before ever beginning tutoring. There are a handful of topics, however, that almost all students report having no knowledge or very limited understanding of. Of these, "generatorsand theyieldkeyword" is one of the biggest culprits. I'm guessing this is the case for most novice Python programmers.

Many report having difficulty understandinggeneratorsand theyieldkeyword even after making a concerted effort to teach themselves the topic. I want to change that. In this post, I'll explain what theyieldkeyword does, why it's useful, and how to use it.

Note: In recent years, generators have grown more powerful as features have been added through PEPs. In my next post, I'll explore the true power ofyieldwith respect to coroutines, cooperative multitasking and asynchronous I/O (especially their use in the "tulip" prototype implementation GvR has been working on). Before we get there, however, we need a solid understanding of how theyieldkeyword andgeneratorswork.

在开始课程之前,我要求学生们填写一份调查表,这个调查表反映了它们对Python中一些概念的理解情况。一些话题("if/else控制流" 或者 "定义和使用函数")对于大多数学生是没有问题的。但是有一些话题,大多数学生只有很少,或者完全没有任何接触,尤其是“生成器和yield关键字”。我猜这对大多数新手Python程序员也是如此。

有事实表明,在我花了大功夫后,有些人仍然不能理解生成器和yield关键字。我想让这个问题有所改善。在这篇文章中,我将解释yield关键字到底是什么,为什么它是有用的,以及如何来使用它。

注意:最近几年,生成器的功能变得越来越强大,它已经被加入到了PEP。在我的下一篇文章中,我会通过协程(coroutine),协同式多任务处理(cooperative multitasking),以及异步IO(asynchronous I/O)(尤其是GvR正在研究的 "tulip" 原型的实现)来介绍yield的真正威力。但是在此之前,我们要对生成器和yield有一个扎实的理解.

Coroutines and Subroutines

When we call a normal Python function, execution starts at function's first line and continues until areturnstatement,exception, or the end of the function (which is seen as an implicitreturn None) is encountered. Once a function returns control to its caller, that's it. Any work done by the function and stored in local variables is lost. A new call to the function creates everything from scratch.

This is all very standard when discussing functions (more generally referred to as subroutines) in computer programming. There are times, though, when it's beneficial to have the ability to create a "function" which, instead of simply returning a single value, is able to yield a series of values. To do so, such a function would need to be able to "save its work," so to speak.

I said, "yield a series of values" because our hypothetical function doesn't "return" in the normal sense.returnimplies that the function is returning control of execution to the point where the function was called. "Yield," however, implies that the transfer of control is temporary and voluntary, and our function expects to regain it in the future.

协程与子例程


我们调用一个普通的Python函数时,一般是从函数的第一行代码开始执行,结束于return语句、异常或者函数结束(可以看作隐式的返回None)。一旦函数将控制权交还给调用者,就意味着全部结束。函数中做的所有工作以及保存在局部变量中的数据都将丢失。再次调用这个函数时,一切都将从头创建。

对于在计算机编程中所讨论的函数,这是很标准的流程。这样的函数只能返回一个值,不过,有时可以创建能产生一个序列的函数还是有帮助的。要做到这一点,这种函数需要能够“保存自己的工作”。

我说过,能够“产生一个序列”是因为我们的函数并没有像通常意义那样返回。return隐含的意思是函数正将执行代码的控制权返回给函数被调用的地方。而"yield"的隐含意思是控制权的转移是临时和自愿的,我们的函数将来还会收回控制权。

In Python, "functions" with these capabilities are calledgenerators, and they're incredibly useful.generators(and theyieldstatement) were initially introduced to give programmers a more straightforward way to write code responsible for producing a series of values. Previously, creating something like a random number generator required a class or module that both generated values and kept track of state between calls. With the introduction ofgenerators, this became much simpler.

To better understand the problemgeneratorssolve, let's take a look at an example. Throughout the example, keep in mind the core problem being solved: generating a series of values.

Note: Outside of Python, all but the simplestgeneratorswould be referred to as coroutines. I'll use the latter term later in the post. The important thing to remember is, in Python, everything described here as acoroutineis still agenerator. Python formally defines the termgenerator;coroutineis used in discussion but has no formal definition in the language.

在Python中,拥有这种能力的“函数”被称为生成器,它非常的有用。生成器(以及yield语句)最初的引入是为了让程序员可以更简单的编写用来产生值的序列的代码。 以前,要实现类似随机数生成器的东西,需要实现一个类或者一个模块,在生成数据的同时保持对每次调用之间状态的跟踪。引入生成器之后,这变得非常简单。

为了更好的理解生成器所解决的问题,让我们来看一个例子。在了解这个例子的过程中,请始终记住我们需要解决的问题:生成值的序列。

注意:在Python之外,最简单的生成器应该是被称为协程(coroutines)的东西。在本文中,我将使用这个术语。请记住,在Python的概念中,这里提到的协程就是生成器。Python正式的术语是生成器;协程只是便于讨论,在语言层面并没有正式定义。

Example: Fun With Prime Numbers

Suppose our boss asks us to write a function that takes alistofints and returns some Iterable containing the elements which are prime1 numbers.

Remember, an Iterable is just an object capable of returning its members one at a time.

"Simple," we say, and we write the following:

def get_primes(input_list):
    result_list = list()
    for element in input_list:
        if is_prime(element):
            result_list.append()

    return result_list

# or better yet...

def get_primes(input_list):
    return (element for element in input_list if is_prime(element))

# not germane to the example, but here's a possible implementation of
# is_prime...

def is_prime(number):
    if number > 1:
        if number == 2:
            return True
        if number % 2 == 0:
            return False
        for current in range(3, int(math.sqrt(number) + 1), 2):
            if number % current == 0: 
                return False
        return True
    return False

Eitheris_primeimplementation above fulfills the requirements, so we tell our boss we're done. She reports our function works and is exactly what she wanted.

例子:有趣的素数

假设你的老板让你写一个函数,输入参数是一个int的list,返回一个可以迭代的包含素数1 的结果。

记住,迭代器(Iterable) 只是对象每次返回特定成员的一种能力。

你肯定认为"这很简单",然后很快写出下面的代码:

def get_primes(input_list):
    result_list = list()
    for element in input_list:
        if is_prime(element):
            result_list.append()

    return result_list

# 或者更好一些的...

def get_primes(input_list):
    return (element for element in input_list if is_prime(element))

# 下面是 is_prime 的一种实现...

def is_prime(number):
    if number > 1:
        if number == 2:
            return True
        if number % 2 == 0:
            return False
        for current in range(3, int(math.sqrt(number) + 1), 2):
            if number % current == 0: 
                return False
        return True
    return False

上面 is_prime 的实现完全满足了需求,所以我们告诉老板已经搞定了。她反馈说我们的函数工作正常,正是她想要的。

Dealing With Infinite Sequences

Well, not quite exactly. A few days later, our boss comes back and tells us she's run into a small problem: she wants to use ourget_primesfunction on a very large list of numbers. In fact, the list is so large that merely creating it would consume all of the system's memory. To work around this, she wants to be able to callget_primeswith astartvalue and get all the primes larger thanstart(perhaps she's solving Project Euler problem 10).

Once we think about this new requirement, it becomes clear that it requires more than a simple change toget_primes. Clearly, we can't return a list of all the prime numbers fromstartto infinity (operating on infinite sequences, though, has a wide range of useful applications). The chances of solving this problem using a normal function seem bleak.

处理无限序列

噢,真是如此吗?过了几天,老板过来告诉我们她遇到了一些小问题:她打算把我们的get_primes函数用于一个很大的包含数字的list。实际上,这个list非常大,仅仅是创建这个list就会用完系统的所有内存。为此,她希望能够在调用get_primes函数时带上一个start参数,返回所有大于这个参数的素数(也许她要解决 Project Euler problem 10)。

我们来看看这个新需求,很明显只是简单的修改get_primes是不可能的。 自然,我们不可能返回包含从start到无穷的所有的素数的列表 (虽然有很多有用的应用程序可以用来操作无限序列)。看上去用普通函数处理这个问题的可能性比较渺茫。

Before we give up, let's determine the core obstacle preventing us from writing a function that satisfies our boss's new requirements. Thinking about it, we arrive at the following: functions only get one chance to return results, and thus must return all results at once. It seems pointless to make such an obvious statement; "functions just work that way," we think. The real value lies in asking, "but what if they didn't?"

Imagine what we could do ifget_primescould simply return the next value instead of all the values at once. It wouldn't need to create a list at all. No list, no memory issues. Since our boss told us she's just iterating over the results, she wouldn't know the difference.

在我们放弃之前,让我们确定一下最核心的障碍,是什么阻止我们编写满足老板新需求的函数。通过思考,我们得到这样的结论:函数只有一次返回结果的机会,因而必须一次返回所有的结果。得出这样的结论似乎毫无意义;“函数不就是这样工作的么”,通常我们都这么认为的。可是,不学不成,不问不知,“如果它们并非如此呢?”

想象一下,如果get_primes可以只是简单返回下一个值,而不是一次返回全部的值,我们能做什么?我们就不再需要创建列表。没有列表,就没有内存的问题。由于老板告诉我们的是,她只需要遍历结果,她不会知道我们实现上的区别。

Unfortunately, this doesn't seem possible. Even if we had a magical function that allowed us to iterate fromntoinfinity, we'd get stuck after returning the first value:

def get_primes(start):
    for element in magical_infinite_range(start):
        if is_prime(element):
            return element
Imagineget_primesis called like so:
def solve_number_10():
    # She *is* working on Project Euler #10, I knew it!
    total = 2
    for next_prime in get_primes(3):
        if next_prime < 2000000:
            total += next_prime
        else:
            print(total)
            return

Clearly, inget_primes, we would immediately hit the case wherenumber = 3and return at line 4. Instead ofreturn, we need a way to generate a value and, when asked for the next one, pick up where we left off.

Functions, though, can't do this. When theyreturn, they're done for good. Even if we could guarantee a function would be called again, we have no way of saying, "OK, now, instead of starting at the first line like we normally do, start up where we left off at line 4." Functions have a singleentry point: the first line.

不幸的是,这样做看上去似乎不太可能。即使是我们有神奇的函数,可以让我们从n遍历到无限大,我们也会在返回第一个值之后卡住:

def get_primes(start):
    for element in magical_infinite_range(start):
        if is_prime(element):
            return element
假设这样去调用get_primes:
def solve_number_10():
    # She *is* working on Project Euler #10, I knew it!
    total = 2
    for next_prime in get_primes(3):
        if next_prime < 2000000:
            total += next_prime
        else:
            print(total)
            return

显然,在get_primes中,一上来就会碰到输入等于3的,并且在函数的第4行返回。与直接返回不同,我们需要的是在退出时可以为下一次请求准备一个值。

不过函数做不到这一点。当函数返回时,意味着全部完成。我们保证函数可以再次被调用,但是我们没法保证说,“呃,这次从上次退出时的第4行开始执行,而不是常规的从第一行开始”。函数只有一个单一的入口:函数的第1行代码。

Enter the Generator

This sort of problem is so common that a new construct was added to Python to solve it: thegenerator. Agenerator"generates" values. Creatinggeneratorswas made as straightforward as possible through the concept ofgenerator functions, introduced simultaneously.

Agenerator functionis defined like a normal function, but whenever it needs to generate a value, it does so with theyieldkeyword rather thanreturn. If the body of adefcontainsyield, the function automatically becomes agenerator function(even if it also contains areturnstatement). There's nothing else we need to do to create one.

generator functionscreategenerator iterators. That's the last time you'll see the termgenerator iterator, though, since they're almost always referred to as "generators". Just remember that ageneratoris a special type ofiterator. To be considered aniterator,generatorsmust define a few methods, one of which is__next__(). To get the next value from agenerator, we use the same built-in function as foriterators:next().

走进生成器

这类问题极其常见以至于Python专门加入了一个结构来解决它:生成器。一个生成器会“生成”值。创建一个生成器几乎和生成器函数的原理一样简单。

一个生成器函数的定义很像一个普通的函数,除了当它要生成一个值的时候,使用yield关键字而不是return。如果一个def的主体包含yield,这个函数会自动变成一个生成器(即使它包含一个return)。除了以上内容,创建一个生成器没有什么多余步骤了。

生成器函数返回生成器的迭代器。这可能是你最后一次见到“生成器的迭代器”这个术语了, 因为它们通常就被称作“生成器”。要注意的是生成器就是一类特殊的迭代器。作为一个迭代器,生成器必须要定义一些方法(method),其中一个就是__next__()。如同迭代器一样,我们可以使用next()函数来获取下一个值。

This point bear repeating: to get the next value from agenerator, we use the same built-in function as foriterators:next().

(next()takes care of calling the generator's__next__()method). Since ageneratoris a type ofiterator, it can be used in aforloop.

So whenevernext()is called on agenerator, thegeneratoris responsible for passing back a value to whomever callednext(). It does so by callingyieldalong with the value to be passed back (e.g.yield 7). The easiest way to remember whatyielddoes is to think of it asreturn(plus a little magic) forgenerator functions.**

Again, this bears repeating: yieldis justreturn(plus a little magic) forgenerator functions.

Here's a simplegenerator function:

>>> def simple_generator_function():
>>>    yield 1
>>>    yield 2
>>>    yield 3
And here are two simple ways to use it:
>>> for value in simple_generator_function():
>>>     print(value)
1
2
3
>>> our_generator = simple_generator_function()
>>> next(our_generator)
1
>>> next(our_generator)
2
>>> next(our_generator)
3

为了从生成器获取下一个值,我们使用next()函数,就像对付迭代器一样。

(next()会操心如何调用生成器的__next__()方法)。既然生成器是一个迭代器,它可以被用在for循环中。

每当生成器被调用的时候,它会返回一个值给调用者。在生成器内部使用yield来完成这个动作(例如yield 7)。为了记住yield到底干了什么,最简单的方法是把它当作专门给生成器函数用的特殊的return(加上点小魔法)。**

yield就是专门给生成器用的return(加上点小魔法)。

下面是一个简单的生成器函数:

>>> def simple_generator_function():
>>>    yield 1
>>>    yield 2
>>>    yield 3
这里有两个简单的方法来使用它:
>>> for value in simple_generator_function():
>>>     print(value)
1
2
3
>>> our_generator = simple_generator_function()
>>> next(our_generator)
1
>>> next(our_generator)
2
>>> next(our_generator)
3

Magic?

What's the magic part? Glad you asked! When agenerator functioncallsyield, the "state" of thegenerator functionis frozen; the values of all variables are saved and the next line of code to be executed is recorded untilnext()is called again. Once it is, thegenerator functionsimply resumes where it left off. Ifnext()is never called again, the state recorded during theyieldcall is (eventually) discarded.

Let's rewriteget_primesas agenerator function. Notice that we no longer need themagical_infinite_rangefunction. Using a simplewhileloop, we can create our own infinite sequence:

def get_primes(number):
    while True:
        if is_prime(number):
            yield number
        number += 1

魔法?

那么神奇的部分在哪里?我很高兴你问了这个问题!当一个生成器函数调用yield,生成器函数的“状态”会被冻结,所有的变量的值会被保留下来,下一行要执行的代码的位置也会被记录,直到再次调用next()。一旦next()再次被调用,生成器函数会从它上次离开的地方开始。如果永远不调用next(),yield保存的状态就被无视了。

我们来重写get_primes()函数,这次我们把它写作一个生成器。注意我们不再需要magical_infinite_range函数了。使用一个简单的while循环,我们创造了自己的无穷串列。

def get_primes(number):
    while True:
        if is_prime(number):
            yield number
        number += 1
返回顶部
顶部