加载中

Programs need to update files. Although most programmers know that unexpected things can happen while performing I/O, I often see code that has been written in a surprisingly naïve way. In this article, I would like to share some insights on how to improve I/O reliability in Python code.

Consider the following Python snippet. Some operation is performed on data coming from and going back into a file:

with open(filename) as f:
   input = f.read()
output = do_something(input)
with open(filename, 'w') as f:
   f.write(output)

Pretty simple? Probably not as simple as it looks at the first glance. I often debug applications that show strange behaviour on production servers.

程序需要更新文件。虽然大部分程序员知道在执行I/O的时候会发生不可预期的事情,但是我经常看到一些异常幼稚的代码。在本文中,我想要分享一些如何在Python代码中改善I/O可靠性的见解。

考虑下述Python代码片段。对文件中的数据进行某些操作,然后将结果保存回文件中:

with open(filename) as f:
   input = f.read()
output = do_something(input)
with open(filename, 'w') as f:
   f.write(output)

看起来很简单吧?可能看起来并不像乍一看这么简单。我在产品服务器中调试应用,经常会出现奇怪的行为。

Here are examples of failure modes I have seen:

  • A run away server process spills out huge amounts of logs and the disk fills up.write()raises an exception right after truncating the file, leaving the file empty.
  • Several instances of our application happen to run in parallel. After they have finished, the file contents is garbage because it intermingles output from multiple instances.
  • The application triggers some follow-up action after completing the write. Seconds later, the power goes off. After we have restarted the server, we see the old file contents again. The data already passed to other applications does not correspond to what we see in the file anymore.

Nothing of what follows is really new. My goal is to present common approaches and techniques to Python developers who are less experienced in system programming. I will provide code examples to make it easy for developers to incorporate these approaches into their own code.

这是我看过的失效模式的例子:
  • 失控的服务器进程溢出大量日志,磁盘被填满。write()在截断文件之后抛出异常,文件将会变成空的。
  • 应用的几个实例并行执行。在各个实例结束之后,因为混合了多个实例的输出,文件内容最终变成了天书。
  • 在完成了写操作之后,应用会触发一些后续操作。几秒钟后断电。在我们重启了服务器之后,我们再一次看到了旧的文件内容。已经传递给其它应用的数据与我们在文件中看到的不再一致。

下面没有什么新的内容。本文的目的是为在系统编程方面缺少经验的Python开发者提供常见的方法和技术。我将会提供代码例子,使得开发者可以很容易的将这些方法应用到自己的代码中。

What does “reliability” mean anyway?

In the broadest sense, reliability means that an operation is performing its required function under all stated conditions. With regard to file updates, the function in question is to create, replace or extend the contents of a file. It might be rewarding to seek inspiration from database theory here. The ACID properties of the classic transaction model will serve as guidelines to improve reliability.

To get started, let’s see how the initial example can be rated against the four ACID properties:

  • Atomicity requires that a transaction either succeeds or fails completely. In the example shown above, a full disk will likely result in a partially written file. Additionally, if other programs read the file while it is being written, they get a half-finished version even in the absence of write errors.
  • Consistency denotes that updates must bring the system from one valid state to another. Consistency can be subdivided into internal and external consistency: Internal consistency means that the file’s data structures are consistent. External consistency means that the file’s contents is aligned with other data related to it. In this example, it is hard to reason about consistency since we don’t know enough about the application. But since consistency requires atomicity, we can say at least that internal consistency is not guaranteed.
  • Isolation is violated if running transactions concurrently yields different results from running the same transactions sequentially. It is clear that the code above has no protection against lost updates or other isolation failures.
  • Durability means that changes need to be permanent. Before we signal success to the user, we must be sure that our data hits non-volatile storage and not just a write cache. Perhaps the code above has been written with the assumption in mind that disk I/O takes place immediately when we callwrite(). This assumption is not warranted by POSIX semantics.

“可靠性”意味着什么?

广义的讲,可靠性意味着在所有规定的条件下操作都能执行它所需的函数。至于文件的操作,这个函数就是创建,替换或者追加文件的内容的问题。这里可以从数据库理论上获得灵感。经典的事务模型的ACID性质作为指导来提高可靠性。

开始之前,让我们先看看我们的例子怎样和ACID4个性质扯上关系:

  • 原子性(Atomicity)要求这个事务要么完全成功,要么完全失败。在上面的实例中,磁盘满了可能导致部分内容写入文件。另外,如果正当在写入内容时其它程序又在读取文件,它们可能获得是部分完成的版本,甚至会导致写错误
  • 一致性(Consistency) 表示操作必须从系统的一个状态到另一个状态。一致性可以分为两部分:内部和外部一致性。内部一致性是指文件的数据结构是一致的。外部一致性是指文件的内容与它相关的数据是相符合的。在这个例子中,因为我们不了解这个应用,所以很难推断是否符合一致性。但是因为一致性需要原子性,我们至少可以说没有保证内部一致性。
  • 隔离性(Isolation)如果在并发的执行事务中,多个相同的事务导致了不同的结果,就违反了隔离性。很明显上面的代码对操作失败或者其它隔离性失败都没有保护。
  • 持久性(Durability)意味着改变是持久不变的。在我们告诉用户成功之前,我们必须确保我们的数据存储是可靠的并且不只是一个写缓存。上面的代码已经成功写入数据的前提是假设我们调用write()函数,磁盘I/O就立即执行。但是POSIX标准是不保证这个假设的。


Use a database system if you can

If we would be able to gain all four ACID properties, we would have come a long way towards increased reliability. But this requires significant coding effort. Why reinvent the wheel? Most database systems already have ACID transactions.

Reliable data storage is a solved problem. If you need reliable storage, use a database. Chances are high that you will not do it by yourself as good as those who have been working on it for years if not decades. If you do not want to set up a “big” database server, you can use sqlite for example. It has ACID transactions, it’s small, it’s free, and it’s included in Python’s standard library.

尽可能使用数据库系统

如果我们能够获得ACID 四个性质,那么我们增加可靠性方面取得了长远发展。但是这需要很大的编码功劳。为什么重复发明轮子?大多数数据库系统已经有ACID事务

可靠性数据存储已经是一个已解决的问题。如果你需要可靠性存储,请使用数据库。很可能,没有几十年的功夫,你自己解决这方面的能力没有那些已经专注这方面好些年的人好。如果你不想安装一个大数据库服务器,那么你可以使用sqlite,它具有ACID事务,很小,免费的,而且它包含在Python的标准库中。

The article could finish here. But there are valid reasons not to use a database. They are often tied to file format or file location constraints. Both are not easily controllable with database systems. Reasons include:

  • we must process files generated by other applications, which are in a fixed format or at a fixed location
  • we must write files for consumption by other applications (and the same restrictions apply)
  • our files must be human-readable or human-editable

…and so on. You get the point.

If we are set out to implement reliable file updates on our own, there are some programming techniques to consider. In the following, I will present four common patterns of performing file updates. After that, I will discuss what steps can be taken to establish ACID properties with each file update pattern.

文章本该在这里就结束的,但是还有一些有根有据的原因,就是不使用数据。它们通常是文件格式或者文件位置约束。这两个在数据库系统中都不好控制。理由如下:

  • 我们必须处理其它应用产生的固定格式或者在固定位置的文件,
  • 我们必须为了其它应用的消耗而写文件(和应用了同样的限制条件)
  • 我们的文件必须方便人阅读或者修改。

...等等。你懂的。

如果我们自己动手实现可靠的文件更新,那么这里有一些编程技术供参考。下面我将展示四种常见的操作文件更新模式。在那之后,我会讨论采取哪些步骤在每个文件更新模式下满足ACID性质。

File update patterns

Files can be updated in a multitude of ways, but I see at least four common patterns. These will serve as a basis for the rest of this article.

Truncate-Write

This is probably the most basic pattern. In the following example, hypothetical domain model code reads data, performs some computation, and re-opens the existing file in write mode:

with open(filename, 'r') as f:
   model.read(f)
model.process()
with open(filename, 'w') as f:
   model.write(f)

A variant of this pattern opens the file in read-write mode (the “plus” modes in Python), seeks to the start, issues an explicittruncate()call and rewrites the contents:

with open(filename, 'a+') as f:
   f.seek(0)
   model.input(f.read())
   model.compute()
   f.seek(0)
   f.truncate()
   f.write(model.output())

An advantage of this variant is that we open file only once and keep it open all the time. This simplifies locking for example.

文件更新模式

文件可以以多种方式更新,但是我认为至少有四种常见的模式。这四种模式将做为本文剩余部分的基础。

截断-写

这可能是最基本的模式。在下述例子中,假设的域模型代码读数据,执行一些计算,然后以写模式重新打开存在的文件:

with open(filename, 'r') as f:
   model.read(f)
model.process()
with open(filename, 'w') as f:
   model.write(f)

此模式的一个变种以读写模式打开文件(Python中的“加”模式),寻找到开始的位置,显式调用truncate(),重写文件内容。

with open(filename, 'a+') as f:
   f.seek(0)
   model.input(f.read())
   model.compute()
   f.seek(0)
   f.truncate()
   f.write(model.output())

该变种的优势是只打开文件一次,始终保持文件打开。举例来说,这样可以简化加锁。

Write-Replace

Another widely used pattern is to write new contents into a temporary file and replace the original file after that:

with tempfile.NamedTemporaryFile(
      'w', dir=os.path.dirname(filename), delete=False) as tf:
   tf.write(model.output())
   tempname = tf.name
os.rename(tempname, filename)

This method is more robust against errors than the truncate-write method. See below for a discussion of atomicity and consistency properties. It is used by many applications.

These first two patterns are so common that the ext4 filesystem in the Linux kernel even detects them and fixes some reliability shortcomings automatically. But don’t depend on it: you are not always using ext4, and the administrator might have disabled this feature.

写-替换

另外一种广泛使用的模式是将新内容写到临时文件,之后替换原始文件:

with tempfile.NamedTemporaryFile(
      'w', dir=os.path.dirname(filename), delete=False) as tf:
   tf.write(model.output())
   tempname = tf.name
os.rename(tempname, filename)

该方法与截断-写方法相比对错误更具有鲁棒性。请看下面对原子性和一致性的讨论。很多应用使用该方法。

这两个模式很常见,以至于linux内核中的ext4文件系统甚至可以自动检测到这些模式,自动修复一些可靠性缺陷。但是不要依赖这一特性:你并不是总是使用ext4,而且管理员可能会关掉这一特性。

Append

The third pattern is to append new data to an existing file:

with open(filename, 'a') as f:
   f.write(model.output())

This pattern is used for writing log files and other cumulative data processing tasks. Technically, its outstanding feature is its extreme simplicity. An interesting extension is to perform append-only updates during regular operation and to reorganize the file into a more compact form periodically.

追加

第三种模式就是追加新数据到已存在的文件:

with open(filename, 'a') as f:
   f.write(model.output())

这个模式用来写日志文件和其它累积处理数据的任务。从技术上讲,它的显著特点就是极其简单。一个有趣的扩展应用就是常规操作中只通过追加操作更新,然后定期重新整理文件,使之更紧凑。

Spooldir

Here we treat a directory as logical data store and create a new uniquely named file for each record:

with open(unique_filename(), 'w') as f:
   f.write(model.output())

This pattern shares its cumulative nature with the append pattern. A big advantage is that we can put a little amount of metadata into the file name. This can be used, for example, to convey information about the processing status. A particular clever implementation of the spooldir pattern is the maildir format. Maildirs use a naming scheme with additional subdirectories to perform update operations in a reliable and lock-free way. The md and gocept.filestore libraries provide convenient wrappers for maildir operations.

If your file name generation is not guaranteed to give unique results, there is even a possibility to demand that the file must be actually new. Use the low-levelos.open()call with proper flags:

fd = os.open(filename, os.O_WRONLY | os.O_CREAT| os.O_EXCL, 0o666)
with os.fdopen(fd, 'w') as f:
   f.write(...)

After opening the file withO_EXCL, we useos.fdopento convert the raw file descriptor into a regular Python file object.

Spooldir

这里我们将目录做为逻辑数据存储,为每条记录创建新的唯一命名的文件:

with open(unique_filename(), 'w') as f:
   f.write(model.output())

该模式与附加模式一样具有累积的特点。一个巨大的优势是我们可以在文件名中放入少量元数据。举例来说,这可以用于传达处理状态的信息。spooldir模式的一个特别巧妙的实现是maildir格式。maildirs使用附加子目录的命名方案,以可靠的、无锁的方式执行更新操作。mdgocept.filestore库为maildir操作提供了方便的封装。

如果你的文件名生成不能保证唯一的结果,甚至有可能要求文件必须实际上是新的。那么调用具有合适标志的低等级os.open():

fd = os.open(filename, os.O_WRONLY | os.O_CREAT| os.O_EXCL, 0o666)
with os.fdopen(fd, 'w') as f:
   f.write(...)

在以O_EXCL方式打开文件后,我们用os.fdopen将原始的文件描述符转化为普通的Python文件对象。

Applying ACID properties to file updates

In the following, I will try to enhance the file update patterns. Let’s see what we can do to meet each ACID property in turn. I will keep this as simple as possible, since we are not planning to write a complete database system. Please note that the material presented in this section is not exhaustive, but it may give you a good starting point for your own experimentation.

Atomicity

The write-replace pattern gives you atomicity for free since the underlyingos.rename()function is atomic. This means that at any given point in time, any process sees either the old or the new file. This pattern has a natural robustness against write errors: if the write operation triggers an exception, the rename operation is never performed and thus, we are not in the danger of overwriting a good old file with a damaged new one.

应用ACID属性到文件更新

下面,我将尝试加强文件更新模式。反过来让我们看看可以做些什么来满足ACID属性。我将会尽可能保持简单,因为我们并不是要写一个完整的数据库系统。请注意本节的材料并不彻底,但是可以为你自己的实验提供一个好的起点。

原子性

写-替换模式提供了原子性,因为底层的os.rename()是原子性的。这意味着在任意给定时间点,进程或者看到旧的文件,或者看到新的文件。该模式对写错误具有天然的鲁棒性:如果写操作触发异常,重命名操作就不会被执行,所有就没有用损坏的新文件覆盖正确的旧文件的风险。

返回顶部
顶部