Over the last few months, I've been thinking about and implementing transactions for Lua scripting in Redis. Not everyone understands why I'm doing this, so let me explain with a bit of history.

MySQL and Postgres

In 1998-2003 if you wanted to start a serious database driven web site/service and didn't have money to pay Microsoft or Oracle for their databases, you picked either MySQL or Postgres. A lot of people picked MySQL because it was faster, and much of that was due to the MyISAM storage engine that traded performance for a lack of transaction capability - speed is speed. Some people went with Postgres because despite its measurably slower performance on the same hardware, you could rely on Postgres to not lose your data (to be fair, the data loss with MySQL was relatively rare, but data loss is never fun).

A lot of time has passed since then; MySQL moved on from MyISAM as the default storage engine to InnoDB (which has been available for a long time now), gained full transaction support in the storage engine, and more. At the same time, Postgres got faster, and added a continually expanding list of features to distinguish itself in the marketplace. And now the choice of whether to use MySQL or Postgres usually boils down to experience and preference, though occasionally business or regulatory needs dictate other choices.

在刚过去的几个月中,我一直在构思并尝试在 redis 中实现 lua 脚本的事务功能。没有多少人理解我的想法,所以我将通过一些历史为大家做下解释。

MySQL 与 Postgres

在 1998-2003 年间,如果你想运行一个正规的数据库驱动的网站/服务,但又没有足够的资金购买微软或 Oracle 的数据库,你可以选择 MySQL 或 Postgres 。很多人都选择了 MySQL,因为它速度较快——主要是因为 MyISAM 存储引擎没有提供事务功能以此来换取性能,但速度确实很快。另一些人转向 Postgres,因为虽然在相同硬件上其性能明显低于 MySQL,但 Postgres 不会丢失数据(说实话,MySQL 数据丢失的情况非常少见,但丢了可不是闹着玩的)。

就这样凑合着过了很久;MySQL 将其默认的存储引擎从 MyISAM 过渡到了 InnoDB (其实很早就有了),这样它的存储引擎也得到了完整的事务支持和其他功能。与此同时,Postgres 也变快了,并添加了一个持续扩展的功能列表来使自己与众不同。现在对于 MySQL 与 Postgres 的选择只看个人的体验与偏好,除了有时业务需要或领导决定使用其他选择。

TL;DR; data integrity

In a lot of ways, Redis up to now is a lot like MySQL was back before InnoDB was an option. There is already a reasonable best-effort to ensure data integrity (replication, AOF, etc.), and the introduction of Lua scripting in Redis 2.6 has helped Redis grow up considerably in its capabilities and the overall simplification of writing software that uses Redis.

Comparatively, Lua scripting operates very much like stored procedures in other databases, but script execution itself has a few caveats. The most important caveat for this post is that once a Lua script has written to the database, it will execute until any one of the following occurs:

  1. The script exits naturally after finishing its work, all writes have been applied

  2. The script hits an error and exits in the middle, all writes that were done up to the error have occurred, but no more writes will be done from the script

  3. Redis is shut down without saving via SHUTDOWN NOSAVE

  4. You attach a debugger and "fix" your script to get it to do #1 or #2 (or some other heroic deed that allows you to not lose data)

To anyone who is writing software against a database, I would expect that you agree that only case #1 in that list is desirable. Cases #2, #3, and #4 are situations where you can end up with data corruption (cases #2 and #4) and/or data loss (cases #3 and #4). If you care about your data, you should be doing just about anything possible to prevent data corruption and loss. This is not philosophy, this is doing your job. Unfortunately, current Redis doesn't offer a lot of help here. I want to change that.


从很多方面来看,Redis 很像当初采用 InnoDB 前的 MySQL。而 Redis 采用了一种很合理的方式来保证数据完整性(复制,AOF 等),并且从 Redis2.6 开始引入的 Lua 脚本在功能与易用性方面为 Redis 的成长提供了很大助力。

相对来说,Lua 脚本与其他数据库中的存储过程很相似,但脚本的执行有些许不同。在本文中最重要的一点就是一旦将脚本写入数据库,它会一直执行直到以下任一种情况出现:

1. 完成所有工作,所有写操作处理完成后脚本会自动退出。

2. 脚本运行时出错并中途退出,所有以前执行的写操作都已发生,但不会再有其他写操作。

3. Redis 通过 SHUTDOWN NOSAVE 关闭时(不保存)。

4. 你附加了调试器来“使”脚本完成 #1 与 #2 (或其他手段来保证不会丢失数据)。

对于使用数据库开发软件的人,我想你也认同只有情景 #1 是最理想的。情景 #2,#3,#4 都会导致数据异常(#2 与 #4)和/或数据丢失(#3 和 #4)。如果你很重视数据,你应该尽可能地阻止数据异常与丢失。这不是哲学,而是工作(This is not philosophy, this is doing your job)。但很遗憾目前的 Redis 也帮不了你多少。所以我决定改变这种情况。

Transactions in Lua

I am seeking to eliminate cases #2, #3, and #4 above, replacing the entire list with:

  1. The script exits naturally after finishing its work, all writes have been applied

  2. The script exits with an error, no changes have been made (all writes were rolled back)

No data loss. Either everything is written, or nothing is written. This should be the expectation of any database, and I intend to add it to the expectations that we all have about Redis.

The current pull request is a proof of concept. It does what it says it does, removing the need to lose data as long as you either a) explicitly run your scripts using the transactional variants, or b) force all Lua script calls to have transactional semantics with a configuration option.

There are many ways the current patch can be made substantially better, and I hope for help from Salvatore (the author of Redis) and the rest of the community.

实现 Lua 脚本事务

我尝试解决上面列表中的 #2,#3,#4 问题,最终像下面这样:

  1. 脚本完成所有的工作,处理完写操作后正常退出

  2. 脚本执行过程中遇到错误退出,不更改任何数据(所有写操作都回滚)

无论有没有写入数据,都不会有数据丢失。这应该是所有的数据库都希望做到的,我打算把这个加到 Redis 中,因为我们都希望 Redis有 这个功能。

目前的 pull request 只是一个概念性的证明。也就是说,为了避免数据丢失,你要么 a) 显式使用事务的变体运行脚本,要么 b) 强制所有 Lua 脚本调用带配置选项的事务语义。

还有很多的办法使现在这个 patch 变得更好,我希望能得到 Salvatore (Redisw 作者)和其他社区的帮助。