Fault-Tolerance on tzimmermann dot org

Switching to Hugo

blog@tzimmermann.org (Thomas Zimmermann) — Wed, 30 Sep 2026 18:00:00 +0200

This blog has been dormant since I joined SUSE 8 years ago, although I wanted to get back to blogging for quite some time. After changing jobs, at first I simply had too much other tasks to take care of. Later when things settled down a bit, my Jekyll installation was obsolete to a point that it wasn’t usable any longer.

Jekyll has always been a bit of pain to work with. Any updates to the Ruby installation risked breaking some of Jekyll’s dependencies. Any updates to the system risked breaking Ruby itself. Whenever I tried to come back to writing, I found something was not working in my Jekyll installation. Most recently, Ruby’s Bundler installer either failed to set up a user-local installation or wanted to mess around with my Linux’ system libraries. Building Ruby’s BigDecimal module didn’t work at all. So it appeared as if I’d have to move up from mere user to Ruby developer to keep using Jekyll.

As this was not an option, I decided to convert the site to another generator. There are plenty to choose from and Hugo seemed like a good choice. It is light-weight and supports Markdown for writing. In contrast to Jekyll, it appears to be a single core program with less fragile dependencies.

The conversion still was painful. Hugo’s documentation is there; to put it in friendly terms. I found it, however, insufficient for beginners. It took me literally hours to figure how to make links work correctly or how to get data onto the final website. (Data goes into static/, not assets/.) I found the book blogdown by Xie, Thomas and Hill really helpful. The authors seem to considers it out-of-date, but it was still the best guide I could find. When setting up the site’s Atom feed, I loosely followed the description at dmvrtx.me.

After getting through the initial learning curve, things got better. I had to fix up the permalinks, adapt some of the syntax and do little changes here and there. It was also a good opportunity to weed out the last remaining JS scripts and external resources. The website should now be self-contained.

For the theme, I forked from Etch by Lukas Joswiak and applied significant modifications to page layout, fonts and page features. The original Etch is available under MIT license.

With everything in place, the resulting website is quite nice, I think. Hopefully I get back to blogging more often.

Safe Integer Conversion in C

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 20 Apr 2018 15:30:00 +0200

Integer conversion is tricky and error prone. When converting from a larger type to a smaller type, overflows and underflows can happen. For conversion of signed values, the rules are often obscure and the compiler seems to perform some dark magic in the background.

In this post, we’ll looks at the situation in C and develop a general function to cover all the corner cases. The type-conversion feature is part of picotm 0.11 and later, which also covers integration with larger software applications.

There are two central problems in integer type conversion. The first are conversions from types with wider range (i.e., more bits) to types with smaller range (i.e., less bits). Here we have to be careful to not overflow or underflow the result type.

The second problem is the conversion from signed values (i.e., negative numbers) to unsigned types, which cannot represent signs. To compare signed and unsigned types, the C compiler performs some interpretation of the values, which can lead to surprising results.

Case 1: Unsigned to Unsigned

Let’s start with something simple. Let’s cast a value of type unsigned short to unsigned char. In our example, the former type contains 16 bit, the latter contains 8 bit.

    unsigned short val_u16 = 1;
    unsigned char val_u8 = (unsigned char)val_u16;

With a value of 1, this works well. Let’s try with a different number.

    unsigned short val_u16 = 456;
    unsigned char val_u8 = (unsigned char)val_u16;

As unsigned char can only represent values ranging from 0 to 255, the casting operation cannot give a correct result. Thus we test this case and report an error accordingly.

    unsigned short val_u16 = 456;
    unsigned char val_u8;

    if (val_u16 > UCHAR_MAX) {
        report_error_and_abort(EDOM);
    }
    val_u8 = (unsigned char)val_u16;

Our test checks the value stored in val_u16 against the constant UCHAR_MAX, which represents the maximum value that a variable of type unsigned char can store. If the test fails, we stop and report a domain error with the errno code of EDOM.

The function report_error_and_abort() is provided by the application. EDOM is defined in the C standard header <errno.h>, UCHAR_MAX is defined in <limits.h>.

All our unsigned-to-unsigned conversions can be performed this way: test against the upper limit of the result type and abort if the test fails.

Conversion Ranks

There’s one question though: why does this test work? In val_u16 > UCHAR_MAX, we compare the value of a 16-bit type and the value of an 8-bit type. Why does the compiler know how to perform the comparision?

To perform the greater-than operation in the test, the compiler has to convert both operands to the same type. This controlled by the types’ conversion ranks. The conversion rank is a property of each type in C. It specifies the implicit conversions during operations, such as the > in our test.

Each type has it’s own rank, which it only shares with its corresponding signed or unsigned type. The ranks are ordered in the following way:

_Bool < unsigned char < unsigned short < unsigned int < unsigned long < unsigned long long.

If an operation receives two operands with different conversion ranks, the compiler first generates code to that converts the operand with the lesser rank to the type with the higher rank. The resulting temporary value is then used for the operation.

This implicit conversion works because of the way the C types are defined. Larger types always cover the complete range of smaller types: unsigned short covers the complete range of unsigned char, unsigned int covers the complete range of unsigned short, and so on. Converting from a lower-rank type to a higher-rank type is therefore safe.

Coming back to our example, the C compiler first converts the value of UCHAR_MAX from unsigned char to unsigned short, and then performs the test with the intermediate value.¹

Case 2: Signed to Signed

Conversions among unsigned types are simple. Luckily, the same rules apply to signed types as well. Conversion ranks among signed types are the same as for their unsigned counterparts. This means, we can do the same test for the upper limit of the result type as in the case of unsigned.

    short val_s16 = 456;
    signed char val_s8;

    if (val_s16 > SCHAR_MAX) {
        report_error_and_abort(EDOM);
    }
    val_s8 = (signed char)val_s16;

The constant SCHAR_MAX holds the maximum value of signed char. For historical reasons, the type char can be signed or unsigned; so it’s best to avoid it here in favor of an explicit signed char or unsigned char.

With signed types, we can have negative numbers. The minimum value of a signed type is stored in a constant named something like <type>_MIN. For signed char this is SCHAR_MIN. To cover cases where a value is too low for the result type, we have to extend our test a bit.

    short val_s16 = -456;
    signed char val_s8;

    if (val_s16 < SCHAR_MIN) {
        report_error_and_abort(EDOM);
    } else if (val_s16 > SCHAR_MAX) {
        report_error_and_abort(EDOM);
    }
    val_s8 = (signed char)val_s16;

We’re now also testing against SCHAR_MIN, but the overall logic remains the same.

Case 3: Unsigned to Signed

In the case of an converting a value from an unsigned type to a signed type, we have to make sure that the result is not larger than the maximum value of the signed type. Our previous code already covers these cases.

    unsigned short val_u16 = 456;
    signed char val_s8;

    if (val_u16 > SCHAR_MAX) {
        report_error_and_abort(EDOM);
    }
    val_s8 = (signed char)val_u16;

There’s really not much difference to the unsigned-to-unsigned case. The C compiler converts the constant SCHAR_MAX to unsigned short for comparing it to val_u16. Because chars are never larger than shorts the temporary value is always representable by unsigned short.

However, if we compile such code, we might get a compiler warning about mixing signed and unsigned values in the test’s condition. This requires a closer look at how signed and unsigned types are compared.

Case 4: Signed to Unsigned

Comparing signed and unsigned values is tricky, because unsigned types cannot represent negative values. Let’s take the following example.

    short val_s16 = -1;
    unsigned char val_u8;

    if (val_s16 < 0ull) {
        report_error_and_abort(EDOM);
    } else if (val_s16 > UCHAR_MAX) {
        report_error_and_abort(EDOM);
    }
    val_u8 = (unsigned char)val_s16;

Here we compare a signed 16-bit value against 0ull and UCHAR_MAX, which are both unsigned. If we’d compile and run this code, we’d be surprised by the fact that both tests succeed and that the result value of val_u8 is 255. We would have expected the first test to fail.

The exact rules for comparing signed and unsigned types in C are complex, but generally speaking, the compiler picks the operand type with the highest rank, converts both operands to the unsigned version of this type, and finally compares these intermediate unsigned operands.

In the first test, 0ull is of type unsigned long long, which has a higher rank than short. So the compiler converts the value of val_s16 to long long and interprets the resulting bits as unsigned long long. When stored as long long, the bit pattern of -1 is 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111, assuming 64-bit long long values. Interpreting this bit pattern as unsigned gives a value of 2⁶⁴-1. And suddenly -1 is larger than 0.

We can fix this problem by comparing against 0ll, a signed zero, in the first test.

    short val_s16 = -1;
    unsigned char val_u8;

    if (val_s16 < 0ll) {
        report_error_and_abort(EDOM);
    } else if (val_s16 > UCHAR_MAX) {
        report_error_and_abort(EDOM);
    }
    val_u8 = (unsigned char)val_s16;

The first test is now a signed-to-signed conversion, which is always safe to perform. The second test is also safe to perform, as the range of short can represent UCHAR_MAX.

Putting It All Together

Ideally, we’d have a single function for each conversion, something like cast_int_to_ushort(), cast_char_to_int(), cast_long_to_bool() and so on. Even better, if we could have a generator macro that creates these functions for us.

The blog post is getting rather long already, so please forgive that I’m not going to explain all the details of how to do this. Rather, I’d like to point you to the picotm source code, which contains exactly this.

The C pre-processor macro PICOTM_CAST_TX() generates conversion functions from one C type to another C type. For example, running

PICOTM_CAST_TX(uint, unsigned int, short, short, SHRT_MAX, SHORT_MIN)

produces

static inline short
cast_uint_to_short_tx(unsigned int value);

which safely casts any value of type unsigned int to type short. The function detects the signess of the input and result types, makes sure that it’s not comparing values of signed and unsigned types in the wrong places, and reports errors.

A lot of the analysis depends on the involved types and can be done by the C compiler while building the program. An optimizing compiler can easily eliminate most of the cases. For example, if both types have the same signess, all the cases for handling differences in signess are removed.

As picotm provides system-level transactions for safe and reliable programming, errors are reported to the picotm transaction manager, which will roll-back the transaction and run the error-recovery code. But this is entirely separated and unrelated to the type conversion.

Summary

In this blog post, we’ve looked at the different cases of converting integer types among each other. This includes

unsigned to unsigned,
signed to signed,
unsigned to signed, and
signed to unsigned.

There are a lot of corner cases and the blog post covers all of the tricky parts. Finally, we’ve seen a generator macro that creates conversion functions during the C compiler’s pre-processing stage. Using such a macro, a large number of safe type-casting functions can be created easily.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

C compilers don’t have to follow this procedure step-by-step. A compiler is allowed to optimize the code as long as the result is the same as in the non-optimized variant. ↩︎

System-Level Transactions with picotm

blog@tzimmermann.org (Thomas Zimmermann) — Mon, 12 Feb 2018 11:00:00 +0100

I’ll speak about picotm at this year’s Chemnitzer Linux-Tage. The conference committee asks each presenter to submit a shortpaper containing an overview of the presentation’s content. Now that I have this nice single-page overview of picotm, I decided to post a cleaned-up version here in my blog. A PDF of the text is also available.

The Problems with Traditional Code

Our goal is to write software that works reliably under all circumstances. Besides correctness at the algorithmic level, this requires correct handling of concurrency and run-time errors. Both are notoriously hard to test and implement.

int fd0, fd1; /* file descriptors */
char obuf[100]; /* output buffer */

write(fd0, obuf, sizeof(obuf));
write(fd1, obuf, sizeof(obuf));

This example program writes an output buffer to two files. If the first write() operation succeeds but the second fails, only one file will be updated and the program will enter an inconsistent, and probably erroneous, state. Correctly returning to the previous consistent state will become impossible at this point. If there’s concurrent access to the files, the program’s result will become unpredictable. Similar examples can be constructed for memory access, data structures, or any other non-constant resource.

Transactional Execution

A transaction is a transition from one consistent state to another consistent state; without the intermediate steps being visible outside of the transaction’s context. This is known as the ACID properties atomicity, consistency, isolation and durability. To achieve ACID properties, a transaction has to handle run-time errors and concurrency of its contained operations; exactly those pieces of the software that we just identified as being problematic.

Phases of each Operation

Most operations, such as write(), can be split into different phases. We call them execute, apply and undo.

When first called, an operation enters the execute phase, where it allocates resources, acquires locks and checks for potential run-time errors. If successful, the operation will enter the apply phase where its effects become globally visible. If unsuccessful, the operation will enter the undo phase, where all setup from the execute phase is reverted.

For n operations in a row, phases can be rearranged. First we perform all execute phases e_1, e_2, … e_n. If successful, we will perform all apply phases a_1, a_2, … a_n. In unsuccessful, we will revert execution with the undo phases u_n, … u_2, u_1. This scheme allows us to compose transactions from arbitrary operations.

System-Level Transactions with picotm

picotm is a system-level transaction manager implemented in highly portable C. It provides a generic framework to formalize the execute-apply-undo scheme and to integrate a large number of different modules and interfaces. All concurrency control and error detection is performed internally, users only have to implement application-specific algorithms and error recovery.

Re-implementing the example program puts all execute phases between picotm_begin and picotm_commit. Note each operation’s _tx suffix.

picotm_begin
  /* execution phase */
  write_tx(fd0, obuf, sizeof(obuf));
  write_tx(fd1, obuf, sizeof(obuf));
picotm_commit /* apply during commit */
  /* recovery phase */
  put_error_recovery_here();
  picotm_restart();
picotm_end

Execute phases perform concurrency control and error detection internally. Each operation’s apply phase is performed by picotm_commit. Until commit, all changes are local and can be reverted via the corresponding undo phase. After reverting from a detected error, the transaction enters the recovery phase where the application can attempt to repair the error and restart the transaction.

picotm Modules

The picotm core library provides the building blocks for generic error handing and concurrency control. On top of these primitives, users can implement modules for their specific use case and application. Out of the box, picotm comes with a large set of available modules, such as

Software Transactional Memory: Transactional access to shared memory
Data structures: Implements transactional lists, queues, multisets, stacks
Memory management: Provides transaction-safe C memory allocation with malloc(), free(), et al.
C string and memory operations: Provides C Standard Library functions that operate on transactional memory, such as memset(), strcpy(), et al.
File-descriptor I/O: Offers transaction-safe access to files and file-like structures on POSIX systems, with functions such as open(), close(), read(), write(), et al.
C math functions: Transactional tan(), sqrt(), et al.

All available modules co-operate and can be mixed freely. More modules are planned and will be added over time.

Obtaining picotm

picotm is available at http://picotm.org. It’s implemented in plain C and currently supports Linux, MacOS X, Windows (Cygwin) and FreeBSD. More systems are planned. The source code is distributed under the terms of the MIT license.

A picotm Demo Application

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 12 Jan 2018 10:00:00 +0100

Happy new near to you! I spent the first week of 2018 on writing a demo application for picotm. The intention is to show what picotm has to offer and how system-level transactions work in a real program. In this blog post we’ll go through it and see what it does.

The demo application reads random data from Linux, puts it into several memory buffers, and visualizes the buffers’ content on screen. This is split among multiple threads, which segment the application into three major parts: an input thread, a number of processing threads, and a (text-mode) visualization thread.

As much as possible, this functionality is implemented using picotm transactions. All the shown transaction functionality is part of picotm and ready to use out-of-the-box.

But let’s start with a screen shot. At first, all buffers are empty.

The Input Thread

First of all there’s an input thread that periodically reads data from the operating system. It does so by opening /dev/urandom and calling read() on it once per second.

Here’s the opening code. It imports the filename into the transaction, opens the file and exports a file descriptor from the transaction to main memory.

static int
open_input_file(const char* filename)
{
    int fd;

    picotm_begin

        // Privatize the memory of the argument and filename
        privatize_c_tx(filename, '\0', PICOTM_TM_PRIVATIZE_LOAD);

        // Open input file
        int tx_fd = open_tx(filename, O_RDONLY);

        // Export the file descriptor from the transaction
        store_int_tx(&fd, tx_fd);

    picotm_commit
        int res = recover_from_tx_error(__FILE__, __LINE__);
        if (res < 0) {
            return -1;
        }
        picotm_restart();
    picotm_end

    return fd;
}

Remember that we don’t require error detection in transactional code. This is done fully automatically by the transaction framework. If anything goes wrong, the framework rolls back the transaction and jumps to the recovery code that sits between the commit and end statements. From there, we can try to repair the error and restart the transaction.

Next is the reading code. Again it’s implemented as a transaction.

struct queue_entry {
    struct txqueue_entry entry;
    struct hdr msg;
};

static struct queue_entry*
read_file(int fd)
{
    struct queue_entry* entry;

    picotm_begin

        struct queue_entry* tx_entry = malloc_tx(sizeof(*entry));

        // Read message header from input stream. The value of `fd` is a
        // constant on the stack; no need to load or privatize. The first
        // 4 byte are considered meta data.
        read_tx(fd, &tx_entry->msg, 4);

        // Read data into buffer. With `read_tx()` the buffer `msg.buf` is
        // automatically privatized by the TM module.
        read_tx(fd, tx_entry->msg.buf, tx_entry->msg.len);

        // Export message from transaction context.
        store_ptr_tx(&entry, tx_entry);

    picotm_commit
        int res = recover_from_tx_error(__FILE__, __LINE__);
        if (res < 0) {
            return NULL;
        }
        picotm_restart();
    picotm_end

    return entry;
}

Not trying to bore you with details, this transaction allocates a message structure named queue, reads the random data into the structure, and exports it from the transactional context. Again, if anything goes wrong, the transaction framework detects the error for us and goes to recovery. Because transactions cleanly separate the error detection from the recovery, we can re-use the exact recovery code from the opening transaction.

We now have a packet containing random data. Each data packet is forwarded to one of multiple processing threads. The forwarding transaction is trivial (as it should be).

picotm_begin
    struct txqueue* queue = txqueue_of_state_tx(&q->queue);
    txqueue_push_tx(queue, &entry->entry);
picotm_commit
    int res = recover_from_tx_error(__FILE__, __LINE__);
    if (res < 0) {
        goto out;
    }
    picotm_restart();
picotm_end

The variable named entry is the data packet we just created, the variable named queue is a transactional queue that connects the input thread with a processing thread.

The Processing Threads

Processing threads receive data from the input thread for further processing. Each maintains an in-memory buffer, where it stores the received data.

The following code removes the input thread’s data packet from the transactional queue. This is the ownership hand-over step of the data and the memory accociated with it. Once the processing thread reads the packet, it owns it.

bool continue_loop;

do {

    continue_loop = false;

    picotm_begin

        // Acquire transactional queue for queue state and get
        // next message from queue.
        //

        struct txqueue* queue = txqueue_of_state_tx(&q->queue);
        if (txqueue_empty_tx(queue)) {
            goto commit;
        }
        struct queue_entry* entry =
            queue_entry_of_txqueue_entry_tx(txqueue_front_tx(queue));

        // Copy message buffer into correct field and fill trailing
        // bytes with 0.
        //

        uint8_t* field = buf->field[entry->msg.off];
        memcpy_tx(field, entry->msg.buf, entry->msg.len);
        memset_tx(field + entry->msg.len, 0, 256 - entry->msg.len);

        // Remove message from queue and free memory.
        //

        txqueue_pop_tx(queue);
        free_tx(entry);

        // Continue loop until queue runs empty
        store__Bool_tx(&continue_loop, true);

    commit:
    picotm_commit
        int res = recover_from_tx_error(__FILE__, __LINE__);
        if (res < 0) {
            return;
        }
        picotm_restart();
    picotm_end

} while (continue_loop);

The data is then copied into the memory buffer buf->field. The processing thread doesn’t really do much processing, but could in a real-world application. The transaction applies Transactional Memory for the copy operation (i.e., memcpy_tx() and memset_tx()). Our output thread will therefore be able to read the buffer content without interfering with the processing.

In the final step, the transaction removes the data packet from the transactional queue and frees its memory.

Again, you can see how we reuse the error recovery. Recovery strategies hardly change within an application, so the recovery code should be separate from the error detection.

At this point, we have received data from an input source forwarded it to another thread, processed it, and stored the results in main memory. A fairly common pattern in most applications.

The Output Thread

The output thread, actually the application’s main thread, visualizes the buffer content. It periodically reads from each buffer and reduces the buffer’s content to a string of hexadecimal numbers. These strings are then displayed on the screen.

Just as in the case of the processing thread, access to each buffer is performed using Transactional Memory. Conflicts among output and processing threads are resolved automatically. Over time each buffer fills with content; a process that the user can follow by watching the screen. After a while the application buffers contain data.

Summary

In this blog post, we’ve looked picotm’s demo application.

The demo application reads random data into memory buffers and visualizes the buffers’ content on the screen.
Input, processing and visualization is performed on different threads.
Transactional code is employed for communication among threads and detecting errors.

The full source code for the demo is available in a git repository on GitHub.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Porting picotm to Mac OS, Windows and FreeBSD

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 15 Dec 2017 12:30:00 +0100

I spent the previous weeks on porting picotm from Linux to a number of other operating systems. Ports to Mac OS X, Windows and FreeBSD will be available in picotm’s next release. in this blog post we go through them one by one and look at the major points and pitfalls.

In it’s current release, picotm, the system-level transaction manager, only runs on Linux systems. Supporting more operating systems was high on my TODO list for quite a while. During the previous weeks I finally got to work on ports to Mac OS, Windows and FreeBSD.

Picotm provides transactional semantics (i.e., automatic locking and error handling) for a large set of low-level functionality, such as

memory operations (transactional memory),
C memory and string functions (concurrent mem*() and str*() calls on shared memory),
memory allocation (malloc() and free() on shared memory),
file-descriptor I/O (concurrent reading and writing within the same file),
process working directory,
C math functions, and
transactional data structures (shared lists, multisets, etc).

All this functionality is implemented as a set of shared libraries, built on top of a single core library that manages all transactions in the application. It’s all C code.

The goal for each new port is to get the core transaction manager running on the new system and then get (most of) the actual functionality working.

FreeBSD

The first port was from Linux to FreeBSD. In terms of systems programming, FreeBSD is very similar to Linux. It has the same or similar tools and compilers. It’s a POSIX-compatible system and supports most of the same programming interfaces that Linux uses. For interfaces that are missing, there’s usually a replacement with equivalent functionality.

Getting FreeBSD was really easy. The FreeBSD project even provides a complete pre-configured disk image for virtual machines.

The first thing to port was the build system. Picotm uses GNU autotools, which are designed for portability among Unix systems. Many people love to hate the autotools, but they are actually quite good. I resolved an additional dependency on the realpath tool by changing the affected build scripts to work without any additional tool, so this change benefits the original Linux build scripts as well.

Next after the build system was the core transaction manager. This library is written in C11 with dependencies on the operating system’s mutexes, condition variables and time sources. FreeBSD comes with a recent version of clang, which provides a good C11 compiler. As a POSIX system, FreeBSD provides all of the required mutex, condition variables and time interfaces. So no problems here.

One major difficulty in porting picotm is the difference in supported interfaces on each operating system. If a system provides a specific interface, picotm tries to offer a transactional variant. If an interface is not available, picotm does to not offer it. For example, on Linux the GNU libc provides the non-standard function rawmemchr() for searching a character within a memory buffer. Therefore picotm provides rawmemchr_tx(), a transactional variant. The C standard library on FreeBSD does not provide rawmemchr(), so picotm doesn’t provide rawmemchr_tx().

Supporting system-specific interfaces is easy with the GNU autotools. With autoconf, the macro AC_CHECK_DECL adds a test for the declaration of a specific function or macro in a specific header file. In the source code one can test for the result and change the program’s behaviour accordingly.

For example, the package configuration scripts contain the following test for rawmemchr().

AC_CHECK_DECL([rawmemchr],
              [AC_DEFINE([PICOTM_LIBC_HAVE_RAWMEMCHR],
                         [1],
                         [Define to one of you have rawmemchr.])],
              [],
              [@%:@include <string.h>])

This test searches for a declaration of rawmemchr() in the header file <string.h>. If it find the declaration, it defines the C pre-processor token PICOTM_LIBC_HAVE_RAWMEMCHR to 1.

In the picotm source code, we can do a test and only provide a transactional implementation if PICOTM_LIBC_HAVE_RAWMEMCHR is defined to 1.

#if defined(PICOTM_LIBC_HAVE_RAWMEMCHR) && PICOTM_HAVE_RAWMEMCHR
void*
rawmemchar_tx(const void*s, int c)
{
    /* transactional variant of rawmemchr() */
}
#endif

This works fairly well and I use this scheme for all supported system interfaces of the C standard and math libraries, and the POSIX thread library. It’s not complicated, just very tedious to do this for the several hundred interfaces that are supported by picotm.

I had to add a few minor changes throughout picotm’s source code to turn non-portable C code into portable one, and that’s it! After this, the test cases succeeded on FreeBSD and I consider it supported.

Mac OS

The next system to support was Mac OS X. Like Linux and FreeBSD, it’s a Unix system and mostly compatible with the POSIX standard.

Because Mac OS only runs on Apply hardware, I bought an old Mac mini on Ebay. Setting up the system for the first time was a bit frustrating. The problem was that it’s a model from 2007 and has long been discontinued. The final version of Mac OS X that run on the device is 10.6.8, with development tools being out of date and compilers not supporting C11.

Thanks to the excellent work of the MacPorts project, a lot of the current Unix software is available on Mac OS X. So I was able to install recent releases of gcc and build tools even on this old Mac mini.

Even though Mac OS X is a POSIX system, it is more different from Linux than FreeBSD, and required another set of changes and fixes.

For example, C11 provides static assertions that are evaluated at compile time. Doing something like

static_assert(sizeof(int) == 4, "Values of type 'int' are not 4 bytes wide.");

tests if the size of an integer is 4 bytes. If the assertion evaluates to false, the compiler stops the build.

The C standard library of the Mac OS’s SDK (i.e., Xcode 3.6) does not support static compile-time assertions. In this and other cases of missing functions, I reimplemented the minimum functionality within picotm. The system’s implementation is prefered, but picotm can fall back to it’s internal code if necessary. For the missing static_assert() macro, there’s now a macro of the same name which looks like this.

#define __PICOTM_LIBC_STATIC_ASSERT(expr, stmt)                          \
    (__extension__({                                                     \
        int assertion_holds[1-(2*!(expr))] __attribute__((__unused__));  \
     }))

#ifndef static_assert
#define static_assert   __PICOTM_LIBC_STATIC_ASSERT
#endif

Note that the actual implementation is in __PICOTM_LIBC_STATIC_ASSERT. The code only defines static_assert if it’s not already defined by the system.

A similar problem happend with Pthread barriers. A barrier is a data structure for synchronizing the execution of multiple threads. It waits to be acquired by a specific number of threads until the acquiring threads can pass. In picotm, it’s used to synchronize the threads of each test case.

The implementation of a barrier is a combination of a mutex and a condition variable. All of them; mutices, condition variables and barriers; are provided by POSIX threads, but barriers are optional. While the implementation of a barrier is trivial, it’s missing at least on this old version of the Mac OS SDK. I had to implement the POSIX barrier interface within picotm, using the system’s POSIX mutices and condition variables.

Finally there was one major problem with time and clock interfaces. The POSIX standard defines the function clock_gettime(), which retrieves a time stamp from one of the system clocks. This interface is missing on Mac OS. I didn’t write a portable implementation within picotm, but instead changed the function that acquired the time stamp in the first place. On Mac OS X, it reads time stamps from non-portable interfaces, on all other systems it reads time stamps using clock_gettime.

Because of the use of system-dependent interfaces for reading time stamps, I had extend the error-handling interface as well. The Mac OS kernel returns low-level error codes in the type kern_return_t. If a time stamp could not be acquired from the system, this error code is now reported to the transaction’s error recovery phase, so that applications can act upon it.

With these changes and some additional minor fixes, the test cases completed successfully on Mac OS. Another system supported!

Windows

Windows is not Unix, so any Unix-specific interface is not natively supported. Fortunately, there is Cygwin a POSIX interface implemented on top of the Windows API.

This means that the Windows port does currently not support Windows’ native interfaces. For example, it’s not possible to perform a transaction write operation to a file using WriteFile_tx(). But using Cygwin, it is possible to achieve the same result with write_tx().

With Cygwin and all the previous fixes for FreeeBSD and Mac OS, the port itself was straight-forward. I only had to fix a few minor issues and the test cases completed successfully within a few hours of work. I’m happy to have Windows support on the feature list.

A few issues remain though. I would be interested in supporting Windows' native interfaces, but it’s a lot of work. At this point it doesn’t make sense for me to spend time on this unless there is real demand for these interfaces.

The other issue is with dynamic linking support. Picotm comes as package of multiple libraries that depend on each other. Linking a DLL requires all its dependencies (i.e., other DLLs) to be already installed. This makes linking picotm libraries as DLLs complicated. It’s not hard to fix, but the result is rather ugly and therefore left-out; and dynamic linking on Windows is disabled for now. Statically linking against picotm’s libraries is no problem.

Next Steps

The current ports lay the ground work for supporting even more operating systems. Coming from FreeBSD and Mac OS, possible next targets are other Unix operating systems, such as Android and OpenBSD.

A very interesting target is FreeRTOS, an operating system for low-power, single-board computers and the IoT. Running code transactionally has the potential of significantly increasing the reliability of these devices.

Then there are the more obscure targets, such as FreeDOS or AmigaOS. And of course, maybe one day I can port picotm to opsys, my own little operating-system project.

Summary

In this blog post, we’ve looked at the different ports that are supported by the next release of picotm.

picotm is developed mainly on Linux, but supposed to be portable to a wide range of different operating systems.
FreeBSD is very similar to Linux, so mostly minor changes were required for a port.
Mac OS X is a Unix system like Linux and FreeBSD. It required the reimplementation of a few interfaces, but it’s still mostly compatible.
Windows uses a different interface than Linux or any other Unix system. While the native interfaces require a significant amount of work to be supported, all of the actual functonality is supported by using the Cygwin compatibility layer.
With these operating systems supported, more can be added easily.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Transactional Multisets

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 24 Nov 2017 12:00:00 +0100

Besides lists, queues and stacks picotm 0.8 will feature a forth transactional data structures: multisets. A multiset is a sorted set that can contain the same entry multiple times. In this blog post, we’ll go through the interface and how to use multisets from within transactions.

The struct txmultiset data structure represents a transactional multiset (i.e., a set that can hold the same value multiple times). As all the other transactional data structures it implements concurrency control and error deteciton in order to provide ACID properties for the multiset.¹ The multiset interface resembles C++’s Standard Template Library. If you know C++ multisets, you’ll find many similarities.

Multiset Entries

To create a multiset, we first need multiset entries. These are represented by struct txmultiset_entry. It works the same way as for lists, queues and stacks. We can add an instance of this data structure to any data value to turn it into a multiset entry. Here’s an example for multisets of values of type unsigned long.

    struct ulong_item {
        struct txmultiset_entry multiset_entry;

        unsigned long value;
    };

    void
    ulong_item_init(struct ulong_item* item, unsigned long value)
    {
        txmultiset_entry_init(&item->multiset_entry);
        item->value = value;
    }

    struct ulong_item item;

    ulong_item_init(&item, 0);

This code initializes the multiset entry using txmultiset_entry_init(). The macro TXMULTISET_ENTRY_INITIALIZER initializes static or stack-allocated multiset entries. The example below illustrates this.

    struct ulong_item {
        struct txmultiset_entry multiset_entry;

        unsigned long value;
    };

    #define ULONG_ITEM_INITIALIZER(_value)  \
    {                                       \
        TXMULTISET_ENTRY_INITIALIZER,       \
        (_value)                            \
    }

    struct ulong_item item = ULONG_ITEM_INITIALIZER(0);

When both, macro and function initialization, is possible, the macro form is prefered. Multiset entries are uninitialized with txmultiset_entry_uninit().

    void
    ulong_item_uninit(struct ulong_item* item)
    {
        txmultiset_entry_uninit(&item->multiset_entry);
    }

    ulong_item_uninit(&item);

Multiset States

To store the multiset entries, we need a non-transactional multiset state, which represents the shared state of a multiset. It’s implemented by struct txmultiset_state. We can define and initialize a multiset state as illustrated in the example below.

    struct ulong_item*
    ulong_item_of_entry(struct txmultiset_entry* entry)
    {
        return picotm_containerof(entry, struct ulong_item, multiset_entry);
    }

    const void*
    ulong_item_key(struct ulong_item* item)
    {
        return &item->value;
    }

    int
    ulong_compare(const unsigned long* lhs, const unsigned long* rhs)
    {
        return (*rhs < *lhs) - (*lhs < *rhs);
    }

    const void*
    key_cb(struct txmultiset_entry* entry)
    {
        return ulong_item_key(ulong_item_of_entry(entry));
    }

    int
    compare_cb(const void* lhs, const void* rhs)
    {
        return ulong_compare(lhs, rhs);;
    }

    struct txmultiset_state multiset_state;

    txmultiset_state_init(&multiset_state, key_cb, compare_cb);

The example shows the first major difference to the other data structures. The initializer function takes the multiset state and two additional call-back functions, which are required for sorting the multiset’s entries. The key_cb() call-back function returns a pointer to the key for a given entry. In the example above, it’s the value itself. The compare_cb() call-back function compares two keys. It returns a value less than, equal to, or greater than zero if the left-hand-side value is less than, equal to, or greater than the right-hand-side value. Entries in a multiset are sorted by their key in ascending order. These call-back functions provide the key and the compare operation.

The code uses the initializer function txmultiset_state_init(). For static or stack-allocated multiset states, there’s the initializer macro TXMULTISET_STATE_INITIALIZER().

    struct txmultiset_state multiset_state = TXMULTISET_STATE_INITIALIZER(multiset_state, key_cb, compare_cb);

When both forms are possible, the initializer macro is prefered.

Multiset-state clean-up is performed by txmultiset_state_uninit(). The multiset state may not contain entries when the clean-up happens. This means that entries have to be removed from within a transaction.

For many uses this requirement just imposes unnecessary overhead. A call to txmultiset_state_clear_and_uninit_entries() provies a non-transactional way of clearing the multiset from its entries. It erases each element from the multiset and calls a clean-up function on it. The example below shows the clean-up code for a multiset of struct ulong_item entries.

    void
    ulong_item_uninit_cb(struct txmultiset_entry* entry, void* data)
    {
        ulong_item_uninit(ulong_item_of_entry(entry));

        // call free() if item was malloc()'ed
    }

    txmultiset_state_clear_and_uninit_entries(&multiset_state, ulong_item_uninit_cb, NULL);
    txmultiset_state_uninit(&multiset_state);

Acquiring a Multiset

At this point we have a multiset state and multiset entries to add to the state. So far all code was non-transactional. Actual multiset access and manipulation is performed by transactional code.

To perform multiset operations within a transaction, we first need a multiset data structure for our transaction. It’s represented by struct txmultiset. A call to txmultiset_of_state_tx() returns an instance.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

    picotm_commit
    picotm_end

Calling txmultiset_of_state_tx() multiple times for the same multiset state within the same transaction returns the same multiset. Multisets are undefined after their transaction committed or aborted, or within other, concurrent transactions.

Inserting Entries

With the multiset, we can now insert entries into the multiset state using txmultiset_insert_tx(). The example below illustrates this.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        txmultiset_insert_tx(multiset, &item->multiset_entry);

        // more transactional code

    picotm_commit
    picotm_end

After this transaction committed, the ulong data item item will be the in multiset_state. If the transactions has to abort after the call to txmultiset_insert_tx(), the transaction framework will automatically remove the entry during the rollback; thus restoring the original state.

The inserted entry will be sorted in ascending order into the multiset, using the key the compare call-back functions. The insert function compares the key of the new entry to the keys of existing entries to determines the new entry’s position. The set is implemented as a tree, so inserting requires a compare operation with only a small fraction of existing items.

Removing Entries

To remove an entry from the multiset, we can call txmultiset_erase_tx(), as illustated in the example below.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        // The multiset state already contains the entry.
        txmultiset_erase_tx(multiset, &item->multiset_entry);

        // more transactional code

    picotm_commit
    picotm_end

Removing an entry keeps the entries sorted. The multiset’s implementation re-arranges the entries automatically with very little overhead. As usual, all errors are detected and handled by the transaction framework. The benefits of transactional code show when we move entries between multisets.

    // init and ulong code here

    picotm_begin

        struct txmultiset* src_multiset = txmultiset_of_state_tx(&src_multiset_state);
        struct txmultiset* dst_multiset = txmultiset_of_state_tx(&dst_multiset_state);

        // The multiset state already contains the entry.
        txmultiset_erase_tx(src_multiset, &item->multiset_entry);

        txmultiset_insert_tx(dst_multiset, &item->multiset_entry);

        // more transactional code

    picotm_commit
    picotm_end

In this example, we removed an entry from a source multiset and inserted it into a destination multiset. The transaction framework automatically isolates these operations from concurrent transactions until the transaction commits. So concurrent transactions see the entry in either the source multiset or the destination multiset, but never in both. If the transaction has to roll back after the insert operation, the transaction framework automatically removes the multiset entry from the destination multiset and returns it to its old position in the source multiset.

Multiset Size and Emptieness

A call to txmultiset_size_tx() returns the number of multiset entries, a call to txmultiset_empty_tx() returns true if a multiset is empty. The former function might have linear complexity, the later function always has constant complexity. It’s therefore better to use txmultiset_empty_tx() if it’s only relevant whether there are entries.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        bool is_empty = txmultiset_empty_tx(multiset);

        size_t size = txmultiset_size_tx(multiset);

        // more transactional code

    picotm_commit
    picotm_end

Iterating over the Multiset’s Entries

We can iterate over the entries of a multiset. The first entry of the multiset is returned by txmultiset_begin_tx(). The terminator entry after the final entry is returned by txmultiset_end_tx(). The terminator entry is not a real entry and should not be dereferenced. Calls to txmultiset_entry_next_tx() and txmultiset_entry_prev_tx() return an entry’s successor or predecessor. Here’s an example that iterates over a multiset of values of type unsigned long and sums up the individual values.

    // init and ulong code here

    unsigned long g_sum; // global variable containg sum of multiset entries

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        unsigned long sum = 0;

        struct txmultiset_entry* beg = txmultiset_begin_tx(multiset);
        struct txmultiset_entry* end = txmultiset_end_tx(multiset);

        while (beg != end) {

            struct ulong_item* item = ulong_item_of_entry(beg);

            sum += item->value;

            beg = txmultiset_entry_next_tx(beg);
        }

        // more transactional code

        // Picotm's modules collaborate! The value stored in
        // `sum` is exported from the transaction state using
        // `store_ulong_tx()` from the Transactional Memory
        // module.

        store_ulong_tx(&g_sum, sum);

    picotm_commit
    picotm_end

Searching Entries and Ranges

Being a sorted data structure, multisets offer efficient search operations. A call to txmultiset_find_tx() looks-up an entry by a key. A multiset can contain multiple entries with the same key. In this case txmultiset_find_tx() returns one of them. The exact entry can vary among calls.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        unsigned long key = 0;

        struct txmultiset_entry* entry = txmultiset_size_tx(multiset, &key);

        // more transactional code

    picotm_commit
    picotm_end

The beginning and end of a range of entries with the same key is be obtained by txmultiset_lower_bound_tx() and txmultiset_upper_bound_tx(). The former returns the first entry with the specified key; the latter returns the entry after the final entry with the specified key. By iterating over the range, all entries with the key can be obtained.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        unsigned int key_count = 0;

        unsigned long key = 0;

        struct txmultiset_entry* beg = txmultiset_lower_bound_tx(multiset, &key);
        struct txmultiset_entry* end = txmultiset_upper_bound_tx(multiset, &key);

        while (beg != end) {

            ++key_count;

            beg = txmultiset_entry_next_tx(beg);
        }

        // ... more transactional code ...

    picotm_commit
    picotm_end

The above example counts the number of entries for the given key. The function txmultiset_count_tx() performs this operation more efficiently.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        unsigned long key = 0;

        unsigned int key_count = txmultiset_count_tx(multiset, &key);

    picotm_commit
    picotm_end

Multiset Clean-up

Finally, to clear the whole multiset at once, there’s txmultiset_clear_tx(). It’s equivalent to a continuous erase operation, but prefered for its reduced overhead.

    // init and ulong code here

    picotm_begin

        struct txmultiset* multiset = txmultiset_of_state_tx(&multiset_state);

        txmultiset_clear_tx(multiset);

        // more transactional code

    picotm_commit
    picotm_end

Summary

In this blog post, we’ve looked at transactional multisets.

Multisets are sorted sets that can contain the same entry multiple times.
Like transactional lists, queues and stacks, multisets provide concurrency control and error detection to implement the ACID properties.
A C++-like interface offers access to the multiset’s state and allows for inserting, removing, iterating and conting the contained entries.
Multiset access can be filtered by key, such that only relevant entries are processed by a transaction.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

If you’re interested in the detail of the transaction manager’s internals, you should go back and read the installments about the simpletm toy transaction manager. ↩︎

Transactional Queues and Stacks

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 17 Nov 2017 12:00:00 +0100

In last week’s blog post, we’ve examined transactional linked lists, a new feature coming in picotm 0.8. In this blog post, we are going to talk about transactional queues and stacks. Like lists, both data structures will be available in picotm at the end of November.

A Brief Recap of Transactional Lists

Like traditional linked lists, a transactional linked list is a data structure for storing data elements. In contrast to the traditional list, a transactional list can be shared by multiple threads safely and handles internal errors automatically. As the name suggests, programs use transactional lists from within a transaction, where they provide transactional ACID properties for all their operation.

Here’s an example program that uses a transactional list to store values of type unsigned long. There’s a full explanation of this code in the previous blog post,¹ so let’s just go through the most important points. If you are familiar with the interface of C++’s Standard Template Library, you’ll find many similarities.

    struct ulong_item {
        struct txlist_entry list_entry;

        unsigned long value;
    };

    struct ulong_item item = ULONG_ITEM_INITIALIZER(0);

    struct txlist_state list_state = TXLIST_STATE_INITIALIZER(list_state);

The variable item is a list entry that holds a value of type unsigned long. The value is initialized to 0 by the initializer macro. The variable list_state is the global state of a transactional linked list. It stores all entries for this list. All actual list access is performed from within a transaction.

    // Inserting transaction
    //

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        txlist_push_back_tx(list, &item->list_entry);

        // ... more transactional code ...

    picotm_commit
    picotm_end

    // ... more non-transactional code ...

The example’s first transaction inserts the entry item at the end of the transactional list. It first acquires an actual list from the list state by calling txlist_of_state_tx(). The returned txlist structure list provides the interface for operating on the list state, and it also holds the transaction-local state of the list. With the transactional list, the first transaction appends the entry to the list state by calling txlist_push_back_tx(). If other transactions access the list state concurrently, the list’s implementation detects and resolves all resulting conflicts automatically. When the transction commits, the list entry becomes part of the shared list state.

    // Iterating transaction
    //

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        struct txlist_entry* beg = txlist_begin_tx(list);
        struct txlist_entry* end = txlist_end_tx(list);

        while (beg != end) {

            struct ulong_item* item = ulong_item_of_entry(beg);

            // ... do something with local item ...

            beg = txlist_entry_next_tx(beg);
        }

        // ... more transactional code ...

    picotm_commit
    picotm_end

    // ... more non-transactional code ...

The example’s second transaction iterates over the elements of a transactional list. Like the first transaction, it acquires a transactional list from the list state. It then gets the list’s beginning and end entries, which it uses to traverse the whole list from the first to the final entry. If other transactions try to modify the list state concurrently, the list detects and resolves any resulting conflicts automatically.

    // Removing transaction
    //

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        // The list state already contains the entry.
        txlist_erase_tx(list, &item->list_entry);

        // ... more transactional code ...

    picotm_commit
    picotm_end

Finally, the example’s third transaction removes our list entry. It acquires a list from the list state and calls txlist_erase_tx() to remove the entry. As before, conflicts are detected and resolved automatically.

Transactional Queues

The list is a very generic data structure that serves a wide range of use cases. If all we want is to insert entries at one end and later retrieve them in the same order at the other end, we can use a transactional queue instead. The semantics of a queue allows for more concurrency and can thus improve performance in some situation. But first, an example.

    struct ulong_item {
        struct txqueue_entry queue_entry;

        unsigned long value;
    };

    struct ulong_item item = ULONG_ITEM_INITIALIZER(0);

    struct txqueue_state queue_state = TXQUEUE_STATE_INITIALIZER(queue_state);

    // Inserting transaction
    //

    picotm_begin

        struct txqueue* queue = txqueue_of_state_tx(&queue_state);

        txqueue_push_tx(queue, &item->queue_entry);

        // ... more transactional code ...

    picotm_commit
    picotm_end

    // ... more non-transactional code ...

As with lists, there are queue entries and shared queue states. In the example’s first transaction, we acquire a queue for the declared queue-state variable. And as with lists, the queue is the transactional interface for modifying the shared queue state. With a call to txqueue_push_tx(), we insert an entry into the queue. When the transaction commits, the entry becomes part of the shared queue state. As lists, queues detect and resolve conflicts with concurrent transactions automatically.

    // Removing transaction
    //

    picotm_begin

        struct txqueue* queue = txqueue_of_state_tx(&queue_state);

        struct txqueue_entry* item = txqueue_front_tx(queue);
        txqueue_pop_tx(queue);

        // ... do something with item ...

        // ... more transactional code ...

    picotm_commit
    picotm_end

The second transaction in the example removes the entry from the queue. It reads the next queue entry by calling txqueue_front_tx(). This is the entry we pushed into the queue in the first transaction. Then the second transaction removes the entry with a call to txqueue_pop_tx(). After the pop operation, the entry itself is still valid, so the transaction can use it’s stored value for further computation. After the successful commit of the second transaction, the entry is not part of the shared queue state any longer.

In contrast to lists, entries in a queue cannot be iterated over. Only the first or final entry can be retrieved. Furthermore, entries can only be inserted at one end and removed at the other end. These semantics allow for more concurrency an some situations.

    picotm_begin

        struct txqueue* queue = txqueue_of_state_tx(&queue_state);

        txqueue_push_tx(queue, &item->queue_entry);

        // ... more transactional code ...

    picotm_commit
    picotm_end

In this example we push an item into the queue. This is a local operation and because queues don’t allow for iteration, there’s no reason to integrate it into the global state just yet. An optimzation applied by picotm is to keep the push operation in the transaction-local state and only integrate it into the shared state during a commit. The transaction does not acquire a reader lock for the queue until it actually commits. Hence concurrent transactions can retrieve or even remove entries on the queue without being affected by our uncommitted push.

Transactional Stacks

Like queues, stacks have certain properties we can use for reducing conflicts and increasing concurrency among transactions. But let’s look at some code first. Here’s the queue example, modified for stacks.

    struct ulong_item {
        struct txstack_entry stack_entry;

        unsigned long value;
    };

    struct ulong_item item = ULONG_ITEM_INITIALIZER(0);

    struct txstack_state stack_state = TXSTACK_STATE_INITIALIZER(stack_state);

    // Inserting transaction
    //

    picotm_begin

        struct txstack* stack = txstack_of_state_tx(&stack_state);

        txstack_push_tx(stack, &item->stack_entry);

        // ... more transactional code ...

    picotm_commit
    picotm_end

    // ... more non-transactional code ...

    // Removing transaction
    //

    picotm_begin

        struct txstack* stack = txstack_of_state_tx(&stack_state);

        struct txstack_entry* item = txstack_top_tx(stack);
        txstack_pop_tx(stack);

        // ... do something with item ...

        // ... more transactional code ...

    picotm_commit
    picotm_end

Again we have entries and shared stack state. The stack is acquired with txstack_of_stack_tx() within each transaction. The first transaction in the example pushes an entry onto the stack, the second transaction retrieves the stack’s top-most entry and removes it. The interfaces are very similar to those of transactional lists and queues. Of course, each interface detects and resolves conflicts among transactions automatically.

Similar to queues, the stack allows us to reduce transaction conflicts and increase concurrency.

    picotm_begin

        struct txstack* stack = txstack_of_state_tx(&stack_state);

        txstack_push_tx(stack, &item->stack_entry);

        // ... more transactional code ...

        txstack_pop_tx(stack);

    picotm_commit
    picotm_end

In this example, the transaction pushes an entry onto the stack. As with queues, this operation is entirely local at this point. There’s no reason to acquire a writer lock on the stack state before the transaction actually commits the change. Concurrent transactions can read the stack state meanwhile.

Even more concurrency can be achieved within the pop operation. Like a push, a pop operation modifies the stack’s top. In the example above, we push an entry onto the transaction’s local stack top. When we later pop the entry, we again operate on the local stack top. So unless we pop an entry from the shared stack state, we don’t require any writer locks. If a transaction performs a number of push operations and afterwards an equal number of pop operations, effectively no change to the shared state was performed. In this case, the transaction can commit without acquiring a lock on the stack state at all.

Summary

In this blog post, we’ve looked at transactional queues and stacks.

Like transactional lists, transactional queues and stacks provide concurrency control and error detection to implement the ACID properties.
The semantics of queues and stack can be used to reduce conflicts and increase concurrency among transactions.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

If you are curious what this is all about and what transactions can do for your software, you should go back and read the installments about the simpletm toy transaction manager. ↩︎

Transactional Linked Lists

blog@tzimmermann.org (Thomas Zimmermann) — Tue, 07 Nov 2017 10:00:00 +0200

The November release of picotm will feature transactional lists. These lists can be accessed and modified concurrently by multiple transactions while respecting the transaction’s ACID properties. In this blog posts, we’re going to look at the lists’ features and interface.

The Transaction Interface

Transactional code runs isolated from other transactions in the system. All concurrency and errors are handled by a transaction manager. If anything fails the transaction framework rolls the transaction back into the original state and either retries or invokes error recovery. To the transaction it looks as if it had the whole system for itself and errors never happen. This is generally known as the ACID properties.

We can build this easily with picotm, a transaction manager for C applications and system software. Without going into details, a picotm transaction looks as shown in the example below. The transactional code is located between picotm_begin and picotm_commit.

    picotm_begin

        // Transactional code

    picotm_commit

        // Error-recovery code

    picotm_end

Earlier in this blog we had a series of articles about constructing a transaction manager in C. If you want to know how all this works at the fundamental level, you might want to go back and read the installments about the simpletm toy transaction manager.

Why Do We Want Lists?

The code in a transactions runs in its own, isolated space. To interact with the outside world (i.e., the rest of the program) it has to perform operations on shared data structurs. Such a data structure could be a double-linked list.

For an example, imagine one transaction generates list entries with values received over the network from another computer or read from a file. It enqueues the entries into a globally shared list where they are picked up by other transactions for further processing of the contained data.

The list has to be transactional to make this work. Therefore it must automatically handle concurrency among transactions and detect internal errors. On any error it has to be able to go back into its original state. In the rest of this blog article well examine how to do this in practice with the new transactional lists in picotm.

The interfaces are similar to the lists of C++ by design. You’ll probably recogonize some if you have experience with C++’ Standard Template Library.

List Entries

To create a list, we first need list entries. These are represented by struct txlist_entry. We can add an instance of this data structure to any data value to turn it into a list entry. Here’s an example for lists of values of type unsigned long.

    struct ulong_item {
        struct txlist_entry list_entry;

        unsigned long value;
    };

    void
    ulong_item_init(struct ulong_item* item, unsigned long value)
    {
        txlist_entry_init(&item->list_entry);
        item->value = value;
    }

    struct ulong_item item;

    ulong_item_init(&item, 0);

This code initializes the list entry using txlist_entry_init(). There’s also an initializer macro. TXLIST_ENTRY_INITIALIZER initializes static or stack-allocated list entries. The example below illustrates this.

    struct ulong_item {
        struct txlist_entry list_entry;

        unsigned long value;
    };

    #define ULONG_ITEM_INITIALIZER(_value)  \
    {                                       \
        TXLIST_ENTRY_INITIALIZER,           \
        (_value)                            \
    }

    struct ulong_item item = ULONG_ITEM_INITIALIZER(0);

When both, macro and function initialization, is possible, the macro form is prefered. List entries are uninitialized with txlist_entry_uninit().

    void
    ulong_item_uninit(struct ulong_item* item)
    {
        txlist_entry_uninit(&item->list_entry);
    }

    ulong_item_uninit(&item);

The Shared List State

To store the list entries we need non-transactional list state, which represents the shared state of a double-linked list. It’s implemented by struct txlist_state. We can define and initialize a list state as illustrated in the example below.

    struct txlist_state list_state;

    txlist_state_init(&list_state);

This code uses the initializer function txlist_state_init(). For static or stack-allocated list states, there’s the initializer macro TXLIST_STATE_INITIALIZER().

    struct txlist_state list_state = TXLIST_STATE_INITIALIZER(list_state);

When both forms are possible, the initializer macro is the prefered form.

List-state clean up is performed by txlist_state_uninit(). The list state may not contain entries when the clean-up happens. This means that entries have to be cleaned up from within a transaction.

For many uses this requirement just imposes unnecessary overhead. A call to txlist_state_clear_and_uninit_entries() provies a non-transactional way of clearing the list from its entries. It erases each element from the list and calls a clean-up function on it. The example below shows the clean-up code for a list of struct ulong_item entries.

    struct ulong_item*
    ulong_item_of_entry(struct txlist_entry* entry)
    {
        return picotm_containerof(entry, struct ulong_item, list_entry);
    }

    void
    ulong_item_uninit_cb(struct txlist_entry* entry, void* data)
    {
        ulong_item_uninit(ulong_item_of_entry(entry));

        // call free() if item was malloc()'ed
    }

    txlist_state_clear_and_uninit_entries(&list_state, ulong_item_uninit_cb, NULL);
    txlist_state_uninit(&list_state);

At this point we have a list state and list entries to add to the state. So far all code was non-transactional. Actual list access and manipulation is performed by transactional code.

Finally, Lists!

To perform list operations within a transaction, we first need a list data structure for our transaction. It’s represented by struct txlist. A call to txlist_of_state_tx() returns an instance.

    // init and ulong code here

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

    picotm_commit
    picotm_end

Calling txlist_of_state_tx() multiple times for the same list state within the same transaction returns the same list. Lists are undefined after their transaction committed or aborted, or within other, concurrent transactions.

Adding And Removing List Entries

With the list, we can now append or prepend entries to the list state using txlist_push_back_tx() and txlist_push_front_tx(). The example below illustrates the use of txlist_push_back_tx().

    // init and ulong code here

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        txlist_push_back_tx(list, &item->list_entry);

        // more transactional code

    picotm_commit
    picotm_end

After this transaction committed, the ulong data item item will be the final entry in list_state. If the transactions has to abort after the call to txlist_push_back_tx(), the transaction framework will automatically remove the appended entry during the rollback; thus restoring the original state.

To remove an entry from the list, we can call txlist_erase_tx(), illustated in the example below.

    // init and ulong code here

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        // The list state already contains the entry.
        txlist_erase_tx(list, &item->list_entry);

        // more transactional code

    picotm_commit
    picotm_end

As usual, all errors are detected and handled by the transaction framework. The benefits of transactional code show when we move entries between lists.

    // init and ulong code here

    picotm_begin

        struct txlist* src_list = txlist_of_state_tx(&src_list_state);
        struct txlist* dst_list = txlist_of_state_tx(&dst_list_state);

        // The list state already contains the entry.
        txlist_erase_tx(src_list, &item->list_entry);

        txlist_push_back_tx(dst_list, &item->list_entry);

        // more transactional code

    picotm_commit
    picotm_end

In this example, we remove an entry from a source list and append it to a destination list. The transaction framework automatically isolates these operations from concurrent transactions until the transaction commits. So concurrent transactions see the entry in either the source list or the destination list, but never in both. If the transaction has to roll back after the append operation, the transaction framework automatically removes the list entry from the destination list and returns it to its old position in the source list.

List Capacity

A call to txlist_size_tx() returns the number of list entries, a call to txlist_empty_tx() returns true if a list is empty. The former function might have linear complexity, the later function always has constant complexity. It’s therefore better to use txlist_empty_tx() if it’s only relevant whether there are entries.

    // init and ulong code here

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        bool is_empty = txlist_empty_tx(list);

        size_t size = txlist_size_tx(list);

        // more transactional code

    picotm_commit
    picotm_end

Iterating Over List Entries

We can also iterate over the entries of a list. The first entry of the list is returned by txlist_begin_tx(). The terminator entry after the final entry is returned by txlist_end_tx(). The terminator entry is not a real entry and should not be dereferenced. Calls to txlist_entry_next_tx() and txlist_entry_prev_tx() return an entry’s successor or predecessor. Here’s and example that iterates over a list of values of type unsigned long and sums up the individual values.

    // init and ulong code here

    unsigned long g_sum; // global variable containg sum of list entries

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        unsigned long sum = 0;

        struct txlist_entry* beg = txlist_begin_tx(list);
        struct txlist_entry* end = txlist_end_tx(list);

        while (beg != end) {

            struct ulong_item* item = ulong_item_of_entry(beg);

            sum += item->value;

            beg = txlist_entry_next_tx(beg);
        }

        // more transactional code

        // Picotm's modules collaborate! The value stored in
        // `sum` is exported from the transaction state using
        // `store_ulong_tx()` from the Transactional Memory
        // module.

        store_ulong_tx(&g_sum, sum);

    picotm_commit
    picotm_end

If we have an entry in a list, we can insert another entry before the existing one with txlist_insert_tx(). In the example below, we insert before the terminator entry, which is equivalent to an append operation.

    // init and ulong code here

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        txlist_insert_tx(list, &item->list_entry, txlist_end_tx(list));

        // more transactional code

    picotm_commit
    picotm_end

To append or prepend entries to a list, it’s better use the appropriate push functions, though. The transaction framework might be able to optimize concurrency control for each. Also, inserting or removing from a list can make iteration loops invalid. It’s therefore better to separate these operations cleanly.

Clearing The List

Finally, to clear the whole list at once there’s txlist_clear_tx(). It’s equivalent to a continuous erase operation, but prefered for its reduced overhead.

    // init and ulong code here

    picotm_begin

        struct txlist* list = txlist_of_state_tx(&list_state);

        txlist_clear_tx(list);

        // more transactional code

    picotm_commit
    picotm_end

Next Steps

The current transactional lists already provide everything we need. But of course, there’s always room for optimizations.

One possible next step is the introduction of an iterator data type that abstracts access to list entries. Such an abstraction would allow for more concurrency when operating on lists, as the order of list entries would be determined by the iterator and not the actual list order.

With iterators in place, appending and prepending to a list is another possible area of optimization. Each operation means modify the entry at either end of the list, but neither depends on where the actual end is. Each transaction can have its own local beginning and end of the list. Only during a commit would this local state become part of the global list state.

Summary

In this blog post, we’ve looked at transactional lists and how to use them.

Transactional lists provide concurrency control and error detection to implement the ACID properties.
Non-transactional list state and entries are separate from the transactional list.
All list access and modification is performed by transactions.
A transaction creates a list by acquiring global list state.
Transactions can add and remove entries in a list, retrieve the number of entries in a list, and iterate over a list’s entries.
Sophisticated list implementations can optimize concurrency control depending on each list operation’s semantics.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Scaling Lock Performance

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 20 Oct 2017 19:00:00 +0200

Release 0.7 of picotm will feature several major improvements of the locking code. In this blog post, we’re going to look at each change and how it impacts performance and scalability. We’ll go from exclusive locks to reader/writer locks to exclusive mode to a dedicated lock manager.

The Test Setup

Picotm is a transaction manager for system-level software. For example, we can write transactions that concurrently perform operations on file descriptors and file buffers, and picotm will detect reader/writer conflicts and errors for us. In the case of a conflict, one of the conflicting transactions has to roll back into its original state and retry. In the case of an error, picotm executes application-specific error handling code.

Our tools for measuring the performance is picotm-perf. It’s a simple test tool and comes with a benchmark script for running a large set of tests.

Tests within picotm-perf operate on transactional memory. Each test

begins a transaction,
performs a number of load operations from shared memory,
performs a number of store operations into shared memory, and
commits the transactions.

All load and store locations are randomized. The shared memory is a memory buffer that is used for all transactions’ loads and stores. With only 1024 bytes it’s fairly small, so conflicts are more or less guaranteed.

Picotm-perf is a synthetic benchmark; all results are only meaningful relative to each other, but they probably don’t represent real-world performance in any way. Change any parameter and the results could be entirely different.

The test environment is outlined in the PDF files with the results. Note that the test machine has 4 logical processor cores, but only 2 physical cores. The performance won’t scale linearly with the number of cores on such a system.

The Original Code of picotm 0.6

Let’s start with the original implementation that shipped with picotm’s previous release 0.6. It divides the shared memory buffer into small records of 8 bytes and locks each record with an exclusive lock. This means that two transactions observe a conflict when both access the same record concurrently; whether it’s a load or a store operation does not matter.

The benchmark results (pg. 3-8) for random memory access reflect this. Independently from the number of loads and stores, all graphs look the same. Here’s an example of a transaction with 50 loads and 50 stores.

The blue bars and the scale on the left show the number of committed transactions per second; the red bars and the right scale show the number of restarts per second.

We already mentioned the effect of exclusive locks on concurrency. For a single transaction we start with around 50,000 commits per seconds, but go down as we add concurrent transactions ot the test. And that’s even the case if we only run transactions with load operations, which don’t require exclusive locks.

Another problem that becomes obvious from the test results is the extremely high number of restarts. There are up to 4 million restarts per second!

These results gives us our first targets for improvement: enable concurrent access to memory records for transactions that only perform load operations, and reduce the number of restarts.

Adding Reader/Writer Locks

Pull request #140 on GitHub converts the transactional-memory module to use reader/writer locks. These locks are already provided by picotm and used throughout the file-I/O code to protect concurrent access to files and file buffers.

Here’s an example from the test results for reader/writer locks. The transaction again consists of 50 load and 50 store operations.

The results are two-fold. We have reduced the number of restarts by a factor of 10 to 15 to something in the 200 thousands. Unfortunately, this didn’t increase the number of commits per seconds. Quite the opposite actually. With more than one transaction present, the result drops to a few 1.000 commits per second.

An exception is the case of the load-only transactions, shown on page 8 of the PDF file. With the new reader/writer locks, reader transactions finally start scaling.

At this point it’s worth remembering that our shared memory buffer is only 1 KiB in size. With a buffer size that small, as soon as we have write operations in our transaction, conflicting access among transactions are likely.

The low number of concurrent commits is the result of the small buffer size and the interactions between concurrent transactions, competing for locks. We have hit a pathological corner case. If we could guarantee progress to some transactions, things might improve. One simple way of doing this is a limit on the number of restarts per transaction.

Adding Exclusive Mode

Exclusive mode is a variant of irrevocability. In exclusive mode, we run one single transaction exclusively. All other transactions are blocked and consequently conflicts among transcations cannot occur.

Pull request #141 on GitHub enables exclusive mode on top of the reader/writer patch set. After a transaction has reached 10 retries, picotm stops all other transactions, which now have to wait until completion of the exclusive one. The limit has been choosen arbitrarily and its optimum depends on the workload and access patterns.

The complete results are in this PDF file. Here’s an example.

The results look really promising. By limiting the number of restarts per transaction, we have reduced the overall number of restarts by factor 10 to 100,000 to 140,000 restarts per second. At the same time we increased the number of concurrent commits by a factor of 10 to up to 20,000.

On top of the scalability improvements, exclusive mode offers another important feature: transactions are now guaranteed to commit and our overall system will make progress in any case. Imagine a scenario where transactions constantly conflict with each other, restart, and conflict again without ever getting out of this pattern. This is called a livelock. With the previous implementation this was very much possible; with the new implementation not so. With this patch set, we did not just improve scalability, we resolved a whole class of concurrency problems.

Even with our current patch set we still observe 100,000 to 140,000 restarts per second. These restarts are required to avoid cyclic dependencies among transactions, as all locks held by a transaction are released if the transaction restarts. By simply never waiting, no cyclic dependencies, and therefore no deadlocks, can occur.

Our next improvement will try to reduce the number of restarts by not restarting after all. That’s what a lock manager does.

Adding a Lock Manager

The lock manager is a software component within picotm that

maintains lists of transactions waiting for each reader/writer lock, and
schedules waiting transactions when a lock becomes available.

The actual implemenation is quite complex and worth a separate blog post. The overall algorithm can be summarized like this

A transaction tries to acquire a reader/writer lock.
If the transaction fails to acquire the lock, it instructs the lock manager to make it wait for the lock to become available.
The lock manager inserts the transaction into a waiting queue for the lock.
When the lock becomes available, the unlocking transaction instructs the lock manager to select one or more of the waiting transactions and wake them up.
In the waiters
1. the woken-up transactions try again to acquire the lock.
2. If a transaction has not been woken up after a certain timeout, it wakes up automatically and a tries to acquire the lock.
Only if a transaction fails a second time to acquire the lock, it restarts; otherwise it continues as if lock acquisition had been successful on the first try.

Point 5.2 is required to avoid cyclic dependencies, and therefore deadlocks, among waiting transactions. If the waiting time has an upper bound, cyclic dependencies will be resolved as soon as the timeout has been reached and one of the transactions restarts.

Lock management gives us a new quality of locking: we now know which transaction waits for which lock, we can keep statistics about each transaction’s locking patterns, and we have a central place that coordinates locks among transactions.

At the moment the results are again two-fold. Here’s our example result.

Compared to the previous patch set, we have reduced the number of restarts by another factor of 8 to 14.000 to 16.000 restarts per second. Compared to the 4 million restarts per second with picotm 0.6 that’s an overall improvement by factor 250. Amazing!

Unfortunately, lock management actually decreased the number of commits per second significantly. The reason is in the high overhead of maintaining the waiter lists and suspending the waiting transactions, which involves a number of system calls. That’s significant compared to the fast load and store operations that are required for simply restarting.

Where To Go From Here?

picotm 0.7 will feature all of the improvements we discussed in this blog post. Reader/writer locks and the exclusive mode are definitely useful changes.

The lock manager will be available, but probably not be used for the most part. For these memory-only transactions of our test program, maintaining waiter lists and suspending transactions makes no sense, because simply restarting conflicting transactions in exclusive mode gives much better results.

The situation changes with longer transactions that may even have a higher system-wide overhead. For example, file I/O requires system calls for reading and writing data in the file buffer. The longer such a transaction runs, the more likely it is that it reaches a tiping point where waiting is the better option over restarting immediately.

Our current lock manager’s transaction scheduler always wakes up the longest waiting transaction when the lock becomes available. That’s probably not optimal in most situation. By maintaining better locking statistics for each transaction, we can make better decisions about which transaction to wake up. There are zillions of strategies and tuning parameters to explore!

Summary

In this blog post, we’ve looked at the impact of different locking patch sets to the scalability and performance of picotm’s transactions.

Exclusive locks don’t scale well, lead to a large number of conflict restarts, and don’t allow for concurrent readers.
Reader/writer locks improve concurrency in setups with many reader transactions and only few conflicts. They are not that helpful if conflicts with writer transactions are likely.
Switching transactions to an exclusive mode after they reached a limited number of restarts further improves scalability and prevents livelocks.
Lock management can significantly reduce the number of restarts, but imposes overhead on the transactions.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

When realloc() Doesn't Allocate

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 06 Oct 2017 08:00:00 +0200

I recently wrote about correct error handling for malloc(). Coincidently I came across an older defect report on the behavior of C’s realloc() function just a few days ago. In this blog post, we’re going to look at realloc()’s behavior if it’s out of memory or if the requested size is zero.

A Simple `realloc()`

In C11, new memory blocks are dynamically allocated using one of malloc(), calloc(), or aligned_alloc().

If you read this blog post, you’ve probably seen calls to these functions frequently, including calls to realloc(). A call to realloc() changes the size of a memory block while preserving the block’s existing content. Here’s a naive implementation.

void*
realloc(void* old_mem, size_t new_size)
{
    if (!old_mem) {
        return malloc(new_size);
    } else if (!new_size) {
        return old_mem;
    }

    void* new_mem = malloc(new_size);
    if (!new_mem) {
        return NULL;
    }

    size_t old_size = ; // non-portable: get size of 'old_mem' from allocator

    memcpy(new_mem, old_mem, old_size < new_size ? old_size : new_size);
    free(old_mem);

    return new_mem;
}

What does it do? First of all, we get two corner cases out of the way. If no old memory buffer is specified, realloc() shall behave like malloc(). Our first condition handles this. Then we test if the new size is not zero. If it is zero, we return the old memory. C11 with DR 400 say that successful zero-size allocations shall behave as if the size were some nonzero value […]. We already know that old_mem contains a non-NULL value, so we return this. It’s not supposed to be dereferenced after this point, though.

With the corner cases handled, we can perform the real realloc(), which is malloc()-memcpy()-free() semantically. And finally we return the new buffer, which contains the content of the old buffer.

For the memcpy() operation, we need the size of the old buffer. There’s no such functionality standardized by ISO C or POSIX, but most allocators provide an interface for this: malloc_usable_size() in the GNU libc, BSD and Bionic, _msize() on Windows, or malloc_size() on MacOS.

Handling the Returned Value

Let’s now call realloc() and see what it does. One common programming mistake is illustrated below.

void* mem = malloc(10); /* non-zero allocation */
...

mem = realloc(mem, 20);
if (!mem) {
    /* handle allocation failure */
}

This pattern can often be found, although it is incorrect. What happens if the re-allocation fails? Of course, NULL is returned. If we immediately store realloc()’s returned value in mem, the old mem buffer leaks if realloc() fails. The correct code looks like this.

void* mem = malloc(10); // non-zero allocation
...

void* tmp = realloc(mem, 20);
if (!tmp) {
    /* handle allocation failure */
}
mem = tmp; /* Only update 'mem' is it's safe to do so */

Even if realloc() failed to allocate a new buffer, we still have our pointer to the old buffer and can free the old buffer later on.

Another, although less common, mistake is to access memory out-of-bounds after shrinking a buffer. Here’s an example.

char* mem = malloc(10); /* non-zero allocation */
...

void* tmp = realloc(mem, 5); /* shrink buffer */
if (!tmp) {
    /* handle allocation failure */
}
mem = tmp;

mem[7] = 0; /* This memory location is now outside the allocated buffer! */

Remember that semantically realloc() allocates a new buffer, copies the old buffer into the new buffer, and then frees the old buffer. Depending on the requested buffer size, the actual buffer size can also shrink. Accessing memory that was allocated with the old buffer, but not with the new buffer is therefore undefined behavior.

Finally, let’s see what happens if the requested new size is zero. Quoting DR 400 again:

	`realloc(NULL, 0)`	`realloc(non-NULL ptr, 0)`
AIX	always returns `NULL`, errno is `EINVAL`	always returns `NULL`, frees `ptr`, errno is `EINVAL`
BSD	returns `NULL` on error, errno is `ENOMEM`	returns NULL on error, `ptr` unchanged, errno is `ENOMEM`
glibc	returns `NULL` on error, errno is `ENOMEM`	always returns `NULL`, `ptr` freed, errno unchanged

This is very inconsistent among older C implementations before C11. Every implementation behaves in a different way. I can’t bring myself to even try to write error-checking code that works correctly with all these variants. If you do, please send me some example code and I’ll post it here.

If we’re on C11, we’re lucky. For zero-size reallocations, DR 400 specifies that realloc() shall either return a NULL pointer to indicate an error, or that the old buffer remains in place. For all pre-C11 code that requires portability between different implementations, it’s best to write a wrapper around realloc() that handles the corner cases specifically.

void*
portable_realloc(void* mem, size_t siz)
{
    if (!mem) {
        return malloc(siz);
    } else if (!siz) {
        return mem;
    }
    return realloc(mem, siz);
}

Summary

In this blog post we looked at the corner cases of realloc().

If the old memory buffer is NULL, realloc() behaves like malloc().
If the requested size is zero, behavior is implementation defined.
On allocation errors, pre-C11 implementations of realloc() handle errno, the old memory buffer and their return value in different ways.
Portable code should either switch to C11, or wrap realloc() in a function that handles the corner cases in a portable way.
Callers must hold a pointer to the old buffer until they tested for the success of realloc(). Otherwise the application leaks the old memory buffer on allocation failures.
Calls to realloc may shrink memory buffers. Memory that was present in the old buffer, may not be available in the new shrinked memory buffer.

If you like this blog post about memory allocation, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Benchmark Visualization with LaTeX and gnuplot

blog@tzimmermann.org (Thomas Zimmermann) — Wed, 27 Sep 2017 18:00:00 +0200

Visualizing benchmark results is an important step in performance testing. In this blog post, we’re going to use LaTeX, gnuplot and Perl to convert raw performance data into a nice-looking PDF document.

We’re going to produce a script named run_benchmark.pl, which executes a set of tests and produces a PDF document with a set of nicely formated diagrams that illustrate the test results. Of course we cannot go through the script line-by-line, we all the major points are explained.

The produced PDF looks like in this example. The full source code is available in picotm’s picotm-perf repository.

Turning Raw Numbers into Actionable Data

During September your author was busy with the implementation of a new testing infrastructure for picotm’s unit tests and continuous integration.

Part of this infrastructure is a set of performance tests. Each test run a predefined workloads and generate a test result that gives information about the performance of the system. Picotm is a manager for system-level transactions, which means that the most interesting performance information is the number of transactions it’s able to perform per unit of time.

The first interesting information is the number of completed transactions per time unit. Our test constantly begins and commits transactions for a certain amount of time and measures the number of successfully committed transactions. This is the throughput measured in commits per second. The single-thread throughput mostly depends on the transaction manager’s overhead, so this information is relevant during optimization.

Occationally (or maybe frequently) the transaction manager stops, rolls back and restarts a transaction. This happens for error handling and conflict resolution. Restarting transactions can have quite a bit of overhead and minimizing the number of restarts will improve the system’s performance. Therefore the second interesting information is the number of restarted transactions per time, measured in restarts per second. Minimizing the number of restarts should have positive impact on throughput.

Our test program picotm-perf executes a number of load and store transactions in the main memory. The resulting output on the console looks like this:

0 60000 10000 0

These numbers are

the thread id. 0 as there’s only one thread in this example.
The number of milliseconds that the test ran. The equivalent of 60 seconds in this case.
The number of commited transactions on this thread. 10000.
And finally the number of restarts. There are no restarts because a single transaction cannot conflict.

This test program can be executed for various combinations of load and store operations and I/O patterns. This only depends on the command-line parameters.

Running all these combinations result is an endless list of such single-line performance results. This is unintuitive, so we have to feed the data into further processing stages.

The tools your author has picked are

Perl for processing the test-program’s information,
gnuplot for converting raw data into diagrams, and
LaTeX for assembling the individual graphs into a PDF document.

You can find the source code of the script in the picotm-perf repository. If you want to use it for other performance visualization, all you have to replace is the test program, and adopt the data parser in the Perl script and the gnuplot instructions. The rest should be independent from the actual performance tests.

Generating the Raw Data

As mentioned before, test results are generated by picotm-perf. To generate larger amounts of test data, the benchmark script run_bechmark.pl calls picotm-perf with different combinations of parameters for the number of concurrent transactions, the number of load operations per transaction, the number of store operations per transaction, and the I/O pattern. It’s a set of nested loops in Perl.

foreach my $pattern ('random', 'sequential') {

    foreach my $nloads (0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100) {

        my $nstores = 100 - $nloads;

        foreach my $ncores (1 .. 4) {

            run_picotm_perf($pattern, $nloads, $nstores, $ncores);

        }
    }
}

sub run_picotm_perf() {

    my ($pattern, $nloads, $nstores, $ncores) = @_;

    return `picotm-perf -T 60000 -P $pattern -L $nloads -S $nstores -t $ncores`;
}

The example code is mostly self-explaining. It executes the function run_picotm_perf() for all different combinations of I/O patterns, numbers of loads and stores, and number of concurrent transactions / processor cores.

The function run_picotm_perf() simply runs the external test program with the given parameters and returns the resulting output string of the format we’ve seen before.

Generating the Diagram Data

With the raw data at hand, we can now generate the information required by gnuplot to create the diagrams. The question is what our diagrams shall tell us.

We are interested in the number of commits and restarts per second for each combination of I/O pattern and loads and stores. For each of these combinations, we’re also interested in how the values change if we add or remove concurrent transactions in the workload.

So we want a single diagram for each combination of I/O pattern, loads and stores, and have the number of transactions change along each diagram’s x axis. The y axis’ display the result.

We can build this by replacing the inner-most loop of the previous example by the function generate_dat(), which is shown below.

sub generate_dat {

    my ($pattern, $nloads, $nstores) = @_;

    foreach my $ncores (1 .. 4) {

        my $result_string = run_picotm_perf($pattern, $nloads, $nstores, $ncores);

        my dat_string

        my $ncommits = 0;
        my $nretries = 0;

        my @lines = split /\n/, $result_string;

        foreach my $line (@lines) {

            # <thread id> <nmsecs> <ncommits> <nretries>
            $line =~ m/^(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s*$/ or die;

            # Sum up normalized results
            $ncommits += $3 * 1000 / $2;
            $nretries += $4 * 1000 / $2;
        }

        dat_string .= "\n" . "$ncores $ncommits $nretries";
    }

    return dat_string;
}

This code produces the data for a single diagram, but it’s probably non-obvious what happens.

First of all the function runs the test program 4 times: with 1, 2, 3, or 4 concurrent transactions. Each run produces a result that is stored in result_string.

For each test run, the result string is parsed line by line for the number of seconds that the test ran, the number of committed transactions and the number of restarted transactions. The result numbers are normalized by time and added up. The result is appended to the string dat_string in the following format.

1 123400 0
2 12300  100
3 1200   1200
4 100    12300

Each line in dat_string now contains the number of concurrent transactions, the number of committed transactions per second, and the number of restarted transactions per second. This is what we want to display in each diagram and it’s in a format that gnuplot understands.

Generating a LaTeX document

At this point we have performed a large number of tests for different combinations of I/O patterns, loads and stores, and for each combination we know how the results change when the number of concurrent transactions increases.

In principle the hardest work is done and all that’s left to do is to put the data into a nice format.¹ As mentioned before, we use LaTeX and gnuplot for the data visualization. It’s fair to say that both programs are fantastic in terms of features, but less than optimal in terms of usability. Both have a file format that is rather inelegant and breaks easily. It’s something most people write and debug once, and then try to leave alone.

The input for gnuplot is a format that describes the resulting diagram. It allows to connect the input data, which we generated in the previous step, to the individual axes, and allows for configuring colors and style of the graph. The x axis on each of our diagrams shows the number of concurrent transactions. The y axis on the left shows the number of commits per second; the y axis on the right shows the number of restarts per second. Running gnuplot on each set of input data produces the corresponding image of the diagram.

We won’t go through the details of the LaTeX document source code here, but your author likes to highlight the use of LaTeX’ gnuplot package. The gnuplot tool typically gets it input from an external file. One can run gnuplot to generate an image file from such an external file and then reference the image’s filename in the LaTeX source code.

The gnuplot package for LaTeX simplifies this significantly. It allows for storing the gnuplot diagram description directly within the LaTeX document. The gnuplot tool is run automatically by the package while processing the LaTeX document and the resulting image is included where the description was located before. This does away with all the file-name juggling. Your author appreciated the gnuplot package a lot while working on the benchmark script.

The rest of the LaTeX document is set in the standard format, which is widely documented on the web. The end result is a file named results.tex, which can be converted into a PDF document.

Generating a PDF Document

The PDF output is produced by the LaTeX tool pdflatex. Running LaTeX tools seems just as arkward as writing LaTeX documents. We have to run pfdlatex 3 times on results.tex to get a correct PDF document. The benchmark script does this internally.

Ideas for Future Improvements

Finally we have a [PDF document][site:results_pdf] named results.pdf. With picotm’s current implementation the performance numbers don’t have much meaning yet, but from here we can easily change and optimize and compare different versions against each other.

For the benchmark script itself, your author has a number of ideas.

Of course, more combinations of our test parameters are always possible. Adding these is a low-hanging fruit.

Since we already have a lot of performance data, we could archive it for future use. Then a user could point the script to a certain directory with test results and have it generate performance diagrams from these.

With the old results stored away, the script could also compute the differences between different test runs, visualize them, and automatically warn about performance regressions above a certain threshold.

Summary

In this blog post, we’ve developed a script for creating and visualizing performance data.

The actual test utility is an external program that generates performance data for different simulated workloads.
The test result is returned in the test utility’s output.
The benchmark script is written in Perl, which is well-suited for text processing.
The benchmark script runs the test program with various combinations of workload parameters and converts the resulting performance data to a format that is compatible with gnuplot.
The gnuplot tool generates images of diagrams from raw data.
The benchmark script assembles all gnuplot-generated diagrams into a LaTeX source file.
The LaTeX tool pdflatex generates the final PDF document from the LaTeX sources.

If you like this blog post, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

One could argue that collecting the data is easy, but writing LaTeX source files is hard. :p ↩︎

malloc()'s Tricky Error Reporting

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 08 Sep 2017 12:30:00 +0200

Most error checks for malloc() are incorrect. In this blog post, we’re going to look at the details of malloc()’s error reporting semantics and how to test if a call to malloc() succeeded.

The Semantics of `malloc()`

If you have seen at least a few lines of C code, you’ve probably come across malloc(), the interface for allocating memory from the operating system at runtime.

size_t size = calc_amount_of_bytes();

void* ptr = malloc(size);

In this example, the function calc_amount_of_bytes() is a placeholder that returns the number of bytes to be allocated. This memory size is given to malloc(), which then acquires a large-enough memory buffer from the operating system. Finally, malloc() returns a pointer to the beginning of the buffer.

First Try: Testing the Returned Value

In the case that malloc() failed to allocate the requested amount of memory, it returns a NULL pointer. Quoting malloc()’s man page:

The malloc() and calloc() functions return a pointer to the allocated memory, which is suitably aligned for any built-in type. On error, these functions return NULL.

Similar descriptions can be found in the ISO C Standard, the POSIX Standard and the malloc() page on MSDN.

The usual, almost ideomatic way of testing for allocation errors therefore looks like this.

size_t size = calc_amount_of_bytes();

void* ptr = malloc(size);
if (!ptr) {
    // allocation error detected!
}

And this code is… wrong.

Again, here’s what malloc()’s man page says. Note the additional sentence.

The malloc() and calloc() functions return a pointer to the allocated memory, which is suitably aligned for any built-in type. On error, these functions return NULL. NULL may also be returned by a successful call to malloc() with a size of zero […].

So if the allocation size is 0, malloc() is allowed to return NULL even though it actually succeeded. For some reason, everyone seems to ignore this additional sentence. I admit that I’m guilty of this as well.

Second Try: Testing `errno`

Just testing the returned pointer for NULL doesn’t work. My solution to this problem was to test the value of errno instead. If the allocation fails, malloc() sets the value of errno to ENOMEM before it returns NULL.

The pattern for testing malloc() errors via errno looks like this.

size_t size = calc_amount_of_bytes();

errno = 0;
void* ptr = malloc(size);
if (errno) {
    // allocation error detected!
}

Here, we clear errno to a neutral value before calling malloc(). If errno is set afterwards, an allocation error has accured.

Unfortunately, it’s not a correct solution either. The pattern worked very well on all kinds of Linux systems until I came across a malloc() implementation on one of Sony’s Android phones.

On Firefox OS, we used to have a number of system daemons for running the Android drivers separately from the main system. When reviewing patches to these daemons, I asked patch authors to use the errno pattern and avoid the test for the NULL pointer.

On Sony’s Android phone, this didn’t work. We had this pattern in our Bluetooth daemon, but Sony’s implementation of malloc() sets errno even if the allocation succeeds. The Bluetooth daemon incorrectly interpreted this as an error.

I don’t blame Sony for this problem. It was a mistake on my side, as the POSIX standard says the following about errno:

No function in this volume of POSIX.1-2008 shall set errno to 0. The setting of errno after a successful call to a function is unspecified unless the description of that function specifies that errno shall not be modified.

In other words, even in the case of success, malloc() is allowed to modify the value of errno. It’s just not allowed to set it to 0.

Third Try: Testing the Allocation Size and the Returned Pointer

This brings us back to testing the returned pointer. Since it’s only allowed to be legally NULL if the requested size is zero, we have to test specifically for this combination. This pattern looks as shown below.

size_t size = calc_amount_of_bytes();

void* ptr = malloc(size);
if (size && !ptr) {
    // allocation error detected!
}

And this is the solution to the problem. The pointer test is only allowed if we requested a memory buffer with a size greater than zero. And only then it’s an error. Allocations of zero-size buffers always succeed.

It’s important to note that the behavior for zero-size allocations is completely up to the implementation. It’s also legal that malloc() returns a pointer different from NULL. It could even set errno in this case. Our current pattern handles these implementations as well.

A nice detail about releasing the allocated memory with free() is, that free() accepts NULL as a valid argument. So it’s always possible to use a code pattern like the one shown below.

size_t size = calc_amount_of_bytes();

void* ptr = malloc(size);
if (size && !ptr) {
    // allocation error detected!
}

free(ptr);

If you know a recent system where this code fails for allocation of zero size, I’d like to hear about it.

Summary

In this blog post, we investigated different patterns for testing for failed allocations.

malloc() returns NULL on errors.
malloc() may return NULL for zero-size allocations.
Testing the returned value does not work reliably with zero-size allocations.
malloc() is allowed to modify errno during successful allocations.
Testing the returned error code does not work reliably.
Testing the returned value iff the allocation size is larger than zero works reliably on modern systems.
A NULL pointer returned by a zero-size allocation is accepted by free().

Errata

2017-10-06: C11 defect report DR 400 changes the specification for zero-size allocations to return NULL to indicate an error, or behave as if a non-zero value had been requested. So iff you’re using a conforming implementation of C11, only testing for NULL should work in all situatons.

If you like this blog post about memory allocation, please subscribe to the RSS feed, follow on Twitter or share on social networks.

The Internals of Unix Pipes and FIFOs

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 01 Sep 2017 14:30:00 +0200

There is a series of blog posts examining the details of files and file descriptors, etc on Linux and Unix. This installment continues with the details of Unix FIFOs, how they work internally, and how they are used for communicating between processes.

First-In First-Out

A recent series of posts on this blog describes the data structures surrounding Unix file I/O, such as file buffers, open file descriptions, and file descriptors; how they work together and interact with the file system. In these entries we mostly focused on regular files, which have a representation in the file system and a data buffer that is stored on a disk.

From the previous blog entries, remember how we opened a file twice and got a number of unique and shared data structures.

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

int fd1 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

In this example we opened the file /home/joe_user/my_file.txt twice and got two file descriptors, each refering to its own open file description. Both open file descriptions refer to the same file buffer, which holds the file’s content. Using a FIFO is similar to this example, but the involved buffer data structure is very different.

A FIFO (first-in first-out) is a data buffer with a write end and a read end. Data is written on one end of the data buffer, and read from the other end.

FIFOs were invented in the early 1970s by Douglas McIlroy. They were first included in Unix Version 5, released in 1974. As we’ll see below, FIFOs allow for connecting the output of one program to the input of another program. This makes them one of the building blocks of what is considered to be Unix.

Using and Creating FIFOs

Let’s first create a FIFO and see what happens. FIFOs are also called pipes, so the Unix function to create a pipe is called pipe() as well.

int pipefd[0];
pipe(pipe[fd]);

A call to pipe() creates two file descriptors, which are returned in the array that pipe() takes as its argument. In the example, the file descriptor for reading is stored in pipefd[0] and the file descriptor for writing is stored in pipefd[1].

After calling pipe(), the data structures look very similar to the case where we opened a file twice. For the purpose of illustration, process-local data structures are shown in light-blue color.

As mentioned we have created two new file descriptors. Each refers to an open file description. Each open file description of the FIFO stores whether it’s on the read end or the write end. Both open file descriptions refer to the same FIFO buffer, which the kernel just created.

Let’s write to our new pipe. The interface for writing and reading is the same as for files. Each end of a FIFO is represented by a file descriptor. A call to write() writes to the write end, a call to read() read from the read end.

const char string[] = "Hello world!";

write(pipefd[1], string, sizeof(string));

The FIFO buffer now contains the string Hello world!. We can read it using the other file descriptor.

char buf[13];
read(pipefd[0], buf, sizeof(buf));

// buf now contains `Hello world!`

Reading transfers the content of the FIFO buffer into the read buffer and removes the read data from the FIFO buffer.

Although the FIFO buffer might look similar to a file buffer at first, it works in a different way.

First of all, there is no actual buffer on the disk drive. A FIFO buffer is only contained in memory. So when the system shuts down, its content is gone.

A FIFO is a channel with a single direction and the order of data is preserved. So when we write Hello world! on the write end, we will later read Hello world! on the read end; but never world! Hello or Ehlol Rwold!.

When accessing a file buffer, we can select the location of where the read or write operation takes place. Because there is only a write end and a read end for a FIFO, it’s not possible to select the location. If we wrote Hello world! in the example, there’s no way to only read world! without reading Hello first.

Duplicating and Closing FIFOs

A file can be duplicated by simply copying it to a new location in the file system. For FIFO buffers it’s not possible to duplicate them in any similar way. All we can do is to duplicate the file descriptor using dup().

read_fd = dup(pipefd[0]);
write_fd = dup(pipefd[0]);

These duplicated file descriptors work exactly the way that the original file descriptors worked.

As with all file descriptors, the read and write ends of a pipe are closed using close(). The kernel releases the FIFO buffer after all its file descriptors have been closed.

close(read_fd);
close(write_fd);

Inter-Process Communication using FIFOs

Now that we’ve looked at the interface, let’s see what FIFOs are actually used for. By far the most common usage of FIFOs is to communicate between processes.

In the previous example, we already created a FIFO with a call to pipe(). Let’s now duplicate the process using fork().

int pipefd[0];
pipe(pipefd);

fork();

As we have seen in previous blog entries, a call to fork() only copies the process-local file descriptors, but the underlying buffers an open file descriptions are shared by the old and the new process. So our data structures now look like this.

It might seem confusing at first, but what this diagram shows is a FIFO with read and write end, and two processes that each have a file descriptor refering to either end of the FIFO. The old process’ data structures are again shown in light-blue color, the new process’ data structures are shown in orange color.

We can now communicate between processes using read() and write() as in the examples above. If we write to the FIFO in the old process

const char string[] = "Hello world!";

write(pipefd[1], string, sizeof(string));

the new process can read the data from the FIFO.

char buf[13];
read(pipefd[0], buf, sizeof(buf));

// buf now contains `Hello world!`

Currently both processes can access both ends of the FIFO buffer. In practice each process would close one end of the FIFO, such that there is one writer and one reader. For bi-directional communication, there would be a second FIFO and processes would act in opposite roles.

An Excursus on the Shell

Using FIFOs, a Unix terminal allows us connect the input and output of different programs.

cat /home/joe_user/my_file.txt | less

This command starts two programs cat and less. The call to cat reads the content of the file /home/joe_user/my_file.txt and writes it to its standard output. The call to less reads text from its standard input and display it to the user.

There’s also a pipe symbol | between them. This instructs the Unix shell to connect both programs using a FIFO. The shell creates a pipe, creates the two processes and makes the FIFO’s write end the standard output of cat. The FIFO’s read end becomes the standard input of less. After that, each process executes either the cat or the less program. Anything written by cat goes into the FIFO, and anything read by less comes out of the FIFO.

Named Pipes

With the pipes that we’ve seen so far, there has to be some kind of relationship between communicating processes. One must be a copy of the other, or both must have the same parent process. Otherwise it would not be possible for them to refer the same FIFO buffer. So if we have two unrelated processes, they cannot communicate this way.

The solution to this problem are named pipes. A named pipe is a special kind of file that has a name in the file system, but refers to a FIFO buffer.

We can create a named pipe with a call to mkfifo().

mkfifo("/home/joe_user/my_fifo", 0);

Here we create a named pipe with the name /home/joe_user/my_fifo and the default file-mode flags. It’s path is now present in the file system.

And just like with a regular file, a process opens a named pipe with a call to open().

int fd0 = open("/home/joe_user/my_fifo", O_RDONLY);

int fd1 = open("/home/joe_user/my_fifo", O_WRONLY);

After opening the named pipe at least once for reading and at least once for writing, it is usable. Because these calls to open() can happen in any process with permission to access the file path, arbitrary programs can communicate using a named pipe.

The big difference to regular files is that the name pipe refers to a FIFO buffer. There is still no file buffer on the disk and as soon as the system shuts down, any un-read content of the named pipe is lost.

Summary

In this blog post, we discussed usage and data structures of FIFOs on Unix systems.

FIFOs, also known as pipes, are data buffers with a write end and a read end.
Data written to a FIFOs write end is read from the read end in the same order.
A pipe is created using the pipe() function.
Reading and writing on a FIFO is performed with read() and write() in the same way as with regular files.
It is not possible to skip or ignore data while reading.
Pipes are used for communication between related programs.
A named pipe is a FIFO with a representation in the file system.
Names pipes can be opened by arbitrary processes and allow unrelated programs to communicate with each other.

If you like this blog post about the basics of Unix file I/O, please subscribe to the RSS feed, follow on Twitter or share on social networks.

The Strange strerror_r() of Dr POSIX and Mr GNU

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 25 Aug 2017 10:00:00 +0200

There are several variants of the C function strerror_r() that differ in their return value and error handling. This blog post describes how to support all of them in a transactional interface, while still being compatible with either internal implementation. As such, strerror_r() serves as a case study for transactional interfaces with multiple or variable semantics.

This blog post builds on previous entries about transactional C code and system-level transactions. If you missed earlier entries in the series, you might want to go back and read the installments so far.

The Old, Single-threaded Times

The original C strerror() function returns a descriptive string for each errno constant, as shown below.

char* str = strerror(ENOMEM);
printf("%s\n", str); // prints "Out of memory"

The example retrieves and prints a description of the errno code ENOMEM, which results in an output like Out of memory or a similar text.

This works well in a single-threaded program, but fails in multi-threaded software. strerror() is allowed to use an internal buffer for constructing and storing the returned string. The buffer is shared among all threads, so if multiple threads execute the function at the same time, they overwrite each other’s error strings.

The Multi-threaded Interface

The function strerror_r() is a variant of strerror that is suitable for multi-threaded environments. As typical for thread-safe interfaces of the C standard library, strerror_r() takes an additional argument that provides thread-local memory for its internal operations. In this case it’s a string buffer and the buffer length.

Rewriting our previous example with strerror_r() we get the following code.

char buf[256]; // large-enough string buffer
char* str = strerror_r(ENOMEM, buf, sizeof(buf));
printf("%s\n", str); // prints "Out of memory"

Any temporary error string is stored in the supplied memory of buf, which is owned by the current thread.

The GNU Interface

The function strerror_r() in the example returns a pointer to the error string, which is possibly stored in the memory of buf. This is the GNU definition of strerror_r().

The GNU variant returns an internal string for known errors, or copies something like Unknown error code into the provided buffer and returns the buffer’s address. In any case no error is returned.

The POSIX Interface

The GNU interface probably existed before strerror_r() was standardized by POSIX in 2001. Unfortunately the POSIX interface works differently. Here again our example.

char buf[256]; // large-enough string buffer
int err = strerror_r(ENOMEM, buf, sizeof(buf));
if (err) {
    // handle error stored in err
}
printf("%s\n", buf); // prints "Out of memory"

The variant of strerror_r() as standardized by POSIX returns an integer value that signals the success or failure of the operation. If strerror_r() succeeds, it returns a value of 0. The provided buffer now contains the error string. If it returns the constant EINVAL, the provided errno constant is unknown and the buffer is still empty. If strerror_r() returns the constant ERANGE, the provided buffer space is too small to store the error string and the caller is supposed to provide a larger buffer.

The Semi-POSIX Interface in Older glibc

In the GNU C library before release 2.13, there exists yet another variant of strerror_r(). It’s similar to the POSIX variant, but returns -1 on errors and stores the error code in errno. Using that variant, our example looks like this.

char buf[256]; // large-enough string buffer
int res = strerror_r(ENOMEM, buf, sizeof(buf));
if (res < 0) {
    // handle error stored in errno
}
printf("%s\n", buf); // prints "Out of memory"

I tried to find out more about the origins of this variant, but was not successful so far. I guess that the use of different semantics were either an oversite, or it was part of some other standard or system that pre-dates POSIX 2001.

The Transactional Interface

For using strerror_r() from within transactions, we have to provide a transaction-safe wrapper function. This function has to protect all input parameters from concurrent access and handle returned errors by starting the transaction’s recovery code.

Let’s start with providing a transactional interface that users can call. In the GNU C library, which we’ll use for the rest of this discussion, all variants of strerror_r() exist in one way or the other. To make them work, we create two internal interfaces; one for GNU and one for POSIX.

char*
__strerror_r_gnu_tx(int errnum, char* buf, size_t buflen);

int
__strerror_r_posix_tx(int errnum, char* buf, size_t buflen);

Which one is used by the caller depends on the compile-time flags of the caller’s program. The POSIX declararion is provided by defining _POSIX_C_SOURCE >= 200112L) and undefining _GNU_SOURCE in the C preprocessor. This means that if we follow POSIX and disallow GNU extenions, we get the POSIX variant, otherwise we get the GNU variant.

Our transactional strerror_r() is called strerror_r_tx(). We provide it as a C pre-processor macro that switches between the variants.

#if (_POSIX_C_SOURCE >= 200112L) && !_GNU_SOURCE
#define strerror_r_tx   __strerror_r_posix_tx
#else
#define strerror_r_tx   __strerror_r_gnu_tx
#endif

This was easy. If we follow POSIX strictly, a call to strerror_r_tx() will result in invoking __strerror_r_posix_tx(), otherwise __strerror_r_gnu_tx().

The Transactional GNU Implementation

We implement the transactional GNU and POSIX interfaces with either __strerror_r_posix_tx() or __strerror_r_gnu_tx(). These functions are part of the transaction system and we always have to provide both because we don’t know if the caller’s program is build against one or the other. Depending on the transaction system’s compile-time flags, either function’s implementation could in turn use the GNU or the POSIX implementation internally.

For simplicity, let’s strip down the example code to the minimum. Here’s the transactional GNU implementation.

char*
__strerror_r_gnu_tx(int errnum, char* buf, size_t buflen);
{
    // protect buf from concurrent access here

    // save errno here

    char* str;

#if (_POSIX_C_SOURCE >= 200112L) && !_GNU_SOURCE

    // POSIX

    char tmpbuf[256]; // large-enough buffer for all error strings

    int res = strerror_r(errnum, tmpbuf, sizeof(tmpbuf));

    // TODO: error handling

    if (buflen) {
        size_t tmplen = strlen(tmpbuf);
        str = memcpy(buf, tmpbuf, MIN(tmplen + 1, buflen));
        str[MIN(tmplen, buflen - 1)] = '\0';
    }

#else

    // GNU

    str = strerror_r(errnum, buf, buflen);

#endif

    return str;
}

In the example, we first protect the input buffer against concurrent access and save the state of errno. Other modules of the transactional system do this for us, so it’s left out here. If you’re curious, previous entries in this blog explain how it works.

If we implement __strerror_r_gnu_tx with the GNU variant, there’s nothing we have to do. Just wrap the function.

If we implement __strerror_r_gnu_tx with the POSIX variant, we execute it with the temporary buffer tmpbuf and copy the result into the caller-provided buffer. The point of using a temporary buffer is to avoid strerror_r() return the error code ERANGE. For this reason the memory in tmpbuf must be large enough to hold any string returned by POSIX' strerror_r(). When we later copy the temporary buffer to the output buffer, we can cut-off any trailing bytes that do not fit into the output buffer.

There’s one TODO comment left in the example. It’s the error handling for strerror_r(). Let’s add this.

First of all, there are three cases to handle here. A return value of 0 always signals success. A negative return value signals an error with glibc before 2.13 and the error is stored in errno. A positive return value signals an error with glibc 2.13 and later, and the error is the returned value itself.

Second, we can handle the error code of EINVAL, which signals an unknown error code, by manually creating an appropriate string. The error code ERANGE, which signals that the provided buffer is too small, is avoided by choosing the size of tmpbuf accordingly. If we see any other error, we have to run the transaction’s error recovery, as required by transactional semantics.

Let’s fill in the error handling.

static void
handle_errnum(int errnum, char* buf)
{
    if (errnum == EINVAL) {
        memcpy(buf, "Unknown error code", sizeof("Unknown error code"));
    } else {
        // invoke transactional error recovery here
    }
}

char*
__strerror_r_gnu_tx(int errnum, char* buf, size_t buflen)
{
    // protect buf from concurrent access here

    // save errno here

    char* str;

#if (_POSIX_C_SOURCE >= 200112L) && !_GNU_SOURCE

    // POSIX

    char tmpbuf[256]; // large-enough buffer for all error strings

    int res = strerror_r(errnum, tmpbuf, sizeof(tmpbuf));

    if (res < 0) { // glibc < 2.13
        handle_errnum(errno, tmpbuf);
    } else if (res) { // glibc >= 2.13
        handle_errnum(res, tmpbuf);
    }

    if (buflen) {
        size_t tmplen = strlen(tmpbuf);
        str = memcpy(buf, tmpbuf, MIN(tmplen + 1, buflen));
        str[MIN(tmplen, buflen - 1)] = '\0';
    } else {
        str = buf;
    }

#else

    // GNU

    str = strerror_r(errnum, buf, buflen);

#endif

    return str;
}

The Transactional POSIX Implementation

After building the transactional GNU implementation, we need a similar implementation for the POSIX variant __strerror_r_posix_tx. This time we implement the POSIX declaration with either the GNU implementation or the POSIX implementation itself. Therefore we have to return an integer value.

There’s one interesting corner case here. While ERANGE, the buffer-too-small signal, is technically an error, it’s not actually used as such. A transaction might want to call the function with an initial buffer, check for ERANGE, and grow the buffer until the error string fits in. So ERANGE is not an error, but a status code. That’s a valid pattern and our transactional implementation should be able to support it.

int
__strerror_r_posix_tx(int errnum, char* buf, size_t buflen)
{
    // protect buf from concurrent access here

    // save errno here

#if (_POSIX_C_SOURCE >= 200112L) && !_GNU_SOURCE

    // POSIX

    int res = strerror_r(errnum, buf, buflen);

    if (res < 0) { // glibc < 2.13
        if (errno == ERANGE) {
            return res;
        } else {
            // invoke transactional error recovery here
        }
    } else if (res) { // glibc >= 2.13
        if (res == ERANGE) {
            return res;
        } else {
            // invoke transactional error recovery here
        }
    }

#else

    // GNU

    char tmpbuf[256];

    char* str = strerror_r(errnum, tmpbuf, tmplen);
    if (str == tmpbuf) {
        // invoke transactional error recovery for EINVAL
    }

    if (strlen(str) + 1 > buflen) {
        if (glibc version >= 2.13) {
            return ERANGE;
        } else {
            errno = ERANGE;
            return -1;
        }
    }

    mempcy(buf, str, strlen(str) + 1);

#endif

    return 0;
}

As before, we first protect the input input buffer against concurrent access and save the value of errno.

If the first case, where we use the POSIX implementation of strerror_r(), we check the return value for possible errors. As before, we have to handle different versions of the glibc differently. If we see an error of type ERANGE, we return it to the calling transaction. That’s the case were the error is actually a status code. The other possible error, EINVAL, is actually an error and triggers transactional error recovery.

In the second case, we use the GNU implementation of strerror_r(). As mentioned before, the GNU code does not return errors at all. To detect ERANGE and EINVAL, we need other means.

The GNU implementation has an interesting property that we can use to our advantage: it only uses the provided buffer for unknown error codes. For every known error code it returns a pointer to an internal static string. In our example we call the GNU implementation with a temporary buffer. If the returned string pointer has the same address as the temporary buffer, GNU’s strerror_r() generated an Unknown error code message. In this case we invoke the transaction’s recovery code as if we had seen EINVAL. If the addresses differ, we got an internal static string, so the error code is a known one.

The other error, ERANGE, is detected by comparing the size of the provided buffer with the length of the error string. If the error string does not fit into the buffer, we return ERANGE to the calling transaction. The transaction can now resize the buffer and try again.

Summary

In blog post, we examined the details of implementing a transactional variant of strerror_r(), a C function with varying interfaces and semantics.¹ It serves as a case study for code with similar properties.

strerror_r() generates an error string for a given errno node. It possibly stores the string in a caller-provided output buffer.
The GNU implementation returns a pointer to the error string.
The POSIX implementation return an integer signalling errors and stores the string in the provided buffer.
The semi-POSIX impementation returns an integer signalling success or failure and stores the error code in errno.
Depending on their compile-time flags, users may call the GNU or POSIX variant. Therefore we have to provide a transactional implementation for each.
The transactional implementations have to ensure compatiblity with the semantics of the non-transactional implementations.
Some error codes, such as ERANGE, might not be errors, but allow for specific programming patterns. The transactional implementations have to support these corner cases.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

To add a personal opinion here, the only useful variant of these interfaces is the one specified by GNU. When we call strerror_r() we’re probably already in the middle of handling an error. The last thing we want is to deal with additional errors coming from the error-reporting code. To me, returning a static error string or whatever fits into the provided output buffer is acceptable and sane semantics for strerror_r(). ↩︎

File Descriptors During fork() and exec()

blog@tzimmermann.org (Thomas Zimmermann) — Thu, 17 Aug 2017 16:00:00 +0200

In the previous blog posts, we examined the relationship between files, file names, file descriptors, and open file descriptions. This time we look at what happens to file descriptors when we start a new program by calling fork() and exec().

Opening a File

As in previous examples, let’s open a file on the file system.

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

This creates a file descriptor, an open file description, and a file buffer. The following graph illustrates this, highlighting the per-process structures in light blue color. A previous blog post describes the open procedure in detail.

File Descriptors and Process Creation

Right now there’s only one Unix process in our example. New processes are created by a call to fork(). Invoking fork() duplicates the calling process, such that the new process is a copy of the caller.

What happens to the file descriptors during a fork? They are part of the process, so they are copied for the new process. So is the file-descriptor table.

Let’s call fork() in our example process.

fork();

We now have two processes and each has its own file descriptors and file-descriptor table.

The process-specific data structures of the original process are still displayed in a light blue color. The new process’ data structures are displayed in orange color. The file descriptor has been duplicated and refers to the new process’ file-descriptor table. The file descriptor table itself is part of the pre-process state and has also been copied by the call to fork().

An important detail of the design of Unix is that the open file description and the file buffer are are kernel data structures. Therefore they are not duplicate by fork(). Consequently, whenever one of the processes reads from or writes to it’s local file descriptor, the effects in the open file description and file buffer are seen by the other process.

On the other hand, operations that work solely on file descriptors, such as close() or dup() don’t effect the other process. So if either process closes its file descriptor, the other process doesn’t notice.

Executing a Different Program

After a call to fork(), both the original and the new process continue execution at the next instruction after fork(). To really execute a different program, a process invokes execl().

The execl() system call instructs the kernel to replace the user-space program of a process with the program stored in an executable file. A call to execl() takes the path of the executable file and a list of command-line arguments for the new program.

Let’s call execl() in the new process that we just created with fork().

execl("/bin/my_executable", NULL);

Our new process will now abandon the original program and execute the program stored in /bin/my_executable.

What does this mean for the process’ file descriptors and file-descriptor table? Nothing! They are still in place and working as before. Remember that the user-space file descriptor is really just an integer index into the file-descriptor table. If the new program reads and writes using the existing file-descriptor value, it reads and writes the file opened by the old program. So even after executing a new program, our diagram looks as before.

The good thing about this is that it’s a nice, generic way of communicating between programs. For example, when starting a new program from the Unix terminal, the terminal sets up the program’s default input and output file descriptors to refer to what the terminal itself uses; let’s say keyboard and text screen. If instead the terminal uses a serial port for input and output, a new program started from the terminal also uses the serial port. This way the new program inherits these settings from the terminal. And likewise, as a user we can direct a program’s default input and output file descriptors to anything that looks like a file, such as pipes, sockets or regular files.

Unfortuantely, there is also one major drawback of this whole design. It’s too easy to leak file descriptors into a newly executed program. That’s what happened here.

The only standardized file descriptors are those for default input and output, and the one for error reporting. They are the file descriptors 0, 1 and 2 respectively. The newly executed program does not know about any other file descriptors opened by the process’ old program.

This is bad for two reasons. First of all it unnecessarily consumes resources. The maximum number of file descriptors per process is limited; typically to 1024. A program can run out of available file descriptors quickly if it doesn’t close file descriptors after their final use.

An even more sever problem is that the old program can leak information into the new program that the new program is not supposed to see. A file descriptor might refer to a file with sensitive data that only the original program was supposed to access. After the call to execl() the new program would be able to read this data as well.

To prevent this, file descriptors have to be closed before the new program gets loaded by the kernel. Fortunately, there’s a simple way of ensuring this constraint: the O_CLOEXEC flag.

At the very beginning of this blog entry, we opened a file with the following code fragment.

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

By or-ing O_CLOEXEC into the flags argument, the returned file descriptor will be closed before the process, or any forked process, starts executing a new program.

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT|O_CLOEXEC, S_IRUSR|S_IWUSR|S_IRGRP);

How would our example look if we opened the file like this? During the fork the file descriptor and file-descriptor table are copied into the new process. The file descriptor carries the O_CLOEXEC flag in its entry in the file-descriptor table, but nothing else changed.

The big change comes when the new process executes execl(). Before loading the new program into the process, the kernel goes through the process' file-descriptor table and closes all file descriptors that have the O_CLOEXEC flag set. Afterwards the resulting file data structures look like this.

The O_CLOEXEC flag can also be set and cleared with a call to ~~fstat()~~ fcntl()¹ after the file descriptor has been created. But setting it later creates the possibility of meanwhile calling execl() and leaking file descriptors.

In practice, the safest strategy is to always set O_CLOEXEC when a new file descriptor gets created; and explictly clear the flag right before the call to execl() for those file descriptors that are supposed to be handed over to the new program.

Summary

In this blog post, we examined the behavior of Unix file-I/O data structures during fork() and execl().

A call to fork() duplicates the calling porcess.
During the fork operation, the kernel copies the process’ file descriptors and file-descriptor table, but not the underlying open file descriptions and file buffers.
Processes can communicate with each other by using the shared open file descriptions and file buffers.
A call to execl() starts a new program within the calling process.
By default, executing a new program has no effect on file-I/O data structures. The new program will have access to all file descriptors, open file descriptions and file buffers left open by the old program.
Leaving file descriptors open before executing a new program can leak resources and/or sensitive information to the new program.
Creating a file descriptor with the flag O_CLOEXEC marks the file descriptor to be closed when a new program gets executed. This automatically prevents file-descriptor leaks.
File descriptors should always be created with O_CLOEXEC and the flag should be cleared explictly before executing a new program if required.

If you like this blog post about the basics of Unix file I/O, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

Edit: The text originally said that fstat() updates a file descriptor’s O_CLOEXEC flag. The correct function is fcntl(). The command parameter is named FD_CLOEXEC. ↩︎

picotm at FrOSCon 2017

blog@tzimmermann.org (Thomas Zimmermann) — Wed, 09 Aug 2017 15:00:00 +0200

I’m very excited to speak about picotm at this year’s FroSCon conference! The presentation is on August 19, 2pm. I’ll cover the use cases of picotm, some programming, and some internals. I look forward to see you there!

Unix Hard Links, Soft Links, and Files

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 04 Aug 2017 10:00:00 +0200

This blog post describes the relation ship between file names and files on Unix file systems. We’ll also look at soft links and how all this interacts with file descriptors.

Creating a New File

Let’s create a new file my_file.txt our application. We do this by calling open() with the file’s name.

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

As we’ve seen in the previous blog post this creates a file descriptor, an open file description, and a file buffer. You should read that entry if you are interested in the relationship among file descriptors, open file descriptions and file buffers.

In the file system, there’s now a file name that refers to the file buffer we just opened.

A file name that refers to a file buffer is also called hard link. There can be multiple links to the same file buffer. Or in other words, the same file buffer can be made available under different names.

Adding Hard Links

An additional name for a file is created with the command link(). The function takes an existing file name and a new file name. If successful, the new name refers to the existing file name’s buffer.

Let’s create an additional hard link of the name /home/joe_user/my_hard_link.txt.

link("/home/joe_user/my_file.txt", "/home/joe_user/my_hard_link.txt");

Our file system now looks like this.

What happens if we open the new file name from within our application?

int fd1 = open("/home/joe_user/my_hard_link.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

It’s the same file buffer as before, so the result is the same as if we’d opened the file by the original name.

The application has opened two file descriptors, two open file descriptions, and they both refer to the same file buffer.

This is also the reason why we cannot go through all files on a file system to sum up how much space they consume on the disk. Since multiple file names can refer to the same file buffer, we could calculate a cumulative size that might be larger than the actually consumed size; maybe even larger than the maximum size of the disk.

Unlinking, Closing, and the Removal of File Buffers

A call to unlink() removes a file name from the file system. Let’s call unlink() on both file names and the file system will be empty.

unlink("/home/joe_user/my_file.txt");
unlink("/home/joe_user/my_hard_link.txt");

Once the final link is gone, there’s no way to create a new link to the file buffer. But the application still has these files open! How does that work?

For each file buffer, Unix maintains two counters: the link counter and the open counter.

The link counter contains the number of links refering to a file buffer. After being created, each file starts with a link count of 1. When we added a hard link with link() the counter went up by one. When we removed a hard link with unlink() the counter went down by one.

The open counter countains the number of times the file buffer was opened, or more specifically the number of open file descriptions refering to the file buffer. When a file buffer is not opened, the open counter is 0. Each time we opened a file buffer with open() that counter went up by one. Only when we later close the file descriptors and the open file descriptions get removed, the counter goes down.

When both, link counter and open counter, reach zero at the same time Unix removes the file buffer from the disk.

Let’s close both file descriptors to remove the file buffer from the disk.

close(fd0);
close(fd1);

Soft Links

Hard links have one major drawback, which is that they only work on the file system that also maintains the buffer. So if you have one partition for your computer’s operating system, an another partition for your personal data, it’s not possible to create a hard link from one file system to the other.

Unix file systems provide a solution called soft links, also called symbolic links or symlinks. Although it has a file name, a soft link is not a regular file, but a special file that stores a full file path. When an application opens the soft links, it actually opens the file at the path that is stored in the soft link. Because a file path is stored (instead of a reference to a file buffer), it’s independent from the underlying file system.

Let’s go back to the example of /home/joe_user/my_file.txt. We can create a soft link with a call to symlink(). Like link() before, it takes the original file’s path and the path of the symlink.

symlink("/home/joe_user/my_file.txt", "/home/joe_user/my_soft_link.txt");

With the soft link in place, the file system looks like this.

We have a file name my_file.txt, which refers to a file buffer. And we have a file name my_soft_link.txt, which refers to the soft-link file, which in turn refers to the original file name.

The existance of a soft link does not have any impact on the actual file, or that file buffers’s link counter. So a file can be removed, even if it is the target of a soft link. If this happens, the soft link goes stale and trying to open it returns an error.

Summary

In this blog post, we went through a tour of how hard links and soft links work on Unix, and how they affect the life time of file buffers.

Initially each file buffer is referenced by a file name.
The file name is called hard link.
There can be multiple hard links ot the same file buffer.
Opening any of a file buffer’s hard links opens the file buffer.
A file buffer can not be opened any longer after all hard links have been removed.
A file buffer is removed from the disk after all hard links have been removed and the file buffer is not opened by any application.
Hard links don’t work across file-system boundaries.
Soft links are a special type of file that reference other files by their path names.
Opening a soft link opens the referenced file.
Soft links can reference files on other file systems.

If you like this blog post about the basics of Unix File I/O, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

Data Structures of Unix File I/O

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 28 Jul 2017 10:00:00 +0200

This blog post describes the basic data structures involved in file I/O on Unix and Linux. We’ll go through file descriptors, open file descriptions and file buffers, and see how they relate to each other.

Over the last few weeks I’ve been reworking and cleaning up the transactional file I/O in picotm’s libc component. It occures to me that while Unix systems dominate the operating-system landscape, many details of how they actually work are not widely documented¹ and consequently not widely understood. With programmers coming from backgrounds other than C and Unix we now get to hear stories like this one.

In this blog post, we’ll look at what happens when an application opens and closes file descriptors on a Unix system. Whether it calls from C source code, Node.js, or any other Unix-like interface is not really important.

File Descriptors, Open File Descriptions, and Regular Files

On Unix systems, such as Linux, the BSDs, or MacOS, there are three major data structures that are relevant to file I/O. These are file descriptors, open file descriptions, and the file buffers themselves.

A call to open() takes the name of the file and the file-access flags that describe whether the file should be opened for reading or writing, etc. If a new file might be created, an optional third argument specifies the new file buffer’s file mode.

Let’s look at an example by calling

int fd0 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

We open the file buffer associated with the filename /home/joe_user/my_file.txt for reading and writing (i.e., O_RDWR). If no file buffer with this name exists, the kernel creates a new one (i.e., O_CREAT). The new file is readable by the process’ user and group, and writable only by the process’ user (i.e., S_IRUSR|S_IWUSR|S_IRGRP).

If we assume that a process has no files open yet, calling open() on /home/joe_user/my_file.txt creates the following hierarchy of these data stuctures.

Let’s start with the file buffer. The file buffer is represented by File buffer no. 0 in our example. It’s the raw data stored in the regular file.

From now on we’ll also refer to file buffers as regular files. The term regular file is Unix-speak for a data buffer that is located on a file system and represented by a name in the file-system tree. The byte offset for reading from and writing to a regular file is freely choosable. Unix file I/O also knowns about multiple types of special files, directories, sockets, multiple types of memory objects, and more. A regular file is what we’d commonly expect to be a file.

Next, the kernel creates a new open file description, labeled Open file description no. 0. in the example. The open file description stores all state related to the opened file. That is the file-access flags supplied with open() and the byte offset where the next read or write operation takes place. In our example, the file-access mode is set to readable/writable and the initial byte offset is 0. The open file description refers to the file buffer we just retrieved from the file system.

The file buffer and the open file description are system-wide resources. Given the correct sequence of operations, multiple processes can get a hold on either of them and share their state.

To do so, each process maintains a so-called file-descriptor table. As the name suggests, it’s a table of all file descriptors of the process. A file descriptor is a process’ means of refering to an open file description. Each file descriptor refers to an open file description, which in turn refers to a file buffer.

The kernel always allocates the lowest available entry in the file-descriptor table and associates it with the open file description it just created. In the case of our invocation of open() this is entry 0.

Finally, open() returns the table index of the allocated file descriptor to the process’s user-space application. Using the returned index, the application can access the regular file via read() or write(), etc. So if we now do

write(fd0, "Hello world!", strlen("Hello world!")); // write 12 byte

we write Hello world! to the beginning of File buffer no. 0 and increment the byte offset of Open file description no. 0 by 12 byte.

There’s a bit of confusing terminology here. When talking about the user-space integer value, one usually speaks of a file descriptor. But the actual file descriptor is in the process’ file-descriptor table in the kernel. The user-space value is just an index into this table.

Opening The Same File Multiple Times

Let’s open the file a second time.

int fd1 = open("/home/joe_user/my_file.txt", O_RDWR|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP);

Our hierarchy of data structures now looks like this.

We already have a file buffer for the regular file with the name of the user’s my_file.txt file. The Unix kernel uses this buffer for the second call to open(). The O_CREAT flag is only relevant if the regular file does not yet exist.

Every call to open() creates a new open file description. It’s labeled Open file description no. 1 in our example. As before it’s initialized with a byte offset of 0 and the file-access flags are taken from open().

For the new open file description, a new entry in the file-descriptor table is allocated and its index is returned to the user-space application. It’s called File descriptor no. 1 in our example.

Our application now has two file descriptors that refer to the same file buffer, but these file descriptors don’t share any state besides the file buffer’s content. So writing with fd1 starts at byte offset 0! Doing

write(fd1, "Salut monde!", strlen("Salut monde!"));

effectively overwrites the output we already wrote using fd0. The file buffer now contains Salut monde! and the byte offset of Open file description no. 1 is at position 12.

Duplicating File Descriptors

We can also have file descriptors that share their open file descriptions. For this we use dup(). A call to dup() duplicates a file descriptor, but reuses the open file description.

Let’s duplicate fd1.

int fd2 = dup(fd1);

This will setup our data structures as shown below.

The call to dup() allocates a new entry in the process’ file-descriptor table and copies File descriptor no. 1 to the new entry. The result is the new File descriptor no. 2. Its table index is returned to the process' user-space application.

In contrast to the previous example of opening a file twice, this time both file descriptors refer to the same open file description, which is Open file description no. 1.

Let’s write using both, fd1 and fd2.

write(fd2, " Ca", strlen(" Ca")); // write 3 byte
write(fd1, " va?", strlen(" va?")); // write another 4 byte

Both file descriptors share the same open file description, writing with either therefore modifies the same byte offset. From the previous write operation, the byte offset is already at position 12. We write another 3 byte using ‘fd2’ and yet another 4 byte using fd1. Our file buffer now contains the output Salut monde! Ca va? and the byte offset in Open file description no. 1 is at position 19.

For a final exampleon writing, let’s again write using fd0. File descriptor no. 0 refers to Open file description no. 0. Last time we used it for writing Hello world! and the open file description’s byte offset is still at position 12. That’s where the next write() operation starts

write(fd0, " How's it going?", strlen(" How's it going?")); // write 16 byte

This command writes 16 byte at the offset of 12, effectively overwriting the previous output at this location. Our file buffer now contains the output Salut monde! How’s it going?.

Closing File Descriptors

Our process has created 3 file descriptors. Let us close fd1. Closing is performed by calling close().

close(fd1);

Here’s how our hierarchy of data structures looks now.

We can see that File descriptor no. 1 is not available to the application code any longer and the entry has been removed from the file-descriptor table.

We always close file descriptors, not open file descriptions or file buffers. Therefore the open file descriptions and the file buffer are still there! Each is still referenced and available for use with File descriptor no. 0 or File descriptor no. 2.

Now let’s close fd0 as well.

close(fd0);

As with the previous close operation, the file descriptor and the entry in the file-descriptor table are gone. And because no other file descriptor references Open file description no. 0, the kernel deletes this open file description and releases the memory. The file buffer File buffer no. 0 is still there, as it’s still referenced by Open file description no. 2.

If we would close the final file descriptor File descriptor no. 2 as well, the kernel would release all remaining data structures including File buffer no. 0.

Summary

This blog post describes the basic data structures of Unix file I/O.

The content of regular files is stored in file buffers.
Opening a regular file opens or creates the file’s file buffer.
Each call to open() creates a new open file description that stores the file-access flags and the byte offset for the next read or write operation.
Each open file description refers to a file buffer.
Processes interact with open file descriptions and file buffers by using file descriptors.
Each process contains its own table of file descriptors.
File descriptors are duplicated using dup().
Duplicated file descriptors share the same open file description.
File descriptors are closed using close().
The kernel releases the open file descriptions and file buffers automatically when they are not longer needed.

If you like this blog post about the basics of Unix File I/O, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Maybe I turn this into a little series of blog posts. There’s interesting stuff happening during process forks; when using sockets; or in other interfaces that look like files, but aren’t.

Footnotes

There are a number of good books on Linux and Unix programming. If you’re interested in Linux system programming I’d recommend Michael Kerrisk’s The Linux Programming Interface. ↩︎

Implementing Fault-Tolerant Software With Transactions

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 21 Jul 2017 11:00:00 +0200

This is a series of blog posts to build a transaction manager in C. Last time, we looked at fault tolerance, and how to build fail-safe transactional software. In this installment, we’re going to implement support for error detection and recovery in our transaction manager simpletm. As usual, the complete example code for this blog post is available on GitHub.

If you missed earlier entries in the series, you might want to go back and read the installments so far. You might especially want to read the previous installment, which in detail lays the foundation for transactional error handling.

A Brief Recap

Here’s a very brief recap of the previous blog post. In this installment, we had the following example program written in C.

enum result {
    failure,
    success
};

enum result
func()
{
    if (call_1() == failure) {
        goto err_call_1;
    }
    if (call_2() == failure) {
        goto err_call_2;
    }
    if (call_3() == failure) {
        goto err_call_3;
    }

    return success;

err_call_3:
    revert_call_2();
err_call_2:
    revert_call_1();
err_call_1:
    return failure;
}

void
repair_func()
{
    /* repair errors of func() */
}

int
main(int argc, char* argv[])
{
try_func:
    if (func() == failure) {
        repair_func()
        goto try_func;
    }

    return EXIT_SUCCESS;
}

The function main() calls func() until func() signals success. In func() itself, there are 3 calls to the functions named call_1(), call_2() and call_3(). If all invocations succeed, func() signals success; if either invocation fails, func() reverts the already completed calls and signals failure. In case of a failure, main() tries to repair the error by calling repair_func(). After completing the repair, main() restarts func().

The revert-repair-and-retry scheme is a typical pattern in C software. It resembles the rollback-and-retry pattern found in transactions. If we add the repair functionality to the transaction manager, we can write the code as a transaction.

void
repair_func()
{
    /* repair errors of func() */
}

int
main(int argc, char* argv[])
{
    tm_begin

        /* execution phase */

        call_1_tx();
        call_2_tx();
        call_3_tx();

    tm_commit

        /* recovery phase */

        repair_func();
        tm_restart();

    tm_end

    return EXIT_SUCCESS;
}

The functions call_1_tx(), call_2_tx() and call_3_tx() are now invoked during the transaction’s execution phase. They internally test for errors and roll back the transaction if they detect one. This functionality is implemented by the transaction manager, so there’s nothing that the application programmer has to add.

After an error has been detected and the transaction manager performed the rollback, it restarts the transaction in the recovery phase. During recovery, the application can try to repair the error and then restart the transaction’s execution phase.

You can already see how much cleaner the transactional implementation is. Everything that can be automated is handled by the transaction manager’s framework. The only code that has to be written by the application programmer is the recovery phase, as it’s specific to the application and use case.

Additional Interfaces for Recovery

For the implementation, we’re going to modify our standard example of producer and a consumer transactions. Let’s start with the transaction manager’s public interfaces.

So far, our transactions have been contained by tm_begin and tm_commit

#define tm_begin                            \
    _tm_begin(setjmp(_tm_get_tx()->env));   \
    {

#define tm_commit                           \
        _tm_commit();                       \
    }

When used in application code and expanded by the C preprocessor, they form a block that looks this.

_tm_begin(setjmp(_tm_get_tx()->env));
{

    /* execution phase */


    _tm_commit();
}

The code for our new recovery phase is located between tm_commit and the new tm_end macro. To run either the execution phase or the recovery, we implement the tm_begin-tm_commit-tm_end combo as if-else branch.

#define tm_begin                                \
    if (_tm_begin(setjmp(_tm_get_tx()->env)))   \
    {

#define tm_commit                           \
        _tm_commit();                       \
    } else {

#define tm_end  \
    }

Expanded by the C preprocessor, the code looks like this.

if (_tm_begin(setjmp(_tm_get_tx()->env)))
{
    /* execution phase */

    _tm_commit();
} else {

    /* recovery phase */

}

Whether the transaction enters execution or recovery depends on the value returned by _tm_begin(). And this value is controlled by the transaction manager. If we start the transaction for the first time or rolled back because of a conflict, the returned value is true and we enter the execution phase. After we rolled back because of an error, the returned value is false and we enter the recovery phase.

At the end of the recovery phase, the transaction defaults to ending without retrying execution. To restart we also need a new public interface. It’s called tm_restart() and invokes the transaction’s execution phase.

void
tm_restart(void);

Error Detection

Our transaction manager contains the function malloc_tx(), a transactional implementation of malloc(). Here’s the implementation.

static void
undo_malloc_tx(uintptr_t data)
{
    void* ptr = (void*)data;
    free(ptr);
}

void*
malloc_tx(size_t size)
{
    save_errno();

    void* ptr = malloc(size);

    append_to_log(NULL, undo_malloc_tx, (uintptr_t)ptr);

    return ptr;
}

Since malloc() fails if no memory is available for allocation, malloc_tx() should detect the error and invoke recovery.

static void
undo_malloc_tx(uintptr_t data)
{
    void* ptr = (void*)data;
    free(ptr);
}

void*
malloc_tx(size_t size)
{
    save_errno();

    void* ptr = malloc(size);
    if (!ptr) {
        tm_recover(errno); /* does not return */
    }

    append_to_log(NULL, undo_malloc_tx, (uintptr_t)ptr);

    return ptr;
}

This implementation of malloc_tx() tests the returned pointer and invokes tm_recover() if no memory could be allocated. To understand tm_recover(), let’s look at the restart-and-recovery machinery as a whole.

bool
_tm_begin(int value)
{
    return value != 2;
}

static void
rollback_tx(struct _tm_tx* tx, int value)
{
    release_resources(arraybeg(g_resource),
                      arrayend(g_resource), false);

    /* Revert logged operations */
    undo_log(tx->log, tx->log + tx->log_length);
    tx->log_length = 0;

    /* Restore errno */
    if (tx->errno_saved) {
        errno = tx->errno_value;
        tx->errno_saved = false;
    }

    /* Jump to the beginning of the transaction */
    longjmp(tx->env, value);
}

void
tm_restart()
{
    rollback_tx(_tm_get_tx(), 1);
}

void
tm_recover(int errno_code)
{
    struct _tm_tx* tx = _tm_get_tx();

    tx->recovery_errno_code = errno_code;

    rollback_tx(tx, 2);
}

The roll back is performed by the internal function rollback_tx(). Its implementation is mostly uninteresting here, but it’s important to note that the second parameter value contains the value that is passed to _tm_begin().

If the transaction manager detects a conflict it rolls back the transaction with a call to tm_restart(). This function is also the one to call at the end of the recovery phase. It uses a value of 1, which means something like regular restart.

If a transactional function detects an error it invokes tm_recover(). This function saves the errno code and restarts the transaction with a value of 2. This value signals the request for error recovery.

An invocation of _tm_begin() can now receive three possible values. A value of 0 signals the initial execution of the transaction. This is the default value returned by setjmp(). A value of 1 signals a restart into the execution phase and a value of 2 signals a restart into the recovery phase. _tm_begin() tests the value and returns true or false accordingly.

in addition to the public interfaces, we also add the function tm_recovery_errno(). It returns the stored errno code, so that the recovery phase can query information about the failure.

int
tm_recovery_errno(void)
{
    struct _tm_tx* tx = _tm_get_tx();

    return tx->recovery_errno_code;
}

A Producer-Consumer Example

In previous blog posts we used an example with a producer transaction and a consumer transaction. The producer looked something like this.

static void
producer_func(void)
{
    unsigned int seed = 1;

    tm_save int i[2] = {0, 0};

    while (true) {

        sleep(1);

        ++i[0];
        i[1] = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i[0], i[1]);

        tm_begin

            int* buf = NULL;
            load((uintptr_t)&g_i, &buf, sizeof(g_i));

            if (!buf) {

                buf = malloc_tx(2 * sizeof(*buf));
                buf[0] = i[0];
                buf[1] = i[1];

                store((uintptr_t)&g_i, &buf, sizeof(g_i));
            }

        tm_commit
    }
}

It produces a pseudo-random value and hands it over to the consumer. The memory is allocated dynamically using malloc_tx().

Let’s extend this example with the new functionality. We can add error recovery by appending only 3 lines of code to the transaction.

static void
recover_from_errno(int errno_code)
{
    fprintf(stderr, "Recovering from error %d (%s)\n", errno_code,
            strerror(errno_code));
}

static void
producer_func(void)
{
    unsigned int seed = 1;

    tm_save int i[2] = {0, 0};

    while (true) {

        sleep(1);

        ++i[0];
        i[1] = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i[0], i[1]);

        tm_begin

            int* buf = NULL;
            load((uintptr_t)&g_i, &buf, sizeof(g_i));

            if (!buf) {

                buf = malloc_tx(2 * sizeof(*buf));
                buf[0] = i[0];
                buf[1] = i[1];

                store((uintptr_t)&g_i, &buf, sizeof(g_i));
            }

        tm_commit
            recover_from_errno(tm_recovery_errno());
            tm_restart();
        tm_end
    }
}

The new function recover_from_errno() is specific to the application. In a real-world scenario it could try to free some memory, for example by flushing caches or running a garbage collector. In the example, it simply prints a message.

To test the code, we can instrument the implementation of malloc_tx() to fail spuriously, just as if the application was low on memory.

static void*
malloc_with_low_mem(size_t size)
{
    /* simulate spurious allocation failures */
    clock_t cputime = clock();
    if (!(cputime % 3)) {
        errno = ENOMEM;
        return NULL;
    }

    return malloc(size);
}

void*
malloc_tx(size_t size)
{
    save_errno();

    void* ptr = malloc_with_low_mem(size);
    if (!ptr) {
        tm_recover(errno); /* does not return */
    }

    append_to_log(NULL, undo_malloc_tx, (uintptr_t)ptr);

    return ptr;
}

The full example code is available on GitHub. If you run it, it should output something like this to the terminal.

Storing i0=1, i1=476707713
Loaded i0=0, i1=0
Storing i0=2, i1=1186278907
Loaded i0=1, i1=476707713
Storing i0=3, i1=505671508
Loaded i0=2, i1=1186278907
Storing i0=4, i1=2137716191
Loaded i0=3, i1=505671508
Storing i0=5, i1=936145377
Loaded i0=4, i1=2137716191
Storing i0=6, i1=1215825599
Loaded i0=5, i1=936145377
Recovering from error 12 (Cannot allocate memory)
Recovering from error 12 (Cannot allocate memory)
Loaded i0=6, i1=1215825599
Storing i0=7, i1=589265238
Recovering from error 12 (Cannot allocate memory)
Loaded i0=7, i1=589265238

This is the output of the values stored and loaded by the producer and consumer transactions. Occationally the producer’s memory allocation failed and triggered the recovery phase.

The final three lines illustrate this nicely. The producer transaction intended to store value no. 7, but had to recover. The error message tells that memory allocation failed (as expected). After completing recovery the producer restarted execution and successfully stored the 7th value. The consumer transaction then loaded the value correctly, as signalled by the final line of the output.

Summary

In this blog post, we’ve added support for error detection and recovery to our simpletm transaction manager.

Error detection and rollback is provided by the transaction manager’s framework.
Only the error recovery is application-specific and has to be provided by the application developer.
Transactional functions, such as malloc_tx, detect errors and invoke recovery.
The transaction either restarts into the execution phase or the recovery phase.
The code of the recovery phase is located between tm_commit and the new token tm_end.

As usual, the complete example code for this blog post is available on GitHub.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Building Fault-Tolerant Software With Transactions

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 14 Jul 2017 11:00:00 +0200

This is a series of blog posts to build a transaction manager in C. Having implemented thread isolation, we will now look at error handling. Our goal is to automate error handling as much as possible and make building fault-tolerant software easy.

If you missed earlier entries in the series, you might want to go back and read the installments so far.

Overview

In the very first installment of the series, I claimed that the transaction manager provides thread isolation and error handling to its transactions; with a clear implication that transactional code would do better than its traditional counter part. But so far we’ve only looked at thread isolation. We have built a basic implementation of Software Transactional Memory and added support for a few C functions on top.

In this and the next blog post, we’re going to discuss error handling and automate it as much as possible. This time we cover the theory, and next time we add an implementation to our simpletm transaction manager. In the end, we’ll have infrastructure that allows for easy creation of fault-tolerant software.

The Tradional Approach

A typical pattern of C code looks like this.

enum result {
    failure,
    success
};

enum result
func()
{
    if (call_1() == failure) {
        goto err_call_1;
    }
    if (call_2() == failure) {
        goto err_call_2;
    }
    if (call_3() == failure) {
        goto err_call_3;
    }

    return success;

err_call_3:
    revert_call_2();
err_call_2:
    revert_call_1();
err_call_1:
    return failure;
}

The function func() invokes 3 other functions named call_1(), call_2(), and call_3(). If either fails, the execution moves to the end of func() and executes a block of clean-up code to revert the effects of the already completed functions.

Although hard coded, these goto statements and labels act like an undo log in a transaction: all the executed operations are un-done and the system moves back into its previous state.

When func() returns failure, the calling code could try to repair the problem and retry the invocation.

void
repair_func()
{
    /* repair errors of func() */
}

int
main(int argc, char* argv[])
{
try_func:
    if (func() == failure) {
        repair_func()
        goto try_func;
    }

    return EXIT_SUCCESS;
}

Here, the main() function invokes func() and tries to repair any errors that might happen. Erroneous invocations are re-started after the repair.

This pattern works very well in traditional C programs, but has a few drawbacks.

The error handling interleaves with the application code. The application logic, the error detection and the error recovery is all mixed in the same location.
The error handling is hard coded and cannot be re-used easily. Similar code requires similar error handling, which has to be re-written. Any changes later on have to be applied to all error-handling code.
The code is not well structured. There are quite a few goto statements and labels. In more complex programs, it’s not always easy to follow these jumps.

The Transactional Approach

Transactions can help to solve these problems. As with thread isolation, application logic is separate from the error detection. The former is given by the application programmer, the latter is entirely implemented within the transaction framework. Error recovery also is provided by the application programmer, but separated from the application logic.

To implement error handling, let us first extend our transaction manager’s interface a bit. So far, a transaction is only confined by tm_begin and tm_commit. All transactional application logic is listed between these two statements. This is called execution phase. Once the transaction reached tm_commit, it entered the commit phase.

We now extend this interface with a new statement called tm_end.

tm_begin

    /* execution phase */

tm_commit   /* commit phase */

    /* recovery phase */

tm_end

tm_end follows after tm_commit and marks the end of the transaction. The code between tm_commit and tm_end is provided by the application programmer to repair possible errors. We call this the recovery phase. The transaction framework only invokes recovery if it detects an error.

An error-free transaction still looks like before.

Perform tm_begin.
Perform the execution phase.
Perform tm_commit.
Leave the transaction by executing the next instruction after tm_end.

An erroneous transaction instead runs the recovery phase.

Perform tm_begin
Perform the execution phase up to the point were the error happens. The error is detected within the framework. All transactional operations; such as load(), store(), malloc_tx() or free_tx(); are provided by the transaction manager. Their implementation detects the error internally. The transactional application logic does not have to do error detection.
Roll-back the transaction log. In traditional code, an equivalent operation is performed by the clean-up code at the end of func().
Perform the recovery phase. The recovery code is application specific and has to be provided by the application programmer.
After successful recovery, restart the transaction’s execution phase.
Perform tm_commit.
Leave the transaction by executing the next instruction after tm_end.

If done correctly, there is no difference between the result of an error-free run, and the result of an erroneous run with successful recovery.

Although rather abstract, let’s re-write the original example as transactional code.

void
repair_func()
{
    /* repair errors of func() */
}

int
main(int argc, char* argv[])
{
    tm_begin

        call_1_tx();
        call_2_tx();
        call_3_tx();

    tm_commit

        repair_func();
        tm_restart();

    tm_end

    return EXIT_SUCCESS;
}

We’ve replaced the function func() with a transaction. During the execution phase, the transaction invokes call_1_tx() to call_3_tx() These operations run the respective call_?() function and append them to the transaction log, together with an undo function. We’ve seen how this works in detail in the blog post on transactional malloc(). Shown below is a pseudo implementation of call_1_tx().

static void
undo_call_1_tx(uintptr_t data)
{
    revert_call_1();
}

void
call_1_tx()
{
    if (call_1() == success) {
        append_to_log();
        append_to_log(NULL, undo_call_1_tx, 0);
    } else {
        tm_recover(); /* does not return */
    }
}

The functions call_1_tx() to call_3_tx() also perform error detection internally. If either function fails, it instructs the transaction manager to perform a rollback and invoke the recovery phase.

The recovery phase starts with a call to repair_func(). What exactly it does is dependent on the application; maybe it runs the garbage collector to free memory, or cleans up files to free disk space. In any case it can at least send an email to the administrator and shut down the software gracefully, so even sever errors don’t go unnoticed.

After a successful repair, the recovery code invokes tm_restart(), which restarts the transaction’s execution phase. It now either succeeds by committing the transaction or detects another error and re-runs the recovery phase.

You can see from this abstract example, how we’ve solved the traditional code’s problems with error handling.

We cleanly separated application logic from error handling. For the error handling, we further separated error detection (i.e., done within the framework) from error recovery (i.e., done during the recovery phase).
Error detection is implemented once by each transactional operation. The detection code is re-used for each transaction and freely combined with other operation’s error detection.
Error recovery has to be implemented only once by the application programmer. The code in repair_func() could repair all kinds of errors, not just those specific to our example.
All goto statements and labels are gone, making the transactional code much more readable.

Summary

In this blog post, we’ve discussed the problems of error handling, and looked at the solution the transaction’s provide.

Traditional code interleaves application logic, error detection and error recovery; making the result hard to read and maintain.
Transactional code detects errors within the transaction manager’s framework.
Each transaction contains a specific recovery phase that implements recovery strategies for specific errors.
If the transaction framework detects an error, it rolls back the transaction’s execution and runs the recovery code.
After a successful recovery, the transction can be re-executed.
Transactional code cleanly separates application logic, error detection and error recovery.
The result of an erroneous transaction with successful recovery is the same as the result of the same transasction after and an error-free run.

This blog post was fairly high-level and abstract. In the next installment, we’ll add an implementation to our simpletm toy transaction manager. All this (and more) is already provided by picotm, the system-level transaction manger.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Types of Transactional Operations

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 07 Jul 2017 10:00:00 +0200

This is a series of blog posts to build a transaction manager in C. In this installment, we’re going to take a closer look at the different types of functions that exist and what we have to do to handle them in a transaction.

We’ve already implemented support for several interfaces from the C Standard Library in our transaction manager. We’re now going to sort and formalize this a bit.

If you missed earlier entries in the series, you might want to go back and read the installments so far.

Overview

Initially we only had support for memory operations in our transaction manager. On top of that, we added memory and string helper functions, such as memcpy(). With the introduction of the transaction log, we have also added support for malloc() and free(), which required either reversal during roll back or delay to commit.

We can distiguish 4 different classes of transactional functions. A transactional function is either

pure,
deferable,
revocable, or
irrevocable.

Sorted roughly by their impact on the transaction system, these classes differ in the kind of support they require from the transaction manager and consequently the amount of work they require for an implementation.

Pure Functions

Pure functions don’t have any dependencies or side effects that interfere with the transaction.

One example on Linux systems is pthread_self(), which returns the calling thread’s id. The function doesn’t depend on any parameters or mutable state. The returned id is constant for each thread. In a transaction, we can call pthread_self() without further considerations.

Another example is fmin(), a function that returns the smaller of two values of type double. The parameters it receives are the two values to compare. Again, no pointers or mutable state that this function depends on. A call to fmin doesn’t even generate errors. There’s a result for all possible input values, even NaN.

A load() operation on transactional memory is also pure. While its result depends on the memory’s content, load() doesn’t change non-transactional mutable state.

Handling pure functions is easy. We can call them directly from within the transaction.

Deferable Functions

The next class after pure functions are deferable functions. A deferable function does not have an immediate effect on the transaction and can be defered until commit time.

A common deferable function that we’ve already seen is free(). Free releases a block of dynamically allocated memory. It accepts a pointer to the memory block as its only argument. After free() returns, the content of the memory block is invalid and accessing it is an undefined operation.

If we’d free the memory during the transaction’s execution, there’d be no way of rolling back the transaction later, as free operations cannot be reverted.

However, there’s no need to actually free the memory at this point. Instead we defer the free operation until commit time. In the case of a roll back of the transaction, we simply ignore the call to free().

Another example of a deferable function is the store() operation we implemented for transactional memory. Stores in write-back mode buffer the written data and defer it until commit time.

The main criteria for a deferable function is that it has an effect on non-transactional mutable state, but the effect is either not immeditely visible, or can be simulated within the transaction.

Revocable Functions

In some way, a revocable function is the opposite of a deferable function. A revocable function is executed during the transaction, but it’s effects are revoked (or reverted) during the transaction’s roll back.

In our case, we’ve already seen malloc() as an example of a revokable function. A call to malloc() returns a dynamically allocated block of memory. This operation has to be executed immediately during the transaction as the memory might be required during subsequent operations within the transaction.

To support malloc(), the transaction executes the operation immediately and returns the allocated memory block. If it rolls back the transaction later, the transaction manager frees the allocated block automatically.

Another example is a read() operation on a file. Whatever gets read might have an effect on the transaction, so the data returned by the read operation is immediately required. But reading has no side-effects on the data itself. It mostly changes the file offset at which the next read operation takes place. The change to the offset can be revoked by restoring the old offset during a roll back. So reading from a file is a revocable operation.

As a third example, again store() can act like a revocable function. When executed in write-through mode, store() immediately writes the data to memory. It keeps a copy of the old data, which is restored during a roll back.

Revocable functions have an effect on the transaction’s execution and therefore have to be performed immediately. The effects on non-transactional mutable state can be revoked during a roll back.

Irrevocable Functions

Finally we have irrevocable functions. The result of an irrevocable function is immeditely required, but there’s no way of revoking it’s effects on non-transactional mutable state.

An example of an irrevocable function is a read() operation on a Unix pipe, which is often used to transfer data from one program to another program. Similar to a file, invoking read() on a pipe returns the next block of data. Unlike a file, a pipe doesn’t provide any means of reverting the effect of the read operation. Once it has read data from the pipe, the transaction has no way of putting back the data into a pipe during a roll back. The effect of calling read() on a pipe is therefore not revocable. The function is thus irrevocable.

The only way to support irrevocable functions is to guarantee that a transaction is not rolled back after it executed an irrevocable function. This has a number of consequences for the transactional system as a whole.

A transaction has to explicitly request irrevocability from the transaction manager. This requires support by the transaction manager.
Once it a transaction is irrevocable, it wins any conflict with concurrent transactions.
At any time, there can only be one irrevocable transaction. Imagine there were two irrevocable transactions at the same time. If there’s a conflict between these two transactions, it would not be possible to pick a loser without violating irrevocability.

The easiest way of supporting irrevocability is to run the irrevocable transaction exclusively. When a transaction requests irrevocability, all other transactions are stopped, rolled back, and suspended until the irrevocable transaction committed. So there are either multiple revocable transactions running, or at most one irrevocable transaction. This is the same semantics as for reader/writer lock and that’s how it’s implemented.

A more elaborated implementation would use a lock manager. This component of the transaction manager observes the state of resource locks, and resolves conflicts between transactions. When it has to resolve a conflict between the irrevocable transaction and a revocable transaction, it would always pick the irrevocable transaction as winner.

To summarize, irrevocable functions have an effect on the transaction’s execution and therefore have to be performed immeditely. The effects on non-transactional mutable state cannot be revoked during a roll back, so the transaction may not perform a roll back at all.

Special Cases

Aside from the 4 basic types, there are a number of special cases.

First of all, there are functions that are both deferable and recovable. One such function is realloc(), which changes the size of an already allocated block of memory. A call to realloc() has to be performed immediately as the allocated memory has to be returned into the transaction. This memory has to be released during a roll back. The memory block of the old size also has to be released, but only at commit time. An implementing of a transactional realloc() is a combination of malloc() and free().

Another special case happens when functions change their type depending on some internal state. We’ve already seen store() and read(). These functions’ type depends on parameters and internal state.

Finally, there are operations that simple don’t make sense in the context of a transaction. These usually concern program state as a whole. Calling functions like exit(), exec(), or abort() will end the current program in some way. Calling them within a transaction would probably lead to an inconsistent state of the system.

Summary

In this blog post, we’ve discussed the different types of transactional functions.

Pure functions have no effect on non-transactional state.
Deferable functions are defered until commit time. Their effects on non-transactional state are simulated to the transaction.
Revocable functions are executed immediately during the transaction, but their effects on non-tranactional state are reverted during a roll back.
Irrevocable functions are executed immediately, but their effects on non-transactional state cannot be reverted. A transaction may not have to roll back after executing an irrevocable function.
Functions can change their type, depending on the parameters or the transaction’s state.
Some functions have no meaningful semantics in the context of transactions.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Everything About errno; plus Transactions

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 30 Jun 2017 10:00:00 +0200

This is a series of blog posts to build a transaction manager in C. In this entry, we’r going to take a look at errno and how it works behind the scenes. Afterwards we’re going to add support for errno to our transaction manager simpletm.

If you missed earlier entries in the series, you might want to go back and read the installments so far. Entries are self-contained as much as possible, but knowning the big picture might by helpful in some cases.

Communicating Problems

The C and POSIX standards specify a multitude of ways for reporting errors to a program.

If you do something critical, such as accessing unmapped memory regions, you’ll get a signal distributed from the operating system; for example SIGSEGV in the case of unmapped memory. Usually these critical signals cannot be ignored. You have to handle them, or the operating system will abort the program.

But signals are rather uncommon. Most interfaces in C or POSIX return the integer value of -1 on errors and provide an additional way of retreiving information about the actual error.

For floating-point operations, there are floating-point exceptions. These can be queries with fgetexceptflag(). Divide by zero and you’ll get FE_DIVBYZERO, for example.

One of the oldest methods for communicating errors is by setting an error code. Traditional error codes in C are provided in the header file errno.h. There you’ll find short identifier for all kinds of possible errors: ENOMEM for out-of-memory situations, EACCES when file access has been denied, or EINVAL if an incorrect argument was given to a function. Each identifier maps to a positiv integer value, but the actual value is system-dependent.

Some more-recent interfaces, such as POSIX threads, return error codes directly, but most POSIX interfaces store them in the variable named errno.

The History of errno

The variable errno is quite old. I searched through the source code of The Unix Heritage Society and found its first in-code appearance in Unix Release V5. According to the git history, the commit was made on Nov 27 1974! Back then, there was no file named errno.h. Instead errno was implemented as part of perror(), a function that outputs a descriptive string for each error code.

The error codes are even older than errno. The earliest error codes I could find were in the source code of Unix Release V4. This file dates back to Aug 31 1973. The error codes were stored in the field u_error of struct user. Even though the errno variable was not present in the source code of V4, it’s already mentioned on the man page for perror(). This file dates back to Nov 6 1973, so errno must have been introduced at some point between Aug 31 and Nov 6 of 1973.

If you compare these old error codes to what is present on today’s Unix-like system, you’ll find that they are mostly the same. In fact all of the old error codes are still present. New releases and standards only added more error codes when required.

So back in Unix Release V5, errno was an integer value. Error codes were still returned in the field u_error of struct user, but automatically copied to errno.

C was first standardized in the late 1980s and errno was part of this standard. It was specified to be a modifiable lvalue of type int, just as it was in Unix.

Compared to its prominent place in Unix, in the C Standard errno feels as if it is either under-used or out of place. Only 3 error codes are part of the C standard: EDOM, EILSEQ and ERANGE. Many traditional Unix functions don’t have error codes specified. For example, even though malloc() is part of the C Standard Library, its error code ENOMEM is not.

The implementation of errno remained of type int for roughly 22 years, and then… multithreading happend.

In a multithreaded program, having a global variable that signals the error code is obviously a problem. In the time between one thread observing an error (i.e., -1 being returned from a function call) and reading the error code from errno, an error on another thread could change the value of errno. A global errno variable of type int is basically useless in this environment.

The solution to this problem is illustrated in the example code shown below.

#ifdef	_THREAD_SAFE
extern	int *		__error();
#define	errno		(* __error())
#else
extern int errno;			/* global error number */
#endif

In a thread-safe program, errno is defined as macro that resolves to a function call. The function returns a pointer to a thread-local copy of the error variable and the errno macro internally dereference the pointer. This way, accessing errno returns a thread-local error code.

The code snipped above has been taken from a rather confusing commit to FreeBSD. The commit was made on Jan 22 1996 and first shipped in FreeBSD 2.2.0.

Other systems, such as GNU or Linux, traditionally used different implementations of POSIX and the C Standard Library, but have gone through a similar transition.

Transactional errno

After this long excurse into history, let’s get back to transactions. For errno to be transaction-safe, all we have to do is to save its value before it’s being changed by any function. If a transaction aborts, we reset errno to its original value, so the transactions always restarts with the same error code. As the variable itself is already thread-local, no concurrency control is required.

Building upon malloc() and free() support from the previous blog entry, let us first add an interface to safe errno during the transaction.

void save_errno(void);

Too easy.

For the implementation, we add an errno value to the transaction context as well as a flag showing the value’s validity.

struct _tm_tx {
    jmp_buf env;

    unsigned long        log_length;
    struct _tm_log_entry log[256];

    bool errno_saved;
    int errno_value;
};

We further have to save errno and restore its value during a rollback.

void
save_errno()
{
    struct _tm_tx* tx = _tm_get_tx();

    if (tx->errno_saved) {
        return;
    }

    tx->errno_value = errno;
    tx->errno_saved = true;
}

void
tm_restart()
{
    release_resources(arraybeg(g_resource),
                      arrayend(g_resource), false);

    struct _tm_tx* tx = _tm_get_tx();

    /* Revert logged operations */
    undo_log(tx->log, tx->log + tx->log_length);
    tx->log_length = 0;

    /* Restore errno */
    if (tx->errno_saved) {
        errno = tx->errno_value;
        tx->errno_saved = false;
    }

    /* Jump to the beginning of the transaction */
    longjmp(tx->env, 1);
}

We only save errno on the first time that save_error() gets called in a transaction. That’s the error code we’re interested in. Later invocations may not replace is. During a transaction’s roll-back and restart, we check if we saved errno and possibly restore it to its original value.

So far, our transaction manager only supports one function that could potentially modify errno, which is malloc_tx(). Modifying malloc_tx() to save errno is trivial.

void*
malloc_tx(size_t size)
{
    save_errno();

    void* ptr = malloc(size);

    append_to_log(NULL, undo_malloc_tx, (uintptr_t)ptr);

    return ptr;
}

That’s the same implementation as in the previous blog post, extended by a call to save_errno().

Summary

We’ve looked at errno and how to support it in a system transaction.

errno has a long history, dating back to the earliest Unix releases.
Today, errno is implemented as a thread-local variable.
Before executing an operation that modifies the value of errno, the original value has to be saved in the transaction context.
During a roll-back of the transaction, errno is restored to its original value.

As usually, you can find the full source code for this blog post on GitHub. If you’re interested in a more sophisticated C transaction manager, take a look at picotm.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Transactional malloc() and free()

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 23 Jun 2017 12:00:00 +0200

This is a series of blog posts to build a transaction manager in C. In this installment we’re going to implement support for malloc() and free(). With these functions we can allocate and free blocks of memory, and automatically garbage-collect allocated blocks if an error or conflict occures during the transaction.

If you missed earlier entries in the series, you might want to go back and read the installments so far. You should especially read last week’s entry, which introduces transaction logs and gives some information about use cases and scenarios.

The Problems of malloc() and free()

One of last week’s example transactions looked like this.

tm_begin

    void* buf = malloc(size);

    load(...);
    store(...);

tm_commit

There’s an invocation to malloc() near the beginning of the transaction, and several load() and store() operations later in the transaction. If the transaction has to roll back and restart, the malloc’ed memory buffer would leak. We therefore need a way of freeing the memory during the rollback.

Another example illustrated the problem with free().

void* buf = malloc(size);

tm_begin

    load(...);

    free(buf);

    store(...);

tm_commit

Here we free a memory buffer that was allocated outside the transaction. If the transaction has to roll back afterwards, there’s no way to un-do the effects of the free() operation. Hence, we’d work on invalid data after a restart. We can only release memory if we know that the transaction commits.

The solution that we saw in the previous blog post was to keep a log of operations that either require rollback, or have to be delayed until commit.

Extending the Transaction Manager

To extend our transaction manager with a log, we first have to know how the log entry has to look like. We require a function to apply an operation during commit, un-do an operation during rollback, and we need some additional user data for each operation.

So the log entry might look like this.

struct _tm_log_entry {
    void        (*apply)(uintptr_t data);
    void        (*undo)(uintptr_t data);
    uintptr_t    data;
};

We add an array of these structures to our transaction’s internal data structure.

struct _tm_tx {
    jmp_buf env;

    unsigned long        log_length;
    struct _tm_log_entry log[256];
};

The log’s initial length at the start of the transaction is 0. Using a static array limits our number of log entries to a certain maximum, 256 in this case, but that’s OK for an example. The log in picotm can grow until it reaches system limits.

How do we append to this log? We have to add a new public interface to the transaction manager.

void
append_to_log(void (*apply)(uintptr_t),
              void (*undo)(uintptr_t), uintptr_t data)
{
    struct _tm_tx* tx = _tm_get_tx();

    assert(tx->log_length < arraylen(tx->log));

    struct _tm_log_entry* entry = tx->log + tx->log_length;

    entry->apply = apply;
    entry->undo  = undo;
    entry->data  = data;

    ++tx->log_length;
}

The implementation of append_to_log() is fairly trivial. It acquires the next available element of the log array and fills it with the arguments it got from the caller. These are the apply() and undo() functions, and some user data.

Finally what we need is some extra code to execute these apply() and undo() functions. Here’s the apply code.

static void
apply_log(struct _tm_log_entry* beg, const struct _tm_log_entry* end)
{
    while (beg < end) {
        if (beg->apply) {
            beg->apply(beg->data);
        }
        ++beg;
    }
}

void
_tm_commit()
{
    release_resources(arraybeg(g_resource),
                      arrayend(g_resource), true);

    struct _tm_tx* tx = _tm_get_tx();

    /* Perform logged operations */
    apply_log(tx->log, tx->log + tx->log_length);
    tx->log_length = 0;
}

The function apply_log() is called by the transaction manager’s main commit function(). It walks over the entries in the log and applys them one by one. Afterwards the log length is set to 0 again.

The undo code is similar.

static void
undo_log(struct _tm_log_entry* beg, const struct _tm_log_entry* end)
{
    while (end > beg) {
        --end;
        if (end->undo) {
            end->undo(end->data);
        }
    }
}

void
tm_restart()
{
    release_resources(arraybeg(g_resource),
                      arrayend(g_resource), false);

    struct _tm_tx* tx = _tm_get_tx();

    /* Revert logged operations */
    undo_log(tx->log, tx->log + tx->log_length);
    tx->log_length = 0;

    /* Jump to the beginning of the transaction */
    longjmp(tx->env, 1);
}

The function undo_log() walks backwards over the log and un-does each log entry’s effects. It’s invoked from the transaction manager’s main restart function. Afterwards the log length is set to 0 again. That’s all fairly trivial, isn’t it?

There’s one small caveat that we have to watch out for. It’s important that we first apply all operations on memory resources and then apply the logged operations. Otherwise it could happen that a transaction first frees a block of memory and then performs store operations on the just free’d memory. The operation would be undefined.

Transactional malloc()

Here’s the implementation of our transactional malloc function malloc_tx().

static void
undo_malloc_tx(uintptr_t data)
{
    void* ptr = (void*)data;
    free(ptr);
}

void*
malloc_tx(size_t size)
{
    void* ptr = malloc(size);

    append_to_log(NULL, undo_malloc_tx, (uintptr_t)ptr);

    return ptr;
}

A call to malloc_tx() allocates memory using standard malloc(). It then puts an entry into the log. That entry contains a pointer to the un-do function and a pointer to the allocated memory. The undo function undo_malloc_tx() simply frees the allocated memory buffer.

Using malloc_tx() we can rewrite our first example transaction.

tm_begin

    void* buf = malloc_tx(size);

    load(...);
    store(...);

tm_commit

Any roll-back of this transaction will un-do the effects of malloc_tx() by freeing the allocated memory.

Transactional free()

The transactional free function free_tx() looks like this.

static void
apply_free_tx(uintptr_t data)
{
    void* ptr = (void*)data;
    free(ptr);
}

void
free_tx(void* ptr)
{
    append_to_log(apply_free_tx, NULL, (uintptr_t)ptr);
}

It puts an entry into the log, containing a pointer to the apply function apply_free_tx() and a pointer to the buffer that is to be free’d. The apply function later executes the free operation during commit.

And our second example using free_tx() looks like this.

void* buf = malloc(size);

tm_begin

    load(...);

    free_tx(buf);

    store(...);

tm_commit

This transaction can roll back at any time. The memory buffer would only be free’d during commit.

Freeing Buffers That Are In Use

There’s one possible scenario that we haven’t covered yet. What happens if the free’d buffer is in use by a concurrent transaction?

Let’s say we free a buffer in one transaction’s commit, but another transaction concurrently performs a store operation on the buffer’s memory. The buffer is gone so the store operation should observe a conflict. The current implementation wouldn’t detect this.

This problem can be solved fairly easy with memory privatizations. In free_tx() we write-privatize the buffer. This will show up any conflict between the freeing transaction and any transaction that uses the buffer concurrently.

void
free_tx(void* ptr)
{
    size_t siz = malloc_usable_size(ptr);

    privatize((uintptr_t)ptr, siz, false, true);

    append_to_log(apply_free_tx, NULL, (uintptr_t)ptr);
}

The function malloc_usable_size() is provided by the GNU C Library. For a given allocated buffer, it returns the size of the allocation. Most allocators provide an interface like this, but it’s unfortunately not standardized.

Summary

In this block post, we’ve implemented support for transactional malloc() and free().

Allocated memory has to be released during rollbacks.
Freeing memory cannot be undone, so this operation has to be delayed until commit.
Invoked operations add themselves to the transaction log.
The transaction manager uses the log to un-do malloc() operations.
The transaction manager uses the log to apply free() operations.
Conflicts between a freeing transaction and users of the free’d buffer can be detected by write-privatizing the buffer before performing the free.

As usually, you can find the full source code for this blog post on GitHub. If you’re interested in a more sophisticated C transaction manager, take a look at picotm.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Transaction Logs

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 16 Jun 2017 11:00:00 +0200

This is series of blog posts to build a transaction manager in C. We have already implemented support for transactional memory and simple functions that operate only on memory, such as memcpy(). In this installment we’re going to look at logs, transactional logging and transactional memory allocation.

If you missed earlier entries in the series, you might want to go back and read the installments so far.

This is the first part of a two-part mini-series on transaction logs. In this entry we’re going to examine the uses for transaction logs and how they work in theory. In the next entry, we’ll implement a log for our transaction manager simpletm.

If you’ve been following this series over the course of the last weeks, you might have noticed how we’re going from more specific use cases to more broad scenarios, and how we’re going from memory operations to arbitrary functions.

When we added memory privatization to our transaction manager, we layed the foundation for calling C functions directly from within a transaction. C functions often take pointers as their arguments, and memory privatization is required allow this in a transaction-safe manner.

In this blog post, we’ll do the next step towards support for arbitrary functions. We’ll look at transaction logs and functions that require some form of transactional log.

Operations That Have To Be Undone

Let’s take a look at the following code fragment.

tm_begin

    void* buf = malloc(size);

    load(...);
    store(...);

tm_commit

Like many C programs, this transaction allocates memory using malloc(). This memory could be a temporary buffer required for an algorithm executed by the transaction. The transaction further performs several load and store operations. It could be a producer transaction that creates data for a consumer.

Looks good so far, however there’s a problem lurking.

Imaging a conflict happens during one of the load or store operations. If the transaction tries to acquire a memory location that is owned by a concurrent transaction, the transaction manager will resolve the conflict usually by rolling back the transaction that came last.

This rollback is a problem because of the memory buffer we allocated. Simply reverting all load and stores and restarting the transaction will leak the allocated memory.

To free allocated memory during a rollback, we have to inform the transaction manager about every allocation we performed during the transaction. This information is stored in the transaction log.

Operations That Have To Be Delayed

Let’s further take a look at another example.

void* buf = malloc(size);

tm_begin

    load(...);

    free(buf);

    store(...);

tm_commit

In this program, memory is allocated outside of the transaction using malloc(). After the transaction started, it loads several memory locations and performs computations. As part of the transaction the allocated memory buffer is released using free(), and finally some stores are performed to save the transaction’s results. This transaction could be the consumer to our producer from the previous example.

This example illustrates another problem. Imaging the transaction frees the buffer and then one of the store operations observes a conflict. The transaction would have to rollback and restart. The free() operation, however, cannot be rolled back. It has already been performed and the memory might have already been allocated by another part of the program.

To allow the rollback of a free operation, we have to delay the invocation of free() until the commit happens. This information is also stored in the transaction log.

Undo and Redo

The examples above illustrate the two use case that require a log: some operations require to be reverted during a rollback, some operations require to be delayed until commit time.

One can distiguish between undo and redo logs, although implementaions might not be so strict.

An undo log is a log that for each operation stores the old value before that operation. It is useful for reverting operations. In the case of malloc() it would store a pointer to the allocated memory, so that the transaction manager knows that before this invocation of malloc(), the memory had not been allocated.

A redo log stores the changes for each operation that affects the state of the transaction. In the case of free() it stores the pointer that needs to be freed during commit. With this type of log it’s possible to re-do the transaction step by step.

We’ve already seen something like these logs before. Let’s for a moment remember the write-back and write-through semantics that we used for store operations on transactional memory. For each memory resource, we had a transaction-local buffer that either stored the original value or the transaction’s modified copy.

In write-back mode the transaction performed all changes in its local buffer, which was synchronized with the shared global buffer during commit. So the local buffer acted like a redo log, although it merges all store operations into one.

A similar thing is true for write-through mode. For a store, the transaction first copys the shared global value into its local buffer as a backup. Then it performed modifications directly on the global buffer. During a rollback the global buffer is reverted to its original state from the content of the local buffer. So the local buffer acts like an undo log.

Implementation Outline

A database usually has an abstract interface to the stored data and the implemented algorithms. Because access happens though abstract languages such as SQL, the database’s implementation might be able to only go with an undo or redo log only.

But as mentioned before, an implementation might not be too strict about distiguishing between undo and redo logs. In our case, we’ve already seen that we require both semantics: the undo semantics for malloc() and the redo semantics for free().

Generally speaking, we can say that we require

undo semantics for all operations that allocate resources, and
redo semantics for all operaionss that release resources.

An operation that allocates a resource can also be a call to open() for opening a file. An operation for releasing a resource can also be a call to close() for closing a file descriptor.

We can simply combine both types of logs into the same data structure. For each allocating operation we have to

perform the operation, because the result is required immediately by the transaction, and
put an entry into the log.

For each releasing operation we have to

put and entry into the log, and
don’t do anything else; especially no the operation itself.

During a commit we can be sure that the transaction is about to complete. So we go through the log entries one-by-one and

ignore the entry if it refers to an allocating operation. We already performed these during the transaction’s execution phase.
Apply the entry if it is a releasing operation.

During a rollback we go backwards through the log entries one-by-one and

undo the entry’s effects if it refers to an allocating operation, or
ignore the entry if it refers to a releasing operation. We’ll certainly require the released resource during after the transaction’s restart.

As system transactions are about concurrency control and error handling, each transaction can get by with its own private log. If we have multiple transactions running concurrently, they won’t have to content for access to a global log. Databases have different use cases and stricter requirements for their transaction’s durability. Therefore they often require logs that are either shared globally among transactions, or allow for restoring a global history from the individual transaction’s logs. Fortunately none of this is required in our case.

Summary

In this block post, we’ve investigated the use of logs in our transaction manager.

Resource-allocating operations have to be undone when the transaction rolls back.
Resource-releasing operations have to be delayed when the transaction commits.
A transaction log keeps track of all changes performed by a transaction.
An undo log stores the state before performing an operation.
A redo log stores the state change peformed by an operation.
Both types of logs can be combined into a single data structure.
Each executed operations puts an entry into the transaction’s log.
During commit the transaction applys all defered operations.
During rollback the transaction reverts all performed operations.

So far we only covered some theory and concepts. In the next installment of this series of blog posts, we will implement a simple transactional log and implement transactional malloc() and free() on top of it.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

picotm at Tübix 2017

blog@tzimmermann.org (Thomas Zimmermann) — Tue, 13 Jun 2017 16:00:00 +0200

On June 24, I’ll be at Tübix 2017. I’ll do a presentation on picotm and system-level transactions. Hope to see you there!

More on Resource Privatization

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 09 Jun 2017 11:00:00 +0200

This is a follow-up post to last week’s piece on resource privatization for transactional memory. We’re going to support C strings and look at combining load and store operations with privatization.

Privatizing C Strings

Privatazion provides a way of directly accessing a resource from within a transaction. For memory this means direct access to the memory region, instead of working on a transaction-local copy.

The interface for privatizing regions of memory looks like this.

void
privatize(uintptr_t addr, size_t siz, bool load, bool store);

This function takes the address of the first byte of the memory region to be privatized, the number bytes in the region, and two booleans enabling load and store access. If it succeeds, the memory region is owned by the transaction and can be accessed directly.

The interface is fine for memory regions of known size. In last week’s example code we implemented a transactional memcpy(); and anything that is typically accessed with memcpy() by be privatized this way.

There is one noteable exception: C strings. Strings in C have their length implicitly signalled with a trailing \0 character. The get a string’s length a call to strlen() is required.

A first attempt to privatize a C string will fail quickly.

char* str = "Hello World!";

tm_begin

    privatize(str, ???, true, false);

tm_commit

When using the current interface, we don’t know of which length the string is. Maybe using strlen() could help.

char* str = "Hello World!";

tm_begin

    size_t siz = strlen(str);
    privatize(str, siz, true, false);

tm_commit

We now have all parameters required by privatize() but the call to strlen() does not protect against concurrent access to str. Another transaction can modify the content of str concurrently, which would make the returned length incorrect. Maybe we can solve this problem by providing a transactional implementation of strlen().

size_t
strlen_tx(const char* str)
{
    privatize(str, ???, true, false);

    return strlen(str);
}

char* str = "Hello World!";

tm_begin

    size_t siz = strlen_tx(str);
    privatize(str, siz, true, false);

tm_commit

This approach creates a chicken-and-egg problem. Our transactional implementation of strlen() requires privatize() to protect the string against concurrent modification. But privatize() is what we’re trying to support in the first place.

Clearly a different approach is required.

The solution implemented by picotm is to have a privatize function that stops at a caller-specified terminating character. For a C string the terminating character would be \0.

Our function is called privatize_c() with c for character. The interface is similar to the regular privatize(), but it receives a character instead of the length.

void
privatize_c(uintptr_t addr, int chr, bool load, bool store);
{
    bool found_chr = false

    while (!found_chr) {

        struct resource* res = acquire_resource(addr & BASE_BITMASK);
        if (!res) {
            tm_restart();
        }

        unsigned long index = addr & RESOURCE_BITMASK;
        unsigned long bits = 1ul << index;

        uint8_t* beg = arraybeg(res->local_value) + index;
        uint8_t* end = arrayend(res->local_value);

        while (!found_chr && (beg < end)) {
            /* If we're about to store, we first have to
             * save the old value for possible rollbacks. */
            if (store && !(res->local_bits & bits) ) {
                *beg = *((uint8_t*)addr);
            }

            bits <<= 1;
            --siz;
            ++addr;
            ++beg;

            found_chr = (*beg == chr);
        }

        if (store) {
            res->flags |= RESOURCE_FLAG_WRITE_THROUGH;
        }
    }
}

The code is again very similar to the implementation of the original privatize(), but instead of testing for the size of the supplied memory region, it tests for the existence of the terminating character. Once the memory location containing the character has been privatized, privatize_c() returns.

Our original example now becomes trivial to implement.

char* str = "Hello World!";

tm_begin

    privatize_c(str, '\0', true, false);

tm_commit

Loading and Storing Privatized Memory

Remember that load and store operations used a transaction-local buffer to work on. We called this write-back, as the buffer is only written back to the shared memory resource during a commit. Privatization instead required write-through semantics, where transactions operate directly on the shared resource.

Because of this difference, our implementation of privatize() can not be combined with load() and store() calls on the same memory regions. Here’s an example transaction.¹

int value = 0;

tm_begin

    privatize(&value, sizeof(value), true, true);

    value = 1;

    load_int(&value); /* returns 0 */

tm_commit

This transaction privatizes an integer variable and stores 1. It then transactionally loads the variable, but this gives an incorrect result of 0.

Loads and stores currently use a write-back scheme. When acquiring a resource, they operate on a transaction-local buffer. Only at commit time this buffer is written back to the shared value. The call to privatize() initializes the transaction-local buffer with the original value of 0, so this is what load_int() returns.

The solution to this problem is to support write-through mode for load and store operation. Without going into details, an implementation would have to provide the following semantics.

For initial loads and stores, acquire the resource and use write-back.
For initial privatize, acquire the resource and use write-through.
For load and stores on an already-privatized resource, operate on the shared buffer, instead of the transaction-local buffer.
For privatizing an already-loaded or already-stored resource, swap the content of transaction-local and shared buffer, and set write-through mode for the resource.

This behavior is implemented in picotm, but probably not worth the effort for our simpletm transaction manager.

A further optimization would be the use of write-through mode for all load operations until the transaction executed a store operation on the resource. This would spare reader transactions from the overhead of filling transaction-local buffers that they are never going to modify.

Summary

In this block post, we looked at two corner cases of memory privatization.

Memory regions with terminating character, such as C strings, can be privatized by stopping privatization at the specific character.
Load and store operations can be combined with privatization by supporting write-through semantics when loading and storing shared resources.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

There might be other limitations in our current implementation that prevent this example from working. One is that acquiring the same resource multiple times from within the same transaction is currently not supported. ↩︎

Transactional memcpy() and Resource Privatization

blog@tzimmermann.org (Thomas Zimmermann) — Thu, 01 Jun 2017 11:00:00 +0200

So far we have implemented a simple transaction manager that loads and stores values in shared memory locations. These load and store operations copy values in and out of the transactional context. In this blog post we’re going to implement direct access to the shared memory. We also lay the foundation for operations besides memory access, such as function calls.

For our transaction manager, we created an implementation that loads and stores values by copying them in and out of the transaction context. The interface is shown below.

void
load(uintptr_t addr, void* buf, size_t siz);

void
store(uintptr_t addr, const void* buf, size_t siz);

The function load() takes an address in main memory and copies siz bytes into a transaction-local buffer. The function store() does a similar thing by copying siz bytes from the transactional buffer to an arbitrary address in memory.

This interface is good for most use cases, but sometimes we need access to the actual shared resource. When we acquire a resource for direct access, it becomes part of the transaction’s private context. We call this privatization.¹

Why Privatize Resources?

Imaging we have a non-transactional function that takes a memory location for it’s arguments, let’s say memcpy(). If we what to call memcpy() from within a transaction, we require an implementation that can handle transactional memory.

Here’s a very simple implementation based on the load and store primitives we’ve developed so far.

void*
memcpy_tx(void* dst, const void* src, size_t siz)
{
    uint8_t* dst8 = dst;
    const uint8_t* src8 = src;

    while (siz) {

        uint8_t value;

        load(src8, &value, 1);
        store(dst8, &value, 1);

        ++src8;
        ++dst8;
        --siz;
    }

    return dst;
}

This function transactionally loads a byte from the source memory, and transactionally stores it in the destination memory. We could optimize the implementation by loading and storing memory words instead of bytes, but the fundamental problem is that the intermediate value exists in the first place.

To get around this limitation we have to add privatization to our transaction manager.

Privatizing Memory

In these blog posts, we usually start with the implementation of a new feature and then work our way outwards to the interface. This time let’s start with the interface and look at the implementation afterwards.

privatize(uintptr_t addr, size_t siz, bool load, bool, store);

This looks similar to the load() and store() function we already have, doesn’t it? The privatize() function takes the first address of a memory buffer and the buffer’s length. This is the memory that will be privatized. The load and store arguments tell the transaction manager whether we want to load or store or both.

Using this interface, we can re-implement memcpy_tx().

void*
memcpy_tx(void* dst, const void* src, size_t siz)
{
    privatize(dst, siz, false, true);
    privatize(src, siz, true, false);

    return memcpy(dst, src, siz);
}

Here we first privatize dst for storing, then privatize src for loading. At this point the transaction owns both buffers. Finally we execute a plain memcpy().

Not only does the new implementation look much nicer, it has a number of advantages.

Single calls to privatize() have less overhead than repeated calls to load() or store(); simply by the fact that there are less function invocations.
We got rid of the intermediate value and instead copy directly from src to dst.
We use the system’s memcpy() operation. C compilers or standard libraries often replace memory and string functions by highly optimized implementations or even single assembly instructions. Using the native implementation of memcpy() allows these optimization to take place.

Implementing Memory Privatization

Let’s take a look at the implementation of privatize(), especially how we provide direct access to memory. Let’s first add a flags field back to the resource structure.

struct resource {
    uintptr_t       base;
    uint8_t         local_value[RESOURCE_NBYTES];
    uint8_t         local_bits;
    uint8_t         flags;
    pthread_t       owner;
    pthread_mutex_t lock;
};

Loading is the same as it was before. When we privatize for loading the transaction acquires the memory resource. The load() function would return a copy of the value, but after privatizing we can just read it directly.

Storing is different than before. So far storing worked by storing to a transaction-local buffer and later copying the content of this buffer to the shared memory during a commit. We can call this a write-back scheme, because we first keep a local copy and later write its content back to the global memory location.

When privatizing for store operations the transaction also acquires the memory. The difference to regular store() calls is that for privatizations, we require what we can call a write-through scheme. During the transaction’s execution, we directly write through to the shared memory.

This means that when the transaction commits, the shared values are already updated. For a rollback we have to restore the old value. Here’s an implementation of privatize().

void
privatize(uintptr_t addr, size_t siz, bool load, bool store)
{
    while (siz) {

        struct resource* res = acquire_resource(addr & BASE_BITMASK);
        if (!res) {
            tm_restart();
        }

        unsigned long index = addr & RESOURCE_BITMASK;
        unsigned long bits = 1ul << index;

        uint8_t* beg = arraybeg(res->local_value) + index;
        uint8_t* end = arrayend(res->local_value);

        while (siz && (beg < end)) {
            /* If we're about to store, we first have to
             * save the old value for possible rollbacks. */
            if (store && !(res->local_bits & bits) ) {
                *beg = *((uint8_t*)addr);
            }

            bits <<= 1;
            --siz;
            ++addr;
            ++beg;
        }

        if (store) {
            res->flags |= RESOURCE_FLAG_WRITE_THROUGH;
        }
    }
}

It’s very similar to the load and store functions we developed so far. The main difference is that for store privatizations, we save the shared value in the transaction-local buffer in the inner while loop. We will use this buffer for restoring previous values during a rollback.

For each store-privatized resource, we also set RESOURCE_FLAG_WRITE_THROUGH, which modifies the commit and rollback behavior. Commit and rollback is both implemented in the same function release_resource().

void
release_resource(struct resource* res, bool commit)
{
    pthread_t self = pthread_self();

    pthread_mutex_lock(&res->lock);

    if (res->owner && res->owner == self) {

        if (res->local_bits) {

            /* We have to store if we either commit in write-back
             * mode, or revert in write-through mode.
             */
            bool store_local_bits = commit != !!(res->flags & RESOURCE_FLAG_WRITE_THROUGH);

            if (store_local_bits) {
                unsigned long bit = 1ul;

                uint8_t* mem = (uint8_t*)res->base;
                uint8_t* beg = arraybeg(res->local_value);
                uint8_t* end = arrayend(res->local_value);

                while (beg < end) {
                    if (res->local_bits & bit) {
                        *mem = *beg;
                    }
                    bit <<= 1;
                    ++mem;
                    ++beg;
                }
            }

            res->local_bits = 0;
            res->flags = 0;
        }

        res->owner = 0;
    }

    pthread_mutex_unlock(&res->lock);
}

Here we modified the ‘write condition.’ Before, we only used write-back mode and unconditionally stored all updates. Now we have to distinguish between write-back and write-through mode. We have to write the transaction-local buffer to the shared memory, if

we commit in write-back mode (stores), or if
we revert in write-through mode (privatizations).

With these modifications we’ve implemented basic memory privatization. It should be noted that in the current implementation a transaction can either load and store or privatize a memory resource. The real-world implementation in picotm handles all combinations of operations automatically.

Producer-Consumer Transactions with Privatizations

What’s missing is an update to our example program to use the new functionality. Here’s how the producer looked until now with stores.

static void
producer_func(void)
{
    unsigned int seed = 1;

    tm_save int i0 = 0;

    while (true) {

        sleep(1);

        ++i0;
        int i1 = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i0, i1);

        tm_begin

            store_int(&g_i0, i0);
            store_int(&g_i1, i1);

        tm_commit
    }
}

We replace the invocations of store_int() with privatize() and memcpy().

static void
producer_func(void)
{
    unsigned int seed = 1;

    tm_save int i[2] = {0, 0};

    while (true) {

        sleep(1);

        ++i[0];
        i[1] = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i[0], i[1]);

        tm_begin

            privatize((uintptr_t)g_i, sizeof(g_i), false, true);

            memcpy(g_i, (const void*)i, sizeof(g_i));

        tm_commit
    }
}

The transaction privatizes the globally shared memory in g_i with a store privatization and copies the updated values there.

The consumer looks similar, but copies out of the shared memory.

static void
consumer_func(void)
{
    while (true) {

        sleep(1);

        int i[2];

        tm_begin

            privatize((uintptr_t)g_i, sizeof(g_i), true, false);

            memcpy(i, g_i, sizeof(i));

            verify_load(i[0], i[1]);

        tm_commit

        printf("Loaded i0=%d, i1=%d\n", i[0], i[1]);
    }
}

When not to Privatize?

In theory one could replace all loads and stores with privatizations, although that’s usually not a good idea.

Our implementation of privatization requires a transaction-local backup of the shared memory’s value. For workloads with mostly loads and a low number of conflicts, these copies could add a non-trivial overhead.

For storing, only one transaction can own the privatized shared resource. With workloads with many store operations to the same memory location, this could create unnecessary contention. For write-back stores, such overhead can be reduced by limiting the time stored memory is owned to each transaction’s commit phase.

Summary

With this blog post we add memory privatizations to our transaction manager.

Privatizations give transactions direct access to shared resources.
This can reduce overhead when wrapping non-transactional code. We used memcpy() as an example.
Memory resources can be privatized by providing a write-through mode for store operations.
A resource’s previous state has to be restored during a rollback in write-through mode.
Depending on the workload, privatization might have undesirable overhead.

You’ll find the full source code for this blog post on GitHub. If you’re interested in a more sophisticated C transaction manager, take a look at picotm.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Footnotes

Wikipedia describes a slightly different meaning of privatization in a slightly different context. ↩︎

picotm 0.2.0 released

blog@tzimmermann.org (Thomas Zimmermann) — Wed, 31 May 2017 15:00:00 +0200

Today I released version 0.2.0 of picotm, the system-level transaction manager. Feature highlights of the new release are consistent error reporting and much improved documentation with tutorials and interface references.

You can find the full announcement on the picotm homepage. The software is available for download from the GitHub release page.

Transactional Access to Arbitrary Memory Locations

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 26 May 2017 09:00:00 +0200

In this blog post we are going to implement transactional semantics for arbitrary locations in main memory. So far our simpletm transaction manager can only handle integer values that we store in a dedicated data structure. At the end of this installment, we’ll be able to work with values of arbitrary types that are located (almost) anywhere in memory.

In the previous installments we finished our transaction manager simpletm to contain miminal functionality for detecting and resolving conflicts among transactions, and we separated the transaction manager from the application logic by creating a generic C interface.

Loads and Stores for Integer Resources

For transactional memory access, our example program executes load and store operations on dedicted data structures of type struct int_resource, which is shown below.

struct int_resource {
    int             value;
    int             local_value;
    unsigned long   flags;
    pthread_t       owner;
    pthread_mutex_t lock;
};

The shared integer value is stored in the field value. Each instance of the structure is owned by a single transaction that stores in the field local_value and loads from either value or local_value, depending whether a local value is present. The flags field tells us about this. During the owning transaction’s commit the local value is copied to the actual value, on rollback the local value is discarded.

The corresponding load/store interface for an application looks like this.

void
load_int(struct int_resource* res, int* value);

void
store_int(struct int_resource* res, int value);

It’s specific to integer values and the application has to know about the struct int_resource data structure. Instead we should prefer an interface that can store a value of any type at any location in memory.

Working With Arbitrary Types

To support arbitrary types, let us rework struct int_resource a bit. We call this new structure struct resource.

#define RESOURCE_BITSHIFT   (3)
#define RESOURCE_NBYTES     (1ul << RESOURCE_BITSHIFT)

/**
 * An integer value with an associated owner.
 */
struct resource {
    uintptr_t       base;
    uint8_t         local_value[RESOURCE_NBYTES];
    uint16_t        local_bits;
    pthread_t       owner;
    pthread_mutex_t lock;
};

First of all, we remove the field value. Instead the base field tells us where in memory the resource is located. The value in base is the address of the first byte of a transactional value.

The field local_value is now a fixed-size byte array. It still stores the local value, but it’s not any longer type specific. In our case the array is 8 byte in size, so we can operate on values up to 8 bytes. Further below we will see how to arrange the resources such that values of arbitrary size can be supported.

Finally we replaced the flags field by local_bits. The latter is a bit field, where each bit signals the present of a transaction-local value in local_value. So if bit 0 is set local_value[0] contains a local value, if bit 1 is set local_value[1] contains a local value, and so on.

Working With Arbitrary Memory Locations

Our local_value stores 8 bytes and base could be any address. If we align base such that it always contains an address at an 8-byte boundary, its lowest three bytes will always be zero. For a memory address, these bits contain the index into local_value.

Here are the bits of a 32-bit memory address with their meaning for access to struct resource.

    | 31 .. 13  12 11 10  9  8  7  6  5  4  3 |  2  1  0 |
    |                  Base                   |   Index  |

For mapping a memory address to a byte in a resource structure, first we have to make sure that the base values of address and resource are the same, then use the address’ lowest three bits as index into local_value.

So far we’ve had two values of type struct int_resource. If we had a single resource structure in our transaction manager, we could load and store 8 consecutive bytes for the transactions. To load and store more memory, we have to add more instances; let’s say 1024.

The code fragment below creates an array of 1024 resource structures.

#define NRESOURCES_BITSHIFT (10)
#define NRESOURCES          (1ul << NRESOURCES_BITSHIFT)

struct resource g_resource[NRESOURCES];

We could assign a base address to each array element and search in the array whenever we want to map an address to a resource with corresponding base address. While this would work, there’s an easy way to compute the correct element directly.

Storing the position of element of the array requires exactly 10 bit. These 10 bit are located in the memory address directly before the index; or in other words, they are the next higher bits above the index.

    | 31 .. 13 | 12 11 10  9  8  7  6  5  4  3 |  2  1  0 |
    |          |             Element           |          |
    |                   Base                   |   Index  |

For mapping a memory address to a byte in a resource structure, we can extract the element position (bits 3 to 12), get the element from the element array we declared, make sure that the base values of address and resource are the same, and then use the address’ lowest three bits as index into local_value.

This look-up scheme is very fast and ensures that resources are consecutive in memory as well as the bytes of each resource. It is therefore possible to operate on values that are larger than 8 bytes. For example, if we load or store a value of 16 bytes, we first operate on the resource structure covering the value’s first bytes, then operate on the resource structure covering the next bytes, and so on.

Loading and Storing

Before we go into the details of the load and store functions, let us look at the new interfaces. Because we don’t use int directly any more, the new interfaces take memory addresses and buffers.

void
load(uintptr_t addr, void* buf, size_t siz);

void
store(uintptr_t addr, const void* buf, size_t siz)

The value addr is the address of the shared value. The application logic is not required to know about the resource any more. The transaction manager maintains these internally. The values buf and siz give a transaction-local buffer and the number of bytes in the buffer.

The complexity is now in the implementation of the load and store function. Here is store().

void
store(uintptr_t addr, const void* buf, size_t siz)
{
    const uint8_t* mem = (const uint8_t*)buf;

    while (siz) {

        struct resource* res = acquire_resource(addr & BASE_BITMASK);
        if (!res) {
            tm_restart();
        }

        unsigned long index = addr & RESOURCE_BITMASK;
        unsigned long bits = 1ul << index;

        uint8_t* beg = arraybeg(res->local_value) + index;
        uint8_t* end = arrayend(res->local_value);

        while (siz && (beg < end)) {
            *beg = *mem;
            res->local_bits |= bits;

            bits <<= 1;
            --siz;
            ++addr;
            ++mem;
            ++beg;
        }
    }
}

store() transactionally stores all bytes in buf to the resources that cover addr and successiv memory locations. The function call to acquire_resource() acquires the resource for a specific base address and returns the resource’s data structure. If the resource cannot be acquired, the transaction is in conflict, NULL is returned and the transaction aborts.

Once we have the resource, we start filling its local_value field with the contents of buf. For each byte we copy, we set the corresponding bit in local_bits. We do this until we reach the end of either buf or local_value. If we reached the end of local_value and there are still bytes left in buf we continue the outer while loop with the next resource.

The load function is identical, except for the actual copy operation.

void
load(uintptr_t addr, void* buf, size_t siz)
{
    uint8_t* mem = (uint8_t*)buf;

    while (siz) {

        struct resource* res = acquire_resource(addr & BASE_BITMASK);
        if (!res) {
            tm_restart();
        }

        unsigned long index = addr & RESOURCE_BITMASK;
        unsigned long bits = 1ul << index;

        uint8_t* beg = arraybeg(res->local_value) + index;
        uint8_t* end = arrayend(res->local_value);

        while (siz && (beg < end)) {
            if (res->local_bits & bits) {
                *mem = *beg;
            } else {
                *mem = *((uint8_t*)addr);
            }

            bits <<= 1;
            --siz;
            ++addr;
            ++mem;
            ++beg;
        }
    }
}

Here we walk over the local values of the involved resources and copy into buf. In the inner while loop, we check if there’s a local value and either copy this or the globally shared value.

Our current setup can support 8192 bytes of transactional memory. If this limit is reached or if multiple memory locations map to the same resource, the transaction manager aborts the process. This would be detected by acquire_resource(). As a workaround to this limit the size of the resource array can be doubled until all variables are covered. A real-world implementation could also replace the array by a sparse look-up tree. That’s how it’s done in picotm.

For convenience, we can keep the load_int() and store_int() functions around. They are now mere wrappers around load() and store().

int
load_int(const int* addr)
{
    int value;
    load((uintptr_t)addr, &value, sizeof(value));
    return value;
}

void
store_int(int* addr, int value)
{
    store((uintptr_t)addr, &value, sizeof(value));
}

With helpers as simple as these, there can be plenty of type-specific functions that load and store float, long, complex double, or any other type.

Converting the Application

The old producer code looked like this.

struct int_resource g_int_resource[2];

static void
producer_func(void)
{
    unsigned int seed = 1;

    tm_save int i0 = 0;

    while (true) {

        sleep(1);

        ++i0;
        int i1 = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i0, i1);

        tm_begin

            store_int(g_int_resource + 0, i0);
            store_int(g_int_resource + 1, i1);

        tm_commit
    }
}

This translates into the following code.

static int g_i0;
static int g_i1;

static void
producer_func(void)
{
    unsigned int seed = 1;

    tm_save int i0 = 0;

    while (true) {

        sleep(1);

        ++i0;
        int i1 = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i0, i1);

        tm_begin

            store_int(&g_i0, i0);
            store_int(&g_i1, i1);

        tm_commit
    }
}

The global values of type struct int_resource are gone. Instead we work directly on integer types.

The consumer side looks similar.

static void
consumer_func(void)
{
    while (true) {

        sleep(1);

        int i0, i1;

        tm_begin

            i1 = load_int(&g_i1);
            i0 = load_int(&g_i0);

            verify_load(i0, i1);

        tm_commit

        printf("Loaded i0=%d, i1=%d\n", i0, i1);
    }
}

As in the previous post, we added a certain complexity to the transaction manager, but the application logic got a lot simpler. And just as before, we had to implement the transaction manager exactly once. Further application-specific changes are not required.

Summary

In this blog post, we removed the dedicated resource data structures from the interface of simpletm. The new interface loads and stores at arbitrary memory locations and all resource management is done internally.

We can load and store arbitrary types.
We can protect concurrent access to arbitrary memory locations.
Applications don’t have to know about dedicated data structures for resource management. The interface for transactional memory access uses memory addresses of the involved variables.
We can further optimize transactional memory access without interfering with application logic.

You’ll find the full source code for this blog post on GitHub. If you’re interested in a more sophisticated C transaction manager, take a look at picotm.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks.

Writing the Beginning and End of a Transaction

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 19 May 2017 09:00:00 +0200

Last time we finished our transaction manager and example program simpletm to contain the minimum functionality for conflict detection and transaction rollback. In this blog post we are going to separate application logic and transaction logic from each other. We will move the latter into a distinct module with a generic C interface for beginning and committing transactions.

Towards a Generic Transaction Interface

Simpletm has two threads. A producer thread generates pairs of integer values and stores them in two globally shared variables. The consumer thread loads the stored values from the shared variables. All load and store operations are performed transactionally.

We ended the previous blog post with a producer function that looked like the code shown below.

static void
producer_func(void)
{
    unsigned int seed = 1;

    int i0 = 0;

    while (true) {

        sleep(1);

        ++i0;
        int i1 = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i0, i1);

        bool commit = false;

        while (!commit) {

            bool succ = store_int(g_int_resource + 0, i0);
            if (!succ) {
                goto release;
            }
            succ = store_int(g_int_resource + 1, i1);
            if (!succ) {
                goto release;
            }

            commit = true;

        release:
            release_int_resource(g_int_resource + 0, commit);
            release_int_resource(g_int_resource + 1, commit);
        }
    }
}

While this code works correctly, it’s poorly designed and not generally re-useable in other software.

The inner while loop serves as transaction control block. This is not obvious. We should prefer clear statements that indicate when we begin and commit a transaction.
We explicitly test the return value of each call to store_int() and abort if necessary. This is error prone and obfuscates the appliction logic. It would be preferable if store_int() could do this automatically for us.
We maintain the variable commit by hand. That is also error prone. If we forget to set commit to true, we end up in an end-less loop. If we set it too early (i.e., before all loads and stores have been executed), we might commit a conflicting transaction.
We track which resources the transaction uses and release them explicitly at the end of the transaction. It would be better if this could be done once for all transactions.

The consumer has similar problems. If we can move all this behind a common interface, we can reduce the overall code to the pure application logic.

The Interface

First of all we have to mark the beginning of a transactional block. Once a transaction has began, there are currently two cases we must handle: either our transaction commits and leaves the transaction, or the transaction aborts at some point and returns to the beginning of the transaction block. In both cases we have to release the transaction’s acquired resources.

Let’s start with implementing two statements. We call them tm_begin and tm_commit.

As the name indicates, tm_begin is where the transaction starts. Here’s the code.

void
_tm_begin()
{
    /* Nothing to do */
}

#define tm_begin    \
    _tm_begin();    \
    {

The function _tm_begin() is the place where we can put any set-up code that we might require. It’s currently empty.¹ The function is not supposed to be called directly. Instead there’s tm_begin. It’s a macro that calls _tm_begin() and then begins a new C code block. Note that tm_begin intentionally has no round brackets. It’s not a function call. It’s an element of code structuring.

The code for tm_commit is where we update and release the shared resources acquired by the transaction.

#define arraylen(_array)    \
    ( sizeof(_array) / sizeof(*(_array)) )

#define arraybeg(_array)    \
    ( _array )

#define arrayend(_array)    \
    ( arraybeg(_array) + arraylen(_array) )

void
release_int_resources(struct int_resource* beg,
                const struct int_resource* end, bool commit)
{
    while (beg < end) {
        release_int_resource(beg, commit);
        ++beg;
    }
}

void
_tm_commit()
{
    release_int_resources(arraybeg(g_int_resource),
                          arrayend(g_int_resources), true);
}

#define tm_commit       \
        _tm_commit();   \
    }

If tm_begin marks the beginning of our transactional code, tm_commit marks the end. It’s again a macro. It calls the function _tm_commit() and ends the C code block that was started by tm_begin. The intended side effect of this block is that all variables declared in the transactional code are transaction-local by default.

The commit operation’s implementation is in _tm_commit(). This function releases all shared integer resources. Then we have completed the transaction. The rest of the shown code is simply helpers.

The primitives shown so far is all we need for beginning and committing transactions. But what about conflict handling?

Non-Local Gotos in C

When a transaction observes a conflict during resource access, we want it to release the resources it acquired so far and restart from the beginning.

Conflicts are detected within store_int(), or load_int() for the consumer transaction. To restart from there we have to implement a non-local goto that jumps from store_int() to where we called tm_begin. It’s called non-local goto because we’re doing a goto operation over function-call boundaries.

Non-local gotos in C are implemented with the functions setjmp() and longjmp().² A call to setjmp() stores the processors current instruction counter and stack pointer for this thread. A later call to longjmp() restores these values, effectivly transfering program control back to the point where setjmp() had been called. The state values are stored in a jump buffer of type jmp_buf.

Here’s how the code looks like. First we need a thread-local jump buffer.

struct _tm_tx {
    jmp_buf env;
};

struct _tm_tx*
_tm_get_tx()
{
    /* Thread-local transaction structure */
    static __thread struct _tm_tx t_tm_tx;

    return &t_tm_tx;
}

We put the jump buffer into struct _tm_tx. The function _tm_get_tx() simply returns a thread-local instance of this structure.

To save the thread state in the jump buffer upon entering the transaction, we have to call setjmp() at the right time. Therefore we modify tm_begin as shown below.

void
_tm_begin(int value)
{
    /* Nothing to do */
}

#define tm_begin                            \
    _tm_begin(setjmp(_tm_get_tx()->env));   \
    {

The call to setjmp() is now done at the very beginning. It saves the instruction counter and stack pointer in the thread-local jump buffer and returns a value. When we later call longjmp() with the saved jump buffer, the transaction performs the non-local goto and setjmp() will return again.

Upon entering the transaction for the first time, setjmp() returns 0. At the time we call longjmp() we have the option of passing a different value, which setjmp() will return. Its return value is passed as an argument to _tm_begin(). This way we can distinguish between a new transaction and a restarted transaction in _tm_begin().

What we finally need is the restart function that is executed by store_int() upon detected conflicts.

void
tm_restart()
{
    release_int_resources(arraybeg(g_int_resource),
                          arrayend(g_int_resource), false);

    /* Jump to the beginning of the transaction */
    longjmp(_tm_get_tx()->env, 1);
}

void
store_int(struct int_resource* res, int value)
{
    bool succ = acquire_int_resource(res);
    if (!succ) {
        tm_restart();
    }

    res->local_value = value;
    res->flags |= RESOURCE_HAS_LOCAL_VALUE;
}

Like _tm_commit() the function tm_restart() releases the transaction’s resources, but of course without updating the global values. Instead of returning, tm_restart() invokes longjmp() with the saved jump buffer and an additional value. This transfers program control back to the transaction’s beginning where setjmp() saved the jump buffer. This time, setjmp() returns the second argument to longjmp(). In our case this is 1.

In store_int(), we now call tm_restart() upon detected conflicts. Notice that a call to store_int() cannot fail anymore. If it returns it has been successful, otherwise it automatically restarts the transaction.

Putting It All Together

While the details of setjmp() and longjmp() are certainly more complex than the while loop, it’s all contained in the transaction manager. And we’re effectivly done with this problem. There’s hardly any reason to ever change the code.

The application logic, however, looks much clear than before. Let’s put everything together. Here’s our new producer.

static void
producer_func(void)
{
    unsigned int seed = 1;

    int i0 = 0;

    while (true) {

        sleep(1);

        ++i0;
        int i1 = rand_r(&seed);

        printf("Storing i0=%d, i1=%d\n", i0, i1);

        tm_begin

            store_int(g_int_resource + 0, i0);
            store_int(g_int_resource + 1, i1);

        tm_commit
    }
}

We were able to condense the whole inner while loop into four statements were each statement is relevant to the application! Gone are the while loop, the explicit tests for successful stores, the commit variable, and the explicit releases at the end of the transaction.

After applying similar changes to the consumer, the new consumer looks like this.

static void
consumer_func(void)
{
    while (true) {

        sleep(1);

        int i0, i1;

        tm_begin

            load_int(g_int_resource + 1, &i1);
            load_int(g_int_resource + 0, &i0);

            verify_load(i0, i1);

        tm_commit

        printf("Loaded i0=%d, i1=%d\n", i0, i1);
    }
}

Again, we were able to remove all the transactional book keeping and reduce the code to the application logic!

Optimizing for Optimizers

There’s one final point about non-local gotos that we have to take care of. Remember that setjmp() only saves the processor’s instruction counter and stack pointer. It does not save any program variable’s value.

Program variables are subject to opimizations by the compiler. They can even be removed entirely and the value can be hold in a processor register. If we compile the simpletm binary with the gcc flags ‘-O2 -Wclobbered’, we might get a warning like the one shown below.

main.c: In function ‘producer_func’
main.c:33:9: warning: variable ‘i0’ might be clobbered by ‘longjmp’ or ‘vfork’ [-Wclobbered]

This means that the compiler’s optimizer moves around access to i0 or maybe even removes the variable. Such optimizations depend on the function’s structure and code. The information about how the longjmp() happens is simply not available to the optimizer because the call to longjmp() happens in a different function. After a longjmp() the value of i0 is undefined.

We can prevent this from happening by declaring the i0 as volatile, or as we call it tm_save. The volatile keyword prevents the compiler from optimizing the variable.

#define tm_save volatile

static void
producer_func(void)
{
    unsigned int seed = 1;

    tm_save int i0 = 0;

    [...]
}

From now on, we will compile transactional code with ‘-Wclobbered’ to make the compiler warn about this problem, and we will add tm_save where necessary.

Summary

In this blog post, we created a clean interface that let’s us separate our transaction manager from the application logic and keep both in separate modules.

We begin a transaction with tm_begin and commit with tm_commit.
Both statements are implemented by the transaction manager.
A call to tm_begin saves the instruction counter and stack pointer using setjmp(). For aborting we execute longjmp() to return to the beginning of the transaction.
Transaction aborts are performed internally by the transaction manager.
Resources are released automatically by the transaction manager during commits and aborts. No support from the application is required.
The application code is reduced to the actual application logic.

As always, the full source code for this blog post is available on GitHub. If you’re interested in a more sophisticated C transaction manager, take a look at picotm.

If you like this series about writing a transaction manager in C, please subscribe to the RSS feed, follow on Twitter or share on social networks. In the next installment, we will probably replace struct int_resource with support for arbitrary memory locations.

Footnotes

We will fill _tm_begin() with code when we get to error recovery in a later blog post. ↩︎
As word of warning, try to avoid using setjmp() and longjmp() in your software. It’s perfect for our transactions, but in general it often does more harm than good to program design. ↩︎

Transaction Roll-Back and Serializability

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 12 May 2017 09:00:00 +0200

In this blog post we’ll explore transaction roll-back code and briefly cover serializability. We’ll build upon our simple transaction manager and the theoretical foundation we’ve discussed last time. At the end of this post, we’ll have a simple, but correct transaction manager that can be adapted to arbitrary work loads.

A Closer Look at simpletm

Let’s take a closer look at simpletm, the transaction manager that we made before. Specifically let’s look at the producer and consumer code. Here’s the producer again.

void
producer_func(void)
{
    static int i0;
    int i1;

    /* produce i0 and i1 */
    ++i0;
    i1 = rand();

    print("Storing i0=%d, i1=%d\n", i0, i1);

    bool commit = false;

    while (!commit) {

        bool succ = store_int(g_int_resource + 0, &i0);
        if (!succ) {
            goto release;
        }
        succ = store_int(g_int_resource + 1, &i1);
        if (!succ) {
            goto release;
        }

        commit = true;

    release:
        release_int_resource(g_int_resource + 0);
        release_int_resource(g_int_resource + 1);
    }
}

It creates two values, i0 and i1, and stores them in shared resources; acquiring these resources along the way. If it can not acquire a resource, the producer releases all its resources and restarts.

Here’s the consumer.

void
consumer_func(void)
{
    int i0, i1;

    bool commit = false;

    while (!commit) {

        bool succ = load_int(g_int_resource + 1, &i1);
        if (!succ) {
            goto release;
        }
        succ = load_int(g_int_resource + 0, &i0);
        if (!succ) {
            goto release;
        }

        commit = true;

    release:
        release_int_resource(g_int_resource + 0);
        release_int_resource(g_int_resource + 1);
    }

    /* consume i0 and i1 */

    print("Loaded i0=%d, i1=%d\n", i0, i1);
}

Similar to the producer, the consumer loads two values from shared resources and acquires these resources along the way. If it fails to acquire a resource the consumer releases all its other resources and restarts.

Let’s further condense both code blocks to the load, store, and release functions that operate on shared resources. A transaction won’t see conflicts when accessing its private resources. So for our analysis, it’s only important how transactions interact with shared resources.

void
producer_func(void)
{
    store_int(g_int_resource + 0, &i0);
    store_int(g_int_resource + 1, &i1);
    release_int_resource(g_int_resource + 0);
    release_int_resource(g_int_resource + 1);
}

void
consumer_func(void)
{
    load_int(g_int_resource + 1, &i1);
    load_int(g_int_resource + 0, &i0);
    release_int_resource(g_int_resource + 0);
    release_int_resource(g_int_resource + 1);
}

Producer and consumer run concurrently, so invocations of the load, store, and release functions can interleave in any way. Let’s reorder the individual calls… and a problem will appear.

store_int(g_int_resource + 0, &i0);         /* producer acquires resource 0 and performs store */

/* context switch */

load_int(g_int_resource + 1, &i1);          /* consumer acquires resource 1 and performs load */

/* context switch */

store_int(g_int_resource + 1, &i1);         /* producer fails to acquire resource 1 */
release_int_resource(g_int_resource + 0);   /* producer releases resource 0 */

/* context switch */

load_int(g_int_resource + 0, &i0);          /* consumer acquires resource 0 and performs load */

/* THE CONSUMER JUST LOADED THE VALUE PREVIOUSLY STORED BY THE ABORTED PRODUCER! */

Oh! If we only execute the first store and the first load, and then abort the producer, the consumer will load the value that has just been stored by the producer.

Is this actually correct? Before, we could only have guessed, but in the previous blog post, we established a theoretical foundation and talked about the ACID properties.

Our property of atomicity required transactions to be either executed completely, or not at all. In the example at hand we only executed half of the producer’s transaction, namely the first store; therefore violating the atomicity principle.

Consequently we also violated the isolation property, as we leaked internal changes of the producer into the global state.

We specified early int the original blog post that we want to update either both resources or none. So we also violated our established notion of consistency.

Clearly, the current behavior is not correct.

Test it for Yourself

Usually you won’t see this problem, because the sleep() and printf() statements slow down thread execution considerably. If you want to trigger the bug, take the original code of the simpletm and replace producer_func() and consumer_func() with the code shown below.

static void
producer_func(void)
{
    unsigned int seed = 1;
    int i0 = 0;

    while (true) {

        ++i0;
        int i1 = rand_r(&seed);

        bool commit = false;

        while (!commit) {

            bool succ = store_int(g_int_resource + 0, i0);
            if (!succ) {
                goto release;
            }
            succ = store_int(g_int_resource + 1, i1);
            if (!succ) {
                goto release;
            }

            commit = true;

        release:
            release_int_resource(g_int_resource + 0);
            release_int_resource(g_int_resource + 1);
        }
    }
}

static void
verify_load(int i0, int i1)
{
    unsigned int seed = 1;

    int i;
    int value = 0;

    for (i = 0; i < i0; ++i) {
        value = rand_r(&seed);
    }

    if (value != i1) {
        printf("Incorrect value pair (%d,%d), should be (%d,%d)\n",
               i0, i1, i, value);
    }
}

static void
consumer_func(void)
{
    while (true) {

        int i0, i1;

        bool commit = false;

        while (!commit) {

            bool succ = load_int(g_int_resource + 1, &i1);
            if (!succ) {
                goto release;
            }
            succ = load_int(g_int_resource + 0, &i0);
            if (!succ) {
                goto release;
            }

            verify_load(i0, i1);

            commit = true;

        release:
            release_int_resource(g_int_resource + 0);
            release_int_resource(g_int_resource + 1);
        }
    }
}

The replacement code has the invocations of sleep() and printf() removed and verifies the loaded values. When you run it, you should see plenty of statements like the ones below on the terminal.

Incorrect value pair (17103,1980393415), should be (17103,507428599)
Incorrect value pair (17104,507428599), should be (17104,1058612930)
Incorrect value pair (17111,1861807294), should be (17111,2062867441)
Incorrect value pair (17112,2062867441), should be (17112,1510167188)

This comes from the consumer thread, which failed to verify the correctness of the loaded values.

Notice that if i0 is 17103 the consumer expects i1 to be 507428599, but this value is only loaded later when i0 is 17104. This is because the producer stored 17104 for i0 and then aborted, keeping i1 at the old value. The same happens for 17111 and 17112.

Introducing Serializability

Remember from the previous post that the cumulative effects of a set of transactions shall look as if the transactions had been executed one by one. This is called serializability.

A specific execution of a set of transactions is called a history. For example, the execution near the beginning of this post, the one that showed up the problem, is a history.

In our code we have two transactions that we execute concurrently. If we execute all operations of one transaction strictly before or after all operations of the other transaction, we have a serial history. This generalizes to histories containing any number of transactions.

A history is serializable if the effects of history’s committed transactions are equivalent to an actual serial history. This means that if we execute and commit all our transactions concurrently, the result must be the same as if we had executed and committed the transactions one by one. The exact order of the transactions doesn’t matter.

To make a history look like a serial one, we have ensure is that each transaction always sees the shared state as if no other transaction was present. If a transaction detects a conflict, it has to revert its own changes as if they never happened.

Fixing the Code

Our transaction manager already detects conflicts among transactions that access the same resource concurrently. This means that we have already implemented half of the requirements for serializability. What the transaction manager is still missing is a way of reverting a transaction’s changes when the transaction aborts.

We implement this by introducing transaction-local state for each resources.

#define RESOURCE_HAS_LOCAL_VALUE    (1ul << 0)

struct int_resource {
    int             value;
    int             local_value;    /* the transaction-local state */
    unsigned long   flags;          /* flag set if local state present. */
    pthread_t       owner;
    pthread_mutex_t lock;
};

The owner of the resource always stores only to the local state; and it always loads from the local state if that is present.

static bool
store_int(struct int_resource* res, int value)
{
    bool succ = acquire_int_resource(res);
    if (!succ) {
        return false;
    }

    /* only store to local state */
    res->local_value = value;
    res->flags |= RESOURCE_HAS_LOCAL_VALUE;

    return true;
}

static bool
load_int(struct int_resource* res, int* value)
{
    bool succ = acquire_int_resource(res);
    if (!succ) {
        return false;
    }

    /* return the local state if we have executed a store() before */
    if (res->flags & RESOURCE_HAS_LOCAL_VALUE) {
        *value = res->local_value;
    } else {
        *value = res->value;
    }

    return true;
}

The local state becomes globally visible if, and only if, (!) the transaction commits. This happens just before releasing the resource.

static void
release_int_resource(struct int_resource* res, bool commit)
{
    pthread_t self = pthread_self();

    pthread_mutex_lock(&res->lock);

    if (res->owner && res->owner == self) {

        /* if we commit, copy the local state to the global state */
        if (res->flags & RESOURCE_HAS_LOCAL_VALUE) {
            if (commit) {
                res->value = res->local_value;
            }
            res->flags &= ~RESOURCE_HAS_LOCAL_VALUE;
        }

        res->owner = 0;
    }

    pthread_mutex_unlock(&res->lock);
}

Every time we call release_int_resource(), we now have to tell it whether we’re comitting or not.

static void
producer_func(void)
{
    [...]
            commit = true;

        release:
            release_int_resource(g_int_resource + 0, commit);
            release_int_resource(g_int_resource + 1, commit);
        }
    }
}

If we commit a transaction, its local effects become globally visible. If we abort, the transaction’s effects are thrown away.

These fairly small changes fix the incorrect state that the consumer was observing. If you run this code, even with sleep() and printf() removed, you won’t see any warnings about incorrect value pairs.

Summary

In this post, we looked at serializability and how to implemenent it in our transaction manager.

A serial history is a set of transactions that are executed one by one.
We want concurrent executions of a history to have the same result as a serial one. This is called serializability.
We have to revert a transaction’s changes when the transaction aborts.
We introduced transaction-local state for each resource. The local state becomes globally visible iff the transaction commits.

As before, the full source code for this blog post is available on GitHub. The code should now be complete enough to be useful for an arbitrary amount of producers and consumers, or threads that perform both.

In the next installment, we’ll replace the while loop in the producer and consumer functions with something nicer. The result will be a cleaner and more maintainable code base.

Transactional Semantics in Theory and Practice

blog@tzimmermann.org (Thomas Zimmermann) — Tue, 09 May 2017 18:00:00 +0200

In the previous installment, we implemented a simple transaction manager, but we didn’t really say what it means to run code as a transaction, which features are considered transactional and which aren’t.

In this blog post, we’re first going to examine the theoretical priciples of transactional semantics, and afterwards look at how to put them into practice.

Why Bother?

It’s certainly fair to ask why we should bother.

Image a software program of your choice. If we take a step backwards from the actual code, we can describe the software as a state machine.

As the name implies, the state machine is always in a state. What the state consists of depends on the software. For example, a chess game might be in a state where it waits for the player’s next move. While it does this, it doesn’t change. At the software level, think of state as a combination of the values of all data structures.

Now something happens. Maybe the user pressed a key, or a hardware sensor fired, or a new packet arrived on the network. Our state machine responds according to which state it is in. When it does so, it performs a specific action and moves into a new state. This movement from one state to the next while performing an action in between is called a transition.

In a real-world program, the action is implemented by software. When the state is in data structures, the action is in algorithms.

When the software is in a specific state, a specific input triggers a specific action, which transitions the software into a new state. Normally all states should be well-defined and transitions should always move the state machine from one well-defined state to another well-defined state.

So far, so good. But back in the real world, not all state can handle all input; and if it can, the associated action can be faulty. When this happens, the state machine goes from a well-defined state into an undefined state. What happens next is not specified.

We call this a software bug.

If we could implement our transition such that it detects incorrect input or internal problems automatically, and has the option of going back to the old, well-defined state; we’d be able to rule-out a large number of incorrect transitions into undefined states.

This is were the transaction concept comes in handy.

The ACID Properties

Transactional semantics are defined by four properties, which the transaction manager guarantees to its transactions. Those are called the ACID properties, from their initial letters. If you had at least a minimal exposure to database theory, you have already seen them before.

The ACID properties are

Atomicity,
Consistency,
Isolation, and
Durability.

Let’s go through them one by one. Consistency directly follows from our previous discussion of states and transations, so let’s start with this.

Consistency describes a requirement to move from one well-defined state to another. For example, the HTML documents of a website refer to each other via hyper links. If one of the document’s files gets renamed, the consistency constraints could require that all links in all documents refering to the renamed file should be updated to the new filename.

What is considered well-defined depends on the software at hand; therefore guarantee-ing consistency always requires some help from the application. The transaction system can only help with implementing this. In our example, if the rename-and-update transaction simple ignores some of the HTML documents for an update, the result is probably inconsistent.

To guarantee consistency, it’s obvious that we have to guarantee atomicity as well. Atomicity specifies that we either do all of the operations contained in a transaction, or none. That’s something, the transaction system can help a lot with. In the example with the HTML documents, the transaction system would ensure that all documents are updated, or, if updating is not possible, revert to the old state before the rename.

So far we’ve only considered a single transaction. Now let’s add another, concurent transaction into the mix. This brings us to isolation. The isolation requirement specifies that the effects of a transaction shall not interfere with the effects of another, concurrent transaction.

To give an example of isolation, if both of our transactions rename filenames of HTML documents and update the references, they should not overwrite each others changes. In most cases the cumulative effect of both transactions should look as if the transactions had been executed one-after-the-other.¹

Finally, we have durability. Durability, also known as persistence, specifies that effects of a transaction persist, once the transaction committed the effects. A later transaction will always see these changes and not a previous state missing them. For our HTML example this means that once a document has been successfully renamed, a later transaction will not see the old filename.

Durability is often not achievable by the transactional system alone. It requires an environment that ensures certain constaints. For example, if an HTML document gets renamed and suddenly the disk drive fails, the rename can get lost. The environment must apply sophisticated back-up and replication strategies to avoid this.

Putting Theory into Practice

The ACID properties are most of all requirements, but not practical solutions. This is the job of the transaction manager.

The transaction manager provides the transactions with a number of operations. For each operation it implements two features. These are

error handling, and
concurrency control.

Who really cares about handling all errors correctly? Handling means detecting every error and always being able to move back into a previous consistent state. The latter is called recovery.

Common sense says that error handling is the worst part of each software. Errors are hard to test for and rarely happen. The error handling is therefore the code that gets executed the least. Recovery is often complicated as it requires significant effort to being able to back in state. In traditional software, this is almost impossible.

Transaction managers implement all logic required to detect every possible error for each of its operations, and contain the logic for going back into a previous state; therefore recovering from the detected error.

This provides a significant benefit over traditional approaches where application logic and error handling are written side by side. Because the transaction manager implements error handling once, all transactions automatically employ it. Therefore all transactional application logic automatically employs it as well. This gives us the atomicity and consistency requirements from the ACID properties.

With a good transaction manager, all the error handling that’s left for the application is the pathological cases, when the software reaches hardware limits or sees hardware failure. Even in these cases, error detection is still provided by the transaction manager and the application at least gets a chance of reporting the error to an administrator and shutting down safely.

Concurrency control describes the logic that controls resource access and protects resources against concurrent access from multiple transactions. A resource can be any data structure that transactions might want to use, maybe a file or shared memory.

There’s a lot to concurrency control, but you can think of it as locking. Locking happens to be the most common implementation.

Controlling resource access gives us the atomicity, consistency and isolation requirements of the ACID properties. Without concurrency control, transactions would constantly overwrite each others effects and operate on inconsistent, half-committed state changes made by other transactions.

Summary

There was quite of theory on this blog post. We first

looked at software as a set of states that are connected by transitions.

We can make these transitions far less error prone by guarantee-ing the properties of

atomicity,
consistensy,
isolation, and
durability.

This is called a transaction. A transaction manager helps with implementing these guarantees by providing

error handling, and
concurrency control.

Next time, we’ll get back to simpletm, our toy transaction manager. If you’re interested in a complete transaction manager, take a look at picotm.

Footnotes

This one-after-the-other requirement is called serializability. We’ll examine this in a later post. ↩︎

Implementing a Simple Transaction Manager in C

blog@tzimmermann.org (Thomas Zimmermann) — Fri, 05 May 2017 10:05:01 +0200

In this blog post, we’re going to write a simple transaction manager in C. It will load and store integer variables in memory, while handling locks automatically. The full source code is available on GitHub.

A transaction manager provides two important guarantees to each transaction. These are isolation from the effects of concurrent transactions, and error handling.¹ For now we only go for isolation. Error handling requires a more elaborated software design than what is possible in a single blog post.

Before writing a transaction manager, the first thing to do is to look at the involved resources. In our case these are two global integer values.

    int g_i0;
    int g_i1;

These values are stored in memory. Transaction systems for memory are called Software Transactional Memory. In principle, we could use arbitrary types of resources or even combine resources of different types in a single transaction. For simplicity, we stick with memory.

So what do we do with these two values? For our example, we use two threads in a producer-consumer scenario. The producer thread stores a value in each of g_i0 and g_i1, the consumer thread reads them.

An important constraint is that the values in these variables are dependent on each other: either both change or neither changes. We achieve this by acquiring a lock while accessing g_i0 or g_i1.

A non-transactional Example…

Let’s take a look at some, still non-transactional, example code. Our producer looks like this.

    static int g_i0;
    static int g_i1;

    static pthread_mutex_t g_lock0; /* Lock for g_i0 */
    static pthread_mutex_t g_lock1; /* Lock for g_i1 */

    void
    producer_func(void)
    {
        static int i0;
        int i1;

        /* produce i0 and i1 */
        ++i0;
        i1 = rand();

        print("Storing i0=%d, i1=%d\n", i0, i1);

        pthread_mutex_lock(&g_lock0);
        pthread_mutex_lock(&g_lock1);

        g_i0 = i0;
        g_i1 = i1;

        pthread_mutex_unlock(&g_lock0);
        pthread_mutex_unlock(&g_lock1);
    }

The producer generates two values i0 and i1. The value of i0 increases monotonically. It might be a timestamp. The value of i1 changes arbitrarily; maybe it’s the output of a hardware sensor. By acquiring locks g_lock0 and g_lock1, the producer enters a critical section where it updates the global values.

Our consumer looks like this.

    void
    consumer_func(void)
    {
        int i0;
        int i1;

        pthread_mutex_lock(&g_lock1);
        pthread_mutex_lock(&g_lock0);

        i0 = g_i0;
        i1 = g_i1;

        pthread_mutex_unlock(&g_lock0);
        pthread_mutex_unlock(&g_lock1);

        /* consume i0 and i1 */

        print("Loaded i0=%d, i1=%d\n", i0, i1);
    }

Again we have two local values i0 and i1. This time they are not generated, but updated from their global counterparts. Like the producer, the consumer enters a critical section by acquiring the two locks. It copies the values and prints them to the terminal.

… with a Problem.

This is all nice and good… except it isn’t. You’ve probably spotted the problem already. Our producer excecutes

    pthread_mutex_lock(&g_lock0);
    pthread_mutex_lock(&g_lock1);

while our consumer concurrently executes

    pthread_mutex_lock(&g_lock1);
    pthread_mutex_lock(&g_lock0);

If they interleave, we get

    pthread_mutex_lock(&g_lock0); /* producer */
    pthread_mutex_lock(&g_lock1); /* consumer */
    pthread_mutex_lock(&g_lock1); /* producer */
    pthread_mutex_lock(&g_lock0); /* consumer */

This is the classical ABBA deadlock. The only way to resolve it is to detect it beforehand and have one thread release its lock, so that the other thread can make progress.

Our transaction manager does this for us!

Rewriting as Transactions

In a transactional code, each resource is somehow owned by the transactions that use it. All this is really not the problem of the transactions themselves, but the transaction manager that controls the transactions. Each transaction tells the transaction manager what resources it requires and which operations it performs on them. The transaction manager handles the details of resource ownership.

As a first step, let us wrap shared integer resources and information about their current owner in a data structure. This simplifies the later steps by a large amount.

    struct int_resource {
        int             value;  /* only accessed by owner */
        pthread_t       owner;  /* protected by lock */
        pthread_mutex_t lock;
    };

This structure has three fields. The value field is the value as before. The owner field is the thread of the transaction that currently owns the resource. A transaction has to acquire ownership from the transaction manager before it can access the value field. Finally, the field lock protects the owner field.

In the next step, we introduce the functions for acquiring and releasing ownership of a resource.

    bool
    acquire_int_resource(struct int_resource* res)
    {
        pthread_t self = pthread_self();

        pthread_mutex_lock(&res->lock);

        if (res->owner && res->owner != self) {
            /* Owned by another thread. */
            goto err_has_owner;

        } else if (!res->owner) {
            /* Now owned by us. */
            res->owner = self;
        }

        pthread_mutex_unlock(&res->lock);

        return true;

    err_has_owner:
        pthread_mutex_unlock(&res->lock);
        return false;
    }

The acquire function acquire_int_resource() locks the owner field and checks its value. If someone else is the owner, it reports failure by returning false. This will later lead to an abort of the transaction. If the resource currently has no owner, acquire_int_resource() sets the transaction’s thread as the owner and reports success by returning true. If the transaction already owns the resource, it also reports success.

    void
    release_int_resource(struct int_resource* res)
    {
        pthread_t self = pthread_self();

        pthread_mutex_lock(&res->lock);

        if (res->owner && res->owner == self) {
            /* Release the resource */
            res->owner = 0;
        }

        pthread_mutex_unlock(&res->lock);
    }

The release function release_int_resource() only has to check if the transaction is the current owner of the resource. If so, it clears the owner field. If not, it returns silently.

You probably noticed that there’s not yet a single integer involved here. We could move this code into a separate module and re-use it for any kind of resource.

The integer values are introduced now. We do this by wrapping the load and store operations in two functions load_int() and store_int(). These functions will do all the work required for accessing shared integer resources.

    bool
    load_int(struct integer_res* res, int* value)
    {
        bool succ = acquire_int_res(res);
        if (!succ) {
            return false;
        }

        *value = res->value;

        return true;
    }

    bool
    store_int(struct integer_res* res, int value)
    {
        bool succ = acquire_int_res(res);
        if (!succ) {
            return false;
        }

        res->value = value;

        return true;
    }

Both functions first try to acquire the resource by calling acquire_int_res(). If successful they perform their respective operation by either loading or storing a value. Like the acquire function, they return true or false, depending on whether or not the resource could be acquired in the first place.

Transactional Producers and Consumers

Phew! I hope you’re still with me.

What we’ve done so far is to write a producer-consumer scenario using locks, only to discover that our code is prone to deadlocks.

From their we went towards implementing transaction support. We first wrapped our resource in a data structure, so that is can be maintained by the transaction manager, then we created interfaces for accessing and releasing the resource.

Now you might wonder where that transaction manager actually is. Well, in some way it’s already there. The core of our transaction manager is in the acquire and release functions and in the resource structure. This code does provide us with everything we need to ensure isolation among transactions.

The only thing missing are the producer and consumer threads that use our wanna-be transaction manager. Let’s take a look at the producer.

    static struct int_resource g_int_resource[2];

The global values are now stored in the instances of struct int_resource in g_int_resource.

Like before the producer creates two values and prints them to the terminal. Unlike before, there are no locks anymore. In fact, the whole critical section has been replaced by a while loop.

    void
    producer_func(void)
    {
        static int i0;
        int i1;

        /* produce i0 and i1 */
        ++i0;
        i1 = rand();

        print("Storing i0=%d, i1=%d\n", i0, i1);

        bool commit = false;

        while (!commit) {

            bool succ = store_int(g_int_resource + 0, &i0);
            if (!succ) {
                goto release;
            }
            succ = store_int(g_int_resource + 1, &i1);
            if (!succ) {
                goto release;
            }

            commit = true;

        release:
            release_int_resource(g_int_resource + 0);
            release_int_resource(g_int_resource + 1);
        }
    }

You probably remember that the transaction might not be able to acquire ownership of a resource. In this case, it has to release its locks and restart from the beginning. That’s how we prevent the ABBA deadlock scenario that our original code couldn’t handle.

The transaction starts with commit set to false. If any invocation of store_int() returns false, it was not able to acquire ownership of the resource. The transaction jumps to release where it releases all its resources. These resources are now available for being acquired by a concurrent transaction. Since commit is still false, the while loop continues and the transaction starts anew.

Be warned that the producer code doesn’t yet roll-back correctly. We’ll fix this in one of the next installments, but don’t put this example into practice as it is.

If all invocations of store_int() succeed, the transaction eventually sets commit to true. It will then also release its resources, but now the while loop breaks. The transaction has committed its results.

The while loop is good for educational purposes, not an optimal software design, and not what you’d use in a real-world transaction manager. We’ll build something better in a later blog post.

Now that we have a producer, let’s also look at the consumer side.

    void
    consumer_func(void)
    {
        int i0, i1;

        bool commit = false;

        while (!commit) {

            bool succ = load_int(g_int_resource + 1, &i1);
            if (!succ) {
                goto release;
            }
            succ = load_int(g_int_resource + 0, &i0);
            if (!succ) {
                goto release;
            }

            commit = true;

        release:
            release_int_resource(g_int_resource + 0);
            release_int_resource(g_int_resource + 1);
        }

        /* consume i0 and i1 */

        print("Loaded i0=%d, i1=%d\n", i0, i1);
    }

Again we replaced the critical section by a while loop. The principles of aborting and committing the transaction are the same here as in the producer, just that this transaction calls the load_int() function.

If you run this example, it should display something like the output below on the terminal.

    Loaded i0=0, i1=0
    Storing i0=1, i1=1804289383
    Loaded i0=1, i1=1804289383
    Storing i0=2, i1=846930886
    Storing i0=3, i1=1681692777
    Loaded i0=3, i1=1681692777

Summary

I hope you enjoyed the quick tour through a very simple transaction manager. While the presented code is simple, it already contains many of the basic building blocks of a complete implementation.

The transaction manager provides isolation and error handling to its transactions.
Transactions interact with transactional resources by invoking transactional interfaces.
Non-transactional code is prone to deadlocks, transactional code isn’t.²
Conflicting access is resolved by the transaction manager. This can lead to an abort and restart of the involved transactions.

The full source code for this blog post is available on GitHub.

There are more blog posts to come that talk about all kinds of topics around transactions. If you’re interested in a complete transaction manager take a look at picotm.

Footnotes

In database context, these guarantees are known as the ACID properties. ↩︎
We’ll talk about livelocks in a later blog post. ↩︎

Thank you, Jekyll!

blog@tzimmermann.org (Thomas Zimmermann) — Tue, 02 May 2017 15:05:01 +0200

Just a quick Thank you to everyone involved with Jekyll.

Before I get to my own topics, I think it’s appropriate that I thank everyone who is involved with Jekyll for their work. Jekyll is the CMS software I’m using, and I could not have made this website and blog without it.

A special Thank you goes out to Stijn, who’s Jekyll theme Holo Alfa I’m using.

Fault-Tolerance on tzimmermann dot org

Switching to Hugo

Safe Integer Conversion in C

Case 1: Unsigned to Unsigned

Conversion Ranks

Case 2: Signed to Signed

Case 3: Unsigned to Signed

Case 4: Signed to Unsigned

Putting It All Together

Summary

Footnotes

System-Level Transactions with picotm

The Problems with Traditional Code

Transactional Execution

Phases of each Operation

System-Level Transactions with picotm

picotm Modules

Obtaining picotm

A picotm Demo Application

The Input Thread

The Processing Threads

The Output Thread

Summary

Porting picotm to Mac OS, Windows and FreeBSD

FreeBSD

Mac OS

Windows

Next Steps

Summary

Transactional Multisets

Multiset Entries

Multiset States

Acquiring a Multiset

Inserting Entries

Removing Entries

Multiset Size and Emptieness

Iterating over the Multiset’s Entries

Searching Entries and Ranges

Multiset Clean-up

Summary

Footnotes

Transactional Queues and Stacks

A Brief Recap of Transactional Lists

Transactional Queues

Transactional Stacks

Summary

Footnotes

Transactional Linked Lists

The Transaction Interface

Why Do We Want Lists?

List Entries

The Shared List State

Finally, Lists!

Adding And Removing List Entries

List Capacity

Iterating Over List Entries

Clearing The List

Next Steps

Summary

Scaling Lock Performance

The Test Setup

The Original Code of picotm 0.6

Adding Reader/Writer Locks

Adding Exclusive Mode

Adding a Lock Manager

Where To Go From Here?

Summary

When realloc() Doesn't Allocate

A Simple realloc()

Handling the Returned Value

Summary

Benchmark Visualization with LaTeX and gnuplot

Turning Raw Numbers into Actionable Data

Generating the Raw Data

Generating the Diagram Data

Generating a LaTeX document

Generating a PDF Document

Ideas for Future Improvements

Summary

Footnotes

A Simple `realloc()`

The Semantics of `malloc()`

Second Try: Testing `errno`