Refactoring C Code: Going to Async I/O

Now that I have a good idea on how to use OpenSSL andlibuvtogether , I’m going to change my code to support that mode of operation. I have already thought about this a lot, and the code I already have is ready to receive the change in behavior, I think.

One of the things that I’m going to try to do while I move the code overis properly handleall error conditions. We’ll see how that goes.

I already have the concept of a server_state_run () method that handles all the network activity, dispatching, etc. So that should make it easy. I’m going to start by moving all thelibuvcode there. I’m also going to take the time to refactor everything to an API that is more cohesive and easier to deal with.

There is some trouble here, with having to merge together two similar (but not quite identical) concepts. MylibuvandOpenSSL post dealt with simply exposing a byte stream to the calling code. My network protocol code is working at a higher level. Initially, I tried to layer things together, but that quickly turned out to be a bad idea. I decided to have a single layer that handles both the reading from the network, using OpenSSL and parsing the commands over the network.

The first thing to do was to merge the connection state, I ended up with this code:

struct tls_uv_connection_state_private_members { server_state_t* server; uv_tcp_t* handle; SSL *ssl; BIO *read, *write; struct { tls_uv_connection_state_t** prev_holder; tls_uv_connection_state_t* next; int in_queue; size_t pending_writes_count; uv_buf_t* pending_writes_buffer; } pending; size_t used_buffer, to_scan; int flags; }; #define RESERVED_SIZE (64 - sizeof(struct tls_uv_connection_state_private_members)) #define MSG_SIZE (8192 - sizeof(struct tls_uv_connection_state_private_members) - 64 - RESERVED_SIZE) // This struct is exactly 8KB in size, this // means it is two OS pages and is easy to work with typedef struct tls_uv_connection_state { struct tls_uv_connection_state_private_members; char reserved[RESERVED_SIZE]; char user_data[64]; // location for user data, 64 bytes aligned, 64 in size char buffer[MSG_SIZE]; } tls_uv_connection_state_t; static_assert(offsetof(tls_uv_connection_state_t, user_data) % 64 == 0, "tls_uv_connection_state_t.user should be 64 bytes aligned"); static_assert(sizeof(tls_uv_connection_state_t) == 8192, "tls_uv_connection_state_t should be 8KB");

There are a few things that are interesting here. On the one hand, I want to keep the state of the connection private, but on the other, we need to expose this out to the user to use some parts of it. The waylibuvhandles it is with comments denoting whatareconsidered public/private portions of the interface. I decided to stick it in a dedicated struct. This also allowed me to get the size of the private members, which is important for what I wanted to do next.

The connection state structhavethe following sections:

private/reserved 64 bytes available foruserto use 64 bytes (and aligned on 64 bytes boundary) msg buffer 8,064 bytes

The idea here is that we give the user some space to keep their own datain,and that the overall connection state size is exactly 8KB, so can fit in two OS pages. On linux, in most cases, we’ll not need a buffer that is over 3,968 bytes long, we can even save thesecond pagematerialization (because the OS lazily allocate memory to the process). I’m using 64 bytes alignment for the user’s data to reduce any issues that the user have for storing data about the connection. It will also keep it nicely within the data the userneedto handle the connection nearby the actual buffer.

I’m 99% sure that I won’t need any of these details, but I thought it is best to think ahead, and it was fun to experiment.

Here is how the startup code for the server changed:

connection_handler_t handler = { print_all_errors, on_connection_dropped, create_connection, on_connection_recv }; server_state_init_t options = { cert, key, "0.0.0.0", 4433, &handler, { // allowed certs "1776821DB1002B0E2A9B4EE3D5EE14133D367009" , "AE535D83572189D3EDFD1568DC76275BE33B07F5" }, 2 // number of allowed certs }; srv_state = server_state_create(&options);

I removed pretty much all the functions that were previously used to build it. We have the server_state_init_t struct, which contains everything that is required for the server to run. Reducing the number of functions to build this means that I have to do less and there is a lot less error checking to go through. Most of the code that I had to touch didn’t require anything interesting. Take the code from thelibuv/opensslproject, make sure it compiles, etc. I’m going to skip talking about the boring stuff.

I did run into a couple of issues that are worth talking about. Error handling and authentication. As mentioned, I’m using client certificates for authentication, but unlike my previous code, I’m not explicitly calling SSL_accept() , instead, I rely on OpenSSL to manage the state directly.

This means that I don’t have a good location to put the checks on the client certificate that is used. For that matter, our protocol starts with the server sending an: “OK\r\n” message to the client to indicate a successfulconnection. Where does this go? I put all of this code inside the handle_read() method.

int ensure_connection_intialized(tls_uv_connection_state_t* state) { if (state->flags & CONNECTION_STATUS_INIT_DONE) return 1; if (SSL_is_init_finished(state->ssl)) { state->flags |= CONNECTION_STATUS_INIT_DONE; if (validate_connection_certificate(state) == 0) { state->flags |= CONNECTION_STATUS_WRITE_AND_ABORT; return 0; } return connection_write(state, "OK\r\n", 4); } return 1; } void handle_read(uv_stream_t *client, ssize_t nread, const uv_buf_t *buf) { tls_uv_connection_state_t* state = client->data; if (nread <= 0) { push_libuv_error(nread, "Unable to read"); state->server->options.handler->connection_error(state); abort_connection_on_error(state); return; } int rc = BIO_write(state->read, buf->base, nread); assert(rc == nread); while (1) { int rc = SSL_read(state->ssl, buf->base, buf->len); if (rc <= 0) { rc = SSL_get_error(state->ssl, rc); if (rc != SSL_ERROR_WANT_READ) { push_ssl_errors(); state->server->options.handler->connection_error(state); abort_connection_on_error(state); break; } maybe_flush_ssl(state); ensure_connection_intialized(state); // need to read more, we'll let libuv handle this break; } // should be rare: can only happen if we go for 0rtt or something like that // and we do the handshake and have real data in one network roundtrip if (ensure_connection_intialized(state) == 0) break; if (state->flags & CONNECTION_STATUS_WRITE_AND_ABORT) { // we won't accept anything from this kind of connection // just read it out of the network and let's give the write // a chance to kill it continue; } if (read_message(state, buf->base, rc) == 0) { // handler asked to close the socket if (maybe_flush_ssl(state)) { state->flags |= CONNECTION_STATUS_WRITE_AND_ABORT; break; } abort_connection_on_error(state); break; } } free(buf->base); }

This method is called wheneverlibuvhas more data to give us on the connection. The actual behavior is on ensure_connection_intialized() , where we check a flag on the connection, and if we haven’t done the initialization of the connection, we checkiOpenSSL consider the connection established. If it is established, we validate the connection and then send the OK to start the ball rolling.

You might have noticed a bunch of work with flags CONNECTIO