Writing Java With a Dash of Functional Programming

2018-08-31T00:00:00+00:00
After learning a little bit of Haskell, writing idiomatic Java leaves me wanting a little more from the normal object orientated constructs and traditional imperative programming paradigms. I realized during one of my projects that I could apply some functional concepts to it to make the code more elegant and readable, which I’ll share in this post using a slimmed down example!
Problem</h2>
A typical data migration to replace one data source with another with zero downtime might look like the following:
Flip application reads to be only from the new source, remove the double writes</li> </ol>
We will be looking at step 3 of a data migration, where we modify our application to load records from both data sources to compare for discrepancies during reads, ensuring no data loss was made during this migration.
In this example, we will be building an indirection to load books from two data sources, comparing them, and returning them to our application upstream, all while handling possible exceptions during the load.
The following code snippet specifies a Book</code> object with just one title</code> field. It implements Comparable<Book></code></a> for us to make assertions that two Book</code> instances are the same.
public class</span> Book</span> implements</span> Comparable</span><</span>Book</span>>{</span></span>
    private final String</span> title</span>;</span></span>
</span>
    public</span> Book</span>(</span>final String</span> title</span>) {</span></span>
        this</span>.</span>title</span> =</span> title</span>;</span></span>
    }</span></span>
</span>
    public String</span> getTitle</span>() {</span></span>
        return</span> title</span>;</span></span>
    }</span></span>
</span>
    @Override</span></span>
    public int</span> compareTo</span>(</span>final Book</span> o</span>) {</span></span>
        return</span> title</span>.</span>compareTo</span>(</span>o</span>.</span>getTitle</span>());</span></span>
    }</span></span>
}</span></span></code></pre>
Here, we have the interface for a BookLoader</code>.</p>
public interface</span> BookLoader</span> {</span></span>
    Book</span> load</span>()</span> throws Exception</span>;</span></span>
}</span></span></code></pre>
Note that loading a book can throw an exception, so we will have to handle it appropriately. In the happy path where loads from both tables are successful, we simply want to compare these results. Should there be any discrepancies, or errors with loading the data, we can call incrementCompareFailedMetric()</code> on our Metrics</code> interface.</p>
public interface</span> Metrics</span> {</span></span>
    void</span> incrementCompareFailedMetric</span>();</span></span>
}</span></span></code></pre>Requirements</h3>
Typically, we do not want integration with the new database to add another failure point on our system, so our function should ignore failures from the new database in the unhappy path. The following table specifies the requirements that we will code out:</p>
oldLoader.load()</code></th> newLoader.load()</code></th> Action</th></tr></thead>

Success</td> Success</td> compareBooks(oldBook, newBook)</code>, return oldBook</code></td></tr>
Success</td> Failure</td> metrics.incrementCompareFailedMetric()</code>, return oldBook</code></td></tr>
Failure</td> Success</td> Throw exception from oldLoader.load()</code></td></tr>
Failure</td> Failure</td> Throw exception from oldLoader.load()</code></td></tr>
</tbody></table>
First Iteration</h2>
Now that we have the requirements set, let’s write some code.</p>
We first declare a constructor to inject in our Metrics</code> interface, and one BookLoader</code> interface for the old and new database. Also, we create a simple helper function to compare books and update metrics.</p>
public class</span> ValidatingBookLoader</span> implements</span> BookLoader</span> {</span></span>
    private final Metrics</span> metrics</span>;</span></span>
    private final BookLoader</span> oldLoader</span>;</span></span>
    private final BookLoader</span> newLoader</span>;</span></span>
</span>
    public</span> ImperativeBookLoader</span>(</span>final Metrics</span> metrics</span>,</span> final BookLoader</span> oldLoader</span>,</span> final BookLoader</span> newLoader</span>) {</span></span>
        this</span>.</span>metrics</span> =</span> metrics</span>;</span></span>
        this</span>.</span>oldLoader</span> =</span> oldLoader</span>;</span></span>
        this</span>.</span>new</span>Loader </span>=</span> newLoader</span>;</span></span>
    }</span></span>
</span>
    private void</span> compareBooks</span>(</span>final Book</span> oldBook</span>,</span> final Book</span> newBook</span>) {</span></span>
        if</span> (</span>oldBook</span>.</span>compareTo</span>(</span>newBook</span>)</span> !=</span> 0</span>) {</span></span>
            metrics</span>.</span>incrementCompareFailedMetric</span>();</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>
We create a tryLoadFromLoader</code> helper to return a tuple</a> holding two types, Book</code> and Exception</code>. It’s a little weird because exceptions are typically thrown. Since we have to handle possible exceptions from both loaders, it helps to catch them and pass them as values.</p>
private Tuple</span><</span>Book</span>,</span> Exception</span>></span> tryLoadFromLoader</span>(</span>final BookLoader</span> loader</span>) {</span></span>
    try</span> {</span></span>
        return new Tuple</span><>(</span>loader</span>.</span>load</span>(),</span> null</span>);</span></span>
    }</span> catch</span> (</span>Exception</span> e</span>) {</span></span>
        return new Tuple</span><>(</span>null</span>,</span> e</span>);</span></span>
    }</span></span>
}</span></span></code></pre>
Now, we can implement the load</code> function.</p>
public Book</span> load</span>()</span> throws Exception </span>{</span></span>
    final Tuple</span><</span>Book</span>,</span> Exception</span>></span> oldLoaderResult</span> =</span> tryLoadFromLoader</span>(</span>oldLoader</span>);</span></span>
    final Tuple</span><</span>Book</span>,</span> Exception</span>></span> newLoaderResult</span> =</span> tryLoadFromLoader</span>(</span>newLoader</span>);</span></span>
</span>
    final Exception</span> oldLoadError</span> =</span> oldLoaderResult</span>.</span>right</span>;</span></span>
    final Exception</span> newLoadError</span> =</span> newLoaderResult</span>.</span>right</span>;</span></span>
</span>
    final boolean</span> oldLoaderSuccessful</span> =</span> (</span>oldLoadError </span>==</span> null</span>);</span></span>
    final boolean</span> newLoaderSuccessful</span> =</span> (</span>newLoadError </span>==</span> null</span>);</span></span>
</span>
    if</span> (</span>oldLoaderSuccessful </span>&&</span> newLoaderSuccessful</span>) {</span></span>
        compareBooks</span>(</span>oldLoaderResult</span>.</span>left</span>,</span> newLoaderResult</span>.</span>left</span>);</span></span>
        return</span> oldLoaderResult</span>.</span>left</span>;</span></span>
    }</span></span>
</span>
    // old loader success, new loader fail</span></span>
    if</span> (</span>oldLoaderSuccessful</span>) {</span></span>
        metrics</span>.</span>incrementCompareFailedMetric</span>();</span></span>
        return</span> oldLoaderResult</span>.</span>left</span>;</span></span>
    }</span></span>
</span>
    // old loader fail. new loader fail</span></span>
    throw</span> oldLoadError</span>;</span></span>
}</span></span></code></pre>
It a little more verbose than it needs to be, but the gist of it is that we use the absence of an exception (on the right side of the tuple) to indicate success of the operation. Using conditionals, we simply perform checks and operations based on the specifications in the table above.</p>
Either</h2>
The above implementation of load</code>, is rather imperative. Without the comments, it can add a bit of cognitive load when one tries to understand the implementation. Also, what happens if there was a bug in the implementation of tryLoadFromLoader</code>, where the returned tuple contains both</em> a result and</em> an exception? It sure would be helpful if such mistakes can be caught at compile tome.</p>
Functional programming to the rescue! In languages like Haskell, Either</code> is a kind of “container” holding just one</em> value that can be of two types, on the left</code>, and the right</code>. Either</em> left</em> or right</em>, get it? It can be used to specify and chain results of execution where the successful value is typically on the left, and error value on the right.</p>
Sounds familiar? This sounds like the Tuple<Book, Exception></code> that we have declared earlier, except with additional helper functions that we can use to transition these results along. Also, it is impossible to initialize an Either</code> type with both the left and right values – doing so should cause a compilation error.</p>
Fold</h3>
Java 8 has introduced some functional concepts such as Optionals, lambdas and more, but it still does not give us this Either</code> type as a built-in. We can still do so with the vavr</a> library though!</p>
Of interest to us is the API</a> of Either.fold</code> which I will illustrate with a simple example. fold</code> takes two parameters, a leftMapper</code> function to transform the left</em> value, if</em> the value is a left</em> type, and a rightMapper</code> function to transform the right</em> value, should it be a right</em> type. In the example below, we transform an Either<String, Integer></code> type to another Either<Integer, String></code> by using the fold</code> method and lambda functions as parameters.</p>
Either</span><</span>String</span>,</span> Integer</span>></span> doSomething</span>() {</span></span>
    // dummy method stub</span></span>
}</span></span>
</span>
Either</span><</span>Integer</span>,</span> String</span>></span> foldResult</span> =</span> doSomething</span>().</span>fold</span>((</span>someString</span>)</span> -></span> {</span></span>
    // what to return when the result of doSomething() is a string (on the left)</span></span>
    return</span> Either</span>.</span>left</span>(</span>someString</span>.</span>length</span>());</span></span>
}, (</span>someInteger</span>)</span> -></span> {</span></span>
    // what to return when the result of doSomething() is a integer (on the right)</span></span>
    return</span> Either</span>.</span>right</span>(</span>Integer</span>.</span>toString</span>(</span>someInteger</span>));</span></span>
});</span></span></code></pre>More Declarative Implementation</h2>
Let us now iterate on our earlier solution. First, we rewrite tryLoadFromLoader</code> to return an Either</code> object instead of a tuple.</p>
private Either</span><</span>Book</span>,</span> Exception</span>></span> tryLoadFromLoader</span>(</span>final BookLoader</span> loader</span>) {</span></span>
    try</span> {</span></span>
        return</span> Either</span>.</span>left</span>(</span>loader</span>.</span>load</span>());</span></span>
    }</span> catch</span> (</span>Exception</span> e</span>) {</span></span>
        return</span> Either</span>.</span>right</span>(</span>e</span>);</span></span>
    }</span></span>
}</span></span></code></pre>
Now, we can rewrite our load</code> function by using a nested fold</code> over the results:</p>
public Book</span> load</span>()</span> throws Exception </span>{</span></span>
    final Either</span><</span>Book</span>,</span> Exception</span>></span> oldLoaderResult</span> =</span> tryLoadFromLoader</span>(</span>oldLoader</span>);</span></span>
    final Either</span><</span>Book</span>,</span> Exception</span>></span> newLoaderResult</span> =</span> tryLoadFromLoader</span>(</span>newLoader</span>);</span></span>
</span>
    final Either</span><</span>Book</span>,</span> Exception</span>></span> result</span> =</span> oldLoaderResult</span>.</span>fold</span>(</span></span>
        (</span>oldBook</span>)</span> -></span> newLoaderResult</span>.</span>fold</span>(</span></span>
            (</span>newBook</span>)</span> -></span> {</span></span>
                compareBooks</span>(</span>oldBook</span>,</span> newBook</span>);</span></span>
                return</span> Either</span>.</span>left</span>(</span>oldBook</span>);</span></span>
            },</span></span>
            (</span>newLoaderError</span>)</span> -></span> {</span></span>
                metrics</span>.</span>incrementCompareFailedMetric</span>();</span></span>
                return</span> Either</span>.</span>left</span>(</span>oldBook</span>);</span></span>
            }),</span></span>
            (</span>oldLoaderError</span>)</span> -></span> newLoaderResult</span>.</span>fold</span>(</span></span>
                (</span>newBook</span>)</span> -></span> Either</span>.</span>right</span>(</span>oldLoaderError</span>),</span></span>
                (</span>newLoaderError</span>)</span> -></span> Either</span>.</span>right</span>(</span>oldLoaderError</span>)</span></span>
            )</span></span>
    );</span></span>
    return</span> result</span>.</span>left</span>().</span>getOrElseThrow</span>((</span>exception</span>)</span> -></span> exception</span>);</span></span>
}</span></span></code></pre>
This breakdown explains what actually happens here if you’re confused:</p>
final Either</span><</span>Book</span>,</span> Exception</span>></span> result</span> =</span> oldLoaderResult</span>.</span>fold</span>(</span></span>
    (</span>oldBook</span>)</span> -></span> newLoaderResult</span>.</span>fold</span>(</span></span>
        (</span>newBook</span>)</span> -></span> {</span></span>
            // old load success, new load success</span></span>
            // return something</span></span>
        },</span></span>
        (</span>newLoaderError</span>)</span> -></span> {</span></span>
            // old load success, new load failure</span></span>
            // return something</span></span>
        }),</span></span>
    (</span>oldLoaderError</span>)</span> -></span> newLoaderResult</span>.</span>fold</span>(</span></span>
        (</span>newBook</span>)</span> -></span> {</span></span>
            // old load failure, new load success</span></span>
            // return something</span></span>
        },</span></span>
        (</span>newLoaderError</span>)</span> -></span> {</span></span>
            // old load failure, new load failure</span></span>
            // return something</span></span>
        })</span></span>
    );</span></span></code></pre>
Let me paste the requirements table</a> from above.</p>
oldLoader.load()</code></th> newLoader.load()</code></th> Action</th></tr></thead>

Success</td> Success</td> compareBooks(oldBook, newBook)</code>, return oldBook</code></td></tr>
Success</td> Failure</td> metrics.incrementCompareFailedMetric()</code>, return oldBook</code></td></tr>
Failure</td> Success</td> Throw exception from oldLoader.load()</code></td></tr>
Failure</td> Failure</td> Throw exception from oldLoader.load()</code></td></tr>
</tbody></table>
Doesn’t it look so much easier to read in a table format? Even though we have two layers of nested lambdas, it is still very clear to see how we enumerate and reduce the 2 × 2 possible exceptions and results into what we want. There we have it!</p>
In Haskell?</h2>
Without going into the syntax of Haskell, here’s how the book loading function can be written. Using pattern matching, we can elegantly define the table above with one line for each row:</p>
-- compareBooks :: Book -> Book -> Bool</span></span>
</span>
handleBookLoad</span> ::</span> Either Book BookLoadError</span> -></span> Either Book BookLoadError</span> -></span> (</span>Either Book BookLoadError</span>,</span> Bool</span>)</span></span>
handleBookLoad </span>(</span>Left oldBook</span>) (</span>Left newBook</span>)</span> =</span> (</span>Left oldBook</span>,</span> compareBooks oldBook newBook</span>)</span></span>
handleBookLoad </span>(</span>Left oldBook</span>) (</span>Right _</span>)</span> =</span> (</span>Left oldBook</span>,</span> False</span>)</span></span>
handleBookLoad </span>(</span>Right oldLoaderError</span>) (</span>Left _</span>)</span> =</span> (</span>Right oldLoaderError</span>,</span> True</span>)</span></span>
handleBookLoad </span>(</span>Right oldLoaderError</span>) (</span>Right _</span>)</span> =</span> (</span>Right oldLoaderError</span>,</span> True</span>)</span></span></code></pre>
Neat eh?</p>
Summing Up</h2>
Using functional programming doesn’t mean that we have to restructure our entire application to use outlandish and academic concepts. As seen in the examples, it’s possible to simply rewrite a method or two in a small surface area, giving it a functional touch for more declarative code.</p>
These concepts are not strictly tied to functional programming languages – I’d strongly recommend picking up one of these languages to broaden your paradigm of programming, and it’ll be really fun too. I’m still pretty much a beginner in Haskell, so feel free to share any tips on how to improve this!</p>
Appendix</h2>
Full Source Code</a></p>
Tuple</h3>
Simple tuple implementation:</p>
public class</span> Tuple</span><</span>L</span>,</span> R</span>> {</span></span>
    public final L</span> left</span>;</span></span>
    public final R</span> right</span>;</span></span>
</span>
    public</span> Tuple</span>(</span>L</span> l</span>,</span> R</span> r</span>) {</span></span>
        this</span>.</span>left</span> =</span> l</span>;</span></span>
        this</span>.</span>right</span> =</span> r</span>;</span></span>
    }</span></span>
}</span></span></code></pre>BookLoader in Full</h3>
import</span> io</span>.</span>vavr</span>.</span>control</span>.</span>Either</span>;</span></span>
</span>
public class</span> DeclarativeBookLoader</span> implements</span> BookLoader</span> {</span></span>
    private final Metrics</span> metrics</span>;</span></span>
    private final BookLoader</span> oldLoader</span>;</span></span>
    private final BookLoader</span> newLoader</span>;</span></span>
</span>
    public</span> DeclarativeBookLoader</span>(</span>final Metrics</span> metrics</span>,</span> final BookLoader</span> oldLoader</span>,</span> final BookLoader</span> newLoader</span>) {</span></span>
        this</span>.</span>metrics</span> =</span> metrics</span>;</span></span>
        this</span>.</span>oldLoader</span> =</span> oldLoader</span>;</span></span>
        this</span>.</span>new</span>Loader </span>=</span> newLoader</span>;</span></span>
    }</span></span>
</span>
    public Book</span> load</span>()</span> throws Exception</span> {</span></span>
        final Either</span><</span>Book</span>,</span> Exception</span>></span> oldLoaderResult</span> =</span> tryLoadFromLoader</span>(</span>oldLoader</span>);</span></span>
        final Either</span><</span>Book</span>,</span> Exception</span>></span> newLoaderResult</span> =</span> tryLoadFromLoader</span>(</span>newLoader</span>);</span></span>
</span>
        final Either</span><</span>Book</span>,</span> Exception</span>></span> result</span> =</span> oldLoaderResult</span>.</span>fold</span>(</span></span>
                (</span>oldBook</span>)</span> -></span> newLoaderResult</span>.</span>fold</span>(</span></span>
                        (</span>newBook</span>)</span> -></span> {</span></span>
                            compareBooks</span>(</span>oldBook</span>,</span> newBook</span>);</span></span>
                            return</span> Either</span>.</span>left</span>(</span>oldBook</span>);</span></span>
                        },</span></span>
                        (</span>newLoaderError</span>)</span> -></span> {</span></span>
                            metrics</span>.</span>incrementCompareFailedMetric</span>();</span></span>
                            return</span> Either</span>.</span>left</span>(</span>oldBook</span>);</span></span>
                        }),</span></span>
                (</span>oldLoaderError</span>)</span> -></span> newLoaderResult</span>.</span>fold</span>(</span></span>
                        (</span>newBook</span>)</span> -></span> Either</span>.</span>right</span>(</span>oldLoaderError</span>),</span></span>
                        (</span>newLoaderError</span>)</span> -></span> Either</span>.</span>right</span>(</span>oldLoaderError</span>)</span></span>
                )</span></span>
        );</span></span>
</span>
        return</span> result</span>.</span>left</span>().</span>getOrElseThrow</span>((</span>exception</span>)</span> -></span> exception</span>);</span></span>
    }</span></span>
</span>
    private Either</span><</span>Book</span>,</span> Exception</span>></span> tryLoadFromLoader</span>(</span>final BookLoader</span> loader</span>) {</span></span>
        try</span> {</span></span>
            return</span> Either</span>.</span>left</span>(</span>loader</span>.</span>load</span>());</span></span>
        }</span> catch</span> (</span>Exception</span> e</span>) {</span></span>
            return</span> Either</span>.</span>right</span>(</span>e</span>);</span></span>
        }</span></span>
    }</span></span>
</span>
    private void</span> compareBooks</span>(</span>final Book</span> oldBook</span>,</span> final Book</span> newBook</span>) {</span></span>
        if</span> (</span>oldBook</span>.</span>compareTo</span>(</span>newBook</span>)</span> !=</span> 0</span>) {</span></span>
            metrics</span>.</span>incrementCompareFailedMetric</span>();</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>Haskell Parallels</h3>
Without using pattern matching, we can also use either</code></a> to handle the loader results, synonymous with how we used fold</code> earlier in Java.  Note that this is a function, and is different from Either</code> as a type constructor.</p>
either</span> ::</span> (</span>a</span> -></span> c</span>)</span> -></span> (</span>b</span> -></span> c</span>)</span> -></span> Either</span> a b</span> -></span> c</span></span></code></pre>handleBookLoad</span> ::</span> Either Book BookLoadError</span> -></span> Either Book BookLoadError</span> -></span> (</span>Either Book BookLoadError</span>,</span> Bool</span>)</span></span>
handleBookLoad oldLoadResult newLoadResult </span>=</span> either</span></span>
    (</span>\</span>oldBook </span>-></span></span>
        either</span></span>
            (</span>\</span>newBook </span>-></span> (</span>Left oldBook</span>,</span> compareBooks oldBook newBook</span>))</span></span>
            (</span>\</span>_ </span>-></span> (</span>Left oldBook</span>,</span> False</span>))</span></span>
            newLoadResult</span></span>
    )</span></span>
    (</span>\</span>oldLoaderError </span>-></span></span>
        either</span></span>
            (</span>\</span>_ </span>-></span> (</span>Right oldLoaderError</span>,</span> True</span>))</span></span>
            (</span>\</span>_ </span>-></span> (</span>Right oldLoaderError</span>,</span> True</span>))</span></span>
            newLoadResult</span></span>
    )</span></span>
    oldLoadResult</span></span></code></pre>


Optimistic Locking with the DynamoDB Mapper
2018-08-04T00:00:00+00:00
Amazon DynamoDB</a> is a non-relational managed database on Amazon Web Services that provides the DynamoDB Mapper</a> Java library, to map Plain Old Java Objects (POJOs)</a> to the non-relational representation of the document data in storage. With this object-orientated interface, it is easy to ensure that records are updated correctly on DynamoDB, through optimistic locking</em></a>. This is a strategy to ensure that the value committed to DynamoDB during an update request is the intended</em> update desired by the client, even when this request can be nondeterministically interweaved with updates by other clients on the same record.</p>
In this post, we will first go over a short summary on simple usage of this library, and subsequently explore how such locks are implemented using DynamoDB primitives.</p>
Using the Mapper</h2>
The following class describes a Book</code> with two fields used by the application, ISBN</code> and title</code>, with an additional version</code> field that we will use for locks..</p>
@DynamoDBTable</span>(</span>tableName</span>=</span>"Books"</span>)</span></span>
public class</span> Book</span> {</span></span>
    // Primary key</span></span>
    private String</span> ISBN</span>;</span></span>
</span>
    private String</span> title</span>;</span></span>
    private Long</span> version</span>;</span></span>
</span>
    @DynamoDBAttribute</span>)</span></span>
    public String</span> getTitle</span>() {</span> return</span> title</span>; }</span></span>
    public void</span> setTitle</span>(</span>String</span> title</span>) {</span> this</span>.</span>title</span> =</span> title</span>; }</span></span>
</span>
    @DynamoDBAttribute</span></span>
    public String</span> getISBN</span>() {</span> return</span> ISBN</span>; }</span></span>
    public void</span> setISBN</span>(</span>String</span> ISBN</span>) {</span> this</span>.</span>ISBN</span> =</span> ISBN</span>;}</span></span>
</span>
    @DynamoDBVersionAttribute</span></span>
    public Long</span> getVersion</span>() {</span> return</span> version</span>; }</span></span>
    public void</span> setVersion</span>(</span>Long</span> version</span>) {</span> this</span>.</span>version</span> =</span> version</span>;}</span></span>
}</span></span></code></pre>
Using the @DynamoDBVersionAttribute</code> annotation has the following impact on the operations (taken verbatim from the docs</a>):</p>

save</strong> — For a new item, the DynamoDBMapper assigns an initial version number 1. If you retrieve an item, update one or more of its properties and attempt to save the changes, the save operation succeeds only if the version number on the client-side and the server-side match. The DynamoDBMapper increments the version number automatically.</p>
</blockquote>

delete</strong> — The delete method takes an object as parameter and the DynamoDBMapper performs a version check before deleting the item. The version check can be disabled if DynamoDBMapperConfig.SaveBehavior.CLOBBER is specified in the request.</p>
</blockquote>

The internal implementation of optimistic locking within DynamoDBMapper uses conditional update and conditional delete support provided by DynamoDB.</p>
</blockquote>
The next code snippet shows how we can load a record from DynamoDB, change the title, and save it again.</p>
DynamoDBMapper</span> mapper</span> =</span> new</span> DynamoDBMapper</span>(</span>client</span>);</span></span>
</span>
// Load a book from DynamoDB by ISBN</span></span>
Book</span> item</span> =</span> mapper</span>.</span>load</span>(</span>Book</span>.</span>class</span>,</span> "978-3-16-148410-0"</span>);</span></span>
item</span>.</span>setTitle</span>(</span>"New Title"</span>);</span></span>
</span>
// Save the item.</span></span>
mapper</span>.</span>save</span>(</span>item</span>);</span></span></code></pre>
What happens when someone updates the record after we load it into memory, but before the save is committed into DynamoDB?</p>

Record loaded in our application as the following (JSON format for simplicity)</li>
</ol>
{</span></span>
  "</span>ISBN</span>":</span> "978-3-16-148410-0"</span>,</span></span>
  "</span>title</span>":</span> "Old Title"</span>,</span></span>
  "</span>version</span>":</span> 2</span></span>
}</span></span></code></pre>

Another client performs and commits a update to change the title of the record. If the DynamoDB mapper is used by this client as well, the version</code> field will be incremented to 3</code>.</li>
</ol>
# New value in DynamoDB</span></span>
{</span></span>
  "ISBN"</span>:</span> "978-3-16-148410-0"</span>,</span></span>
  "title"</span>:</span> "Changed By Someone Else"</span>,</span></span>
  "version"</span>:</span> 3</span></span>
}</span></span></code></pre>

Now, our application tries to perform the a save</code> of the following record:</li>
</ol>
{</span></span>
  "</span>ISBN</span>":</span> "978-3-16-148410-0"</span>,</span></span>
  "</span>title</span>":</span> "New Title"</span>,</span></span>
  "</span>version</span>":</span> 2</span></span>
}</span></span></code></pre>
Conditional expressions</a> are special conditions that can be specified and need to be fulfilled before data is committed to the DynamoDB store to prevent data races between clients.</p>
When the save</code> method is executed, it automatically creates a conditional expression to ensure that the record is persisted only if the field annotated with @DynamoDBVersionAttribute</code> (version</code> in our example) in the request matches the same field in storage. Since the version has been changed earlier to 3</code> by another client, DynamoDB will raise a conditional check failed exception, preventing the update.</p>
How It works</h2>
From the Java docs</a> , the @DynamoDBVersionAttribute</code> annotation does the following:</p>

On a save()</code> operation, the DynamoDBMapper will attempt to increment the version property and assert that the service’s value matches the client’s. New objects will be assigned a version of 1 when saved.</p>
</blockquote>
Let’s try to better understand this on a lower level by looking at the internals of the library. The SaveObjectHandler</code></a> is responsible for carrying out the save operation of an annotated POJO with the following steps:</p>

Construct a request from the primary keys and other fields</li>
Execute the UpdateItem</code></a> low-level request</a></li>
If the request is successful, apply any auto-generated changes to the initial POJO parameter</li>
</ol>
Handling of Version Fields</h3>
The UpdateItem request is constructed by iterating through all the annotated fields of the model object. A special handler is invoked when a field annotated with @DynamoDBVersionAttribute</code> is encountered:</p>
// https://github.com/aws/aws-sdk-java/blob/9d9bc2fb303181b60f404af9a84b2300e8c20550/aws-java-sdk-dynamodb/src/main/java/com/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapper.java#L916-L931</span></span>
</span>
/**</span></span>
 * Auto-generates the version.</span></span>
 * </span>@param</span> mapping</span> The mapping details.</span></span>
 */</span></span>
private void</span> onVersionAttribute</span>(</span>DynamoDBMapperFieldModel</span><</span>Object</span>,</span>Object</span>></span> field</span>) {</span></span>
    if</span> (</span> getLocalSaveBehavior</span>()</span> !=</span> SaveBehavior</span>.</span>CLOBBER</span></span>
            && !</span>internalExpectedValueAssertions</span>.</span>containsKey</span>(</span>field</span>.</span>name</span>())) {</span></span>
        // First establish the expected (current) value for the</span></span>
        // update call</span></span>
        // For new objects, insist that the value doesn't exist.</span></span>
        // For existing ones, insist it has the old value.</span></span>
        final Object</span> current</span> =</span> field</span>.</span>get</span>(</span>object</span>);</span></span>
        if</span> (</span>current </span>==</span> null</span>) {</span></span>
            internalExpectedValueAssertions</span>.</span>put</span>(</span>field</span>.</span>name</span>(),</span></span>
                new</span> ExpectedAttributeValue</span>().</span>withExists</span>(</span>false</span>));</span></span>
        }</span> else</span> {</span></span>
            internalExpectedValueAssertions</span>.</span>put</span>(</span>field</span>.</span>name</span>(),</span></span>
                new</span> ExpectedAttributeValue</span>().</span>withExists</span>(</span>true</span>).</span>withValue</span>(</span>field</span>.</span>convert</span>(</span>current</span>)));</span></span>
        }</span></span>
    }</span></span>
</span>
    // Generate the new version value</span></span>
    onAutoGenerate</span>(</span>field</span>);</span></span>
}</span></span></code></pre>
Essentially, it performs a check to determine if the POJO models a fresh new record, or one that already exists (previously loaded from mapper.load</code>) based on the nullability of the version field, and creates a ExpectedAttributeValue</a> object. This is a high level object-orientated API for describing conditional expressions. In addition, note the different behavior if the mapper is configured with the CLOBBER</code></a> save behavior – no conditional expressions will be used and the value in DynamoDB will simply be replaced.</p>
Subsequently, the onAutoGenerate</code> function is called:</p>
// https://github.com/aws/aws-sdk-java/blob/9d9bc2fb303181b60f404af9a84b2300e8c20550/aws-java-sdk-dynamodb/src/main/java/com/amazonaws/services/dynamodbv2/datamodeling/DynamoDBMapper.java#L889-L893</span></span>
</span>
/**</span></span>
* Auto-generates the attribute value.</span></span>
* </span>@param</span> mapping</span> The mapping details.</span></span>
*/</span></span>
private void</span> onAutoGenerate</span>(</span>DynamoDBMapperFieldModel</span><</span>Object</span>,</span>Object</span>></span> field</span>) {</span></span>
    AttributeValue</span> value</span> =</span> field</span>.</span>convert</span>(</span>field</span>.</span>generate</span>(</span>field</span>.</span>get</span>(</span>object</span>)));</span></span>
    updateValues</span>.</span>put</span>(</span>field</span>.</span>name</span>(),</span>  new</span> AttributeValueUpdate</span>().</span>withAction</span>(</span>"PUT"</span>).</span>withValue</span>(</span>value</span>));</span></span>
    inMemoryUpdates</span>.</span>add</span>(</span>new</span> ValueUpdate</span>(</span>field</span>,</span> value</span>,</span> object</span>));</span></span>
}</span></span></code></pre>
This function calls a generator for the field type. Typically, if the value is null</code>, it will be initialized with 1</code>, or it will be incremented. An example for the Long</code> type is below.</p>
// https://github.com/bbsage/aws/blob/9243e0716bc2a649085480d3a4fb7606b6541da3/aws-java-sdk-dynamodb/src/main/java/com/amazonaws/services/dynamodbv2/datamodeling/DynamoDBAutoGeneratorRegistry.java#L333-L345</span></span>
</span>
/**</span></span>
* Version generator for {@code Long} types.</span></span>
*/</span></span>
static final class</span> LongVersionGenerator</span> extends</span> NeverVersionGenerator</span><</span>Long</span>> {</span></span>
    private</span> LongVersionGenerator</span>() {</span></span>
        super</span>(</span>Long</span>.</span>class</span>);</span></span>
    }</span></span>
</span>
    @Override</span></span>
    public final Long</span> generate</span>(</span>final Long</span> currentValue</span>) {</span></span>
        if</span> (</span>currentValue </span>==</span> null</span>) {</span></span>
            return</span> Long</span>.</span>valueOf</span>(</span>1L</span>);</span></span>
        }</span></span>
        return</span> (</span>long</span>)(</span>currentValue </span>+</span> 1L</span>);</span></span>
    }</span></span>
}</span></span></code></pre>
To better distill how the version fields is handled, we can transform it into pseudocode describing how parameters are added to the UpdateItem</code> request for @DynamoDBVersionAttribute</code> fields.</p>
# `$version_key` refers to the field name of the DynamoDBVersionAttribute</span></span>
# `$version_value` refers to the client side value of the DynamoDBVersionAttribute</span></span>
</span>
if</span> the mapper save behavior</span> is not</span> `CLOBBER`</span>:</span></span>
    if</span> `$version_value`</span> in</span> the provided</span> object</span> is</span> null</span>:</span></span>
        # new object is being created here</span></span>
        add to updateItem request</span>: {</span></span>
            "ConditionExpression"</span>:</span> "attribute_not_exists($version_key)"</span>,</span></span>
        }</span></span>
    else</span>:</span></span>
        # the object already exists in DynamoDB</span></span>
        add to updateItem request</span>: {</span></span>
           "ConditionExpression"</span>:</span> "$version_key = :version"</span>,</span></span>
           "ExpressionAttributeValues"</span>: {</span></span>
               ":version"</span>:</span> "$version_value"</span></span>
        }</span></span>
</span>
if</span> the `$version_value`</span> is</span> null</span>:</span></span>
    add update parameter to request</span>:</span> set</span> it to identity value</span></span>
else</span>:</span></span>
    add update parameter to request</span>:</span> increment the value by</span> 1</span></span></code></pre>
This results in the following edge cases that we have to be aware of:</p>

Saving POJOs to DynamoDB with a non-null version</em> when it does not exist will result in a conditional check failed exception</li>
Irregardless of the save behavior, the version attribute will be initialized / incremented when it is saved.</li>
</ul>
As seen in the pseudocode, the CLOBBER</code></a> save behavior can be used to configure the mapper so that DynamoDB skips the conditional checks. This can be useful when you are writing to multiple tables at once, say during a data migration.</p>
Updating the Client-side POJO</h3>
After the update is completed successfully, the DynamoDBMapper updates the parameter object</a> with any fields that have autogenerated changes, such as incremented / initialized version attributes. This updates the state of the local POJO model instance in an attempt to keep it in sync with the latest change made from this client.</p>
Book</span> item</span> =</span> mapper</span>.</span>load</span>(</span>Book</span>.</span>class</span>,</span> "978-3-16-148410-0"</span>);</span></span>
item</span>.</span>setTitle</span>(</span>"New Title"</span>);</span></span>
</span>
item</span>.</span>getVersion</span>()</span> // 1</span></span>
</span>
// Save increments the version of `item` in DynamoDB</span></span>
mapper</span>.</span>save</span>(</span>item</span>);</span></span>
</span>
item</span>.</span>getVersion</span>()</span> // 2 - Value is updated to reflect new value in DynamoDB</span></span></code></pre>
It can be useful to have the updated state of the object available for the application locally for further processing, but note the mutability pitfall here. Calling mapper.save(book)</code> multiple times with the same parameter is not idempotent, as the @DynamoDBVersionAttribute</code> on the input parameter will be mutated on each call to the mapper. In my case, I was making parallel calls across two DynamoDB tables, and this unexpected mutability of the input object interfered with upstream retry elements of my application, causing data inconsistencies.</p>
Wrapping Up</h2>
In this post, we have investigated the source code of the DynamoDB Mapper to shed some light on how locks on objects are implemented by the library, and highlighting some pitfalls at the same time.</p>
It’s important to understand your abstractions when picking a tool to solve a problem; when you need more clarity from documentation, don’t be afraid to dive into the source (code) of truth! Online tools like Sourcegraph</a> make it much easier to search and traverse source code on GitHub without having to setup an IDE and downloading the code. Here’s a link</a> to the corresponding page for the DynamoDB Mapper on Sourcegraph if you’re interested to explore more!</p>


Deep Dive into Database Timeouts in Rails
2018-01-29T00:00:00+00:00
This article was originally published on Grab’s Engineering Blog</a></em></p>
A couple of weeks ago, we had a production outage for one of our internal Ruby on Rails application servers. One of the databases that the application connects to had a failover event. It was expected that the server should continue functioning for endpoints which do not depend on this database, but it was observed that our server slowed down to a crawl, and was unable to function properly even after the failover completed, until we manually restarted the servers.</p>
Background</h2>
ActiveRecord</a> is the canonical ORM for Rails to access a database. Different requests are handled on different threads, so a connection pool is necessary to maintain a limited set of connections to the database and also to skip the additional latency of establishing a TCP connection.</p>

A connection pool synchronizes thread access to a limited number of database connections. The basic idea is that each thread checks out a database connection from the pool, uses that connection, and checks the connection back in.</p>
</blockquote>

It will also handle cases in which there are more threads than connections: if all connections have been checked out, and a thread tries to checkout a connection anyway, then ConnectionPool will wait until some other thread has checked in a connection.</p>
</blockquote>
Source: The ActiveRecord::Connection Pool</code></a> .</p>
Options for the Connection Pool</h3>
In Rails, database configurations are set in the config/database.yml</code> file. These options are either native to the ActiveRecord::ConnectionPool</code> module, or passed to the underlying adapter, depending on whether MySQL or PostgreSQL is used.</p>
ActiveRecord uses connection adapters to make database calls. For MySQL, it uses the mysql2</a> library, which depends on the libmysqlclient</code></a> C library. The following options affect the behaviour of the library:</p>
pool</code></h4>

Native to the ActiveRecord ConnectionPool</a></li>
Default 5 - Source</a></li>
</ul>
This specifies the maximum number of connections to the database that ActiveRecord will maintain per server.</p>
checkout_timeout</code></h4>

Native to the ActiveRecord ConnectionPool</a></li>
Default 5 seconds - Source</a></li>
</ul>
When making a ActiveRecord call, ActiveRecord tries to checkout a database connection from the pool. If the pool is at maximum capacity, ActiveRecord will wait for this timeout to elapse before raising an ActiveRecord::ConnectionTimeoutError</code> exception.</p>
connect_timeout</code></h4>

Native to the mysql2</a> library, passed to libmysqlclient</code> as MYSQL_OPT_CONNECT_TIMEOUT</code></a></li>
Default 120 seconds - Source</a></li>
</ul>
If there are no available connections to the database in the connection pool, a new connection will have to be established. connect_timeout</code> , specifies the timeout to establish a new connection to the database before failing.</p>
read_timeout</code></h4>

Native to the mysql2</a> library, passed to libmysqlclient</code> as MYSQL_OPT_READ_TIMEOUT</code></a></li>
Default 3 × 10 minutes - Source</a></li>
</ul>
Read timeout is used by the libmysqlclient</code> library to identify whether the MySQL client is still alive and sending data. As we know that TCP sends data in chunks, the client waits for this timeout when reading from the socket, before deeming that there is an error and closing the connection.</p>
Connection Pooling Algorithm</h3>
The following pseudocode is the algorithm for how ActiveRecord retrieves connections from the pool to perform database queries.</p>
if</span> there are existing connections to the database available</span>:</span></span>
    return</span> one of the existing connections</span></span>
</span>
if</span> the pool</span> is</span> at capacity</span>:</span></span>
    wait on the queue</span>,</span> raise</span> exception</span> if</span> `checkout_timeout` has elapsed</span></span>
    return</span> one of the now available connections</span></span>
</span>
# pool is not at capacity</span></span>
try</span> to create a new connection</span>,</span> raise</span> exception</span> if</span> `connect_timeout` has elapsed</span></span>
</span>
# connection to database established</span></span>
return</span> new connection</span></span></code></pre>
This is loosely translated from the source code</a>.</p>
Replicating and Debugging</h2>
Let’s try to replicate the problem in a small Rails application. We will create a new Rails application, connect it to a database, run it in a Docker container and finally run some experiments to replicate the problem. In production, we use Puma</a> to run our Rails server and connect to a few MySQL databases managed by Amazon Relational Database Service (RDS)</a>, so we will try to follow that on our local setup.</p>
Step 1: Create a new Rails Application</h3>
First, we will scaffold a fresh Rails application and connect it to two databases that we will call as db_main</code> and db_other</code>:</p>
# the flags removes unwanted boilerplate code</span></span>
rails</span> new rails-mysql-timeouts --database=mysql --api -M -C -S -J -T</span></span></code></pre>
For simplicity, we will set the thread_count</code> of our Puma server to 2</code>, in config/puma.rb</code>.</p>
threads_count</span> =</span> 2</span></span></code></pre>
Using rails generate scaffold</code>, we set up a Driver</code> model to talk to our main database, and a Passenger</code> model to talk to another database we want to test the failure on. This can be done by adding the following line to our Passengers</code> model.</p>
class</span> Passenger</span> <</span> ApplicationRecord</span></span>
  # connect to #{Rails.env}_other database specified in the database.yml</span></span>
  establish_connection </span>"</span>#{</span>Rails</span>.</span>env</span>}</span>_other"</span>.</span>to_sym</span></span>
end</span></span></code></pre>
We now have the following HTTP routes:</p>
# connects to db_main</span></span>
GET /drivers/1</span></span>
</span>
# connects to db_other</span></span>
GET /passengers/1</span></span></code></pre>
Now we will run our Rails server with the following environment variables</p>
export</span> RAILS_ENV</span>=</span>production</span></span>
export</span> RAILS_LOG_TO_STDOUT</span>=</span>1</span></span>
</span>
rails</span> server</span></span></code></pre>
By using a docker container to run the Rails application, we can isolate the process namespace and focus directly on our application. We run ps</code> and observe the two threads we have configured puma — puma 001</code> and puma 002</code>.</p>
$</span> ps -T -e</span></span>
  PID</span>  SPID TTY          TIME CMD</span></span>
    1</span>     1</span> ?        00:00:00 sleep</span></span>
   30</span>    30</span> pts/1    00:00:00 bash</span></span>
   63</span>    63</span> pts/0    00:00:00 bash</span></span>
   97</span>    97</span> pts/1    00:00:03 ruby</span></span>
   97</span>    99</span> pts/1    00:00:00 ruby-timer-thr</span></span>
   97</span>   105</span> pts/1    00:00:00 tmp_restart.rb</span>*</span></span>
   97</span>   106</span> pts/1    00:00:00 puma</span> 001</span></span>
   97</span>   107</span> pts/1    00:00:00 puma</span> 002</span></span>
   97</span>   108</span> pts/1    00:00:00 reactor.rb:152</span></span>
   97</span>   109</span> pts/1    00:00:00 thread_pool.rb</span>*</span></span>
   97</span>   110</span> pts/1    00:00:00 thread_pool.rb</span>*</span></span>
   97</span>   111</span> pts/1    00:00:00 server.rb:327</span></span>
  112</span>   112</span> pts/0    00:00:00 ps</span></span></code></pre>
Note that PID 1 is sleep</code> because in docker-compose.yml</code></a>, we specified that the container should start with cmd: sleep infinity</code> so that we can attach to the running container at any time, not unlike a ssh</code> to a machine.</p>
Step 2: Verify Our Application</h3>
We make the following requests to ensure that our server is working correctly:</p>
$</span> curl localhost:3000/drivers</span></span>
[</span>{</span>"id"</span>:1,</span>"name"</span>:</span>"test driver"</span>,</span>"created_at"</span>:</span>"2017-11-05T11:59:15.000Z"</span>,</span>"updated_at"</span>:</span>"2017-11-05T11:59:15.000Z"</span>}</span>]</span></span>
</span>
$</span> curl localhost:3000/passengers</span></span>
[</span>{</span>"id"</span>:1,</span>"name"</span>:</span>"test"</span>,</span>"created_at"</span>:</span>"2017-01-01T00:00:00.000Z"</span>,</span>"updated_at"</span>:</span>"2017-01-07T00:00:00.000Z"</span>}</span>]</span></span></code></pre>
Great! We are now able to see the records generated in the database by the above curl requests.</p>
The entire source code for this application can be found here</a>.</p>
Step 3: Simulating the Production Issue</h3>
We will now try to simulate the production issue by using a proxy to monitor all our TCP connections from our Rails application to our database. Finally, we will run some experiments by sending requests that hit the backend database and analyse the behaviour of both connect_timeout</code> and read_timeout</code> settings.</p>
First, we use Toxiproxy</a> as a transport layer proxy to db_other</code> which allows us to manipulate the pipe between the client and the upstream database. The following command stops all data from getting the proxy, and closes the connection after timeout.</p>
toxiproxy-cli</span> toxic add db_other_proxy --toxicName timeout --type timeout --attribute=timeout=100000</span></span></code></pre>
Now we test if things are still working for endpoints that access the unaffected database.</p>
$</span> curl localhost:3000/drivers</span></span>
[</span>{</span>"id"</span>:1,</span>"name"</span>:</span>"test driver"</span>,</span>"created_at"</span>:</span>"2017-11-05T11:59:15.000Z"</span>,</span>"updated_at"</span>:</span>"2017-11-05T11:59:15.000Z"</span>}</span>]</span></span></code></pre>
This is expected, as the db_main</code> is still running. Let’s trigger a request to db_other</code>.</p>
$</span> curl localhost:3000/passengers</span></span></code></pre>
We notice that the command does not exit and our terminal blocks while waiting for the command to terminate.</p>
Let’s trigger another call to db_main</code>.</p>
$</span> curl localhost:3000/drivers</span></span>
[</span>{</span>"id"</span>:1,</span>"name"</span>:</span>"test driver"</span>,</span>"created_at"</span>:</span>"2017-11-05T11:59:15.000Z"</span>,</span>"updated_at"</span>:</span>"2017-11-05T11:59:15.000Z"</span>}</span>]</span></span></code></pre>
Seems like it still works! Now let’s make another request to the db_other</code> to lock up the two threads our server is configured to use.</p>
$</span> curl localhost:3000/passengers</span></span></code></pre>
And make another request to db_main</code>.</p>
$</span> curl localhost:3000/drivers</span></span></code></pre>
Notice that the call to /drivers</code> is stuck and does not complete now. Because we have set the thread count to 2</code>, and have two /passengers</code> requests in flight, both threads are stuck waiting for the database and we do not have any more threads available to handle the new request, hence the stalled /drivers</code> request.</p>
This is exactly what happened during our production outage, except on a much larger scale.</p>
Experiments</h3>
Let’s perform some experiments to better understand how connect_timeout</code> and read_timeout</code> work. We will set the timeouts to the following:</p>
+ connect_timeout</span>:</span> 10</span></span>
+ read_timeout</span>:</span> 5</span></span></code></pre>
In the following section we will perform two experiments.</p>
Experiment 1: Application has no Existing Connections before Database Failure</h4>

Stop data transmission to db_other</code></li>
Start Rails</li>
GET /passengers</code></li>
</ol>
We first block data to db_other</code> , so that on the first ActiveRecord call to retrieve some data from the database, there are no available connections in the connection pool and it needs to establish a fresh connection to the database when it receives the first GET /passengers</code> request.</p>
Experiment 2: Application has Existing Connections before Database Failure</h4>

Start Rails</li>
GET /passengers</code></li>
Stop data transmission to db_other</code></li>
GET /passengers</code></li>
</ol>
We’ve started Rails and make a call to GET /passengers</code>. A connection to the database is established to retrieve the data, and checked back into the pool as an available connection after the request.</p>
Now, when the proxy stops sending data to db_other</code>, ActiveRecord does not know that the database is unavailable and believes that the previously checked in connection is available for use with the second GET /passengers</code>.</p>
We can use the ss</code></a> command to observe the TCP connections. When Rails has just been started, there are no existing TCP connections .</p>
# shows TCP connections with the PID</span></span>
$</span> ss -tnp</span></span></code></pre>
After a GET /passengers</code> completes, a TCP connection can be seen in the ESTAB</code> state.</p>
$</span> ss -tnp</span></span>
State</span>  Recv-Q  Send-Q  Local Address:Port  Peer Address:Port</span></span>
ESTAB</span>  0       0</span>       172.18.0.4:54304    172.18.0.3:3306   users:</span>((</span>"ruby",pid</span>=11683,fd=13</span>))</span></span></code></pre>
Now we stop the database, and make another call to GET /passengers</code>. We run ss</code> when the request is in flight, and observe another TCP connection for the request to the port Rails listens on, port 3000</code>.</p>
$</span> ss -tnp</span></span>
State</span>  Recv-Q  Send-Q  Local Address:Port  Peer Address:Port</span></span>
ESTAB</span>  0       0</span>       172.18.0.4:54304    172.18.0.3:3306   users:</span>((</span>"ruby",pid</span>=11683,fd=13</span>))</span></span>
ESTAB</span>  0       0</span>       172.18.0.4:3000     172.18.0.1:60878  users:</span>((</span>"ruby",pid</span>=11683,fd=12</span>))</span></span></code></pre>
After read_timeout</code> has elapsed, we see that a new connection is established to the database, and the first one has transitioned to a FIN-WAIT</code> state. This new TCP connection is in the ESTAB</code> state (line 3), because we have only stopped the database on the application layer, but the sockets to the container still accept the TCP handshake on the transport layer.</p>
$</span> ss -tnp</span></span>
State</span>       Recv-Q  Send-Q  Local Address:Port  Peer Address:Port</span></span>
FIN-WAIT-2</span>  0       0</span>       172.18.0.4:54304    172.18.0.3:3306</span></span>
ESTAB</span>       0       0</span>       172.18.0.4:54308    172.18.0.3:3306   users:</span>((</span>"ruby",pid</span>=11683,fd=13</span>))</span></span>
ESTAB</span>       0       0</span>       172.18.0.4:3000     172.18.0.1:60878  users:</span>((</span>"ruby",pid</span>=11683,fd=12</span>))</span></span></code></pre>
After connect_timeout</code> has elapsed, the request terminates with a 500 error, and we observe that all the connections are in the FIN-WAIT</code> state.</p>
$</span> ss -tnp</span></span>
State</span>       Recv-Q  Send-Q  Local Address:Port  Peer Address:Port</span></span>
FIN-WAIT-2</span>  0       0</span>       172.18.0.4:54310    172.18.0.3:3306</span></span>
FIN-WAIT-2</span>  0       0</span>       172.18.0.4:54304    172.18.0.3:3306</span></span>
FIN-WAIT-2</span>  0       0</span>       172.18.0.4:54308    172.18.0.3:3306</span></span></code></pre>
The experimental data can be found below.</p>
Findings</h4>
It’s worth noting that when setting connect_timeout</code> and read_timeout</code> in the database.yml</code>, there is a difference between empty values and the case where the key is missing entirely in the file. If the values are empty, scenario 1 will fail to terminate after 5 minutes, but if the keys are absent, scenario 1 will fail after 120 seconds, which is the default for connect_timeout</code>.</p>
Experiment 1 Findings</h5>
The request waits for connect_timeout</code> to connect to the database, where the default value (when not specified) is indeed 120 seconds.</p>
As expected, connecting to the database with no existing connections is independent of the read_timeout</code>.</p>
Experiment 2 Findings</h5>
The request waits for read_timeout</code> + connect_timeout</code> before failing. This is because the connection pool waits for read_timeout</code> on the existing connection before terminating it, and then waits for connect_timeout</code> as it tries to establish a new connection to db_other</code>.</p>
Analysis</h2>
With these findings, we can try to understand how the lack of these timeouts affected our Rails server in production during and after the database failover.</p>
Establishing Terms</h3>
Our application server constantly receives requests, out of which a certain percentage of requests will trigger the code to connect to the affected database, which we’ll call x</em>-type requests. The other requests, that do not trigger a database connection, we’ll call x’</em>-type requests.</p>
Analysis</h3>
With the background knowledge gathered in our experiments, let’s try to analyse all the steps that happened during our production outage.</p>

Rails started from a clean state, with no connections set up to the database initially</li>
Rails handles the first few x</em> request types, opens a connection to the database</li>
Subsequent requests of x</em> type can reuse the same connections from the connection pool</li>
At a certain time, due to a hardware fault out of our control, a failover of the database is triggered</li>
At the same time requests of x</em> type comes in — and ActiveRecord reuses the same database connection from the pool, but there is no response. It then waits for read_timeout</code>, causing the thread to be stuck waiting for the default timeout</li>
Even though Rails can process requests of the x’</em> type normally, more and more requests of x</em> type come in and cause more and more threads to be stuck waiting</li>
Eventually, all the available threads to handle requests are stuck waiting on the TCP connection to the failed database, and Rails can no longer respond to new requests</li>
After the default read_timeout</code> has elapsed (3 × 10 minutes), some threads will be released to handle new requests</li>
Subsequent requests of x</em> type will cause a new connection to be opened to the database

If the failover is complete and the DNS records for the new instance has been updated, the new connections will be established</li>
If the failover is not complete or the DNS records were not updated, the TCP connections will still try to connect to the old IP address with the failed database instance. The connections will wait for the connect_timeout</code> (default 120 seconds) to elapse before failing</li>
</ul>
</li>
Finally, once all the threads are stuck, our Rails application stops responding to all requests until it was restarted manually</li>
</ol>
Solution</h4>
To fix the problem, we have to prevent our database connections from being stuck in trying to read from an unresponsive socket, and trying to connect to a closed socket.</p>
This can be done by simply setting the read_timeout</code> so that when the database fails, existing connections and threads will be released. The connect_timeout</code> also has to be set so that when the existing connections are released, new connections and threads handling the requests will not be stuck trying to connect to the same unavailable database.</p>
We set the following values in our staging environment and manually triggered a database failover via the AWS console, and observed that requests of the x’</em> type are no longer stalled during the failover.</p>
The following is a snippet for our current database.yml</code> configuration before the outage, and the changes to resolve the problem.</p>
# Config for the non-primary `db_other` database</span></span>
production_other</span>:</span></span>
  adapter</span>:</span> mysql2</span></span>
  encoding</span>:</span> utf8</span></span>
  reconnect</span>:</span> false</span></span>
  database</span>:</span> …</span></span>
  pool</span>:</span> …</span></span>
  reaping_frequency</span>:</span> 120</span></span>
  username</span>:</span> …</span></span>
  password</span>:</span> …</span></span>
  host</span>:</span> …</span></span></code></pre># New changes</span></span>
+ connect_timeout</span>:</span> 5</span></span>
+ read_timeout</span>:</span> 5</span></span></code></pre>Conclusion</h2>
In this post, we have gone over how timeouts are handled by the ActiveRecord ORM with our MySQL database and how failing to configure them brought down some of our production systems.</p>
Timeouts are very important configurations when setting up distributed systems and they are easily overlooked in the initial deployments of such applications.</p>
These principles are not just limited to Rails or MySQL, and the experiments and their findings can be easily extended to other technologies as well. Needless to say, these timeout settings are extremely important for the resiliency of applications in the world of micro services.</p>
References</h2>

ankane/the-ultimate-guide-to-ruby-timeouts</a></li>
ankane/production_rails</a></li>
MySQL Reference Manual</a></li>
MySQL Source Code Mirror</a></li>
TCP Connection States</a></li>
</ul>
Big thanks to Joel Low</a> for helping out with this investigation and clarifying ambiguities in Rails and MySQL, and my manager Amit Saini for his helpful review of this post!</p>
Cover photo by Artem Verbo</a> on Unsplash</a>.</p>
Source code for the test rails application can be found here</a>.</p>
Jia Hao

Writing Java With a Dash of Functional Programming

More Declarative Implementation</h2> Let us now iterate on our earlier solution. First, we rewrite tryLoadFromLoader</code> to return an Either</code> object instead of a tuple.</p>

Appendix</h2> Full Source Code</a></p>

Optimistic Locking with the DynamoDB Mapper

Handling of Version Fields</h3> The UpdateItem request is constructed by iterating through all the annotated fields of the model object. A special handler is invoked when a field annotated with @DynamoDBVersionAttribute</code> is encountered:</p>

Deep Dive into Database Timeouts in Rails

Appendix</h2>
Full Source Code</a></p>
