Deniz Eren An ordinary person

Writing Tests for Dynamodb Application

While developing my side project I needed to write tests for my backend which was using Dynamodb. So connecting to Amazon in my test code was not the solution (obviously), but what was the right way to write tests? I looked at goamz library, which I used for dynamodb interaction in my project.

DYNAMODB_LOCAL_VERSION = 2014-10-07

launch: DynamoDBLocal.jar
    cd dynamodb_local_$(DYNAMODB_LOCAL_VERSION) && java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar

DynamoDBLocal.jar: dynamodb_local_$(DYNAMODB_LOCAL_VERSION).tar.gz
    mkdir -p dynamodb_local_$(DYNAMODB_LOCAL_VERSION)
    [ -f dynamodb_local_$(DYNAMODB_LOCAL_VERSION)/DynamoDBLocal.jar ] || tar -C dynamodb_local_$(DYNAMODB_LOCAL_VERSION) -zxf dynamodb_local_$(DYNAMODB_LOCAL_VERSION).tar.gz

dynamodb_local_$(DYNAMODB_LOCAL_VERSION).tar.gz:
    curl -O https://s3-us-west-2.amazonaws.com/dynamodb-local/dynamodb_local_$(DYNAMODB_LOCAL_VERSION).tar.gz

clean:
    rm -rf dynamodb_local_$(DYNAMODB_LOCAL_VERSION)*

Voila! This was the trick. Guys at Amazon build a small client-side version of dynamodb using Java so that people like me can test their applications without connecting to Amazon.

Using this local dynamodb application I was able to write a small dynamodb backend for gorilla sessions and write tests for my side project.

Happy hacking.

Flask file descriptor inheritance problem

Last month my friend was writing a service monitoring and controlling app using flask. The app was simply checking status of service in the server and also start, stop and restart functions were available to users in this app.

In about a week my friend came across with an expected bug, file descriptor inheritance. He was complaining about port already in use errors. When I checked who was listening to that port using netstat it was interesting, the services managed by the app were listening to app’s port.

crazy-server# netstat -nlp|grep 8000
tcp        0      0 0.0.0.0:8000           0.0.0.0:*               LISTEN      1958/mysql

The problem was obviously file descriptor inheritance. after service monitor app starts or restarts a service its open file descriptors were inherited by the new started processes.

Solution was setting CLOEXEC flag using fcntl. To do that we first had to get all open file descriptors since flask was only creating sockets we needed to find only socket objects. Using python’s garbage collector we fetched all socket objects.

filter(lambda x: type(x) == socket._socketobject, gc.get_objects())

After that we set CLOEXEC flag on all socket file descriptors.

for sock in filter(lambda x: type(x) == socket._socketobject, gc.get_objects()):
    fd = sock.fileno()
    old_flags = fcntl.fcntl(fd, fcntl.F_GETFD)
    fcntl.fcntl(fd, fcntl.F_SETFD, old_flags | fcntl.FD_CLOEXEC)

And our fd inheritance bug was solved.

Happy hacking.