Passing our tests

If you recall from the last tutorial, we have a few tests that are currently failing. This is a good thing, because now we know exactly what we need to do to get the tests passing. For reference, here are the tests from last time:

(ns restful-clojure.handler-test
  (:use clojure.test
        ring.mock.request
        restful-clojure.handler))

(deftest test-app
  (testing "users endpoint"
    (let [response (app (request :get "/users"))]
      (is (= (:status response) 200))
      (is (= (get-in response [:headers "Content-Type"]) "application-json"))))

  (testing "lists endpoint"
    (let [response (app (request :get "/lists"))]
      (is (= (:status response) 200))
      (is (= (get-in response [:headers "Content-Type"]) "application-json"))))

  (testing "not-found route"
    (let [response (app (request :get "/bogus-route"))]
      (is (= (:status response) 404)))))

We need to expose /users and /lists as JSON api endpoints, and we also need to have our app send an HTTP 404 response for undefined routes. Since the last test will be the simplest to pass, let’s go ahead and tackle that one first. Open up handler.clj and get rid of the code from the dummy counting routes that we created before, and replace them instead with a handler that only has a not-found route:

(defroutes app-routes
  (route/not-found "{\"message\":\"Page not found\"}"))

(def app app-routes)

If you’re like me, you might be wondering how the not-found route works. Well, since we’re dealing with a Clojure library, we’re in luck! Go ahead and start up a REPL (lein repl from the restful_clojure directory), and we can delve into the code:

(doc compojure.route/not-found)
; => ([body])
; =>   A route that returns a 404 not found response, with its argument as the
; =>   response body.
; => nil
(source compojure.route/not-found)
; => (defn not-found
; =>   "A route that returns a 404 not found response, with its argument as the
; =>   response body."
; =>   [body]
; =>   (wrap-head
; =>     (fn [request]
; =>       (-> (response/render body request)
; =>           (status 404)))))
; => nil

Expecting more? Sorry to disappoint! That is the beauty of lisp - that something like handling 404s can be handled with a 4 line function.

If you run the tests again, you’ll notice that the last one is passing! However, we’re creating a JSON string directly in our code - yuck! Before we move on to implementing the /users and /lists endpoints, let’s introduce some middleware that will create JSON response bodies from Clojure data structures.

Wiring custom Ring middleware

In most applications, there are certain concerns (e.g. logging, authentication, content nedotiation, etc.) that we want to take care of at a high level. In some frameworks, these concerns are taken care of by a Front Controller or plugin. With Ring, however, we can simply wrap our routes in another handler that does… well, pretty much anything we want it to. Can you feel the power? I can feel the power.

Middleware are higher-order functions that take in a handler function and return a handler function. You can modify the request or the response, or you can perform some side-effect like logging the request to a file. Let’s extend our application by adding some extremely basic logging - we’ll log the request map to the server’s console:

; handler.clj
; ...

; Behold, our middleware! Note that it's common to prefix our middleware name
; with "wrap-", since it surrounds any routes an other middleware "inside"
(defn wrap-log-request [handler]
  (fn [req] ; return handler function
    (println req) ; perform logging
    (handler req))) ; pass the request through to the inner handler

; We can attach our middleware directly to the main application handler. All
; requests/responses will be "filtered" through our logging handler.
(def app
  (-> app-routes
    wrap-log-request))

If you fire up the server with lein ring server-headless (or just server if you’re testing it out locally). And hit it with any URL, you should see your request map printed out to the console. This can be helpful for debugging in development but is probably not something that we want in production.

As you can see, writing middleware is not terribly complicated. That doesn’t mean, however, that we should re-invent the wheel just because we can. For our application, we want to convert Clojure data structures to JSON responses (and we probably want to parse JSON request bodies as well). For this we can use ring-json from James Reeves (the maintainer of Ring and a number of other projects). Let’s add it to our project.clj as a dependency:

(defproject restful-clojure "0.1.0-SNAPSHOT"
; ...
:dependencies [
; ... other dependencies ...
		[ring/ring-json "0.2.0"]])

Now over in handler.clj, we can add the middleware provided by ring-json. Here’s what the entire file should look like at this point:

(ns restful-clojure.handler
  (:use compojure.core
        ring.middleware.json)
  (:require [compojure.handler :as handler]
            [ring.util.response :refer [response]]
            [compojure.route :as route]))

; Notice that the body of out not-found route is now a Clojure map.
; The wrap-json-response middleware will take care of converting this to a
; JSON string.
(defroutes app-routes
  (route/not-found
    (response {:message "Page not found"})))

(defn wrap-log-request [handler]
  (fn [req]
    (println req)
    (handler req)))

(def app
  (-> app-routes
    wrap-log-request
    wrap-json-response
    wrap-json-body))

With this middleware in place, we are all set to parse JSON request bodies and serve up JSON responses. Now we are finally ready to implement the core of our application, starting with the tests.

Persisting to a database

In order to persist our data to a database, we’ll add the JDBC driver for PostgreSQL, the Korma library (which provides a nice DSL for querying SQL databases), and Ragtime for database migrations. Let’s add the dependencies to project.clj. I’ll go ahead and display the entire source of the file since we’re adding code in a few places:

(defproject restful-clojure "0.1.0-SNAPSHOT"
  :description "An example RESTful shopping list application back-end written
in Clojure to accompany a tutorial series on kendru.github.io"
  :url "https://github.com/kendru/restful-clojure"
  :license {:name "MIT"
            :url "http://opensource.org/licenses/MIT"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [ring/ring-core "1.2.1"]
                 [ring/ring-jetty-adapter "1.2.1"]
                 [compojure "1.1.6"]
                 [cheshire "5.3.1"]
                 [ring/ring-json "0.2.0"]
                 [korma "0.3.0-RC5"]
                 [org.postgresql/postgresql "9.2-1002-jdbc4"]
                 [ragtime "0.3.4"]]

  :plugins [[lein-ring "0.8.10"]
            [ragtime/ragtime.lein "0.3.6"]]

  :ring {:handler restful-clojure.handler/app
         :nrepl {:start? true
                 :port 9998}}

  ; Have ragtime default to loading the database URL from an environment
  ; variable so that we don't keep production credentials in our
  ; source code. Note that for our dev environment, we set this variable
  ; with Puppet (see default.pp).
  :ragtime {:migrations ragtime.sql.files/migrations
            :database (System/getenv "DB_URL")}

  :profiles
  {:dev {:dependencies [[javax.servlet/servlet-api "2.5"]
                        [ring-mock "0.1.5"]]}
   :test {:ragtime {:database
"jdbc:postgresql://localhost:5432/restful_test?user=restful_test&password=pass_test"}}})

Now we can create the directory to hold our migrations and create our initial migrations.

$ pwd
# /home/andrew/restful-clojure/restful-clojure
$ mkdir migrations
$ NOW=$(date "+%Y-%m-%d-%H%M%S")
$ touch migrations/$NOW-add-initial-tables.up.sql
$ touch migrations/$NOW-add-initial-tables.down.sql

The migrations are in the project repo, but we won’t go into them in detail here. Basically, they create tables for users, lists, products, and product_lists (and a couple other things). If you’re following along with the project, you can just copy the files from the repo.

And with that, we should be able to run the initial migration from our VM to create the users table:

$ lein ragtime migrate
# Applying 2014-03-03-213517-add-initial-tables

And let’s go ahead and migrate the test database as well:

$ lein with-profile test ragtime migrate
# Applying 2014-03-03-213517-add-initial-tables

This will use the comfiguration specified under the :test profile specified in project.clj for migrating the database. We’re set up. We’re migrated. We’re ready to code.

Getting to the good stuff

Let’s begin by writing the tests for our CRUD operations on both users and lists:

; tests/restful_clojure/users_test.clj
(ns restful-clojure.users-test
  (:use clojure.test
        restful-clojure.test-core)
  (:require [restful-clojure.models.users :as users]
            [environ.core :refer [env]]))

; Run each test in an isolated db transaction and rollback
; afterwards
(use-fixtures :each with-rollback)

(deftest create-read-users
  (testing "Create user"
    (let [count-orig (users/count-users)]
      (users/create {:name "Charlie" :email "charlie@example.com"})
      (is (= (inc count-orig) (users/count-users)))))

  (testing "Retrieve user"
    (let [user (users/create {:name "Andrew" :email "me@mytest.com"})
          found-user (users/find-by-id (user :id))]
      (is (= "Andrew" (found-user :name))
      (is (= "me@mytest.com" (found-user :email))))))

  (testing "Find by email"
    (users/create {:name "John Doe" :email "j.doe@ihearttractors.com"})
    (let [user (users/find-by-email "j.doe@ihearttractors.com")]
      (is (= "John Doe" (user :name))))))

(deftest multiple-user-operations
  (testing "Find all users"
    (doseq [i (range 10)]
      (users/create {:name "Test user"
                     :email (str "user." i "@example.com")}))
    (is (= 10 (count (users/find-all))))))

(deftest update-users
  (testing "Modifies existing user"
    (let [user-orig (users/create {:name "Curious George" :email
"i.go.bananas@hotmail.com"})
          user-id (user-orig :id)]
      (users/update-user (assoc user-orig :name "Chiquita Banana"))
      (is (= "Chiquita Banana" (:name (users/find-by-id user-id)))))))

(deftest delete-users
  (testing "Decreases user count"
    (let [user (users/create {:name "Temporary" :email
"ephemerial@shortlived.org"})
          user-count (users/count-users)]
      (users/delete-user user)
      (is (= (dec user-count) (users/count-users)))))

  (testing "Deleted correct user"
    (let [user-keep (users/create {:name "Keep" :email "important@users.net"})
          user-del (users/create {:name "Delete" :email
"irrelevant@users.net"})]
      (users/delete-user user-del)
      (is (= user-keep
             (users/find-by-id (user-keep :id))))
      (is (= nil
             (users/find-by-id (user-del :id)))))))

The basic tests for lists are pretty similar, so I won’t post them here, but they are in the tutorial repository. I have also included a users+lists_test.clj for testing the operations that affect both models. Later in this tutorial we will take a look at the tests and implementation code for dealing with managing the items on a list.

Now the implementation for both users and lists is fairly straightforward. We’re using Korma to simplify database transactions. We can put all of the boilerplace Korma code that we will use to define the entities in our system in a restful-clojure.entities namespace. The models themselves will go in separate namespaces under restful-clojure.models.

; src/restful_clojure/entities.clj
(ns restful-clojure.entities
  (:use korma.core
        restful-clojure.db))

(declare users lists)

(defentity users
  (pk :id)
  (table :users)
  (has-many lists)
  (entity-fields :name :email))

(defentity lists
  (pk :id)
  (table :lists)
  (belongs-to users {:fk :user_id})
  (entity-fields :title))
; src/restful_clojure/models/users.clj
(ns restful-clojure.models.users
  (:use korma.core)
  (:require [restful-clojure.entities :as e]))

(defn find-all []
  (select e/users))

(defn find-by [field value]
  (first
    (select e/users
      (where {field value})
      (limit 1))))

(defn find-by-id [id]
  (find-by :id id))

(defn for-list [listdata]
  (find-by-id (listdata :user_id)))

(defn find-by-email [email]
  (find-by :email email))

(defn create [user]
  (insert e/users
    (values user)))

(defn update-user [user]
  (update e/users
    (set-fields (dissoc user :id))
    (where {:id (user :id)})))

(defn count-users []
  (let [agg (select e/users
              (aggregate (count :*) :cnt))]
    (get-in agg [0 :cnt] 0)))

(defn delete-user [user]
  (delete e/users
    (where {:id (user :id)})))

Again, since the boilerplate code for lists is not too interesting (and it’s very similar to the code for users, I am not posting the entire code here. I’ll only cover what’s different and interesting about lists - adding and removing items.

You know the drill. We start with the tests:

(By the way, I know that unit tests are not the most riveting read, so please do not feel beholden to take in every line. I post the tests here in case you are interested in seeing some examples of ordinary tests in Clojure.)

; test/restful_clojure/lists_test.clj
; ...
(deftest add-products
  (let [user (users/create {:name "Test user" :email "me@mytest.com"})
        my-list (lists/create {:user_id (:id user) :title "My list"})
        pintos (products/create {:title "Pinto Beans"
                                 :description "Yummy beans for burritos"})]
    (testing "Adds product to existing list"
      (let [modified-list (lists/add-product my-list (:id pintos))]
        (is (= [pintos] (:products modified-list)))))

    (testing "Creates new list with products"
      (let [listdata (lists/create {:user_id (:id user)
                                    :title "Most interesting"
                                    :products [pintos]})]
        (is (= [pintos] (:products listdata)))
        (is (= [pintos] (:products (lists/find-by-id (:id listdata)))))))

    (testing "Creates products added with an update"
      (let [listdata (lists/create {:user_id (:id user)
                                    :title "Things to update"
                                    :products [pintos]})
            coffee (products/create {:title "Coffee Beans"
                                     :description "No, not *THAT* Java"})
            updated (lists/update-list (update-in listdata [:products] conj coffee))]
        (is (= [pintos coffee] (:products updated)))
        (is (= [pintos coffee] (:products (lists/find-by-id (:id listdata)))))))))

(deftest remove-products
  (let [user (users/create {:name "Test user" :email "me@mytest.com"})
        kidneys (products/create {:title "Kidney Beans"
                                  :description "Poor Charlie the Unicorn..."})
        limas (products/create {:title "Lima Beans"
                                :description "Yuck!"})
        my-list (lists/create {:user_id (:id user)
                               :title "My list"
                               :products [kidneys limas]})]
   (testing "Does not remove a product from the database entirely"
     (let [fresh-list (lists/create {:user_id (:id user)
                                     :title "My list"
                                     :products [kidneys limas]})]
       (lists/remove-product fresh-list (:id kidneys))
       (is (not (nil? (products/find-by-id (:id kidneys)))))))
    (testing "Removes a product from a list"
      (let [modified-list (lists/remove-product my-list (:id kidneys))]
        (is (= [limas] (:products modified-list)))))

    (testing "Removes products absent from an update"
      (let [coffee (products/create {:title "Coffee Beans"
                                     :description "No, not *THAT* Java"})
            listdata (lists/create {:user_id (:id user)
                                    :title "Things to update"
                                    :products [limas coffee]})
            updated (lists/update-list (assoc listdata :products [coffee]))]
        (is (= [coffee] (:products updated)))
        (is (= [coffee] (:products (lists/find-by-id (:id listdata)))))))))

Although I just pasted all of these tests in at once, keep in mind that when writing code, I try to only write one (failing) test at a time, then write the code to implement it, but it would be distracting to have so many tiny code samples littering the post.

There are a couple of interesting things going on here. First, we want to be able to associate products with a list when it is created and associate/disassociate them when the list is updated. Second, when we remove a product from a list, we want to be sure that we are just breaking the relationship between lists and products, not deleting the product entirely. Now the application code:

; src/restful_clojure/models/lists.clj
; ...
(defn add-product
  "Add a product to a list with an optional status arg"
  ([listdata product-id]
    (add-product listdata product-id "incomplete"))
  ([listdata product-id status]
    (let [sql (str "INSERT INTO lists_products ("
                   "list_id, product_id, status"
                   ") VALUES ("
                   "?, ?, ?::item_status"
                   ")")]
      (exec-raw [sql [(:id listdata) product-id status] :results])
      (find-by-id (:id listdata)))))

(defn remove-product [listdata product-id]
  (delete "lists_products"
    (where {:list_id (:id listdata)
            :product_id product-id}))
   (update-in listdata [:products]
     (fn [products] (remove #(= (:id %) product-id) products))))

(defn- get-product-ids-for
  "Gets a set of all product ids that belong to a particular list"
  [listdata]
  (into #{}
    (map :product_id
      (select "lists_products"
        (fields :product_id)
        (where {:list_id (:id listdata)})))))

(defn update-list [listdata]
  (update e/lists
    (set-fields (dissoc listdata :id :products))
    (where {:id (:id listdata)}))
  (let [existing-product-ids (get-product-ids-for listdata)
        updated-product-ids (->> (:products listdata)
                                 (map :id)
                                 (into #{}))
        to-add (difference updated-product-ids existing-product-ids)
        to-remove (difference existing-product-ids updated-product-ids)]
    (doseq [prod-id to-add]
      (add-product listdata prod-id))
    (doseq [prod-id to-remove]
      (remove-product listdata prod-id))
    (find-by-id (:id listdata))))

Notice that in the add-product function, we build up a query to execute against the database directly. This is because we are taking advantage of Postgres’s Enumerated Types, which do not play nice with Korma. I am slowly becoming convinced that SQL is best written in SQL, so if I were to start this tutorial again, I would probably use something like yesql.

Additionally, when we update a list we compare the set of products that are on the list at update time to the products that were previously on the list, adding the new products, and removing the products that are no longer present.

Coming up next…

Whew. That was a lot of code! If you have made it this far, I applaud you. In the next tutorial, I will cover adding some basic authentication and authorization to the API using buddy, which is a very flexible security library. After that, we’ll create a simple ClojureScript client to consume our API - after all, what good is a service without anything to use it? Finally, we’ll deploy the app to a DigitalOcean server (disclaimer: affiliate link) using nginx as a reverse proxy and SSL terminator.

Go To