Alex Richey

Software Engineer @ Meural

Bringing Meural's Frontend to the Bleeding Edge

My article on my.meural's redesign originally published on Meural's Product Blog.

When we decided to redesign our flagship web app at Meural, we took the opportunity to rethink our approach to React and other frontend infrastructure. We ultimately decided to migrate from our own home-cooked server side rendering solution to Next.js, to change our approach to data handling with Redux, and to adopt styled-components. These decisions enabled us to develop and deploy our new frontend in just two months. In this article, I’ll talk about the upshot of these choices and about how things have worked out in production.

Next.js

Next.js is a framework for React apps that handles both server and client-side code. It offers a number of features, including server side rendering and automatic code-splitting in production; and hot module replacement in development. I know from experience that implementing these features from scratch is far from trivial. Therefore, the notion of starting a new project with Next.js at the foundation was attractive.

In addition to these flagship features, there are also positive externalities. Since Next.js is open source and has an active community behind it, common problems are faced at the community-level and standard, well-tested solutions emerge. These solutions can be found in the Next.js repo in a folder called /examples. Inside are skeleton projects with commonly desired integrations such as with Redux, styled-components, and others. Having these examples available significantly reduces mental overhead in implementing such integrations in one’s own project. This accelerates development as a result.

The price one pays for these features is to surrender control of routing and considerable portions of application architecture and configuration. Next.js jettisons the de facto routing standard of React-Router and requires your React application to be divided into what it calls pages. A page is a top-level React component that is associated with a URL. The notion of a page is also what enables automatic code-splitting — bundles are split at each page and preloaded on the client with <link rel="preload">.

We decided that the features of Next.js, coupled with its positive externalities, outweigh the loss of control of our application architecture. Moreover, we found that adopting Next.js’ architecture simplified our application. By distinguishing between pages — those are, top-level components — and ordinary components, the directory structure of our frontend codebase became more easily comprehensible.

Redux

One of the challenges we faced in adopting Next.js is how to persist data across pages. Since Next.js’ architecture is based on pages, and since pages themselves are React components, which are unmounted on navigation events, their state is lost with every page navigation. We solved this problem by persisting state across page navigation events in a Redux store. What’s more, we broke away from standard Redux use patterns by using it only for persistent data.

Roughly speaking, use patterns of Redux can be placed on a continuum where, at one extreme, every bit of application state is stored in Redux and every mutation of state is achieved through a Redux action. At the other extreme, state is managed entirely at the component level and there is no store of global state.

In our previous frontend, we tended toward the former use pattern. This pattern offered some advantages — all data was fetched consistently through Redux actions with Redux-Thunk; and every connected component could gain visibility to any other area of the application. But we decided to abandon this approach for two main reasons. First, the excessive boilerplate that Redux requires makes extending functionality more work than it needs to be. Second, we often found ourselves dispatching actions that essentially mimic ordinary component lifecycle methods. For example, suppose that on componentDidMount, I fetch some data and save it in the Redux store. Then, in order to prevent the user from seeing stale data when this component loads again, I dispatch an action that clears this area of the Redux store on componentWillUnmount. It seems much simpler to store this data in the component state, where it will be naturally destroyed when the component unmounts and naturally fetched on componentDidMount, than to store it in the Redux store where it needs to be manually destroyed.

These considerations led us to the strategy of using Redux only for data that is truly global. As a result, we now have an extremely thin Redux store that stores only user, device, and configuration data. All other data is fetched and managed at the component level. We’ve found that this approach both makes our React components easer to comprehend and dramatically reduces boilerplate.

Styled-Components

The final change to our frontend stack was to adopt styled-components. In some ways, styled-components represent the terminus of a new paradigm in web development. In the old paradigm, the best-practice was to keep markup, JavaScript, and styling separate, on the grounds that these were separate concerns that should be isolated from each other. React changed this by bringing markup into JavaScript which made programmatically rendering markup less cumbersome and more easy to reason about. Styled-Components takes the philosophy that React developed and extends it to styles. It brings styling into JavaScript as well, where it can also receive the benefits of programmatic handling. The result of these developments is that concerns are grouped by component rather than by language.

Prior to using styled-components, our strategy was to include a sass file for every component and to bundle everything together with Webpack’s sass and css-loaders. This strategy was successful, but it often made styling components overly cumbersome. There were also the typical issues of globals and specificity in css.

Adopting styled-components solved these problems for us and, more significantly, it has made predicting and reasoning about the visual behavior of each component easier. We now keep all of the code related to one component in a single .jsx file that contains markup, JavaScript, and styling. This makes each component much easier to comprehend and, from a logistical point of view, easier to change because one no longer needs to open three or four files at the same time to handle just one locus of behavior. Now, I can open several components at once and think about their interrelations and the signatures that each one should have. In other words, I can reason about the application in a more abstracted and powerful way.

Production

The strategies that I described above enabled the web team at Meural, which, at the time, consisted of myself and one other developer, to build and deploy our new frontend in just two months. I believe that our focus on reducing mental overhead by making our codebase easier to comprehend enabled us to deliver on schedule. Nevertheless, we have noticed some issues in production.

The primary issues that we have seen are related to performance and stem from Next.js and styled-components. For all of their virtues, the combination of Next.js and styled-components creates large JavaScript bundles. At the time of this writing, our main.js bundle is just under 800kb and other split bundles range from less than 10kb to more than 100kb. This increased our page time to interactive from under 2.5 seconds to over 5 seconds, which is not acceptable.

With gzip, we can reduce the size of these bundles by roughly 75%, which would get our page load time back into an acceptable range. The problem is that Next.js does not serve static files in a straightforward way. Static assets are built and saved to the /.next directory, but the directory structure does not match the URLs that the client requests. These requests must be routed through the Next.js application to be resolved to the correct static asset. This prevents us from easily gzipping and serving these files independently from the Next.js application.

The solution that we developed is to leverage NGINX’s caching capabilities. For many reasons, we run our Next.js application behind an NGINX server. We decided to use the proxy_cache directive to cache responses at the /_next/ location. The way this works is that the first request for any asset at /_next/ will be forwarded to the application with proxy_pass, but the resulting response will be saved in NGINX’s cache. Subsequent requests will then be served out of the cache. Here are the relevant parts of the configuration.

http {
    ...

    # Cache setup

    proxy_cache_path /data/nginx/cache keys_zone=one:20m; 
    proxy_cache_valid 200 60m;


    # Gzip settings

    gzip_static on;
    gzip on;
    gzip_comp_level 5;
    gzip_types application/javascript ...;
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store;

    server {
        listen 80;

        ...

        location /_next/ {
            proxy_pass http://application:8000;
            gzip_proxied any;
            proxy_cache one;
            break;
        }

        ...

    }
}

Conclusion

We have been running our new frontend stack in production for a little over two months and, so far, have had an easy time updating and extending our application. All of the decisions we made regarding Next.js, Redux, and styled-components seem to have paid off and it has been easy to live with the few inflexible areas of Next.js. After we deployed our new NGINX configuration, page load performance was restored to our target range and our application, at least in my opinion, has a wonderful, speedy, and modern feel.

Setting Up Server Side Rendering with React, Redux, and Django

My article on server side rendering originally published on Meural's Product Blog.

At Meural, we decided to implement server side rendering in order to increase our SEO exposure and to make social media sharing more effective. In this post, I’ll talk about how server side rendering works and the implementation that I developed.

Background

In traditional web applications, web pages are rendered on the client side. The browser receives a blob of JavaScript from the server, processes it, and paints the UI that the user sees. In server rendered applications, on the other hand, the first render of a web page is done on the server. The browser receives a pre-rendered page, which it can display without running any JavaScript. This enables better SEO exposure, since many web crawlers cannot run JavaScript, and faster perceived page load times.

There are also some downsides to server side rendering. These include slower server response time and, in lower bandwidth environments, slower time to interactive. For apps that run behind login screens, whose content is private, these downsides might make server side rendering less than worthwhile. For our use case, however, the benefits well outweigh the costs. Most of our app’s pages are public facing, so exposure to web crawlers is essential.

Preliminary Considerations

Implementing server side rendering is not a trivial task. In response to this fact, a number of libraries have emerged to make implementation a bit easier. These include Next.js, Razzle, numerous boilerplates, and others.

These libraries and frameworks can be a good choice for rapidly prototyping a feature or when starting a new app, but I would not recommend them for use in production or for extant apps. The reason for this is that using them requires surrendering a large portion of your codebase to them, which makes debugging and optimization more difficult, if not impossible, and makes you ignorant of how your app is really working. What’s more, integrating such a framework into an extant application is often more trouble than it’s worth.

Therefore, I developed a custom implementation of server side rendering at Meural. I hope that our stack is similar enough to that of other companies so that other developers might find this article useful. Our frontend uses React, React Router 3, and Redux, while our backend is a monolithic Django application.

The High-Level View

At a high level, setting up server side rendering consists in setting up the following chain of events.

SSR Schema

Since our primary application server is a Django application, which cannot understand JavaScript, we need a JavaScript runtime to render our React frontend. For this, we’ll use a Node server. When a request hits our primary Django server, we’ll query our database to get the info we need. Next we’ll send that info in an HTTP POST request to our Node server, which will return our markup, plus the final state of our Redux store. Finally, we’ll embed this information into the HTML response of our Django app and send it to the client.

Node-Django Interaction

Let’s begin by setting up the Node server. I decided to use Express.js because it is battle-tested and very easy to use. Note that we are reading our NODE_HOST and NODE_PORT variables from our runtime environment.

I recommend writing a simple render and buildInitialState functions for testing purposes that simply return some valid output of any kind. I also recommend testing this server with cURL before moving on to anything else.

Now let’s wire up the Django app and test it’s interaction with the Node server.

Here’s how we insert the rendered HTML payload into our Django template. Note that we use Webpack and django-webpack-loader to handle our client-side JavaScript.

We can now test the interaction between Node and Django. Let’s start the Node server and the Django server, open up a browser, and go to the url that corresponds to our sandwich view. To prevent our React frontend from taking over the page on load, we’ll disable JavaScript in DevTools. If you see a page with the output that you defined in your render and buildInitialState functions, then all is well.

Defining the Render Function

It will be instructive to first look at the code of the render function and then to explain how it works.

The first thing the render function does is configure the Redux store. I used the same configureStore function that I had already defined in following the usual Redux API pattern.

The second thing the render function does is get the frontend routes of my React app, by calling a function I wrote called getRoutes. This function takes the Redux store as an argument and returns all of the routes to my app. It prevents code duplication because I can call it in both server and browser environments (See the Handling the Client Side section below to see how it is used in a browser environment).

The name getRoutes, though accurate, is somewhat incomplete. The function does not just get routes. It also gets the React components of which my app is composed, since they are embedded in the definitions of the routes themselves. Therefore, the getRoutes function is what links my existing React app to the render function.

Next, in order for the render function to match the desired route, I use React Router’s match function. This function’s first argument is an object containing all of my app’s routes as its first key — which we have from the getRoutes function — and the desired route as the second key. The match function’s second argument is a callback that gets evoked after matching is complete. In a successful matching, this callback’s third argument is a renderProps object. These renderProps represent the state of my app’s props at a given route. I pass this object into React-Router’s <RouterContext> component (which is a static version of its more familiar <Router> component) to render the state of our app at the matched route.

To give the components of my app access to the Redux store, I wrap the <RouterContext> component with the <Provider> component from the React-Redux library.

Next, I call ReactDOMServer#renderToString with this wrapped component as its argument to render the state of my application at the matched route to HTML.

Finally, I call getState on my Redux store to extract its final state in case the rendering process changed anything.

If the match function fails to match the desired route, the second argument of its callback is a redirectLocation argument. In this case, I recursively call the render function with this new desired route. I am confident that there will never be a chain of infinite redirects because I have defined a wildcard route to handle such cases.

Calling the Render Function

The render function will not work as it is currently defined. The reason for this is that the JSX used in the function itself, as well as in the rest of my app, is not understood in Node runtimes. Therefore, I use Webpack and Babel to transpile my render.jsx file into Node compliant code. All I had to do to make this work was to copy my existing webpack config, change the entry point to render.jsx, and replace the target parameter with target: ‘node’.

When transpiling, I ran into some errors. Since this version of my React app will not be running in the browser, window and document will not be defined. Therefore, I had to move all references to window and document to functions that are executed only after the DOM is accessible. This involved moving references to window and document from methods like Component#constructor to methods like Component#componentDidMount. Unlike Component#constructor, Component#componentDidMount will only be called after the component has been mounted to the DOM. I also had to abandon some third-party libraries that relied on window or document in problematic places.

Handling the Client Side

Now that my server responds with a fully rendered page of my app, I need to adjust my client side JS to expect this. Here’s the code I wrote.

You might have noticed that I added an async attribute to my script tag in base.html. The async attribute makes the tag non-render-blocking, which means that the browser won’t wait for the entire script to download before rendering. This produces a considerable speed increase, especially with large JavaScript bundles. However, it also means that it is possible for the script to be executed at any time during the load process, which means that the standard procedure of waiting for DOMContentLoaded before rendering with ReactDOM might not always work, since DOMContentLoaded might have already fired, in which case, the React app would never get executed and the page would never become interactive. Therefore, I check the document.readyState when the bundle is initially executed. If the readyState is complete or interactive, I initialize my React app right away. Otherwise, I add listener for DOMContentLoaded and use my initialize function as the callback.

In my initializeApp function, I get the current state of the Redux store from the window and pass it into configureStore to setup Redux. Next, I match the current route, just as I did on the server side, using the same getRoutes and match functions which I discussed above. Instead of calling ReactDOM#render, which is the usual pattern in client rendered apps, I call ReactDOM#hydrate, which sets up React’s virtual DOM and installs listeners to make the page interactive.

Conclusion

Since deploying this project, we have seen much better SEO at Meural. Now a Google search for the terms meural and some artist’s name will yield a result of that artist’s page or one of her playlists, if that artist is in our collection. Prior to this deployment, this was not possible, since we had exposed only one webpage, which contained our single-page React app.

From Philosophy to Computer Science

image When I say that I’m a software developer with a background in academic philosophy, many people react with surprise, as if I had jumped between completely divergent fields. Philosophy and computer science are in fact closely related and it is easy to transition from one to the other. In this article I would like to highlight a few reasons why this is so, with the hopes of dispelling the common misperception that they are opposites and of giving some background for students of philosophy who are interested in transitioning to computer science.

Logical Similarities

The basis of both philosophy and computer science is logic. In philosophy, it’s often used to evaluate arguments, whereas in computer science, it is the means by which the processor is given instructions. Having a good grasp of logic’s core concepts is essential to do well in either.

Consider this example to see what I mean. Suppose I say,

If Trump or Hillary wins, then all is lost.

This statement can be represented in philosophical notation as follows:

(1) (A ∨ B) → C

where A is "Trump wins", B is "Hillary wins", C is "All is lost," and and are shorthand for "or" and "If … then …," respectively.

Now let’s write a simple program that captures the content of our claim. Let’s make our computer print "All is lost," if either Hillary or Trump win, and print "Not all is lost," if neither of them wins.

if trump_wins || hillary_wins
    print "All is lost."
else
    print "Not all is lost."
end

Cool. If we run this program after the election, it will print "All is lost," or "Not all is lost," depending on the result. Notice that, as far as the logic is concerned, we’ve used the exact same concepts that we used in expressing (1). The only difference is syntax. Here, the is represented as a multi-line command and is represented as ||. The other addition that we’ve made is to add an imperative, namely, print. This is one of the main differences between pure logic and programming: Computers do things in the physical world; pure logic does not.

The advantage of a background in philosophy is that you are already able to translate colloquial statements into logical formulae, just as programmers do. As a student of philosophy, moreover, this kind of knowledge allows you to craft and criticize arguments. As a computer programmer, the same knowledge enables you to see several ways of building the same program; and the ability to see the far reaching, logical entailments of your claims often makes it easier to spot bugs.

Analytical Similarities

The way that one evaluates a philosophical position is surprisingly similar to the way a computer processor evaluates code. Both involve analytically determining what follows from what.

Consider the position of utilitarianism. According to this view,

What is good is what maximizes happiness. The virtuous choice is the one that maximizes happiness.

In order to evaluate this position, we need to consider its implications and ask if they are sound. If there is an unsound consequence, then the following logic will undermine the position. Schematically, where ¬ is 'not,' A is "What is good is what maximizes happiness," and B is some unsound consequence, we have:

Assume:            A → B
Assume:            A
By Modus Ponens:   B
Assume:           ¬B
By Modus Tollens: ¬A ∎

This argument shows that, if there is some unsound consequence B, then A cannot be true.

In computer programming, the style of thinking involved is remarkably similar. Suppose you are writing a computer program that plays chess and you need to write a function that checks the board to see if there is a checkmate. You might start out with something like this:

def checkmate?
  if current_player.king.is_capturable?
    return true
  else
    return false
  end
end

This function checks to see if the current player's king can be captured. If it is, it returns true; otherwise, it returns false.

In order to evaluate this code, we perform a mental stack trace, that is, we imagine, as best we can, how the computer will process the code that we have just written. If our code is successful, all checkmated boards will be identified; otherwise, they won't be. Similarly, in the philosophical case, we evaluate our position by imagining all possible entailments. If our position is successful, we'll agree with all of the entailments; otherwise, we won't.

As it turns out, both of these cases are deficient. The problem with utilitarianism (at least in its current crude form) is that it ends up implying that slavery is not wrong, on the grounds that the intense suffering of the few may result in the highest levels of happiness for the many. This is unacceptable, so the argument above obtains. Analogously, the checkmate function will misidentify checked boards as being checkmated. The fact that the current player's king can be captured just means that she must move her king out of check, not that its capture is inevitable.

The point, however, is that both computer programming and philosophy involve analytically determining the consequences of one's statements; and both necessitate a certain creativity in subsequently modifying one's statements accordingly.

Historical Development

Besides these concrete similarities, academic philosophy played a fascinating role in the historical development of computing.

The conceptual work that made the advent of modern computing possible took place in the early seventeen hundreds, when the philosopher and mathematician G.W.Leibniz developed an algebraic logic. In this logic, ones and zeros represent true or false states. The reason this is significant is that Leibniz’s ones and zeros would later be rendered mechanically as on or off switches in the first primitive computers. Although it took several hundred more years and the work of countless other philosophers, mathematicians, and engineers to develop our modern notion of computing, its conceptual foundation can be traced back to Leibniz’s philosophical logic.

The influence of philosophy on computer science did not stop at Leibniz, however. In the early nineteen hundreds, the philosophers Bertrand Russell and Alfred North Whitehead wrote Principia Mathematica, a nearly 2,000 page book written largely in formal logic that aimed to couch all of mathematics in terms of logic. Although the primary aim of this book was later demonstrated to be impossible by Gödel, the Principia also set out a theory of types that was later put to use by computer scientists. According to Constable’s brief history, Russell and Whitehead’s theory of types

…provided a basis for both a precise semantics and elegant programming logics. It was in this context that computer scientists and logicians created the type theories that are deeply connected to Principia Mathematica and serve now as comprehensive logical accounts of computing, computational mathematics, and programming logics (Constable 3).

Today the influence of philosophy on computer science continues, especially in the ares of artificial intelligence, the representation of information, language, and other things.