Custom Ember Data the easy way

One of the best features of EmberJS is Ember Data. One of the most infuriating features of EmberJS is Ember Data. If you live in an ideal world, and your backend data format of choice is JSON:API, then you're on the happy path, and the world is your oyster.

But not many of us work on greenfields environments where JSON:API is the planned choice. The truth is that most applications have a history and therefore their own custom JSON format to suit.

It makes sense that EmberJS can support any type of data format using the RESTSerializer. The problem is finding a nice easy guide that will walk the entire way through with actual examples of massaging some custom data.

Hopefully this guide help because I've wallowed in a pit here myself.

I want to state a couple of things up front to keep in mind as we work through to help:

1. RESTAdapter gives us the opportunity to massage data a little bit to fit our expected model, and to help Ember Data convert our data into JSON:API format behind the scenes (perhaps we're not using JSON:API, but EmberJS is internally).
2. Understanding the use of plurals versus singular in our code and data is important when massaging the data itself. This is the area that I think is the biggest challenge when starting out.

A couple of extra points before we get going. We're going to start with dummy data just to get something on the screen. While it's usual to give some sort of API to use, at times it's simpler to experiment offline and see a result with some basic JSON.

The second point is that Mirage is sometimes recommended to start out mocking for Ember Data. But then, if knew what Mirage was or had experience with it, you wouldn't be reading this article. So let's skip both of those things for now.

As we go, we'll improve each step until we're got something more like a real application.

Create a new project with:

ember new custom-data-workshop

Open the custom-data-workshop using VSCode, then open a new terminal.

I like to use a different port instead of the default 4200. You might be using the default for another EmberJS application. Therefore change package.json's start script as:

"start": "ember serve --port 4300",

Now run as usual with:

npm start

Open up in your browser to run your app. Finally, replace in app/templates/application.hbs:

<h1>Data</h1>
 
<p>Nothing</p>
 
{{outlet}}

We're good to go.

Oh just before that, would be nice to improve the styles just a little. Add this to the app/styles/app.css:

body {
    padding: 10px;
	font-family: sans-serif;
}
 
ul {
    border: 1px solid #BBBBBB;
    width: 260px;
    border-radius: 4px;
    padding: 7px 7px 7px 40px;
}
 
span {
    display: inline-block;
    background-color: black;
    border-radius: 25px;
    color: white;
    width: 28px;
	height: 28px;
	line-height: 28px;
    text-align: center;
}

We'll start by returning some hard-coded JSON from a route for the default template to display. We already have an application.hbs template, so we will make an application.js route in which to load our data with:

ember g route application 

Say no to overwriting the app\templates\application.hbs file.

Add a new model hook to create and return some JSON that will be used to display information about a game:

import Route from '@ember/routing/route';
 
export default class ApplicationRoute extends Route {
 
    model (){
        const game = { 
            "id": 1,
            "title": "The Secret Of Monkey Island",
            "year": "1990"
        }
 
        return game;
    }
}

Now to use those data values in our default application.hbs template. Replace the content with:

<h1>Data</h1>
 
<ul>
    <li>{{@model.title}} <span>{{@model.id}}</span></li>
    <li>{{@model.year}}</li>
</ul>

Result:

Great! First result with some data showing on screen. On the downside, we are not using Ember Data just yet!

It is also good to see failure. Let's comment out the return in our model hook:

    //return game;

Notice we get nothing rendered. We need to return the data object back otherwise the model hook considers it has received nothing. Best to uncomment that line again before moving.

To start feeling more like a regular application we should Fetch our data rather than hard-code it within a route.

Create a JSON file in the public folder called game.json.

Take the data from the model hook and copy it into game.json as:

{ 
    "id": 1,
    "title": "The Secret Of Monkey Island",
    "year": "1990"
}

Change your application model hook to load the data from the file:

import Route from '@ember/routing/route';
 
export default class ApplicationRoute extends Route {
 
    model (){
        let promise = fetch('game.json');
        return promise.then(function(response){
            return response.json();
        });
    }
}

Alternatively, you could use async/await syntax like this:

import Route from '@ember/routing/route';
 
export default class ApplicationRoute extends Route {
 
    async model (){
 
        let response = await fetch('game.json');
        let data = await response.json();
        return data;
 
    }
}

Slightly easier on the eyes.

But either way, the data should be loading into the page from the file using fetch. Nice one.

For the sake of showing that this step can be difficult to get your resolved Promises right, here is some code that does not work for returning the data from a model hook:

    model (){
        let promise = fetch('game.json');
        promise.then(function(response){
            return response.json()
            .then(function(game){
                return game;
            });
        });
        return promise;
    }

What about a similar API that is remote? Let's see how we go with that. At this point we need be online. I'll provide a RESTful test API at https://retrogames.waynejohnson.net/api

You can call the entire list of games with: https://retrogames.waynejohnson.net/api/games Or you can call a single game with: https://retrogames.waynejohnson.net/api/games/{id}

First port of call, lets swap out the fetch call so that it makes a call to the API to get one record out of many:

import Route from '@ember/routing/route';
 
export default class ApplicationRoute extends Route {
 
    model (){
        let promise = fetch('https://retrogames.waynejohnson.net/api/games/7');
        return promise.then(function(response){
            return response.json();
        });
    }
}

And that works too. But notice there is a subtle problem that will catch us up later. It's the id.

{
  "gameId": 7,
  "title": "Beneath a Steel Sky",
  "year": 1994
}

Notice that the ID is no longer displaying on our page. This is because we are expecting a property called id but our API is returning it as gameId.

We could work around the problem by altering the JSON with:

    model (){
        let promise = fetch('https://retrogames.waynejohnson.net/api/games/7');
        return promise.then(function(response){
            return response.json().then(function(json){
                json.id = json.gameId;
                return json;
            });
        });
    }

…but… as we get into collections (more than just a single record) and more complex results and patching, this is going to get very messy quickly. It's probably time to utilise Ember Data and have it do all the heavy lifting for us.

Now for the meatier stuff: actually using Ember Data to call the API for us and to bring in the data. For this, we'll start to use models and calls from our routes.

Begin by defining the model, which is the cornerstone of helping Ember Data help us. Create a model file with:

ember g model game

This will create the game.js model file in the app/models folder. Change the content to be:

import Model, { attr } from '@ember-data/model';
 
export default class GameModel extends Model {
    @attr('string') title;
    @attr('number') year;
}

You might observe that we don't have an id property. You should not specify an id in your Ember data models. More on this later.

Now because Ember Data prefers to consume JSON:API, we are going to need a custom Adapter to handle where the data is sourced from, and a custom Serializer to handle any massaging that we might need to fix up the JSON data we load in.

Make a new Adapter for our game model with:

ember g adapter game

By default your new adapter is going to be a JSONAPIAdapter but that's obviously no good for our customised purposes. So let's change it into a RESTAdapter and add the host and namespace properties to point to our own API:

import RESTAdapter from '@ember-data/adapter/rest';
 
export default class GameAdapter extends RESTAdapter {
    host = 'https://retrogames.waynejohnson.net';
    namespace = "api"
}

Let's now make our first attempt to load in data from the model hook in the app/routes/application.js route file. We need to bring in the store service by adding it to our includes, and also declare it inside the Route class:

import Route from '@ember/routing/route';
import { inject as service } from '@ember/service';
 
export default class ApplicationRoute extends Route {
    @service store;
...

Replace the model() function with:

import Route from '@ember/routing/route';
import { inject as service } from '@ember/service';
 
export default class ApplicationRoute extends Route {
    @service store;
 
    model (){
        let data = this.store.findRecord('game', 3);
        return data;
    }
}

There's some good news and bad news. If you check the Network tab, you'll notice there was a successful call to our API to retrieve back a record:

200 GET 3	json	https://retrogames.waynejohnson.net/api/games/3

{
	"gameId": 3,
	"title": "Twinsen's Odyssey",
	"year": 1997
}

Fantastic. That means that the adapter is doing it's job. If you want to know how the adapter works it's magic: it takes the host and namespace properties from the adapter settings, and the game from our this.store.findRecord(“game”, 3);, and pluralises game into games. Therefore:

https://retrogames.waynejohnson.net + api + game + plural + index = https://retrogames.waynejohnson.net/api/games/3

Now the bad news… the page crashed with a blank page, and you will get an error in the console:

Error while processing route: index payload.data is null

This is an indicator that we don't have a serializer for Ember Data that understands anything about our JSON structure. So we need to make a Serializer.

We are going to spend some time here because this is the part that can be a complete mongrel to get right. Let's get started by making a serializer file with:

ember g serializer game

Now with an empty serializer, we will get the following error because the serializer we created defaults to a JSONAPISerializer:

Error while processing route: index Assertion Failed: normalizeResponse must return a valid JSON API document

Let's turn it into a RESTAdapter:

import RESTSerializer from '@ember-data/serializer/rest';
 
export default class GameSerializer extends RESTSerializer {
}

Hmmm.. things appear to be worse:

WARNING: Encountered "gameId" in payload, but no model was found for model name "game-id"
WARNING: Encountered "title" in payload, but no model was found for model name "title"
WARNING: Encountered "year" in payload, but no model was found for model name "year" 
Error while processing route: index payload.data is null

Or

Error: Assertion Failed: The 'findRecord' request for game:3 resolved indicating success but contained no primary data. To indicate a 404 not found you should either reject the promise returned by the adapter's findRecord method or throw a NotFoundError.

So far, Ember Data doesn't know how to convert our incoming data calls into something that suits our model.

A normalizeFindRecordResponse function is required in our serializer. Let's add a plain one that does little more than send our payload to super (super is in charge of converting our data into JSON:API behind the scenes internally):

import RESTSerializer from '@ember-data/serializer/rest';
 
export default class GameSerializer extends RESTSerializer {
    normalizeFindRecordResponse(store, type, payload, id) {
        return super.normalizeFindRecordResponse(store, type, payload, id);
    }
}

Same error. Seems like we haven't progressed. Why don't we seem to getting anywhere?

Let's take another look at the JSON that is coming back over the wire:

{
	"gameId": 3,
	"title": "Twinsen's Odyssey",
	"year": 1997
}

Notice that the data has no root node. It's just an object. The serializer doesn't know what to make of it or which model it needs to map to. It's true we did tell the GameAdapter that we wanted a game object, but unless the server can show that it has returned one or more game data items, Ember will be upset. The server may have returned an error message, or worse, junk response from a crashed service. We wouldn't want Ember Data to proceed modelling on that “data”.

The simple fix is to wrap the incoming data item with a game node. Let's update the normalizer function to be:

import RESTSerializer from '@ember-data/serializer/rest';
 
export default class GameSerializer extends RESTSerializer {
    normalizeFindRecordResponse(store, type, payload, id) {
 
        const alteredPayload = {
            game: { payload }
        }
        return super.normalizeFindRecordResponse(store, type, alteredPayload, id);
    }
}

Hey we've progressed! A different error this time:

Error while processing route: index Assertion Failed: You must include an 'id' for game in an object passed to 'push'

And you might be able to guess the reason for this? Remember how our JSON that was returned from the API had a gameId, but our template application.hbs is expecting it to be id? So we need to map the ID property. We can massage this within the normalizeFindRecordResponse function:

import RESTSerializer from '@ember-data/serializer/rest';
 
export default class GameSerializer extends RESTSerializer {
    normalizeFindRecordResponse(store, type, payload, id) {
        const alteredPayload = {
            game: {
                id: payload.gameId,
                title: payload.title,
                year: payload.year
            }
        }
        return super.normalizeFindRecordResponse(store, type, alteredPayload, id);
    }
}

Woah! Hey! We got rendered output onto our page! Fantastic stuff. This means we have a complete working flow for our application for single records.

Now that we have been successful with a single data record, and we understand that pluralisation and JSON root nodes can make all the difference… let's move on to working with multiple data records.

We'll need to change both our template page, and the route to expect multiple records. Let's start with the routes/application.js:

import Route from '@ember/routing/route';
import { inject as service } from '@ember/service';
 
export default class ApplicationRoute extends Route {
    @service store;
 
    model (){
        let data = this.store.findAll('game');
        return data;
    }
}

Oooh, a findAll call. You'll notice we don't have an error, but even still: our data doesn't populate on the page. However… the Network tab will show that the call successully loaded all our records because the GameAdapter constructed a URL like this: https://retrogames.waynejohnson.net/api/games.

How? Again, the adapter takes the host and namespace properties, and the game from our this.store.findAll('game');, and pluralises game into games. Therefore:

https://retrogames.waynejohnson.net + api + game + plural = https://retrogames.waynejohnson.net/api/games

The trick to all this Ember Data stuff is to be aware of what convention needs to be a single and what needs to be a plural. After that, everything is smooth sailing.

There are a bunch of warnings in the developer console like this:

WARNING: Encountered "0" in payload, but no model was found for model name "0".

This means Ember Data is trying to parse each record, but the RESTSerializer doesn't know what to do with a collection of game objects coming in. We better teach it by supplying a new normalizeFindAllResponse function into the RESTSerializer:

	normalizeFindAllResponse(store, type, payload) {
		return super.normalizeFindAllResponse(store, type, payload);
	}

The same warnings occur because we did nothing to massage in the incoming rows. Let's try wrapping the collection of JSON objects with a root node:

    normalizeFindAllResponse(store, type, payload) {
        const newPayload = {
            games: payload
        }
        return super.normalizeFindAllResponse(store, type, newPayload);
    }

Notice the plural games node for a collection? This is good. The warnings are gone, however and we're back to an error:

Error while processing route: index Assertion Failed: You must include an 'id' for game in an object passed to 'push'

Looks like we need to ensure we massage the ids for each row coming in. Change the RESTSerializer to do this with:

    normalizeFindAllResponse(store, type, payload) {
 
        const newPayload = payload.map( g => {
            return {
                id: g.gameId,
                title: g.title,
                year: g.year
            }
        } );
 
        const wrappedPayload = {
            games: newPayload
        }
 
        return super.normalizeFindAllResponse(store, type, wrappedPayload);
    }

That's fixed the data but we can't display it just yet.

That code was a little long too, and could be optimised. But you can easily follow what's happening here. There is an easier way to do this for collections: you can add a primaryKey property to the RESTSerializer:

import RESTSerializer from '@ember-data/serializer/rest';
 
export default class GameSerializer extends RESTSerializer {
    primaryKey = 'gameId'; //<------ note!!
 
    normalizeFindRecordResponse(store, type, payload, id) {
        const alteredPayload = {
            game: {
                gameId: payload.gameId, // <- Set prop to gameId again. primaryKey will sort this out for us!
                title: payload.title,
                year: payload.year
            }
        }
        return super.normalizeFindRecordResponse(store, type, alteredPayload, id);
    }
 
    normalizeFindAllResponse(store, type, payload) {
 
        const wrappedPayload = {
            games: payload
        }
 
        return super.normalizeFindAllResponse(store, type, wrappedPayload);
    }    
}

The errors and warnings are gone, and so therefore all that is left is to update our application.hbs template to ensure each record is rendered:

<h1>Data</h1>
{{#each @model as |game|}}
<ul>
    <li>{{game.title}} <span>{{game.id}}</span></li>
    <li>{{game.year}}</li>
</ul>
{{/each}}

And viola! We have a collection of games rendering in a list. And I think we'll stop there, that's plenty enough. But there's much more to learn, such as relationships, also POSTs, PUTs and PATCHes weren't covered. So have fun experimenting!

Thank you to Latha.K & Andrew.W for proofing and tips, and Julien.P for wallowing with me with every possible permutation of Promises.

1. https://emberigniter.com/fit-any-backend-into-ember-custom-adapters-serializers
2. https://javascript.info/promise-chaining
3. https://guides.emberjs.com/release/routing/specifying-a-routes-model
4. https://emberigniter.com/render-promise-before-it-resolves