Creating a Real Time Chat API with Node, Express, Socket.io, and MongoDB

Datetime:2016-08-22 22:30:17          Topic:          Share

In my last post, I broke down creating an API with JWT authentication. In this post, we are going to set up the server-side work for a real time chat system. We will be using Node, Express, Socket.io, and MongoDB to make it happen.

We are going to build a threaded chat/instant messaging system, much like Facebook Messenger or Google Hangouts. Every message sent will belong to a conversation.

Whether you are following along from where we left off or starting fresh, you can refer to this repository for code if you get lost.

We will start by creating the conversation schema. We will be working exclusively in the server directory today. Create and open models/conversation.js. Add the following:

const mongoose = require('mongoose'),  
      Schema = mongoose.Schema;

// Schema defines how chat messages will be stored in MongoDB
const ConversationSchema = new Schema({  
  participants: [{ type: Schema.Types.ObjectId, ref: 'User'}],
});

module.exports = mongoose.model('Conversation', ConversationSchema);

Our conversation schema will essentially just hold the participants of the conversation and generate an ID for the conversation. We are specifying that our participants field will contain an array of IDs generated by MongoDB. The 'ref' you see is setting up a Mongoose populate, which sort of simulates a join from MySQL, if you're familiar with that. There are no real joins in MongoDB, to clarify. Read more here .

Next, we will create and open models/message.js. Add the following:

const mongoose = require('mongoose'),  
      Schema = mongoose.Schema;

const MessageSchema = new Schema({  
  conversationId: {
    type: Schema.Types.ObjectId,
    required: true
  },
  body: {
    type: String,
    required: true
  },
  author: {
    type: Schema.Types.ObjectId,
    ref: 'User'
  }
},
{
  timestamps: true // Saves createdAt and updatedAt as dates. createdAt will be our timestamp.
});

module.exports = mongoose.model('Message', MessageSchema);

MongoDB would have allowed us to simply embed the messages as an array in the conversation collection, but that is an anti-pattern, as the array could grow without bounds. This would affect performance. Read more about that here .

With the schemas created, we can start creating our API endpoints, which will allow us to send, delete, edit, and view messages. We will start by creating our controllers. Create and open controllers/chat.js. The first route we create will be our get route to view a list of conversations. If you open up your inbox on Facebook Messenger, you can see a list of conversations you are in. You can also see an excerpt of each conversation you are involved in. The following is how I solved this:

"use strict"
const Conversation = require('../models/conversation'),  
      Message = require('../models/message'),
      User = require('../models/user');

exports.getConversations = function(req, res, next) {  
  // Only return one message from each conversation to display as snippet
  Conversation.find({ participants: req.user._id })
    .select('_id')
    .exec(function(err, conversations) {
      if (err) {
        res.send({ error: err });
        return next(err);
      }

      // Set up empty array to hold conversations + most recent message
      let fullConversations = [];
      conversations.forEach(function(conversation) {
        Message.find({ 'conversationId': conversation._id })
          .sort('-createdAt')
          .limit(1)
          .populate({
            path: "author",
            select: "profile.firstName profile.lastName"
          })
          .exec(function(err, message) {
            if (err) {
              res.send({ error: err });
              return next(err);
            }
            fullConversations.push(message);
            if(fullConversations.length === conversations.length) {
              return res.status(200).json({ conversations: fullConversations });
            }
          });
      });
  });
}

First, we search our conversation collection for conversations in which our authenticated user is a participant. Next, we create an empty array, fullConversations, which we will push the results of our next query onto. Our next query takes the results of our first one (which are stored in an array), and searches the message collection for any messages which are a part of the given conversation. We sort those by most recent, limit the results to one, and populate the author path. Finally, we push those results to our fullConversations array, and when the array is equal in size to the length of the conversations array, we are done, so we respond to the request with the array of conversations.

Note: This doesn't feel optimal to me and is still a work in progress. I am also not sure this feature is entirely necessary. I will update if I find a better solution. If you have one in mind, please share in the comments.

Next, we will set up our controller to get all the messages in a single conversation. This will be pretty straight forward:

exports.getConversation = function(req, res, next) {  
  Message.find({ conversationId: req.params.conversationId })
    .select('createdAt body author')
    .sort('-createdAt')
    .populate({
      path: 'author',
      select: 'profile.firstName profile.lastName'
    })
    .exec(function(err, messages) {
      if (err) {
        res.send({ error: err });
        return next(err);
      }

      res.status(200).json({ conversation: messages });
    });
  }

Now we can move on to starting a new conversation. First we will write the code, then we will talk about it.

exports.newConversation = function(req, res, next) {  
  if(!req.params.recipient) {
    res.status(422).send({ error: 'Please choose a valid recipient for your message.' });
    return next();
  }

  if(!req.body.composedMessage) {
    res.status(422).send({ error: 'Please enter a message.' });
    return next();
  }

  const conversation = new Conversation({
    participants: [req.user._id, req.params.recipient]
  });

  conversation.save(function(err, newConversation) {
    if (err) {
      res.send({ error: err });
      return next(err);
    }

    const message = new Message({
      conversationId: newConversation._id,
      body: req.body.composedMessage,
      author: req.user._id
    });

    message.save(function(err, newMessage) {
      if (err) {
        res.send({ error: err });
        return next(err);
      }

      res.status(200).json({ message: 'Conversation started!', conversationId: conversation._id });
      return next();
    });
  });
}

First, we did some simple checking to make sure the required fields were sent with the request.

Next, we created a new conversation with the authenticated user and their specified recipient as the participants. Following that, we created and saved a new message, which we affiliated with the conversation we just created.

Our final chat controller will be for sending a reply, or adding a new message to an existing conversation.

exports.sendReply = function(req, res, next) {  
  const reply = new Message({
    conversationId: req.params.conversationId,
    body: req.body.composedMessage,
    author: req.user._id
  });

  reply.save(function(err, sentReply) {
    if (err) {
      res.send({ error: err });
      return next(err);
    }

    res.status(200).json({ message: 'Reply successfully sent!' });
    return(next);
  });
}

This section is pretty easy to follow, so we'll move on to setting up our routes in a moment. Note, if you want to add DELETE and PUT routes for your messages and/or conversations, this is where you will add the controllers. In my app, I don't want to allow users to delete their messages. That could look something like this:

// DELETE Route to Delete Conversation
exports.deleteConversation = function(req, res, next) {  
  Conversation.findOneAndRemove({
    $and : [
            { '_id': req.params.conversationId }, { 'participants': req.user._id }
           ]}, function(err) {
        if (err) {
          res.send({ error: err });
          return next(err);
        }

        res.status(200).json({ message: 'Conversation removed!' });
        return next();
  });
}

// PUT Route to Update Message
exports.updateMessage = function(req, res, next) {  
  Conversation.find({
    $and : [
            { '_id': req.params.messageId }, { 'author': req.user._id }
          ]}, function(err, message) {
        if (err) {
          res.send({ error: err});
          return next(err);
        }

        message.body = req.body.composedMessage;

        message.save(function (err, updatedMessage) {
          if (err) {
            res.send({ error: err });
            return next(err);
          }

          res.status(200).json({ message: 'Message updated!' });
          return next();
        });
  });
}

Open router.js and add your chat controller to your imports:

const AuthenticationController = require('./controllers/authentication'),  
      UserController = require('./controllers/user'),
      ChatController = require('./controllers/chat'),
      express = require('express'),
      passportService = require('./config/passport'),
      passport = require('passport');

We will need to set up a new route group for chat routes as well:

const apiRoutes = express.Router(),
        authRoutes = express.Router(),
        chatRoutes = express.Router();

In our last step prior to setting up Socket.io for real time capability, we will set up our new routes:

// Set chat routes as a subgroup/middleware to apiRoutes
  apiRoutes.use('/chat', chatRoutes);

  // View messages to and from authenticated user
  chatRoutes.get('/', requireAuth, ChatController.getConversations);

  // Retrieve single conversation
  chatRoutes.get('/:conversationId', requireAuth, ChatController.getConversation);

  // Send reply in conversation
  chatRoutes.post('/:conversationId', requireAuth, ChatController.sendReply);

  // Start new conversation
  chatRoutes.post('/new/:recipient', requireAuth, ChatController.newConversation);

Now you can test out your new API endpoints by creating a new conversation.

In Postman, you will need to login by sending a POST request to http://localhost:3000/api/auth/login with your email and password. Once you get a JWT response, copy that and put it in your Authorization headers. (Instructions for this can be found inthis tutorial toward the bottom). Change the body of the request to composedMessage: Test message here! then change your URL to /api/chat/new/ idOfUser , but enter the ID of one of the users saved in your database. Hit send and you should get a success message.

Feel free to also test your GET routes on all conversations (/api/chat) and then on the individual conversation (/api/chat/ idOfConversation )you just created. You can also add a reply with your other POST route (/api/chat/ idOfConversation ).

Here is what your list of conversations (a GET request to /api/chat) should look like:

Our final step is to get Socket.io set up on the server-side. This won't do much for us until we build the client-side, but it will be nice to be able to isolate the client-side when we wrap up with our API.

We will need to install Socket.io.

npm install --save socket.io

Create and open socketEvents.js in the server folder. Add the following:

exports = module.exports = function(io) {  
  // Set socket.io listeners.
  io.on('connection', (socket) => {
    //console.log('a user connected');

    // On conversation entry, join broadcast channel
    socket.on('enter conversation', (conversation) => {
      socket.join(conversation);
      // console.log('joined ' + conversation);
    });

    socket.on('leave conversation', (conversation) => {
      socket.leave(conversation);
      // console.log('left ' + conversation);
    })

    socket.on('new message', (conversation) => {
      io.sockets.in(conversation).emit('refresh messages', conversation);
      });

    socket.on('disconnect', () => {
      //console.log('user disconnected');
    });
  });
}

This won't mean a lot until we have built the client-side of this app, but basically, it's telling Socket to listen for and react to certain events, like a client connecting, leaving, or sending a message.

Now open up index.js and import the socketEvents file you just finished.

socketEvents = require('./socketEvents');

Next, look for where you console logged that your server is running on a port. Below that, add the following:

const io = require('socket.io').listen(server);

socketEvents(io);

That's it! Your server is all set up for a messaging app. Soon, we will be getting to the client-side, where we will build a real time front-end using React, Redux, and the Socket.io client-side library.

Any questions, comments, concerns, or pointers? I would love to hear them in the comments below.