Skip to content

Setup MongoDB Sharding Cluster

Recently, I have success in setup MongoDB cluster in 3 local machine for development and test purpose. And this post will share with you how did I do that.

To follow this post, you will need:

  • Minimum of 3 Linux server (or you can replace with 3 Linux computer with local LAN setup)
  • A little experience with Linux command and mongo shell command.

Getting Started

Before you go there for setting up MongoDB cluster, I’m assume that you already got the basic knowledge with MongoDB (If you’re not, go to official site and get it yourself).

First of all, you have to understand about scaling up your server and why you have to do that. A very simple reason is your data is group up quickly and your server can not handle it (and google it for more). There are two broad ways of scaling up your server including:

  • Vertical scaling: simply understand it as you will add more and more resource to your server to handle lager data.
  • Horizontal scaling: instead of adding more resource to a single server, you will add more server to handle the same job. This is a preferred way for scaling up server and this post will present how to do horizontal scaling your server.

Ok, to setup a horizontal scaling you need to have your cluster first (server machine or local machine as my case). For development and test purpose only, I have used 3 local Linux machine with LAN connected (borrow your friends or your co-workers if you don’t have it 🙂 ). And first step, of course, you have to install MongoDB on all your server, this doc will help you how to do it (if you install MongoDB for server, please find another link). Please install all of it on the same version to make sure that you will not get any problem later. In my case, I have installed MongoDB version 3.4.17 on my 3 Ubuntu machines.

Cluster Architecture

Before starting to install the system, let’s have a quick overview of the architecture and the components we will setup.

Image result for mongodb sharding

  • Query Router: The mongos daemon acts as an interface between the client application and the cluster shards. Since data is distributed among multiple servers, queries need to be routed to the shard where a given piece of information is stored. The query router is run on the application server. In this guide, we’ll only be using one query router, although you should put one on each application server in your cluster.
  • Config server: This stores metadata and configuration settings for the rest of the cluster. In this post, I’ll use one config server for simplicity but in production environments, this should be a replica set of at least three server.
  • Shard: A shard is simply a database server that holds a portion of your data. Items in the database are divided among shards either by range or hashing, which we’ll explain later in this guide. For simplicity, I’ll use one single-server shard.

Configure hosts File

Fire up your terminal in each server and change your /ect/hosts  file:

Replace the IP addresses to yours

Setup MongoDB Authentication

In this section you’ll create a key file that will be used to secure authentication between your servers. While in this example you’ll be using a key file generated with openssl, MongoDB recommends using an X.509 certificate to secure connections between production systems.

Create an Administrative User

On the server you intend to use as config server, log in to the mongo  shell:

Connect to admin  database:

Create an administrative user with root privileges:

Generate a Key file

  1. Go back to your terminal by quit()  command. Now type this command to generate your key file:
  2. The generated mongo-keyfile will be use as the security key to communicate between your servers. So that, copy your key file to all the servers and the rest steps in this section must be perform on each server. Create the /opt/mongo  directory to store your key file:
  3. Now move your mongo-keyfile  to /opt/mongo  directory and assign it the correct permissions:
  4. Next step, change the ownership of your key file to mongodb  user (or mongod  user on CentOS):
  5. Let’s enable security section for your server can communicate correctly with each other. Open  /ect/mongod.conf  file and change the following value (remember the space, without it you will face the trouble):

    This step should be skipped on the router server, because we don’t use  mongod  on router, we use  mongos  instead.

Setup Config Server

This step will setup our config server.

  1. First step, change the port  and the bindIp  address of the config server in /ect/mongod.conf  file. The bindIp  must match your config server’s IP:
  2. Uncomment the replication  section and add the following value to create replica set for your server:

    rs0  is just the name of your replica set, you can change it to what ever you want and remember it.
  3. Enable sharding by uncomment the sharding  section and set the following value:
  4. Now, restart your mongod  service to apply changes have been made:
  5. Check if your service is working correctly (ACTIVE status), if not then double check if you have missed any step or any typo mistake (Once again, remember the space character in your config file. I have forgot it then spent a lot of time to figure it out):
  6. If your mongod  service is working correctly, then connect to it!
  7. Initialize the replica set:

    The result should be:  { "ok" : 1 } . If not, check the message then troubleshooting 🙂
  8. Let’s check your replica set status:

    if your config is correct then you will see your server as a member of replica set in the result.

Now you have done setup for config server, let’s move on the query router server.

Setup Query Router Server

Because router server is use the mongos  service instead of mongod , so that we must create a config file for mongos service and enable it. Fire up your query router server terminal and follow the steps below.

  1. Create new config file called /ect/mongos.conf  and add the following values:

    Replace your bindIp  and the configDB  to yours.
  2. Create a new systemd unit file for mongos called /lib/systemd/system/mongos.service  with the following values:
  3. You’re done, let’s stop the mongod  service and then enable and start the mongos  service.
  4. Check if your mongos service is ACTIVE now:

    If your service is active and running now, then move to the next big step! Setup your shard

Setup Shard Server

  1. Fire up your shard server and modify the /etc/mongod.conf  file:

    Replace the IP with yours, and then enable the security section:

    Enable sharding section too:
  2. Now connect to your query router:
  3. If you cannot log in to your router server, then check the log file for detail error. The log file of mongod  service is located at /var/log/mongodb/mongod.log , and the log file of mongos  service is located at /var/log/mongodb/mongos.log
  4. Now you successfully connect to your router server, let’s register a shard

     

All done. You have successfully setup your first simple MongoDB cluster with 3 local machine! Let’s play around with your cluster and make a step into big data world!

In the next post, I will share about how do I play around with mongodb sharding cluster. Please check it out!

Published inBig DataMongoDB

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *