Mongo Operations

This is with assumption of you're using either AWS or Google Cloud. Also, this isn't elaborate purposely.

Installation on Centos/Amazonlinux:

• Create a new repo file: sudo vi /etc/yum.repos.d/mongodb-org.repo
• Drop in following contents in the file:

[mongodb-org-3.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.2/x86_64/
enabled=1
fastestmirror_enabled=0
gpgcheck=0
sslverify=false

• sudo yum install mongodb-org -y
• sudo systemctl start mongod

Run mongo in the shell, clear out all warnings.

Filesystem considerations for performance:

• Use XFS if you're going with wiredtiger. For other storage engines, please refer to mongodb documentation for recommended methods.
• Ensure fstab entries are intact.
• Set ulimit to 64000 (mongodb installation should automatically do it).

Replica Set:

• Keep oplog considerably high. 50gb is good for most cases.
• Distribute replica sets across Availability Zones.
• Make sure to use DNS. That might require you to do a rs.stepDown() at least once in initial setup.
• Mount disks instead of using root disks. If you've some experience, you may try to add multiple disks and separate it out into journal and data.
• Dry run init.d scripts.

Sharded Cluster:

• Configure mongos and config servers (in a replica set).
• Make sure to add init.d script for mongos.
• sh.status() gives a sanity check.
• For performance considerations, you can enable a balancer window to tell mongos to do shard allocation only when the cluster is relatively free.
• You can add multiple mongos.

Potential issues and fixes:

• Replica set not syncing to primary and being stuck in recovery state refers to a situation where your failover failed. To fix the issue, take backup of primary and restore to a member. Add it back to replica set and let it sync.
• Monitor the number of requests and asserts in mongo. Asserts can be errors or warnings.
• Based on index size and read/write ratio, you can decide on the right sizing of mongodb instance. Keep room for scaling up. While in theory mongodb can scale horizontally by adding more shards, the practical experience says otherwise. You need to plan ahead to add a shard and when you discover the issue, you've better chances of improvement with immediate vertical scaling. Having secondary-primary in sync makes it easy to scale up as well.
• Balancer can block operations. When sec-pri goes out of sync for whatsoever reason, it's best to stop mongos balancing immediately. Since the option writeConcern is by default set to ensure writes are replicated before accepting new writes, it's important to stop balancing.