Mongodb with Chef and Terraform
I want to launch mongodb sharded cluster with chef and terraform on AWS.. show me steps!
Let's name our mongos cluster as "Roads".
Assumptions: There's a private VPC in AWS with subnets in multiple AZs. There's a bastion host launched in same VPC for ssh connectivity, as mentioned here: GHOST_URL/launch-bastion/
Mongo components:
Mongos: Server responsible for routing requests, balancing chunks in shards and entry point for all management.
Config servers: A replica set of primary-secondary-secondary nodes that store data of chunk-shard mapping in addition to other metadata required for cluster to function.
Shard servers: Replica sets (each set with primary-secondary-arbiter/secondary). They hold the relevant data and respond to queries from mongos.
Chef
For the sake of this post, I assume you've a chef server running and can upload cookbooks and roles.
For mongodb, I'm using cookbook from here: https://github.com/sunggun-yu/chef-mongodb3
In the current tutorial, I set mongo version in attributes/default.rb to 3.4.21. Also, add this line in attributes/default.rb file:
# sharding options: http://docs.mongodb.org/manual/reference/configuration-options/#storage-options
default['mongodb3']['config']['mongod']['sharding']['clusterRole'] = nil # shardsvr or configsvr
Once I've uploaded cookbooks to my server with knife cookbook upload <cookbook_name>, I shall create the following roles in my chef.
- For config servers,
roadsMongoConfig.json
{
"name": "roads_config",
"description": "Installs mongo config servers for roads cluster",
"default_attributes": {
"mongodb3": {
"config": {
"mongod": {
"replication": {
"replSetName": "roads-mongo-config"
},
"storage" : {
"dbPath" : "/vol/mongo"
},
"sharding": {
"clusterRole" : "configsvr"
}
}
}
}
},
"run_list": [
"recipe[mongodb3::package_repo]",
"recipe[mongodb3::default]"
]
}
Using the above role, I'll launch three servers in different AZs or subnets and make DNS entries like roads-mongo-config-1, roads-mongo-config-2, roads-mongo-config-3.
- For mongos server,
roadsMongos.json
{
"name": "roads_mongos",
"description": "Role for roads mongos",
"default_attributes":
{
"mongodb3" : {
"config" : {
"mongos" : {
"sharding" : {
"configDB" : "roads-mongo-config/roads-mongo-config-1:27017,roads-mongo-config-2:27017,roads-mongo-config-3:27017"
}
}
}
}
},
"run_list": [
"recipe[mongodb3::package_repo]",
"recipe[mongodb3::mongos]"
]
}
- For shards, I create two roles.
roadsArbiter-1.json:
{
"name": "roads_arbiter1",
"description": "Creates arbiter for roads set 1",
"default_attributes": {
"mongodb3": {
"config": {
"mongod": {
"replication": {
"replSetName": "roads-replica-1"
},
"storage" : {
"journal": {
"enabled": "false"
}
}
}
}
}
},
"run_list": [
"recipe[mongodb3::package_repo]",
"recipe[mongodb3::default]"
]
}
roadsReplica-1.json
{
"name": "roads_replica1",
"description": "sets up roads replica set",
"default_attributes": {
"mongodb3": {
"config": {
"mongod": {
"replication": {
"replSetName": "roads-replica-1"
},
"storage" : {
"dbPath" : "/vol/mongo"
},
"sharding" : {
"clusterRole" : "shardsvr"
}
}
}
}
},
"run_list": [
"recipe[mongodb3::package_repo]",
"recipe[mongodb3::default]"
]
}
Upload all roles to chef server with knife role from file <filename>
- Once the roles and cookbooks are uploaded, launch servers in AWS. If you choose not to do with terraform, you can launch them manually and bootstrap servers with:
knife bootstrap <private_ip> --ssh-user ec2-user --sudo --identity-file ~/.ssh/<private_key> -N <instance-id> --run-list 'role[roads_config]'
Similarly, for arbtier and replicaset, replace roles with respective ones.
Terraforming
These terraform scripts are dirty and incomplete. I'm still learning terraform to full extent.
Here's one for launching 3 config servers.
pending here: this terraform script isn't multi-AZ
One can use similar script to launch replica sets, arbiters by replacing role in provisioner.
resource "aws_instance" "roads_mongo_config" {
count = 3
ami = "ami-00e782930f1c3dbc7"
instance_type = "t2.micro"
associate_public_ip_address = false
vpc_security_group_ids = ["<security_group_id>"]
key_name = "<my_private_key>"
subnet_id = <subnet_to_be_used>
provisioner "chef" {
environment = "_default"
run_list = ["role[roads_config]"]
node_name = "${aws_instance.roads_mongo_config[count.index].id}"
server_url = "https://chef-server-url/organizations/orgName"
recreate_client = true
user_name = "<your_user_name>"
user_key = "${file("/Path/to/your_private_key")}"
ssl_verify_mode = ":verify_none"
client_options = ["chef_license 'accept'"]
version = "12.21.3"
connection {
type = "ssh"
host = "${aws_instance.roads_mongo_config[count.index].private_ip}"
user = "ec2-user"
private_key = "${file("/Path/to/my_private_key")}"
bastion_host = "bastion-server-url"
bastion_host_key = "<copy-bastion-host-key from ~/.ssh/known_hosts file>"
}
}
Connecting the dots
While the above scripts launched independent mongodb instances with appropriate roles, the weaving of network is still pending. We'll do it manually for now as terraform script is dirty.
- Make DNS entries for all the above servers (3 config servers, 1 mongos, 1 shard with primary, secondary, arbiter)
roads-config-1.example.vpc, roads-config-2.example.vpc, roads-config-3.example.vpc, roads-mongos.example.vpc, roads-replica-1.example.vpc, roads-replica-2.example.vpc, roads-arbiter-1.example.vpc - Login to config server, do
rs.initiate()followed by doingrs.add("config-1")andrs.add("config-2") - Login to replica set, do
rs.initiate()and addrs.add("replica-1")andrs.add(replica-2)
