Due to various reasons, mongodb secondary goes out of sync and might be dead/stuck in recovered state or replication lag grows beyond acceptable limit. Instructions 1. Make sure primary is up and running. Login to primary mongo, let’s say calendar-mongo-rs-1-1. 2. Check acceptable mongodb replication lag. You can

In a typical setup, we need a resilient way to track down the errors. To think of it, we're writing code and we're generating the error. Thus, we should know it even before a user reports. While tracking logical errors might be a technical challenge, it's not difficult to track

I want to launch mongodb sharded cluster with chef and terraform on AWS.. show me steps! Let's name our mongos cluster as "Roads". Assumptions: There's a private VPC in AWS with subnets in multiple AZs. There's a bastion host launched in same VPC for ssh connectivity, as mentioned here: GHOST_

Here's a quick guide on how to upgrade mongodb version on a sharded cluster hosted on AWS, AmazonLinux OS. This is done, manually, on a cluster with ~30 TB data with minimal downtime (less than a minute). Mongodb components: * mongos: Server responsible for routing requests, balancing chunks in shards and

Meant for self reference. This is a gist of launching an RDS mysql instance with terraform. It uses module from here: https://github.com/terraform-aws-modules/terraform-aws-rds resource "aws_security_group" "sample-mysql" { name = "sample-mysql" description = "Allows services to talk to sample mysql" vpc_id = "vpc-xxxx" ingress { from_port = 3306 to_port

Down to dirty blog on setting up VPC with terraform. This is to setup a new VPC in ap-south-1 with three public and three private subnets. It'll also take care of NAT Gateways, Internet Gateways, etc. What the script does/creates: * There is 1 vpc in ap-south-1 * There are 3

Assumption: You've a kubernetes cluster running on EKS, and all the kubeconfig, kubectl, aws authentication are handled. What's EFK? Elasticsearch: A search database with support for REST api queries. Fluentd: Data collector typically used for collecting logs in a unified manner. Kibana: Visualisation tool for Elasticsearch data. Data flow: pods

This is for self reference. You may find it useful too. Objective is to deploy a kafka + zookeeper cluster in a dirty way. This is going to be on AWS. Assumes you can launch an ec2 instance with latest amazonlinux AMI. Quick intro of kafka and zookeeper: * Kafka is an

Context switch of an engineer usually comes on the top when we're talking of productivity. The problem, unlike CPU, humans can't do context switches by simply putting things in RAM, etc. That's a limitation. Solutions usually work towards reducing context switches. Building enough layers of communication so that engineer gets

Opinion on how to handle configuration drift and identifying the problems.

This block redirects all http requests to https ``` server { listen         80; if ($http_x_forwarded_proto != "https") { rewrite ^(.*) https://$host$1 permanent; } }```

Webhooks explained with analogy

Problem Statement: Whenever mongos balancer is running, the cluster gets insanely slow.. Latencies are back to normalcy immediately if the balancer stopped. Mongos balancing process for reference: 1. Balancer takes a lock 2. Identify chunk to migrate. This is based on several criteria 3. Send command to "Source" shard. 4.

This is with assumption of you're using either AWS or Google Cloud. Also, this isn't elaborate purposely. Installation on Centos/Amazonlinux: * Create a new repo file: sudo vi /etc/yum.repos.d/mongodb-org.repo * Drop in following contents in the file: [mongodb-org-3.2] name=MongoDB Repository baseurl=https://repo.mongodb.

To discard an old set of kafka brokers and migrate them to newer set, 1. Get the existing list of topics. 2. Make a file topics-to-move.json with contents in the format: {"topics": [{"topic": "sampleTopic_A"}],"version":1} 3. In any of the kafka brokers, run the following command ./$PATH_

A few strategies for migrating Kafka to new cluster from classic to VPC: Strategy 1: Launch a new cluster (Zookeeper+Kafka brokers), point producers to new cluster and drain the older cluster once consumers are also pointed to new cluster. pros: * Existing cluster isn't impacted, and hence helps in easy

> How few services of hostedgraphite not hosted on AWS, got affected by AWS outage: https://blog.hostedgraphite.com/2018/03/01/spooky-action-at-a-distance-how-an-aws-outage-ate-our-load-balancer/ > How codespaces had to shutdown their business after loss of data: http://www.itwriting.com/blog/8498-resilience-is-not-backup-how-codespaces-com-lost-its-data-and-its-business.html > TravisCI truncated production database by human error: https://blog.travis-ci.

The following command can be used to pause mongodb ttl for 86400 seconds (1 day) db.adminCommand({setParameter:1, ttlMonitorSleepSecs: 86400}) The following to disable TTL and enable it later on: db.adminCommand({setParameter:1, ttlMonitorEnabled:false});

Uploading custom ssl certificate to AWS Cloudfront: aws iam upload-server-certificate --server-certificate-name --certificate-chain --private-key --certificate-body --path /cloudfront/