Engineering Most interesting outage postmortems How few services of hostedgraphite not hosted on AWS, got affected by AWS outage: https://blog.hostedgraphite.com/2018/03/01/spooky-action-at-a-distance-how-an-aws-outage-ate-our-load-balancer/ How codespaces had to shutdown their business after loss of data:
Engineering mongodb pause ttl The following command can be used to pause mongodb ttl for 86400 seconds (1 day) db.adminCommand({setParameter:1, ttlMonitorSleepSecs: 86400}) The following to disable TTL and enable it later on: db.adminCommand(
Hibernate+Spot fleet on AWS AWS has been constantly working on solutions to make computing less expensive and latest development on spot instances with option to hibernate has been an incredible feature. In short, we can run stateful
Stories The loner This is a scribbling I've done, sitting on marine drive Mumbai the other night. I wanted to keep the image as it is, instead of typing it or editing it. Please give your
Stories Sar Pass lapse Nowhere close to infinity, we were at 13,800 ft. A few foot steps on the canvas of snow, imprinted memories deep within, lived an experience. It was life beyond comforts. Limits pushed
Engineering Quick AWS commands Uploading custom ssl certificate to AWS Cloudfront: aws iam upload-server-certificate --server-certificate-name <Certificate name> --certificate-chain <interim-certificate> --private-key <private-key-file> --certificate-body <certificate-secret> --path /cloudfront/
Engineering Few Logrotate Configs Logrotate is a linux utility which rotates files upon certain conditions.If it's not installed, you can do via apt-get install logrotate or yum install logrotate The usual tricky part of logrotate is
Engineering Rolling update on AWS Autoscaling AWS autoscaling groups let you manage a set of instances and scale them in or out, based on various parameters. Deployment strategy: To launch new instances with latest code version and deprecate old
Stories Moonrise Alone in the misty woods, the river flow being echoed, he stood on the cliff watching the distant stars fade away in brightness. The golden line on the sky made itself vivid, cleared
Stories One Last Letter Staring at stars, a voice whispered, “Did you get response yet?” I held my tears, kept my voice low, took a heavy breath. I tricked my mind to travel a few years ago.
Engineering My RAM :O :( At it's core, Linux has a simple philosophy, "Everything is a file" Linux communicates with everything including Block devices, Serial devices, Network devices as if they're files (same as one'd open
Engineering Resizing disk size without restarting process Often, we end up adding disks to increase space. However, we have to mount them on a new partition. Change our data modules everywhere. Ex: Mongo disk is originally on /dev/sdb1. Now,
Engineering linux-debugging Debugging in Linux! Who is the guy talking? What’s Debugging in DevOps? Usual Methods Usual things Tools strace uptime dstat top/htop/atop perf opensnoop/execsnoop/iolatency. telnet netstat lsof Basic Intermediate
Engineering checking ulimit To check ulimit (which generally causes too many files open error), don’t use ulimit -n ulimit of each process varies and when you issue the command, it’s issued by shell you’
Engineering Edit file vi, not opened by sudo Opened a file in vi without sudo. Edited some stuff, realized you can't save it. You close the file, open with sudo, redo the edits. How about, switching to sudo while vi is
Engineering Debugging git slowness Here's the command to understand where is it running slow and what's happening at each step. set -x; GIT_TRACE=2 GIT_CURL_VERBOSE=2 GIT_TRACE_PERFORMANCE=2 GIT_TRACE_PACK_ACCESS=
Engineering Deleting a node from Chef On workstation knife node delete --yes NODENAME knife client delete --yes NODENAME On client yum -y remove chef rm -rf /var/chef rm -rf /etc/chef rm -rf /opt/chef Bootstrap again, if
Engineering mysql config check mysqld --help --verbose will show variables that'd be loaded if process mysqld (re)started. This will also throw errors if config file /etc/my.cnf is invalid. To get variables of current running
Engineering mysql reset root password sudo service mysqld stop sudo su - mysql /usr/sbin/mysqld --skip-grant-tables --skip-networking & mysql -u root > flush privileges > SET PASSWORD FOR [email protected]'localhost' = PASSWORD('myn3wpa$$word'); > quit exit sudo
Engineering Migrate mysql with minimal downtime Login to existing mysql server. Change the configuration /etc/my.cnf, add the following server-id=1, log-bin=mysql-bin To make sure your configuration is correct, run mysqld --verbose For the mysql process to
Engineering Change log from git commits Git commits aren't the perfect way to communicate change logs, however it gets better each phase. Phase 1: Tell developers to add #changelog in their commit messages, if they want a message to
Engineering Amazon EBS backup + LVM No-nonsense steps: Backup mechanism: Tag volumes of the attached instance with appropriate volume_group names. For each volume group, take lvm snapshot For each ebs in the volume group, take ebs snapshot. Delete
Engineering Debugging Connections To find number of unique connections per IP address, sorted from max to min: netstat -antu | grep :80 | grep -v LISTEN | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort
Engineering Setting AWS Cloudwatch alarms aws cloudwatch put-metric-alarm --alarm-name "application-name-cpu" --metric-name CPUUtilization --namespace AWS/EC2 --statistic Average --period 300 --threshold 80 --comparison-operator GreaterThanThreshold --dimensions "Name=InstanceId,Value=i-******" --alarm-actions arn:aws:sns:us-east-1:account_
Engineering Debugging Disk space To find top 10 files eating up disk space -- du -kh |sort -nr |head -10 To find disk space per folder/file in current directory. -- du -sch .[!.]* * |sort -h To find
