TIL, 2017-05-17

Sourcepad Staging Server Setup

[SP setup] vs [old setup]

VPC/Virtual Private Cloud - Each Sourcepad project has a staging VPC and a production VPC. This is a set of virtual machines (which Amazon calls instances) which make up the project.

VPC Components (Each of these are usually their own virtual machines)

Component	What Does It Do	Comparison to Local Environment	Privacy
Bastion	This is the only machine that has shell access to the other machines	Iterm	Public
Web Server/Nginx/Load Balancer	Routes `hello.sourcepadstage.com` to the correct app server. If more than one app server, the load balancer distributes requests to each of these servers.	Browser	Private
App Server	Runs the Rails app	`localhost:3000`	Private
AWS Relational Database Service (RDS)	Contains the relational database.	Your local RDBMS engine (PostgreSQL or MySQL)	Private
AWS ElastiCache (Optional)	Contains a key-value store (usually Redis). Redis is usually used for job queues like Resque and as a caching layer for your Rails app.	Your local Redis/`redis-server`.	Private
AWS Route 53/DNS/Domain Naming System (Optional)	An alternative to other DNS providers like `namecheap.com` and `godaddy.com`.	`ngrok`	Public
	You can choose to not set this up, but setting this up allows you to have one admin panel for both managing instances and managing DNS stuff.
AWS Simple Storage Service (S3)	Stores basically anything (photos, reports/PDFs, file uploads)	Your hard drive.	Private
NAT/Network Address Translation	Grants Internet access to the rest of the VPC.	Your Wifi modem.	Public

Why We Split Each Role Up

Theoretically, all of this can be in one machine, but we do not put it in one machine, because:

Multiple points of failure–we want to split things up so that even if things like the caching layer goes down, the web app can still function because the web app and RDS machine is still up.
Can scale specific parts of the app–different machines have different requirements. Bastion doesn’t need much, it literally is just a platform for deploying the apps, so we can leave it at a micro-level instance. A Rails server needs more CPU and an RDS server needs more memory (database accesses are memory intensive). If we need a faster Rails server, we do not need to scale the other components up.
Continuing with different requirements for each machine, there are also different prices for each Amazon service.

On Bastion

Even if you have your staging server’s database username and password, you won’t be able to access them from your local machine. Why is that? This is because your staging server’s VPC only allows very few hosts to have shell access.
These hosts are the Bastion server(s) (usually just one for staging).

Accessing the Private Instances

# ssh into the Bastion host
$ ssh [email protected]

# ssh into a VPC instance (in this case, the app server) from inside the Bastion host.
[email protected]$ ssh [email protected]

This will not work:

# You are not inside the Bastion host yet.
$ ssh [email protected]
# ssh: connect to host 10.10.10.10 port 22: Operation timed out

When you are deploying your code via Capistrano, your Capistrano settings, if they have been set by another SP developer, already do this process (so you can deploy with ease). To confirm this, check out /config/deploy/staging.rb or config/deploy/production.rb.

Scaling Basics

2 primary ways to scale:

Vertical scaling: Increasing the computing power of a machine (either CPU or memory). Most probably, you’ll do this with RDS servers first.
Horizontal scaling: Increasing the number of machines that do a specific role. Most probably, you’ll do this with app servers first.

Example of a project that is vertically scaled: WF. WF needs a big database since it has millions of database rows. It is vertically scaled in the RDS layer (it is also horizontally scaled hehe).

Example of a project that is horizontally scaled: ?.

This is why it is better to have efficient code since efficient code is cheaper, computing wise. You don’t need to scale immediately :).

Instance Types

Reference

EC2 instances: Bastion, nginx, App, front-end. No hard drives yet.
EBS Volumes:.
EBS:
Elastic IP: Static IPs of the EC2 instances. If you do not have this, if you restart server, new address.

Devops tools

Elastic Beanstalk: Easiest to run and like Heroku.
OpsWorks: You need chef recipes.
Full management: CloudFormation.

IAM Management

Identity and Access Management–per user.

Creating a VPC

IPv4 - 10.0.0.0/16
VPS/Virtual Private Server – old way.
NAT is automatically included.
Why not Digital Ocean/Heroku? Because they are also attached to AWS, and you are hiring the labor.
RDS needs 2 subnets. Public - 0, first - 1, second - 2. First private subnet: web server/front-end. Second private subnet: RDS/Redis.
Security Groups - pota hahaha.

Create Security Group - Network/Seceurity Group.

Create Security Group - Source is the Private IPs.
Type PostgreSQL.
Type. 5432 PostgreSQL. 3306 MySQL. 6379 Redis.
Default VPC security group firewall of ISP.
Create Security Group for PostgreSQL, another SG for Bastion–Type SSH, protocol TCP, port range 22, source. Sometimes anywhere.
Create SG for Nginx. TCP 80, TCP 443, TCP 22, anywhere.
Create SG for API. Port 3000.
NAT: Source: Itself. From the cloud to itself.
Private IP, pem file.
Creating an RDS.