Automated Deployment with EC2 and Bitbucket

First, I am going to split this into several parts in order to be able to handle the wordpress editing process.

1. I am going to describe how I bring up an ec2 instance with boto3
2. I describe the oauth process to the bitbuckt REST API and the transfer of a deploy-key
3. Bring it all together and wrap it up

So, to start lets create an EC2 instance:


import boto3

def start_ec2_app():

c = get_client()
res = get_resource()

app_sg = create_app_sg(c, 'app')

keypair = c.create_key_pair(KeyName='app_key')
with open('keys/'+keypair['KeyName']+'.pem','w+') as keyfile:
keyfile.write(keypair['KeyMaterial'])

inst = res.create_instances(
ImageId = AMI,
KeyName = 'app_key',
InstanceType = INSTANCE_TYPE,
SecurityGroups = ['app'],
MinCount = 1,
MaxCount = 1
)

# let's wait for the instance
runningWaiter = c.get_waiter("instance_running")
runningWaiter.wait(InstanceIds = [inst[0].id])

instances = res.instances.filter(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
i = 0
for instance in instances:
tags = [{"Key" : 'instanceName', "Value" : 'app_%s' %i}]
print(instance.public_ip_address+' '+instance.public_dns_name)
c.create_tags(
Resources = [instance.id],
Tags = tags)

allow_ssh(c, 'app')

def allow_ssh(c, name):
sg = c.describe_security_groups(Filters=[{'Name': 'group-name', 'Values': [name]}])
group = sg['SecurityGroups'][0].get('GroupId')
c.authorize_security_group_ingress(
IpProtocol = "tcp",
CidrIp = "0.0.0.0/0",
FromPort = 22,
ToPort = 22,
GroupId = group)

def get_client():
return boto3.client(
'ec2',
region_name = AWS_REGION,
aws_access_key_id = AWS_ACCESS_KEY,
aws_secret_access_key = AWS_SECRET_KEY,
)

def get_resource():
return boto3.resource(
'ec2',
region_name = AWS_REGION,
aws_access_key_id = AWS_ACCESS_KEY,
aws_secret_access_key = AWS_SECRET_KEY,
)

def create_app_sg(c, name):
sg = c.describe_security_groups(Filters=[{'Name': 'group-name', 'Values': [name]}])
if not sg['SecurityGroups']:
c.create_security_group(
GroupName = name,
Description = '%s Security Group' %name)

start_ec2_app()

Okay, okay… let’s go slowly:
a) we need a client to work (or at least I prefer a client, you could use a resource with some fiddling), so we create one
b) this client now creates a security group that gets the fabulous name ‘app’
c) we run an instance (a single one in this case)
d) we get the id of the instance so we can
e) wait until the instance is running
f) then we can retrieve the public ip and the dns-name
g) lastly we create a pair of keys and save the private key

So, this is done. Off to bitbucket and oauth next…

Advertisements

Perl vs. Python RegEx Shootout

I am constantly told that Perl has much better regex performance than python. When I ask people how they know they answer with “everybody knows that” or “because it’s native” or I am shown some obscure benchmarks whcih seem to test anything but regex performance (hardcoded regex vs interpolated etc.). I wanted to know, and I wanted to fiddle around with performance analysis since I am dealing with Big-O lately. So, without putting an end to the discussion and more as a base for discussions with colleagues and friends here is what I did:

1. I took a large text (Moby Dick at archive.org

2. I wrote a very small programs in perl and python

3. I read in the whole file and measured the time (to be able to see whether one program takes longer to read or not)

4. I ran the code with regex

5. I changed the regex and ran it again

6. I measured with linux’s “time”

I am however not interested in absolute performance (which is machine dependent) but relative.
Version were
perl 5, version 18, subversion 2 (v5.18.2) built for darwin-thread-multi-2level
and
Python 2.7.6 (default, Jan 17 2014, 15:43:59) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin

The first two scripts were these

import re;

count = 0
with open('mobydick.txt','r') as f:
data = f.read();


#!/usr/sbin/perl -w
use utf8;
use strict;
use warnings;

my $string;

open FILE, "<", "mobydick.txt";
$string = join("", );
close FILE;

Ran them both and got
python py_regex.py 0,02s user 0,02s system 53% cpu 0,069 total
perl pl_regex.pl 0,01s user 0,02s system 70% cpu 0,047 total

Pretty close. So, I don’t have to concern myself with reading speed in the next measurements.

Then I changed the code to include some regexes. I just counted how many times the word “Pequod” was used.

import re;

count = 0
with open('mobydick.txt','r') as f:
data = f.read();

m = re.findall('(Pequod)', data);

for find in m:
print find
count+=1

print "%d" %count

#!/usr/sbin/perl -w
use utf8;
use strict;
use warnings;

my $count = 0;
my $string;

open FILE, "<", "mobydick.txt";
$string = join("", );
close FILE;

my @m = $string =~ /(Pequod)/g;

foreach(@m){
print "$_\n";
$count++;
}

print $count."\n";

Ran them again and got:

Pequod
[...]
Pequod
66
python py_regex.py 0,02s user 0,01s system 89% cpu 0,033 total

And

Pequod
[...]
Pequod
66
perl pl_regex.pl 0,01s user 0,01s system 89% cpu 0,021 total

Okay, that was a little surprising since in the discussions I had before “outperforms” was a term used quite often.
Maybe it was just that the regex was simply not complex enough or something…

Change the regex and keep everything else.

m = re.findall('(.*Pequod:*)\s', data);

my @m = $string =~ /(.*Pequod.*)\s/g;

And run it again

the Pequod. Devil-Dam, I do not know the origin of ;
[...]
SLOWLY wading through the meadows of brit, the Pequod
66
python py_regex.py 0,07s user 0,01s system 95% cpu 0,082 total

Not too bad an increase.

the Pequod. Devil-Dam, I do not know the origin of ;
[...]
SLOWLY wading through the meadows of brit, the Pequod
66
perl pl_regex.pl 18,16s user 0,09s system 99% cpu 18,347 total

GOODNESS ME!!

I still don’t know what happened, but I will ask around…

Flask migrations with alembic without flask-migrate

The starting point for this setup was, that I wanted to be able to automigrate and to be able to move my models in a folder. And since flask-migrate is using a manager I opted out of that and used plain alembic with flask-sqlalchemy.

So, this is my Flask setup:

app.py
alembic.ini
models/
      shared_model.py
      ... the models ...
alembic/
      env.py
      ... versions/ and rest of alembic ...

So where is the magic happening?
Actually (spoilers!) there is no magic. Maybe I’m just a little daft, for not getting this right sooner. So here is what I do
In my shared_model.py I put my db declaration like this


from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()

and the rest of the code I like to use in all models.

In my app.py I initialize my app via function


from flask import Flask
app = Flask(__name__)

def start_app():
      ... configuration magic ...
      db.init_app(app)
      return app

if __name__ == "__main__":
      app = start_app()
      app.run()

Finally the adjustments in alembic/env.py somewhere underneath the MetaData


from start import start_app
app = start_app()
from models.shared_model import db
db.init_app(app)
config.set_main_option("sqlalchemy.url", app.config["SQLALCHEMY_DATABASE_URI"])
target_metadata = db.metadata

An that’s it. Blueprints, autogenerate and all the fancies 🙂

flask, alembic and blueprints

For some time I could easily do without autogenerated migrations. Now I wanted them and I wanted to use Flask and not Django. I started, very naively, by installing and importing either flask-alembic and flask-migrate but they all seemed (at that time) to support patterns that I didn’t want (e.g. manager, single models.py) or couldn’t understand. At some points I didnt’t get migrations to work at all or they were empty or blueprints wouldn’t work or…

What I wanted was
* a folder “models” containing all models with a file for each model
* plain alembic
* a single start file with my setup and configs

After installing alembic via pip migrations didn’t work and even importing model in env.py didn’t solve it, fiddeling with target_metadata didn’t help as well as several other solutions outlined in StackOverflow. So here is what worked for me:

In my start/setup file (start.py in my case) has a function:


start_app():
app = Flask(__name__)
# config stuff
db.init_app(app)
return app

and


if __name__ == "__main__":
app.start_app()
app.run()

The app is started by just running python start.py without need of a manager.

I created a my shared_model that all model import:


from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()

This makes it easier since all the models just import this shared model and I can also put some other stuff in here that I want to have access to in my models.

The last thing to do is editing the alembic env.py:
1. Import the start_app function and start the app
2. Import the db from the shared model and initialize it
3. Configure and set target_metadata


from start import start_app
app = start_app()
from models.shared_model import db
db.init_app(app)
config.set_main_option("sqlalchemy.url", app.config["SQLALCHEMY_DATABASE_URI"])
target_metadata = db.metadata

That’s about it models go into the models folder and can be used in blueprints, alembic revision –autogenerate produces more than “pass” and the app starts like usual.