Categories
python ruby Technology

An open letter to SmugMug

TL;DR

SmugMug is great, but its developer ecosystem is, in my humble opinion, crumbling, and can use some serious love — or put out of its misery and die…

Dear SmugMug, there are lots of people, myself included, who want to see you thrive and succeed. People who are spending their free time, resources and energy on sharing their tools with the community. People who can build great things on top of SmugMug, and can make SmugMug even more successful than it currently is. Please don’t forget us. We are the potential evangelists, multipliers, and we do this for free. Please treat our free gifts with respect. These gifts might be free, but they are precious. They should be cherished, rather than ignored, or discarded.

Categories
monitoring optimization Performance python Technology

a scalable Analytics backend with Google BigQuery, AWS Lambda and Kinesis

On my previous post, I described the architecture of Gimel – an A/B testing backend using AWS Lambda and redis HyperLogLog. One of the commenters suggested looking into Google BigQuery as a potential alternative backend.

It looked quite promising, with the potential of increasing result accuracy even further. HyperLogLog is pretty awesome, but trades space for accuracy. Google BigQuery offers a very affordable analytics data storage with an SQL query interface.

There was one more thing I wanted to look into and could also improve the redis backend – batching writes. The current gimel architecture writes every event directly to redis. Whilst redis itself is fast and offers low latency, the AWS Lambda architecture means we might have lots of active simultaneous connections to redis. As another commenter noted, this can become a bottleneck, particularly on lower-end redis hosting plans. In addition, any other backend that does not offer low-latency writes could benefit from batching. Even before trying out BigQuery, I knew I’d be looking at much higher latency and needed to queue and batch writes.

Categories
monitoring optimization Performance python Technology

a Scaleable A/B testing backend in ~100 lines of code (and for free*)

(updated: 2016-05-07)

tip-toeing on the shoulders of giants

Before I dive into the reasons for writing Gimel in the first place, I’d like to cover what it’s based on. Clearly, 100 lines of code won’t get you that far on their own. There are two (or three) essential components this backend is running on, which makes it scalable and also light-weight in terms of actual code:

  1. AWS Lambda (and Amazon API Gateway) – handle the requests to both store experiment data and to return the experiment results.
  2. Redis – using Sets and HyperLogLog data structures to store the experiment data. It provides an extremely efficient memory footprint and great performance.

For free?

Categories
django python Security

Django-Tastypie Authorization glitch

TL;DR

If a request using django tastypie is not authorized, please make sure to raise Unauthorized() exception in your _detail authorization methods in Tastypie v0.9.12.

The longer version

On one of my previous posts I wrote at length about django-tastypie authorization and gave some tips and tricks on how to work more flexibly and securely with this framework. A lot has happened since, and it was hard to keep track of all the various changes and updates to Tastypie.

Since version 0.9.12, the authorization mechanisms in tastypie changed rather radically, and that’s a very good improvement. It plugged some holes with nested resources and authorization, and made authorization decisions more granular. From a simple is_authorized and apply_limits, now each operation can be authorized, broken down to CRUD elements (create, read, update, delete). Each element is authorized for _list and _detail operations (I’ll try to cover this in more depth on a follow-up post at some stage).

For now, I just wanted to highlight an important pitfall you might want to avoid when using the new tastypie authorization that could leave you exposed. There’s a fix in the pipeline very soon, but until then, you should protect yourself by making a small change to your authorization methods, and the _detail ones in particular

The crux of the issue is that the _detail authorization methods should make a binary decision – is this authorized? (yes/no). If the method returns True, or does nothing, the request is authorized. If the method returns False or raises an Unauthorized exception, the request should be blocked.

The glitch is that if your authorization _detail functions return False, the request still goes through and is effectively authorized. Until the fix is in place, please make sure to raise Unauthorized() exception if you’re using Tastypie v0.9.12.

Categories
monitoring python Security Technology

Statsd and Carbon security

I’ve written about installing and using Graphite and it’s a really great tool for measuring lots of kinds of metrics. Most of the guides online don’t touch on the security aspects of this setup, and there was at least one thing that I thought should be worth writing about.

How are we measuring

Metrics we gather from our applications have the current characteristics / requirements:

  • We want to gather lots of data over time.
  • Any single data-point isn’t significant on its own. Only in aggregate.
  • Measuring is important, but not if it slows down our application in any way.
Categories
django monitoring python Technology

Fabric Installer for Graphite

fabric-graphite is a fabric script to install Graphite, Nginx, uwsgi and all dependencies on a debian-based host.

Why?

I was reading a few interesting posts about graphite. When I tried to install it however, I couldn’t find anything that really covered all the steps. Some covered it well for Apache, others covered Nginx, but had steps missing or assumed the reader knows about them etc.

I’m a big fan of fabric, and try to do all deployments and installations using it. This way I can re-run the process, and also better document what needs to be done. So instead of writing another guide, I created this fabric script.

Categories
django optimization python Security

Keep your hands off my tastypie

Update

Please note that since Tastypie v0.9.12 the authorization framework was rewritten. Lots of information on this post no longer applies. I’m hoping to write a follow-up post at some stage.

Original Post

I’ve been using tastypie, the very awesome django REST API framework for a little while now (btw, that’s not the official title, but it might as well be). I’m not going to write yet another comparison between tastypie and django-piston. My reasons for choosing tastypie were that its code looked nicer, and it seemed a much more active project.

One of the things that I immediately liked about tastypie, being a security-geek and all, was the security framework built into it. Primarily the authentication and authorization classes. They make it very easy to extend, and almost a no-brainer to apply to any resource. This means that providing resource-level authorization is also very easy and clean.

However, whilst working with tastypie and applying some authorization rules to my resources, I noticed a couple of pitfalls. Those are quite easy to miss if you’re not very familiar with the tastypie codebase. I wouldn’t say it’s a vulnerability or a bug as such, perhaps more of a (sub-optimal) design choice from a security-perspective. That said, if you use tastypie incorrectly, or unaware of those pitfalls, you might create a security vulnerability on your otherwise delicious API.

Categories
django monitoring optimization python Technology

django memory leaks, part II

On my previous post I talked about django memory management, the little-known maxrequests parameter in particular, and how it can help ‘pop’ some balloons, i.e. kill and restart some django processes in order to release some memory. On this post I’m going to cover some of the things to do or avoid in order to keep memory usage low from within your code. In addition, I am going to show at least one method to monitor (and act automatically!) when memory usage shoots through the roof.

Categories
django monitoring optimization python Technology

django memory leaks, part I

A while ago I was working on optimizing memory use for some django instances. During that process, I managed to better understand memory management within django, and thought it would be nice to share some of those insights. This is by no means a definitive guide. It’s likely to have some mistakes, but I think it helped me grasp the configuration options better, and allowed easier optimization.

Does django leak memory?

In actual fact, No. It doesn’t. The title is therefore misleading. I know. However, if you’re not careful, your memory usage or configuration can easily lead to exhausting all memory and crashing django. So whilst django itself doesn’t leak memory, the end result is very similar.

Memory management in Django – with (bad) illustrations

Lets start with the basics. Lets look at a django process. A django process is a basic unit that handles requests from users. We have several of those on the server, to allow handling more than one request at the time. Each process however handles one request at any given time.

But lets look at just one.

cute, isn’t it? it’s a little like a balloon actually (and balloons are generally cute). The balloon has a certain initial size to allow the process to do all the stuff it needs to. Lets say this is balloon size 1.

Categories
python Security Technology

Once upon a time

One-Time-Passwords always fascinated me. Long long time ago in a land far far away I suddenly had this idea. The idea was simple and in today’s terms pretty common, perhaps trivial. One-Time-Password without the need for an extra token. After the user keys in their username and password, they get sent a random password via SMS. Ten years ago there wasn’t anything that did that. I created a basic RADIUS implementation with support for different SMS gateways, all in Java. Sadly however, with no funding, no clue how to turn it into a business, and just finishing my computer science degree, it had to be abandoned for an easier day job.

I was recently pulled into looking at two-factor-authentication (2FA) solutions. I used SecurID at a previous job, and know of several solutions in this area. I was quite pleased to discover many soft-token solutions working on mobile phones (iphone, blackberry, HTC, Nokia) and USB-based ideas like Yubikey. I was even more pleased to discover open source initiatives in this area, and OATH HOTP in particular.