RUN@cloud » Amazon Web Services

Amazon Web Services

Last modified by Tracy Kennedy on 2013/12/11 20:01

aws.png

Amazon has a wide variety of services available for developers right now. For the CloudBees platform on AWS, this means that you can make low latency reuse of these services in your applications.

The following are a set of services that are available to developers for the applications today (and many have been in use for some time used by both us and customers). 

AWS accounts

Customers with their own Amazon Web Services accounts can provision Amazon cloud resources from services like RDS, SQS, and EC2 and connect to them from applications that are running on CloudBees.

Geographic regions and data centres

CloudBees runs in multiple regions, so for some services you want to ensure that the region that it is running in is the same one your CloudBees services are in - for latency and data sovereignty issues - when you set up.

Billing

You will receive a separate bill for these services from Amazon.

Provisioning and setup

You will need your own AWS account, and to set up resources using each services control panel as described below.

RDS

RDS (Relational Data Services) is a hosted relational database service for the following systems that compliment the CloudBees and partner hosted databases:

  • MySQL 
  • Oracle RDBMS 
  • Microsoft SQL Server 

When to use

When you need a relational database (one of the above!). Many people have an existing RDS instance and wish to reuse it with their CloudBees application (this is also fine). All licensing costs are taken care of in your monthly billing of the servers. These databases run as dedicated instances. 

How to use

Use the RDS component of the Amazon console to create your database. Ideally - please pick multi-zone configurations to ensure high availability, and the region that contains the CloudBees applications you wish to access the database from (we also recommend "provisioned IOPS" for consistent throughput).

To let your RDS servers be accessed from CloudBees applications, you will need to let them be accessed by cloudbees application server instances. Currently for new AWS accounts, there is no way to use security groups (pre 2013 AWS accounts can use the technique below) - we are awaiting Amazon to rectify this omission shortly. We are also looking at VPN based solutions in the longer term.

If you have dedicated servers on cloudbees - you can open up the database port to your cloudbees IPs - otherwise you will need to open up RDS to all IPs or at least all the IPs used by the amazon region you are running in. 

For pre-2013 AWS accounts ONLY, you can use security groups: 

you will need to open up to the following CloudBees security groups:
Security Groups: appserver, genapp
AWS Account ID: 063356183961

You can find that in your AWS RDS console: 

aws-change-awsid.png

How to configure with CloudBees applications

Once you have your credentials for your RDS service, it is recommended that you bind them into your application as resources or configuration variables so that they will be injected into your app at runtime.

Binding RDS as a Resource

If you have registered your RDS database as an external resource on your CloudBees account, you can use the application bind command to inject it into your application container so that it is available at runtime. If your app is running in a container that supports database resource injections (like the tomcat or jboss stacks), you'll be able to use access the database as a Java EE DataSource using JNDI.

For example, you could inject your RDS database as a app resource named mydb:
bees resource:create  -t database  rds1 DATABASE_URL=mysql://acme.mdnahwad.us-east-1.rds.amazonaws.com:3306/mydb   DATABASE_USERNAME=someusername  DATABASE_PASSWORD=somepassword

bees app:bind -a acme -r resource:rds1 -as mydb

See: Generic Resources to learn more about configuring RDS databases a resources.

Binding RDS as Configuration Parameters

To directly inject the credentials for your RDS database, you can use configuration parameters that are read by your application at runtime:

bees config:set -a yourapp rds_user=X rds_password=Y
You can then configure your connection pool to access these variables when connecting to the database. You can read more here. Please contact us if you need help with this.

A DB connection pool should always be used - and this is available with the drivers for the respective database and language you are using.

See: Configuration Parameters to learn more about securely adding configuration parameters to your apps.

VPC

VPC is Amazon's Virtual Private Cloud service. This is designed for running isolated environments, where the CloudBees platform is running in an isolated virutal network with your corporate services. This is a successful alternative to "private cloud", and functions as a special case of the CloudBees dedicated server offering.

VPC.png

When to use

VPC gives you a lot more control over the internal/external network setup of your instances - this is very close to running the CloudBees platform on servers on your own network (think of it like that). This requires more expertise to setup but with the benefit of more control. 

VPC does allow things like IPSec traffic and GRE tunnels - allowing you to link your behind-the-firewall servers securiely with your virutal private cloud (the cloud becomes an extension of your network).

VPC servers can have IP addresses that are internal to your organization and completely isolated.

How to use

As VPC is complex, the best bet is to contact us to ensure this is what you want before you set it up. VPC would generally be setup in a dedicated AWS account (which may be managed by either CloudBees, or your organization, depending on your needs).

How to integrate with CloudBees

The hard bit is setting up the VPC - once set up, as developers, there is no real difference to the experience (you can have some applications effectively running in a private cloud and others in the public cloud for public users) - there is not much to do here!

Dynamo DB

DynamoDB is a very, very fast NoSQL database (also Amazon's fastest growing service, in popularity). This service is similar to the much-written about "big table" from Google - a very highly scalable service for storing semi-structured data (it is non-relational). Dynamo spreads its load and data over many machines to achieve its scalability and uses SSD drives for storing all data for extremely low latency.

When to use

Dynamo DB would be considered whenever you would consider a NoSQL database and want very low latency and high speed. There is some learning curve with the tunable features of DyanmoDB, so spend some time reading before jumping into it.

How to use

Using the AWS console you can create and tune your DynamoDB service - in the region that is where your CloudBees services are running. 

How to integrate with CloudBees applications 

Similar to RDS - you will obtain some credentials that allow your application to access DynamoDB - you can provide this as configuration items via bees config:set  which will inject them at runtime (and be stored encrypted).

You will also need to use a DynamoDB client - Amazon ships clients (or "drivers") in various languages in their SDK (notably for Ruby and Java). The Java one is shown in action in this sample. In that example, the credentials are stored in a properties file. This can be set as a configuration parameter in CloudBees - and then accessed via System.getProperty - for example.

CloudFront CDN

CloudFront is a Content Delivery Network (CDN). A CDN uses servers all over the world to cache content so that when users access your application, their local DNS server will likely direct them to a CDN store which is geographically close to them. This takes a load off your main application, as well as providing a far better experience for the user.

We make good use of the CloudFront CDN at CloudBees and it works very well.

When to use

Almost all web apps can make use of a CDN in some way, typically for:

  • CSS
  • JavaScript
  • images
  • any large, mostly static files

It is not to be used for assets that you will need to change often. Generally, if you want to post a new version to a CDN, it can take a while for the change to take effect. It is often easier to provide an asset as a new name/path (and let the CDN cache it indefinitely).

How to use

The two main ways you can use CloudFront are:

  • via an S3 bucket
  • in front of your application

In the first case - you proactively upload your content to an S3 bucket (that you create from the AWS console) - e.g., very large files, sounds, large images that you know will be slow to download. When you release a new version, you upload it. In the AWS console, in the CloudFront section - you create a new "distribution" that points to your S3 bucket - it will then provide a unique URL to access the contents of that bucket.

In the second case, you place your assets in your application package as normal, but tell CloudFront (via the management console) to create a new "distribution" for CloudFront based on the URL to your applicaiton. It will then provide URLs that you can place in your application so that the assets are loaded from the CDN, instead of directly from your application (CloudFront will then load from your application as needed - and cache the assets). 

How to integrate with your CloudBees application

In this case, it is mostly updating the base URLs (the hostnames) to the assets you are shifting to the CDN. The path component of the URL will remain the same, but the hostname will change (some web frameworks can help automate this process).

CDNs like CloudFront are global - so they aren't restricted to a geographic region.

Simple Queue Service(SQS)

SQS is a hosted message queueing service for distributing work (messages) amongst machines. It has an http interface and client libraries for most languages (for convenience).

SQS.png

When to use

When you have messages or work to transfer between applications, or an application that is distributed amongst many machines, SQS can be a simple way to spread work across the instances (it also allows for locking and timeout of messages that are being worked on). This includes asychronous and long-running workflows. 

How to use

The hardest part of any queuing system is the designing of your application to cope with asynchornous message-based programming. The actual consumption of SQS is the easy part! Typically you have a unit of work which is represented as a message placed on a queue, and then one or more consumers that monitors this queue, locks the item and performs work on it (it then removes the item from the queue).

The SQS service is enabled in your AWS account via the console. Queues can then be programmatically created by the http API or client libraries.

The Amazon SDK for Java provides a JVM-friendly client library - but you can also use the query the API directly if you wish (similar SDK exists for Ruby).

The API reference is here and the request-driven API is documented here.

How to integrate with your CloudBees applications

Similar to other services - all that is needed here are your credentials via the bees config:set command. You are free to use a client library based on your language platform, or use http directly (SQS is designed to be language agnostic) - even communicating between applications writtin in different languages is possible. 

Simple Storage Service (S3)

S3 is one of the most popular and well known services - it is a "blob store" - or a store for binary artifacts (often large files). This can be used in a semi-transactional capacity, or a kind of bulk store for downloadable content (often used in concert with the CDN CloudFront for content delivery). At CloudBees, we make heavy use of this API. This is a name/value type database - but used for documents or large content (not for searching or querying).

When to use

Many applications have "blob" content. It is far better to store this in S3 than a database or even a file system (however, S3 can be higher latency in some cases - at least to write data). S3 is extremely durable and available (generally far better than any other system you could imagine). Store files, images, scripts, documents. 

How to use

In your AWS console, you navigate to the S3 section and define buckets. You can then set security policies for the bucket (contents can be private, encrypted or publicly addressable). You can upload content from the console and even set security constraints on each item in a bucket. Content can be streamed directly from S3 to clients - saving on application bandwidth and load, if needed.

How to integrate with your application

You can use the S3 services as a highly available blob/document store for your application or your can store data and serve it directly (or a combination of both). Content can be uploaded via: the Amazon command line, a variety of client tools, or client libraries (in every language).
Some client libraries are:

These are for when you programmatically want to control S3 from your application. In this case, as with others, you would inject the S3 credentials via bees set:info

Resources you want to expose directly to clients you can by setting permissions on the S3 item and bucket, and providing a URL to download the artifact (a common use is signed URLs that have an expiration - so a piece of content is only available for a certain amount of time).

To create a S3FS resource that will represent the Amazon S3 bucket, use the following command:

bees resource:create -t s3fs  RESOURCE_NAME  s3_bucket=AWS_BUCKET_NAME   s3_passwd=AWS_KEY:AWS_SECRET  mount_link=APP_MOUNT_POINT

Once the resource is bounded to an application, the application  access the s3fs file system via the local path to the directory specified by the "mount_link" parameter after you bind it to an application.

For example to create a s3fs resource name "fs1" for the aws S3 bucket "acme-bucket" mounted in the application local directory under "my_s3".
bees resource:create -t s3fs  fs1  s3_bucket=acme-bucket   s3_passwd=yourAWSKey:yourAWSSecret mount_link=my_s3

Bind this resource to an application
To mount a s3fs resource called "fs1" for the application "acme" with the alias "fs-acme".
bees app:bind -a acme -r resource:fs1 -as fs-acme

Roadmap

We will shortly be integrating these services under the CloudBees services platform. This will allow you to "bind" services to your application, vastly simplifying the above setup. Provisioning and billing will also be done on your behalf, so you will not require an Amazon account in all cases.

Please contact us with any queries or if you need help.

Created by Michael Neale on 2012/11/22 23:06