Full Stack Python

Application Performance Monitoring AWS Lambda Functions with Sentry

Matt Makai — Thu, 26 Aug 2021 00:00:00 -0400

Amazon Web Services (AWS) Lambda is a usage-based computing infrastructure service that can execute Python 3 code. One of the challenges of this environment is ensuring efficient performance of your Lambda Functions. Application performance monitoring (APM) is particularly useful in these situations because you are billed based on how long you use the resources.

In this post we will install and configure Sentry's APM that works via a Lambda layer. Note that if you are looking for error monitoring rather than performance monitoring, take a look at How to Monitor Python Functions on AWS Lambda with Sentry rather than following this post.

First steps with AWS Lambda

A local development environment is not required to follow this tutorial because all of the coding and configuration can happen in a web browser through the AWS Console.

Sign into your existing AWS account or sign up for a new account. Lambda gives you the first 1 million requests for free so that you can execute basic applications without no or low cost.

When you log into your account, use the search box to enter "lambda" and select "Lambda" when it appears to get to the right page.

If you have already used Lambda before, you will see your existing Lambda functions in a searchable table. We're going to create a new function so click the "Create function" button.

The create function page will give you several options for building a Lambda function.

Click the "Browse Serverless App Repository" selection box, then choose the "hello-world-python3" starter app from within the "Public applications" section.

The hello-world-python3 starter app details page should look something like the following screen:

Fill in some example text such as "test" under IdentityNameParameter and click the "Deploy" button:

The function will now be deployed. As soon as it is ready we can customize it and test it out before adding Sentry to capture any errors that occur during execution.

Go back to the Lambda functions main page and select your new deployed starter app from the list.

Find the orange "Test" button with a down arrow next to it like you see in the image below, and then click the down arrow. Select "Configure Test Event".

Fill in the Event name as "FirstTest" or something similar, then press the "Create" button at the bottom of the modal window.

Click the "Test" button and it will run the Lambda function with the parameters from that new test event. You should see something like the following output:

Response
"value1"

Function Logs
START RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Version: $LATEST
value1 = value1
value2 = value2
value3 = value3
END RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914
REPORT RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914  Duration: 0.30 ms   Billed Duration: 1 ms   Memory Size: 128 MB Max Memory Used: 43 MB  Init Duration: 1.34 ms

Request ID
62fa2f25-669c-47b7-b4e7-47353b0bd914

The code was successfully executed, so let's add Sentry's performance monitoring and test some code that uses it.

Performance monitoring with Sentry

Go to Sentry.io's homepage.

Sign into your account or sign up for a new free account. You will be at the main account dashboard after logging in or completing the Sentry sign up process.

Select "Performance" on the left navigation bar, it will take you to the performance monitoring page.

Click "Start Setup" then go back over to AWS Lambda to complete the steps for adding Sentry's Python layer to your Lambda function.

The easiest way to add Sentry to Lambda for this application is to configure an AWS Lambda Layer with the necessary dependency for Sentry. Sentry has concise documentation on adding via Lambda Layers so we will walk through that way to configure it and test it out.

Scroll down to the "Layers" section while in your Lambda function configuration. Click the "Add a layer" button":

In the "Add layer" screen, select the "Specify an ARN" option.

Now to specify the Amazon Resource Name (ARN), we need to use the Sentry documentation to get the right configuration string.

US-East-1 is the oldest and most commonly-used region so I'll use that here in this tutorial but you should check which one you are in if you are not certain.

Copy that value into the Lambda Layer configuration, like this:

Then press the "Add" button. You now have the Sentry dependency in your environment so code that relies upon that library can be used in the Lambda function.

Testing performance monitoring

Let's change our Python code in the Lambda function and test out the APM agent.

Make sure you are signed into your Sentry account and go to this specific AWS Lambda set up guide.

You will see a "DSN string" that we need to set as an environment variable on AWS Lambda to finish our setup. Copy the string that matches your project as shown on that page in the highlighted green section:

We will use environment variables on AWS Lambda to store and access values like this Sentry DSN key.

Go into the Lambda console to create a new environment variable. To do that, click the "Configuration" tab within Lambda like you see here:

Then click "Edit" and add a new environment variable with the key of SENTRY_DSN and the value of the DSN string that you copied from the Sentry screen.

Click the "Save" button and go back to your Lambda function's code editor.

Replace the code in your Lambda function with the following code:

import json
import os
import sentry_sdk
import time
from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration
from sentry_sdk import start_transaction

SENTRY_DSN = os.environ.get('SENTRY_DSN')
sentry_sdk.init(
    dsn=SENTRY_DSN,
    traces_sample_rate=1.0,
    integrations=[AwsLambdaIntegration()]
)

print('Loading function')


def lambda_handler(event, context):
    calc = 1000

    # this is custom instrumentation, see docs: https://bit.ly/2WjT3AY
    with start_transaction(op="task", name="big calculation"):
        for i in range(1, 1000):
            calc = calc * i

    print(calc)
    return event['key1']  # Echo back the first key value

The above code imports the Sentry dependencies, and then runs both automatic instrumentation and custom instrumentation on the code. Click the "Deploy" button and then "Test". The code will successfully execute and when we go back to our Sentry performance monitoring dashboard we will see some initial results, like this following screenshot.

Looks good, you have both the default and the specified transaction performance recordings in the dashboard, and you can toggle between them (or other transactions you record) through the user interface.

What's Next?

We just wrote and executed a Python 3 function on AWS Lambda that used the basics of Sentry APM to get some initial performance monitoring data.

Check out the AWS Lambda section for more tutorials by other developers.

Further questions? Contact me on Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

How to Monitor Python Functions on AWS Lambda with Sentry

Matt Makai — Fri, 23 Apr 2021 00:00:00 -0400

Amazon Web Services (AWS) Lambda is a usage-based compute service that can run Python 3 code. Errors can happen in any environment you are running your application in, so it is necessary to have reliable monitoring in place to have visibility when a problem occurs.

In this post we will install and configure Sentry's application monitoring service that works specifically for code running on AWS Lambda.

Application Dependencies

A local development environment is not required to follow this tutorial because all of the coding and configuration can happen in a web browser through the AWS Console.

The example code can be copy and pasted from this blog post or you can access it on GitHub under the Full Stack Python blog-post-examples repository within the monitor-python-aws-lambda-sentry directory.

Accessing the AWS Lambda Service

Sign into your existing AWS account or sign up for a new account. Lambda gives you the first 1 million requests for free so that you can execute basic applications without no or low cost.

When you log into your account, use the search box to enter "lambda" and select "Lambda" when it appears to get to the right page.

If you have already used Lambda before, you will see your existing Lambda functions in a searchable table. We're going to create a new function so click the "Create function" button.

The create function page will give you several options for starting a new Lambda function.

Click the "Browse Serverless App Repository" selection box, then choose the "hello-world-python3" starter app from within the "Public applications" section.

The hello-world-python3 starter app details page should look something like the following screen:

Fill in some example text such as "test" under IdentityNameParameter and click the "Deploy" button:

The function will now be deployed. As soon as it is ready we can customize it and test it out before adding Sentry to capture any errors that occur during execution.

Testing the starter Python app

Go back to the Lambda functions main page and select your new deployed starter app from the list.

Find the orange "Test" button with a down arrow next to it like you see in the image below, and then click the down arrow. Select "Configure Test Event".

Fill in the Event name as "FirstTest" or something similar, then press the "Create" button at the bottom of the modal window.

Click the "Test" button and it will run the Lambda function with the parameters from that new test event. You should see something like the following output:

Response
"value1"

Function Logs
START RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914 Version: $LATEST
value1 = value1
value2 = value2
value3 = value3
END RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914
REPORT RequestId: 62fa2f25-669c-47b7-b4e7-47353b0bd914  Duration: 0.30 ms   Billed Duration: 1 ms   Memory Size: 128 MB Max Memory Used: 43 MB  Init Duration: 1.34 ms

Request ID
62fa2f25-669c-47b7-b4e7-47353b0bd914

That means the test case was successful, but what happens even if there is a straightforward mistake in the code, such as trying to access an undeclared variable?

Go into the code editor and you should see the starter code like this:

Update the code with the new highlighted line, which tries to access a fourth variable, which does not exist in the test configuration we try to run it with.

import json

print('Loading function')


def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event['key1'])
    print("value2 = " + event['key2'])
    print("value3 = " + event['key3'])
~~    print("value4 = " + event['key4'])
    return event['key1']  # Echo back the first key value
    #raise Exception('Something went wrong')

After adding that one new line of code, hit the "Deploy" button, then the "Test" button. You should see some error output:

Response
{
  "errorMessage": "'key4'",
  "errorType": "KeyError",
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      11,
      "lambda_handler",
      "print(\"value4 = \" + event['key4'])"
    ]
  ]
}

Function Logs
START RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb Version: $LATEST
value1 = value1
value2 = value2
value3 = value3
'key4': KeyError
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 11, in lambda_handler
    print("value4 = " + event['key4'])
KeyError: 'key4'

END RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb
REPORT RequestId: a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb  Duration: 0.81 ms   Billed Duration: 1 ms   Memory Size: 128 MB Max Memory Used: 43 MB  Init Duration: 1.61 ms

Request ID
a4e956bd-cce4-403e-b5e7-e95bc3ffa2cb

It is obvious when we are working in the Console that an error just occurred. However, in most cases an error will happen sporadically which is why we need a monitoring system in place to catch and report on those exceptions.

AWS Lambda function monitoring with Sentry

The easiest way to add Sentry to Lambda for this application is to configure an AWS Lambda Layer with the necessary dependency for Sentry. Sentry has concise documentation on addin gvia Lambda Layers so we will walk through that way to configure it and test it out.

First, scroll down to the "Layers" section while in your Lambda function configuration. Click the "Add a layer" button":

In the "Add layer" screen, select the "Specify an ARN" option.

Now to specify the Amazon Resource Name (ARN), we need to use the Sentry documentation to get the right configuration string.

US-East-1 is the oldest and most commonly-used region so I'll use that here in this tutorial but you should check which one you are in if you are not certain.

Copy that value into the Lambda Layer configuration, like this:

Then press the "Add" button. Now you have the Sentry dependency in your environment so code that relies upon that library can be used in the Lambda function.

Next we need to go into the Sentry dashboard to create a project, get our unique identifer, and connect it to our Lambda function.

Sentry can be self-hosted or used as a cloud service through Sentry.io. We will use the cloud hosted version because it is quicker than setting up your own server as well as free for smaller projects.

Go to Sentry.io's homepage.

Sign into your account or sign up for a new free account. You will be at the main account dashboard after logging in or completing the Sentry sign up process.

There are no errors logged on our account dashboard yet, which is as expected because we have not yet connected our account to our Lambda function.

Click "Projects" on the left navigation bar, then "Create Project" in the top right corner.

Under "Choose a Platform", select "Serverless" and then "AWS Lambda (Python)" as shown below:

Decide under what criteria it should send error information out of Lambda. For this tutorial, we will have it send every exception. Then click the "Create Project." button.

You can have Sentry handle the instrumentation automatically but we will handle it manually for our function. On the next screen, Sentry will provide you with your unique DSN string, which we will need for our function.

Typically you will want to use environment variables on AWS Lambda to store and access values like your Sentry key.

Copy the contents of the Sentry DSN string, and go into the Lambda console to create a new environment variable. To do that, click the "Configuration" tab within Lambda like you see here:

Then click "Edit" and add a new environment variable with the key of SENTRY_DSN and the value of the DSN string that you copied from the Sentry screen.

Click the "Save" button and go back to your Lambda function code.

Update your Lambda function with the following highlighted new lines of code to send errors to Sentry.

import json
~~import os
~~import sentry_sdk
~~from sentry_sdk.integrations.aws_lambda import AwsLambdaIntegration

~~SENTRY_DSN = os.environ.get('SENTRY_DSN')
~~sentry_sdk.init(
~~    dsn=SENTRY_DSN,
~~    integrations=[AwsLambdaIntegration()]
~~)

print('Loading function')


def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))
    print("value1 = " + event['key1'])
    print("value2 = " + event['key2'])
    print("value3 = " + event['key3'])
    print("value4 = " + event['key4'])
    return event['key1']  # Echo back the first key value
    #raise Exception('Something went wrong')

Click the "Deploy" button and then "Test". The code will throw an error and when we go back to our Sentry dashboard we will see it captured and viewable for further inspection.

It works! Next you will likely want to tune your exception reporting criteria to make sure you get alerted to the right number of exceptions if you do not want to see all of them.

What's Next?

We just wrote and executed a Python 3 function on AWS Lambda then captured the exception message into the Sentry logs. You can now continue building out your Python code knowing that when something goes wrong you will have full visibility on what happened.

Check out the AWS Lambda section for more tutorials by other developers.

Further questions? Contact me on Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Using Django & AssemblyAI for More Accurate Twilio Call Transcriptions

Matt Makai — Mon, 13 Sep 2021 00:00:00 -0400

Recording phone calls with one or more participants is easy with Twilio's Programmable Voice API, but the speech-to-text accuracy can be poor, especially for transcription of words from niche domains such as healthcare and engineering. AssemblyAI's API for transcription provides much higher accuracy by default and through optional keyword lists. accuracy for recordings.

In this tutorial, we'll record an outbound Twilio call recording to AssemblyAI's API to get significantly more accurate speech-to-text output.

Tutorial Prerequisites

Ensure you have Python 3 installed, because Python 2 reached its end-of-life at the beginning of 2020 and is no longer supported. Preferrably, you should have Python 3.7 or greater installed in your development environment. This tutorial will also use:

We will use the following dependencies to complete this tutorial:

Django version 3.1.x, where x is the latest security release
A Twilio account and the Python Twilio helper library version 6.45.2 or newer
requests version 2.24.0
An AssemblyAI account, which you can sign up for a free key API access key here

All code in this blog post is available open source under the MIT license on GitHub under the django-accurate-twilio-voice-transcriptions directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Configuring our development environment

Change into the directory where you keep your Python virtual environments. Create a new virtualenv for this project using the following command.

Start the Django project by creating a new virtual environment using the following command. I recommend using a separate directory such as ~/venvs/ (the tilde is a shortcut for your user's home directory) so that you always know where all your virtualenvs are located.

python3 -m venv ~/venvs/djtranscribe

Activate the virtualenv with the activate shell script:

source ~/venvs/djtranscribe/bin/activate

After the above command is executed, the command prompt will change so that the name of the virtualenv is prepended to the original command prompt format, so if your prompt is just $, it will now look like the following:

(djtranscribe) $

Remember, you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the Django package into the activated but otherwise empty virtualenv.

pip install django==3.1.3 requests==2.24.0 twilio==6.45.2

Look for output similar to the following to confirm the appropriate packages were installed correctly from PyPI.

(djtranscribe) $ pip install django==3.1.3 requests==2.24.0 twilio=6.45.2
pip install django requests twilio
Collecting django
  Downloading Django-3.1.3-py3-none-any.whl (7.8 MB)
     |████████████████████████████████| 7.8 MB 2.6 MB/s 
Collecting requests
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting twilio
  Downloading twilio-6.47.0.tar.gz (460 kB)
     |████████████████████████████████| 460 kB 19.6 MB/s 
Collecting sqlparse>=0.2.2
  Downloading sqlparse-0.4.1-py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 4.8 MB/s 
Collecting pytz
  Downloading pytz-2020.4-py2.py3-none-any.whl (509 kB)
     |████████████████████████████████| 509 kB 31.0 MB/s 
Collecting asgiref<4,>=3.2.10
  Downloading asgiref-3.3.0-py3-none-any.whl (19 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Downloading urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
     |████████████████████████████████| 127 kB 24.5 MB/s 
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting PyJWT>=1.4.2
  Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
Using legacy 'setup.py install' for twilio, since package 'wheel' is not installed.
Installing collected packages: sqlparse, pytz, asgiref, django, chardet, idna, certifi, urllib3, requests, six, PyJWT, twilio
    Running setup.py install for twilio ... done
Successfully installed PyJWT-1.7.1 asgiref-3.3.0 certifi-2020.6.20 chardet-3.0.4 django-3.1.3 idna-2.10 pytz-2020.4 requests-2.24.0 six-1.15.0 sqlparse-0.4.1 twilio-6.47.0 urllib3-1.25.11

We can get started coding the application now that we have all of our required dependencies installed.

Starting our Django project

Let's begin coding our application.

We can use the Django django-admin tool to create the boilerplate code structure to get our project started. Change into the directory where you develop your applications. For example, I typically use /Users/matt/devel/py/ for all of my Python projects. Then run the following command to start a Django project named djtranscribe:

django-admin.py startproject djtranscribe

Note that in this tutorial we are using the same name for both the virtualenv and the Django project directory, but they can be different names if you prefer that for organizing your own projects.

The django-admin command creates a directory named djtranscribe along with several subdirectories that you should be familiar with if you have previously worked with Django.

Change directories into the new project.

cd djtranscribe

Create a new Django app within djtranscribe named caller.

python manage.py startapp caller

Django will generate a new folder named caller in the project. We should update the URLs so the app is accessible before we write our views.py code.

Open djtranscribe/djtranscribe/urls.py. Add the highlighted lines so that URL resolver will check the caller app for additional routes to match with URLs that are requested of this Django application.

# djtranscribe/djtranscribe/urls.py
~~from django.conf.urls import include
from django.contrib import admin
from django.urls import path


urlpatterns = [
~~    path('', include('caller.urls')),
    path('admin/', admin.site.urls),
]

Save djtranscribe/djtranscribe/urls.py and open djtranscribe/djtranscribe/settings.py. Add the caller app to settings.py by inserting the highlighted line:

# djtranscribe/djtranscribe/settings.py
# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
~~    'caller',
]

Make sure you change the default DEBUG and SECRET_KEY values in settings.py before you deploy any code to production. Secure your app properly with the information from the Django production deployment checklist so that you do not add your project to the list of hacked applications on the web.

Save and close settings.py.

Next change into the djtranscribe/caller directory. Create a new file named urls.py to contain routes for the caller app.

Add all of these lines to the empty djtranscribe/caller/urls.py file.

# djtranscribe/caller/urls.py
from django.conf.urls import url
from . import views

urlpatterns = [
    url(r'', views.index, name="index"),
]

Save djtranscribe/caller/urls.py. Open djtranscribe/caller/views.py to add the following two highlighted lines.

# djtranscribe/caller/views.py
from django.http import HttpResponse


~~def index(request):
~~    return HttpResponse('Hello, world!', 200)

We can test out that this simple boilerplate response is correct before we start adding the meat of the functionality to the project. Change into the base directory of your Django project where the manage.py file is located. Execute the development server with the following command:

python manage.py runserver

The Django development server should start up with no issues other than an unapplied migrations warning.

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

November 15, 2020 - 14:07:03
Django version 3.1.3, using settings 'djtranscribe.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Open a web browser and go to localhost:8000.

You should see 'Hello, world!' rendered in the browser. That means everything is working properly so far and we can now add the dialing, recording and transcribing capabilities to our Django project.

Adding Twilio to the Django project

Time to add Twilio's Voice API into the mix so we can dial a phone call from our Django project and make a recording out of it.

Start by opening up djtranscribe/djtranscribe/settings.py and modifying it with the following highlighted import os line:

# djtranscribe/djtranscribe/settings.py
~~import os
from pathlib import Path


# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

Then at the bottom of the settings.py file, add the following highlighted lines, which will be settings that are pulled from environment variables we will configure later.

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.1/howto/static-files/

STATIC_URL = '/static/'


~~BASE_URL = os.getenv("BASE_URL")
~~TWIML_INSTRUCTIONS_URL = "{}/record/".format(BASE_URL)
~~TWILIO_PHONE_NUMBER = os.getenv("TWILIO_PHONE_NUMBER")

Save settings.py and change into the caller Django app directory.

Update djtranscribe/caller/urls.py with the the following new code:

# djtranscribe/caller/urls.py
from django.conf.urls import url 
from . import views

urlpatterns = [ 
~~    url(r'dial/(\d+)/$', views.dial, name="dial"),
~~    url(r'record/$', views.record_twiml, name="record-twiml"),
~~    url(r'get-recording-url/([A-Za-z0-9]+)/$', views.get_recording_url,
~~        name='recording-url'),
]

Next, open djtranscribe/views.py and update it with the following code, replacing what already exists within the file:

# djtranscribe/caller/views.py
from django.conf import settings
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt

from twilio.rest import Client
from twilio.twiml.voice_response import VoiceResponse


def dial(request, phone_number):
    """Dials an outbound phone call to the number in the URL. Just
    as a heads up you will never want to leave a URL like this exposed
    without authentication and further phone number format verification.
    phone_number should be just the digits with the country code first,
    for example 14155559812."""
    # pulls credentials from environment variables
    twilio_client = Client()
    call = twilio_client.calls.create(
            to='+{}'.format(phone_number),
            from_=settings.TWILIO_PHONE_NUMBER,
            url=settings.TWIML_INSTRUCTIONS_URL,
    )   
    print(call.sid)
    return HttpResponse("dialing +{}. call SID is: {}".format(
                            phone_number, call.sid))


@csrf_exempt
def record_twiml(request):
    """Returns TwiML which prompts the caller to record a message"""
    # Start our TwiML response
    response = VoiceResponse()

    # Use <Say> to give the caller some instructions
    response.say('Ahoy! Call recording starts now.')

    # Use <Record> to record the caller's message
    response.record()

    # End the call with <Hangup>
    response.hangup()

    return HttpResponse(str(response), content_type='application/xml')


def get_recording_url(request, call_sid):
    """Returns an HttpResponse with plain text of the link to one or more
    recordings from the specified Call SID."""
    # pulls credentials from environment variables
    twilio_client = Client()
    recording_urls = ""
    call = twilio_client.calls.get(call_sid)
    for r in call.recordings.list():
        recording_urls="\n".join([recording_urls, "".join(['https://api.twilio.com', r.uri])])
    return HttpResponse(str(recording_urls), 200)

Each of the above view functions performs one of the steps needed to create a call recording of a phone call dialed by Twilio, and then retrieve it as a file. dial programmatically initiates the outbound call, record_twiml returns instructions to play a message that the call is being recorded, records it, and then hangs up when the call is done. get_recording_url only returns the URL location of the recorded phone call so that in the next step we can send the file over to AssemblyAI.

Our Django project modifications are done. Next, we need to use two services, Twilio and Ngrok, to enable some of the machine to happen of phone calling and running the application from our local machine.

Twilio credentials and environment variables

Sign up for Twilio or log into your existing account. Once you get to the Twilio Console, you can obtain your TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN on the right side of the page:

When you sign up you should have a phone number assigned to your account. You can use that or purchase a new phone number to use.

Set three environment variables with the names TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, and TWILIO_PHONE_NUMBER using the export command in your terminal. Make sure to replace the values with your own Account SID, Auth Token and Twilio phone number.

export TWILIO_ACCOUNT_SID=xxxxxxxxxxxxx    # found in twilio.com/console
export TWILIO_AUTH_TOKEN=yyyyyyyyyyyyyy    # found in twilio.com/console
export TWILIO_PHONE_NUMBER=+17166382453    # replace with your Twilio number

Note that you must use the export command in every command line window that you want this key to be accessible. The scripts we are writing will not be able to access the Twilio APIs if you do not have the tokens exported in the environment where you are running the script.

There is one more environment variable to set before we can run app.py. We need to use Ngrok as a localhost tunnel so that Twilio's webhook can send an HTTP POST request to our Django application running on our local development environment.

Run Ngrok in a new terminal window, because you will need to keep it running while we run our other Python code:

./ngrok http 8000

Copy the HTTPS version of the "Forwarding" URL and set the BASE_URL environment variable value to it. For example, in this screenshot you would set BASE_URL to https://7764c1810ad3.ngrok.io using the following command:

export BASE_URL=https://7764c1810ad3.ngrok.io    # use your ngrok URL, or domain. no trailing slash

We also need to update djtranscribe/djtranscribe/settings.py's ALLOWED_HOSTS list to include the Ngrok Forwarding URL otherwise the webhook from Twilio asking for instructions on how to handle the phone call will fail. Open the settings.py file and update the ALLOWED_HOSTS with your Ngrok Forwarding hostname list the following:

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = os.getenv('SECRET_KEY', 'development key')

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

~~ALLOWED_HOSTS = ['7764c1810ad3.ngrok.io','127.0.0.1','localhost']


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'caller',
]

Okay, we can finally re-run our Django web app. Ensure Ngrok is still running in a different window, your virtualenv is active and that in this terminal you have your four environment variables set, then run the runserver command in the root project directory where manage.py is located:

python manage.py runserver

Let's make our phone ring by testing the application.

Testing Twilio Programmable Voice Recording

We can test our application by going to localhost on port 8000. Go to this URL in your web browser, replacing the "14155551234" with the phone number you want to call, where the person on the line will be recorded: http://localhost:8000/dial/14155551234.

That number should now receive a phone call from your Twilio number. Pick up, record a message that you want to use to test the transcription, and then hang up.

If you get an error, make sure all of your environment variables are set. You can check the values by using the echo command like this:

echo $BASE_URL

When the call is over, copy the call SID show on the web page so that we can use it to look up where the recording audio file is stored.

Go to "localhost:8000/get-recording-url/" with the call SID at the end. For example, "localhost:8000/get-recording-url/CAda3f2f49ff4e8ef2be6b726edb998c92".

Copy the entire output except for the ".json" at the end, then paste it into the web browser's URL bar, prepended with "api.twilio.com". For example, "https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300". This will bring up the recording. Copy the entire URL and we will use it as input into the AssemblyAI service.

Transcribing with the AssemblyAI API

We can now use the AssemblyAI API for speech-to-text transcription on the call recording that was just made.

Sign up for an AssemblyAI account and log in to the AssemblyAI dashboard, then copy "Your API token" as shown in this screenshot:

We need to export our AssemblyAI API key as an environment variable so that our Python application can use it to authenticate with their API. We also need to pass the publicly-accessible URL for the recording, so we'll set that as an environment variable as well.

# make sure to replace this URL with the one for your recording
export ASSEMBLYAI_KEY=your-api-key-here
export RECORDING_URL=https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300

Create a new file named transcribe.py and write the following code in it:

import os
import requests

endpoint = "https://api.assemblyai.com/v2/transcript"

json = {
  "audio_url": os.getenv("RECORDING_URL")
}

headers = {
    "authorization": os.getenv("ASSEMBLYAI_KEY"),
    "content-type": "application/json"
}

response = requests.post(endpoint, json=json, headers=headers)

print(response.json())

The above code calls the AssemblyAI transcription service using the secret key and passes it the URL with the file recording. The script prints out the JSON response from the service, which will contain a transcription ID that we'll use to access the results after they finish processing.

Run the script using the python command:

python transcribe.py

You will get back some JSON as output, similar what you see here:

{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'text': None, 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'redact_pii': False, 'confidence': None, 'webhook_status_code': None, 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'status': 'queued',
'boost_param': None, 'words': None, 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': None, 'auto_highlights': False, 'word_boost': [], 'dual_channel': None, 'audio_start_from': None}

Find the value contained with the id field of the JSON response. We need that value to look up the final result of our transcription. Copy the transcription ID and set it as an environment variable to use as input by the final script:

# replace with what's found within `id` from the JSON response
export TRANSCRIPTION_ID=aksd19vwmx-82ce-476c-85a7-e82c09c67daf

We just need a little more Python that looks up the result and we'll be all done.

Retrieve the AssemblyAI Transcription

AssemblyAI will be busy transcribing the recording. Depending on the size of the file it can take anywhere from a few seconds to a few minutes for the job to complete. We can use the following code to see if the job is still in progress or it has completed. If the transcription is done it will print the results to the terminal.

Create a new file named print_transcription.py with the following code:

import os
import requests

endpoint = "https://api.assemblyai.com/v2/transcript/{}".format(os.getenv("TRANSCRIPTION_ID"))

headers = {
    "authorization": os.getenv("ASSEMBLYAI_KEY"),
}

response = requests.get(endpoint, headers=headers)

print(response.json())
print("\n\n")
print(response.json()['text'])

The code above in print_transcription.py is very similar to the code in the previous transcribe.py source file. imports os (operating system) from the Python standard library, as we did in the previous two files, to obtain the TRANSCRIPTION_ID and ASSEMBLYAI_KEY environment variable values.

The endpoint is the AssemblyAI API endpoint for retrieving transcriptions. We set the appropriate authorization header and make the API call using the requests.get function. We then print out the JSON response as well as just the text that was transcribed.

Time to test out this third file. Execute the following command in the terminal:

python print_transcription.py

Your output will be different based on your recording but you should see a result in the terminal similar to the following:

{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'auto_highlights_result': None, 'text': 'An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures.', 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'confidence': 0.931797752808989, 'webhook_status_code': None, 'status': 'completed', 'boost_param': None, 'redact_pii': False, 'words': [{'text': 'An', 'confidence': 1.0, 'end': 90, 'start': 0}, {'text': 'object', 'confidence': 0.94, 'end': 570, 'start': 210}, {'text': 'relational', 'confidence': 0.89, 'end': 1080, 'start': 510}, {'text': 'mapper', 'confidence': 0.97, 'end': 1380, 'start': 1020}, {'text': 'is', 'confidence': 0.88, 'end': 1560, 'start': 1350}, {'text': 'a', 'confidence': 0.99, 'end': 1620, 'start': 1500}, {'text': 'code', 'confidence': 0.93, 'end': 1920, 'start': 1620}, {'text': 'library', 'confidence': 0.94, 'end': 2250, 'start': 1860}, {'text': 'that', 'confidence': 0.99, 'end': 2490, 'start': 2220}, {'text': 'automates', 'confidence': 0.93, 'end': 2940, 'start': 2430}, {'text': 'the', 'confidence': 0.95, 'end': 3150, 'start': 2910}, {'text': 'transfer', 'confidence': 0.98, 'end': 3510, 'start': 3090}, {'text': 'of', 'confidence':
0.99, 'end': 3660, 'start': 3480}, {'text': 'data', 'confidence': 0.84, 'end': 3960, 'start': 3630}, {'text': 'stored', 'confidence': 0.89, 'end': 4350, 'start': 3900}, {'text': 'in', 'confidence': 0.98, 'end': 4500, 'start': 4290}, {'text': 'a', 'confidence': 0.85, 'end': 4560, 'start': 4440}, {'text': 'relational', 'confidence': 0.87, 'end': 5580, 'start': 4500}, {'text': 'database', 'confidence': 0.92, 'end':
6030, 'start': 5520}, {'text': 'tables', 'confidence': 0.93, 'end': 6330, 'start': 5970}, {'text': 'into', 'confidence': 0.92, 'end': 7130, 'start': 6560}, {'text': 'objects', 'confidence': 0.96, 'end': 7490, 'start': 7100}, {'text': 'that', 'confidence': 0.97, 'end': 7700, 'start': 7430}, {'text': 'are', 'confidence': 0.9, 'end': 7850, 'start': 7640}, {'text': 'more', 'confidence': 0.97, 'end': 8030, 'start': 7790}, {'text': 'commonly', 'confidence': 0.92, 'end': 8480, 'start': 7970}, {'text': 'used', 'confidence': 0.86, 'end': 8750, 'start': 8420}, {'text': 'in', 'confidence': 0.94, 'end': 9050, 'start': 8840}, {'text': 'application.', 'confidence': 0.98, 'end': 9860, 'start': 9110}, {'text': 'Code', 'confidence': 0.93, 'end': 10040, 'start': 9830}, {'text': 'or', 'confidence': 1.0, 'end': 11210, 'start': 10220}, {'text': 'MS', 'confidence': 0.83, 'end': 11480, 'start': 11180}, {'text': 'provide', 'confidence': 0.94, 'end': 11870, 'start': 11510}, {'text': 'a', 'confidence': 1.0, 'end': 11960, 'start': 11840}, {'text': 'high', 'confidence': 1.0, 'end': 12200, 'start': 11930}, {'text': 'level', 'confidence': 0.94, 'end': 12440, 'start': 12170}, {'text': 'abstraction', 'confidence': 0.95, 'end': 12980, 'start': 12410}, {'text':
'upon', 'confidence': 0.94, 'end': 13220, 'start': 12950}, {'text': 'a', 'confidence': 1.0, 'end': 13280, 'start': 13160}, {'text': 'relational', 'confidence': 0.94, 'end': 13820, 'start': 13280}, {'text': 'database', 'confidence': 0.95, 'end': 14210, 'start': 13790}, {'text': 'that', 'confidence': 0.96, 'end': 14420, 'start': 14150}, {'text': 'allows', 'confidence': 0.99, 'end': 14720, 'start': 14360}, {'text':
'the', 'confidence': 0.56, 'end': 14870, 'start': 14690}, {'text': 'developer', 'confidence': 0.98, 'end': 15290, 'start': 14810}, {'text': 'to', 'confidence': 0.94, 'end': 15410, 'start': 15230}, {'text': 'write', 'confidence': 0.96, 'end': 15680, 'start': 15380}, {'text': 'Python', 'confidence': 0.94, 'end': 16070, 'start': 15620}, {'text': 'code.', 'confidence': 0.98, 'end': 16310, 'start': 16070}, {'text': 'Instead', 'confidence': 0.97, 'end': 17160, 'start': 16500}, {'text': 'of', 'confidence': 0.93, 'end': 17340, 'start': 17130}, {'text': 'sequel', 'confidence': 0.86, 'end': 17820, 'start': 17280}, {'text': 'to', 'confidence': 0.91, 'end': 18090, 'start': 17880}, {'text': 'create', 'confidence': 0.89, 'end': 18450, 'start': 18090}, {'text': 'read', 'confidence': 0.88, 'end': 18840, 'start': 18480}, {'text': 'update', 'confidence': 0.92, 'end': 19290, 'start': 18870}, {'text': 'and', 'confidence': 0.94, 'end': 19590, 'start': 19230}, {'text': 'delete', 'confidence': 0.89, 'end': 19920, 'start': 19530}, {'text': 'data',
'confidence': 0.95, 'end': 20190, 'start': 19890}, {'text': 'and', 'confidence': 0.92, 'end': 20490, 'start': 20250}, {'text': 'schemas', 'confidence': 0.86, 'end': 21000, 'start': 20430}, {'text': 'in', 'confidence': 0.94, 'end': 21210, 'start': 21000}, {'text': 'their', 'confidence': 0.98, 'end': 21510, 'start': 21150}, {'text': 'database', 'confidence': 0.97, 'end': 21900, 'start': 21450}, {'text': 'developers', 'confidence': 0.83, 'end': 23200, 'start': 22420}, {'text': 'can', 'confidence': 0.95, 'end': 23440, 'start': 23200}, {'text': 'use', 'confidence': 0.97, 'end': 23650, 'start': 23410}, {'text': 'the', 'confidence': 0.99, 'end': 23890, 'start': 23590}, {'text': 'programming', 'confidence': 0.97, 'end': 24370, 'start': 23830}, {'text': 'language', 'confidence': 1.0, 'end': 24700, 'start': 24310}, {'text': 'that', 'confidence': 1.0, 'end': 24880, 'start': 24640}, {'text': 'they', 'confidence': 0.99, 'end': 25060, 'start': 24820}, {'text': 'are', 'confidence': 0.85, 'end': 25210, 'start': 25000}, {'text': 'comfortable', 'confidence': 0.92, 'end': 25780, 'start': 25180}, {'text': 'with', 'confidence': 1.0, 'end': 25960, 'start': 25720}, {'text': 'comfortable', 'confidence': 0.94, 'end': 29090, 'start': 28090}, {'text': 'to', 'confidence': 0.84, 'end': 29840, 'start': 29180}, {'text': 'work', 'confidence': 0.95, 'end': 30050, 'start': 29780}, {'text': 'with', 'confidence': 0.98, 'end': 30290, 'start': 30020}, {'text': 'the', 'confidence': 0.69, 'end': 30440, 'start': 30230}, {'text': 'database', 'confidence': 0.98, 'end': 30860, 'start': 30380}, {'text': 'instead', 'confidence': 1.0, 'end': 32780, 'start': 31780}, {'text': 'of', 'confidence': 0.98, 'end': 32900, 'start': 32720}, {'text': 'writing', 'confidence': 0.87, 'end': 33320, 'start': 32870}, {'text': 'sequel', 'confidence': 0.88, 'end': 33860, 'start': 33290}, {'text': 'statements', 'confidence': 0.95, 'end': 34310, 'start': 33800}, {'text': 'or', 'confidence': 0.9, 'end': 34460, 'start': 34250}, {'text': 'short', 'confidence': 0.9, 'end': 34790, 'start': 34430}, {'text': 'procedures.', 'confidence': 0.98, 'end': 35270, 'start': 34760}], 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': 36.288, 'auto_highlights': False, 'word_boost': [],
'dual_channel': None, 'audio_start_from': None}


An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures.

That's a lot of output. The first part contains the results of the transcription and the confidence in the accuracy of each word transcribed. The second part is just the plain text output from the transcription.

You can take this now take this base code and add it to any application that needs high quality text-to-speech transcription. If the results aren't quite good enough for you yet, check out this tutorial on boosting accuracy for keywords or phrases.

Additional resources

We just finished building a highly accurate transcription application for recordings.

Next, try out some of these other related Django tutorials:

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai. See something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Higher Accuracy Twilio Voice Transcriptions with Python and Flask

Matt Makai — Sat, 10 Oct 2020 00:00:00 -0400

Twilio's Programmable Voice API is commonly used to initiate and receive phone calls, but the transcription accuracy for recordings often leaves a lot to be desired. In this tutorial, we'll see how to connect an outbound phone call powered by the Twilio Voice API with AssemblyAI's deep learning transcription API to get significantly more accurate speech-to-text output.

Required Tools for this Application

Ensure you have Python 3 installed, because Python 2 reached its end-of-life at the beginning of 2020 and is no longer supported. Preferrably, you should have Python 3.6 or newer installed in your development environment. This tutorial will also use:

We will use the following dependencies to complete this tutorial:

requests, version 2.24.0, for accessing the AssemblyAI transcription API
Flask, version 1.1.2, to respond to Twilio's webhooks
A Twilio account, of which a free trial version is good enough to test this tutorial
Twilio Python helper library, version 6.45.4 or newer, for interacting with the REST API
An AssemblyAI account, which you can sign up for a free key API access key here
Ngrok if you need a localhost tunnel to expose a public URL that webhooks can send a POST request to

All code in this blog post is available open source under the MIT license on GitHub under the accurate-twilio-voice-call-recording-transcriptions-assemblyai directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Configuring our development environment

Change into the directory where you keep your Python virtual environments. Create a new virtualenv for this project using the following command.

Start this Python project by creating a new virtual environment using the following command. I recommend using a separate directory such as ~/venvs/ (the tilde is a shortcut for your user's home directory) so that you always know where all your virtualenvs are located.

python3 -m venv ~/venvs/record-transcribe

Activate the virtualenv with the activate shell script:

source ~/venvs/record-transcribe/bin/activate

After the above command is executed, the command prompt will change so that the name of the virtualenv is prepended to the original command prompt format, so if your prompt is simply $, it will now look like the following:

(record-transcribe) $

Remember, you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the required packages package into the activated but otherwise empty virtualenv.

pip install Flask==1.1.2 requests==2.24.0 twilio==6.45.4

Look for output similar to the following to confirm the appropriate packages were installed correctly from PyPI.

(recordtranscribe) $ pip install Flask==1.1.2 requests==2.24.0 twilio=6.45.4
Collecting Flask
  Using cached https://files.pythonhosted.org/packages/f2/28/2a03252dfb9ebf377f40fba6a7841b47083260bf8bd8e737b0c6952df83f/Flask-1.1.2-py2.py3-none-any.whl
Collecting requests
  Using cached https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl
Collecting twilio
  Using cached https://files.pythonhosted.org/packages/d0/4e/7c377eb1a1d57f011dc1bee2fee77cf1e9a08407b8d44ea25a187a30c78d/twilio-6.45.4.tar.gz
Collecting Werkzeug>=0.15 (from Flask)
  Using cached https://files.pythonhosted.org/packages/cc/94/5f7079a0e00bd6863ef8f1da638721e9da21e5bacee597595b318f71d62e/Werkzeug-1.0.1-py2.py3-none-any.whl
Collecting itsdangerous>=0.24 (from Flask)
  Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting click>=5.1 (from Flask)
  Using cached https://files.pythonhosted.org/packages/d2/3d/fa76db83bf75c4f8d338c2fd15c8d33fdd7ad23a9b5e57eb6c5de26b430e/click-7.1.2-py2.py3-none-any.whl
Collecting Jinja2>=2.10.1 (from Flask)
  Using cached https://files.pythonhosted.org/packages/30/9e/f663a2aa66a09d838042ae1a2c5659828bb9b41ea3a6efa20a20fd92b121/Jinja2-2.11.2-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl
Collecting idna<3,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl
Collecting chardet<4,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting six (from twilio)
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting pytz (from twilio)
  Using cached https://files.pythonhosted.org/packages/4f/a4/879454d49688e2fad93e59d7d4efda580b783c745fd2ec2a3adf87b0808d/pytz-2020.1-py2.py3-none-any.whl
Collecting PyJWT>=1.4.2 (from twilio)
  Using cached https://files.pythonhosted.org/packages/87/8b/6a9f14b5f781697e51259d81657e6048fd31a113229cf346880bb7545565/PyJWT-1.7.1-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10.1->Flask)
  Using cached https://files.pythonhosted.org/packages/0c/12/37f68957526d1ec0883b521934b4e1b8ff3dd8e4fab858a5bf3e487bcee9/MarkupSafe-1.1.1-cp38-cp38-macosx_10_9_x86_64.whl
Installing collected packages: Werkzeug, itsdangerous, click, MarkupSafe, Jinja2, Flask, urllib3, idna, certifi, chardet, requests, six, pytz, PyJWT, twilio
  Running setup.py install for twilio ... done
Successfully installed Flask-1.1.2 Jinja2-2.11.2 MarkupSafe-1.1.1 PyJWT-1.7.1 Werkzeug-1.0.1 certifi-2020.6.20 chardet-3.0.4 click-7.1.2 idna-2.10 itsdangerous-1.1.0 pytz-2020.1 requests-2.24.0 six-1.15.0 twilio-6.45.4 urllib3-1.25.10

We can get started coding the application now that we have all of our required dependencies installed.

Building our application

Time to dig into the code! We're going to write three source files in this application:

app.py: a Flask app that will handle the phone call and recording
transcribe.py: a short Python script to invoke AssemblyAI with the recording and start the transcription process
print_transcription.py: a script to print the output of the transcription to the terminal

Remember that you can get access to all three of the completed files in the accurate-twilio-voice-call-recording-transcriptions-assemblyai directory of the blog-code-examples Git repository if you do not want to type or copy from the blog post itself.

Create a new directory named record-transcribe to store your source files and change into the new directory.

mkdir record-transcribe
cd record-transcribe

Create a new file named app.py with the following code:

import os                                                                                                                                                                                                                                     
from flask import Flask, request
from twilio.twiml.voice_response import VoiceResponse
from twilio.rest import Client


app = Flask(__name__)

# pulls credentials from environment variables
client = Client()

BASE_URL = os.getenv("BASE_URL")
twiml_instructions_url = "{}/record".format(BASE_URL)
recording_callback_url = "{}/callback".format(BASE_URL)
twilio_phone_number = os.getenv("TWILIO_PHONE_NUMBER")


@app.route("/record", methods=["GET", "POST"])
def record():
    """Returns TwiML which prompts the caller to record a message"""
    # Start our TwiML response
    response = VoiceResponse()

    # Use <Say> to give the caller some instructions
    response.say('Ahoy! Call recording starts now.')

    # Use <Record> to record the caller's message
    response.record()

    # End the call with <Hangup>
    response.hangup()

    return str(response)

There are a couple more functions we'll need to add to app.py but first let's take a look at what the above code does.

We imported parts of both the Flask and Twilio helper libraries, which will enable us to programmatically create and control phone calls that Twilio records. Note that when we instantiate the Twilio helper library with the empty Client() constructor, it automatically looks to read two environment variables, TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN to gain appropriate permissions to your Twilio account. If those two environment variables are not set with those exact names then you will need to explicitly pass the Account SID and Auth Token for your account into the constructor.

After the import are the Flask and Twilio library instantiations. Then we configure the BASE_URL by reading from an environment variable. In this tutorial the BASE_URL will be from Ngrok, but it can also be your domain where your application is deployed, such as "https://www.twilio.com". We have not yet set these environment variables, but we will shortly after we finish writing app.py.

After setting BASE_URL, and the three other variables set by environment variables, we have the record function. This function is a Flask route that generates the TwiML that tells Twilio how to handle a phone call. First, an automated voice alerts the person who picks up that the phone call is being recorded. Then the recording starts. Whatever the person on the call says will be recorded and stored by Twilio.

Finish app.py by adding these two following functions after the record function:

@app.route("/dial/<int:phone_number>")
def dial(phone_number):
    """Dials an outbound phone call to the number in the URL. Just
    as a heads up you will never want to leave a URL like this exposed
    without authentication and further phone number format verification.
    phone_number should be just the digits with the country code first,
    for example 14155559812."""
    call = client.calls.create(
            to='+{}'.format(phone_number),
            from_=twilio_phone_number,
            url=twiml_instructions_url,
    )
    print(call.sid)
    return "dialing +{}. call SID is: {}".format(phone_number, call.sid)


@app.route("/get-recording-url/<call_sid>")
def get_recording_url(call_sid):
    recording_urls = ""
    call = client.calls.get(call_sid)
    for r in call.recordings.list():
        recording_urls="\n".join([recording_urls, r.uri])
    return str(recording_urls)

The dial function creates a Flask route that takes a phone number input as part of the second level path. Note that in a production application you must have better phone number validation or you will have a security issue with unsanitized inputs. We are doing this here to easily grab a phone number as input rather than having to build a whole user interface with an HTML form just to grab a phone number. dial calls the Twilio Voice API using our Twilio account credentials so that we can dial an outbound phone call to the number sent in through the URL. The twiml_instructions_url should be set to the record function URL so that it can give the proper dialing and recording TwiML instructions for how Twilio's service should handle dialing the phone call.

Once we dial the outbound phone call, the call SID is printed to the terminal. We'll need that call SID to get the recording after the call is finished.

Our app.py file is all done. We just need to export our environment variables for our Twilio credentials.

Sign up for Twilio or log into your existing account. Once you get to the Twilio Console, you can obtain your TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN on the right side of the page:

When you sign up you should have a phone number assigned to your account. You can use that or purchase a new phone number to use.

export TWILIO_ACCOUNT_SID=xxxxxxxxxxxxx    # found in twilio.com/console
export TWILIO_AUTH_TOKEN=yyyyyyyyyyyyyy    # found in twilio.com/console
export TWILIO_PHONE_NUMBER=+17166382453    # replace with your Twilio number

There is one more environment variable to set before we can run app.py. We need to use Ngrok as a localhost tunnel so that Twilio's webhook can send an HTTP POST request to our app.py Flask application running on our local development environment.

Run Ngrok in a new terminal window, because you will need to keep it running while we run our other Python code:

./ngrok http 5000

Copy the HTTPS version of the "Forwarding" URL and set the BASE_URL environment variable value to it. For example, in this screenshot you would set BASE_URL to https://7f9139eaf445.ngrok.io using the following command:

export BASE_URL=https://7f9139eaf445.ngrok.io    # use your ngrok URL, or domain. no trailing slash

Okay, we can finally run app.py. Make sure you are still running Ngrok in a different window, your virtualenv is active and that in this terminal you have your four environment variables set, then run the flask run command:

flask run

You should see Flask output something like the following text:

 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

That is a legitimate warning: only use this command for development purposes and when you want to deploy to production you need to use a real WSGI server like Gunicorn.

Time to test out our application.

Testing Twilio Programmable Voice Recording

We can test our application by going to localhost on port 5000. Go to this URL in your web browser, replacing the "14155551234" with the phone number you want to call, where the person on the line will be recorded: http://localhost:5000/dial/14155551234.

That number should now receive a phone call from your Twilio number. Pick up, record a message that you want to use to test the transcription, and then hang up.

If you get an error, make sure all of your environment variables are set. You can check the values by using the echo command like this:

echo $BASE_URL

When the call is over, copy the call SID show on the web page so that we can use it to look up where the recording audio file is stored.

Go to "localhost:5000/get-recording-url/" with the call SID at the end. For example, "localhost:5000/get-recording-url/CAda3f2f49ff4e8ef2be6b726edb998c92".

Transcribing with the AssemblyAI API

We can now use the AssemblyAI API for speech-to-text transcription on the call recording that was just made.

Sign up for an AssemblyAI account and log in to the AssemblyAI dashboard, then copy "Your API token" as shown in this screenshot:

# make sure to replace this URL with the one for your recording
export ASSEMBLYAI_KEY=your-api-key-here
export RECORDING_URL=https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300

Create a new file named transcribe.py and write the following code in it:

import os
import requests

endpoint = "https://api.assemblyai.com/v2/transcript"

json = {
  "audio_url": os.getenv("RECORDING_URL")
}

headers = {
    "authorization": os.getenv("ASSEMBLYAI_KEY"),
    "content-type": "application/json"
}

response = requests.post(endpoint, json=json, headers=headers)

print(response.json())

Run the script using the python command:

python transcribe.py

You will get back some JSON as output, similar what you see here:

{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'text': None, 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'redact_pii': False, 'confidence': None, 'webhook_status_code': None, 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'status': 'queued',
'boost_param': None, 'words': None, 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': None, 'auto_highlights': False, 'word_boost': [], 'dual_channel': None, 'audio_start_from': None}

# replace with what's found within `id` from the JSON response
export TRANSCRIPTION_ID=aksd19vwmx-82ce-476c-85a7-e82c09c67daf

We just need a little more Python that looks up the result and we'll be all done.

Retrieve the AssemblyAI Transcription

Create a new file named print_transcription.py with the following code:

import os
import requests

endpoint = "https://api.assemblyai.com/v2/transcript/{}".format(os.getenv("TRANSCRIPTION_ID"))

headers = {
    "authorization": os.getenv("ASSEMBLYAI_KEY"),
}

response = requests.get(endpoint, headers=headers)

print(response.json())
print("\n\n")
print(response.json()['text'])

The endpoint is simply the AssemblyAI API endpoint for retrieving transcriptions. We set the appropriate authorization header and make the API call using the requests.get function. We then print out the JSON response as well as just the text that was transcribed.

Time to test out this third file. Execute the following command in the terminal:

python print_transcription.py

Your output will be different based on your recording but you should see a result in the terminal similar to the following:

{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'auto_highlights_result': None, 'text': 'An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures.', 'audio_url': 'https://api.twilio.com/2010-04-01/Accounts/ACe3737affa0d2e17561ad44c9d190e70c/Recordings/RE3b42cf470bef829c3680ded961a09300', 'speed_boost': False, 'language_model': 'assemblyai_default', 'id': 'zibe9vwmx-82ce-476c-85a7-e82c09c67daf', 'confidence': 0.931797752808989, 'webhook_status_code': None, 'status': 'completed', 'boost_param': None, 'redact_pii': False, 'words': [{'text': 'An', 'confidence': 1.0, 'end': 90, 'start': 0}, {'text': 'object', 'confidence': 0.94, 'end': 570, 'start': 210}, {'text': 'relational', 'confidence': 0.89, 'end': 1080, 'start': 510}, {'text': 'mapper', 'confidence': 0.97, 'end': 1380, 'start': 1020}, {'text': 'is', 'confidence': 0.88, 'end': 1560, 'start': 1350}, {'text': 'a', 'confidence': 0.99, 'end': 1620, 'start': 1500}, {'text': 'code', 'confidence': 0.93, 'end': 1920, 'start': 1620}, {'text': 'library', 'confidence': 0.94, 'end': 2250, 'start': 1860}, {'text': 'that', 'confidence': 0.99, 'end': 2490, 'start': 2220}, {'text': 'automates', 'confidence': 0.93, 'end': 2940, 'start': 2430}, {'text': 'the', 'confidence': 0.95, 'end': 3150, 'start': 2910}, {'text': 'transfer', 'confidence': 0.98, 'end': 3510, 'start': 3090}, {'text': 'of', 'confidence':
0.99, 'end': 3660, 'start': 3480}, {'text': 'data', 'confidence': 0.84, 'end': 3960, 'start': 3630}, {'text': 'stored', 'confidence': 0.89, 'end': 4350, 'start': 3900}, {'text': 'in', 'confidence': 0.98, 'end': 4500, 'start': 4290}, {'text': 'a', 'confidence': 0.85, 'end': 4560, 'start': 4440}, {'text': 'relational', 'confidence': 0.87, 'end': 5580, 'start': 4500}, {'text': 'database', 'confidence': 0.92, 'end':
6030, 'start': 5520}, {'text': 'tables', 'confidence': 0.93, 'end': 6330, 'start': 5970}, {'text': 'into', 'confidence': 0.92, 'end': 7130, 'start': 6560}, {'text': 'objects', 'confidence': 0.96, 'end': 7490, 'start': 7100}, {'text': 'that', 'confidence': 0.97, 'end': 7700, 'start': 7430}, {'text': 'are', 'confidence': 0.9, 'end': 7850, 'start': 7640}, {'text': 'more', 'confidence': 0.97, 'end': 8030, 'start': 7790}, {'text': 'commonly', 'confidence': 0.92, 'end': 8480, 'start': 7970}, {'text': 'used', 'confidence': 0.86, 'end': 8750, 'start': 8420}, {'text': 'in', 'confidence': 0.94, 'end': 9050, 'start': 8840}, {'text': 'application.', 'confidence': 0.98, 'end': 9860, 'start': 9110}, {'text': 'Code', 'confidence': 0.93, 'end': 10040, 'start': 9830}, {'text': 'or', 'confidence': 1.0, 'end': 11210, 'start': 10220}, {'text': 'MS', 'confidence': 0.83, 'end': 11480, 'start': 11180}, {'text': 'provide', 'confidence': 0.94, 'end': 11870, 'start': 11510}, {'text': 'a', 'confidence': 1.0, 'end': 11960, 'start': 11840}, {'text': 'high', 'confidence': 1.0, 'end': 12200, 'start': 11930}, {'text': 'level', 'confidence': 0.94, 'end': 12440, 'start': 12170}, {'text': 'abstraction', 'confidence': 0.95, 'end': 12980, 'start': 12410}, {'text':
'upon', 'confidence': 0.94, 'end': 13220, 'start': 12950}, {'text': 'a', 'confidence': 1.0, 'end': 13280, 'start': 13160}, {'text': 'relational', 'confidence': 0.94, 'end': 13820, 'start': 13280}, {'text': 'database', 'confidence': 0.95, 'end': 14210, 'start': 13790}, {'text': 'that', 'confidence': 0.96, 'end': 14420, 'start': 14150}, {'text': 'allows', 'confidence': 0.99, 'end': 14720, 'start': 14360}, {'text':
'the', 'confidence': 0.56, 'end': 14870, 'start': 14690}, {'text': 'developer', 'confidence': 0.98, 'end': 15290, 'start': 14810}, {'text': 'to', 'confidence': 0.94, 'end': 15410, 'start': 15230}, {'text': 'write', 'confidence': 0.96, 'end': 15680, 'start': 15380}, {'text': 'Python', 'confidence': 0.94, 'end': 16070, 'start': 15620}, {'text': 'code.', 'confidence': 0.98, 'end': 16310, 'start': 16070}, {'text': 'Instead', 'confidence': 0.97, 'end': 17160, 'start': 16500}, {'text': 'of', 'confidence': 0.93, 'end': 17340, 'start': 17130}, {'text': 'sequel', 'confidence': 0.86, 'end': 17820, 'start': 17280}, {'text': 'to', 'confidence': 0.91, 'end': 18090, 'start': 17880}, {'text': 'create', 'confidence': 0.89, 'end': 18450, 'start': 18090}, {'text': 'read', 'confidence': 0.88, 'end': 18840, 'start': 18480}, {'text': 'update', 'confidence': 0.92, 'end': 19290, 'start': 18870}, {'text': 'and', 'confidence': 0.94, 'end': 19590, 'start': 19230}, {'text': 'delete', 'confidence': 0.89, 'end': 19920, 'start': 19530}, {'text': 'data',
'confidence': 0.95, 'end': 20190, 'start': 19890}, {'text': 'and', 'confidence': 0.92, 'end': 20490, 'start': 20250}, {'text': 'schemas', 'confidence': 0.86, 'end': 21000, 'start': 20430}, {'text': 'in', 'confidence': 0.94, 'end': 21210, 'start': 21000}, {'text': 'their', 'confidence': 0.98, 'end': 21510, 'start': 21150}, {'text': 'database', 'confidence': 0.97, 'end': 21900, 'start': 21450}, {'text': 'developers', 'confidence': 0.83, 'end': 23200, 'start': 22420}, {'text': 'can', 'confidence': 0.95, 'end': 23440, 'start': 23200}, {'text': 'use', 'confidence': 0.97, 'end': 23650, 'start': 23410}, {'text': 'the', 'confidence': 0.99, 'end': 23890, 'start': 23590}, {'text': 'programming', 'confidence': 0.97, 'end': 24370, 'start': 23830}, {'text': 'language', 'confidence': 1.0, 'end': 24700, 'start': 24310}, {'text': 'that', 'confidence': 1.0, 'end': 24880, 'start': 24640}, {'text': 'they', 'confidence': 0.99, 'end': 25060, 'start': 24820}, {'text': 'are', 'confidence': 0.85, 'end': 25210, 'start': 25000}, {'text': 'comfortable', 'confidence': 0.92, 'end': 25780, 'start': 25180}, {'text': 'with', 'confidence': 1.0, 'end': 25960, 'start': 25720}, {'text': 'comfortable', 'confidence': 0.94, 'end': 29090, 'start': 28090}, {'text': 'to', 'confidence': 0.84, 'end': 29840, 'start': 29180}, {'text': 'work', 'confidence': 0.95, 'end': 30050, 'start': 29780}, {'text': 'with', 'confidence': 0.98, 'end': 30290, 'start': 30020}, {'text': 'the', 'confidence': 0.69, 'end': 30440, 'start': 30230}, {'text': 'database', 'confidence': 0.98, 'end': 30860, 'start': 30380}, {'text': 'instead', 'confidence': 1.0, 'end': 32780, 'start': 31780}, {'text': 'of', 'confidence': 0.98, 'end': 32900, 'start': 32720}, {'text': 'writing', 'confidence': 0.87, 'end': 33320, 'start': 32870}, {'text': 'sequel', 'confidence': 0.88, 'end': 33860, 'start': 33290}, {'text': 'statements', 'confidence': 0.95, 'end': 34310, 'start': 33800}, {'text': 'or', 'confidence': 0.9, 'end': 34460, 'start': 34250}, {'text': 'short', 'confidence': 0.9, 'end': 34790, 'start': 34430}, {'text': 'procedures.', 'confidence': 0.98, 'end': 35270, 'start': 34760}], 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': 36.288, 'auto_highlights': False, 'word_boost': [],
'dual_channel': None, 'audio_start_from': None}


An object relational mapper is a code library that automates the transfer of data stored in a relational database tables into objects that are more commonly used in application. Code or MS provide a high level abstraction upon a relational database that allows the developer to write Python code. Instead of sequel to create read update and delete data and schemas in their database developers can use the programming language that they are comfortable with comfortable to work with the database instead of writing sequel statements or short procedures.

You can take this now take this base code and add it to any application that needs high quality text-to-speech transcription. If the results aren't quite good enough for you, check out this tutorial on boosting accuracy for keywords or phrases as well as better matching your data with topic detection.

What now?

We just finished building a highly accurate transcription application for recordings.

Next, try out some of these other related Python tutorials:

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai. If you see an issue or error in this tutorial, please fork the source repository on GitHub and submit a pull request with the fix.

Using Sentry to Handle Python Exceptions in Django Projects

Matt Makai — Sun, 23 Aug 2020 00:00:00 -0400

Web applications built in Django can become sprawlingly complex over time, which is one reason why centralized error handling is important. This tutorial will guide you through adding a free, basic Sentry configuration to a new Django project.

When we're done, you will be able to view centralized error reports in the Sentry dashboard like you see in this screenshot:

Tutorial Requirements

Throughout this tutorial we are going to use the following dependencies, which we will install in just a moment. Make sure you also have Python 3, preferrably 3.7 or newer installed, in your environment:

We will use the following dependencies to complete this tutorial:

All code in this blog post is available open source under the MIT license on GitHub under the sentry-handle-exceptions-django-projects directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Development environment configuration

Change into the directory where you keep your Python virtual environments. Create a new virtualenv for this project using the following command.

python3 -m venv ~/venvs/djsentry

Activate the virtualenv with the activate shell script:

source ~/venvs/djsentry/bin/activate

(djsentry) $

Remember, you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the Django package into the activated but otherwise empty virtualenv.

pip install django==3.1 sentry-sdk==0.16.5

Look for output similar to the following to confirm the appropriate packages were installed correctly from PyPI.

(djsentry) $ pip install django==3.1 sentry-sdk==0.16.5
Collecting django
  Downloading https://files.pythonhosted.org/packages/2b/5a/4bd5624546912082a1bd2709d0edc0685f5c7827a278d806a20cf6adea28/Django-3.1-py3-none-any.whl (7.8MB)
    100% |████████████████████████████████| 7.8MB 6.3MB/s 
Collecting sentry-sdk
  Downloading https://files.pythonhosted.org/packages/f4/4c/49f899856e3a83e02bc88f2c4945aa0bda4f56b804baa9f71e6664a574a2/sentry_sdk-0.16.5-py2.py3-none-any.whl (113kB)
    100% |████████████████████████████████| 122kB 33.7MB/s 
Collecting asgiref~=3.2.10 (from django)
  Using cached https://files.pythonhosted.org/packages/d5/eb/64725b25f991010307fd18a9e0c1f0e6dff2f03622fc4bcbcdb2244f60d6/asgiref-3.2.10-py3-none-any.whl
Collecting sqlparse>=0.2.2 (from django)
  Using cached https://files.pythonhosted.org/packages/85/ee/6e821932f413a5c4b76be9c5936e313e4fc626b33f16e027866e1d60f588/sqlparse-0.3.1-py2.py3-none-any.whl
Collecting pytz (from django)
  Using cached https://files.pythonhosted.org/packages/4f/a4/879454d49688e2fad93e59d7d4efda580b783c745fd2ec2a3adf87b0808d/pytz-2020.1-py2.py3-none-any.whl
Collecting urllib3>=1.10.0 (from sentry-sdk)
  Using cached https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl
Collecting certifi (from sentry-sdk)
  Using cached https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl
Installing collected packages: asgiref, sqlparse, pytz, django, urllib3, certifi, sentry-sdk
Successfully installed asgiref-3.2.10 certifi-2020.6.20 django-3.1 pytz-2020.1 sentry-sdk-0.16.5 sqlparse-0.3.1 urllib3-1.25.10

We can get started coding the application now that we have all of our required dependencies installed.

Coding the initial application

We have everything we need to start building our application.

django-admin.py startproject djsentry

Note that in this tutorial we are using the same name for both the virtualenv and the Django project directory, but they can be different names if you prefer that for organizing your own projects.

The django-admin command creates a directory named djsentry along with several subdirectories that you should be familiar with if you have previously worked with Django.

Change directories into the new project.

cd djsentry

Create a new Django app within djsentry.

python manage.py startapp errors

Django will generate a new folder named errors for the project. We should update the URLs so the app is accessible before we write our views.py code.

Open djsentry/djsentry/urls.py. Add the highlighted lines so that URL resolver will check the errors app for additional routes to match with URLs that are requested of this Django application.

# djsentry/djsentry/urls.py
~~from django.conf.urls import include
from django.contrib import admin
from django.urls import path


urlpatterns = [
~~    path('', include('errors.urls')),
    path('admin/', admin.site.urls),
]

Save djsentry/djsentry/urls.py and open djsentry/djsentry/settings.py. Add the errors app to settings.py by inserting the highlighted line:

# djsentry/djsentry/settings.py
# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
~~    'errors',
]

Save and close settings.py.

Next change into the djsentry/errors directory. Create a new file named urls.py to contain routes for the errors app.

Add all of these lines to the empty djsentry/errors/urls.py file.

# djsentry/errors/urls.py
from django.conf.urls import url
from . import views

urlpatterns = [
    url(r'^$', views.errors_index, name="index"),
]

Save djsentry/errors/urls.py. Open djsentry/errors/views.py to add the following two highlighted lines. You can keep the boilerplate comment "# Create your views here." or delete like I usually do.

# djsentry/errors/views.py
from django.shortcuts import render


~~def errors_index(request):
~~    return render(request, 'index.html', {})

Next, create a directory for your template files named templates under the djmaps/maps app directory.

mkdir templates

Create a new file named index.html within djsentry/errors/templates that contains the following Django template language markup.

<!DOCTYPE html>
<html>
  <head>
    <title>First step for errors</title>
  </head>
  <body>
   <h1>Hello, world!</h1>
  </body>
</html>

We can test out this static page to make sure all of our code is correct before we start adding the meat of the functionality to the project. Change into the base directory of your Django project where the manage.py file is located. Execute the development server with the following command:

python manage.py runserver

The Django development server should start up with no issues other than an unapplied migrations warning.

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 18 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.
August 15, 2020 - 17:26:57
Django version 3.1, using settings 'djsentry.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Open a web browser and go to localhost:8000.

Our code works, but it sure does not do much yet. Let's add sentry-sdk so we can understand how it works.

Adding Sentry and the sentry-sdk library

We can now add Sentry and test it with a bunch of errors to make sure it is working properly.

Sentry can either be self-hosted or used as a cloud service through Sentry.io. In this tutorial we will use the cloud hosted version because it's faster than setting up your own server as well as free for smaller projects.

Go to Sentry.io's homepage.

Sign into your account or sign up for a new free account. You will be at the main account dashboard after logging in or completing the Sentry sign up process.

There are no errors logged on our account dashboard yet, which is as expected because we have not yet connected our account to our Django project.

Create a new Sentry Project just for this application by clicking "Projects" in the left sidebar to go to the Projects page.

On the Projects page, click the "Create Project" button in the top right corner of the page.

You can either choose "Django" or select "Python". I usually just choose "Python" if I do not yet know what framework I'll be using to build my application. Next, give your new Project a name and then press the "Create Project" button. Our new project is ready to integrate with our Python code.

We need the unique identifier for our account and project to authorize our Python code to send errors to this Sentry instance. The easiest way to get what we need is to go to the Python+Django documentation page and read how to configure the SDK.

Copy the string parameter for the init method and set it as an environment variable rather than having it exposed in your project's code.

export SENTRY_DSN='https://yourkeygoeshere.ingest.sentry.io/project-number'

Make sure to replace "yourkeygoeshere" with your own unique identifier and "project-number" with the ID that matches the project you just created.

Check that the SENTRY_DSN is set properly in your shell using the echo command:

echo $SENTRY_DSN

Next, update settings.py with the following highlighted new lines:

# settings.py
~~import os
~~import sentry_sdk

from pathlib import Path
~~from sentry_sdk.integrations.django import DjangoIntegration


# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve(strict=True).parent.parent

At the bottom of the file after the line with STATIC_URL, add the Sentry configuration:

STATIC_URL = '/static/'

~~sentry_sdk.init(
~~    dsn=os.getenv('SENTRY_DSN'),
~~    integrations=[DjangoIntegration()],

~~    # If you wish to associate users to errors (assuming you are using
~~    # django.contrib.auth) you may enable sending PII data.
~~    send_default_pii=True
~~)

Now that we have the configuration in place we can deliberately make some errors happen to test the connection to Sentry's service.

Testing Sentry's error catching

We'll change some of the existing code to deliberately throw exceptions to make sure everything is working properly.

Start by opening errors/views.py and updating it with one new highlighted line that will automatically throw a generic Exception when this function is called.

# djsentry/errors/views.py
from django.shortcuts import render


def errors_index(request):
~~    raise Exception('testing exception')
    return render(request, 'index.html', {})

Go to localhost:8000 in your browser and you will immediately get this exception page when running the development server:

We can also try out code that does not simply raise an exception but instead will definitely create one when executed, like this division by zero operation:

# djsentry/errors/views.py
from django.shortcuts import render


def errors_index(request):
~~    division_by_zero = 1 / 0
    return render(request, 'index.html', {})

If those exceptions both appear in the Sentry dashboard like this, you're all set:

The above exceptions were just a couple of generic ways to test that everything is working to send error information to Sentry. This configuration will also handle the many other Django exceptions you are likely to see when building the rest of your Django project.

Additional resources

We just finished building a Django project that uses Sentry for centralized error handling.

Next, try out some of these other related Django tutorials:

If you have questions or comments about this tutorial, please contact me via Twitter @fullstackpython, or on GitHub @mattmakai. See something wrong with this post? Fork this page's source on GitHub and submit a pull request.

How to Transcribe Speech Recordings into Text with Python

Matt Makai — Mon, 13 Sep 2021 00:00:00 -0400

When you have a recording where one or more people are talking, it's useful to have a highly accurate and automated way to extract the spoken words into text. Once you have the text, you can use it for further analysis or as an accessibility feature.

In this tutorial, we'll use a high accuracy speech-to-text web application programming interface called AssemblyAI to extract text from an MP3 recording (many other formats are supported as well).

With the code from this tutorial, you will be able to take an audio file that contains speech such as this example one I recorded and output a highly accurate text transcription like this:

An object relational mapper is a code library that automates the transfer of 
data stored in relational, databases into objects that are more commonly used
in application code or EMS are useful because they provide a high level 
abstraction upon a relational database that allows developers to write Python 
code instead of sequel to create read update and delete, data and schemas in 
their database. Developers can use the programming language. They are 
comfortable with to work with a database instead of writing SQL...

(the text goes on from here but I abbreviated it at this point)

Tutorial requirements

Throughout this tutorial we are going to use the following dependencies, which we will install in just a moment. Make sure you also have Python 3, preferably 3.6 or newer installed, in your environment:

We will use the following dependencies to complete this tutorial:

requests version 2.24.0 to make HTTP requests to the AssemblyAI speech-to-text API
An AssemblyAI account, which you can sign up for a free API access key here

All code in this blog post is available open source under the MIT license on GitHub under the transcribe-speech-text-script directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Setting up the development environment

Change into the directory where you keep your Python virtual environments. I keep mine in a subdirectory named venvs within my user's home directory. Create a new virtualenv for this project using the following command.

python3 -m venv ~/venvs/pytranscribe

Activate the virtualenv with the activate shell script:

source ~/venvs/pytranscribe/bin/activate

(pytranscribe) $

Remember, you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the requests package into the activated but otherwise empty virtualenv.

pip install requests==2.24.0

Look for output similar to the following to confirm the appropriate packages were installed correctly from PyPI.

(pytranscribe) $ pip install requests==2.24.0
Collecting requests==2.24.0
  Using cached https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests==2.24.0)
  Using cached https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests==2.24.0)
  Using cached https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl
Collecting chardet<4,>=3.0.2 (from requests==2.24.0)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<3,>=2.5 (from requests==2.24.0)
  Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl
Installing collected packages: certifi, urllib3, chardet, idna, requests
Successfully installed certifi-2020.6.20 chardet-3.0.4 idna-2.10 requests-2.24.0 urllib3-1.25.10

We have all of our required dependencies installed so we can get started coding the application.

Uploading, initiating and transcribing audio

We have everything we need to start building our application that will transcribe audio into text. We're going to build this application in three files:

upload_audio_file.py: uploads your audio file to a secure place on AssemblyAI's service so it can be access for processing. If your audio file is already accessible with a public URL, you don't need to do this step, you can just follow this quickstart
initiate_transcription.py: tells the API which file to transcribe and to start immediately
get_transcription.py: prints the status of the transcription if it is still processing, or displays the results of the transcription when the process is complete

Create a new directory named pytranscribe to store these files as we write them. Then change into the new project directory.

mkdir pytranscribe
cd pytranscribe

We also need to export our AssemblyAI API key as an environment variable. Sign up for an AssemblyAI account and log in to the AssemblyAI dashboard, then copy "Your API token" as shown in this screenshot:

export ASSEMBLYAI_KEY=your-api-key-here

Note that you must use the export command in every command line window that you want this key to be accessible. The scripts we are writing will not be able to access the API if you do not have the token exported as ASSEMBLYAI_KEY in the environment you are running the script.

Now that we have our project directory created and the API key set as an environment variable, let's move on to writing the code for the first file that will upload audio files to the AssemblyAI service.

Uploading the audio file for transcription

Create a new file named upload_audio_file.py and place the following code in it:

import argparse
import os
import requests


API_URL = "https://api.assemblyai.com/v2/"


def upload_file_to_api(filename):
    """Checks for a valid file and then uploads it to AssemblyAI
    so it can be saved to a secure URL that only that service can access.
    When the upload is complete we can then initiate the transcription
    API call.
    Returns the API JSON if successful, or None if file does not exist.
    """
    if not os.path.exists(filename):
        return None

    def read_file(filename, chunk_size=5242880):
        with open(filename, 'rb') as _file:
            while True:
                data = _file.read(chunk_size)
                if not data:
                    break
                yield data

    headers = {'authorization': os.getenv("ASSEMBLYAI_KEY")}
    response = requests.post("".join([API_URL, "upload"]), headers=headers,
                             data=read_file(filename))
    return response.json()

The above code imports the argparse, os and requests packages so that we can use them in this script. The API_URL is a constant that has the base URL of the AssemblyAI service. We define the upload_file_to_api function with a single argument, filename that should be a string with the absolute path to a file and its filename.

Within the function, we check that the file exists, then use Request's chunked transfer encoding to stream large files to the AssemblyAI API.

The os module's getenv function reads the API that was set on the command line using the export command with the getenv. Make sure that you use that export command in the terminal where you are running this script otherwise that ASSEMBLYAI_KEY value will be blank. When in doubt, use echo $ASSEMBLY_AI to see if the value matches your API key.

To use the upload_file_to_api function, append the following lines of code in the upload_audio_file.py file so that we can properly execute this code as a script called with the python command:

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("filename")
    args = parser.parse_args()
    upload_filename = args.filename
    response_json = upload_file_to_api(upload_filename)
    if not response_json:
        print("file does not exist")
    else:
        print("File uploaded to URL: {}".format(response_json['upload_url']))

The code above creates an ArgumentParser object that allows the application to obtain a single argument from the command line to specify the file we want to access, read and upload to the AssmeblyAI service.

If the file does not exist, the script will print a message that the file couldn't be found. In the happy path where we do find the correct file at that path, then the file is uploaded using the code in upload_file_to_api function.

Execute the completed upload_audio_file.py script by running it on the command line with the python command. Replace FULL_PATH_TO_FILE with an absolute path to the file you want to upload, such as /Users/matt/devel/audio.mp3.

python upload_audio_file.py FULL_PATH_TO_FILE

Assuming the file is found at the location that you specified, when the script finishes uploading the file, it will print a message like this one with a unique URL:

File uploaded to URL: https://cdn.assemblyai.com/upload/463ce27f-0922-4ea9-9ce4-3353d84b5638

This URL is not public, it can only be used by the AssemblyAI service, so no one else will be able to access your file and its contents except for you and their transcription API.

The part that is important is the last section of the URL, in this example it is 463ce27f-0922-4ea9-9ce4-3353d84b5638. Save that unique identifier because we need to pass it into the next script that initiates the transcription service.

Initiate transcription

Next, we'll write some code to kick off the transcription. Create a new file named initiate_transcription.py. Add the following code to the new file.

import argparse
import os
import requests


API_URL = "https://api.assemblyai.com/v2/"
CDN_URL = "https://cdn.assemblyai.com/"


def initiate_transcription(file_id):
    """Sends a request to the API to transcribe a specific
    file that was previously uploaded to the API. This will
    not immediately return the transcription because it takes
    a moment for the service to analyze and perform the
    transcription, so there is a different function to retrieve
    the results.
    """
    endpoint = "".join([API_URL, "transcript"])
    json = {"audio_url": "".join([CDN_URL, "upload/{}".format(file_id)])}
    headers = {
        "authorization": os.getenv("ASSEMBLYAI_KEY"),
        "content-type": "application/json"
    }
    response = requests.post(endpoint, json=json, headers=headers)
    return response.json()

We have the same imports as the previous script and we've added a new constant, CDN_URL that matches the separate URL where AssemblyAI stores the uploaded audio files.

The initiate_transcription function essentially just sets up a single HTTP request to the AssemblyAI API to start the transcription process on the audio file at the specific URL passed in. This is why passing in the file_id is important: that completes the URL of the audio file that we are telling AssemblyAI to retrieve.

Finish the file by appending this code so that it can be easily invoked from the command line with arguments.

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("file_id")
    args = parser.parse_args()
    file_id = args.file_id
    response_json = initiate_transcription(file_id)
    print(response_json)

Start the script by running the python command on the initiate_transcription file and pass in the unique file identifier you saved from the previous step.

# the FILE_IDENTIFIER is returned in the previous step and will
# look something like this: 463ce27f-0922-4ea9-9ce4-3353d84b5638
python initiate_transcription.py FILE_IDENTIFIER

The API will send back a JSON response that this script prints to the command line.

{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'text': None, 
 'audio_url': 'https://cdn.assemblyai.com/upload/463ce27f-0922-4ea9-9ce4-3353d84b5638', 
 'speed_boost': False, 'language_model': 'assemblyai_default', 'redact_pii': False, 
 'confidence': None, 'webhook_status_code': None, 
 'id': 'gkuu2krb1-8c7f-4fe3-bb69-6b14a2cac067', 'status': 'queued', 'boost_param': None, 
 'words': None, 'format_text': True, 'webhook_url': None, 'punctuate': True, 
 'utterances': None, 'audio_duration': None, 'auto_highlights': False, 
 'word_boost': [], 'dual_channel': None, 'audio_start_from': None}

Take note of the value of the id key in the JSON response. This is the transcription identifier we need to use to retrieve the transcription result. In this example, it is gkuu2krb1-8c7f-4fe3-bb69-6b14a2cac067. Copy the transcription identifier in your own response because we will need it to check when the transcription process has completed in the next step.

Retrieving the transcription result

We have uploaded and begun the transcription process, so let's get the result as soon as it is ready.

How long it takes to get the results back can depend on the size of the file, so this next script will send an HTTP request to the API and report back the status of the transcription, or print the output if it's complete.

Create a third Python file named get_transcription.py and put the following code into it.

import argparse
import os
import requests


API_URL = "https://api.assemblyai.com/v2/"


def get_transcription(transcription_id):
    """Requests the transcription from the API and returns the JSON
    response."""
    endpoint = "".join([API_URL, "transcript/{}".format(transcription_id)])
    headers = {"authorization": os.getenv('ASSEMBLYAI_KEY')}
    response = requests.get(endpoint, headers=headers)
    return response.json()


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("transcription_id")
    args = parser.parse_args()
    transcription_id = args.transcription_id
    response_json = get_transcription(transcription_id)
    if response_json['status'] == "completed":
        for word in response_json['words']:
            print(word['text'], end=" ")
    else:
        print("current status of transcription request: {}".format(
              response_json['status']))

The code above has the same imports as the other scripts. In this new get_transcription function, we simply call the AssemblyAI API with our API key and the transcription identifier from the previous step (not the file identifier). We retrieve the JSON response and return it.

In the main function we handle the transcription identifier that is passed in as a command line argument and pass it into the get_transcription function. If the response JSON from the get_transcription function contains a completed status then we print the results of the transcription. Otherwise, print the current status which is either queued or processing before it is completed.

Call the script using the command line and the transcription identifier from the previous section:

python get_transcription.py TRANSCRIPTION_ID

If the service has not yet started working on the transcript then it will return queued like this:

current status of transcription request: queued

When the service is currently working on the audio file it will return processing:

current status of transcription request: processing

When the process is completed, our script will return the text of the transcription, like you see here:

An object relational mapper is a code library that automates the transfer of 
data stored in relational, databases into objects that are more commonly used
in application code or EMS are useful because they provide a high level 

...(output abbreviated)

That's it, we've got our transcription!

You may be wondering what to do if the accuracy isn't where you need it to be for your situation. That is where boosting accuracy for keywords or phrases comes in. You can use either of those two methods to boost the accuracy of your recordings to an acceptable level for your situation.

What's next?

We just finished writing some scripts that call the AssemblyAI API to transcribe recordings with speech into text output.

Next, take a look at some of their more advanced documentation that goes beyond the basics in this tutorial:

Questions? Let me know via an issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai. See something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Tracking Daily User Data in Django with django-user-visit

Matt Makai — Sun, 19 Jul 2020 00:00:00 -0400

It can be tedious to figure out what data to track, create data models and build middleware for your Django project if you just want to collect some basic information about clients that connect to your web application . Fortunately, the library django-user-visit is a handy Django project that quickly handles all of this complexity for you. In this tutorial, we'll learn how to use django-user-visit in a new Django project to add daily visit data tracking to Django projects.

When we're done, we can view information like the following in the Django Admin:

Project Requirements

We will use the following dependencies to complete this tutorial:

All code in this blog post is available open source under the MIT license on GitHub under the track-daily-user-data-django-user-visit directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Development environment set up

Change into the directory where you keep your Python virtual environments. Create a new virtualenv for this project using the following command.

python3 -m venv ~/venvs/djuservisit

Activate the virtualenv with the activate shell script:

source ~/venvs/djuservisit/bin/activate

(djuservisit) $

Remember, you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the Django package into the activated but otherwise empty virtualenv.

pip install django==3.0.8 django-user-visit==0.4

Look for output similar to the following to confirm the appropriate packages were installed correctly from PyPI.

(djuservisit) $ pip install django==3.0.8 django-user-visit==0.4
Collecting django==3.0.8
  Using cached https://files.pythonhosted.org/packages/ca/ab/5e004afa025a6fb640c6e983d4983e6507421ff01be224da79ab7de7a21f/Django-3.0.8-py3-none-any.whl
Collecting django-user-visit==0.4
  Downloading https://files.pythonhosted.org/packages/23/ef/d3ec22c3a897192e267389d6ee59ce1858f5ede262b078f93211aff110e7/django_user_visit-0.4-py3-none-any.whl
Collecting sqlparse>=0.2.2 (from django==3.0.8)
  Using cached https://files.pythonhosted.org/packages/85/ee/6e821932f413a5c4b76be9c5936e313e4fc626b33f16e027866e1d60f588/sqlparse-0.3.1-py2.py3-none-any.whl
Collecting asgiref~=3.2 (from django==3.0.8)
  Using cached https://files.pythonhosted.org/packages/d5/eb/64725b25f991010307fd18a9e0c1f0e6dff2f03622fc4bcbcdb2244f60d6/asgiref-3.2.10-py3-none-any.whl
Collecting pytz (from django==3.0.8)
  Using cached https://files.pythonhosted.org/packages/4f/a4/879454d49688e2fad93e59d7d4efda580b783c745fd2ec2a3adf87b0808d/pytz-2020.1-py2.py3-none-any.whl
Collecting user-agents<3.0,>=2.1 (from django-user-visit==0.4)
  Using cached https://files.pythonhosted.org/packages/1b/be/82e4d20a7716d8e5de98b948edcecff9bb237e6718c3831bd92794fe9821/user-agents-2.1.tar.gz
Collecting ua-parser>=0.9.0 (from user-agents<3.0,>=2.1->django-user-visit==0.4)
  Using cached https://files.pythonhosted.org/packages/9d/22/4d16b08db329fd440eed366d35e4dd7195c9babb4ecac5218f28081522a2/ua_parser-0.10.0-py2.py3-none-any.whl
Installing collected packages: sqlparse, asgiref, pytz, django, ua-parser, user-agents, django-user-visit
  Running setup.py install for user-agents ... done
Successfully installed asgiref-3.2.10 django-3.0.8 django-user-visit-0.4 pytz-2020.1 sqlparse-0.3.1 ua-parser-0.10.0 user-agents-2.1

Our dependencies are installed so we can now create our project and start coding.

Creating the application

We have everything we need to start building our application.

django-admin.py startproject djuservisit

Note that in this tutorial we are using the same name for both the virtualenv and the Django project directory, but they can be different names if you prefer that for organizing your own projects.

The django-admin command creates a directory named djuservisit along with several subdirectories that you should be familiar with if you have previously worked with Django.

Change directories into the new project.

cd djuservisit

Open djuservisit/djuservisit/settings.py. Add the user_visits app and its middleware to settings.py by inserting the two highlighted lines:

# djuservisit/djuservisit/settings.py
# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
~~    'user_visit',
]

MIDDLEWARE = [ 
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
~~    'user_visit.middleware.UserVisitMiddleware',
]

Save and close settings.py.

Create a Django superuser so you can access the Django Admin. Go to the base directory of this project and use the manage.py file with the following command:

python manage.py createsuperuser

Follow the prompts and enter values for the username, email address and password that you want your local Django superuser to have. Next, we'll test out how this library works when a user visits a page created by our Django web app.

Testing django-user-visit

Let's test out our bare-bones application. Execute the development server with the following command:

python manage.py runserver

The Django development server should start up with no issues.

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
July 19, 2020 - 13:01:41
Django version 3.0.8, using settings 'djuservisit.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Open a web browser and go to "http://localhost:8000".

That's the default page provided by Django in the absence of any other URLs to serve at the root URL, but it works for our purposes.

Go to the Django Admin by changing the URL in your browser to "http://localhost:8000/admin". The Django Admin login page will appear.

Enter the username and password of the superuser you just created with the manage.py command to log in. Next, you will see the Django admin dashboard.

The "User visit log" has already been added to the Admin. Click on the "User visits" link.

The list of all users that have visited by day will show up.

Click on any of the visits to see more detailed data about the record, just like you would with any Django Admin extension.

That library was pretty easy to install for the information that it aggregates for you. Next, let's take a closer look at the Django ORM model that powers this library.

Inspecting the django-user-visit model

We confirmed that django-user-visit is properly installed. Let's take a closer look at the model the library provides to store the user data.

Take a look at the source code for django-user-visit/user_visit/models.py on GitHub. Below is an excerpt with the relevant lines of that source file. I've highlighted a few lines that will be discussed below the code excerpt.

## ... source code abbreviated ...

class UserVisit(models.Model):
    """
    Record of a user visiting the site on a given day.
    This is used for tracking and reporting - knowing the volume of visitors
    to the site, and being able to report on someone's interaction with the site.
    We record minimal info required to identify user sessions, plus changes in
    IP and device. This is useful in identifying suspicious activity (multiple
    logins from different locations).
    Also helpful in identifying support issues (as getting useful browser data
    out of users can be very difficult over live chat).
    """

~~    user = models.ForeignKey(
~~        settings.AUTH_USER_MODEL, related_name="user_visits", on_delete=models.CASCADE
~~    )
    timestamp = models.DateTimeField(
        help_text="The time at which the first visit of the day was recorded",
        default=timezone.now,
    )
    session_key = models.CharField(help_text="Django session identifier", max_length=40)
    remote_addr = models.CharField(
        help_text=(
            "Client IP address (from X-Forwarded-For HTTP header, "
            "or REMOTE_ADDR request property)"
        ),
        max_length=100,
        blank=True,
    )
~~    ua_string = models.TextField(
~~        "User agent (raw)", help_text="Client User-Agent HTTP header", blank=True,
~~    )
    uuid = models.UUIDField(default=uuid.uuid4, editable=False, unique=True)
    hash = models.CharField(
        max_length=32,
        help_text="MD5 hash generated from request properties",
        unique=True,
    )
    created_at = models.DateTimeField(
        help_text="The time at which the database record was created (!=timestamp)",
        auto_now_add=True,
    )

    objects = UserVisitManager()

    class Meta:
        get_latest_by = "timestamp"

    def __str__(self) -> str:
        return f"{self.user} visited the site on {self.timestamp}"

    def __repr__(self) -> str:
        return f"<UserVisit id={self.id} user_id={self.user_id} date='{self.date}'>"

    def save(self, *args: Any, **kwargs: Any) -> None:
        """Set hash property and save object."""
~~        self.hash = self.md5().hexdigest()
        super().save(*args, **kwargs)

    @property
    def user_agent(self) -> user_agents.parsers.UserAgent:
        """Return UserAgent object from the raw user_agent string."""
        return user_agents.parsers.parse(self.ua_string)

    @property
    def date(self) -> datetime.date:
        """Extract the date of the visit from the timestamp."""
        return self.timestamp.date()

    # see https://github.com/python/typeshed/issues/2928 re. return type
    def md5(self) -> hashlib._Hash:
        """Generate MD5 hash used to identify duplicate visits."""
~~        h = hashlib.md5(str(self.user.id).encode())  # noqa: S303
        h.update(self.date.isoformat().encode())
        h.update(self.session_key.encode())
        h.update(self.remote_addr.encode())
        h.update(self.ua_string.encode())
        return h

A few things to note based on the highlighted above:

The UserVisit model matches up with the Django user model using the user = models.ForeignKey... line
The code uses the save function to ensure the some of the fields are automatically populated, such as the hash property using the hashlib module
This library has a dependency on the user_agents library to parse the User-Agent of the browser the client is using

Reading the source code for libraries like django-user-visit is helpful not only to know what it is doing under the covers, but also to learn new ways to code your own applications.

Take a look at the Django code examples and Django extensions pages to see more projects with good Python example code that you can learn from.

Additional resources

We just finished building an app that tracks daily user visits with the django-user-visit library.

Next, try out some of these other related Django tutorials:

Quickly Use Bootstrap 4 in a Django Template with a CDN

Matt Makai — Sun, 05 Jul 2020 00:00:00 -0400

The Django web framework makes it easy to render HTML using the Django template engine. However, the default styling on HTML pages usually need a Cascading Style Sheet (CSS) framework such as Bootstrap to make the design look decent. In this beginner's tutorial, we'll use the Bootstrap Content Delivery Network (CDN) to quickly add Bootstrap to a rendered HTML page.

Here is what the <h1> element styling will look like at the end of this tutorial:

Tutorial Requirements

We will use the following dependencies to complete this tutorial:

Django web framework, version 3.0.8

All code in this blog post is available open source under the MIT license on GitHub under the bootstrap-4-django-template directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Development environment set up

Change into the directory where you keep your Python virtual environments. Create a new virtualenv for this project using the following command.

python3 -m venv ~/venvs/djbootstrap4

Activate the virtualenv with the activate shell script:

source ~/venvs/djbootstrap4/bin/activate

(djbootstrap4) $

Remember, you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the Django package into the activated but otherwise empty virtualenv.

pip install django==3.0.8

Look for output similar to the following to confirm the appropriate packages were installed correctly from PyPI.

Collecting django
  Using cached https://files.pythonhosted.org/packages/ca/ab/5e004afa025a6fb640c6e983d4983e6507421ff01be224da79ab7de7a21f/Django-3.0.8-py3-none-any.whl
Collecting sqlparse>=0.2.2 (from django)
  Using cached https://files.pythonhosted.org/packages/85/ee/6e821932f413a5c4b76be9c5936e313e4fc626b33f16e027866e1d60f588/sqlparse-0.3.1-py2.py3-none-any.whl
Collecting asgiref~=3.2 (from django)
  Using cached https://files.pythonhosted.org/packages/d5/eb/64725b25f991010307fd18a9e0c1f0e6dff2f03622fc4bcbcdb2244f60d6/asgiref-3.2.10-py3-none-any.whl
Collecting pytz (from django)
  Using cached https://files.pythonhosted.org/packages/4f/a4/879454d49688e2fad93e59d7d4efda580b783c745fd2ec2a3adf87b0808d/pytz-2020.1-py2.py3-none-any.whl
Installing collected packages: sqlparse, asgiref, pytz, django
Successfully installed asgiref-3.2.10 django-3.0.8 pytz-2020.1 sqlparse-0.3.1

We can get started coding the application now that we have all of our required dependencies installed.

Building our application

Let's begin coding our application.

django-admin.py startproject djbootstrap4

Note that in this tutorial we are using the same name for both the virtualenv and the Django project directory, but they can be different names if you prefer that for organizing your own projects.

The django-admin command creates a directory named djbootstrap4 along with several subdirectories that you should be familiar with if you have previously worked with Django.

Change directories into the new project.

cd djbootstrap4

Create a new Django app within djbootstrap4.

python manage.py startapp bootstrap4

Django will generate a new folder named bootstrap4 for the project. We should update the URLs so the app is accessible before we write our views.py code.

Open djbootstrap4/djbootstrap4/urls.py. Add the highlighted lines so that URL resolver will check the bootstrap4 app for additional routes to match with URLs that are requested of this Django application.

# djbootstrap4/djbootstrap4/urls.py
~~from django.conf.urls import include
from django.contrib import admin
from django.urls import path


urlpatterns = [
~~    path('', include('bootstrap4.urls')),
    path('admin/', admin.site.urls),
]

Save djbootstrap4/djbootstrap4/urls.py and open djbootstrap4/djbootstrap4/settings.py. Add the bootstrap4 app to settings.py by inserting the highlighted line:

# djbootstrap4/djbootstrap4/settings.py
# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
~~    'bootstrap4',
]

Save and close settings.py.

Next change into the djbootstrap4/bootstrap4 directory. Create a new file named urls.py to contain routes for the bootstrap4 app.

Add all of these lines to the empty djbootstrap4/bootstrap4/urls.py file.

# djbootstrap4/bootstrap4/urls.py
from django.conf.urls import url
from . import views

urlpatterns = [
    url(r'', views.bootstrap4_index, name="index"),
]

Save djbootstrap4/bootstrap4/urls.py. Open djbootstrap4/bootstrap4/views.py to add the following two highlighted lines. You can keep the boilerplate comment "# Create your views here." or delete like I usually do.

# djbootstrap4/bootstrap4/views.py
from django.shortcuts import render


~~def bootstrap4_index(request):
~~    return render(request, 'index.html', {})

Next, create a directory for your template files named templates under the djmaps/maps app directory.

mkdir templates

Create a new file named index.html within djbootstrap4/bootstrap4/templates that contains the following Django template language markup.

<!DOCTYPE html>
<html>
  <head>
    <title>First step for bootstrap4</title>
  </head>
  <body>
   <h1>Hello, world!</h1>
  </body>
</html>

python manage.py runserver

The Django development server should start up with no issues other than an unapplied migrations warning.

Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).

You have 17 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

July 05, 2020 - 10:59:58
Django version 3.0.8, using settings 'djbootstrap4.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Open a web browser and go to "http://localhost:8000".

With our base application working, we can now add Bootstrap.

Integrating Bootstrap

Time to add Bootstrap into the template so we can use its styling.

Open djbootstrap4/bootstrap4/templates/index.html back up and add or modify the following highlighted lines, which are very similar to what you will find in the Bootstrap introduction guide:

<!DOCTYPE html>
<html lang="en">
  <head>
~~    <meta charset="utf-8">
~~    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
~~    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/css/bootstrap.min.css" integrity="sha384-9aIt2nRpC12Uk9gS9baDl411NQApFmC26EwAOH8WgZl5MYYxFfc+NcPb1dKGj7Sk" crossorigin="anonymous">
~~    <title>bootstrap4</title>
  </head>
  <body>
    <h1>Hello, world!</h1>

~~    <!-- Optional JavaScript -->
~~    <!-- jQuery first, then Popper.js, then Bootstrap JS -->
~~    <script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha384-DfXdz2htPH0lsSSs5nCTpuj/zy4C+OGpamoFVy38MVBnE+IbbVYUew+OrCXaRkfj" crossorigin="anonymous"></script>
~~    <script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js" integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo" crossorigin="anonymous"></script>
~~    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/js/bootstrap.min.js" integrity="sha384-OgVRvuATP1z7JjHLkuOU7Xw704+h835Lr+6QL9UvYjZE3Ipu6Tp75j7Bh/kR0JKI" crossorigin="anonymous"></script>
  </body>
</html>

The above new lines in the <head> section add a couple of meta elements that are important to Bootstrap's styling, and add the mandatory Bootstrap stylesheet.

We keep the same <h1> header, which will automatically get the CSS styling. Then there are 3 optional script elements that pull in Bootstrap JavaScript for more advanced features. We are not using them in this tutorial because we just wanted to quickly show how to use the CDN and with this in place you can see in the Bootstrap content docs what you want to add to the template next.

Refresh the page at "http://localhost:8000" and you should see "Hello, world!" change fonts.

If you see that, it means everything works as expected.

What now?

We just added Bootstrap via the CDN so we can use it in our Django template. This was the absolute simplest way to add Bootstrap to a single Django page and now there's a ton more you can do with it.

Next, try out some of these other related Django tutorials:

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai. If you see an issue or error in this tutorial, please fork the source repository on GitHub and submit a pull request with the fix.

How to Report Errors in Flask Web Apps with Sentry

Matt Makai — Thu, 02 Jul 2020 00:00:00 -0400

Flask web applications are highly customizable by developers thanks to the framework's extension-based architecture, but that flexibility can sometimes lead to more errors when you run the application due to rough edges between the libraries.

Reporting errors is crucial to running a well-functioning Flask web application, so this tutorial will guide you through adding a free, basic Sentry configuration to a fresh Flask project.

Tutorial Requirements

Flask web framework, version 1.1.2
a hosted Sentry instance on sentry.io, which we'll need an account to access
the Sentry Python helper library to send exception data to our Sentry instance, with the Flask integration

All code in this blog post is available open source under the MIT license on GitHub under the report-errors-flask-web-apps-sentry directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Development environment set up

Change into the directory where you keep your Python virtual environments. Create a new virtualenv for this project using the following command.

Install the Flask and Sentry-SDK code libraries into a new Python virtual environment using the following commands:

python -m venv sentryflask
source sentryflask/bin/activate

pip install flask>=1.1.2 sentry-sdk[flask]==0.15.1

Note that we installed the Flask integration as part of the Sentry SDK, which is why the dependency is sentry-sdk[flask] rather than just sentry-sdk.

Now that we have all of our dependencies installed we can code up a little application to show how the error reporting works.

Creating the application

We have everything we need to start building our application. Create a new directory for your project. I've called mine report-errors-flask-web-apps-sentry in the examples repository but you can use a shorter name if you prefer. Open a new file named app.py and write the following code in it.

# app.py
from flask import Flask, escape, request


app = Flask(__name__)


@app.route('/divide/<int:numerator>/by/<int:denominator>/')
def hello(numerator, denominator):
    answer = numerator / denominator
    return f'{numerator} can be divided by {denominator} {answer} times.'

The above code is a short Flask application that allows input via the URL for two integer values: a numerator and a denominator.

Save the file and run it using the flask run command:

env FLASK_APP=app.py flask run

If you see the following output on the command line that means the development server is working properly:

 * Serving Flask app "app.py"
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Test it by going to http://localhost:5000/divide/50/by/10/ and you will get the following output in your web browser:

With our base application working, we can now add error reporting for the situations that do not work as expected.

Adding Sentry to the Flask app

It's time to add Sentry with the Flask integration into the mix, so that we can easily see when the route errors out due to bad input.

Go to Sentry.io's homepage.

Sign into your account or sign up for a new free account. You will be at the main account dashboard after logging in or completing the Sentry sign up process.

There are no errors logged on our account dashboard yet, which is as expected because we have not yet connected our account to our Python application.

You'll want to create a new Sentry Project just for this application so click "Projects" in the left sidebar to go to the Projects page.

On the Projects page, click the "Create Project" button in the top right corner of the page.

You can either choose "Flask" or select "Python". I usually just choose "Python" if I do not yet know what framework I'll be using to build my application. Next, give your new Project a name and then press the "Create Project" button. Our new project is ready to integrate with our Python code.

We need the unique identifier for our account and project to authorize our Python code to send errors to this Sentry instance. The easiest way to get what we need is to go to the Python+Flask documentation page and read how to configure the SDK.

Copy the string parameter for the init method and set it as an environment variable rather than having it exposed in your project's code.

export SENTRY_DSN='https://yourkeygoeshere.ingest.sentry.io/project-number'

Make sure to replace "yourkeygoeshere" with your own unique identifier and "project-number" with the ID that matches the project you just created.

Check that the SENTRY_DSN is set properly in your shell using the echo command:

echo $SENTRY_DSN

Update app.py with the following highlighted lines of code.

# app.py                                                                                                                                                                                
~~import os
~~import sentry_sdk
from flask import Flask, escape, request
~~from sentry_sdk.integrations.flask import FlaskIntegration


~~sentry_sdk.init(
~~    dsn=os.getenv('SENTRY_DSN'), integrations=[FlaskIntegration()]
~~)


app = Flask(__name__)


@app.route('/divide/<int:numerator>/by/<int:denominator>/')
def hello(numerator, denominator):
    answer = numerator / denominator
    return f'{numerator} can be divided by {denominator} {answer} times.'

The above new lines of code initialize the Sentry client and allow it to properly send any errors that occur over to the right Sentry service.

Testing the Sentry Integration

The Sentry dashboard shows that the service is still waiting for events.

Let's make an error happen to see if we've properly connected the Flask integration with our application.

Try to divide by zero, by going to http://localhost:5000/divide/50/by/0/ in your web browser. You should get an "Internal Server Error".

Back over in the Sentry dashboard, the error appears in the list.

We can drill into the error by clicking on it and get a ton more information, not just about our application but also about the client that visited the site. This is handy if you have an issue in a specific browser or other type of client when building an API.

With that in place, you can now build out the rest of your Flask application knowing that all of the exceptions will be tracked in Sentry.

What's next?

We just finished building a Flask app to show how quickly the hosted version of Sentry can be added to applications so you do not lose track of your error messages.

Next, you can try one of these tutorials to add other useful features to your new application:

You can also determine what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai.

If you see an issue or error in this tutorial, please fork the source repository on GitHub and submit a pull request with the fix.

Reporting Exceptions in Python Scripts with Sentry

Matt Makai — Mon, 25 May 2020 00:00:00 -0400

Python scripts are the glue that keep many applications and their infrastructure running, but when one of your scripts throws an exception you may not know about it immediately unless you have a central place to aggregate the errors. That's where adding Sentry can solved this distributed error logging problem.

In this tutorial, we'll see how to quickly add Sentry to a new or existing Python script to report errors into a centralized location for further debugging.

Development environment setup

Make sure you have Python 3 installed. As of right now, Python 3.8.3 is the latest version of Python.

During this tutorial we're also going to use:

a hosted Sentry instance on sentry.io, which we'll need an account to access
the Sentry Python helper library to send exception data to our Sentry instance

Install the above code libraries into a new Python virtual environment using the following commands:

python -m venv sentryscript
source sentryscript/bin/activate

pip install sentry-sdk>=0.14.4

Our development environment is now ready and we can write some code that will throw exceptions to demonstrate how to use Sentry.

Note that all of the code for this tutorial can be found within the blog-code-examples Git repository on GitHub under the python-script-sentry directory.

An Example Script for Loading Python Modules

We'll start by writing a small but useful script that prints out the names of all modules within a Python package, then add Sentry to it when it becomes apparent that capturing exceptions would be a useful addition.

Create a new file named module_loader.py and write the following lines of code in it to allow us to easily execute it on the command line.

import argparse

def import_submodules(package):
    return {}


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("package")
    args = parser.parse_args()

    package_to_load = args.package
    results = import_submodules(package_to_load)
    for r in results:
        print(str(r))

The above code takes an argument when the script is invoked from the command line and uses the value as an input into the stub import_submodules function that will contain code to walk the tree of modules within the package.

Nextt, add the following highlighted lines of code to use importlib and pkgutil to recursively import modules from the package if one is found that matches the name sent in as the package argument.

import argparse
~~import importlib
~~import pkgutil


def import_submodules(package):
~~    """Import all submodules of a module, recursively, including subpackages.

~~    :param package: package (name or actual module)
~~    :type package: str | module
~~    :rtype: dict[str, types.ModuleType]
~~    """
~~    if isinstance(package, str):
~~        package = importlib.import_module(package)
~~    results = {}
~~    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
~~        full_name = package.__name__ + '.' + name
~~        try:
~~            results[full_name] = importlib.import_module(full_name)
~~            if is_pkg:
~~                results.update(import_submodules(full_name))
~~        except ModuleNotFoundError as mnfe:
~~            print("module not found: {}".format(full_name))
~~        except Exception as general_exception:
~~            print(general_exception)
~~    return results


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("package")
    args = parser.parse_args()

    package_to_load = args.package
    results = import_submodules(package_to_load)
    for r in results:
        print(str(r))

The new code above loops through all packages with the walk_package function in the pkgutil standard library module and tries to import it using the import_module on the package name plus package as a string. If the result is successful, the function will recursively call itself to import submodules within the imported package. If a module is not found, or some other issue occurs, exceptions are caught so that the script does not fail but instead can continue processing potential modules.

Test the full script to see what it prints out with an arbitrary package on the command line:

python module_loader.py importlib

The above example generates the output:

importlib._bootstrap
importlib._bootstrap_external
importlib.abc
importlib.machinery
importlib.resources
importlib.util

Trying to inspect a package that is not installed will give an error. Use the script with a package that is not installed in your current environment.

python module_loader.py flask

The above command produces the following traceback due to an expected ModuleNotFoundError.

Traceback (most recent call last):
  File "module_loader.py", line 35, in <module>
    results = import_submodules(package_to_load)
  File "module_loader.py", line 14, in import_submodules
    package = importlib.import_module(package)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flask'

If you install Flask into your current environment the module is found and the application will go through the list of modules and submodules.

Our example script is usable but what if we run this code or something similar on one or more servers that we don't check that often? That's where it would be helpful to have a way to aggregate one or more scripts' exception output in a single place. Sentry can help us to accomplish that goal.

Adding Exception Reporting with Sentry

Go to Sentry.io's homepage.

Sign into your account or sign up for a new free account. You will be at the main account dashboard after logging in or completing the Sentry sign up process.

There are no errors logged on our account dashboard yet, which is as expected because we have not yet connected our account to the Python script.

You'll want to create a new Sentry Project just for this application so click "Projects" in the left sidebar to go to the Projects page.

On the Projects page, click the "Create Project" button in the top right corner of the page.

Select Python, give your new Project a name and then press the "Create Project" button. Our new project is ready to integrate with our Python script.

We need the unique identifier for our account and project to authorize our Python code to send errors to this Sentry instance. The easiest way to get what we need is to go to the Python getting started documentation page and scroll down to the "Configure the SDK" section.

Copy the string parameter for the init method and set it as an environment variable rather than exposing it directly in your application code.

export SENTRY_DSN='https://yourkeygoeshere.ingest.sentry.io/project-number'

Make sure to replace "yourkeygoeshere" with your own unique identifier and "project-number" with the ID that matches the project you just created.

Check that the SENTRY_DSN is set properly in your shell using the echo command:

echo $SENTRY_DSN

Modify the application to send exception information to Sentry now that we have our unique identifier. Open module_loader.py again and update the following highlighted lines of code.

import argparse
import importlib
~~import os
import pkgutil
~~import sentry_sdk
~~from sentry_sdk import capture_exception

~~# find on https://docs.sentry.io/error-reporting/quickstart/?platform=python
~~sentry_sdk.init(dsn=os.getenv('SENTRY_DSN'))


def import_submodules(package):
    """Import all submodules of a module, recursively, including subpackages.

    :param package: package (name or actual module)
    :type package: str | module
    :rtype: dict[str, types.ModuleType]
    """
    if isinstance(package, str):
        package = importlib.import_module(package)
    results = {}
    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
        full_name = package.__name__ + '.' + name
        try:
            results[full_name] = importlib.import_module(full_name)
            if is_pkg:
                results.update(import_submodules(full_name))
        except ModuleNotFoundError as mnfe:
            print("module not found: {}".format(full_name))
~~            capture_exception(mnfe)
        except Exception as general_exception:
            print(general_exception)
~~            capture_exception(general_exception)
    return results


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("package")
    args = parser.parse_args()

    package_to_load = args.package
    results = import_submodules(package_to_load)
    for r in results:
        print(str(r))

These new lines of code import the Sentry Python SDK and os library (to read system environment variables). The application then initializes the Sentry SDK with the string found in the SENTRY_DSN environment variable. Down in the import_submodules function we then call the capture_exception SDK function whenever a ModuleNotFoundException is thrown or another exception which would be caught within the broader Exception bucket.

Now that our code is in place, let's test out the new Sentry integration.

Testing the Script and Viewing Exceptions

The easiest way to test out whether the Sentry code is working or not is to try to import a module that does not exist. Let's say you make a typo in your command and try to run the script on importliba instead of importlib (maybe because you are using an awful Macbook Pro "butterfly" keyboard instead of a durable keyboard). Try it out and see what happens:

python module_loader.py importliba

The script will run and finish but there will be errors because that module does not exist. Thanks to our new code, we can view the errors in Sentry.

Check the Sentry dashboard to see the error.

We can also click into the error to learn more about what happened.

You can also receive email reports on the errors that occur so that you do not have to always stay logged into the dashboard.

With that all configured, we've now got a great base to expand the script and build better error handling with Sentry as our Python application becomes more complex.

What's Next?

We just created an example script that outputs all of the modules and submodules in a package, then added Sentry to it so that it would report any exceptions back to our central hosted instance.

That's just a simple introduction to Sentry, so next you'll want to read one of the following articles to do more with it:

You can also get an idea of what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Exporting pandas DataFrames into SQLite with SQLAlchemy

Matt Makai — Mon, 30 Mar 2020 00:00:00 -0400

It is common when performing exploratory data analysis, for example when examining COVID-19 data with pandas, to load from files like a CSV, XML, or JSON into a pandas DataFrame. You may then do some work with the data in the DataFrame and want to store it in a more durable location like a relational database.

This tutorial walks through how to load a pandas DataFrame from a CSV file, pull out some data from the full data set, then save the subset of data to a SQLite database using SQLAlchemy.

Configuring our development environment

Make sure you have Python 3 installed. As of right now, Python 3.8.2 is the latest version of Python.

During this tutorial we're also going to use:

pandas (project homepage and source code), version 1.0.3 in this tutorial
SQLAlchemy (project homepage and source code), version 1.3.15 for this tutorial
SQLite (project homepage and source code), which Python includes a connector for as part of the Python standard library

Install the above code libraries into a new Python virtual environment using the following commands:

python -m venv pandasexport
source pandasexport/bin/activate

pip install pandas==1.0.3 sqlalchemy==1.3.15

Our development environment is now ready to download an example COVID-19 data set, load it into a pandas DataFrame, perform some analysis on it then save into a SQLite database.

Obtaining COVID-19 data

Go to the download today’s data on the geographic distribution of COVID-19 cases worldwide page in your web browser. It should look something like the following screenshot.

There should be a link to download the data in CSV format, but the organization has changed the page layout several times in the past few weeks, which makes it difficult to find formats other than Excel (XLSX). If you have trouble obtaining the CSV version, just download this one from GitHub which is pegged to a copy downloaded on March 28th, 2020.

Importing the CSV into pandas

The raw data is in a CSV file and we need to load it into memory via a pandas DataFrame.

Start by running the Python Read-Evaluate-Print Loop (REPL) on the command line:

python

>>>

The REPL is ready to execute code, but we first need to import the pandas library so we can use it.

from pandas import read_csv

df = read_csv("covid-19-cases-march-28-2020.csv", encoding="ISO-8859-1")

The data is now loaded into the df variable which is an instance of the pandas DataFrame class.

When we run the count function on this DataFrame, we get back that it has 7320 rows.

df.count()

Next, we'll take this set of 7320 rows of data and slice out only the rows that pertain to the United States.

Creating a new DataFrame from the original DataFrame

We can pick out all of the rows of data for a single country using a pandas function to match the countriesAndTerritories column to the country of our choice.

save_df = df[df['countriesAndTerritories']=="United_States_of_America"]

The save_df variable contains the smaller subset of data. You can find out what's in it by having it print itself:

save_df

You should see something like the following output:

         dateRep  day  month  year  cases  deaths   countriesAndTerritories geoId countryterritoryCode  popData2018
7082  28/03/2020   28      3  2020  18695     411  United_States_of_America    US                  USA  327167434.0
7083  27/03/2020   27      3  2020  16797     246  United_States_of_America    US                  USA  327167434.0
7084  26/03/2020   26      3  2020  13963     249  United_States_of_America    US                  USA  327167434.0
7085  25/03/2020   25      3  2020   8789     211  United_States_of_America    US                  USA  327167434.0
7086  24/03/2020   24      3  2020  11236     119  United_States_of_America    US                  USA  327167434.0
...          ...  ...    ...   ...    ...     ...                       ...   ...                  ...          ...
7166  04/01/2020    4      1  2020      0       0  United_States_of_America    US                  USA  327167434.0
7167  03/01/2020    3      1  2020      0       0  United_States_of_America    US                  USA  327167434.0
7168  02/01/2020    2      1  2020      0       0  United_States_of_America    US                  USA  327167434.0
7169  01/01/2020    1      1  2020      0       0  United_States_of_America    US                  USA  327167434.0
7170  31/12/2019   31     12  2019      0       0  United_States_of_America    US                  USA  327167434.0

[89 rows x 10 columns]

89 rows of data out of the original 7320 rows. Let's proceed with saving this subset to a SQLite relational database.

Saving the DataFrame to SQLite

We are going to use SQLAlchemy to create a connection to a new SQLite database, which in this example will be stored in file named save_pandas.db. You can of course save the file with whatever name you want and in any location, not just the directory where you are executing the Python REPL.

Start by importing the create_engine function from the sqlalchemy library.

from sqlalchemy import create_engine

Create the connection using the imported create_engine function and then invoking the connect method on it.

engine = create_engine('sqlite:///save_pandas.db', echo=True)
sqlite_connection = engine.connect()

We set echo=True to see all of the output that comes from our database connection. When the connection is successful you will see output similar to the following:

2020-03-29 20:44:08,198 INFO sqlalchemy.engine.base.Engine SELECT CAST('test plain returns' AS VARCHAR(60)) AS anon_1
2020-03-29 20:44:08,198 INFO sqlalchemy.engine.base.Engine ()
2020-03-29 20:44:08,199 INFO sqlalchemy.engine.base.Engine SELECT CAST('test unicode returns' AS VARCHAR(60)) AS anon_1
2020-03-29 20:44:08,199 INFO sqlalchemy.engine.base.Engine ()
<sqlalchemy.engine.base.Connection object at 0x7fd4d932ec88>

Set a variable name with the string of a table name you would like to create. Then use that variable when invoking the to_sql method on the save_df object, which is our pandas DataFrame that is a subset of the original data set with 89 rows filtered from the original 7320.

Note that in this case we are going to fail if the table already exists in the database. You can change if_exists to to replace or append and add your own exception handling in a more robust version of this program. Check the pandas.DataFrame.to_sql documentation for the extensive details on your options.

sqlite_table = "Covid19"
save_df.to_sql(sqlite_table, sqlite_connection, if_exists='fail')

The echo output should spin up with a bunch of output.

2020-03-29 20:45:09,066 INFO sqlalchemy.engine.base.Engine PRAGMA main.table_info("Covid19")
2020-03-29 20:45:09,066 INFO sqlalchemy.engine.base.Engine ()
2020-03-29 20:45:09,067 INFO sqlalchemy.engine.base.Engine PRAGMA temp.table_info("Covid19")
2020-03-29 20:45:09,067 INFO sqlalchemy.engine.base.Engine ()
2020-03-29 20:45:09,069 INFO sqlalchemy.engine.base.Engine 
CREATE TABLE "Covid19" (
    "index" BIGINT, 
    "dateRep" TEXT, 
    day BIGINT, 
    month BIGINT, 
    year BIGINT, 
    cases BIGINT, 
    deaths BIGINT, 
    "countriesAndTerritories" TEXT, 
    "geoId" TEXT, 
    "countryterritoryCode" TEXT, 
    "popData2018" FLOAT
)


2020-03-29 20:45:09,069 INFO sqlalchemy.engine.base.Engine ()
2020-03-29 20:45:09,070 INFO sqlalchemy.engine.base.Engine COMMIT
2020-03-29 20:45:09,070 INFO sqlalchemy.engine.base.Engine CREATE INDEX "ix_Covid19_index" ON "Covid19" ("index")
2020-03-29 20:45:09,070 INFO sqlalchemy.engine.base.Engine ()
2020-03-29 20:45:09,071 INFO sqlalchemy.engine.base.Engine COMMIT
2020-03-29 20:45:09,072 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2020-03-29 20:45:09,074 INFO sqlalchemy.engine.base.Engine INSERT INTO "Covid19" ("index", "dateRep", day, month, year, cases, deaths, "countriesAndTerritories", "geoId", "countryterritoryCode", "popData2018") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2020-03-29 20:45:09,074 INFO sqlalchemy.engine.base.Engine ((7082, '28/03/2020', 28, 3, 2020, 18695, 411, 'United_States_of_America', 'US', 'USA', 327167434.0), (7083, '27/03/2020', 27, 3, 2020, 16797, 246, 'United_States_of_America', 'US', 'USA', 327167434.0), (7084, '26/03/2020', 26, 3, 2020, 13963, 249, 'United_States_of_America', 'US', 'USA', 327167434.0), (7085, '25/03/2020', 25, 3, 2020, 8789, 211, 'United_States_of_America', 'US', 'USA', 327167434.0), (7086, '24/03/2020', 24, 3, 2020, 11236, 119, 'United_States_of_America', 'US', 'USA', 327167434.0), (7087, '23/03/2020', 23, 3, 2020, 8459, 131, 'United_States_of_America', 'US', 'USA', 327167434.0), (7088, '22/03/2020', 22, 3, 2020, 7123, 80, 'United_States_of_America', 'US', 'USA', 327167434.0), (7089, '21/03/2020', 21, 3, 2020, 5374, 110, 'United_States_of_America', 'US', 'USA', 327167434.0)  ... displaying 10 of 89 total bound parameter sets ...  (7169, '01/01/2020', 1, 1, 2020, 0, 0, 'United_States_of_America', 'US', 'USA', 327167434.0), (7170, '31/12/2019', 31, 12, 2019, 0, 0, 'United_States_of_America', 'US', 'USA', 327167434.0))
2020-03-29 20:45:09,074 INFO sqlalchemy.engine.base.Engine COMMIT
2020-03-29 20:45:09,075 INFO sqlalchemy.engine.base.Engine SELECT name FROM sqlite_master WHERE type='table' ORDER BY name
2020-03-29 20:45:09,075 INFO sqlalchemy.engine.base.Engine ()

Our table with all of its data should now be all set. Close the database connection.

sqlite_connection.close()

We can take a look at the data through the sqlite3 command line viewer to make sure it was properly saved to the SQLite file.

On the command line (not in the Python REPL), type:

sqlite3

This will open up the command line prompt to interact with SQLite databases. However, we are not yet connected to our save_pandas.db file.

SQLite version 3.28.0 2019-04-15 14:49:49
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite>

Use the .open command with our save_pandas.db file name to access the database. Then use a standard SQL query to obtain all of the records from the Covid19 table.

sqlite> .open save_pandas.db
sqlite> select * from Covid19;

The SQLite explorer should produce output like you see below:

7082|28/03/2020|28|3|2020|18695|411|United_States_of_America|US|USA|327167434.0
7083|27/03/2020|27|3|2020|16797|246|United_States_of_America|US|USA|327167434.0
7084|26/03/2020|26|3|2020|13963|249|United_States_of_America|US|USA|327167434.0
7085|25/03/2020|25|3|2020|8789|211|United_States_of_America|US|USA|327167434.0
7086|24/03/2020|24|3|2020|11236|119|United_States_of_America|US|USA|327167434.0
7087|23/03/2020|23|3|2020|8459|131|United_States_of_America|US|USA|327167434.0
7088|22/03/2020|22|3|2020|7123|80|United_States_of_America|US|USA|327167434.0
7089|21/03/2020|21|3|2020|5374|110|United_States_of_America|US|USA|327167434.0
7090|20/03/2020|20|3|2020|4835|0|United_States_of_America|US|USA|327167434.0
7091|19/03/2020|19|3|2020|2988|42|United_States_of_America|US|USA|327167434.0
7092|18/03/2020|18|3|2020|1766|23|United_States_of_America|US|USA|327167434.0
7093|17/03/2020|17|3|2020|887|16|United_States_of_America|US|USA|327167434.0
7094|16/03/2020|16|3|2020|823|12|United_States_of_America|US|USA|327167434.0
7095|15/03/2020|15|3|2020|777|10|United_States_of_America|US|USA|327167434.0
7096|14/03/2020|14|3|2020|511|7|United_States_of_America|US|USA|327167434.0
7097|13/03/2020|13|3|2020|351|10|United_States_of_America|US|USA|327167434.0
7098|12/03/2020|12|3|2020|287|2|United_States_of_America|US|USA|327167434.0
7099|11/03/2020|11|3|2020|271|2|United_States_of_America|US|USA|327167434.0
7100|10/03/2020|10|3|2020|200|5|United_States_of_America|US|USA|327167434.0
7101|09/03/2020|9|3|2020|121|4|United_States_of_America|US|USA|327167434.0
7102|08/03/2020|8|3|2020|95|3|United_States_of_America|US|USA|327167434.0
7103|07/03/2020|7|3|2020|105|2|United_States_of_America|US|USA|327167434.0
7104|06/03/2020|6|3|2020|74|1|United_States_of_America|US|USA|327167434.0
7105|05/03/2020|5|3|2020|34|2|United_States_of_America|US|USA|327167434.0
7106|04/03/2020|4|3|2020|22|3|United_States_of_America|US|USA|327167434.0
7107|03/03/2020|3|3|2020|14|4|United_States_of_America|US|USA|327167434.0
7108|02/03/2020|2|3|2020|20|1|United_States_of_America|US|USA|327167434.0
7109|01/03/2020|1|3|2020|3|1|United_States_of_America|US|USA|327167434.0
7110|29/02/2020|29|2|2020|6|0|United_States_of_America|US|USA|327167434.0
7111|28/02/2020|28|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7112|27/02/2020|27|2|2020|6|0|United_States_of_America|US|USA|327167434.0
7113|26/02/2020|26|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7114|25/02/2020|25|2|2020|18|0|United_States_of_America|US|USA|327167434.0
7115|24/02/2020|24|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7116|23/02/2020|23|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7117|22/02/2020|22|2|2020|19|0|United_States_of_America|US|USA|327167434.0
7118|21/02/2020|21|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7119|20/02/2020|20|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7120|19/02/2020|19|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7121|18/02/2020|18|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7122|17/02/2020|17|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7123|16/02/2020|16|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7124|15/02/2020|15|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7125|14/02/2020|14|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7126|13/02/2020|13|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7127|12/02/2020|12|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7128|11/02/2020|11|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7129|10/02/2020|10|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7130|09/02/2020|9|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7131|08/02/2020|8|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7132|07/02/2020|7|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7133|06/02/2020|6|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7134|05/02/2020|5|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7135|04/02/2020|4|2|2020|0|0|United_States_of_America|US|USA|327167434.0
7136|03/02/2020|3|2|2020|3|0|United_States_of_America|US|USA|327167434.0
7137|02/02/2020|2|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7138|01/02/2020|1|2|2020|1|0|United_States_of_America|US|USA|327167434.0
7139|31/01/2020|31|1|2020|1|0|United_States_of_America|US|USA|327167434.0
7140|30/01/2020|30|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7141|29/01/2020|29|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7142|28/01/2020|28|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7143|27/01/2020|27|1|2020|3|0|United_States_of_America|US|USA|327167434.0
7144|26/01/2020|26|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7145|25/01/2020|25|1|2020|1|0|United_States_of_America|US|USA|327167434.0
7146|24/01/2020|24|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7147|23/01/2020|23|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7148|22/01/2020|22|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7149|21/01/2020|21|1|2020|1|0|United_States_of_America|US|USA|327167434.0
7150|20/01/2020|20|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7151|19/01/2020|19|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7152|18/01/2020|18|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7153|17/01/2020|17|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7154|16/01/2020|16|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7155|15/01/2020|15|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7156|14/01/2020|14|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7157|13/01/2020|13|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7158|12/01/2020|12|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7159|11/01/2020|11|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7160|10/01/2020|10|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7161|09/01/2020|9|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7162|08/01/2020|8|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7163|07/01/2020|7|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7164|06/01/2020|6|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7165|05/01/2020|5|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7166|04/01/2020|4|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7167|03/01/2020|3|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7168|02/01/2020|2|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7169|01/01/2020|1|1|2020|0|0|United_States_of_America|US|USA|327167434.0
7170|31/12/2019|31|12|2019|0|0|United_States_of_America|US|USA|327167434.0
sqlite>

All of the data with the countriesAndTerritories column matching United_States_of_America is there! We successfully exported the data from the DataFrame into the SQLite database file.

What's Next?

We just imported data from a CSV into a pandas DataFrame, selected a subset of that data then saved it to a relational database.

You should take a look at the Learning pandas by Exploring COVID-19 Data tutorial to learn more about how to select subsets of data from a larger DataFrame, or head to the pandas page for more tutorials by the rest of the Python community.

You can also get an idea of what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Learning pandas by Exploring COVID-19 Data

Matt Makai — Sat, 28 Mar 2020 00:00:00 -0400

The European Centre for Disease Prevention and Control provides daily-updated worldwide COVID-19 data that is easy to download in JSON, CSV or XML formats. In this tutorial, we will use the pandas data analysis tool on the comma-separated values (CSV) data to learn some of the basic pandas commands and explore what is contained within the data set.

Configuring our development environment

Make sure you have Python 3 installed. As of right now, Python 3.8.2 is the latest.

During this tutorial we're also going to use pandas.

Install it now into a new virtual environment with the following commands:

python -m venv covidpandas
source covidpandas/bin/activate

pip install pandas

We are now ready to get the COVID-19 data and start analyzing it with pandas.

Obtaining the COVID-19 data

Go to the download today’s data on the geographic distribution of COVID-19 cases worldwide page in your web browser. It should look something like the following screenshot.

Importing the CSV into pandas

We have the data in a CSV now we need to import it into a pandas DataFrame.

Start by running the Python REPL:

python

>>>

The REPL is ready to go, now we need to import pandas so we can read the data we downloaded.

from pandas import read_csv

df = read_csv("covid-19-cases-march-28-2020.csv")

Don't worry if you get an error like UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe7.... Run this command instead which explicitly sets the file encoding so pandas can properly read the CSV.

# make sure the file name of the csv matches your file's name!
df = read_csv("covid-19-cases-march-28-2020.csv", encoding="ISO-8859-1")

We now have our data loaded into a pandas DataFrame and can start running code to poke and prod and what's inside the data set.

Running pandas commands

Let's first take a peek at what a sample of the data looks like. I typically run the head and tail functions when I open something up to find out what are contained in the first five and last five rows.

df.head()

You should see six lines of output: one as the columns header and the first five rows of data from the CSV:

      dateRep  day  month  year  cases  deaths countriesAndTerritories geoId countryterritoryCode  popData2018
0  28/03/2020   28      3  2020     16       1             Afghanistan    AF                  AFG   37172386.0
1  27/03/2020   27      3  2020      0       0             Afghanistan    AF                  AFG   37172386.0
2  26/03/2020   26      3  2020     33       0             Afghanistan    AF                  AFG   37172386.0
3  25/03/2020   25      3  2020      2       0             Afghanistan    AF                  AFG   37172386.0
4  24/03/2020   24      3  2020      6       1             Afghanistan    AF                  AFG   37172386.0

The tail function looks at the last five rows in a DataFrame.

df.tail()

tail output will look something like this:

         dateRep  day  month  year  cases  deaths countriesAndTerritories geoId countryterritoryCode  popData2018
7315  25/03/2020   25      3  2020      0       0                Zimbabwe    ZW                  ZWE   14439018.0
7316  24/03/2020   24      3  2020      0       1                Zimbabwe    ZW                  ZWE   14439018.0
7317  23/03/2020   23      3  2020      0       0                Zimbabwe    ZW                  ZWE   14439018.0
7318  22/03/2020   22      3  2020      1       0                Zimbabwe    ZW                  ZWE   14439018.0
7319  21/03/2020   21      3  2020      1       0                Zimbabwe    ZW                  ZWE   14439018.0

Note that you can also pass an integer into head or tail like df.head(10) to get the first or last n number of rows.

It looks like based on the tail function we have around 7320 rows of data (since the first row is 0 indexed). We can confirm how much data is in each column with the count function.

df.count()

count's output will look like:

dateRep                    7320
day                        7320
month                      7320
year                       7320
cases                      7320
deaths                     7320
countriesAndTerritories    7320
geoId                      7306
countryterritoryCode       7254
popData2018                7311
dtype: int64

What if we want to look at one of those columns and find, for example, the highest value of cases?

df.cases.max()

In this data set we get 18695 as the output. What about looking at standard statistical measures across all columns? That's where the describe function comes in handy.

df.describe()

describe presents standard statistical measures such as min, max, median and mean for everything in your data set. In this case we receive as output:

               day        month         year         cases       deaths   popData2018
count  7320.000000  7320.000000  7320.000000   7320.000000  7320.000000  7.311000e+03
mean     16.828142     2.249454  2019.990847     80.870355     3.687158  7.130483e+07
std       8.322981     1.256463     0.095239    608.270244    35.327689  2.140624e+08
min       1.000000     1.000000  2019.000000     -9.000000     0.000000  1.000000e+03
25%      10.000000     1.000000  2020.000000      0.000000     0.000000  4.137309e+06
50%      18.000000     2.000000  2020.000000      0.000000     0.000000  1.072767e+07
75%      24.000000     3.000000  2020.000000      5.000000     0.000000  5.139301e+07
max      31.000000    12.000000  2020.000000  18695.000000   971.000000  1.392730e+09

How about a quick view into whether or not columns' data are correlated with each other? The corr function is what we need.

df.corr()

For our data set, corr outputs:

                  day     month      year     cases    deaths  popData2018
day          1.000000  0.203006 -0.163665  0.063629  0.060075    -0.040677
month        0.203006  1.000000 -0.745912  0.062494  0.052707    -0.039131
year        -0.163665 -0.745912  1.000000  0.012715  0.010032    -0.006294
cases        0.063629  0.062494  0.012715  1.000000  0.716968     0.136580
deaths       0.060075  0.052707  0.010032  0.716968  1.000000     0.082229
popData2018 -0.040677 -0.039131 -0.006294  0.136580  0.082229     1.000000

Not surprisingly, we see 1.000000 correlation between a column and itself. We'd have to worry if we didn't see that result! For other columns it may not make sense to look at their correlation. This is where you need to think about the data. There is often correlation between completely unrelated columns just because the data is structured a certain way.

If you are a developer like me without a rigorous background in statistics (Stats 200 in college was a long time ago), you may need to brush up on your stats knowledge before you are able to say whether something in the data matters or not.

Let's keep going exploring the data. We can select columns and determine how many unique items are held within it. For example, how many unique countries and territories are listed?

df.countriesAndTerritories.nunique()

In this case the result should be 196.

Asking questions of the data

Those functions are fine for basic querying to learn what's in the data set, but how do we ask real questions by stringing together some commands?

We now know there are 7320 rows in this set since we used the count function above. Each row represents a single day within a country. Now to ask a question. How many days across these countries were there 10 or more cases reported?

Let's create a new dataframe named df2 with the rows that only have 10 or more cases reported on that day, then count the number of rows within it.

df2 = df[df['cases']>=10]
df2.count()

That should give us the value 1531. There have been 1531 instances of 10 or more COVID-19 cases reported on a single day, across the 196 countries or terrorities listed. But the 1531 is hard to explain to people. We should pick out a single country and show how many times 10 or more cases were reported on one day. How about a smaller country like Vietnam that is not being reported on as much as China, the United States or Italy?

df2[df2['countriesAndTerritories']=='Vietnam']

This will give us the full output of data by column:

         dateRep  day  month  year  cases  deaths countriesAndTerritories geoId countryterritoryCode  popData2018
7217  28/03/2020   28      3  2020     16       0                 Vietnam    VN                  VNM   95540395.0
7219  26/03/2020   26      3  2020     14       0                 Vietnam    VN                  VNM   95540395.0
7220  25/03/2020   25      3  2020     11       0                 Vietnam    VN                  VNM   95540395.0
7222  23/03/2020   23      3  2020     24       0                 Vietnam    VN                  VNM   95540395.0
7226  19/03/2020   19      3  2020     15       0                 Vietnam    VN                  VNM   95540395.0

We can also use the count function here to confirm there have been five days in which 10 or more new cases have been reported in Vietnam so far:

df2[df2['countriesAndTerritories']=='Vietnam'].count()

We get the output of 5 for the columns. Unfortunately, when you look at the full data it appears these rows are all very recent and the virus is just beginning to spread more widely there. Let's hope they along with every other country is able to turn the tide, flatten the curve and keep more people from getting sick as we continue onwards.

That's a good spot to leave off, but we covered a lot of pandas ground in this tutorial!

What's next?

We just imported and took a look at what's in the European Centre for Disease Prevention and Control's COVID-19 data set using pandas. That was a quick tour of some basic pandas commands and I strongly recommend you peruse the DataFrame documentation list to learn about all of the other handy functions that this tool provides to developers.

You can also get an idea of what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

The Best Resources for Developers to Learn Finance

Matt Makai — Sun, 08 Mar 2020 00:00:00 -0500

Software developers should understand the basics of finance not only to manage their own money but also to understand how businesses' software projects are funded.

Understanding how other people who work in accounting, finance and project management think about business and finance in particular can help you make better architectural decisions when trying to build maintainable systems. Code is only one aspect of a large software project so working with others and viewing the world through their discipline will help you immensely as you advance your career.

Newsletters & Podcasts on Finance

The fastest way to take a first step in improving your financial literacy is to subscribe to a few free newsletters that regularly hit your inbox, or a podcast if listening better fits your daily routine. I read and listen to each of the following newsletters and podcasts to pick up on unfamiliar topics then do more of my own research if I do not understand what they are talking or writing about.

Money Stuff by Matt Levine of Bloomberg (newsletter sign up form) is a hilarious must-read daily newsletter that covers the world of finance and breaks down many absurd situations such as financial fraud, insider trading, or competing interests in credit default swaps. Amazingly, the author stays out of political topics, which I find very refreshing because many other journalists seem to force their own biases about finance down your throat even if you do not want their opinions.
Endless Metrics explains financial topics in a way that's easy for anyone without a finance background to understand. For example, what the heck is GDP and how do you read a GDP chart?. What I love most about this newsletter is that the author will often venture into finance-related topics he's interested in and then explain those subjects while grounding them with useful charts and data. This analytic approach closely matches how my developer brain processes information!
Points of Return by John Auther (newsletter sign up form). This author is incredibly knowledgeable about finance and typically provides a solid grounding in long-term fundamentals rather than the short-term hyperbole that is pervasive in cable television financial journalism.
Odd Lots covers kind of whatever topics the hosts find interesting such as pandemic bonds, repo market disruption, sovereign debt restructuring and emerging markets. That's why it's so good - the hosts bring on an expert in that topic and ask a ton of great questions because they want to learn what's going on for themselves. You follow along with them as they try to understand some of the oft-esoteric subject areas of finance.

Books, Websites and Magazines for Finance

Newsletters and podcasts are great for prodding you into discovering topics you did not know you needed to learn. When you discover something that you want to go deeper on in finance, here are a few of my favorite books and websites that range from the very basics of finance to broader macroeconomic data trends.

I learned most of my basic finance knowledge when I read Financial Intelligence for IT Professionals in graduate school (go Hoos!). The book is well-written, straightforward and accessible, particularly because it clearly targets its software developer audience.
Don't Quit Your Day Job uses a ton of metrics and statistics to ground their articles on financial topics that are often relevant specifically to software developers. For example, the article on How Many Developers are There in America, and Where Do They Live? is fascinating and especially useful because they explain their data sources and analysis methodology.
Money Magazine can be useful to pick up in paper edition for a few months to understand personal finance basics. After a few months you'll discover the articles and topics tend to recycle so there are diminishing returns to reading it after you have familiarized yourself with most of the topics.
Longtermtrends aggregates long term high-level financial data and displays it. I find looking at these charts gets me away from the day-to-day "oh the stock market is down" and towards thinking about what happens when you invest money over many years or decades.

Specific Articles on Financial Topics

The following individual articles I have found to be both well-written and extremely useful for specific scenarios such as evaluating stock-based equity compensation, or negotiating your salary.

Basic Data Types in Python 3: Booleans

Kevin Whinnery — Fri, 15 Nov 2019 00:00:00 -0500

Welcome back to our ongoing series of blog posts on basic data types in Python 3! Last time, we explored the functionality of strings. Today, we dive in to another key data type - booleans. Booleans (and "boolean logic") are an important concept in programming, representing the concept of "true" and "false".

If you're learning Python, you might also want to check out TwilioQuest 3. You'll learn about basic data types like the boolean, and much more about Python programming.

Ready to learn how to use booleans in Python 3? Let's get started!

Booleans in Python 3

Booleans are a concept that exists in every programming language. A boolean represents the idea of "true" or "false". When you are writing a program, there are often circumstances where you want to execute different code in different situations. Booleans enable our code to do just that.

You can declare a boolean value in your code using the keywords True and False (note the uppercase). The following code would create two boolean values and assign them to variables.

mullet_looks_good = False
python_is_fun = True

More commonly, a boolean value is returned as a result of some kind of comparison. The following code example would store a boolean value of False in the have_same_name variable after using the equality comparison operator, the == symbol.

my_name = "Wammu"
your_name = "Kars"

have_same_name = my_name == your_name

Boolean logic

Booleans are used in your code to make it behave differently based on current conditions within your program. You can use boolean values and comparisons in conjunction with the if, elif, and else keyoards as one means to achieve this.

my_age = 10

if my_age >= 100:
  print("One hundred years old! Very impressive.")
elif my_age <= 3:
  print("Awwww. Just a baby.")
else:
  print("Ah - a very fine age indeed")

In addition to testing for truth, you can also check if conditions are not true using the not keyword.

favorite_team = "Vikings"

if not favorite_team == "Vikings":
  print("Oh - how unfortunate.")
else:
  print("Skol, Vikings!")

More complex boolean logic

Sometimes you will need to evaluate multiple conditions in your boolean logic. For this purpose, you'll combine the and and or keywords. The and keyword compares two boolean values and returns True if both are true. The or keyword compares two values and returns True if any of the statements are true.

Let's look at an example. That uses the in keyword to see if a string is inside a list of values (we'll cover lists in a future article).

favs = ["Donatello", "Raphael"]

if "Michelangelo" in favs and "Donatello" in favs:
  print("Those are my favorite ninja turtles too!")
elif "Michelangelo" in favs or "Donatello" in favs:
  print("Well, one out of two isn't bad...")
else:
  print("Huh - not what I would have chosen.")

Wrapping up

Booleans are an important tool in any programming language. Using boolean logic, your code can react to data inside your program, and carry out different instructions under different circumstances. Hopefully, you've learned a bit about how to work with booleans in Python 3! Stay tuned for more blog posts in this series to learn more about basic data types like strings, numbers, booleans, lists, and dictionaries.

Also, be sure to download and play TwilioQuest 3 to learn even more about Python!

Basic Data Types in Python 3: Strings

Kevin Whinnery — Fri, 18 Oct 2019 00:00:00 -0400

There is a lot to learn on your Python journey when you are new to the programming language. Once you are comfortable writing and executing code, your first stop becomes understanding how to represent data in your code. No matter the language, there are a few basic data types you'll use all the time - strings, numbers, booleans, lists, and dictionaries.

Those data types, and how to use them in Python 3, are the topic of this blog post series. Today, we're starting with strings.

If you're learning Python, you might also want to check out TwilioQuest 3. You'll learn about basic data types and much more about Python programming.

Ready to learn how to use strings in Python 3? Let's get started!

Strings in Python 3

One of the most common data types in any programming language is a string. A string represents a series of characters, which you would use to represent usernames, blog posts, tweets, or any text content in your code. You can create a string and assign it to a variable like this.

my_name = "Jonathan Joestar"

Strings are "immutable"

In Python, strings are considered immutable - once you create them, they can't be changed. You can, however, use a variety of methods to create new strings from existing strings. This type of work in programming is called string manipulation. Some web developers joke that at the end of the day, their job is just mashing strings together - and this isn't far from the truth!

Here are some common tasks you might undertake when using strings in your code.

Common task - combining strings together

Combining strings together - concatenating them - is a very common task. In Python 3, you can use the + operator for this purpose. You can use the + operator multiple times to concatenate multiple strings.

first_name = "Jonathan"
last_name = "Joestar"

full_name = first_name + " " + last_name

Common task - inserting data into strings

Another common task with strings is inserting data into a specific place within a string. In programming, we call this string interpolation. Python 3 provides a handy tool for doing this called "f" strings. The "f" in "f strings" stands for format - you can insert other data from your program into a string when you define it rather than doing complex string concatenation as demonstrated previously.

Here is an example of creating a formatted string - note the letter f is included just before the first double quote when defining the message variable. When you want to insert data from your program into the string, you can include it between two "curly braces" - the { and } characters.

first_name = "Jonathan"
last_name = "Joestar"
age = 24

message = f"My name is {first_name} {last_name}, and I am {age} years old."
print(message)

Common task - using built-in string methods to manipulate strings

String objects have a number of methods to perform common tasks, like changing the case of strings or trimming their content. Below, you'll find a few examples. In two of these examples, we are creating a string variable, and then assigning the same variable a new value, which is the result of calling a method on a string object.

Example 1: Convert a string to all caps using the upper method.

example_string = "am I stoked enough yet?"
example_string = example_string.upper()
print(example_string) # prints "AM I STOKED ENOUGH YET?"

Example 2: Replace all instances of the word kale with tacos.

example_string = "We're having kale for dinner! Yay kale!"
example_string = example_string.replace("kale", "tacos")
print(example_string) # prints "We're having tacos for dinner! Yay tacos!"

Example 3: Split a comma-delimited string into a list of strings.

example_string = "Apples,Oranges,Pears"
groceries = example_string.split(',')

# Code below prints:
# Apples
# Oranges
# Pears
for item in groceries:
    print(item)

Check our more strings can do in the Python 3 docs!

Type casting

Frequently, you will want to convert data from one type into another. In programming, we call this process type casting. There are a number of functions built in to Python which allow us to do these type conversions on basic data types.

Example 1: Convert a number into a string using the str function.

example_number = 42
converted = str(example_number)
message = "The meaning of life is " + converted

Example 2: Convert a string into a whole number (integer) using int.

example_string = "2"
converted = int(example_string)
message = f"Two plus two equals { converted + 2 }"

Wrapping up

Strings of text are one of the most common pieces of data you will work with in programming. Hopefully, you've learned a bit about how to work with strings in Python 3! Stay tuned for more blog posts in this series to learn more about basic data types like strings, numbers, booleans, lists, and dictionaries.

Also, be sure to download and play TwilioQuest 3 to learn even more about Python!

Developer-led Sales for Startups

Matt Makai — Thu, 11 Jul 2019 00:00:00 -0400

This blog post contains the slides along with a loose transcript from my talk on the promises and perils of developer-led sales as an early-stage company method to acquire customers.

I gave this talk remotely to Ubiquity.VC portfolio company startup founders and the Extended Team on June 26, 2019.

Hey folks, my name is Matt Makai. I serve the Developer Network at Twilio. I'm also part of the Extended Team at Ubiquity Ventures.

I often meet startup founders who ask me "how can my company do what Twilio does with developers?"

I respond by asking, "what do see Twilio doing and want to replicate for developers who interact with your business?"

That's when they usually tell me about their "the dev-led sales dream".

Their dream is that software developers go to tech events, like hackathons, conferences and meetups. While at an event, developers see a new API or technical product that looks interesting.

Immediately those developers implement the API in their own projects.

Boom, success!

You make more money as the company that the developer works for uses increasing amounts of your product. Take that money, reinvest in the business to grow and improve your product.

Rinse and repeat until the IPO and beyond.

There are examples of this developer-led marketing and sales success story across many early-stage and mid-stage companies. For example, Rollbar and Datadog are doing well by focusing on developer adoption with great technical content for their monitoring and analytics tools.

Postman recently raised $50 million in a Series B round of venture capital to expand their API testing developer tools. Postman was founded in large part because the original tool was virally adopted by developers building APIs and microservices.

DigitalOcean's extensive community-written tutorials have endeared them to developers and has allowed them to compete in a world where providing cloud infrastructure pits you against AWS, Google and Microsoft.

Citus Data's developer-focused database content and PostgreSQL offerings led them to a successful exit via Microsofta acquisition.

Then you have the massive success stories of developer-focused companies, such as publicly-traded Okta Pluralsight, Slack and Twilio. Still-private Stripe recently raised more venture capital money at a $22 billion valuation which is more than those four public companies.

Amazon Web Services (AWS), while not independent of its mega cap parent company, arguably is by far the leader in cloud computing infrastructure due to its initial and continuing focus on grassroots developer product adoption.

But how does that startup dream match the reality of the work and skills required to execute on the vision?

You, as the founder, should serve as the chief evangelist for your product during the early stages of your company. That role never stops regardless of how large and successful you become.

But your time is split in a hundred different ways so eventually you need to hire someone with a nuanced understanding of how to appropriately market your company to developers. Hiring the wrong person for the job is going to be the equivalent of adding a net negative producing programmer to your development team at a critical stage of growth. A poor result could sink your entire company.

The right person for the job likely has a combination of significant heads-down software development experience, humility, patience, strong writing and solid speaking abilities.

That combination of skills is exceedingly rare, which according to supply and demand dictates a large compensation premium over a programmy-only skill set. If you've done any recruiting for top notch developer relations folks, you may have already discovered this issue which is prohibitive for startup companies trying to compete with better-capitalized firms.

We've done a bunch of talking already about the developer-led sales dream for startups, but what exactly is developer-led sales?

Developer-led sales mean that developers are your primary customers. The goal is to have developers see the value in your product, sign up, start using it and spend some amount of money that is relevant to your business. This approach typically works best with a usage-based model where development is cheap or free then costs scale with the consumption of your product.

For example, Twilio's SMS API is free to get started with during development. Then for testing and production text messages can be sent and received for .00075 US dollars each. A developer can sign up with their credit card to test that your product actually does what you say it can do, then scale up as their application goes to production.

Developer-led sales stands in contrast to the typical business model where a business development representative (often shortened to "BDR") prospects for potential customers, outbound emails and calls those prospects, then hands off to a more senior sales representative to work on closing a deal that is large enough to justify the time invested in the sales cycle.

In this traditional model, your startup sells directly to stakeholders who control a relatively large budget and are authorized to spend that money. Developers usually do not have significant budget to spend but can often spend a small amount on a credit card and expense it without an issue. That's why dev-led sales is possible as an alternative sales model.

We should also confirm what developer-led sales is not. It's not sales engineering, where your technical folks support a traditional top-down large sales deal.

Developer-led sales is also not developer-only sales. Other stakeholders remain important. When your product is successfully implemented then you still have to ensure non-developer stakeholders are part of your sales process.

Here's another way to think about a developer-led approach to product adoption and sales.

Before your product exists, no developers anywhere are using and paying for what you are offering, as represented by this dark world map.

A few risk-taking developers who have a problem you are solving "raise their hands" with code by using your product or service once you launch.

When you are successful, more and more developers use your products and serve as indicators that your product is solving a real problem. Those developers can be your first hook into larger deals with organizations where those developers work.

When should you consider a developer-led sales strategy instead of a tried-and-true traditional sales motion?

First, you need to be solving an actual problem that developers recognize is an issue for them that they themselves would not want to solve. Solving a meaty technical problem with an easy-to-use solution is a high bar that non-technical founders often take for granted when pitching their product.

Is the problem your product solves... actually a problem worth solving for developers? It must be if you want to be successful with a developer-led sales model.

Second, you need to be able to properly articulate the problem that you solve to developers. That is an unexpectedly large challenge if no one on the founding team of your tech startup is a developer themselves.

That link contains the slides and a loose transcript to another talk that provides some guidance around telling your product's story to developers.

Third, even at an early stage you should have some understanding of your own business model. The business model should include a working hypothesis for both the cost of customer acquisition (CAC) and lifetime value of a customer (LTV). Also, does the revenue derived from your customers match a standard distribution or a power law distribution?

What do CAC, LTV and standard vs. power law distribution have to do with a developer-led sales motion?

Ideally, your CAC will be significantly lower with a developer-led sales strategy versus a traditional sales motion. This is one reason established companies such as Okta acquire smaller firms such as Stormpath to create a different sales cycle based on usage from developers instead of trying to convince companies to sign large deals before the software has proven useful.

Note that I am making a lot of assumptions about your understanding of developer platforms. If what I am saying about CAC, LTV, usage-based models and related terms are unclear, you definitely need to read Bessemer's articles on 10 Laws of Cloud Computing and 8 Laws for Developer Platforms.

They are both incredibly helpful foundational resources for understanding how to build, market and sell software products for developers.

Finally, if you have a product that solves a real problem for developers, can explain it to them and have a working model of your CAC and LTV, then a massive bonus comes into play when you have a broad platform useful across many industries. Broad platforms that developers can use in creative ways can potentially create new use cases for company to sell.

For example, when Twilio started the SMS API, 2-factor authentication was rare. The ease of integrating the API to send unique codes to pre-qualified phone numbers to enhance security beyond a password became a new use case for the company to sell to customers. Developers came up with the use case and deserve the credit for figuring out what was valuable about an SMS API.

Most startup founders I talk to go straight into questions about the tactics that Twilio and other companies use for selling to developers. It's a bit of a "the cart before the horse" situation because you really need to have figured out what we talked about in the previous section as it applies to your business before you dig into the specific tactics you will use to engage developers.

Let's assume you have determined a developer-led sales motion fits well with your company strategy. Now we can dig into the myriad tactics you can apply to grow your sales funnel.

Most of these tactics are better described as "developer marketing" rather than sales. You are not going to be cold calling developers to pitch them your service. Please make sure you never do that.

Let's dig into several of the specific tactics that can work well to jump start developers adopting your product when it solves a problem that they have.

We'll start with a few online tactics then review in-person "offline" event-related activities like startup founders' dream hackathons.

By far your most important marketing asset is having stellar developer documentation. However, documentation rarely comes up in conversations as part of marketing and sales efforts. Why is that?

Historically, documentation has been a by-product of an actual product or service being offered. Docs are considered a necessary evil and cost center that needs to be minimized as a time suck for engineers to work on "real" problems. Sometimes writers with minimal or no development experience are hired to just get it done.

The "necessary evil" attitude makes sense in a world where documentation lives in a large paper binder that is mailed out after a purchase is complete. The quality has to be just enough that the customer doesn't complain about the complexity being so high their developers cannot eventually (after some long unpaid additional work nights and weekends) complete an implementation.

Documentation is your evergreen always-to-date content, but there are additional formats such as blog post walkthroughs, example use cases and brand awareness tutorials related to hot topics that developers are currently interested in learning.

Alright, I am going to start talking about online social with the caveat that I personally think it's difficult if not impossible to pull off as a decent developer adoption tactic. It's usually done so poorly that it is better to not do social channels like Twitter, Facebook and your own Slack instance at all.

Video tutorials that teach how to use your product which you can post on YouTube and other sites are more difficult to create than straight text and code, but they can be well worth the effort. YouTube is currently the second largest search engine in the world (amazing how both number 1 and number 2 are the same company) and the video search results are also cross-posted in Google standard search.

The wide reach presents a strong acquisition model if you can create useful technical content.

Live coding on Twitch and YouTube are a nascent but promising teaching method.

Evergreen documentation, timely technical content, online social, asynchronous videos and streaming are the most common online developer adoption tactics. Next we'll look at some offline, in-person tactics.

Conferences split into a couple of categories: community-run and vendor-run. For example, WWDC is Apple's vendor-run developer conference. They run the show and control the messaging. PyCon US is the community-run Python developer conference. There are a ton of vendors there as sponsors but no one company controls what happens at the conference.

Conferences, especially community-run ones, work well for brand awareness. Vendor-run conferences can work well for brand awareness if your product augments an existing company's products and solves a problem for the developers who attend that event.

Non-developer conferences are great for sales lead generation. Think of badge scanning and prize raffles in exchange for hearing a product pitch. However, badge scanning and lead generation activities do not work very well because most developers are skeptical and just want to avoid overly pushy situations. Be considerate of your audience and respectful.

Generally, I find you can spend a ton of money on conference sponsorships without getting a ton of value out of them. If you want to be successful with this tactic you should figure out your secret developer marketing sauce on smaller, regional conferences then expand from there.

Tech meetups are good for brand awareness among primarily less experienced developers. While there are some niche, senior developer and tech lead-focused meetups, most of the content at these events is aimed at less experienced folks.

Professional hackathons have undergone a transition over the past several years. Broadly speaking there is less excitement among developers to spend a long sleep-deprived weekend working on a project with an unclear payoff. I could be projecting my own feelings on this one but it appears that the majority of developers in 2019 would prefer to work on their projects outside the pressure of an arbitrary forced deadline. Hackathons can be still useful for getting a focused product shipped in a weekend but there seems to be less excitement around them than a few years ago.

College hacakthons, like Hack MIT, MHacks and Hoo Hacks, on the other hand, are going stronger than ever. There are always new classes of students who are excited to code their ideas. However, you are playing the long game with college hackathons and assuming that a few years down the road a student who learned your APIs will pick it back up to solve a problem at work. For most early startups, college hackathons won't make a ton of sense unless you want to get quick immediate feedback from junior programmers about your product's developer experience.

That covers the most common online and offline developer marketing and sales tactics. Is that it?

No, there are many other tactics. However, the ones we just listed are the most likely to be successful based on the relatively small investment you can make as a startup.

There are an additional set of tactics that are essentially "magic" to you because they rely on an existing successful foundation to properly execute. Building a new foundation of developer awareness is dramatically different from expanding developer adoption among those are not early adoptors as well as doing a better job of serving developers who already use your service.

Let's wrap with a quick review of the important concepts presented in this talk.

There is a startup dream of developer-led sales which is an easy path from launching your service to developers, having them find out and start to use it, then becoming large paying customers.

The reality is clearly more difficult than the dream. You as the startup founder need to be the "chief developer evangelism officer" and it will be expensive when you aim to hire a seasoned developer relations employee.

We define developer-led sales as a different sales motion than a traditional sales outreach approach. Instead, you appropriately market to developers who need a solution to a problem they are facing, typically with great documentation, timely tutorials or another respectful tactic rather than cold calling or empty clickbait content.

That's all the prepared materials for today. Thank you very much. Again, my name is Matt Makai, I am a software developer at Twilio on the Developer Network and the creator of Full Stack Python (source code).

Introduction to Ansible video course released!

Matt Makai — Sun, 13 Jan 2019 00:00:00 -0500

Check out the just-launched video course, Introduction to Ansible on Talk Python Training. This is the perfect course for you if you want to learn to configure servers and deploy web apps with the Ansible configuration management tool.

My approach in this course is in less than 3 hours to teach you the core concepts then get a ton of hands-on time creating Ansible Playbooks and learning modules for practical applications. I also show you the errors I frequently run into when using Ansible and how to fix them rather than only showing the happy path.

Now that this course has been published I'll be turning my attention back to the Full Stack Python Guide to Deployments book update that uses the latest version of Ansible, Python 3 and Ubuntu 18.04 LTS. More news about the update coming as soon as possible. In addition, the Ansible course pairs very well with the deployments book as they use the same tools but give a different angle on how to learn and use them.

Got questions or comments about Full Stack Python? Send me an email or submit an issue ticket on GitHub to let me know how to improve the site as I continue to fill in the table of contents with new pages and new tutorials.

Adding Okta Authentication to an Existing Flask Web App

Matt Makai — Fri, 02 Nov 2018 00:00:00 -0400

It can be a lot of work to piece together a full authentication system if you have an existing Flask web application that you are coding. Okta makes it much easier to drop-in a complete user authentication system without a lot of additional effort. In this tutorial we will take the Flask Git Dashboard project as an example and add Okta to it.

Libraries

Python 3 is required for this tutorial and we will also use:

Flask web framework version 1.0.2
Flask-OIDC where OIDC stands for "OpenID Connect". It provides support to use OpenID Connect in Flask applications.
Okta Python helper library
A free Okta developer account

All of the finished code in this blog post is provided as open source under the MIT license on GitHub under the auth-existing-flask-app/finished directory of the blog-code-examples repository. Use and abuse the source code for your own applications.

Installing Dependencies

We will start out with an existing Flask web application. If you do not have your own that you are modifying, clone this Git repository:

git clone git@github.com:fullstackpython/blog-code-examples.git

Next, create a new Python virtualenv for this project:

python3 -m venv flaskauth

Activate the virtual environment with the activate script:

. ./flaskauth/bin/activate

The command prompt should change after activation:

Remember that you will have to activate the virtualenv in every terminal window where you want to use the dependencies contained in this virtualenv.

Change into the project directory within the block-code-examples Git repository that you cloned.

cd blog-code-examples/auth-existing-flask-app/start/

Now we can install the dependencies for the existing project.

pip install -r requirements.txt

Look for output similar to the following to confirm that the dependencies successfully installed:

...
Collecting amqp<3.0,>=2.1.4 (from kombu<5.0,>=4.0.2->Celery==4.1.0->-r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/7f/cf/12d4611fc67babd4ae250c9e8249c5650ae1933395488e9e7e3562b4ff24/amqp-2.3.2-py2.py3-none-any.whl (48kB)
    100% |████████████████████████████████| 51kB 10.7MB/s 
Collecting six>=1.5 (from python-dateutil->alembic>=0.6->Flask-Migrate==2.2.0->-r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting vine>=1.1.3 (from amqp<3.0,>=2.1.4->kombu<5.0,>=4.0.2->Celery==4.1.0->-r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/10/50/5b1ebe42843c19f35edb15022ecae339fbec6db5b241a7a13c924dabf2a3/vine-1.1.4-py2.py3-none-any.whl
Installing collected packages: click, itsdangerous, Werkzeug, MarkupSafe, Jinja2, Flask, SQLAlchemy, Flask-SQLAlchemy, Mako, python-editor, six, python-dateutil, alembic, Flask-Migrate, billiard, pytz, vine, amqp, kombu, Celery, redis, WTForms
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for SQLAlchemy ... done
  Running setup.py install for Mako ... done
  Running setup.py install for python-editor ... done
  Running setup.py install for alembic ... done
  Running setup.py install for billiard ... done
  Running setup.py install for WTForms ... done
Successfully installed Celery-4.1.0 Flask-1.0.2 Flask-Migrate-2.2.0 Flask-SQLAlchemy-2.3.2 Jinja2-2.10 Mako-1.0.7 MarkupSafe-1.0 SQLAlchemy-1.2.12 WTForms-2.1 Werkzeug-0.14.1 alembic-1.0.1 amqp-2.3.2 billiard-3.5.0.4 click-7.0 itsdangerous-1.1.0 kombu-4.2.1 python-dateutil-2.7.5 python-editor-1.0.3 pytz-2018.7 redis-2.10.6 six-1.11.0 vine-1.1.4

We need a couple of additional dependencies for our project to work, flask-oidc and okta:

pip install flask-oidc>=1.4.0 okta==0.0.4

The dependencies are now properly installed into our virtual environment. Let's test out the application to see if we can get it running properly.

export FLASK_APP=flaskdash.py
export FLASK_ENV=development
flask run

We should see the application start up with some default development time values:

 * Serving Flask app "flaskdash.py" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 203-814-092

Head to localhost:5000 in your web browser and we should see a work-in-progress dashboard:

It's time to get to setting up an Okta developer account so we can get the appropriate configuration information for our application.

Okta for Authentication

Head to the Okta developers sign up page.

The interesting bit about the Okta developer sign up flow is that now you should check your email to finish creating your account. Look for an email like this one:

Click the "Sign In" button and log into developer account using the temporary password found in the email. Set a new password and challenge question. Then pick an image to match your account login process.

Click the "Create Account" button and you will be wisked away to the Okta developer dashboard.

Find the "Org URL" as shown in the following image.

We are going to use that URL in our secret credentials file so that our Flask web app can properly connect to the Okta service.

Create a new file in your project directory named openidconnect_secrets.json with the following contents:

{
  "web": {
    "client_id": "{{ OKTA_CLIENT_ID }}",
    "client_secret": "{{ OKTA_CLIENT_SECRET }}",
    "auth_uri": "{{ OKTA_ORG_URL }}/oauth2/default/v1/authorize",
    "token_uri": "{{ OKTA_ORG_URL }}/oauth2/default/v1/token",
    "issuer": "{{ OKTA_ORG_URL }}/oauth2/default",
    "userinfo_uri": "{{ OKTA_ORG_URL }}/oauth2/default/userinfo",
    "redirect_uris": [
      "http://localhost:5000/oidc/callback"
    ]
  }
}

Replace the four {{ OKTA_ORG_URL }} placeholders with the Org URL value found in your dashboard. We will fill in the rest of the placeholders with actual values as we proceed through the tutorial. My openidconnect_secret.json file would currently have the following values based on my developer dashboard Org URL. Remember that your URL values will be different!

{
  "web": {
    "client_id": "{{ OKTA_CLIENT_ID }}",
    "client_secret": "{{ OKTA_CLIENT_SECRET }}",
~~    "auth_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/authorize",
~~    "token_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/token",
~~    "issuer": "https://dev-860408.oktapreview.com/oauth2/default",
~~    "userinfo_uri": "https://dev-860408.oktapreview.com/oauth2/default/userinfo",
    "redirect_uris": [
      "http://localhost:5000/oidc/callback"
    ]
  }
}

Okay awesome, we have our Okta account set up so we can add the authentication code to our Flask application.

Updating the Flask App with Okta

We need to connect our Flask code to our new Okta account. The recommended way of including variables such as account credentials in a Flask application is through configuration handling.

Update config.py the Flask code with the following highlighted lines.

import os


class Config(object):
    SECRET_KEY = os.getenv('SECRET_KEY') or 'development key'

    # Redis
    REDIS_SERVER = os.getenv('REDIS_SERVER') or 'localhost'
    REDIS_PORT = os.getenv('REDIS_PORT') or 6379
    REDIS_DB = os.getenv('REDIS_DB') or 1
    REDIS_URL = 'redis://{}:{}'.format(REDIS_SERVER, REDIS_PORT)

    # Celery task queue
    CELERY_BROKER_URL = os.getenv('CELERY_BROKER_URL') or REDIS_URL
    CELERY_RESULT_BACKEND = os.getenv('CELERY_RESULT_BACKEND') or REDIS_URL

    # database settings
    SQLALCHEMY_DATABASE_URI = os.getenv('DATABASE_URL') or \
      'sqlite:///' + os.path.join(os.path.abspath(os.path.dirname(__file__)),
      'flaskdash.db')
    SQLALCHEMY_TRACK_MODIFICATIONS = False
~~
~~    OIDC_CLIENT_SECRETS = "openidconnect_secrets.json"
~~    OIDC_COOKIE_SECURE = False
~~    OIDC_CALLBACK_ROUTE = "/oidc/callback"
~~    OIDC_SCOPES = ["openid", "email", "profile"]
~~    OIDC_ID_TOKEN_COOKIE_NAME = "oidc_token"

We first add three import lines, one to pull values from environment variables, and the next two imports to make it possible to use OpenID Connect and Okta in our application.

The rest of the new code sets Flask application configuration values that can be used to instantiate the OpenID Connect and Okta clients.

OIDC_CLIENT_SECRETS: the location of the OpenID Connect secrets file
OIDC_COOKIE_SECURE: allows development mode for testing user login and registration without SSL. Your application must set this to True in a production application.
OIDC_CALLBACK_ROUTE: URL in the web app for handling user logins
OIDC_SCOPES: what data to request about the user when they log in. Our application requests the basic email, name and profile information
SECRET_KEY: this is a Flask setting to keep sessions secure. The key must never be made public or your web application user sessions will be compromised.

Where do we get those application configuration values though? We need to obtain them from our Okta account so go back to the dashboard to create a new OpenID Connect application.

OpenID Connect applications use a client ID and client secret in place of traditional usernames and passwords. The client ID and client secret will tell your authorization server to recognize your application. Press the "Add Application" button.

On the new application screen choose "Web" and then press "Next".

On the next page there are numerous configuration options but only a few values we need to fill in before we can get our credentials. Set the following values to the Name, Base URIs and Login redirect URIs properties:

FlaskApp for Name
http://localhost:5000 for Base URIs
http://localhost:5000/oidc/callback for Login redirect URIs

Those are the three values you need to fill in for now so save the application to create it.

On the next page scroll down to find your client and secret keys.

Copy and paste the client ID and client secret into the following highlighted lines to replace the {{ OKTA_CLIENT_ID }} and {{ OKTA_CLIENT_SECRET }} placeholders.

{
  "web": {
~~    "client_id": "{{ OKTA_CLIENT_ID }}",
~~    "client_secret": "{{ OKTA_CLIENT_SECRET }}",
    "auth_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/authorize",
    "token_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/token",
    "issuer": "https://dev-860408.oktapreview.com/oauth2/default",
    "userinfo_uri": "https://dev-860408.oktapreview.com/oauth2/default/userinfo",
    "redirect_uris": [
      "http://localhost:5000/oidc/callback"
    ]
  }
}

Save the file and make sure to keep it out of version control as those secret values need to stay secret.

We have one more step in the Okta developer dashboard before we upgrade our Flask application with the authentication code: creating an API authentication token. Go to the API tab.

Click the "Create Token" button.

Name the token FlaskToken and copy it. Save the token somewhere safe as we will not be able to access it through the dashboard again. We are going to use this token when setting the OKTA_AUTH_TOKEN environment variable in the next section of this tutorial.

Okay, we finally have all the Okta service configuration and tokens in our openidconnect_secret.json file that we need to finish our application.

Update app/__init__.py with these highlighted lines:

import redis
~~from os import environ
from flask import Flask
from app.utils import make_celery
from config import Config
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
~~from flask_oidc import OpenIDConnect
~~from okta import UsersClient


app = Flask(__name__, static_url_path='/static')
app.config.from_object(Config)
db = SQLAlchemy(app)
migrate = Migrate(app, db) 

# connect to Redis instance
redis_db = redis.StrictRedis(host=app.config['REDIS_SERVER'],
                             port=app.config['REDIS_PORT'],
                             db=app.config['REDIS_DB'])
celery = make_celery(app)


~~# instantiate OpenID client to handle user session
~~oidc = OpenIDConnect(app)
~~# Okta client will determine if a user has an appropriate account
~~okta_client = UsersClient(environ.get("OKTA_ORG_URL"),
~~                          environ.get("OKTA_AUTH_TOKEN"))


from app import routes

We can now access the okta_client in our routes. Open app/routes.py and update the following lines:

from flask import send_from_directory, render_template
from flask import redirect, g
~~from app import app, oidc, okta_client


~~@app.before_request
~~def before_request():
~~    if oidc.user_loggedin:
~~        g.user = okta_client.get_user(oidc.user_getfield("sub"))
~~    else:
~~        g.user = None


@app.route('/js/<path:path>')
def send_js(path):
    return send_from_directory('js', path)


@app.route('/css/<path:path>')
def send_css(path):
    return send_from_directory('css', path)


@app.route("/")
def dashboard():
    return render_template('dashboard.html')


@app.route("/repositories")
~~@oidc.require_login
def repositories():
    return render_template('repositories.html')


~~@app.route("/login")
~~@oidc.require_login
~~def login():
~~    return redirect(url_for(".repositories"))
~~
~~
~~@app.route("/logout")
~~def logout():
~~    oidc.logout()
~~    return redirect(url_for(".landing_page"))

The above new highlighted lines check whether or not a user is logged in before each request. If a route requires a logged in user due to the @oidc.require_login decorator then the user will be redirect to the sign in page. We also added routes under /login and /logout to make it possible to log in and out of the application.

Set three environment variables so our application can use them when we run it. Make sure the placeholders ORG_URL and AUTH_TOKEN are set with your actual Org URL value and auth token from the Okta developer dashboard.

On the command line run the following commands, making sure to replace any placeholder values with your own tokens and URLs:

# this tells Flask we want to run the built-in server in dev mode
export FLASK_ENV=development
# make sure to use a very long random string here that cannot be guessed
export SECRET_KEY='a very long string with lots of numbers and letters'
# this is the same Org URL found on your developer dashboard
# for example, https://dev-860408.oktapreview.com
export OKTA_ORG_URL='ORG_URL'
# this is the API authentication token we created
export OKTA_AUTH_TOKEN='AUTH_TOKEN'

Now re-run the Flask application:

set FLASK_APP=app.py
flask run

You should be in good shape if the development server starts up with output like this:

(flaskauth)$ flask run
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 415-920-546

Head to localhost:5000 in a browser where you are not already logged into your Okta account (an incognito window of your web browser works great).

Let's test the redirect functionality when we try to go to the /dashboard route by going to localhost:5000/repositories. We get redirected to the Okta login page.

Enter your Okta developer username and password to log into your application. For development purposes this will work fine for testing but obviously in a production application you will create other accounts for users to log into.

To unauthenticate your user go to localhost:5000/logout. When you go back to localhost:5000/repositories again you will now have to re-authenticate.

What Now?

We configured an existing Flask application to use Okta for user authentication and identity management via the Okta API.

Next you can try one of the following tutorials to add other features to the Flask application:

You can also determine what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Fresh Tutorials on Full Stack Python

Matt Makai — Mon, 22 Oct 2018 00:00:00 -0400

There are a bunch of new tutorials on Full Stack Python that were written since the last time I sent out an email newsletter. These range from getting started with some popular open source projects to integrating third party APIs to build authentication into Flask applications:

Configure Python 3, Flask and Gunicorn on Ubuntu 18.04 LTS shows you how to set up your Python and Flask development environment on the latest Ubuntu Long-Term Support (LTS) release.
How to Add User Authentication to Flask Apps with Okta covers using OpenID Connect and the Okta API in Flask applications to handle user authentication.
How to Provision Ubuntu 18.04 LTS Linux Servers on DigitalOcean is a quick tutorial for developers who have not seen how easy it is to spin up a virtual private server on DigitalOcean for hosting their Python applications.
Running Bottle Apps in Docker Containers on macOS provides just the basics to start using Docker on macOS to run an example Flask web app.
How to Explain Your Products to Developers is based on a talk I gave to a group of technical founders and investors in Silicon Valley. It's a bit different from my usual step-by-step tutorial in that it gives strong advice based on my experience rather than show how to use an open source project or integrate a third-party API.

How to Provision Ubuntu 18.04 LTS Linux Servers on DigitalOcean

Matt Makai — Sun, 14 Oct 2018 00:00:00 -0400

Python web applications need to be deployed to a production server or service so your users have access to the application.

DigitalOcean is one such service that makes it easy to immediately get access to initially free servers which are low cost (~$5 per month depending on the resources) to continue using after the first few months.

In this tutorial we'll learn how to quickly sign up and spin up an Ubuntu-based Linux server that only you will have access to based on a private SSH key.

Obtain Your Virtual Server

These steps sign you up for a DigitalOcean account and guide you through provisioning a virtual private server called a "Droplet" for $5/month which we configure throughout the rest of the book.

Point your web browser to Digitalocean.com's registration page. Note that this link uses a referral code which gives you $100 in free credit. Feel free to just go to digitalocean.com if you do not want to use the referral link (you will not get the $100 in credit though). Their landing page will look something like the following image.

Register for a new DigitalOcean account. Fill out the appropriate information. When your account is registered and active you can create a new DigitalOcean server, which they call "droplets".

After you finish the registration process you will be able to start creating DigitalOcean servers. Select the "Create" button which opens a drop-down menu. Choose "Droplets" to go to the "Create Droplets" page.

The new droplet configuration screen will appear and look like the following image. The default Ubuntu instance is 16.04, but we will use the newer LTS release 18.04 in this book.

Select the 1 GB memory-sized server for $5 per month. This instance size should be perfect for prototypes, side projects and minimum viable products. Feel free to choose a larger instance size if you want more memory and resources for running your application.

Scroll down and choose the data center region where you want your instance to be located. I typically choose New York because I am on the East Coast of the United Statest in Washington, D.C., and you will want the server to be closest to your users' location.

Next, scroll down and click "New SSH Key". Copy and paste in the contents of your public SSH key. If you do not yet have an SSH key here are a couple of guides that will walk you through creating one:

You can see the contents of a public key using the cat command. For example on my system the command:

cat root.pub

Outputs the contents of my public key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqAY/Le17HZpa4+eSoh2L9FMYaQ7EnLOGkYbcbsiQNpnF4FTAemF7tbvMvjpVLU6P9AVGs6qEeJdgTE2gH8fq881AUsQ8it1gla2oAlc+vOZmqWPYaLIl5g9DkGwvbITXayobDcw9wTN5tOITOxp3BV5jqanqoqDAPH1RGfT6A5vkJFsmu4w7cPsn9tiqfZZdge3WkpMNT1M3ou+ogrAwE6Ra531s3zYVG9y1688BGdYzbQFfU0+Pou6Z43Do6xbh2hAfQ5hUuTG0OrE3b/yhGcxEWz0Y9+wPGmxm3/0ioTfMWUG3LOQn+oMtKX/PXX/qOJuUjszbqYBvSYS3kv2IVFGV2KEIKC1xgUDfw+HOV4HlIosIbc97zY83m0Ft+tFavPaiQYrar3wCsVfRUltSR4EwNnLmvNYeMVSS8jSP2ZSPwbL8GO7xxAAS9Oy12set1f4OxdPhEUB9rEfAssU1mE6J5eq+Drs8KX04OasLSLt7kP7wWA27I9pU/y9NRHxEsO0YbLG7DzfHGl4QVXwDjIA5GpwjQMwZLt+lyGc4hpnuXg+IUR6MXI90Hh64ch32nSC8j/hjnWCWgj8Cyuc4Rd/2OtO5dHpbjSyU5Yza2lzIqFbFRo7aQNaIkBIioJnc1d6mrg9mLxfd5Ef2ez9bUjqcq4K7uH/JAm0H2Vk1VFQ== matthew.makai@gmail.com

Copy and paste this key into the DigitalOcean modal window and give it a memorable name for future reference:

Optionally, give your server a nickname such as flask-deploy-manual. Then click the big green "Create" button at the bottom of the screen.

The server provisioning process will begin and our Ubuntu Linux 18.04 LTS-powered will soon be ready to go.

Ubuntu 18.04 is the current Long Term Support (LTS) release and has a 5 year support lifecycle. This version will receive security updates until April 2023 as shown on the Ubuntu release end-of-life page.

You should now be back on the DigitalOcean dashboard.

Our server is now up and ready for SSH access.

Connect to the server using the IP address associated with it:

# make sure to replace 192.168.1.1 with your server's IP address
# and the "private_key" name with the name of your private key
ssh -i ./private_key 192.168.1.1

You should now be connected to your new server and can proceed with development or deployment.

What's Next?

We just stood up a new virtual private server on DigitalOcean that can be used as a production or development environment.

Next up I recommend either configuring the development environment or deploying your application with one of the following tutorials:

You can also figure out what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

How to Add User Authentication to Flask Apps with Okta

Matt Makai — Wed, 10 Oct 2018 00:00:00 -0400

User authentication is a basic feature in web applications so people can create and access their own accounts. Unfortunately, authentication is not always easy to set up and there are many ways to incorrectly implement login and logout features.

This tutorial walks through how to use the secure identity authentication service called Okta, which is free for up to 1,000 active user accounts, to easily handle user data in Flask applications.

Our Tools

Python 3 is strongly recommended for building applications and this tutorial was built with Python 3.7 although earlier versions of Python 3 should also work fine. In addition to Python 3.x we will also use:

Flask web framework version 1.0.2
Flask-OIDC where OIDC stands for "OpenID Connect". It provides support to use OpenID Connect in Flask applications.
Okta Python helper library
A free Okta developer account

All of the code in this blog post is provided as open source under the MIT license on GitHub under the flask-auth-okta directory of the blog-code-examples repository. Use and abuse the source code for applications you want to build.

Installing Dependencies

Create a new Python virtualenv for this project:

python3 -m venv flaskauth

Activate the virtual environment with the activate script:

. ./flaskauth/bin/activate

The command prompt should change after activation:

Remember that you will have to activate the virtualenv in every terminal window where you want to use the dependencies contained in this virtualenv.

Now we can install Flask and the Okta dependencies.

pip install flask>=1.0.2 flask-oidc>=1.4.0 okta==0.0.4

Look for output similar to the following to confirm that the dependencies successfully installed:

...
Collecting idna<2.8,>=2.5 (from requests>=2.5.3->okta)
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 16.6MB/s 
Collecting urllib3<1.24,>=1.21.1 (from requests>=2.5.3->okta)
  Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 14.0MB/s 
Installing collected packages: MarkupSafe, Jinja2, click, itsdangerous, Werkzeug, flask, pyasn1, pyasn1-modules, rsa, httplib2, six, oauth2client, flask-oidc, chardet, certifi, idna, urllib3, requests, python-dateutil, okta
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for itsdangerous ... done
  Running setup.py install for httplib2 ... done
  Running setup.py install for flask-oidc ... done
  Running setup.py install for okta ... done
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 certifi-2018.8.24 chardet-3.0.4 click-6.7 flask-1.0.2 flask-oidc-1.4.0 httplib2-0.11.3 idna-2.7 itsdangerous-0.24 oauth2client-4.1.3 okta-0.0.4 pyasn1-0.4.4 pyasn1-modules-0.2.2 python-dateutil-2.7.3 requests-2.19.1 rsa-4.0 six-1.11.0 urllib3-1.23

We installed our required Flask and the Okta dependencies so let's get to building the Flask application.

Creating A Basic Flask App

The first step before adding authentication to our Flask application is to write some scaffolding functions. The authentication will hook into these functions, such as signin and signout, to ensure the auth process works properly.

Create a directory for your project named thundercats. Why thundercats? Why not Thundercats?

Within the thundercats directly create a file named app.py with the following initial contents:

# imports for Flask
from flask import Flask, Response


app = Flask(__name__)


@app.route("/lair")
def lair():
    return Response("Thundercats (supposed to be hidden) lair.")


@app.route("/")
def landing_page():
    return Response("Thundercats, Thundercats, hoooooooooooo!")

We can run our Flask app using the following command:

set FLASK_APP=app.py
flask run

Go to localhost:5000 in your web browser and you should see:

Now go to our "hidden lair" at localhost:5000/lair/. Eventually this page should require authentication to access, but for now it appears without any login challenge:

Awesome, our basic app is up and running, let's get to the authentication functionality.

Auth-as-a-Service

Head to the Okta developers sign up page.

The interesting bit about the Okta developer sign up flow is that now you should check your email to finish creating your account. Look for an email like this one:

Click the "Create Account" button and you will be wisked away to the Okta developer dashboard.

Find the "Org URL" as shown in the following image.

We are going to use that URL in our secret credentials file so that our Flask web app can properly connect to the Okta service.

Create a new file in your project directory named openidconnect_secrets.json with the following contents:

{
  "web": {
    "client_id": "{{ OKTA_CLIENT_ID }}",
    "client_secret": "{{ OKTA_CLIENT_SECRET }}",
    "auth_uri": "{{ OKTA_ORG_URL }}/oauth2/default/v1/authorize",
    "token_uri": "{{ OKTA_ORG_URL }}/oauth2/default/v1/token",
    "issuer": "{{ OKTA_ORG_URL }}/oauth2/default",
    "userinfo_uri": "{{ OKTA_ORG_URL }}/oauth2/default/userinfo",
    "redirect_uris": [
      "http://localhost:5000/oidc/callback"
    ]
  }
}

{
  "web": {
    "client_id": "{{ OKTA_CLIENT_ID }}",
    "client_secret": "{{ OKTA_CLIENT_SECRET }}",
~~    "auth_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/authorize",
~~    "token_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/token",
~~    "issuer": "https://dev-860408.oktapreview.com/oauth2/default",
~~    "userinfo_uri": "https://dev-860408.oktapreview.com/oauth2/default/userinfo",
    "redirect_uris": [
      "http://localhost:5000/oidc/callback"
    ]
  }
}

Okay awesome, we have our Okta account set up so we can add the authentication code to our Flask application.

Connecting Flask to Okta

We need to connect our Flask code to our new Okta account. The recommended way of including variables such as account credentials in a Flask application is through configuration handling so we will use that in our account.

Update the Flask code with the following highlighted lines.

# imports for both Flask and Okta connection
~~from os import environ
from flask import Flask, Response
~~from flask_oidc import OpenIDConnect
~~from okta import UsersClient


app = Flask(__name__)
~~# secret credentials for Okta connection
~~app.config["OIDC_CLIENT_SECRETS"] = "openidconnect_secrets.json"
~~app.config["OIDC_COOKIE_SECURE"] = False
~~app.config["OIDC_CALLBACK_ROUTE"] = "/oidc/callback"
~~app.config["OIDC_SCOPES"] = ["openid", "email", "profile"]
~~app.config["SECRET_KEY"] = environ.get("SECRET_KEY")
~~app.config["OIDC_ID_TOKEN_COOKIE_NAME"] = "oidc_token"
~~# instantiate OpenID client to handle user session
~~oidc = OpenIDConnect(app)
~~# Okta client will determine if a user has an appropriate account
~~okta_client = UsersClient(environ.get("OKTA_ORG_URL"),
~~                          environ.get("OKTA_AUTH_TOKEN"))


@app.route("/lair")
def lair():
    return Response("Thundercats (supposed to be hidden) lair.")


@app.route("/")
def landing_page():
    return Response("Thundercats, Thundercats, hoooooooooooo!")

We first add three import lines, one to pull values from environment variables, and the next two imports to make it possible to use OpenID Connect and Okta in our application.

The rest of the new code sets Flask application configuration values that can be used to instantiate the OpenID Connect and Okta clients.

OIDC_CLIENT_SECRETS: the location of the OpenID Connect secrets file
OIDC_COOKIE_SECURE: allows development mode for testing user login and registration without SSL. Your application must set this to True in a production application.
OIDC_CALLBACK_ROUTE: URL in the web app for handling user logins
OIDC_SCOPES: what data to request about the user when they log in. Our application requests the basic email, name and profile information
SECRET_KEY: this is a Flask setting to keep sessions secure. The key must never be made public or your web application user sessions will be compromised.

Where do we get those application configuration values though? We need to obtain them from our Okta account so go back to the dashboard to create a new OpenID Connect application.

On the new application screen choose "Web" and then press "Next".

ThunderFlaskCats for Name
http://localhost:5000 for Base URIs
http://localhost:5000/oidc/callback for Login redirect URIs

Those are the three values you need to fill in for now so save the application to create it.

On the next page scroll down to find your client and secret keys.

Copy and paste the client ID and client secret into the following highlighted lines to replace the {{ OKTA_CLIENT_ID }} and {{ OKTA_CLIENT_SECRET }} placeholders.

{
  "web": {
~~    "client_id": "{{ OKTA_CLIENT_ID }}",
~~    "client_secret": "{{ OKTA_CLIENT_SECRET }}",
    "auth_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/authorize",
    "token_uri": "https://dev-860408.oktapreview.com/oauth2/default/v1/token",
    "issuer": "https://dev-860408.oktapreview.com/oauth2/default",
    "userinfo_uri": "https://dev-860408.oktapreview.com/oauth2/default/userinfo",
    "redirect_uris": [
      "http://localhost:5000/oidc/callback"
    ]
  }
}

Save the file and make sure to keep it out of version control as those secret values need to stay secret.

We have one more step in the Okta developer dashboard before we upgrade our Flask application with the authentication code: creating an API authentication token. Go to the API tab.

Click the "Create Token" button.

Name the token ThunderFlaskCatsToken and copy it. Save the token somewhere safe as we will not be able to access it through the dashboard again. We are going to use this token when setting the OKTA_AUTH_TOKEN environment variable in the next section of this tutorial.

Okay, we finally have all the Okta service configuration and tokens in our openidconnect_secret.json file that we need to finish our application.

Protecting the Lair

Our configuration is set so update the app.py file with the following highlighted lines:

# imports for both Flask and Okta connection
from os import environ
~~from flask import Flask, Response, redirect, g, url_for
from flask_oidc import OpenIDConnect
from okta import UsersClient


app = Flask(__name__)
# secret credentials for Okta connection
app.config["OIDC_CLIENT_SECRETS"] = "openidconnect_secrets.json"
app.config["OIDC_COOKIE_SECURE"] = False
app.config["OIDC_CALLBACK_ROUTE"] = "/oidc/callback"
app.config["OIDC_SCOPES"] = ["openid", "email", "profile"]
app.config["SECRET_KEY"] = environ.get("SECRET_KEY")
app.config["OIDC_ID_TOKEN_COOKIE_NAME"] = "oidc_token"
# instantiate OpenID client to handle user session
oidc = OpenIDConnect(app)
# Okta client will determine if a user has an appropriate account
okta_client = UsersClient(environ.get("OKTA_ORG_URL"),
                          environ.get("OKTA_AUTH_TOKEN"))


~~@app.before_request
~~def before_request():
~~    if oidc.user_loggedin:
~~        g.user = okta_client.get_user(oidc.user_getfield("sub"))
~~    else:
~~        g.user = None


@app.route("/lair")
~~@oidc.require_login
def lair():
    return Response("Thundercats (supposed to be hidden) lair.")


@app.route("/")
def landing_page():
    return Response("Thundercats, Thundercats, hoooooooooooo!")


~~@app.route("/login")
~~@oidc.require_login
~~def login():
~~    return redirect(url_for(".lair"))
~~
~~
~~@app.route("/logout")
~~def logout():
~~    oidc.logout()
~~    return redirect(url_for(".landing_page"))

On the command line run the following commands, making sure to replace any placeholder values with your own tokens and URLs:

# this tells Flask we want to run the built-in server in dev mode
export FLASK_ENV=development
# make sure to use a very long random string here that cannot be guessed
export SECRET_KEY='a very long string with lots of numbers and letters'
# this is the same Org URL found on your developer dashboard
# for example, https://dev-860408.oktapreview.com
export OKTA_ORG_URL='ORG_URL'
# this is the API authentication token we created
export OKTA_AUTH_TOKEN='AUTH_TOKEN'

Now re-run the Flask application:

set FLASK_APP=app.py
flask run

You should be in good shape if the development server starts up with output like this:

(flaskauth)$ flask run
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 415-920-546

Head to localhost:5000 in a browser where you are not already logged into your Okta account (an incognito window of your web browser works great).

Let's test the redirect functionality when we try to go to the /lair route by going to localhost:5000/lair. We get redirected to the Okta login page.

Let's tweak one more bit in our application to fix the glaring lack of excitement in successfully completing the authentication code for this tutorial. Update the two highlighted lines to match what is in the code block below:

# imports for both Flask and Okta connection
from os import environ
from flask import Flask, Response, redirect, g, url_for
from flask_oidc import OpenIDConnect
from okta import UsersClient


app = Flask(__name__)
# secret credentials for Okta connection
app.config["OIDC_CLIENT_SECRETS"] = "openidconnect_secrets.json"
app.config["OIDC_COOKIE_SECURE"] = False
app.config["OIDC_CALLBACK_ROUTE"] = "/oidc/callback"
app.config["OIDC_SCOPES"] = ["openid", "email", "profile"]
app.config["SECRET_KEY"] = environ.get("SECRET_KEY")
app.config["OIDC_ID_TOKEN_COOKIE_NAME"] = "oidc_token"
# instantiate OpenID client to handle user session
oidc = OpenIDConnect(app)
# Okta client will determine if a user has an appropriate account
okta_client = UsersClient(environ.get("OKTA_ORG_URL"),
                          environ.get("OKTA_AUTH_TOKEN"))


@app.before_request
def before_request():
    if oidc.user_loggedin:
        g.user = okta_client.get_user(oidc.user_getfield("sub"))
    else:
        g.user = None


@app.route("/lair")
@oidc.require_login
def lair():
~~    thundercats_lair = '<html><head><title>Thundercats, hoooo!</title></head><body><h1>Thundercats now hidden lair.</h1><iframe src="https://giphy.com/embed/ahXtBEbHiraxO" width="480" height="273" frameBorder="0" class="giphy-embed" allowFullScreen></iframe><p><a href="https://giphy.com/gifs/retro-cartoons-thundercats-ahXtBEbHiraxO">via GIPHY</a></p></body></html>'
~~    return Response(thundercats_lair)


@app.route("/")
def landing_page():
    return Response("Thundercats, Thundercats, hoooooooooooo!")


@app.route("/login")
@oidc.require_login
def login():
    """Force user to login and then redirect them to the lair.
    """
    return redirect(url_for(".lair"))


@app.route("/logout")
def logout():
    oidc.logout()
    return redirect(url_for(".landing_page"))

Refresh the lair page.

Alright that's just a little bit better! Go to localhost:5000/logout to unauthenticate your user. When you go to localhost:5000/lair again you will now have to re-authenticate.

What Now?

We just built an example Flask application with user authentication via the Okta API.

Next up try the following tutorials to add other features to your Flask application:

You can also determine what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Configure Python 3, Flask and Gunicorn on Ubuntu 18.04 LTS

Matt Makai — Fri, 15 Jun 2018 00:00:00 -0400

Ubuntu Linux's latest Long Term Support (LTS) operating system version is 18.04 and was released in April 2018. The 18.04 update is code named "Bionic Beaver" and it includes Python 3 by default. However, there are bunch of dependencies you will need to install to get this release set up as a development environment.

In this tutorial we will get Python 3.6 configured with development system packages to start a new Flask web application project and run it with Green Unicorn (Gunicorn).

Our Tools

Our project will use the Ubuntu 18.04 release along with a few other libraries. Note that if you are using the older 16.04 LTS release, there is also a guide that will walk you through setting up that version as your development environment.

We will install the following tools as we step through the rest of the sections in this tutorial:

Ubuntu 18.04 LTS (Bionic Beaver)
Python version 3.6.5 (default in Ubuntu 18.04)
Flask web framework version 1.0.2
Green Unicorn (Gunicorn) version 19.8.1

If you're running on Mac OS X or Windows, use virtualization software such as Parallels or VirtualBox with the Ubuntu .iso file. Either the amd64 or i386 version for 18.04 will work. I am using amd64 for development and testing in this tutorial.

When you boot up to the Ubuntu desktop you should see a screen like this one.

We're ready to get our development environment configured.

System Packages

Open up a terminal window to proceed with the setup.

Use the following two commands to check which version of Python 3 is installed

python3 --version
which python3

The Python version should be 3.6.5 and the location /usr/bin/python3.

Our Ubuntu installation requires a few system packages to do development rather than just run Python scripts. Run the following apt-get command and enter your sudo password to allow restricted system access.

sudo apt-get install python3-dev python3-pip python3-virtualenv

We should see the following prompt requesting sudo access. Enter y to let the system package manager complete the installation.

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-headers-4.15.0-20 linux-headers-4.15.0-20-generic
  linux-image-4.15.0-20-generic linux-modules-4.15.0-20-generic
  linux-modules-extra-4.15.0-20-generic
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  dh-python libexpat1-dev libpython3-dev libpython3.6-dev python3-setuptools
  python3-wheel python3.6-dev
Suggested packages:
  python-setuptools-doc
The following NEW packages will be installed:
  dh-python libexpat1-dev libpython3-dev libpython3.6-dev python3-dev
  python3-pip python3-setuptools python3-virtualenv python3-wheel
  python3.6-dev
0 upgraded, 10 newly installed, 0 to remove and 11 not upgraded.
Need to get 3,617 kB/3,661 kB of archives.
After this operation, 20.2 MB of additional disk space will be used.
Do you want to continue? [Y/n]

The package manager will do the dirty work and should report when the installation finishes successfully.

(...clipped a bunch of installation lines for brevity...)
Unpacking python3-wheel (0.30.0-0.2) ...
Setting up python3-wheel (0.30.0-0.2) ...
Setting up python3-virtualenv (15.1.0+ds-1.1) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1) ...
Setting up libexpat1-dev:amd64 (2.2.5-3) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up python3-setuptools (39.0.1-2) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up libpython3.6-dev:amd64 (3.6.5-3) ...
Setting up python3.6-dev (3.6.5-3) ...
Setting up libpython3-dev:amd64 (3.6.5-3) ...
Setting up python3-dev (3.6.5-3) ...

The packages we need are now installed. We can continue on to install our Python-specific dependencies.

Virtual environment

We installed virtualenv and pip to handle our application dependencies. We can now use them to download and install Flask and Gunicorn.

Create a directory to store your virtualenvs. Then create a new virtualenv within that directory.

# make sure pip and setuptools are the latest version
pip3 install --upgrade pip setuptools
# the tilde ("~") specifies the user's home directory, such as "/home/matt"
cd ~
mkdir venvs
# specify the system python3 installation
python3 -m venv venvs/flask1804

Activate the virtualenv.

source ~/venvs/flask1804/bin/activate

Our prompt will change when the virutalenv is activated.

Our virtualenv is now activated with Python 3. We can install any dependencies we need such as Flask and Gunicorn.

Flask and Gunicorn

We're going to use pip within our new virtualenv but it's a good idea to update it to the latest version. We should also install the wheel package to remove installation warnings when pip tries to use Python wheels, which are the newest standard in an admittedly long line of Python distribution package models.

pip install --upgrade pip
pip install wheel

We can now install Flask and Green Unicorn via the pip command.

pip install flask gunicorn

Look for output similar to the following to ensure the libraries installed without an issue.

(flask1804) matt@ubuntu:~$ pip install flask gunicorn
Collecting flask
  Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl
Collecting gunicorn
  Using cached https://files.pythonhosted.org/packages/55/cb/09fe80bddf30be86abfc06ccb1154f97d6c64bb87111de066a5fc9ccb937/gunicorn-19.8.1-py2.py3-none-any.whl
Collecting click>=5.1 (from flask)
  Using cached https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl
Collecting Werkzeug>=0.14 (from flask)
  Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl
Collecting itsdangerous>=0.24 (from flask)
  Using cached https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz
Collecting Jinja2>=2.10 (from flask)
  Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask)
  Using cached https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Building wheels for collected packages: itsdangerous, MarkupSafe
  Running setup.py bdist_wheel for itsdangerous ... done
  Stored in directory: /home/matt/.cache/pip/wheels/2c/4a/61/5599631c1554768c6290b08c02c72d7317910374ca602ff1e5
  Running setup.py bdist_wheel for MarkupSafe ... done
  Stored in directory: /home/matt/.cache/pip/wheels/33/56/20/ebe49a5c612fffe1c5a632146b16596f9e64676768661e4e46
Successfully built itsdangerous MarkupSafe
Installing collected packages: click, Werkzeug, itsdangerous, MarkupSafe, Jinja2, flask, gunicorn
Successfully installed Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 flask-1.0.2 gunicorn-19.8.1 itsdangerous-0.24

Create a new directory named flask1804 under your home directory (not within the venvs subdirectory) that will store our Flask test project. Change directory into the new folder.

mkdir ~/flask1804
cd ~/flask1804

Create a new file named __init__.py within our flaskproj directory so we can test to make sure Flask is working properly. I usually use Vim but Emacs and other development environments work great as well.

Within __init__.py write the following code.

from flask import Flask, Response


app = Flask(__name__)

@app.route("/")
def index():
    return Response("It works!"), 200

if __name__ == "__main__":
    app.run(debug=True)

We could run our app with the Flask development server using the python __init__.py command. Instead run the Flask app with Gunicorn. Go to the directory above the flask1804 folder, in our case we can enter cd ~ then use the gunicorn command:

gunicorn flask1804.app:app

We should see:

[2018-06-15 15:54:31 -0400] [5174] [INFO] Starting gunicorn 19.8.1
[2018-06-15 15:54:31 -0400] [5174] [INFO] Listening at: http://127.0.0.1:8000 (5174)
[2018-06-15 15:54:31 -0400] [5174] [INFO] Using worker: sync
[2018-06-15 15:54:31 -0400] [5177] [INFO] Booting worker with pid: 5177

Great now we can bring up our shell Flask app in the web browser at the localhost:8000 or 127.0.0.1:8000 address.

Now you're ready for some real Flask development!

Ready to Code

That provides a quick configuration for getting started on 18.04 LTS developing Flask applications with the Gunicorn WSGI server.

Next up you should check out the following tutorials that use this Flask configuration:

Alternatively you can also determine what to code next in your Python project by reading the Full Stack Python table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Running Bottle Apps in Docker Containers on macOS

Matt Makai — Tue, 05 Jun 2018 00:00:00 -0400

It can be confusing to figure out how to use Docker containers in your Python and Bottle development environment workflow. This tutorial will quickly show you the exact steps to get Docker up and running on macOS with a working Bottle web application

Our Tools

This tutorial is written for Python 3. It may work with Python 2 but it has not been testing with that soon-to-be deprecated 2.7 version. You should really be using Python 3, preferrably the latest release which is currently 3.6.5.

Docker for Mac is necessary to run Docker containers. I recommend that you use the stable release unless you have an explicit purpose for the edge channel.

Within the Docker container we will use:

Python 3, specifically the slim-3.6.5 version from Docker Hub
Bottle version 0.12.13

All for the Dockerfile and the Bottle project are available open source under the MIT license on GitHub under the docker-bottle-mac directory of the blog-code-examples repository.

Installing Docker on macOS

We must install Docker before we can spin up our containers. Jump to the next section if you already have Docker for Mac installed and working on your computer.

On your Mac, download the Docker Community Edition (CE) for Mac installer.

Open Finder and go to the downloads folder where the installation file is located. Follow the installation steps and open Terminal when the installer finishes.

Test your Docker installation by running the docker command along with the --version flag:

docker --version

If Docker is installed correctly you should see the following output:

Docker version 18.03.1-ce, build 9ee9f40

Note that Docker runs through a system agent you can find in the menu bar.

Docker is now installed so we can run a container and write a simple Bottle application to test running an app within the container.

Dockerfile

Docker needs to know what we want in our container so we specify an image using a Dockerfile.

# this is an official Python runtime, used as the parent image
FROM python:3.6.5-slim

# set the working directory in the container to /app
WORKDIR /app

# add the current directory to the container as /app
ADD . /app

# execute everyone's favorite pip command, pip install -r
RUN pip install --trusted-host pypi.python.org -r requirements.txt

# unblock port 80 for the Bottle app to run on
EXPOSE 80

# execute the Flask app
CMD ["python", "app.py"]

Save the Dockerfile and then on the commandline run:

docker build -t bottledock .

The above docker build file uses the -t flag to tag the image with the name of bottledock.

If the build worked successfully the shell will show some completed output like the following:

$ docker build -t bottledock .
Sending build context to Docker daemon  16.38kB
Step 1/6 : FROM python:3.6.5-slim
3.6.5-slim: Pulling from library/python
f2aa67a397c4: Pull complete 
19cc085bc22b: Pull complete 
83bd7790bc68: Pull complete 
8b3329adba1b: Pull complete 
d0a8fd6eb5d0: Pull complete 
Digest: sha256:56100f5b5e299f4488f51ea81cc1a67b5ff13ee2f926280eaf8e527a881afa61
Status: Downloaded newer image for python:3.6.5-slim
 ---> 29ea9c0b39c6
Step 2/6 : WORKDIR /app
Removing intermediate container 627538eb0d39
 ---> 26360255c163
Step 3/6 : ADD . /app
 ---> 9658b91b29db
Step 4/6 : RUN pip install --trusted-host pypi.python.org -r requirements.txt
 ---> Running in f0d0969f3066
Collecting bottle==0.12.13 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/bd/99/04dc59ced52a8261ee0f965a8968717a255ea84a36013e527944dbf3468c/bottle-0.12.13.tar.gz (70kB)
Building wheels for collected packages: bottle
  Running setup.py bdist_wheel for bottle: started
  Running setup.py bdist_wheel for bottle: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/76/a0/b4/2a3ee1a32d0506931e558530258de1cc04b628eff1b2f008e0
Successfully built bottle
Installing collected packages: bottle
Successfully installed bottle-0.12.13
Removing intermediate container f0d0969f3066
 ---> 0534575c8067
Step 5/6 : EXPOSE 80
 ---> Running in 14e49938d3be
Removing intermediate container 14e49938d3be
 ---> 05e087d2471d
Step 6/6 : CMD ["python", "app.py"]
 ---> Running in ca9738bfd06a
Removing intermediate container ca9738bfd06a
 ---> 9afb4f01e0d3
Successfully built 9afb4f01e0d3
Successfully tagged bottledock:latest

We can also see the image with the docker image ls command. Give that a try now:

docker image ls

Our tag name should appear in the images list:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
bottledock          latest              9afb4f01e0d3        About a minute ago   145MB

Our image is ready to load as a container so we can code a short Bottle web app for testing and then further development.

Coding A Bottle Web App

It is time to code a simple "Hello, World!"-style Bottle app to test running Python code within our Docker container. Within the current project directory, create a file named app.py with the following contents:

import bottle
from bottle import route, run


app = bottle.default_app()


@route('/')
def hello_world():
    return "Hello, world! (From Full Stack Python)"


if __name__ == "__main__":
    run(host="0.0.0.0", port=8080, debug=True, reloader=True)

The above code returns a simple "Hello, world!" message when executed by the Bottle development server and contacted by a client.

We need just one more file to specify our bottle dependency. Create a requirements.txt file within the same directory as app.py:

bottle==0.12.13

Make sure both the app.py and requirements.txt file are saved then we can give the code a try.

Running the Container

Now that we have our image in hand along with the Python code in a file we can run the image as a container with the docker run command. Execute the following command, making sure to replace the absolute path for the volume to your own directory.

docker run -p 5000:8080 --volume=/Users/matt/devel/py/blog-code-examples/docker-bottle-macapp bottledock

If you receive the error python: can't open file 'app.py': [Errno 2] No such file or directory then you likely did not change /Users/matt/devel/py/bottledocker to the directory where your project files, especially app.py, are located.

Everything worked when you see a simple text-based HTTP response like what is shown above in the screenshot of my Chrome browser.

What's Next?

We just installed Docker and wrote a Bottle web app to run inside a container. That is just the beginning of how you can integrate Docker into your workflow.

Next up take a look at the Bottle, Docker and deployment pages for more tutorials.

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

Do you see a typo, syntax issue or just something that's confusing in this blog post? Fork this page's source on GitHub and submit a pull request with a fix or file an issue ticket on GitHub.

How to Explain Your Products to Developers

Matt Makai — Fri, 25 May 2018 00:00:00 -0400

This blog post contains the slides along with a loose transcript from my talk on appropriately marketing products to software developers that I gave at Silicon Valley Bank during Ubiquity.VC's summit for founders, investors and technical advisors on May 24, 2018.

Hey folks, my name is Matt Makai. I serve the Developer Network at Twilio. We've talked a lot today about making the real, physical world programmable. We ask "what if we could modify the world using GitHub and Jira?" When we succeed in creating programmatic access to the physical world, what then? Is that the end goal?

No, that's only the beginning. We need developers to use those new capabilities and code with them.

How do you get developers to adopt what you are creating? That is a broad question so I am going to zoom in on just one small slice of developer relations that kicks off the whole adoption process. Unfortunately I see upwards of 90% of companies completely screw up explaining their products to developers.

Today we are going to look at how to appropriately explain and demo your product to developers to maximize developer adoption. This is the first step towards getting a developer to care enough to try out what you have built.

In addition to serving the Developer Network at Twilio, I am also a Python and Swift developer as well as the creator of Full Stack Python. My background provides me an opportunity to give insight on this topic because I am a software developer, I market to fellow software developers and I write a community-driven site that is widely read and trusted by software developers.

How do you explain your product? Fred Wilson of Union Square Ventures said it best in this quote, which we will roughly summarize as: show, don't just tell.

With Fred Wilson's quote in mind, it's demo time!

(This is where I do a condensed, approximately two minute version of my Twilio five minute live-coded demo. For a rough approximation of what I showed, check out the NY Tech Meetup Twilio demo from 2010.)

That demo represented the Twilio 5 minute demo from 2008 through part of 2011, when the phone calling voice API was the company's main product.

Let's break down the demo into its component pieces so we can learn from it. The demo narrative fits into a story arc. Yes, a story arc like from a novel. You may not have thought about explaining and showing your product in a couple of minutes to be similiar to a novel, but you should follow the same narrative structure because it is easier for the audience to understand.

The demo we just saw follows the story arc in the beginning when I introduce myself and Twilio. A clear, concise set of intentional words are used to explain what Twilio *can do for a developer*. "Twilio makes it easy for software developers to add phone calling to applications using the programming languages that you already know." Breaking that down further:

software developers: a clear call out to who we are talking to
phone calling: what problem we solve by adding this feature to applications
programming languages that you already know: emphasizing that you do not have to learn some complicated proprietary syntax from the telecommunications world

Next in the exposition, we explain how it works using a diagram that shows inbound and outbound phone calls and how they interact with Twilio's service as well as your web server.

The inciting incident during the demo happens when I finish the explanation of how Twilio works and say "rather than just show you a little diagram, let's build an application together right now".

We move into the demo phase where I buy and configure a phone number then we all test it by calling the number on our own cell phones. The audience learns that to configure the phone number to do something useful in this case only requires two XML elements that can be stored in a static file or generated by an endpoint in their application.

The climax hits when we see outbound phone calling, everyone's phones in the room start ringing and we are all on speaker phone together. Finally, there is a short resolution where I re-explain what Twilio can do for developers and outro with my name and where you can find me.

The whole two minute demo, or however long we need it to be, has a narrative with a clear story arc.

In 2011, Twilio added SMS. This changed the 5 minute demo's explanation to "Twilio makes it easy for developers to send and receive text messages and make and receive phone calls using the programming languages that they already know". The overall structure otherwise remained the same because we used SMS for inbound action and kept phone calling for the outbound action.

Eventually your product line or features within a product line will reach a point where you need to determine if it changes your explanation and demo. In some cases there will be modifications that fit within the existing framework and do not substantially change the narrative.

As you continue to grow you will eventually reach an inflection point where you have too many products or features to explain, regardless of how much time you have for your demo. You reach a situation where if you try to tell the audience everything that your product does, they will zone out and ignore your laundry list of features.

If you are not intentional in the words you say and specific in the products and features you choose to show, then your pitch becomes spread too thin and no developer will care to listen.

Twilio now has dozens of products under the communications umbrella. I talk about specific products and tailor my explanation based on the audience. You should too! For example, if I am talking to a group of web developers, I will still use the classic Twilio 5 minute demo that shows off SMS and phone calling capability. On the other hand, if I am demoing to iOS and Android mobile developers then I will show off Programmable Chat or Programmable Video.

The explanation is tuned to "Twilio makes it easy for developers to add communications, such as phone calling, messaging and video, to their applications using the programming languages that they already know." I draw a broad theme by saying the word "communications" then give three specific examples of products that are the most widely used by developers because they are incredibly useful for implementing common application features.

It's time to reinforce why it is so important for you, as a founder, or as an investor that works with founders, to be the chief evangelist for your product. You cannot ever outsource this role. You cannot hire someone to lead an evangelism team and expect them to figure it all out for you.

If you are not excited about the product you are building or are unable to transfer that excitement to developers with a clear explanation and demo, then all of the other priorities for your company become useless. If developers are your customers and they do not adopt your product then you will not sell anything, you won't be able to set a great company culture and you won't need to worry about what snacks are stocked in your office's kitchen. If developers are the lifeblood of your company then you need to be the chief evangelist, period.

Here are a few more important points for how to perform this role effectively.

When you are early stage, be as specific as possible about what problem that you are solving. You are not "disrupting transportation by blah blah blah". No developer gives a shit. They want to know what problem you will solve for them right now.

Be specific, like the "add phone calling to your applications" line so that it is absolutely clear what you do.

When your company grows and your brand expands, then you may expand to include the general industry your company works in, such as "communications". Do not jump the gun in trying to become too grandiose with your ambitions because your developer audience cares about what problem you are solving for them, not who you imagine yourself to be in your future vision.

Refine the explanation you use and the demo under two situations.

First, when your products and features expand. Think critically if a new feature should be part of your explanation or it can be left as an answer to follow up questions that a developer asks you.

Second, developer ecosystems are constantly changing. If you tried to talk to me about containers ten years ago and I was not a Solaris sysadmin then I would not have any clue what you are talking about. Today, it's generally safe to assume most Bay Area developers have a working knowledge of what containers are and what they are useful for accomplishing. Use that type of context in your pitch to reinforce your technical credibility with your developer audience.

I get asked a lot about live coding because everyone is worried about demo fails. You should not demo fail, ever. To steal a line from Rob Spectre, former head of the Twilio Developer Network:

There is only one demo God, and her name is rehearsal.

You do not just rehearse and practice the happy path, you also practice what can go wrong. What happens when you mistype a character in your code? Find out and get used to it. If and when it happens during your live demo then you can incorporate that mistake into your narrative as a learning opportunity for the audience.

Magicians always have "outs" in their acts, essentially plan B, plan C, plan Z. You should too because something will always go wrong but if you are ready for it and know how to handle it then you will never demo fail.

That sums up my strong recommendations for this one small slice of the field of developer product adoption. To re-iterate, create a narrative for your explanation and demo that follows the classic story arc. The earlier you are as a product, the more specific your explanation should be in what problem you solve. Refine the message as your features and product lines grow, as well as when the industry around you changes. Rehearse your demo including what can and will go wrong.

There is a lot more to developer adoption than a good explanation and demo, but I see greater than 90% of companies never even get to this point so you will be way ahead of the pack if you heed the advice from this talk.

That's all for today. My name is Matt Makai, I am a software developer at Twilio and the author of Full Stack Python. Thank you very much.

How to Add Maps to Django Web App Projects with Mapbox

Matt Makai — Sun, 06 Oct 2019 00:00:00 -0400

Building interactive maps into a Django web application can seem daunting if you do not know where to begin, but it is easier than you think if you use a developer tool such as Mapbox.

In this post we will build a simple Django project with a single app and add an interactive map like the one you see below to the webpage that Django renders with the Mapbox Maps API.

Our Tools

Python 3 is strongly recommended for this tutorial because Python 2 will no longer be supported starting January 1, 2020. Python 3.6.5 to was used to build this tutorial. We will also use the following application dependencies to build our application:

Django web framework, version 2.0.5
pip and virtualenv, which come installed with Python 3, to install and isolate the Django library from your other applications
A free Mapbox account to interact with their web API using JavaScript

If you need help getting your development environment configured before running this code, take a look at this guide for setting up Python 3 and Django on Ubuntu 16.04 LTS.

This blog post's code is also available on GitHub within the maps-django-mapbox directory of the blog-code-examples repository. Take the code and use it for your own purposes because it is all provided under the MIT open source license.

Installing Dependencies

python3 -m venv djangomaps

Activate the virtualenv with the activate shell script:

source djangomaps/bin/activate

The command prompt will change after activating the virtualenv:

Remember that you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the Django package into the activated but otherwise empty virtualenv.

pip install django==2.0.5

Look for the following output to confirm Django installed correctly from PyPI.

  Downloading https://files.pythonhosted.org/packages/23/91/2245462e57798e9251de87c88b2b8f996d10ddcb68206a8a020561ef7bd3/Django-2.0.5-py3-none-any.whl (7.1MB)
      100% |████████████████████████████████| 7.1MB 231kB/s 
      Collecting pytz (from django==2.0.5)
        Using cached https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl
        Installing collected packages: pytz, django
        Successfully installed django-2.0.5 pytz-2018.4

The Django dependency is ready to go so now we can create our project and add some awesome maps to the application.

Building Our Django Project

We can use the Django django-admin.py tool to create the boilerplate code structure to get our project started. Change into the directory where you develop your applications. For example, I typically use /Users/matt/devel/py/. Then run the following command to start a Django project named djmaps:

django-admin.py startproject djmaps

The django-admin.py command will create a directory named djmaps along with several subdirectories that you should be familiar with if you have previously worked with Django.

Change directories into the new project.

cd djmaps

Create a new Django app within djmaps.

python manage.py startapp maps

Django will generate a new folder named maps for the project. We should update the URLs so the app is accessible before we write our views.py code.

Open djmaps/djmaps/urls.py. Add the highlighted lines so that URLs will check the maps app for appropriate URL matching.

""" (comments)
"""
~~from django.conf.urls import include
from django.contrib import admin
from django.urls import path


urlpatterns = [
~~    path('', include('maps.urls')),
    path('admin/', admin.site.urls),
]

Save djmaps/djmaps/urls.py and open djmaps/djmaps/settings.py. Add the maps app to settings.py by inserting the highlighted line:

# Application definition

INSTALLED_APPS = [ 
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
~~    'maps',
]

Save and close settings.py.

Next change into the djmaps/maps directory. Create a new file named urls.py to contain routes for the maps app.

Add these lines to the empty djmaps/maps/urls.py file.

from django.conf.urls import url                                                                                                                              
from . import views

urlpatterns = [ 
    url(r'', views.default_map, name="default"),
]

Save djmaps/maps/urls.py and open djmaps/maps/views.py add the following two highlighted lines. You can keep the boilerplate comment or delete it.

from django.shortcuts import render


~~def default_map(request):
~~    return render(request, 'default.html', {})

Next, create a directory for your template files named templates under the djmaps/maps app directory.

mkdir templates

Create a new file named default.html within djmaps/maps/templates that contains the following Django template markup.

<!DOCTYPE html>
<html>
  <head>
    <title>Interactive maps for Django web apps</title>
  </head>
  <body>
   <h1>Map time!</h1>
  </body>
</html>

We can test out this static page to make sure all of our code is correct, then we'll use Mapbox to embed a customizable map within the page. Change into the base directory of your Django project where the manage.py file is located. Execute the development server with the following command:

python manage.py runserver

The Django development server will start up with no issues other than an unapplied migrations warning.

Performing system checks...

System check identified no issues (0 silenced).

You have 14 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

May 21, 2018 - 12:47:54
Django version 2.0.5, using settings 'djmaps.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Open a web browser and go to localhost:8000.

Our code works, but boy is that a plain-looking HTML page. Let's make the magic happen by adding JavaScript to the template to generate maps.

Adding Maps with Mapbox

Head to mapbox.com in your web browser to access the Mapbox homepage.

Click on "Get Started" or "Get Started for free" (the text depends on whether or not you already have a Mapbox account).

Click the "JS Web" option.

Choose "Use the Mapbox CDN" for the installation method. The next two screens show some code that you should add to your djmaps/maps/templates/default.html template file. The code will look like the following but you will need to replace the mapboxgl.accessToken line with your own access token.

<!DOCTYPE html>
<html>
  <head>
    <title>Interactive maps for Django web apps</title>
~~    <script src='https://api.mapbox.com/mapbox-gl-js/v0.44.2/mapbox-gl.js'></script>
~~    <link href='https://api.mapbox.com/mapbox-gl-js/v0.44.2/mapbox-gl.css' rel='stylesheet' />
  </head>
  <body>
   <h1>Map time!</h1>
~~   <div id='map' width="100%" style='height:400px'></div>
~~   <script>
~~    mapboxgl.accessToken = {{ mapbox_access_token }};
~~    var map = new mapboxgl.Map({
~~     container: 'map',
~~     style: 'mapbox://styles/mapbox/streets-v10'
~~    });
~~   </script>
  </body>
</html>

Re-open djmaps/maps/views.py to update the parameters passed into the Django template.

from django.shortcuts import render


def default_map(request):
~~    # TODO: move this token to Django settings from an environment variable
~~    # found in the Mapbox account settings and getting started instructions
~~    # see https://www.mapbox.com/account/ under the "Access tokens" section
~~    mapbox_access_token = 'pk.my_mapbox_access_token'
~~    return render(request, 'default.html', 
~~                  { 'mapbox_access_token': mapbox_access_token })

The Mapbox access token should really be stored in the Django settings file, so we left a "TODO" note to handle that as a future step.

Now we can try our webpage again. Refresh localhost:8000 in your web browser.

Sweet, we've got a live, interactive map! It's kind of weird thought how it is zoomed out to view the entire world. Time to customize the map using a few JavaScript parameters.

Customizing the Map

We can modify the map by changing parameters for the style, zoom level, location and many other attributes.

We'll start by changing the location that the initial map centers in on as well as the zoom level.

Re-open djmaps/maps/templates/default.html and modify the first highlighted lines so it ends with a commas and add the two new highlighted lines shown below.

<!DOCTYPE html>
<html>
  <head>
    <title>Interactive maps for Django web apps</title>
    <script src='https://api.mapbox.com/mapbox-gl-js/v0.44.2/mapbox-gl.js'></script>
    <link href='https://api.mapbox.com/mapbox-gl-js/v0.44.2/mapbox-gl.css' rel='stylesheet' />
  </head>
  <body>
   <h1>Map time!</h1>
   <div id='map' width="100%" style='height:400px'></div>
   <script>
    mapboxgl.accessToken = {{ mapbox_access_token }};
    var map = new mapboxgl.Map({
     container: 'map',
~~     style: 'mapbox://styles/mapbox/streets-v10',
~~     center: [-77.03, 38.91],
~~     zoom: 9
    });
   </script>
  </body>
</html>

The first number, -77.03, for the center array is the longitude and the second number, 38.91, is the latitude. Zoom level 9 is much closer to the city than the default which was the entire world at level 0. All of the customization values are listed in the Mapbox GL JS API documentation.

Now refresh the page at localhost:8000 to reload our map.

Awesome, now we are zoomed in on Washington, D.C. and can still move around to see more of the map. Let's make a couple other changes to our map before wrapping up.

Again back in djmaps/maps/templates/default.html change the highlighted line for the style key to the mapbox://styles/mapbox/satellite-streets-v10 value. That will change the look from an abstract map style to satellite image data. Update zoom: 9 so that it has a comma at the end of the line and add bearing: 180 as the last key-value pair in the configuration.

<!DOCTYPE html>
<html>
  <head>
    <title>Interactive maps for Django web apps</title>
    <script src='https://api.mapbox.com/mapbox-gl-js/v0.44.2/mapbox-gl.js'></script>
    <link href='https://api.mapbox.com/mapbox-gl-js/v0.44.2/mapbox-gl.css' rel='stylesheet' />
  </head>
  <body>
   <h1>Map time!</h1>
   <div id='map' width="100%" style='height:400px'></div>
   <script>
    mapboxgl.accessToken = {{ mapbox_access_token }};
    var map = new mapboxgl.Map({
     container: 'map',
~~     style: 'mapbox://styles/mapbox/satellite-streets-v10',
~~     center: [-77.03, 38.91],
~~     zoom: 9,
~~     bearing: 180
    });
   </script>
  </body>
</html>

Save the template and refresh localhost:8000.

The map now provides a satellite view with streets overlay but it is also... "upside down"! At least the map is upside down compared to how most maps are drawn, due to the bearing: 180 value, which modified this map's rotation.

Not bad for a few lines of JavaScript in our Django application. Remember to check the Mapbox GL JS API documentation for the exhaustive list of parameters that you can adjust.

What's Next?

We just learned how to add interactive JavaScript-based maps to our Django web applications, as well as modify the look and feel of the maps. Next try out some of the other APIs Mapbox provides including:

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

Do you see a typo, syntax issue or wording that's confusing in this blog post? Fork this page's source on GitHub and submit a pull request with a fix or file an issue ticket on GitHub.

Full Stack Python at PyCon US 2018

Matt Makai — Wed, 09 May 2018 00:00:00 -0400

PyCon US 2018 kicked off today with the first day of tutorials. I am flying in tomorrow and will be there through the end of the weekend. If you're around, come by either the Twilio booth or the community booth where the gang from Talk Python to Me, Real Python, PyBites and Test & Code will be hanging out. I will be at one of those two spots when I am not watching talks! I'd love your feedback on what I can improve on Full Stack Python. It's also great hearing your stories about how the site has helped you improve your development skills.

For those folks who can't make it to PyCon, I'll be tweeting the best stuff that I see throughout the conference via @fullstackpython. Likewise, if I miss something let me know on Twitter or via email so we can highlight it.

One quick update on the Full Stack Python Guide to Deployments book. I have a great update in the works that bumps to the latest versions of Ubuntu (now 18.04 LTS), Ansible 2.5.1 and Flask 1.0.2. It has been a long time coming and will be a free update to all existing purchasers. If you have not bought the book yet, I recommend waiting until the update is out because the existing book's software versions are getting way too out of date to be useful to most projects.

Monitoring Python 3.6 Functions on AWS Lambda

Matt Makai — Wed, 25 Apr 2018 00:00:00 -0400

Amazon Web Services (AWS) Lambda is a usage-based execution environment that can run Python 3.6 code. If you have never previously used AWS Lambda then you can read
How to Create Your First Python 3.6 AWS Lambda Function. However, this tutorial will give you every step to follow even if you are completely new to AWS.

In this post we are going to monitor Python code that is running on AWS Lambda by using a hosted monitoring service, Rollbar.

Required Tools and Code

A local development environment is not required to follow this tutorial. All the work will happen in a web browser through the AWS Console.

The example code can be copy and pasted from this blog post or you can access it on GitHub under the Full Stack Python blog-post-examples repository within the monitor-aws-lambda-python-3-6 directory.

Accessing the AWS Lambda Service

Sign into your existing AWS account or sign up for a new account. AWS Lambda comes with a free tier so you can test code and execute basic applications without cost.

AWS has a boatload of services so use the search box to enter "lambda" and select "Lambda" when it appears to get to the appropriate starting page.

Click the "Create function" button.

Select "Author from Scratch". Fill in a name so you can easily recognize this function for future reference. I chose "monitorPython3". Select "Python 3.6" for Runtime.

Select "Create new role from template(s)", input a Role name, for example "basicEdgeLambdaRole". For Policy templates choose "Basic Edge Lambda Permissions".

Then click "Create function."

Ok, finally we have arrived at the configuration screen where we can write our code.

Coding a Python Function

Scroll down to the "Function code" user interface section.

Paste or type in the following code, replacing what is already in the text box.

import os
import rollbar


ROLLBAR_KEY = os.getenv('ROLLBAR_SECRET_KEY', 'missing Rollbar secret key')
rollbar.init(ROLLBAR_KEY, 'production')


@rollbar.lambda_function
def lambda_handler(event, context):
    message = os.getenv("message")
    print_count = int(os.getenv("print_count"))

    # check if message exists and how many times to print it
    if message and print_count > 0:
        for i in range(0, print_count):
            # formatted string literals are new in Python 3.6
            print(f"message: {message}.")
        return print_count
    return None

The code contains the required lambda_handler function. lambda_handler is Lambda's hook for where to start execution the code.

The Python code expects two environment variables that are read by the os module with the getenv function. The message and print_count variables are set by the environment variables.

Below the code input text box on this function configuration screen there is a section to set environment variable key-value pairs. We need to input two environment variables and then we can run our code.

Enter the keys named message with a value of Hello World!. Then enter print_count as a second key with the value of 5.

Our Python code's error handling is not robust. A value other than a number in the print_count variable will cause the script to throw an exception when it is executed due to the forced casting of print_count via the int() function. We will use the exception that can occur during this forced casting as a trivial example that shows what happens when errors in our code happen during Lambda function execution.

Hit the "Save" button at the top right. Use the default "Hello World" test template values and name it "testHelloWorld". We do not need any of those values for our function.

Click "Create" and your test template will be created. Now click "Test" to run the function. You should see "Execution result: succeeded" with the message variable printed five times.

Now change the value of print_count to i dunno. Save the function and click "Test" again. The function will fail.

Monitoring our Lambda Function

Head over to the Rollbar homepage to obtain a free account and grab the necessary information to add their hosted monitoring service into our Lambda application.

Click "Sign Up" in the upper right-hand corner. Enter your email address, username and desired password.

After the sign up page you will see the onboarding flow where you can enter a project name and select a programming language. For the project name type in "Full Stack Python" and then select that you are monitoring a Python-based application.

Press "Continue" at the bottom of the screen. The next page shows us a few instructions on how to add monitoring.

Take note of that server-side access token as we will need to set it as an environment variable on AWS Lambda.

We can now update our Python function to collect and aggregate the errors that occur in our application. Add the following highlighted lines to your Lambda code:

import os
~~import rollbar
~~
~~
~~ROLLBAR_KEY = os.getenv('ROLLBAR_SECRET_KEY', 'missing Rollbar secret key')
~~rollbar.init(ROLLBAR_KEY, 'production')


~~@rollbar.lambda_function
def lambda_handler(event, context):
    message = os.getenv("message")
    print_count = int(os.getenv("print_count"))

    # check if message exists and how many times to print it
    if message and print_count > 0:
        for i in range(0, print_count):
            # formatted string literals are new in Python 3.6
            print(f"message: {message}.")
        return print_count
    return None

The above highlighted new code lines incorporate the rollbar library into our application, set the ROLLBAR_KEY with our environment variable and use the rollbar.lambda_function decorator to catch all errors in our lambda_handler function.

Add the following third environment variable named ROLLBAR_SECRET_KEY that is the server-side token from your new Rollbar project.

There is just one issue with this function on Lambda as it stands: there is no way for Lambda to know about the Rollbar package code. The external Rollbar dependency needs to be included. There are a couple of ways to handle the issue:

Download this pre-made zip file from the GitHub repository which includes all of the Rollbar package code and our code in the lambda_function.py file.
Re-create the above code on your local system and use pip to obtain the dependencies and create a zip file locally.

I provided the pre-made zip file to save time in this tutorial so try that one now so we can see the final results. Under "Function code", change the "Code entry type" from "Edit code inline" to "Upload a .ZIP file". Hit the "Upload" button under "Function package".

Hit the "Save" button at the top. With our new code we can now see if Rollbar will capture and report the exceptions. Hit the "Save" button and then "Test".

The function will fail as expected. If we move over to our Rollbar dashboard and refresh the page, we see the exceptions.

Now we can track Lambda exceptions across many functions regardless of how frequently they are running.

What's Next?

We just wrote and executed a Python 3.6 function on AWS Lambda then captured the exception message into our Rollbar logs. Now you can continue building out your Python code knowing that when something goes wrong you will have full visibility on what happened.

Check out the AWS Lambda section for more tutorials by other developers.

Further questions? Contact me on Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Developing Flask Apps in Docker Containers on macOS

Matt Makai — Tue, 05 Jun 2018 00:00:00 -0400

Adding Docker to your Python and Flask development environment can be confusing when you are just getting started with containers. Let's quickly get Docker installed and configured for developing Flask web applications on your local system.

Our Tools

This tutorial is written for Python 3. It will work with Python 2 but I have not tested it with the soon-to-be deprecated 2.7 version.

Docker for Mac is necessary. I recommend the stable release unless you have an explicit purpose for the edge channel.

Within the Docker container we will use:

Python 3, specifically the slim-3.6.5 version from Docker Hub
Flask version 1.0.2

All of the code for the Dockerfile and the Flask app are available open source under the MIT license on GitHub under the docker-flask-mac directory of the blog-code-examples repository. Use the code for your own purposes as much as you like.

Installing Docker on macOS

We need to install Docker before we can spin up our Docker containers. If you already have Docker for Mac installed and working, feel free to jump to the next section.

On your Mac, download the Docker Community Edition (CE) for Mac installer.

Find the newly-downloaded install within Finder and double click on the file. Follow the installation process, which includes granting administrative privileges to the installer.

Open Terminal when the installer is done. Test your Docker installation with the --version flag:

docker --version

If Docker is installed correctly you should see the following output:

Docker version 18.03.1-ce, build 9ee9f40

Note that Docker runs through a system agent you can find in the menu bar.

I have found the Docker agent to take up some precious battery life on my Macbook Pro. If I am not developing and need to max battery time I will close down the agent and start it back up again when I am ready to code.

Now that Docker is installed let's get to running a container and writing our Flask application.

Dockerfile

Docker needs to know what we want in a container, which is where the Dockerfile comes in.

# this is an official Python runtime, used as the parent image
FROM python:3.6.5-slim

# set the working directory in the container to /app
WORKDIR /app

# add the current directory to the container as /app
ADD . /app

# execute everyone's favorite pip command, pip install -r
RUN pip install --trusted-host pypi.python.org -r requirements.txt

# unblock port 80 for the Flask app to run on
EXPOSE 80

# execute the Flask app
CMD ["python", "app.py"]

Save the Dockerfile so that we can run our next command with the completed contents of the file. On the commandline run:

docker build -t flaskdock .

The above docker build file uses the -t flag to tag the image with the name of flaskdock.

If the build worked successfully we can see the image in with the docker image ls command. Give that a try now:

docker image ls

We should then see our tag name in the images list:

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
flaskdock           latest              24045e0464af        2 minutes ago       165MB

Our image is ready to load up as a container so we can write a quick Flask app that we will use to test our environment by running it within the container.

Coding A Simple Flask app

Time to put together a super simple "Hello, World!" Flask web app to test running Python code within our Docker container. Within the current project directory, create a file named app.py with the following contents:

from flask import Flask, Response


app = Flask(__name__)


@app.route("/")
def hello():
    return Response("Hi from your Flask app running in your Docker container!")


if __name__ == "__main__":
    app.run("0.0.0.0", port=80, debug=True)

The above 7 lines of code (not counting blank PEP8-compliant lines) in app.py allow our application to return a simple message when run with the Flask development server.

We need just one more file to specify our Flask dependency. Create a requirements.txt file within the same directory as app.py:

flask==1.0.2

Make sure both the app.py and requirements.txt file are saved then we can give the code a try.

Running the Container

docker run -p 5000:80 --volume=/Users/matt/devel/py/flaskdocker:/app flaskdock

If you receive the error python: can't open file 'app.py': [Errno 2] No such file or directory then you likely forgot to chance /Users/matt/devel/py/flaskdocker to the directory where your project files, especially app.py, are located.

Everything worked when you see a simple text-based HTTP response like what is shown above in the screenshot of my Chrome browser.

What's Next?

We just installed Docker and configured a Flask application to run inside a container. That is just the beginning of how you can integrate Docker into your workflow. I strongly recommend reading the Django with PostgreSQL quickstart that will introduce you to Docker Swarm as well as the core Docker container service.

Next up take a look at the Docker and deployment pages for more related tutorials.

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

Do you see a typo, syntax issue or just something that's confusing in this blog post? Fork this page's source on GitHub and submit a pull request with a fix or file an issue ticket on GitHub.

ReportLab and Future Community Project Launches

Matt Makai — Sun, 04 Mar 2018 00:00:00 -0500

Congratulations to fellow Python developer Mike Driscoll for his successful ReportLab: PDF Processing with Python Kickstarter that just concluded with over double his funding goal.

I was excited to back Mike's project for a couple of reasons. First, I've used ReportLab on past projects and it is a handy library for working with PDFs. Second, it is super useful to have entire books written on niche Python code libraries such as ReportLab.

Full Stack Python will gladly back and spread the word about other awesome, legitimate Python community projects. Let me know via email (matthew.makai@gmail.com or mattmakai@fullstackguides.com) when you are getting ready to launch a Python project so I can help give a boost.

Michael Kennedy and I know from our own Kickstarter experience how much work goes into making these ideas come to fruition. It's a big confidence boost to have a community tailwind at your back and I am always happy to be part of that tailwind.

Monitoring Django Projects with Rollbar

Matt Makai — Sun, 20 May 2018 00:00:00 -0400

One fast way to scan for exceptions and errors in your Django web application projects is to add a few lines of code to include a hosted monitoring tool.

In this tutorial we will learn to add the Rollbar monitoring service to a web app to visualize any issues produced by our web app. This tutorial will use Django as the web framework to build the web application but there are also tutorials for the Flask and Bottle frameworks as well. You can also check out a list of other hosted and open source tools on the monitoring page.

Our Tools

Python 3 is strongly recommended for this tutorial because Python 2 will no longer be supported starting January 1, 2020. Python 3.6.4 to was used to build this tutorial. We will also use the following application dependencies to build our application:

Django web framework, version 2.0.4
rollbar monitoring instrumentation library, version 0.13.18, to report exceptions and errors
pip and virtualenv, which come installed with Python 3, to install and isolate these Django and Rollbar libraries from your other applications
A free Rollbar account where we will send error data and view it when it is captured

If you need help getting your development environment configured before running this code, take a look at this guide for setting up Python 3 and Django on Ubuntu 16.04 LTS.

All code in this blog post is available open source on GitHub under the MIT license within the monitor-python-django-apps directory of the blog-code-examples repository. Use and modify the code however you like for your own applications.

Installing Dependencies

Start the project by creating a new virtual environment using the following command. I recommend keeping a separate directory such as ~/venvs/ so that you always know where all your virtualenvs are located.

python3 -m venv monitordjango

Activate the virtualenv with the activate shell script:

source monitordjango/bin/activate

The command prompt will change after activating the virtualenv:

Remember that you need to activate your virtualenv in every new terminal window where you want to use the virtualenv to run the project.

We can now install the Django and Rollbar packages into the activated, empty virtualenv.

pip install django==2.0.4 rollbar==0.13.18

Look for output like the following to confirm the dependencies installed correctly.

Collecting certifi>=2017.4.17 (from requests>=0.12.1->rollbar==0.13.18)
  Downloading certifi-2018.1.18-py2.py3-none-any.whl (151kB)
    100% |████████████████████████████████| 153kB 767kB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests>=0.12.1->rollbar==0.13.18)
  Using cached urllib3-1.22-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=0.12.1->rollbar==0.13.18)
  Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests>=0.12.1->rollbar==0.13.18)
  Using cached idna-2.6-py2.py3-none-any.whl
Installing collected packages: pytz, django, certifi, urllib3, chardet, idna, requests, six, rollbar
  Running setup.py install for rollbar ... done
Successfully installed certifi-2018.1.18 chardet-3.0.4 django-2.0.4 idna-2.6 pytz-2018.3 requests-2.18.4 rollbar-0.13.18 six-1.11.0 urllib3-1.22

We have our dependencies ready to go so now we can write the code for our Django project.

Our Django Web App

Django makes it easy to generate the boilerplate code for new projects and apps using the django-admin.py commands. Go to the directory where you typically store your coding projects. For example, on my Mac I use /Users/matt/devel/py/. Then run the following command to start a Django project named djmonitor:

django-admin.py startproject djmonitor

The command will create a directory named djmonitor with several subdirectories that you should be familiar with when you've previously worked with Django.

Change directories into the new project.

cd djmonitor

Start a new Django app for our example code.

python manage.py startapp billions

Django will create a new folder named billions for our project. Let's make sure our Django URLS work properly before before we write the code for the app.

Now open djmonitor/djmonitor/urls.py and add the highlighted lines so that URLs with the path /billions/ will be routed to the app we are working on.

""" (comments section)
"""
~~from django.conf.urls import include
from django.contrib import admin
from django.urls import path

urlpatterns = [
~~    path('billions/', include('billions.urls')),
    path('admin/', admin.site.urls),
]

Save djmonitor/djmonitor/urls.py and open djmonitor/djmonitor/settings.py. Add the billions app to settings.py by inserting the highlighted line, which will become line number 40 after insertion:

# Application definition

INSTALLED_APPS = [ 
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
~~    'billions',
]

Save and close settings.py.

Reminder: make sure you change the default DEBUG and SECRET_KEY values in settings.py before you deploy any code to production. Secure your app properly with the information from the Django production deployment checklist so that you do not add your project to the list of hacked applications on the web.

Next change into the djmonitor/billions directory. Create a new file named urls.py that will be specific to the routes for the billions app within the djmonitor project.

Add the following lines to the currently-blank djmonitor/billions/urls.py file.

from django.conf.urls import url                                                                                                                              
from . import views

urlpatterns = [ 
    url(r'(?P<slug>[\wa-z-]+)', views.they, name="they"),
]

Save djmonitor/billions/urls.py. One more file before we can test that our simple Django app works. Open djmonitor/billions/views.py.

from django.core.exceptions import PermissionDenied
from django.shortcuts import render


def they(request, slug):
    if slug and slug == "are":
        return render(request, 'billions.html', {})
    else:
        raise PermissionDenied("Hmm, can't find what you're looking for.")

Create a directory for your template files named templates under the djmonitor/billions app directory.

mkdir templates

Within templates create a new file named billions.html that contains the following Django template markup.

<!DOCTYPE html>
<html>
  <head>
    <title>They... are BILLIONS!</title>
  </head>
  <body>
    <h1><a href="http://store.steampowered.com/app/644930/They_Are_Billions/">They Are Billions</a></h1>
    <img src="https://media.giphy.com/media/2jUHXTGhGo156/giphy.gif">
  </body>
</html>

Alright, all of our files are in place so we can test the application. Within the base directory of your project run the Django development server:

python manage.py runserver

The Django development server will start up with no issues other than an unapplied migrations warning.

(monitordjango) $ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).

You have 14 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python manage.py migrate' to apply them.

April 08, 2018 - 19:06:44
Django version 2.0.4, using settings 'djmonitor.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Only the /billions/ route will successfully hit our billions app. Try to access "http://localhost:8000/billions/are/". We should see our template render with the gif:

Cool, our application successfully rendered a super-simple HTML page with a GIF of one of my favorite computer games. What if we try another path under /billions/ such as "http://localhost:8000/billions/arenot/"?

Our 403 Forbidden is raised, which is what we expected based on our code. That is a somewhat contrived block of code but let's see how we can catch and report this type of error without changing our views.py code at all. This approach will be much easier on us when modifying an existing application than having to refactor the code to report on these types of errors, if we even know where they exist.

Monitoring with Rollbar

Go to the Rollbar homepage in your browser to add their tool to our Django app.

Click the "Sign Up" button in the upper right-hand corner. Enter your email address, a username and the password you want on the sign up page.

After the sign up page you will see the onboarding flow where you can enter a project name and select a programming language. For the project name type in "Full Stack Python" (or whatever project name you are working on) then select that you are monitoring a Python-based application.

Press the "Continue" button at the bottom to move along. The next screen shows us a few instructions on how to add monitoring.

Let's change our Django project code to let Rollbar collect and aggregate the errors that pop up in our application.

Re-open djmonitor/djmonitor/settings.py and look for the MIDDLEWARE list. Add rollbar.contrib.django.middleware.RollbarNotifierMiddleware as the last item:

MIDDLEWARE = [ 
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
~~    'rollbar.contrib.django.middleware.RollbarNotifierMiddleware',
]

Do not close settings.py just yet. Next add the following lines to the bottom of the file. Change the access_token value to your Rollbar server side access token and root to the directory where you are developing your project.

ROLLBAR = {
    'access_token': 'access token from dashboard',
    'environment': 'development' if DEBUG else 'production',
    'branch': 'master',
    'root': '/Users/matt/devel/py/blog-code-examples/monitor-django-apps/djmonitor',
    'patch_debugview': False,
}

If you are uncertain about what your secret token is, it can be found on the Rollbar onboarding screen or "Settings" -> "Access Tokens" within rollbar.com.

Note that I typically store all my environment variables in a .env

We can test that Rollbar is working as we run our application. Run it now using the development server.

python manage.py runserver

Back in your web browser press the "Done! Go to Dashboard" button.

If an event hasn't been reported yet we'll see a waiting screen like this one:

Make sure your Django development still server is running and try to go to "http://localhost:8000/billions/arenot/". A 403 error is immediately reported on the dashboard:

We even get an email with the error (which can also be turned off if you don't want emails for every error):

Alright we now have monitoring and error reporting all configured for our Django application!

What now?

We learned to catch issues in our Django project using Rollbar and view the errors in Rollbar's interface. Next try out Rollbar's more advanced monitoring features such as:

There is plenty more to learn about in the areas of web development and deployments so keep learning by reading about web frameworks. You can also learn more about integrating Rollbar with Python applications via their Python documentation.

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

Do you see a typo, syntax issue or wording that's confusing in this blog post? Fork this page's source on GitHub and submit a pull request with a fix or file an issue ticket on GitHub.

5 Years of Full Stack Python

Matt Makai — Sat, 23 Dec 2017 00:00:00 -0500

Full Stack Python began five years ago today, on December 23, 2012, with Git commit 69f5f46.

I originally built the site to help out a group of junior developers that kept asking me similar Python web development questions via email. It seemed like the answers would be useful to more people if I put them in a publicly-accessible location. One day over lunch with a friend before I started writing I sketched out some of my vague ideas on a napkin:

The site started as a single page static website but eventually was split into topic-specific pages such as:

Most pages were on deployment and web framework topics. I have made a concerted effort to write more about data and development environment subjects as I continue to learn and grow my own software development skills. In some ways Full Stack Python's evolution represents my own growth as a programmer.

The site now has over 120,000 words and 150+ pages, split between topics pages and tutorial blog posts. I've also given a few technical talks on many of these topics, such as Full Stack Python at EuroPython 2014 and WebSockets in Python at the San Francisco Python meetup. With so much content on the site, it's time to revamp many of the original pages to ensure they are still accurate and contain solid resources that explain those subjects. It can be sad to see so many awesome blog posts I used to reference that have succumbed to link rot. Maintenance takes up an increasing amount of time spent working on the site so please submit issue tickets whenever you see a 404 or a link that's not the original correct resource.

Full Stack Python has now been read by over 2.5 million developers, but it took a long time to get to that milestone. In fact there were only a few hundred readers within the first year. Over time with daily updates I have been fortunate to grow the readership to around 125,000 developers per month.

Watching the numbers go up has been fun but the best part is receiving "thank you" emails and tweets, as well as talking to readers in person at PyCon. Keep those emails coming as they keep me motivated to continue writing! If you'll be at PyCon in April, I'll definitely be there at the Twilio booth or around the community booths where Michael Kennedy of Talk Python to Me and other Python community folks such as Dan Bader, Adrian Rosebrock of PyImageSearch, Bob Belderbos of PyBites and the Real Python guys will also be hanging out.

It's been a real pleasure working on Full Stack Python over the past five years and I'm really excited for what's coming for the site in the next five years. The change log page contains a complete list of major modifications and future directions has some insight into my thought process for creating additional content.

GitPython and New Git Tutorials

Matt Makai — Mon, 11 Dec 2017 00:00:00 -0500

First Steps with GitPython is a quick tutorial that shows how to get started using the awesome GitPython library for programmatically interacting with Git repositories in your Python applications. In the spirit of the thank you maintainers issue ticket I wrote about last newsletter, I opened a quick "non-issue" ticket for the GitPython developers to thank them. Give them a thank you +1 if you've used the project and also found it useful.

The Git page on Full Stack Python has also just been updated with new resources. A few of my favorite new tutorials list on the Git page are:

I also split out the Git page resources into beginner, more advanced, specific use case and workflow sections so it's easier to parse based on whether you're a Git veteran or still up-and-coming in that area of your development skills.

First Steps with GitPython

Matt Makai — Thu, 30 Nov 2017 00:00:00 -0500

GitPython is a Python code library for programmatically reading from and writing to Git source control repositories.

Let's learn how to use GitPython by quickly installing it and reading from a local cloned Git repository.

Our Tools

This tutorial should work with either Python 2.7 or 3, but Python 3, especially 3.6+, is strongly recommended for all new applications. I used Python 3.6.3 to write this post. In addition to Python, throughout this tutorial we will also use the following application dependencies:

Git, a source (version) control implementation, version 2.15.1
GitPython version 2.1.7
pip and virtualenv, which come packaged with Python 3, to install and isolate the GitPython library from any of your other Python projects

Take a look at this guide for setting up Python 3 and Flask on Ubuntu 16.04 LTS if you need specific instructions to get a base Python development environment set up.

All code in this blog post is available open source under the MIT license on GitHub under the first-steps-gitpython directory of the blog-code-examples repository. Use and abuse the source code as you like for your own applications.

Install GitPython

Start by creating a new virtual environment for your project. My virtualenv is named testgit but you can name yours whatever matches the project you are creating.

python3 -m venv gitpy

Activate the newly-created virtualenv.

source gitpy/bin/activate

The virtualenv's name will be prepended to the command prompt after activation.

Now that the virutalenv is activated we can use the pip command to install GitPython.

pip install gitpython==2.1.7

Run the pip command and after everything is installed you should see output similar to the following "Successfully installed" message.

(gitpy) $ pip install gitpython==2.1.7
Collecting gitpython==2.1.7
  Downloading GitPython-2.1.7-py2.py3-none-any.whl (446kB)
    100% |████████████████████████████████| 450kB 651kB/s 
Collecting gitdb2>=2.0.0 (from gitpython==2.1.7)
  Downloading gitdb2-2.0.3-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 947kB/s 
Collecting smmap2>=2.0.0 (from gitdb2>=2.0.0->gitpython==2.1.7)
  Downloading smmap2-2.0.3-py2.py3-none-any.whl
Installing collected packages: smmap2, gitdb2, gitpython
Successfully installed gitdb2-2.0.3 gitpython-2.1.7 smmap2-2.0.3

Next we can start programmatically interacting with Git repositories in our Python applications with the GitPython installed.

Clone Repository

GitPython can work with remote repositories but for simplicity in this tutorial we'll use a cloned repository on our local system.

Clone a repository you want to work with to your local system. If you don't have a specific one in mind use the open source Full Stack Python Git repository that is hosted on GitHub.

git clone git@github.com:mattmakai/fullstackpython.com fsp

Take note of the location where you cloned the repository because we need the path to tell GitPython what repository to handle. Change into the directory for the new Git repository with cd then run the pwd (present working directory) command to get the full path.

cd fsp
pwd

You will see some output like /Users/matt/devel/py/fsp. This path is your absolute path to the base of the Git repository.

Use the export command to set an environment variable for the absolute path to the Git repository.

export GIT_REPO_PATH='/Users/matt/devel/py/fsp' # make sure this your own path

Our Git repository and path environment variable are all set so let's write the Python code that uses GitPython.

Read Repository and Commit Data

Create a new Python file named read_repo.py and open it so we can start to code up a simple script.

Start with a couple of imports and a constant:

import os
from git import Repo


COMMITS_TO_PRINT = 5

The os module makes it easy to read environment variables, such as our GIT_REPO_PATH variable we set earlier. from git import Repo gives our application access to the GitPython library when we create the Repo object. COMMITS_TO_PRINT is a constant that limits the number of lines of output based on the amount of commits we want our script to print information on. Full Stack Python has over 2,250 commits so there'd be a whole lot of output if we printed every commit.

Next within our read_repo.py file create a function to print individual commit information:

def print_commit(commit):
    print('----')
    print(str(commit.hexsha))
    print("\"{}\" by {} ({})".format(commit.summary,
                                     commit.author.name,
                                     commit.author.email))
    print(str(commit.authored_datetime))
    print(str("count: {} and size: {}".format(commit.count(),
                                              commit.size)))

The print_commit function takes in a GitPython commit object and prints the 40-character SHA-1 hash for the commit followed by:

the commit summary
author name
author email
commit date and time
count and update size

Below the print_commit function, create another function named print_repository to print details of the Repo object:

def print_repository(repo):
    print('Repo description: {}'.format(repo.description))
    print('Repo active branch is {}'.format(repo.active_branch))
    for remote in repo.remotes:
        print('Remote named "{}" with URL "{}"'.format(remote, remote.url))
    print('Last commit for repo is {}.'.format(str(repo.head.commit.hexsha)))

print_repository is similar to print_commit but instead prints the repository description, active branch, all remote Git URLs configured for this repository and the latest commit.

Finally, we need a "main" function for when we invoke the script from the terminal using the python command. Round out our

if __name__ == "__main__":
    repo_path = os.getenv('GIT_REPO_PATH')
    # Repo object used to programmatically interact with Git repositories
    repo = Repo(repo_path)
    # check that the repository loaded correctly
    if not repo.bare:
        print('Repo at {} successfully loaded.'.format(repo_path))
        print_repository(repo)
        # create list of commits then print some of them to stdout
        commits = list(repo.iter_commits('master'))[:COMMITS_TO_PRINT]
        for commit in commits:
            print_commit(commit)
            pass
    else:
        print('Could not load repository at {} :('.format(repo_path))

The main function handles grabbing the GIT_REPO_PATH environment variable and creates a Repo object based on the path if possible.

If the repository is not empty, which indicates a failure to find the repository, then the print_repository and print_commit functions are called to show the repository data.

If you want to copy and paste all of the code found above at once, take a look at the read_repo.py file on GitHub.

Time to test our GitPython-using script. Invoke the read_repo.py file using the following command.

(gitpy) $ python read_repo.py

If the virtualenv is activated and the GIT_REPO_PATH environment variable is set properly, we should see output similar to the following.

Repo at ~/devel/py/fsp/ successfully loaded.
Repo description: Unnamed repository; edit this file 'description' to name the repository.
Repo active branch is master
Remote named "origin" with URL "git@github.com:mattmakai/fullstackpython.com"
Last commit for repo is 1fa2de70aeb2ea64315f69991ccada51afac1ced.
----
1fa2de70aeb2ea64315f69991ccada51afac1ced
"update latest blog post with code" by Matt Makai (matthew.makai@gmail.com)
2017-11-30 17:15:14-05:00
count: 2256 and size: 254
----
1b026e4268d3ee1bd55f1979e9c397ca99bb5864
"new blog post, just needs completed code section" by Matt Makai (matthew.makai@gmail.com)
2017-11-30 09:00:06-05:00
count: 2255 and size: 269
----
2136d845de6f332505c3df38efcfd4c7d84a45e2
"change previous email newsletters list style" by Matt Makai (matthew.makai@gmail.com)
2017-11-20 11:44:13-05:00
count: 2254 and size: 265
----
9df077a50027d9314edba7e4cbff6bb05c433257
"ensure picture sizes are reasonable" by Matt Makai (matthew.makai@gmail.com)
2017-11-14 13:29:39-05:00
count: 2253 and size: 256
----
3f6458c80b15f58a6e6c85a46d06ade72242c572
"add databases logos to relational databases pagem" by Matt Makai (matthew.makai@gmail.com)
2017-11-14 13:28:02-05:00
count: 2252 and size: 270

The specific commits you see will vary based on the last 5 commits I've pushed to the GitHub repository, but if you see something like the output above that is a good sign everything worked as expected.

What's next?

We just cloned a Git repository and used the GitPython library to read a slew of data about the repository and all of its commits.

GitPython can do more than just read data though - it can also create and write to Git repositories! Take a look at the modifying references documentation page in the official GitPython tutorial or check back here in the future when I get a chance to write up a more advanced GitPython walkthrough.

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

See something wrong in this blog post? Fork this page's source on GitHub and submit a pull request.

DevOps, Thank You Maintainers and Contributing to Open Source

Matt Makai — Mon, 13 Nov 2017 00:00:00 -0500

DevOps, Continuous Delivery... and You is a blog post with the slides and notes based on a class I taught at the University of Virginia this past week. The talk is relevant as a brief introduction to DevOps and Continuous Delivery, especially for junior developers and less-technical managers of software teams. I'm experimenting with the "talk as blog post" style so let me know via email or a tweet if you enjoy it and would want to see future technical talks in that format.

Speaking of feedback on projects, this GitHub issue thread named "thank you" is incredible to read. The issue ticket blew up on the front page of Hacker News as an example of how powerful genuine positive comments can be for project maintainers.

Contributing to open source is a recent Talk Python to Me podcast episode in the same vein as thanking your maintainer. Working on open source projects with your own contributions to documentation or simple bug fixes can be a great way to become a better programmer. I particularly enjoyed the recommendations of the panel to cut your teeth on smaller open source projects rather than trying to jump into a massive codebase like Django or the CPython implementation. Take a listen to that podcast episode if you are new to open source or have been wondering how to get involved.

As always, send me an email or submit an issue ticket on GitHub to let me know how to improve Full Stack Python as I continue to fill in the table of contents with new pages and new tutorials.

DevOps, Continuous Delivery... and You

Matt Makai — Sun, 05 Nov 2017 00:00:00 -0400

This blog post contains the slides along with a loose transcript and additional resources from my technical talk on DevOps and Continuous Delivery concepts given at my alma mater, the University of Virginia, to the M.S. in Management of Information Technology program on November 2nd and 4th of 2017.

Links to learn more about the concepts presented in this talk can be found in the sidebar and at the bottom of this page.

Hey folks, my name is Matt Makai. I am a software developer at Twilio and the creator of Full Stack Python, which over 125,000 developers read each month to learn how to build, deploy and operate Python-based applications.

You've talked about using the Agile software development methodology on your teams, but what's the purpose? Why does Agile development matter to you and your organization?

Agile matters because it allows you to ship more code, faster than traditional "waterfall" methodology approaches.

Shipping is a common allegory in software development nowadays, because code that is not in production, in the hands of your users, doesn't create value for anyone.

If code is not running in production, it's not creating value. New code created by your Agile development teams every couple of weeks does not create more value until it is executing in production.

Shipping code is so important to high functioning companies that the maritime theme is used across all sorts of projects, including in the Docker logo.

As well as in the Kubernetes logo in the form of a ship steering wheel.

Here is a super high-level diagram of the ideal scenario we need for Agile development teams. Create working code and get it shipped as soon as possible into production.

Facebook's internal motto used to be "Move fast and break things." They thought that if you aren't breaking things then you aren't moving fast enough.

And eventually if you're constantly shipping to production and you do not have the appropriate processes and tools in place, your applications will break. The breakage has nothing to do with the Agile methodology itself.

Your team and organization will come to a fork in the road when you end up with a broken environment.

Traditionally, organizations have tried to prevent breakage by putting more manual tools and processes in place. Manual labor slows... down... your... ability... to... execute.

This is one path provided by the fork in the road. Put your "Enterprise Change Review Boards" in place. Require production sign-offs by some Executive Vice President who has never written a line of code in his life. Put several dozen "technical architects" in a room together to argue over who gets to deploy their changes to production that month.

The manual path is insanity. Eventually the best developers in your organization will get frustrated and leave. Executives will ask why nothing ever gets done. Why does it take our organization three years to ship a small change to a critical application?

Some development teams try to get around the manual production challenges by shipping everything to a development environment. The dev environment is under their control.

But what's the huge glaring problem in this situation?

If you are not shipping to production, then you are not creating any value for your users. The teams have made a rational decision to ship to development but the organization still suffers due to the manual controls.

The problems we are talking about are created by the Agile methodology because they become acute when your development team is producing code at high velocity. Once code is created faster, you need a way to reliably, consistently put the code into production so that it can create value for its users.

DevOps and Continuous Delivery are the broad terms that encompass how to reliably ship code to production and operate it when the code is running in production.

We are going to use the terms "DevOps" and "Continuous Delivery" a lot today, so let's start by defining what they mean. In fact, the term "DevOps" has already accumulated a lot of buzzword baggage, so we'll start by defining what DevOps is not.

First,DevOps is not a new role. If you go hire a bunch of people and call them "DevOps engineers" then sit them in the middle of your developers and system admin/ops folks, you are going to have a bad time. You just added a new layer between the two groups you need to pull closer together.

Second, DevOps is not a specific tool or application. You do not need to use Docker or Puppet to do DevOps in your organization. The processes that make DevOps work are made much easier by some tools such as cloud platforms where infrastructure is transient, but even those platforms are not required to do DevOps right.

Third, DevOps is not tied to a specific programming language ecosystem. You do not need to use Node.js or Ruby on Rails. You can still use DevOps in a COBOL- or J2EE-only organization.

With those misconceptions out of the way, let's talk about what DevOps IS. First, at the risk of being way too obvious, DevOps is the combination of the two words Development and Operations. This combination is not a random pairing, it's an intentional term.

Second, DevOps means your application developers handle operations. Not necessarily all operations work, but ops work that deals with the code they write and deploy as part of their sprints. The developers also will likely become intimately familiar with the underlying infrastructure such as the web application servers, web servers and deployment code for configuration management tools.

Third, DevOps allows your organization to be more efficient in handling issues by ensuring the correct person is handling errors and application failures.

We are not going to go through Continuous Delivery (CD) by defining what it is not, but there are a couple bits to say about it. First, CD is a collection of engineering practices aimed at automating the delivery of code from version control check-in until it is running in a production environment.

The benefit of the automation CD approach is that your organization will have far greater confidence in the code running in production even as the code itself changes more frequently with every deployment.

Facebook's original motto changed a few years ago to "Move Fast and Build Things" because they realized that breaking production was not a byproduct of moving fast, it was a result of immature organizational processes and tools. DevOps and Continuous Delivery are why organizations can now deploy hundreds or thousands of times to production every day but have increasing, not decreasing, confidence in their systems as they continue to move faster.

Let's take a look at a couple of example scenarios that drive home what DevOps and CD are all about, as well as learn about some of the processes, concepts and tools that fall in this domain.

Here is a beautiful evening picture of the city I just moved away from, San Francisco.

The company I work for, Twilio is located in San Francisco. If you ever fly into the SFO airport and catch a ride towards downtown, you will see our billboard on the right side of the road.

Twilio makes it easy for software developers to add communications, such as phone calling, messaging and video, into their applications. We are a telecommunications company built with the power of software that eliminates the need for customers to buy all the expensive legacy hardware that they used to have to acquire. As a telecomm company, we can never go down, or our customers are hosed and then our business is hosed.

However, we have had challenges in our history that have forced us to confront the fork in the road between manual processes and moving faster via trust in our automation.

In August 2013, Twilio faced an infrastructure failure.

First, some context. When a developer signs up for Twilio, she puts some credit on their account and the credit is drawn upon by making phone calls, sending messages and such. When credit runs low we can re-charge your card so you get more credit.

There was a major production issue with the recurring charges in August 2013. Our engineers were alerted to the errors and the issue blew up on the top of Hacker News, drawing widespread atttention.

So now there is a major production error... what do we do?

(Reader note: this section is primarily audience discussion based on their own experiences handling these difficult technical situations.)

One step is to figure out when the problem started and whether or not it is over. If it's not over, triage the specific problems and start communicating with customers. Be as accurate and transparent as possible.

The specific technical issue in this case was due to our misconfiguration of Redis instances.

We know the particular technical failure was due to our Redis mishandling, but how do we look past the specific bit and get to a broader understanding of the processes that caused the issue?

Let's take a look at the resolution of the situation and then learn about the concepts and tools that could prevent future problems.

In this case, we communicated with our customers as much about the problem as possible. As a developer-focused company, we were fortunate that by being transparent about the specific technical issue, many of our customers gained respect for us because they had also faced similar misconfigurations in their own environments.

Twilio became more transparent with the status of services, especially with showing partial failures and outages.

Twilio was also deliberate in avoiding the accumulation of manual processes and controls that other organizations often put in place after failures. We doubled down on resiliency through automation to increase our ability to deploy to production.

What are some of the tools and concepts we use at Twilio to prevent future failure scenarios?

If you do not have the right tools and processes in place, eventually you end up with a broken production environment after shipping code. What is one tool we can use to be confident that the code going into production is not broken?

Automated testing, in its many forms, such as unit testing, integration testing, security testing and performance testing, helps to ensure the integrity of the code. You need to automate because manual testing is too slow.

Other important tools that fall into the automated testing bucket but are not traditionally thought of as a "test case" include code coverage and code metrics (such as Cyclomatic Complexity).

Awesome, now you only deploy to production when a big batch of automated test cases ensure the integrity of your code. All good, right?

Err, well no. Stuff can still break in production, espcially in environments where for various reasons you do not have the same exact data in test that you do in production. Your automated tests and code metrics will simply not catch every last scenario that could go wrong in production.

When something goes wrong with your application, you need monitoring to know what the problem is, and alerting to tell the right folks. Traditionally, the "right" people were in operations. But over time many organizations realized the ops folks ended up having to call the original application developers who wrote the code that had the problem.

A critical piece to DevOps is about ensuring the appropriate developers are carrying the pagers. It sucks to carry the pager and get woken up in the middle of the night, but it's a heck of a lot easier to debug the code that your team wrote than if you are a random ops person who's never seen the code before in her life.

Another by-product of having application developers carry the "pagers" for alerts on production issues is that over time the code they write is more defensive. Errors are handled more appropriately, because otherwise you know something will blow up on you later on at a less convenient time.

Typically you find though that there are still plenty of production errors even when you have defensive code in place with a huge swath of the most important parts of your codebase being constantly tested.

That's where a concept known as "chaos engineering" can come in. Chaos engineering breaks parts of your production environment on a schedule and even unscheduled basis. This is a very advanced technique- you are not going to sell this in an environment that has no existing automated test coverage or appropriate controls in place.

By deliberately introducing failures, especially during the day when your well-caffeinated team can address the issues and put further safeguards in place, you make your production environment more resilient.

We talked about the failure in Twilio's payments infrastructure several years ago that led us to ultimately become more resilient to failure by putting appropriate automation in place.

Screwing with other people's money is really bad, and so is messing with people's lives.

Let's discuss a scenario where human lives were at stake.

To be explicit about this next scenario, I'm only going to talk about public information, so my cleared folks in the audience can relax.

During the height of U.S forces' Iraq surge in 2007, more improvised explosive devices were killing and maiming soldiers and civilians than ever before. It was an incredible tragedy that contributed to the uncertainty of the time in the country.

However, efforts in biometrics were one part of the puzzle that helped to prevent more attacks, as shown in this picture from General Petraeus' report to Congress.

One major challenge with the project was a terrible manual build process that literally involved clicking buttons in an integrated development environment to create the application artifacts. The process was too manual and the end result was that the latest version of the software took far too long to get into production.

We did not have automated deployments to a development environment, staging or production.

Our team had to start somewhere, but with a lack of approved tools, all we had available to us was shell scripts. But shell scripts were a start. We were able to make a very brittle but repeatable, automated deployment process to a development environment?

There is still a huge glaring issue though: until the code is actually deployed to production it does not provide any value for the users.

In this case, we could never fully automate the deployment because we had to burn to a CD before moving to a physically different computer network. The team could automate just about everything else though, and that really mattered for iteration and speed to deployment.

You do the best you can with the tools at your disposal.

What are the tools and concepts behind automating deployments?

Source code is stored in a source control (or version control) repository. Source control is the start of the automation process, but what do we need to get the code into various environments using a repeatable, automated process?

This is where continuous integration comes in. Continuous integration takes your code from the version control system, builds it, tests it and calculate the appropriate code metrics before the code is deployed to an environment.

Now we have a continuous integration server hooked up to source control, but this picture still looks odd.

Technically, continuous integration does not handle the details of the build and how to configure individual execution environments.

Configuration management tools handle the setup of application code and environments.

Those two scenarios provided some context for why DevOps and Continuous Delivery matter to organizations in varying industries. When you have high performing teams working via the Agile development methodology, you will encounter a set of problems that are not solvable by doing Agile "better". You need the tools and concepts we talked about today as well as a slew of other engineering practices to get that new code into production.

The tools and concepts we covered today were automated testing, monitoring, chaos engineering, continuous integration and configuration management.

There are many other practices you will need as you continue your journey. You can learn about all of them on Full Stack Python.

That's all for today. My name is Matt Makai and I'm a software developer at Twilio and the author of Full Stack Python. Thank you very much.

Additional resources to learn more about the following topics can be found on their respective pages:

PyDev of the Week, Django 2.0 and Twilio Voices

Matt Makai — Mon, 30 Oct 2017 00:00:00 -0400

PyDev of the Week is a developer interview series by Mike Driscoll that asks Python programmers how they started coding, the projects they're working on and what advice they have for beginners. Mike was kind enough to interview me in the latest PyDev of the Week post.

In the PyDev interview I gave a big shoutout to the fine folks working on the Django project, which is currently beta testing the major upcoming Django 2.0 release. Django 2.0 is the first release to support only Python 3, specifically Python 3.4, 3.5 and 3.6. The Django 2.0 beta 1 release needs feedback on bugs in the issue tracker.

One bit I missed calling out in the PyDev interview is a new program I'm working on called Twilio Voices. Twilio Voices pays developers to write great technical tutorials for the Twilio blog. We have already published a slew of awesome Python walkthroughs such as:

How I Hacked My University's Registration System with Twilio SMS by Samuel Taylor
Getting Started on Geospatial Analysis with Python, GeoJSON and GeoPandas by Lesley Cordero
JSON Serialization in Python using serpy by Siddhant Goel
Wedding at Scale: How I Used Twilio, Python and Google to Automate My Wedding by Thomas Curtis
Never Forget A Friend’s Birthday with Python, Flask and Twilio by Bob Belderbos

Take a look at the Twilio Voices page and submit the interest form if you want to get paid to write code tutorials in any programming language of your choice. We'll take care of promoting your posts to the broader developer community.

As always, send me an email or submit an issue ticket on GitHub to let me know how to improve Full Stack Python as I continue to fill in the table of contents with new pages and new tutorials.

PyCon US 2018 CFP, Python Bytes and Pelican

Matt Makai — Mon, 09 Oct 2017 00:00:00 -0400

PyCon US 2018 is coming up in Cleveland, Ohio on May 9th-17th. The call for proposals (CFP) went live in the past few days so now is the time to sharpen your keyboards and get yourself into the proposal writing zone.

When you start working on your proposal, here are some awesome resources on building a great tech talk, public speaking and describing your session by writing a solid proposal:

Looking forward to seeing you all at PyCon US in Cleveland early next year!

Speaking of folks who will definitely be at PyCon, Python Bytes is an awesome weekly Python podcast by Michael Kennedy of Talk Python to Me and Brian Okken of Test and Code. Michael and Brian teamed up to host and produce Python Bytes. I really enjoy listening to the rapid-fire discussion of several programming topics within a single podcast.

Michael and Brian were kind enough to invite me on as a co-host for the 38th episode "Hacking Classic Nintendo Games with Python" while Michael was on vacation. The following episode "The new PyPI" also had a great discussion of the Object-Relational Mappers (ORMs) page on Full Stack Python.

One of the projects I talked about on the Python Bytes podcast episode that I guest hosted is Pelican, the static site generator that turns Markdown and some Jinja templates into the Full Stack Python site. Here are some additional tutorials and resources to get started using Pelican if you've been meaning to build a static site yourself:

One last bit: Python for Entrepreneurs is now fully released with all 20 hours of content. Got non-developer friends who wants you to build them an app? Send them the Python for Entrepreneurs course so they can stop bugging you and build it themselves :)

How to Monitor Python Web Applications

Matt Makai — Fri, 29 Sep 2017 00:00:00 -0400

A quick way to check for errors and issues in your operational Python web application is to drop-in one of many awesome hosted monitoring tools.

Let's learn to quickly add Rollbar monitoring to a web app to visualize when our application is running properly and when it has issues. This tutorial will use Bottle as the example web framework along with Rollbar as the monitoring service but you can also check out the list of other tools on the monitoring page.

Our Tools

We can use either Python 2 or 3 to build this tutorial, but Python 3 is strongly recommended for all new applications. Python 3.6.2 was used to build this tutorial. We will also use the following application dependencies throughout the post:

pip and virtualenv, which come installed with Python 3, to install and isolate the Bottle and Rollbar libraries from your other projects
Bottle web framework, version 0.12.13
pyrollbar monitoring instrumentation library, version 0.13.13 in Bottle applications so pyrollbar can report on all errors
A free Rollbar account where we will send error data and view it when it is captured

If you need help getting your development environment configured before running this code, take a look at this guide for setting up Python 3 and Bottle on Ubuntu 16.04 LTS.

All code in this blog post is available open source under the MIT license on GitHub under the monitor-python-bottle-apps directory of the blog-code-examples repository. Use and abuse the source code as you desire for your own applications.

Installing Dependencies

Create a new virtual environment for this project using the following command. I recommend keeping a separate directory for virtualenvs under ~/Envs/ so you will know where all your project virtualenvs are located.

python3 -m venv monitorpython

Activate the virtualenv with the activate shell script:

source monitorpython/bin/activate

The command prompt will change after activating the virtualenv:

Remember that you need to activate your virtualenv in every new terminal window where you want to use the virtualenv to run the project.

We can now install Bottle and Rollbar into the activated virtualenv.

pip install bottle==0.12.13 rollbar==0.13.13

Look for output like the following to confirm the dependencies installed correctly.

Installing collected packages: bottle, urllib3, certifi, idna, chardet, requests, six, rollbar
  Running setup.py install for bottle ... done
    Running setup.py install for rollbar ... done
    Successfully installed bottle-0.12.13 certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 requests-2.18.4 rollbar-0.13.13 six-1.11.0 urllib3-1.22

We have our dependencies ready to go so now we can build our Python web application.

Our Python Web App

Create a folder for your project named monitor-python-apps. cd into the folder and then create a file named app.py with the following code.

import bottle
import os
import re
from bottle import route, template


TEMPLATE_STRING = """
<html>
 <head>
  <title>Full Stack Python Web App</title>
 </head>
 <body>
  <h1>{{ h1 }}</h1>
 </body>
</html>
"""

MIN_MSG_LENGTH = 2


@route("/<msg>/")
def show_message(msg):
    """Display a message if the msg value is greater than 2 characters
    in the path.
    """
    valid_length = len(msg) >= MIN_MSG_LENGTH
    valid_name = re.match('^[a-z\-]+$', msg.lower()) is not None
    if valid_length and valid_name:
        return template(TEMPLATE_STRING, h1=msg)
    else:
        error_msg = "Sorry, only alpha characters and hyphens allowed."
        raise Exception(error_msg)


if __name__ == "__main__":
    bottle.run(host='localhost', port=8080)

The above application code has a few standard Bottle imports so we can create a Bottle web app and handle URL routes.

We have a single function, show_message, that handles a single Bottle URL route. show_message checks if the URL path contains only alphabetic characters and hyphens for a message to display. If the message passes the conditions then a page is rendered with that message in an h1 element. If msg does not pass the condition test then an exception is thrown that only alpha characters and hyphens are allowed.

Save app.py and we can run our code. Execute app.py using the python command as follows (make sure your virtualenv is still activated in the terminal where you are running this command):

python app.py

The Bottle development server should start up and display a few lines of output.

Try to access a URL with a path that contains only alphabetic characters and hyphens, such as localhost:8080/hello-world/.

The application was successful in displaying "hello-world" but what if we try a URL that contains numbers in addition to the alphabetic characters, such as localhost:8080/fullstackpython123/?

An HTTP 500 error. That is surely not a good user experience.

The 500 error is obvious to us right now because we are testing the application locally during development. However, what happens when the app is deployed and a user gets the error in their own web browser? They will likely quit out of frustration and you will never know what happened unless you add some error tracking and application monitoring.

Time to modify our code to add Rollbar to report errors that occur.

Monitoring for Errors with Rollbar

Go to the Rollbar homepage in your browser to add their tool to our Bottle app.

Click the "Sign Up" button in the upper right-hand corner. Enter your email address, a username and the password you want on the sign up page.

After the sign up page you will see the onboarding flow where you can enter a project name and select a programming language. For the project name type in "Full Stack Python" then select that you are monitoring a Python app.

Press the "Continue" button at the bottom to move along. The next screen shows us a few instructions to add monitoring to a Python application.

Let's change our Bottle code to let Rollbar collect and aggregate the errors that pop up in our application. Modify app.py to include the following highlighted lines.

import bottle
import os
import re
from bottle import route, template
~~from rollbar.contrib.bottle import RollbarBottleReporter


TEMPLATE_STRING = """
<html>
 <head>
  <title>Full Stack Python Web App</title>
 </head>
 <body>
  <h1>{{ h1 }}</h1>
 </body>
</html>
"""

MIN_MSG_LENGTH = 2
~~ROLLBAR_SECRET = os.environ.get("ROLLBAR_SECRET")
~~
~~rb_monitor = RollbarBottleReporter(access_token=ROLLBAR_SECRET,
~~                                   environment='production')
~~bottle.install(rb_monitor)


@route("/<msg>/")
def show_message(msg):
    """Display a message if the msg value is greater than 2 characters
    in the path.
    """
    valid_length = len(msg) >= MIN_MSG_LENGTH
    valid_name = re.match('^[a-z\-]+$', msg.lower()) is not None
    if valid_length and valid_name:
        return template(TEMPLATE_STRING, h1=msg)
    else:
        error_msg = "Sorry, only alpha characters and hyphens allowed."
        raise Exception(error_msg)


if __name__ == "__main__":
    bottle.run(host='localhost', port=8080)

A new import from rollbar.contrib.bottle import RollbarBottleReporter is our conduit between the application and the Rollbar server. rollbar is the library we installed earlier.

The ROLLBAR_SECRET token needs to be set in an environment variable. Save and quit app.py. Run the following command in the terminal where your virtualenv is activated:

export ROLLBAR_SECRET='token here'

If you are uncertain about what your secret token is, it can be found on the Rollbar onboarding screen.

Note that I typically store all my environment variables in a .env file and use a template.env as a template for what I should fill into .env. .env can be invoked from the terminal using the . .env command. Make sure to never commit your secret tokens to a source control repository though, especially if the repository is public!

After exporting your ROLLBAR_SECRET key as an environment variable we can test that Rollbar is working as we run our application. Run it now using python:

python app.py

Back in your web browser press the "Done! Go to Dashboard" button.

If an event hasn't been reported yet we'll see a waiting screen like this one:

Make sure your Bottle development server is running and try to go to localhost:8080/fullstackpython123/. A 500 server error is immediately reported on the dashboard:

We even get an email with the error (which can also be turned off if you don't want emails for every error):

Nice, with just a few lines of code we now have our Bottle app reporting errors for any user that's working with our application.

What now?

We just learned how to catch and handle errors with Rollbar as a hosted monitoring platform in a simple example Bottle application. Next you will want to add monitoring to more complicated web apps, including ones that use Django or Flask. You can also try Rollbar's more advanced features to:

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

Do you see a typo, syntax issue or just something that's confusing in this blog post? Fork this page's source on GitHub and submit a pull request with a fix or file an issue ticket on GitHub.

How to Provision Ubuntu 16.04 Linux Servers on Linode

Matt Makai — Wed, 20 Sep 2017 00:00:00 -0400

Your live web application must be deployed and run somewhere other than your local development environment. That deployment location is known as a "production environment" and it is built out of one or more servers.

Let's learn how to provision an Ubuntu Linux 16.04 LTS virtual private server (VPS) on Linode that can be used for production or development purposes.

Signing up for Linode

We need a Linode account to provision a server, so start by pointing your web browser to Linode.com. Their landing page will look something like the following image.

You should receive an email for account confirmation. Fill out the appropriate information and add initial credit to your account. If you want to enter a referral code, mine is bfeecaf55a83cd3dd224a5f2a3a001fdf95d4c3d. Your account will go for a quick review to ensure you are not a malicious spam bot and then your account will be fully activated.

Once your account is activated refresh the page. The new page will allow you to add a Linode instance.

Provisioning a server for $5 or $10/month (depending on how much memory and storage you want) is more than enough for small-scale Python web applications.

Select the 1024 option and the data center location of your choice. I chose Newark, NJ because I grew up in northern NJ and otherwise the location is not important for my deployment. If your most of your users are located in a specific region then you should select the data center location closest to them.

Click the "Add this Linode!" button and a dashboard will appear that shows the Linode is being provisioned.

Refresh the page and look for the status to change to "Brand New." Write down or copy the IP address as it will be needed later to SSH into the server, then click on the name of the Linode. A page will appear to show more information about your new virtual private server.

Click the "Rebuild" link.

Select Ubuntu 16.04, which is the current Long Term Support (LTS) release and has a 5 year support lifecycle. This version will receive security updates until April 2021 as shown on the Ubuntu wiki page for LTS releases.

Enter a root password. Make sure you type the password in carefully and remember it! The password will be needed when you log into the server as the root user. The "Deployment Disk Size" and "Swap Disk" can be left as their default values.

When the build process begins Linode will send us back to our server's dashboard page. The progress bars will show the status and in a couple of minutes the server will be ready to boot up.

Boot and Log In

Click the "Boot" button and the Ubuntu boot process will get started. Booting should take less than a minute. Bring up your local command line as we will need it to connect to the remote machine.

SSH into your server with ssh root@{ip.address.here} where {ip.address.here} is your server's IP address, which can be found on the Linode dashboard. For example, if your new Linode's IP address is 66.175.209.129, you'll enter ssh root@66.175.209.129.

You'll likely receive a prompt like the following warning. This prompt states that you've never connected to this server before and it asks if you are sure that this host's signature matches the server on which you intend to connect. Enter yes then enter the root password you created during the earlier Linode server provisioning step.

The authenticity of host '66.175.209.192 (66.175.209.192)' can't be established.
RSA key fingerprint is 51:3c:ba:bc:c3:83:1a:36:b1:2d:e3:f6:6d:f0:11:56.
Are you sure you want to continue connecting (yes/no)? yes

A message like "Welcome to Ubuntu 16.04.3 LTS" will appear followed by a prompt. Now we can enter commands on the remote machine to get the server secured and setup.

Next Steps

You are all set to start configuring your server. You will want to immediately create SSH keys and disable password logins as well as install tools like fail2ban.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Creating Bar Chart Visuals with Bokeh, Bottle and Python 3

Matt Makai — Sun, 30 Jul 2017 00:00:00 -0400

The Bokeh open source Python visualization library assists developers with creating web browser visuals. You can build charts for web applications without coding any JavaScript, like you'd need to do to use libraries such as d3.js and plotly.

Bokeh can create many common and custom visualizations using only Python, such as this bar chart we will create in this tutorial:

Let's use the Bottle web framework with Bokeh to build custom Python web app bar charts.

Our Tools

This tutorial works with either Python 2 or 3, but Python 3 is strongly recommended for new applications. I used Python 3.6.2 while writing this post. In addition to Python throughout this tutorial we will also use the following application dependencies:

Bottle web framework, version 0.12.13
Bokeh data visualization library, version 0.12.6
pandas data structures and analysis library, version 0.20.3
pip and virtualenv, which come packaged with Python 3, to install and isolate the Bottle, Bokeh, and pandas libraries from other Python projects you are working on

If you need help getting your development environment configured before running this code, take a look at this guide for setting up Python 3 and Bottle on Ubuntu 16.04 LTS.

All code in this blog post is available open source under the MIT license on GitHub under the bar-charts-bokeh-bottle-python-3 directory of the blog-code-examples repository. Use the source code as you want to for your own projects.

Installing Bottle and Bokeh

Create a new virtual environment for this project to isolate our dependencies using the following command in the terminal. I usually run the venv command within a separate venvs directory where all my virtualenvs are store.

python3 -m venv bottlechart

Activate the virtualenv.

source bottlechart/bin/activate

The command prompt will change after activating the virtualenv:

Keep in mind that you need to activate the virtualenv in every new terminal window where you want to use the virtualenv to run the project.

Bokeh and Bottle are installable into the now-activated virtualenv using pip. Run this command to get the appropriate Bokeh and Bottle versions.

pip install bokeh==0.12.6 bottle==0.12.13 pandas==0.20.3

Our required dependencies will be installed within our virtualenv after a brief download and installation period.

Installing collected packages: bottle, six, chardet, certifi, idna, urllib3, requests, PyYAML, python-dateutil, MarkupSafe, Jinja2, numpy, tornado, bkcharts, bokeh, pytz, pandas
  Running setup.py install for bottle ... done
  Running setup.py install for PyYAML ... done
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for tornado ... done
  Running setup.py install for bkcharts ... done
  Running setup.py install for bokeh ... done
Successfully installed Jinja2-2.9.6 MarkupSafe-1.0 PyYAML-3.12 bkcharts-0.2 bokeh-0.12.6 bottle-0.12.13 certifi-2017.7.27.1 chardet-3.0.4 idna-2.5 numpy-1.13.1 pandas-0.20.3 python-dateutil-2.6.1 pytz-2017.2 requests-2.18.2 six-1.10.0 tornado-4.5.1 urllib3-1.22

We can now begin coding our web app.

Building the Bottle App

First we'll code a basic Bottle application and then we will add the bar charts to the rendered page.

Create a folder for your project named bottle-bokeh-charts. Within bottle-bokeh-charts create a new file named app.py with the following code:

import os
import bottle
from bottle import route, run, template


app = bottle.default_app()

TEMPLATE_STRING = """
<html>
 <head>
  <title>Bar charts with Bottle and Bokeh</title>
 </head>
 <body>
  <h1>Bugs found over the past {{ bars_count }} days</h1>
 </body>
</html>
"""


@route('/<num_bars:int>/')
def chart(num_bars):
    """Returns a simple template stating the number of bars that should
    be generated when the rest of the function is complete.
    """
    if num_bars <= 0:
        num_bars = 1
    return template(TEMPLATE_STRING, bars_count=num_bars)


if __name__ == '__main__':
    run(host='127.0.0.1', port=8000, debug=False, reloader=True)

The code shown above provides a short Bottle application with a single route, defined with the chart function. chart receives an arbitrary integer value as input. The template function within chart uses the HTML template defined in TEMPLATE_STRING to render an HTML page as a response to incoming requests.

The last two lines in the allow us to run the Bottle application in debug mode on port 8000. Never use debug mode for production deployments! WSGI servers like Gunicorn are built for handling real traffic and will be easier to configure without major security holes.

We can now test out our application.

Make sure your virtualenv is still activated and that you are in the base directory of your project where app.py is located. Run app.py using the python command.

(bottlechart)$ python app.py

Go to localhost:8000/16/ in your web browser. You should see a header message about the number of bugs found over the past 16 days. However, there's no bar chart to accompany that message just yet.

Our single Bottle route is in place but it is not very exciting. Time to create a nice-looking bar chart.

Creating A Bar Chart with Bokeh

We'll build on our basic Bottle app foundation using some new Python code to engage the Bokeh library.

Open app.py back up and add the following highlighted import lines.

import os
import bottle
~~import random
~~from bokeh.models import (HoverTool, FactorRange, Plot, LinearAxis, Grid,
~~                          Range1d)
~~from bokeh.models.glyphs import VBar
~~from bokeh.plotting import figure
~~from bokeh.charts import Bar
~~from bokeh.embed import components
~~from bokeh.models.sources import ColumnDataSource
from bottle import route, run, template

The rest of our application will use these imports to generate random data and the bar chart.

Our bar chart will have "software bugs found" for its theme. The data will randomly generate each time the page is generated. In a real application you would of course likely have a more stable and useful data source.

Continue modifying app.py so the section after the imports looks like the following code.

app = bottle.default_app()

TEMPLATE_STRING = """
<html>
 <head>
  <title>Bar charts with Bottle and Bokeh</title>
~~  <link href="http://cdn.pydata.org/bokeh/release/bokeh-0.12.6.min.css" 
~~        rel="stylesheet">
~~  <link href="http://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.6.min.css" 
~~        rel="stylesheet">
 </head>
 <body>
  <h1>Bugs found over the past {{ bars_count }} days</h1>
~~  {{ !the_div }}
~~  <script src="http://cdn.pydata.org/bokeh/release/bokeh-0.12.6.min.js"></script>
~~  <script src="http://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.6.min.js"></script>
~~  {{ !the_script }}
 </body>
</html>
"""


@route('/<num_bars:int>/')
def chart(num_bars):
    """Returns a simple template stating the number of bars that should
    be generated when the rest of the function is complete.
    """
    if num_bars <= 0:
        num_bars = 1
~~    data = {"days": [], "bugs": [], "costs": []}
~~    for i in range(1, num_bars + 1):
~~        data['days'].append(i)
~~        data['bugs'].append(random.randint(1,100))
~~        data['costs'].append(random.uniform(1.00, 1000.00))

~~    hover = create_hover_tool()
~~    plot = create_bar_chart(data, "Bugs found per day", "days",
~~                            "bugs", hover)
~~    script, div = components(plot)
~~    return template(TEMPLATE_STRING, bars_count=num_bars,
~~                    the_div=div, the_script=script)

The chart function gains three new lists that are randomly generated by Python 3's super-handy random module.

chart calls two functions, create_hover_tool and create_bar_chart. We haven't written those functions yet, so let's do that now. Add these two new functions below the chart function, but before the if __name__ == '__main__': line.

def create_hover_tool():
    # we'll code this function in a moment
    return None


def create_bar_chart(data, title, x_name, y_name, hover_tool=None,
                     width=1200, height=300):
    """Creates a bar chart plot with the exact styling for the centcom
       dashboard. Pass in data as a dictionary, desired plot title,
       name of x axis, y axis and the hover tool HTML.
    """
    source = ColumnDataSource(data)
    xdr = FactorRange(factors=data[x_name])
    ydr = Range1d(start=0,end=max(data[y_name])*1.5)

    tools = []
    if hover_tool:
        tools = [hover_tool,]

    plot = figure(title=title, x_range=xdr, y_range=ydr, plot_width=width,
                  plot_height=height, h_symmetry=False, v_symmetry=False,
                  min_border=10, toolbar_location="above", tools=tools,
                  responsive=True, outline_line_color="#666666")

    glyph = VBar(x=x_name, top=y_name, bottom=0, width=.8,
                 fill_color="#6599ed")
    plot.add_glyph(source, glyph)

    xaxis = LinearAxis()
    yaxis = LinearAxis()

    plot.add_layout(Grid(dimension=0, ticker=xaxis.ticker))
    plot.add_layout(Grid(dimension=1, ticker=yaxis.ticker))
    plot.toolbar.logo = None
    plot.min_border_top = 0
    plot.xgrid.grid_line_color = None
    plot.ygrid.grid_line_color = "#999999"
    plot.yaxis.axis_label = "Bugs found"
    plot.ygrid.grid_line_alpha = 0.1
    plot.xaxis.axis_label = "Days after app deployment"
    plot.xaxis.major_label_orientation = 1
    return plot

That's a lot of new code. The create_hover_tool function does not do anything just yet other than returning. None, which is used when no hover tool is desired for the graph.

Within the create_bar_chart function we take in our randomly-generated data source and convert it into a ColumnDataSource object that is one type of input object we can pass to Bokeh functions. We specify two ranges for the chart's x and y axes.

The tools list will remain empty because we do not yet have a hover tool. A lot of the magic happens in the lines where we create plot using the figure function. We specify all the parameters we want our graph to have such as the size, toolbar, borders and whether or not the graph should be responsive upon changing the web browser size.

The VBar object creates vertical bars to add them to the plot with the add_glyph function.

The last lines of the function change the graph's appearance. For example, we took away the Bokeh logo by specifying plot.toolbar.logo = None and added labels to both axes. I recommend keeping the bokeh.plotting documentation open so you know what your options are for customizing the charts and visualizations.

Let's test our app by trying a 6-bar chart. The Bottle app should automatically reload when you save app.py with the new code. If you shut down the development server, start it back up using python app.py.

When you start up the development server you will receive the following warning because we are using the latest (at the time of this writing) 0.12.6 Bokeh release.

/Users/matt/Envs/bottlechart/lib/python3.6/site-packages/bokeh/util/deprecation.py:34: BokehDeprecationWarning: 
The bokeh.charts API has moved to a separate 'bkcharts' package.

This compatibility shim will remain until Bokeh 1.0 is released.
After that, if you want to use this API you will have to install
the bkcharts package explicitly.

Eventually a separate bkcharts project will be required but for now we can keep our code as is.

Open your browser to localhost:8000/6/.

That one looks a bit sparse, so we can crank it up by 3x to 18 bars by going to localhost:5000/18/.

Now another 5x to 90 bars with localhost:5000/90/.

Looking good so far! What about that hover tool we skipped over though? We can add the hover tool with just a few more lines of code in the create_hover_tool function.

Creating a Hover Tool

Add these highlighted lines to app.py within the create_hover_tool function.

def create_hover_tool():
~~    """Generates the HTML for the Bokeh's hover data tool on our graph."""
~~    hover_html = """
~~      <div>
~~        <span class="hover-tooltip">$x</span>
~~      </div>
~~      <div>
~~        <span class="hover-tooltip">@bugs bugs</span>
~~      </div>
~~      <div>
~~        <span class="hover-tooltip">$@costs{0.00}</span>
~~      </div>
~~    """
~~    return HoverTool(tooltips=hover_html)

Embedding HTML within your Python application isn't usually a great idea but it works for small snippets like this hover tool. The hover tool uses $x to show the bar's x axis, @bugs to show the "bugs" field from our data source, and $@costs{0.00} to show the "costs" field formatted as a dollar amount with exactly 2 decimal places.

Ensure that you changed return None to return HoverTool(tooltips=hover_html) in your function so the results of the new code are reflected in the refreshed graph.

Go back to the browser and reload the localhost:8000/122/ page.

Well done! Try playing around with the number of bars in the URL and the window size to see what the graph looks like under different conditions.

The chart gets crowded with more than 100. However, you can try to create as many bars as you want if your computer can handle the rendering. This screenshot shows what the completely impractical amount of 40,000 bars looks like:

You may need to do some more work to get the chart to be useful for displaying more than a couple hundred bars at a time.

What now?

We created a nice little configurable bar chart using the Bokeh code library.

Next you can change the input data source, work with other types of charts or modify the chart color scheme.

There is a lot more than Bokeh can do. Take a look at the official project documentation , GitHub repository, the Full Stack Python Bokeh page or take a look at other topics on Full Stack Python.

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

Do you see something wrong in this blog post? Fork this page's source on GitHub and submit a pull request with a fix.

How to Add Hosted Monitoring to Flask Web Applications

Matt Makai — Sun, 23 Jul 2017 00:00:00 -0400

How do you know whether your application is running properly with minimal errors after building and deploying it? The fastest and easiest way to monitor your operational Flask web application is to integrate one of the many available fantastic hosted monitoring tools.

In this post we will quickly add Rollbar monitoring to catch errors and visualize our application is running properly. There are also many other great hosted monitoring tools, which you can check out on the monitoring page.

Our Tools

We can use either Python 2 or 3 to build this tutorial, but Python 3 is strongly recommended for all new applications. I used Python 3.6.2 to execute my code. We will also use the following application dependencies throughout the post:

Flask web framework, version 0.12.2
pyrollbar monitoring instrumentation library, version 0.13.12
blinker for signaling support in Flask applications so pyrollbar can report on all errors
A free Rollbar account where we will send error data and view it when it is captured
pip and the virtualenv virtual environment library, which come packaged with Python 3, to install and isolate the Flask and Rollbar libraries from other Python projects you are working on

If you need help getting your development environment configured before running this code, take a look at this guide for setting up Python 3 and Flask on Ubuntu 16.04 LTS.

All code in this blog post is available open source under the MIT license on GitHub under the monitor-flask-apps directory of the blog-code-examples repository. Use and abuse the source code as you desire for your own applications.

Installing Dependencies

Change into the directory where you keep your Python virtualenvs. Create a new virtual environment for this project using the following command.

python3 -m venv monitorflask

Activate the virtualenv.

source monitorflask/bin/activate

The command prompt will change after activating the virtualenv:

Remember that you need to activate the virtualenv in every new terminal window where you want to use the virtualenv to run the project.

Flask, Rollbar and Blinker can now be installed into the now-activated virtualenv.

pip install flask==0.12.2 rollbar==0.13.12 blinker==1.4

Our required dependencies should be installed within our virtualenv after a short installation period. Look for output like the following to confirm everything worked.

Installing collected packages: blinker, itsdangerous, click, MarkupSafe, Jinja2, Werkzeug, Flask, idna, urllib3, chardet, certifi, requests, six, rollbar
  Running setup.py install for blinker ... done
  Running setup.py install for itsdangerous ... done
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for rollbar ... done
Successfully installed Flask-0.12.2 Jinja2-2.9.6 MarkupSafe-1.0 Werkzeug-0.12.2 blinker-1.4 certifi-2017.4.17 chardet-3.0.4 click-6.7 idna-2.5 itsdangerous-0.24 requests-2.18.1 rollbar-0.13.12 six-1.10.0 urllib3-1.21.1

Now that we have our Python dependencies installed into our virtualenv we can create the initial version of our application.

Building Our Flask App

Create a folder for your project named monitor-flask-apps. Change into the folder and then create a file named app.py with the following code.

import re
from flask import Flask, render_template, Response
from werkzeug.exceptions import NotFound


app = Flask(__name__)
MIN_PAGE_NAME_LENGTH = 2


@app.route("/<string:page>/")
def show_page(page):
    try:
        valid_length = len(page) >= MIN_PAGE_NAME_LENGTH
        valid_name = re.match('^[a-z]+$', page.lower()) is not None
        if valid_length and valid_name:
            return render_template("{}.html".format(page))
        else:
            msg = "Sorry, couldn't find page with name {}".format(page)
            raise NotFound(msg)
    except:
        return Response("404 Not Found")


if __name__ == "__main__":
    app.run(debug=True)

The above application code has some standard Flask imports so we can create a Flask web app and render template files. We have a single function named show_page to serve a single Flask route. show_page checks if the URL path contains only lowercase alpha characters for a potential page name. If the page name can be found in the templates folder then the page is rendered, otherwise an exception is thrown that the page could not be found. We need to create at least one template file if our function is ever going to return a non-error reponse.

Save app.py and make a new subdirectory named templates under your project directory. Create a new file named battlegrounds.html and put the following Jinja2 template markup into it.

<!DOCTYPE html>
<html>
  <head>
    <title>You found the Battlegrounds GIF!</title>
  </head>
  <body>
    <h1>PUBG so good.</h1>
    <img src="https://media.giphy.com/media/3ohzdLMlhId2rJuLUQ/giphy.gif">
  </body>
</html>

The above Jinja2 template is basic HTML without any embedded template tags. The template creates a very plain page with a header description of "PUBG so good" and a GIF from this excellent computer game.

Time to run and test our code. Change into the base directory of your project where app.py file is located. Execute app.py using the python command as follows (make sure your virtualenv is still activated in the terminal where you are running this command):

python app.py

The Flask development server should start up and display a few lines of output.

What happens when we access the application running on localhost port 5000?

HTTP status 404 page not found, which is what we expected because we only defined a single route and it did not live at the base path.

We created a template named battlegrounds.html that should be accessible when we go to localhost:5000/battlegrounds/.

The application successfully found the battlegrounds.html template but that is the only one available. What if we try localhost:5000/fullstackpython/?

HTTP 500 error. That's no good.

The 404 and 500 errors are obvious to us right now because we are testing the application locally. However, what happens when the app is deployed and a user gets the error in their own web browser? They will typically quit out of frustration and you will never know what happened unless you add some error tracking and application monitoring.

We will now modify our code to add Rollbar to catch and report those errors that occur for our users.

Handling Errors

Head to Rollbar's homepage so we can add their hosted monitoring tools to our oft-erroring Flask app.

Click the "Sign Up" button in the upper right-hand corner. Enter your email address, a username and the password you want on the sign up page.

After the sign up page you will see the onboarding flow where you can enter a project name and select a programming language. For project name enter "Battlegrounds" and select that you are monitoring a Python app.

Press the "Continue" button at the bottom to move along. The next screen shows us a few quick instructions to add monitoring to our Flask application.

Let's modify our Flask application to test whether we can properly connect to Rollbar's service. Change app.py to include the following highlighted lines.

~~import os
import re
~~import rollbar
from flask import Flask, render_template, Response
from werkzeug.exceptions import NotFound


app = Flask(__name__)
MIN_PAGE_NAME_LENGTH = 2


~~@app.before_first_request
~~def add_monitoring():
~~    rollbar.init(os.environ.get('ROLLBAR_SECRET'))
~~    rollbar.report_message('Rollbar is configured correctly')


@app.route("/<string:page>/")
def show_page(page):
    try:
        valid_length = len(page) >= MIN_PAGE_NAME_LENGTH
        valid_name = re.match('^[a-z]+$', page.lower()) is not None
        if valid_length and valid_name:
            return render_template("{}.html".format(page))
        else:
            msg = "Sorry, couldn't find page with name {}".format(page)
            raise NotFound(msg)
    except:
        return Response("404 Not Found")


if __name__ == "__main__":
    app.run(debug=True)

We added a couple of new imports, os and rollbar. os allows us to grab environment variable values, such as our Rollbar secret key. rollbar is the library we installed earlier. The two lines below the Flask app instantiation are to initialize Rollbar using the Rollbar secret token and send a message to the service that it started correctly.

The ROLLBAR_SECRET token needs to be set in an environment variable. Save an quit the app.py. Run export ROLLBAR_SECRET='token here' on the command line where your virtualenv is activated. This token can be found on the Rollbar onboarding screen.

I typically store all my environment variables in a file like template.env and invoke it from the terminal using the . ./template.env command. Make sure to avoid committing your secret tokens to a source control repository, especially if the repository is public!

After exporting your ROLLBAR_SECRET key as an environment variable we can test that Rollbar is working as we run our application. Run it now using python:

python app.py

Back in your web browser press the "Done! Go to Dashboard" button. Don't worry about the "Report an Error" section code, we can get back to that in a moment.

If the event hasn't been reported yet we'll see a waiting screen like this one:

Once Flask starts up though, the first event will be populated on the dashboard.

Okay, our first test event has been populated, but we really want to see all the errors from our application, not a test event.

Testing Error Handling

How do we make sure real errors are reported rather than just a simple test event? We just need to add a few more lines of code to our app.

import os
import re
import rollbar
~~import rollbar.contrib.flask
from flask import Flask, render_template, Response
~~from flask import got_request_exception
from werkzeug.exceptions import NotFound


app = Flask(__name__)
MIN_PAGE_NAME_LENGTH = 2


@app.before_first_request
def add_monitoring():
    rollbar.init(os.environ.get('ROLLBAR_SECRET'))
~~    ## delete the next line if you dont want this event anymore
    rollbar.report_message('Rollbar is configured correctly')
~~    got_request_exception.connect(rollbar.contrib.flask.report_exception, app)


@app.route("/<string:page>/")
def show_page(page):
    try:
        valid_length = len(page) >= MIN_PAGE_NAME_LENGTH
        valid_name = re.match('^[a-z]+$', page.lower()) is not None
        if valid_length and valid_name:
            return render_template("{}.html".format(page))
        else:
            msg = "Sorry, couldn't find page with name {}".format(page)
            raise NotFound(msg)
    except:
~~        rollbar.report_exc_info()
        return Response("404 Not Found")


if __name__ == "__main__":
    app.run(debug=True)

The above highlighted code modifies the application so it reports all Flask errors as well as our HTTP 404 not found issues that happen within the show_page function.

Make sure your Flask development server is running and try to go to localhost:5000/b/. You will receive an HTTP 404 exception and it will be reported to Rollbar. Next go to localhost:5000/fullstackpython/ and an HTTP 500 error will occur.

You should see an aggregation of errors as you test out these errors:

Woohoo, we finally have our Flask app reporting all errors that occur for any user back to the hosted Rollbar monitoring service!

What's Next?

We just learned how to catch and handle errors with Rollbar as a hosted monitoring platform in a simple Flask application. Next you will want to add monitoring to your more complicated web apps. You can also check out some of Rollbar's more advanced features such as:

There is a lot more to learn about web development and deployments so keep learning by reading up on Flask and other web frameworks such as Django, Pyramid and Sanic. You can also learn more about integrating Rollbar with Python applications via their Python documentation.

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

See something wrong in this blog post? Fork this page's source on GitHub and submit a pull request with a fix.

How to Create Your First Static Site with Pelican and Jinja2

Matt Makai — Fri, 09 Jun 2017 00:00:00 -0400

Pelican is an incredibly well-built Python tool for creating static sites.

Full Stack Python is generated with Pelican, Jinja2 templates and Markdown. This site is deployed to Amazon S3 and currently handles over one hundred thousand readers per month. There are never scaling concerns because a static site is pre-generated before deployment and a web server simply responds with existing files rather than executing any code on the server during the HTTP request-response cycle.

In this tutorial you will learn how to create your own static website from scratch using Pelican.

Our simple static site will have pages that look like the above screenshot but the entire site can be easily customized and expanded with your own design and content.

Our Tools

This tutorial should work with either Python 2 or 3, but Python 3 is strongly recommended for all new applications. I used Python 3.6.1 to write this post. In addition to Python, throughout this tutorial we will also use the following application dependencies:

Pelican static site generator, version 3.7.1
Markdown parsing library to handle Markdown as a content input format, version 2.6.8
Jinja2, a Python template engine, version 2.9.6
pip and virtualenv, which come packaged with Python 3, to install and isolate the Pelican, Markdown, and Jinja2 libraries from any of your other Python projects

If you need help getting your development environment configured, take a look at this guide for setting up Python 3 and Flask on Ubuntu 16.04 LTS

All code in this blog post is available open source under the MIT license on GitHub under the generating-static-websites-pelican-jinja2-markdown directory of the blog-code-examples repository. Use and abuse the source code as you like for your own applications.

Install the Pelican and Markdown libraries

Start by creating a new virtual environment for your project. My virtualenv is named staticsite but you can name yours whatever matches the project you are creating.

python3 -m venv staticsite

Activate the virtualenv.

source staticsite/bin/activate

The virtualenv will prepend its name to your command prompt when it is activated.

Install the appropriate dependencies after your virtualenv is activated. Use the pip command to install Pelican and Markdown, which will also install Jinja2 because Pelican specifies it as a dependency.

pip install pelican==3.7.1 markdown==2.6.8

Run the pip command and after everything is installed you should see output similar to the following "Successfully installed" message.

Installing collected packages: pygments, pytz, six, feedgenerator, blinker, unidecode, MarkupSafe, jinja2, python-dateutil, docutils, pelican, markdown
  Running setup.py install for feedgenerator ... done
  Running setup.py install for blinker ... done
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for markdown ... done
Successfully installed MarkupSafe-1.0 blinker-1.4 docutils-0.13.1 feedgenerator-1.9 jinja2-2.9.6 markdown-2.6.8 pelican-3.7.1 pygments-2.2.0 python-dateutil-2.6.0 pytz-2017.2 six-1.10.0 unidecode-0.4.20

Now that our dependencies are installed into the virtualenv we can start building our static site.

Generate a Basic Site

Create a new directory to store your project. My site will contain some of my favorite retro synthwave artists as examples, but of course your site can contain whatever subjects that you want.

Change into the project directory after creating it.

mkdir retrosynth
cd retrosynth

Run the pelican-quickstart command within the new project directory.

(staticsite) $ pelican-quickstart

The quickstart script will rattle off a bunch of questions. Follow along with the answers below or modify them for your own site name and desired configuration.

Welcome to pelican-quickstart v3.7.1.

This script will help you create a new Pelican-based website.

Please answer the following questions so this script can generate the files
needed by Pelican.


> Where do you want to create your new web site? [.]  
> What will be the title of this web site? RetroSynthwave
> Who will be the author of this web site? Matt Makai
> What will be the default language of this web site? [en] 
> Do you want to specify a URL prefix? e.g., http://example.com   (Y/n) n
> Do you want to enable article pagination? (Y/n) n
> What is your time zone? [Europe/Paris] America/New_York
> Do you want to generate a Fabfile/Makefile to automate generation and publishing? (Y/n)y
> Do you want an auto-reload & simpleHTTP script to assist with theme and site development? (Y/n) y
> Do you want to upload your website using FTP? (y/N) n
> Do you want to upload your website using SSH? (y/N) n
> Do you want to upload your website using Dropbox? (y/N) n
> Do you want to upload your website using S3? (y/N) y
> What is the name of your S3 bucket? [my_s3_bucket] 
> Do you want to upload your website using Rackspace Cloud Files? (y/N) n
> Do you want to upload your website using GitHub Pages? (y/N) n
Done. Your new project is available at /Users/matt/devel/py/retrosynth
(staticsite) $

What did we just create using Pelican's quickstart script? Check out the new files in the directory.

(staticsite) $ ls
Makefile        develop_server.sh   pelicanconf.py
content         fabfile.py          publishconf.py

The quickstart created five files and one new directory:

Makefile: make command convenience tasks for common operations such as running a development server, building a site and cleaning extraneous build files
fabfile.py: A Fabric file that has some of the same types of commands as the Makefile. Fabric is a wonderful code library but for now I recommend skipping the Fabric file because unfortunately Fabric does not yet support Python 3.
develop_server.sh: shell script for running the development server
pelicanconf.py: settings file for your Pelican project. If you are used to earlier versions of Pelican this file was instead named settings.py
publishconf.py: another (optional) settings file that can be considered as a "production" settings file when you move past the development phase and want to deploy your site
content: location for your markup files, which should be stored under pages and posts directories

We can use these files as the base for our new static site. Let's see what it looks like by default by running it via the devserver task in the Makefile.

make devserver

The Pelican development server will start serving up your site with a daemon process. Go to localhost:8000 in your web browser and you will see the first version of your static site.

What if you don't have make installed on your system? Change into the output directory and use the python -m http.server command to use the built-in Python 3 HTTP server for your generated files.

When you want to kill the development server look for a file named pelican.pidunder your project directory. The pelican.pid file is created by Pelican and contains the process ID for your development server.

(staticsite) $ cat pelican.pid 
1365

Use the ps and grep commands to view the process then stop the process with the kill command as follows. Remember that your process ID will almost definitely be different from the 1365 ID for my process.

Kill the development server now so that we can use different commands to serve our site after we create our initial content.

(staticsite) $ ps -A | grep 1365
 1365 ttys003    0:01.43 /Library/Frameworks/Python.framework/Versions/3.6/Resources/Python.app/Contents/MacOS/Python /Users/matt/Envs/staticsite/bin/pelican --debug --autoreload -r /Users/matt/devel/py/retrosynth/content -o /Users/matt/devel/py/retrosynth/output -s /Users/matt/devel/py/retrosynth/pelicanconf.py
 1411 ttys003    0:00.00 grep 1365
(staticsite) $ kill 1365
(staticsite) $ ps -A | grep 1365
 1413 ttys003    0:00.00 grep 1365

It is up to you whether you want to use the development server or not while creating your site. Every time I want to view my changes for Full Stack Python I regenerate the site using my own Makefile that wraps the pelican command. The python -m http.server command constantly serves up each build's changes.

Alright, now that we have our starter files we can get to work creating some initial content.

Write Some Content

Pelican can accept both Markdown and reStructureText markup files as input.

Make a new subdirectory under the content named posts. Change into the posts directory. Create a new file named gunship.markdown with the following content.

title: Gunship
slug: gunship
category: bands
date: 2017-06-09
modified: 2017-06-09


[Gunship](https://www.gunshipmusic.com/) is a *retro synthwave* artist out of the UK.

[Revel in Your Time](https://www.youtube.com/watch?v=uYRZV8dV10w), 
[Tech Noir](https://www.youtube.com/watch?v=-nC5TBv3sfU), 
[Fly for Your Life](https://www.youtube.com/watch?v=Jv1ZN8c4_Gs) 
and 
[The Mountain](https://www.youtube.com/watch?v=-HYRTJr8EyA) 
are all quality songs by Gunship. Check out those amazing music videos!

Also take a look at other retro synthwave artists such as
[Trevor Something](https://trevorsomething.bandcamp.com/), 
[Droid Bishop](https://droidbishop.bandcamp.com/),
[FM-84](https://fm84.bandcamp.com/)
and 
[Daniel Deluxe](https://danieldeluxe.bandcamp.com/).

Our make file can also help us regenerate the site when changes occur if we choose to not use the development server.

We used the devserver task earlier, but what other task are available to us via the Makefile?

make

make should show us all of the following tasks we can run.

Makefile for a pelican Web site                                           

Usage:                                                                    
   make html                           (re)generate the web site          
   make clean                          remove the generated files         
   make regenerate                     regenerate files upon modification 
   make publish                        generate using production settings 
   make serve [PORT=8000]              serve site at http://localhost:8000
   make serve-global [SERVER=0.0.0.0]  serve (as root) to :80    
   make devserver [PORT=8000]          start/restart develop_server.sh    
   make stopserver                     stop local server                  
   make ssh_upload                     upload the web site via SSH        
   make rsync_upload                   upload the web site via rsync+ssh  
   make dropbox_upload                 upload the web site via Dropbox    
   make ftp_upload                     upload the web site via FTP        
   make s3_upload                      upload the web site via S3         
   make cf_upload                      upload the web site via Cloud Files
   make github                         upload the web site via gh-pages   

Set the DEBUG variable to 1 to enable debugging, e.g. make DEBUG=1 html   
Set the RELATIVE variable to 1 to enable relative urls

The html task is what we are looking for to invoke the pelican command using our pelicanconf.py settings file.

(staticsite) $ make html
pelican /Users/matt/devel/py/retrosynth/content -o /Users/matt/devel/py/retrosynth/output -s /Users/matt/devel/py/retrosynth/pelicanconf.py 
Done: Processed 1 article, 0 drafts, 0 pages and 0 hidden pages in 0.14 seconds.

Our site has been regenerated and placed in the output directory.

If you used the make devserver command earlier then change into the output directory and give Python's built-in HTTP server a shot with the following command.

cd output
python -m http.server

Our first post in all its glory...

You can change the HTTP server port binding by adding a number after the command, if you want to serve more than one static site at a time or you already have an application bound to port 8000.

python -m http.server 8005

Note that if you are using Python 2 the equivalent HTTP server command is python -m SimpleHTTPServer.

Our site now has some very basic content. We could expand this start into many more posts and pages but let's learn how to modify the site configuration.

Edit Site Configuration

Pelican's quickstart assumed a bunch of defaults that may or may not be applicable to your site. Open up the pelicanconf.py file to change some of the defaults.

Look for the TIMEZONE variable. If it's not right for your location then modify it to your zone. Wikipedia has a handy table of valid time zones values.

Also modify the LINKS tuple to include your site (or Full Stack Python!) instead of including the "you can modify those links" link. Change the last line of LINKS so it looks like the following tuple of tuples.

# Blogroll
LINKS = (('Pelican', 'http://getpelican.com/'),
         ('Python.org', 'http://python.org/'),
         ('Jinja2', 'http://jinja.pocoo.org/'),
         ('Full Stack Python', 'https://www.fullstackpython.com/'),)

Instead of using the make html file, this time we will invoke the pelican command directly from the command line. There is nothing wrong with the Makefile, but it is a good idea to get comfortable with Pelican directly instead of only through build files.

pelican -s pelicanconf.py -o output content

Now run the HTTP server if you do not already have it running in another terminal window.

cd output
python -m http.server

Head back to the browser and refresh to view the updated configuration.

What happens when we click on the blog post title? It takes us to a very similar-looking page with the localhost:8000/gunship.html URL.

Alright, we updated some basic site-wide data, but our site really could use a change of paint.

Modify Site Theme

Changing the site theme is really where you can turn a standard blog into whatever type of site you want to build. While the default Pelican configuration creates a blog template, you do not need to have a chronological structure if it is not right for your website.

Create a new directory under your project directory that is named theme. Within theme create another directory named templates. templates is where our Jinja2 templates will be stored and can override the default theme.

Start by creating a file named base.html which will store the boilerplate used by templates across the site.

<!DOCTYPE html>
<html lang="en">
<head>
 <title>{% block title %}{% endblock %}</title>
</head>
<body>
 <div class="container">
  {% block content %}{% endblock %}
 </div> 
</body>
</html>

Within theme/templates create a file named article.html that will have a different theme for blog posts than the rest of the site. Fill article.html with the following Jinja2 markup.

{% extends "base.html" %}

{% block title %}{{ article.title }}{% endblock %}

{% block content %}
<div class="row">
 <div class="col-md-8">
  <h1>{{ article.title }}</h1>
  <label>Posted on <strong>{{ article.date }}</strong></label>
  {{ article.content }}
 </div>
</div>
{% endblock %}

Next we will use a Jinja2 template to override the default index.html main page. Again within the theme/templates directory, create a file named index.html with the following markup.

{% extends "base.html" %}

{% block title %}{{ SITENAME }}{% endblock %}

{% block content %}
<div class="row">
 <div class="col-md-8">
  <h1>{{ SITENAME }}</h1>
  {% for article in articles %}
   <h2><a href="/{{ article.slug }}.html">{{ article.title }}</a></h2>
   <label>Posted on <strong>{{ article.date }}</strong></label>
   {{ article.content|truncate(110) }}
  {% else %}
   No posts yet!
  {% endfor %}
 </div>
</div>
{% endblock %}

Regenerate the site and make sure you are serving it with the development server or the python -m http.server command.

Make sure to use the new -t theme flag to specify that the Jinja2 templates within the theme directory should be applied to the site.

pelican -s pelicanconf.py -o output -t theme content

Go to localhost:8000 and refresh the page. The styling on the main page is now different because it uses the index.html theme.

Click on the title of the Gunship post. This page uses the article.html template, although it's hard to tell because there is no CSS applied to the page.

Pretty sparse! We can at least add the Bootstrap CSS to the HTML to align our content.

Within base.html, add the following line for Bootstrap under <title>{% block title %}{% endblock %}</title> and above </head>.

<!-- Latest compiled and minified Bootstrap CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">

Regenerate the site and refresh the Gunship page.

Well at least our design has moved from 1996 to 2001. I am sure you can do a whole lot more to improve your own site's design.

The new base.html does not provide much of a theme yet but it at least provides a fresh start for completely customized sites.

What's next?

You generated your first Pelican static website using Markdown and Jinja2. Additional modifications can be made to the Jinja2 templates and the content contained in the Markdown files.

Do you want to deploy your new static website to GitHub Pages or an S3 bucket? Well, that's a story for another Full Stack Python tutorial...

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

See something wrong in this blog post? Fork this page's source on GitHub and submit a pull request.

Responsive Bar Charts with Bokeh, Flask and Python 3

Matt Makai — Sun, 30 Jul 2017 00:00:00 -0400

Bokeh is a powerful open source Python library that allows developers to generate JavaScript data visualizations for their web applications without writing any JavaScript. While learning a JavaScript-based data visualization library like d3.js can be useful, it's often far easier to knock out a few lines of Python code to get the job done.

With Bokeh, we can create incredibly detailed interactive visualizations, or just traditional ones like the following bar chart.

Let's use the Flask web framework with Bokeh to create custom bar charts in a Python web app.

Our Tools

This tutorial works with either Python 2 or 3, but Python 3 is strongly recommended for new applications. I used Python 3.6.1 while writing this post. In addition to Python throughout this tutorial we will also use the following application dependencies:

Flask web framework, version 0.12.2
Bokeh data visualization library, version 0.12.5
pandas data structures and analysis library, version 0.20.1
pip and virtualenv, which come packaged with Python 3, to install and isolate the Flask, Bokeh, and pandas libraries from any other Python projects you might be working on

If you need help getting your development environment configured before running this code, take a look at this guide for setting up Python 3 and Flask on Ubuntu 16.04 LTS

All code in this blog post is available open source under the MIT license on GitHub under the bar-charts-bokeh-flask-python-3 directory of the blog-code-examples repository. Use and abuse the source code as you like for your own applications.

Installing Bokeh and Flask

Create a fresh virtual environment for this project to isolate our dependencies using the following command in the terminal. I typically run this command within a separate venvs directory where all my virtualenvs are store.

python3 -m venv barchart

Activate the virtualenv.

source barchart/bin/activate

The command prompt will change after activating the virtualenv:

Keep in mind that you need to activate the virtualenv in every new terminal window where you want to use the virtualenv to run the project.

Bokeh and Flask are installable into the now-activated virtualenv using pip. Run this command to get the appropriate Bokeh and Flask versions.

pip install bokeh==0.12.5 flask==0.12.2 pandas==0.20.1

After a brief download and installation period our required dependencies should be installed within our virtualenv. Look for output to confirm everything worked.

Installing collected packages: six, requests, PyYAML, python-dateutil, MarkupSafe, Jinja2, numpy, tornado, bokeh, Werkzeug, itsdangerous, click, flask, pytz, pandas
  Running setup.py install for PyYAML ... done
  Running setup.py install for MarkupSafe ... done
  Running setup.py install for tornado ... done
  Running setup.py install for bokeh ... done
  Running setup.py install for itsdangerous ... done
Successfully installed Jinja2-2.9.6 MarkupSafe-1.0 PyYAML-3.12 Werkzeug-0.12.2 bokeh-0.12.5 click-6.7 flask-0.12.2 itsdangerous-0.24 numpy-1.12.1 pandas-0.20.1 python-dateutil-2.6.0 pytz-2017.2 requests-2.14.2 six-1.10.0 tornado-4.5.1

Now we can start building our web application.

Starting Our Flask App

We are going to first code a basic Flask application then add our bar chart to the rendered page.

Create a folder for your project then within it create a file named app.py with these initial contents:

from flask import Flask, render_template


app = Flask(__name__)


@app.route("/<int:bars_count>/")
def chart(bars_count):
    if bars_count <= 0:
        bars_count = 1
    return render_template("chart.html", bars_count=bars_count)


if __name__ == "__main__":
    app.run(debug=True)

The above code is a short one-route Flask application that defines the chart function. chart takes in an arbitrary integer as input which will later be used to define how much data we want in our bar chart. The render_template function within chart will use a template from Flask's default template engine named Jinja2 to output HTML.

The last two lines in the allow us to run the Flask application from the command line on port 5000 in debug mode. Never use debug mode for production, that's what WSGI servers like Gunicorn are built for.

Create a subdirectory within your project folder named templates. Within templates create a file name chart.html. chart.html was referenced in the chart function of our app.py file so we need to create it before our app will run properly. Populate chart.html with the following Jinja2 markup.

<!DOCTYPE html>
<html>
  <head>
    <title>Bar charts with Bokeh!</title>
  </head>
  <body>
    <h1>Bugs found over the past {{ bars_count }} days</h1>
  </body>
</html>

chart.html's boilerplate displays the number of bars passed into the chart function via the URL.

The <h1> tag's message on the number of bugs found goes along with our sample app's theme. We will pretend to be charting the number of bugs found by automated tests run each day.

We can test our application out now.

Make sure your virtualenv is still activated and that you are in the base directory of your project where app.py is located. Run app.py using the python command.

$(barchart) python app.py

Go to localhost:5000/16/ in your web browser. You should see a large message that changes when you modify the URL.

Our simple Flask route is in place but that's not very exciting. Time to add our bar chart.

Generating the Bar Chart

We can build on the basic Flask app foundation that we just wrote with some new Python code that uses Bokeh.

Open app.py back up and change the top of the file to include the following imports.

import random
from bokeh.models import (HoverTool, FactorRange, Plot, LinearAxis, Grid,
                          Range1d)
from bokeh.models.glyphs import VBar
from bokeh.plotting import figure
from bokeh.charts import Bar
from bokeh.embed import components
from bokeh.models.sources import ColumnDataSource
from flask import Flask, render_template

Throughout the rest of the file we will need these Bokeh imports along with the random module to generate data and our bar chart.

Our bar chart will use "software bugs found" as a theme. The data will be randomly generated each time the page is refreshed. In a real application you'd have a more stable and useful data source!

Continue modifying app.py so the section after the imports looks like the following code.

app = Flask(__name__)


@app.route("/<int:bars_count>/")
def chart(bars_count):
    if bars_count <= 0:
        bars_count = 1

    data = {"days": [], "bugs": [], "costs": []}
    for i in range(1, bars_count + 1):
        data['days'].append(i)
        data['bugs'].append(random.randint(1,100))
        data['costs'].append(random.uniform(1.00, 1000.00))

    hover = create_hover_tool()
    plot = create_bar_chart(data, "Bugs found per day", "days",
                            "bugs", hover)
    script, div = components(plot)

    return render_template("chart.html", bars_count=bars_count,
                           the_div=div, the_script=script)

The chart function gains three new lists that are randomly generated by Python 3's super-handy random module.

chart calls two functions, create_hover_tool and create_bar_chart. We haven't written those functions yet so continue adding code below chart:

def create_hover_tool():
    # we'll code this function in a moment
    return None


def create_bar_chart(data, title, x_name, y_name, hover_tool=None,
                     width=1200, height=300):
    """Creates a bar chart plot with the exact styling for the centcom
       dashboard. Pass in data as a dictionary, desired plot title,
       name of x axis, y axis and the hover tool HTML.
    """
    source = ColumnDataSource(data)
    xdr = FactorRange(factors=data[x_name])
    ydr = Range1d(start=0,end=max(data[y_name])*1.5)

    tools = []
    if hover_tool:
        tools = [hover_tool,]

    plot = figure(title=title, x_range=xdr, y_range=ydr, plot_width=width,
                  plot_height=height, h_symmetry=False, v_symmetry=False,
                  min_border=0, toolbar_location="above", tools=tools,
                  responsive=True, outline_line_color="#666666")

    glyph = VBar(x=x_name, top=y_name, bottom=0, width=.8,
                 fill_color="#e12127")
    plot.add_glyph(source, glyph)

    xaxis = LinearAxis()
    yaxis = LinearAxis()

    plot.add_layout(Grid(dimension=0, ticker=xaxis.ticker))
    plot.add_layout(Grid(dimension=1, ticker=yaxis.ticker))
    plot.toolbar.logo = None
    plot.min_border_top = 0
    plot.xgrid.grid_line_color = None
    plot.ygrid.grid_line_color = "#999999"
    plot.yaxis.axis_label = "Bugs found"
    plot.ygrid.grid_line_alpha = 0.1
    plot.xaxis.axis_label = "Days after app deployment"
    plot.xaxis.major_label_orientation = 1
    return plot

There is a whole lot of new code above so let's break it down. The create_hover_tool function does not do anything yet, it simply returns None, which we can use if we do not want a hover tool. The hover tool is an overlay that appears when we move our mouse cursor over one of the bars or touch a bar on a touchscreen so we can see more data about the bar.

Within the create_bar_chart function we take in our generated data source and convert it into a ColumnDataSource object that is one type of input object we can pass to Bokeh functions. We specify two ranges for the chart's x and y axes.

Since we do not yet have a hover tool the tools list will remain empty. The line where we create plot using the figure function is where a lot of the magic happens. We specify all the parameters we want our graph to have such as the size, toolbar, borders and whether or not the graph should be responsive upon changing the web browser size.

We create vertical bars with the VBar object and add them to the plot using the add_glyph function that combines our source data with the VBar specification.

The last lines of the function modify the look and feel of the graph. For example I took away the Bokeh logo by specifying plot.toolbar.logo = None and added labels to both axes. I recommend keeping the bokeh.plottin documentation open to know what your options are for customizing your visualizations.

We just need a few updates to our templates/chart.html file to display the visualization. Open the file and add these 6 lines to the file. Two of these lines are for the required CSS, two are JavaScript Bokeh files and the remaining two are the generated chart.

<!DOCTYPE html>
<html>
  <head>
    <title>Bar charts with Bokeh!</title>
    <link href="http://cdn.pydata.org/bokeh/release/bokeh-0.12.5.min.css" rel="stylesheet">
    <link href="http://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.0.min.css" rel="stylesheet">
  </head>
  <body>
    <h1>Bugs found over the past {{ bars_count }} days</h1>
    {{ the_div|safe }}
    <script src="http://cdn.pydata.org/bokeh/release/bokeh-0.12.5.min.js"></script>
    <script src="http://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.5.min.js"></script>
    {{ the_script|safe }}
  </body>
</html>

Alright, let's give our app a try with a simple chart of 4 bars. The Flask app should automatically reload when you save app.py with the new code but if you shut down the development server fire it back up with the python app.py command.

Open your browser to localhost:5000/4/.

That one looks a bit sparse, so we can crank it up by 4x to 16 bars by going to localhost:5000/16/.

Now another 4x to 128 bars with localhost:5000/128/...

Looking good so far. But what about that hover tool to drill down into each bar for more data? We can add the hover with just a few lines of code in the create_hover_tool function.

Adding a Hover Tool

Within app.py modify the create_hover_tool to match the following code.

def create_hover_tool():
    """Generates the HTML for the Bokeh's hover data tool on our graph."""
    hover_html = """
      <div>
        <span class="hover-tooltip">$x</span>
      </div>
      <div>
        <span class="hover-tooltip">@bugs bugs</span>
      </div>
      <div>
        <span class="hover-tooltip">$@costs{0.00}</span>
      </div>
    """
    return HoverTool(tooltips=hover_html)

It may look really odd to have HTML embedded within your Python application, but that's how we specify what the hover tool should display. We use $x to show the bar's x axis, @bugs to show the "bugs" field from our data source, and $@costs{0.00} to show the "costs" field formatted as a dollar amount with exactly 2 decimal places.

Make sure you changed return None to return HoverTool(tooltips=hover_html) so we can see the results of our new function in the graph.

Head back to the browser and reload the localhost:5000/128/ page.

Nice work! Try playing around with the number of bars in the URL and the window size to see what the graph looks like under different conditions.

The chart gets crowded with more than 100 or so bars, but you can give it a try with whatever number of bars you want. Here is what an impractical amount of 50,000 bars looks like just for the heck of it:

Yea, we may need to do some additional work to display more than a few hundred bars at a time.

What's next?

You just created a nifty configurable bar chart in Bokeh. Next you can modify the color scheme, change the input data source, try to create other types of charts or solve how to display very large numbers of bars.

There is a lot more than Bokeh can do, so be sure to check out the official project documentation , GitHub repository, the Full Stack Python Bokeh page or take a look at other topics on Full Stack Python.

Questions? Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

See something wrong in this blog post? Fork this page's source on GitHub and submit a pull request.

How to Become A Successful Self-Taught Software Developer

Matt Makai — Sun, 14 May 2017 00:00:00 -0400

I received the following question via email from someone spending significant effort learning how to code in anticipation of obtaining full-time job with those skills. The question is also frequently asked by university students and coding bootcamp graduates.

This post provides my current answer on how get your first full-time job as a software developer. My answer assumes that the definition of "successful path" for a self-taught developer is getting a full-time position after investing so much time learning to code.

Note though that as I describe in my answer below, I took a more "traditional" route to become a professional software developer. Therefore my response should be only one of many that you solicit while working towards making the leap from self-taught to professional software developer.

(Paraphrased) Original Question

I'm not sure what I should learn first to become a developer.

Right now the path I am on is/was: Learn basic python fundamentals -> git/github -> ubuntu/linux OS--> flask/jinja2 --> Bootstrap -> SQLalchemy -> Docker -> Celery -> Redis -> AWS -> Django?!

I don't know where JS / Angular2 / ECMAscript6 / HTML / CSS all fit into this...

What is the ideal path to becoming a successful self-taught developer so I can eventually get a job as a software developer?"

My tl;dr answer

Go very deep in one area you really enjoy working after you learn the fundamentals and get a broad overview of the language's ecosystem. Leverage your depth in your targeted expertise area when you find teams that need that skill to land your first full-time job.

Answer Context

Figuring out what order to go in when learning is definitely one of the trickiest problems for self-guided learners. I'm not sure my answer to your question is the best one that you can get because for better or worse I took four years of computer science (CS) in high school, followed by undergrad CS & grad school CS (while working as a full-time developer). That route seems like the "traditional developer" background. However, I will do my best to give an answer. You are definitely not the only person who faces this issue.

I typically see self-taught and developer bootcamp grads feel like to get a job they have to learn everything from the database backend up through the web frameworks to every new JavaScript frontend framework that comes out, but that's definitely not true. When you land that first full-time developer gig it will be because a development team sees you have a particular skill that their team lacks and they need help with on their project(s).

Going deep

If you find yourself coding front-end stuff but wishing you could get back to optimizing the database, you should focus on going much, much deeper in database optimization. Learn as much as you can about SQL, DDLs, DMLs, ORMs, PostgreSQL, database testing and performance tuning. Constantly go deeper. Spend most of your time coding but when possible also teach others what you're learning. Some folks prefer to teach by writing blog posts. Other people enjoy giving tutorials at a meetup. You also mentor others in-person or remote on video chats who are also new to software development.

By teaching others you are not being purely altruistic: explaining programming and answering others' questions will reinforce in your own mind what you have learned and where your gaps remain based on the questions. Experiment with code to learn more and continue to go deeper. Create a feedback loop where you code, learn, write and find new unexplored veins to learn more in that area.

You should be ready to start job hunting once you have a good feedback loop where you are digging into your favorite subject area and are teaching it to others in some way.

Job searching

When you've gone deep in your subject, search for jobs that have a bit of a full-stack flavor with an emphasis on your specialty. Reach out via email to developers on the team or the hiring managers. Ask them for advice on what skills successful developers on their teams have an what unsuccessful candidates were lacking for their positions. Use their answers as data points for what you may still need to learn when their responses are relevant to the area you're going deep in.

When you feel you are ready, see if you can grab lunch or video chat with developers on those teams to learn more about their work. If that goes well, ask them if they'd refer you into the interview queue. Referrals will get you much further than applying through a human resources resume collection system.

Look for both software development junior roles and technical support roles, if the technical support roles are at software-focused companies. For example, Twilio's Support team often hires folks who have limited development experience but over time they can learn how to debug coding issues based on all the support tickets they have to answer (along with continued self-paced learning).

Interviewing and working tips

Enthusiasm is crucial for obtaining and doing well in your first few jobs. In hindsight, a lot of the enterprise software I worked on right out of college was horrible, but it was all new to me so I soaked up as much knowledge as possible while asking the tech leads and architects around me a ton of questions. Enjoy climbing steep learning curves.

Keep your cynicism and any "I'm better than this" attitude in check because companies have a ton of unexciting grunt work that needs to get done. The grunt work will teach you how to become a better software developer.

While looking for your first position, always be working on dozens of potential opportunities and do not pin your hopes up on one specific job. The goal is to get your first development gig that will help you continue to learn, not to land your dream job. The dream job comes later when you actually have enough experience to know what your dream job looks like!

You will eventually land your first development gig. Then you will have to constantly keep learning and the great part is that you'll get paid for it.

Feedback

What other questions can I answer and what additional topics can I add to Full Stack Python that would be immensely helpful to new folks who are struggling to become self-taught developers?

Let me know via a GitHub issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai.

How should I improve this blog post? Fork this page's source on GitHub and submit a pull request.

How to Create Your First Python 3.6 AWS Lambda Function

Matt Makai — Wed, 25 Apr 2018 00:00:00 -0400

Amazon Web Services (AWS) Lambda provides a usage-based compute service for running Python code in response to developer-defined events. For example, if an inbound HTTP POST comes in to API Gateway or a new file is uploaded to AWS S3 then AWS Lambda can execute a function to respond to that API call or manipulate the file on S3.

AWS Lambdas are not related to the Python languages' lambda expressions, which are used to create anonymous functions. The AWS Lambda name just happens to collide with the the lambda keyword's name.

Let's learn how to quickly write and run a Lambda function to execute basic Python 3.6 code which uses environment variables as input. This code, which is also available on GitHub under the blog-post-examples repository can be changed so that you can build much more complicated Python programs.

Our Tools

No local development environment tools are required for this tutorial, other than a web browser. All the work will happen on AWS via their Console.

These steps can also be completed from the command line via the boto3 library, but we won't cover that in this post.

If using Python 2 is still your jam rather than Python 3, take a look at this other post which shows how to execute Python 2.7 code on AWS Lambda.

First Steps with AWS Lambda

After signing up a few tutorials may pop up, but skip past them and go to the main Console. AWS has tons of services, with more being added every month, so using the search box is the best way to get around. Select the search text box, enter "lambda" and select "Lambda" to get to the Lambda starting page.

Click the "Create a Lambda function" button. The "Select Blueprint" page will appear.

Select "Blank Function" and the "Configure triggers" page will come up. It was non-obvious to me at first, but you don't actually need to configure a trigger to move on. A trigger is how the Lambda function typically knows when to execute based on an event from another AWS service such as API Gateway or Cloudwatch.

We won't configure a trigger for this function because we can manually kick off the Lambda to test it when we are finished configuring it. Leave the trigger icon blank and click the "Next" button to move along.

Next we get to the "Configure function" screen where we can finally write some code!

Python Code for Our Lambda Function

Enter a name for the Lambda function, such as "python_3_6_lambda_test", as well as a description. A description is optional but it is useful when you have a dozens or hundreds of different Lambda functions and need to keep them straight. In the Runtime drop-down, select Python 3.6 for the programming language.

Beneath the Runtime drop-down there is a large text box for code, prepopulated with a lambda_handler function definition. The "Code entry type" drop-down can also be changed to allow uploading a ZIP file or inputing a file from an S3 bucket. For our simple first Lambda function we will stick to the "Edit code inline" option. Copy or type in the following code, replacing what is already in the text box. This code is also available on this open source GitHub repository.

import os


def lambda_handler(event, context):
    what_to_print = os.environ.get("what_to_print")
    how_many_times = int(os.environ.get("how_many_times"))

    # make sure what_to_print and how_many_times values exist
    if what_to_print and how_many_times > 0:
        for i in range(0, how_many_times):
            # formatted string literals are new in Python 3.6
            print(f"what_to_print: {what_to_print}.")
        return what_to_print
    return None

The code above contains a required lambda_handler function, which is AWS Lambda's defined hook so it knows where to begin execution. Think of lambda_handler as a main function, like the
if __name__ == "__main__": conditional line commonly used in Python files to ensure a block of code is executed when a script is run from the command line.

The Python code expects two environment variables that are read by the os module with the environ.get function. With the what_to_print and how_many_times variables set by the environment variables, our code then prints a message zero or more times, based on the amount defined in the how_many_times variable. If a message is printed at least once then the function returns the what_to_print string, if nothing is printed then None is returned.

Below the code input text box on this function configuration screen there is a section to set environment variable key-value pairs.

Enter the keys named what_to_print and how_many_times then enter their values. Use a string message for what_to_print's value and an integer whole number above 0 for how_many_times. Our Python code's error handling is not very robust so a value other than a number in the how_many_times variable will cause the script to throw an error when it is executed due to the forced casting of how_many_times via the int() function.

The Python 3.6 code and the environment variables are now in place. We just need to handle a few more AWS-specific settings before we can test the Lambda function.

Executing our Lambda Function

Scroll past the environment variables to the "Lambda function handler and role" section, which contains a few more required function configuration items.

Keep the default handler set to lambda_function.lambda_handler. Select "Create a new Role from template(s)" from the drop-down then for the "Role name" field enter "dynamodb_access". Under "Policy templates" select the "Simple Microservice permissions".

The "Simple Microservice permissions" allows our Lambda to access AWS DynamoDB. We will not use DynamoDB in this tutorial but the service is commonly used either as permanent or temporary storage for Lambda functions.

Our code and configuration is in place so click the "Next" button at the bottom right corner of the page.

The review screen shows us our configuration settings to make sure we selected the appropriate values for our new Lambda function. Scroll down press "Create function".

Success message should appear on the next page below the "Test" button.

Click the "Test" button to execute the Lambda. Lambda will prompt us for some data to simulate an event that would kick off our function. Select the "Hello World" sample event template, which contains some keys but our Lambda will not use that in its execution. Click the "Save and test" button at the bottom of the modal.

Scroll down to the "Execution result" section where we can see our output.

The log output shows us the return value of our function, which in this execution was the string message from what_to_print. We can also see our print function produced output five times as expected based on the amount set in the how_many_times environment variable.

Next Steps

You just configured, coded and executed your first Python 3.6 AWS Lambda function! The real power of Lambda comes in when you use triggers to your Lambda function so it executes based on events that happen. We will take a look at that in the next tutorial.

View the AWS Lambda Full Stack Python page for additional examples and tutorials that other folks have shared for Lambda with Python.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Getting Started with AWS Lambda & Python 2.7

Matt Makai — Tue, 30 Mar 2021 00:00:00 -0400

Amazon Web Services (AWS) Lambda is a "serverless" compute service that executes arbitrary Python code in response to developer-defined events, such as inbound API calls or file uploads to AWS S3. Note that AWS Lambda has nothing to do with the lambda keyword in Python that is used to create anonymous functions, it's just the product name that happens to collide with an existing Python language feature name.

In this tutorial we'll learn how to quickly write and run a Lambda function that executes some simple Python 2.7 code and handles environment variables. The code can then be modified to build far more complicated Python applications.

Note: AWS ended support for Python 2.7 Lambda functions in 2021 and Python 2.7 no longer receives support so you should really be using Python 3.8 or above.

Tools We Need

We do not need any local development environment tools to get through this walkthrough other than a web browser because all the work will happen on AWS.

Grab a new free tier Amazon Web Services account or use your existing AWS account.

First Steps with Lambda

Head to the AWS Lambda landing page in your web browser. Sign into your account, or sign up for a new account which comes with a free tier so you don't have to pay.

If you're not taken directly to the Lambda Console page after logging in you'll see the main Console. AWS has a ridiculous number of services (that seems to expand every week) so the best way to get around is to select the search text box and search for "lambda" as shown in the following screenshot.

Press the "Create a Lambda function" button and you'll see the "Select Blueprint" page.

Choose "Blank Function". The next screen gives the option to select a "trigger", which is how the Lambda function gets executed. A trigger is some event that is integrated with other AWS services and can be exposed externally via an API or device such as Alexa.

However, we aren't going to set up a trigger for this function because we can manually test the Lambda later before connecting it to a trigger. Leave the trigger icon blank and click the "Next" button to move along to the next screen.

Now we're on the screen where we can enter our specific configuration and code for our new Lambda.

Writing Our Python Code

Start by entering a name for your Lambda function, such as "my_first_python_lambda" and a description. The description field is optional but it's handy when you start using Lambda regularly to keep all your functions straight. In the Runtime drop-down, select Python 2.7 as the execution language.

Below the Runtime drop-down you'll see a large text box for writing code. We can also choose to upload a ZIP file with our Python application which is handy for more than simple test Lambdas. However, for our simple starter Lambda application you can copy or type in the following code (or copy it from this GitHub repo). Make sure to replace what's already in the text box.

import os


def lambda_handler(event, context):
    what_to_print = os.environ.get("what_to_print")
    how_many_times = int(os.environ.get("how_many_times"))

    # make sure what_to_print and how_many_times values exist
    if what_to_print and how_many_times > 0:
        for i in range(0, how_many_times):
            print(what_to_print)
        return what_to_print
    return None

The above code has the required lambda_handler function definition that provides a hook for the Lambda service to know where to begin executing the Python code. Think of lambda_handler as a main function when you're using this service.

Our Python code expects and reads two environment variables and then the code prints a message zero to many times, based on the amount defined in the how_many_times variable. If a message is printed then the function returns the what_to_print string, if nothing is printed then None is returned.

Just below the code input text box there are environment variable key-value pairs that can be set. Our code will use two environment variables, named what_to_print and how_many_times.

Our code and environment variables are in place and we just need to set a few more AWS-specific settings before we can test the Lambda function.

Executing the Lambda

Scroll down below the environment variables to the "Lambda function handler and role" section. This section contains the last few required configuration items. Keep the default handler, which should be lambda_function.lambda_handler. Select "Create a new Role from template(s)" from the drop-down then for the "Role name" field enter "dynamodb_permissions". Under "Policy templates" select the "Simple Microservice permissions".

The "Simple Microservice permissions" gives our Lambda access to AWS DynamoDB. We won't use DynamoDB in this tutorial but it's super useful as either permanent or temporary storage when working with Lambda.

Now that our code and configuration is in place, click the "Next" button at the bottom right corner of the page.

The review screen will show us our configuration settings. Scroll down to the bottom and click the "Create function" button to continue.

We should see a success message on the next page just below the "Save and test" button.

Press the "Test" button to execute the Lambda. Lambda prompts us for some data to simulate an event that would trigger our function. Select the "Hello World" sample event template, which contains some example keys. Our Lambda will not those keys in its execution so it does not matter what they are. Click the "Save and test" button at the bottom of the modal.

Scroll down to the "Execution result" section where we can see our output.

We get the log output that shows us the return value of our function. In this case it is the string message from what_to_print. We can also see down below that our print function produced output five times.

What's Next?

Awesome, you just configured, wrote and executed your first Python 2.7 code on AWS Lambda! The real power of Lambda comes in when you connect a trigger to it so your code executes based on events. We'll take a look at that in the next tutorial.

What else can you do with Python and Lambda? Take a look at the AWS Lambda page for more examples and tutorials.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I am also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub.

How to Choose the Right DevOps Tools for You and Your Team

Matt Makai — Wed, 01 Mar 2017 00:00:00 -0500

This blog post contains a loose transcript along with the slides and additional resources from my technical talk that will be given at DC Continuous Delivery within the next couple of months.

Additional resources to learn more about deployments, configuration management and DevOps are listed at the end of the post.

Hey folks, my name is Matt Makai. I'm a Developer Evangelist with Twilio and the creator of Full Stack Python.

Over the past couple of years I've been coding mostly in Python and Swift. I bring that up because the way we build, deploy and operate applications in either ecosystem is different. It would not make sense to forcefully recommend a single way to work in your own ecosystem if it is different than the ones I work in.

I used to do a ton of Java development. That's how I started my professional career before I moved mostly into Python and Swift.

Back in my own software development dark ages of 2004, I learned about a concept that got me interested in DevOps before it was called DevOps: source control, also known as version control. We don't talk much about source control being a DevOps tool, but it really is the foundational layer for everything you want to automate with code.

Nowadays we have amazing open source distributed version control systems and beautiful web application front ends to visualize our code changes over time. Yet there is still a small percentage of developers who don't use source control.

It might seem crazy but I know developers at Fortune 500 companies that still do not use source control! How do you automate building, deploying, testing and operating your application if you don't even have your files versioned?

Let's start off this discussion of DevOps tools with a pat on your own back if you already use source control. Nice work! We've come a long way as an industry in the last couple of decades when source control was an exotic concept for most developers.

...

10 scale, how amenable is your organization to improving the technical environment?">

...

My name is Matt Makai and I'm a Developer Evangelist with Twilio, a Python and Swift developer, as well as the author of Full Stack Python. You can get in touch with me via these channels. Thank you!

Creating SSH Keys on macOS Sierra

Matt Makai — Fri, 28 Apr 2017 00:00:00 -0400

Deploying Python applications typically requires SSH keys. An SSH key has both a public and a private key file. You can use the private key to authenticate when syncing remote Git repositories, connect to remote servers and automate your application's deployments via configuration management tools like Ansible. Let's learn how to generate SSH key pairs on macOS Sierra.

Generating New Keys

Bring up a new terminal window on macOS by going into Applications/Utilities and opening "Terminal".

The ssh-keygen command provides an interactive command line interface for generating both the public and private keys. Invoke ssh-keygen with the following -t and -b arguments to ensure we get a 4096 bit RSA key. Note that you must use a key with 2048 or more bits in macOS Sierra or the system will not allow you to connect to servers with it.

Optionally, you can also specify your email address with -C (otherwise one will be generated off your current macOS account):

ssh-keygen -t rsa -b 4096 -C my.email.address@company.com

The first prompt you will see asks where to save the key. However, there are actually two files that will be generated: the public key and the private key.

Generating public/private rsa key pair.
Enter file in which to save the key (/Users/matt/.ssh/id_rsa):

This prompt refers to the private key and whatever you enter will also generate a second file for the public key that has the same name and .pub appended.

If you already have a key then specify a new filename. I use many SSH keys so I oftne name them "test-deploy", "prod-deploy", "ci-server" along with a unique project name. Naming is one of those hard computer science problems, so take some time to come up with a system that works for you!

Next you will see a prompt for an optional passphrase:

Enter passphrase (empty for no passphrase):

Whether or not you want a passphrase depends on how you will use the key. The system will ask you for the passphrase whenever you use the SSH key, although macOS can store the passphrase in your system Keychain after the first time you enter it. However, if you are automating deployments with a continuous integration server like Jenkins then you will not want a passphrase.

Note that it is impossible to recover a passphrase if it is lost. Keep that passphrase safe and secure because otherwise a completely new key would have to be generated.

Enter the passphrase (or just press enter to not have a passphrase) twice. You'll see some output like the following:

Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/matt/.ssh/deploy_prod.
Your public key has been saved in /Users/matt/.ssh/deploy_prod.pub.
The key fingerprint is:
SHA256:UnRGH/nzYzxUFS9jjd0wOl1ScFGKgW3pU60sSxGnyHo matthew.makai@gmail.com
The key's randomart image is:
+---[RSA 4096]----+
|        ..+o++**@|
|       . +.o*O.@=|
|        . oo*=B.*|
|       . .  =o=+ |
|      . S E. +oo |
|       . .  .  =.|
|              . o|
|                 |
|                 |
+----[SHA256]-----+

Your SSH key is ready to use!

What now?

Now that you have your public and private keys, I recommend building and deploying some Python web apps such as:

Building your first Slack bot
Sending picture or video messages via a REST API
Dialing outbound phone calls with the Bottle web framework

Additional ssh-keygen command resources:

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Creating SSH Keys on Ubuntu Linux 16.04 LTS

Matt Makai — Fri, 28 Apr 2017 00:00:00 -0400

SSH keys are a necessity for Python development when you are working with Git, connecting to remote servers and automating your deployments. Let's walk through how to generate SSH key pairs, which contain both a public and a private key within a single pair, on Ubuntu Linux.

Generating the Public and Private Keys

Open up a new terminal window in Ubuntu like we see in the following screenshot.

The ssh-keygen command provides an interactive command line interface for generating both the public and private keys. Invoke ssh-keygen with the following -t and -b arguments to ensure we get a 4096 bit RSA key. Optionally, you can also specify your email address with -C (otherwise one will be generated off your current Linux account):

ssh-keygen -o -t rsa -b 4096 -C my.email.address@company.com

(Note: the -o option was introduced in 2014; if this command fails for you, simply remove the -o option)

The first prompt you will see asks where to save the key. However, there are actually two files that will be generated: the public key and the private key.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/matt/.ssh/id_rsa):

This prompt refers to the private key and whatever you enter will also generate a second file for the public key that has the same name and .pub appended.

If you already have a key, you should specify a new filename. I use many SSH keys so I typically name them "test-deploy", "prod-deploy", "ci-server" along with a unique project name. Naming is one of those hard computer science problems, so take some time to come up with a system that works for you and the development team you work with!

Next you will see a prompt for an optional passphrase:

Enter passphrase (empty for no passphrase):

Whether or not you want a passphrase depends on how you will use the key. The system will ask you for the passphrase whenever you use the SSH key so it is more secure. However, if you are automating deployments with a continuous integration server like Jenkins then you will not want a passphrase.

Be aware that it is impossible to recover a passphrase if it is lost. Keep that passphrase safe and secure because otherwise a completely new key would have to be generated.

Enter the passphrase (or just press enter to not have a passphrase) twice. You'll see some output like the following:

Your identification has been saved in /home/matt/.ssh/prod_deploy.
Your public key has been saved in /home/matt/.ssh/prod_deploy.pub.
The key fingerprint is:
SHA256:xoCWgk40XfM5mruZQNCVoBKXZ4d0gn09ivVENacb7xw matt@ubuntu
The key's randomart image is:
+---[RSA 2048]----+
|.oo*==oo..o .    |
|.+*.*** =  +     |
|o+.++=.B .o      |
|+ .o. +oo  +     |
| . . o  S . E    |
|  .   ..   o .   |
|   . .      o    |
|    . +          |
|     +           |
+----[SHA256]-----+

Your SSH key is now generated and ready to use!

What now?

Now that you have your public and private keys, I recommend setting up a Python development environment with one of the following tutorials so you can start coding:

Additional ssh-keygen command resources:

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

How to Make Phone Calls in Python

Matt Makai — Sat, 22 Jul 2017 00:00:00 -0400

Good old-fashioned phone calls remain one of the best forms of communication despite the slew of new smartphone apps that have popped up over the past several years. With just a few lines of Python code plus a web application programming interface we can make and receive phone calls from any application.

Our example calls will say a snippet of text and put all incoming callers into a recorded conference call. You can modify the instructions using Twilio's TwiML verbs when you perform different actions in your own application's phone calls.

Our Tools

You should have either Python 2 or 3 installed to build this application. Throughout the post we will also use:

pip and virtualenv to handle application dependencies
A free Twilio account to use their phone calling web API
Twilio's Python helper library, version 5.7.0, which is available on PyPI

You can snag all the open source code for this tutorial in the python-twilio-example-apps GitHub repository under the no-framework/phone-calls directory. Use and copy the code for your own applications. Everything in that repository and in this blog post are open source under the MIT license.

Install App Dependencies

Our application will use the Twilio Python helper library to create an HTTP POST request to Twilio's API. The Twilio helper library is installable from PyPI into a virtual environment. Open your terminal and use the virtualenv command to create a new virtualenv:

virtualenv phoneapp

Invoke the activate script within the virtualenv bin/ directory to make this virtualenv the active Python executable. Note that you will need to perform this step in every terminal window that you want the virtualenv to be active.

source phoneapp/bin/activate

The command prompt will change after activating the virtualenv to something like (phoneapp) $.

Next use the pip command to install the Twilio Python package into the virtualenv.

pip install twilio==5.7.0

We will have the required dependency ready for project as soon as the installation script finishes. Now we can write and execute Python code to dial phone numbers.

Our Python Script

Create a new file named phone_calls.py and copy or type in the following lines of code.

from twilio.rest import TwilioRestClient


# Twilio phone number goes here. Grab one at https://twilio.com/try-twilio
# and use the E.164 format, for example: "+12025551234"
TWILIO_PHONE_NUMBER = ""

# list of one or more phone numbers to dial, in "+19732644210" format
DIAL_NUMBERS = ["",]

# URL location of TwiML instructions for how to handle the phone call
TWIML_INSTRUCTIONS_URL = \
  "http://static.fullstackpython.com/phone-calls-python.xml"

# replace the placeholder values with your Account SID and Auth Token
# found on the Twilio Console: https://www.twilio.com/console
client = TwilioRestClient("ACxxxxxxxxxx", "yyyyyyyyyy")


def dial_numbers(numbers_list):
    """Dials one or more phone numbers from a Twilio phone number."""
    for number in numbers_list:
        print("Dialing " + number)
        # set the method to "GET" from default POST because Amazon S3 only
        # serves GET requests on files. Typically POST would be used for apps
        client.calls.create(to=number, from_=TWILIO_PHONE_NUMBER,
                            url=TWIML_INSTRUCTIONS_URL, method="GET")


if __name__ == "__main__":
    dial_numbers(DIAL_NUMBERS)

There are a few lines that you need to modify in this application before it will run. First, insert one or more phone numbers you wish to dial into the DIAL_NUMBERS list. Each one should be a string, separated by a comma. For example, DIAL_NUMBERS = ["+12025551234", "+14155559876", "+19735551234"].

Next, TWILIO_PHONE_NUMBER and the Account SID and Authentication Token, found on the client = TwilioRestClient("ACxxxxxxxxxx", "yyyyyyyyyy") line, need to be set. We can get these values from the Twilio Console.

In your web browser go to the Twilio website and sign up for a free account or sign into your existing Twilio account.

Copy the Account SID and Auth Token from the Twilio Console and paste them into your application's code:

The Twilio trial account allows you to dial and receive phone calls to your own validated phone number. To handle calls from any phone number then you need to upgrade your account (hit the upgrade button on the top navigation bar).

Once you are signed into your Twilio account, go to the manage phone numbers screen. On this screen you can buy one or more phone numbers or click on an existing phone number in your account to configure it.

After clicking on a number you will reach the phone number configuration screen. Paste in the URL with TwiML instructions and change the dropdown from "HTTP POST" to "HTTP GET". In this post we'll use http://static.fullstackpython.com/phone-calls-python.xml, but that URL can be more than just a static XML file.

The power of Twilio really comes in when that URL is handled by your web application so it can dynamically respond with TwiML instructions based on the incoming caller number or other properties stored in your database.

Under the Voice webhook, paste in http://static.fullstackpython.com/phone-calls-python.xml and change the drop-down to the right from "HTTP POST" to "HTTP GET". Click the "Save" button at the bottom of the screen.

Now try calling your phone number. You should hear the snippet of text read by the Alice voice and then you will be placed into a conference call. If no one else calls the number then hold music should be playing.

Making Phone Calls

We just handled inbound phone calls to our phone number. Now it's time to dial outbound phone calls. Make sure your phone_calls.py file is saved and that your virtualenv is still activated and then execute the script:

python phone_calls.py

In a moment all the phone numbers you write in the DIAL_NUMBERS list should light up with calls. Anyone that answers will hear our message read by the "Alice" voice and then they'll be placed together into a recorded conference call, just like when someone dials into the number.

Here is my inbound phone call:

Not bad for just a few lines of Python code!

Next Steps

Now that we know how to make and receive phone calls from a Twilio number that follows programmatic instructions we can do a whole lot more in our applications. Next you can use one of these tutorials to do more with your phone number:

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub as mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Dialing Outbound Phone Calls with a Bottle Web App

Matt Makai — Fri, 16 Jun 2017 00:00:00 -0400

Python web apps built with the Bottle web framework can send and receive SMS text messages. In this tutorial we will go beyond texting and learn how to dial outbound phone calls. The calls will read a snippet of text then play an MP3 file, but they can then be easily modified to create conference lines and many other voice features in your Python web apps.

Tools We Need

You should have either Python 2 or 3 installed to create your Bottle app, although Python 3 is recommended for new applications. We also need:

pip and virtualenv to handle application dependencies
Ngrok for localhost tunneling to our Bottle application while it's running on our local development environment
Bottle web framework
Free Twilio account to use their phone calling web API
Twilio's Python helper library, which is open source on GitHub and available for download from PyPI

Take a look at this guide on setting up Python 3, Bottle and Gunicorn on Ubuntu 16.04 LTS if you need help getting your development environment configured before continuing on through the remainder of this tutorial.

You can snag all the open source code for this tutorial in the python-bottle-phone GitHub repository under the outbound directory. Use and copy the code however you want - it's all open source under the MIT license.

Installing Our Application Dependencies

Our Bottle app needs a helper code library to make it easy to dial outbound phone calls. Bottle and the Twilio helper library are installable from PyPI into a virtualenv. Open your terminal and use the virtualenv command to create a new virtualenv:

virtualenv bottlephone

Use the activate script within the virtualenv, which makes this virtualenv the active Python installation. Note that you need to do this in every terminal window that you want this virtualenv to be used.

source bottlephone/bin/activate

The command prompt will change after activating the virtualenv to something like (bottlephone) $. Here is a screenshot of what my environment looked like when I used the activate script.

Next use the pip command to install the Bottle and Twilio Python packages into your virtualenv.

pip install bottle twilio==5.7.0

After the installation script finishes, we will have the required dependencies to build our app. Time to write some Python code to dial outbound phone calls.

Bottle and Twilio

Our simple Bottle web app will have three routes:

/ - returns a text string to let us know our Bottle app is running
/twiml - responds with TwiML (a simple subset of XML) that instructs Twilio what to do when someone picks up the call to them from our Bottle web app
/dial-phone/<outbound_phone_number>, where "outbound_phone_number" is a phone number in the format "+12025551234" - this route uses the Twilio helper library to send a POST request to the Twilio Voice API to dial a phone call

We can build the structure of our Bottle app and the first route right now. Create a new file named app.py with the following contents to start our app.

import os
import bottle
from bottle import route, run, post, Response
from twilio import twiml
from twilio.rest import TwilioRestClient


app = bottle.default_app()
# plug in account SID and auth token here if they are not already exposed as
# environment variables
twilio_client = TwilioRestClient()

TWILIO_NUMBER = os.environ.get('TWILIO_NUMBER', '+12025551234')
NGROK_BASE_URL = os.environ.get('NGROK_BASE_URL', 'https://c6c6d4e8.ngrok.io')


@route('/')
def index():
    """
    Returns a standard text response to show the app is up and running.
    """
    return Response("Bottle app running!")


if __name__ == '__main__':
    run(host='127.0.0.1', port=8000, debug=False, reloader=True)

Make sure you are in the directory where you created the above app.py file. Run the app via the Bottle development server with the following command. Make sure your virtualenv is still activated so our code can rely on the Bottle code library.

python app.py

We should see a successful development server start up like this:

(bottlephone) matt@ubuntu:~/bottlephone$ python app.py 
Bottle v0.12.9 server starting up (using WSGIRefServer())...
Listening on http://127.0.0.1:8000/
Hit Ctrl-C to quit.

Here is what the development server message looks like in my environment on Ubuntu:

Let's test out the app by going to "localhost:8000" in the web browser. We should get a simple success message that the app is running and responding to requests.

Next we need to obtain a phone number that our Bottle app can use to call other phone numbers.

Obtain a Phone Number

Our basic Bottle web app is running but what we really want to do is dial outbound calls - which will be handled by Twilio.

In your web browser go to the Twilio website and sign up for a free account. You can also sign into your existing Twilio account if you already have one.

The Twilio trial account allows you to dial and receive phone calls to your own validated phone number. To dial and receive calls from any phone number then you need to upgrade your account (hit the upgrade button on the top navigation bar to do that). Trial accounts are great for initial development before your application goes live but upgraded accounts are where the real power comes in.

There is nothing for us to configure right now on the phone number configuration page because we are making outbound phone calls for this tutorial. Now that we have a phone number in hand, let's add the final bit of code to our Bottle app to get this app working.

Making Phone Calls

We need to add two new routes to our Bottle app so it can dial outbound phone calls. Modify your existing app.py file with the two new functions below, twiml_response and outbound_call. None of the other code in this file needs to change other than adding those two new functions to what we wrote in the previous section.

import os
import bottle
from bottle import route, run, post, Response
from twilio import twiml
from twilio.rest import TwilioRestClient


app = bottle.default_app()
# plug in account SID and auth token here if they are not already exposed as
# environment variables
twilio_client = TwilioRestClient()

# add your Twilio phone number here
TWILIO_NUMBER = os.environ.get('TWILIO_NUMBER', '+16093002984')
# plug in your Ngrok Forwarding URL - we'll set it up in a minute
NGROK_BASE_URL = os.environ.get('NGROK_BASE_URL', 'https://c6c6d4e8.ngrok.io')


@route('/')
def index():
    """
    Returns a standard text response to show the app is up and running.
    """
    return Response("Bottle app running!")


@post('/twiml')
def twiml_response():
    """
    Provides TwiML instructions in response to a Twilio POST webhook
    event so that Twilio knows how to handle the outbound phone call
    when someone picks up the phone.
    """
    response = twiml.Response()
    response.say("Sweet, this phone call is answered by your Bottle app!")
    response.play("https://api.twilio.com/cowbell.mp3", loop=10)
    return Response(str(response))


@route('/dial-phone/<outbound_phone_number>')
def outbound_call(outbound_phone_number):
    """
    Uses the Twilio Python helper library to send a POST request to
    Twilio telling it to dial an outbound phone call from our specific
    Twilio phone number (that phone number must be owned by our Twilio 
    account).
    """
    # the url must match the Ngrok Forwarding URL plus the route defined in
    # the previous function that responds with TwiML instructions
    twilio_client.calls.create(to=outbound_phone_number, 
                               from_=BLOG_POST_NUMBER,
                               url=NGROK_BASE_URL + '/twiml')
    return Response('phone call placed to ' + outbound_phone_number + '!')


if __name__ == '__main__':
    run(host='127.0.0.1', port=8000, debug=False, reloader=True)

There is just one problem with our current setup if you're developing on a local environment: Twilio won't be able to reach that /twiml route. We need to deploy our app to a reachable server, or just use a localhost tunneling tool like Ngrok. Ngrok provides an external URL that connects to a port running on your machine. Download and install the Ngrok application that is appropriate for your operating system.

We run Ngrok locally and expose our Bottle app that is running on port 8000. Run this command within the directory where the Ngrok executable is located.

./ngrok http 8000

Ngrok will start up and provide us with a Forwarding URL, with both HTTP and HTTPS versions.

We can use the Forwarding URL to instruct Twilio how to handle the outbound phone call when someone picks up. Insert the Ngrok forwarding URL into the app.py file where NGROK_BASE_URL is specified.

If Ngrok is useful to you, make sure to read this 6 awesome reasons to use Ngrok when testing webhooks post to learn even more about the tool.

Time to test out our app, let's give it a quick spin.

Making Phone Calls

Make sure your Bottle development server is still running or re-run it with the python app.py command in a shell where your virtualenv is still activated.

Bring up the application in a browser, this time test out the phone calling capabilities. Go to "localhost:8000/dial-phone/my-phone-number", where "my-phone-number" is a number in the "+12025551234" format. For example, here is what happens when I dialed +12023351278:

And here is the inbound phone call!

When we pick up the phone call we also see the /twiml route get called via Ngrok.

With just two routes in our Bottle app and Twilio we were able to make outbound phone calls. Not bad!

What's next?

Sweet, we can now dial outbound phone calls to any phone number from our Bottle web application. Next you may want to try one of these tutorials to add even more features to your app:

Upgrade your Bottle app to also send and respond to text messages
Create a phone-calling Slack bot
Implement call tracking for both inbound and outbound phone calls made through your app

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub as mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Python for Entrepreneurs

Matt Makai — Tue, 25 Apr 2017 00:00:00 -0400

Python for Entrepreneurs is a new video course by the creators of Talk Python to Me and Full Stack Python.

Update: The Kickstarter has been funded! Michael and I are hard at work on the course content. Thank you to everyone who supported us as a backer. The course is available in early access mode on training.talkpython.fm until it is fully released.

We are creating this course and running a Kickstarter for it based on feedback that it's still too damn difficult to turn basic Python programming knowledge into a business to generate income as a side or full time project. Both Michael and I have been able to make that happen for ourselves and we want to share every difficult lesson we've learned through this course.

The Python for Entrepreneurs videos and content will dive into building and deploying a real-world web application, marketing it to prospective customers, handling search engine optimization, making money through credit card payments, getting help from part-time contractors for niche tasks and scaling up to meet traffic demands.

If this course hits the mark for what you want to do with Python, check out the Kickstarter - we've set up steep discounts for early backers.

If you have any questions, please reach out to Michael Kennedy or me, Matt Makai.

Setting Up Python 3, Django & Gunicorn on Linux Mint 17.3

Matt Makai — Thu, 17 Jun 2021 00:00:00 -0400

Linux Mint 17.3 "Rosa" is December 2015 release of the polished and widely-used Linux distribution. This Mint release includes both Python 2.7 and 3.4 by default, but in this tutorial we will download and install the latest Python 3.5.1 version to run our Django application.

If you want to use a different Linux distribution such as Ubuntu instead of Mint, check out the tutorial for Ubuntu 16.04 "Xenial Xerus". If Mint is your desired development environment though, let's get started!

Tools We Need

Our setup will use several system packages and code libraries to get up and running. Do not worry about installing these dependencies just yet, we will get to them as we progress through the tutorial. The tools and their current versions as of June 2016 are:

Linux Mint 17.3 "Rosa" with the default Cinnamon desktop
Python version 3.5.1
Django web framework version 1.9.x
Green Unicorn (Gunicorn) version 19.6

If you are on Mac OS X or Windows, my recommendation is to use virtualization software such as Parallels or VirtualBox with the Linux Mint Cinnamon desktop .iso.

We should see a desktop screen like this one when we boot up the operating system for the first time.

Open up terminal to proceed with the configuration.

System Packages

We can see the Python version Linux Mint comes with, as well as where its executable is stored.

python3 --version
which python3

The output of those two commands should be (these are not commands to run):

Python 3.4.3
/usr/bin/python3

We really want to use the latest Python release instead of the default 3.4 when starting a new Python project, so let's download and install 3.5.1 now.

Run these commands in the terminal to download Python 3.5.1 source code:

cd ~/Downloads
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz

Extract the Python source code:

tar -xvf Python-3.5.1.tgz

Linux Mint is not configured by default to build the Python source code. We need to update our system package lists and install several packages to make building the Python source code possible. If you have a password on your user account, enter it when prompted to allow the installation to proceed.

sudo apt update
sudo apt install build-essential checkinstall
sudo apt install libreadline-gplv2-dev libncursesw5-dev libssl-dev 
sudo apt install libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
sudo apt install python3-dev

Once the packages are installed, we can configure and install Python from source.

cd Python-3.5.1
./configure
sudo make install

Test that the installation worked properly by starting up the Python REPL:

python3.5

If the REPL starts up properly with Python 3.5.1 in the output then we're good to go.

The basic system packages we need are now installed so we can proceed to our Python-specific dependencies.

Virtual environment and pip

Python 3.5 comes with the virtual environment and pip applications so we can use them to handle our application dependencies.

Create a directory to store virtual environments then create a virtualenv for our Django project.

# the tilde "~" specifies the user's home directory, like /home/matt
cd ~
mkdir venvs
# specify the system python3 installation
python3.5 -m venv djangoproj

Activate the virtualenv.

source ~/venvs/djangoproj/bin/activate

Our virtual environment is properly activated when we see (djangoproj) prepended to our prompt.

Our virtualenv with Python 3.5.1 is activated so we can install whatever dependencies we want, such as Django and Gunicorn. Our default python command is also set to use the Python 3.5.1 installation instead of the Python 2.7 version that comes with Linux Mint.

Django and Gunicorn

Now we can install Django and Green Unicorn into our virtual environment.

pip install django==1.9.7 gunicorn==19.6

If there are no errors in the pip output then that is a good sign we can proceed.

Create a new Django project named djangoproj, or whatever you want to name your project. Change into the directory for the new project.

cd ~
django-admin startproject djangoproj
cd djangoproj

We can run Django using the development server with the python manage.py runserver command. However, start Django up with Gunicorn instead.

gunicorn djangoproj.wsgi

Awesome, we can bring up our shell project in the web browser at the http://localhost:8000 or http://127.0.0.1:8000 address.

Now you're ready for Django development!

Ready for Development

Those are the first few steps for beginning development with Django and Gunicorn on Linux Mint 17.3 "Rosa". If you need an even more in-depth walkthrough for deploying your Python web application to a production environment, check out the Full Stack Python Guide to Deployments book.

To figure out what to do next for your Python project, read the topics found on the table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Configuring Python 3, Pyramid and Gunicorn on Ubuntu 16.04

Matt Makai — Wed, 10 Aug 2016 00:00:00 -0400

Canonical's Ubuntu 16.04 Long Term Support (LTS) Linux operating system, also known as "Xenial Xerus", was released in April 2016. It is the first Ubuntu release to include Python 3 instead of Python 2 as its default Python installation.

The Pyramid web framework has long supported Python 3. With just a few short steps we can start a new Pyramid project and run it with Green Unicorn (Gunicorn) on Ubuntu 16.04.

Required Tools

Our project requires Ubuntu 16.04 plus several code libraries. You do not need to install these tools yet - we will get to them in turn as the walkthrough progresses. Our requirements and their current versions are:

Ubuntu 16.04 LTS (Xenial Xerus)
Python version 3.5 (default in Ubuntu 16.04)
Pyramid web framework version 1.7
Gunicorn version 19.6
Waitress version 0.9.0

If you are developing on Mac OS X or Windows, you can use virtualization software such as Parallels or VirtualBox with the Ubuntu .iso file. Either the amd64 or i386 version of 16.04 is fine. While creating this I used the amd64 version.

A desktop screen like this one appears when you boot up Ubuntu.

Open a new terminal window so we can be ready to install required system packages.

Install System Packages

The precise Python version can be shown using the python command with the --version argument.

python3 --version

We can also view where the python3 program is installed on Ubuntu using the which command.

which python3

Ubuntu requires a few system packages before we can properly install Pyramid and Gunicorn. When we run the apt command to install system packages we will be prompted for the superuser password. Restricted system access is necessary to modify files within the system folders.

sudo apt-get install python3-dev

Press y then return to let the system package installation run.

The required system packages are installed. We can now install the Python-specific dependencies.

Set up a virtual environment

Create a directory for the virtual environments. Then create a new virtual environment.

# the tilde "~" specifies the user's home directory, like /home/matt
cd ~
mkdir venvs
# specify the system python3 installation
/usr/bin/python3 -m venv venvs/pyramidproj

Activate the virtual environment.

source ~/venvs/pyramidproj/bin/activate

Our prompt will change after we properly activate the virtual environment to something like (pyramidproj) matt@ubuntu:~$.

Our virtual environment is activated with Python 3.

We should update pip and venv to the latest versions in our virtual environment.

pip install --upgrade pip setuptools

We can install whatever dependencies we want, in our case Pyramid and Gunicorn.

Install Python Packages

We can install Pyramid, Gunicorn and Waitress into our virtual environment using the pip command.

pip install pip install "pyramid==1.7" gunicorn waitress

No errors like we see in the following screenshot is a good sign.

Pyramid comes with a project starter template creation tool named pcreate. Run pcreate to generate the boilerplate for a new Pyramid project named "pyramidproj".

pcreate -s starter pyramidproj

Use the cd (change directory) command to move into the new folder.

cd ~/pyramidproj

A slew of new files have been created within the "pyramidproj" directory. These are the basic files you can customize for the web application you want to build. A good resource for understanding and modifying these files is to follow the quick tutorial for Pyramid.

For now, we just want to use Gunicorn to run our starter pyramidproj app. Install pyramidproj into your virtual environment using the python command on setup.py.

python setup.py develop

Now we can run our app with Gunicorn. Pyramid is a paste-compatible framework, so we use the --paste argument to run the WSGI server with the "development.ini" configuration file. In addition, the -b argument tells Gunicorn which port number to bind on when the server starts.

gunicorn --paste development.ini -b :8080

Cool, we can bring up our starter Pyramid project up in the web browser at the localhost:8000 or 127.0.0.1:8000 address.

Time to develop a full-fledged web application with Pyramid!

Ready to Develop with Pyramid

Now you have a simple setup to develop Pyramid web apps using Gunicorn as the WSGI server on Ubuntu 16.04. If you need a full step-by-step tutorial to deploy your Python web application to a production environment, check out the Full Stack Python Guide to Deployments book.

To decide what to do next with your Python project, check out the Full Stack Python table of contents page.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Replying to SMS Text Messages with Python and Bottle

Matt Makai — Wed, 10 Aug 2016 00:00:00 -0400

Python applications can easily send SMS by using a web API. Web apps built with the Bottle framework can also reply to incoming text messages by handling inbound HTTP POST webhooks. In this post we'll quickly walk through how to set up a Bottle web app to handle SMS data in the form of HTTP POST requests.

Tools We'll Need

This tutorial works with either Python 2 or 3, although Python 3 is recommended by the community for new applications. Install one of those two Python versions on your system to use for this walkthrough. We also need:

pip and virtualenv to handle application dependencies
Bottle web framework
Ngrok for localhost tunneling to our Bottle application while it's running on our local development environment
Free Twilio account to use their SMS web API
Open source Twilio Python helper library, version 5.7.0 or earlier

Check out the guide on how to set up Python 3, Bottle and Gunicorn on Ubuntu 16.04 LTS if you need help getting your development environment configured.

Application Dependency Installation

Our application will use a helper code library to reply to inbound SMS. Bottle and the helper library are installable from PyPI into a virtualenv. Open your terminal and use the virtualenv command to create a new virtualenv:

virtualenv replysms

Invoke the virtualenv's activate script, which makes it the "active" Python installation. Note that you need to do this in every terminal window that you want this virtualenv to be used.

source replysms/bin/activate

The command prompt will change after activating the virtualenv:

Use the pip command to install the Bottle and Twilio Python packages into your virtualenv.

pip install bottle twilio==5.7.0

We have installed the required dependencies so now Python code that is run with the virtualenv activated will be able to use those packages. It's time to build our Bottle web app and reply to incoming text messages.

Coding Our Bottle App

The Bottle web app will have two routes. One route will allow us to test that the app is running. The other route will handle and respond to incoming HTTP POST requests from Twilio. Create a new file named app.py in your in the directory where you want to store this Python project.

Write the following code in the new app.py file. There is also a GitHub Gist with the code that you can copy and paste.

from bottle import (post, request, response, route, run, )
from twilio import twiml


@route('/')
def check_app():
    # returns a simple string stating the app is working
    return "It works!"


@post('/twilio')
def inbound_sms():
    twiml_response = twiml.Response()
    # grab message from the request. could also get the "To" and 
    # "From" phone numbers as well from parameters with those names
    inbound_message = request.forms.get("Body")
    # we can now use the incoming message text in our Python application
    if inbound_message == "Hello":
        twiml_response.message("Hello from Bottle right back at you!")
    else:
        twiml_response.message("Hi! Not quite sure what you meant, but okay.")
    # we return back the mimetype because Twilio needs an XML response
    response.content_type = "application/xml"
    return str(twiml_response)


if __name__ == '__main__':
    run(host='127.0.0.1', port=5000, debug=True, reloader=True)

The lines starting with # are comments that give explanations for what the code lines below them are doing. Bottle web apps define URL routes with the @route and @post decorators, depending on the type of HTTP request the route should handle.

Make sure your virtualenv is still active so that the application can use the Bottle and Twilio code libraries we installed earlier. Give the application a try by running it with python app.py.

Open a web browser and go to localhost:5000 (or 127.0.0.1:5000). We should see "It works!" on the screen.

However, there is an issue with our web app running on our local development environment. Twilio cannot send a the HTTP POST request to the web app server unless a localhost tunnel is created.

Ngrok Localhost Tunneling

Ngrok is a localhost tunneling tool that bridges your local development environment to an external URL. Download and install the Ngrok version that's appropriate for your operating system.

We can run Ngrok locally and expose our Bottle app that is running on port 5000. Run this command within the directory where the Ngrok executable is located.

./ngrok http 5000

Cool, now we can use the Forwarding URL so Twilio can send POST requests to our application when there is an inbound SMS. Replace the URL in the text box with your own Forwarding URL, like I did in this screenshot.

Now we just need a Twilio phone number that will send POST request to our application when there is an inbound SMS.

Obtain a Phone Number

Our Bottle web app's route can respond to incoming POST requests but we need to use Twilio to have a phone number that will convert the inbound SMS data into the POST request. In your web browser go to the Twilio website and sign up for a free account. You can also sign into your existing Twilio account if you already have one.

The Twilio trial account allows you to send and receive text messages to your own validated phone number. To send and reply to SMS to and from any phone number then you need to upgrade your account. Trial accounts are great for initial development before your application goes live.

When you sign up, you receive a free Twilio phone number. We can configure that phone number to forward the SMS information to our web application by setting up the response webhook.

Go to the manage phone numbers screen and click on the phone number you want to configure for replying to text messages.

Scroll down and look for the "Messaging" header. Change the "A Message Comes in" text box to input the ngrok Forwarding URL plus the "/twilio" route, as shown in the screenshot below.

Click the "Save" button so that our changes take effect.

Our application is ready to go - time to give our phone number a try! Send "Hello" or whatever text you want to your phone number. Here is what the result looks like on my iPhone.

The concise Bottle web app is a good start to build more complicated programs such as Choose Your Own Adventure Presentations or SMS Slack bots.

What's next?

Awesome, our Bottle application now replies to inbound SMS text messages!

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

How to Build Your First Slack Bot with Python

Matt Makai — Wed, 13 Dec 2017 00:00:00 -0500

Bots are a useful way to interact with chat services such as Slack. If you have never built a bot before, this post provides an easy starter tutorial for combining the Slack API with Python to create your first bot.

We will walk through setting up your development environment, obtaining a Slack API bot token and coding our simple bot in Python.

Tools We Need

Our bot, which we will name "StarterBot", requires Python and the Slack API. To run our Python code we need:

Either Python 2 or 3
pip and virtualenv to handle Python application dependencies
Free Slack account - you need to be signed into at least one workspace where you have access to building apps.

It is also useful to have the Slack API docs handy while you're building this tutorial.

All the code for this tutorial is available open source under the MIT license in the slack-starterbot public repository.

Establishing Our Environment

We now know what tools we need for our project so let's get our development environment set up. Go to the terminal (or Command Prompt on Windows) and change into the directory where you want to store this project. Within that directory, create a new virtualenv to isolate our application dependencies from other Python projects.

virtualenv starterbot

Activate the virtualenv:

source starterbot/bin/activate

Your prompt should now look like the one in this screenshot.

The official slackclient API helper library built by Slack can send and receive messages from a Slack channel. Install the slackclient library with the pip command:

pip install slackclient==1.3.2

When pip is finished you should see output like this and you'll be back at the prompt.

We also need to create a Slack App to recieve an API token for your bot. Use "Starter Bot" as your App name. If you are signed into more than one workspace, pick a Development Workspace from the dropdown.

After submitting the form, keep the app configuration page open.

Slack APIs and App Configuration

We want our Starter Bot to appear like any other user in your team - it will participate in conversations inside channels, groups, and DMs. In a Slack App, this is called a bot user, which we set up by choosing "Bot Users" under the "Features" section. After clicking "Add a Bot User", you should choose a display name, choose a default username, and save your choices by clicking "Add Bot User". You'll end up with a page that looks like the following:

The slackclient library makes it simple to use Slack's RTM API and Web API. We'll use both to implement Starter Bot, and they each require authentication. Conveniently, the bot user we created earlier can be used to authenticate for both APIs.

Click on the "Install App" under the "Settings" section. The button on this page will install the App into our Development Workspace. Once the App is installed, it displays a bot user oauth access token for authentication as the bot user.

A common practice for Python developers is to export secret tokens as environment variables. Back in your terminal, export the Slack token with the name SLACK_BOT_TOKEN:

export SLACK_BOT_TOKEN='your bot user access token here'

Nice, now we are authorized to use the Slack RTM and Web APIs as a bot user.

Coding Our Starter Bot

We've got everything we need to write the Starter Bot code. Create a new file named starterbot.py and include the following code in it.

import os
import time
import re
from slackclient import SlackClient

With our dependencies imported we can use them to obtain the environment variable values and then instantiate the Slack client.

# instantiate Slack client
slack_client = SlackClient(os.environ.get('SLACK_BOT_TOKEN'))
# starterbot's user ID in Slack: value is assigned after the bot starts up
starterbot_id = None

# constants
RTM_READ_DELAY = 1 # 1 second delay between reading from RTM
EXAMPLE_COMMAND = "do"
MENTION_REGEX = "^<@(|[WU].+?)>(.*)"

The code instantiates the SlackClient client with our SLACK_BOT_TOKEN exported as an environment variable. It also declares a variable we can use to store the Slack user ID of our Starter Bot. A few constants are also declared, and each of them will be explained as they are used in the code that follows.

if __name__ == "__main__":
    if slack_client.rtm_connect(with_team_state=False):
        print("Starter Bot connected and running!")
        # Read bot's user ID by calling Web API method `auth.test`
        starterbot_id = slack_client.api_call("auth.test")["user_id"]
        while True:
            command, channel = parse_bot_commands(slack_client.rtm_read())
            if command:
                handle_command(command, channel)
            time.sleep(RTM_READ_DELAY)
    else:
        print("Connection failed. Exception traceback printed above.")

The Slack client connects to the Slack RTM API. Once it's connected, it calls a Web API method (auth.test) to find Starter Bot's user ID.

Each bot user has a user ID for each workspace the Slack App is installed within. Storing this user ID will help the program understand if someone has mentioned the bot in a message.

Next, the program enters an infinite loop, where each time the loop runs the client recieves any events that arrived from Slack's RTM API. Notice that before the loop ends, the program pauses for one second so that it doesn't loop too fast and waste your CPU time.

For each event that is read, the parse_bot_commands() function determines if the event contains a command for Starter Bot. If it does, then command will contain a value and the handle_command() function determines what to do with the command.

We've laid the groundwork for processing Slack events and calling Slack methods in the program. Next, add three new functions above the previous snippet to complete handling commands:

def parse_bot_commands(slack_events):
    """
        Parses a list of events coming from the Slack RTM API to find bot commands.
        If a bot command is found, this function returns a tuple of command and channel.
        If its not found, then this function returns None, None.
    """
    for event in slack_events:
        if event["type"] == "message" and not "subtype" in event:
            user_id, message = parse_direct_mention(event["text"])
            if user_id == starterbot_id:
                return message, event["channel"]
    return None, None

def parse_direct_mention(message_text):
    """
        Finds a direct mention (a mention that is at the beginning) in message text
        and returns the user ID which was mentioned. If there is no direct mention, returns None
    """
    matches = re.search(MENTION_REGEX, message_text)
    # the first group contains the username, the second group contains the remaining message
    return (matches.group(1), matches.group(2).strip()) if matches else (None, None)

def handle_command(command, channel):
    """
        Executes bot command if the command is known
    """
    # Default response is help text for the user
    default_response = "Not sure what you mean. Try *{}*.".format(EXAMPLE_COMMAND)

    # Finds and executes the given command, filling in response
    response = None
    # This is where you start to implement more commands!
    if command.startswith(EXAMPLE_COMMAND):
        response = "Sure...write some more code then I can do that!"

    # Sends the response back to the channel
    slack_client.api_call(
        "chat.postMessage",
        channel=channel,
        text=response or default_response
    )

The parse_bot_commands() function takes events from Slack and determines if they are commands directed at Starter Bot. There are many event types that our bot will encounter, but to find commands we only want to consider message events. Message events also have subtypes, but the commands we want to find won't have any subtype defined. The function filters out uninteresting events by checking these properties. Now we know the event represents a message with some text, but we want to find out if Starter Bot is being mentioned in the text. The parse_direct_mention() function will figure out of the message text starts with a mention, and then we compare that to the user ID we stored earlier for Starter Bot. If they are the same, then we know this is a bot command, and return the command text with the channel ID.

The parse_direct_mentions() function uses a regular expression to determine if a user is being mentioned at the beginning of the message. It returns the user ID and the remaining message (and None, None if no mention was found).

The last function, handle_command() is where in the future you'll add all the interesting commands, humor, and personality for Starter Bot. For now, it has just one example command: do. If the command starts with a known command, it will have an appropriate response. If not, a default response is used. The response is sent back to Slack by calling the chat.postMessage Web API method with the channel.

Here is how the entire program should look when it's all put together (you can also view the file in GitHub):

import os
import time
import re
from slackclient import SlackClient


# instantiate Slack client
slack_client = SlackClient(os.environ.get('SLACK_BOT_TOKEN'))
# starterbot's user ID in Slack: value is assigned after the bot starts up
starterbot_id = None

# constants
RTM_READ_DELAY = 1 # 1 second delay between reading from RTM
EXAMPLE_COMMAND = "do"
MENTION_REGEX = "^<@(|[WU].+?)>(.*)"

def parse_bot_commands(slack_events):
    """
        Parses a list of events coming from the Slack RTM API to find bot commands.
        If a bot command is found, this function returns a tuple of command and channel.
        If its not found, then this function returns None, None.
    """
    for event in slack_events:
        if event["type"] == "message" and not "subtype" in event:
            user_id, message = parse_direct_mention(event["text"])
            if user_id == starterbot_id:
                return message, event["channel"]
    return None, None

def parse_direct_mention(message_text):
    """
        Finds a direct mention (a mention that is at the beginning) in message text
        and returns the user ID which was mentioned. If there is no direct mention, returns None
    """
    matches = re.search(MENTION_REGEX, message_text)
    # the first group contains the username, the second group contains the remaining message
    return (matches.group(1), matches.group(2).strip()) if matches else (None, None)

def handle_command(command, channel):
    """
        Executes bot command if the command is known
    """
    # Default response is help text for the user
    default_response = "Not sure what you mean. Try *{}*.".format(EXAMPLE_COMMAND)

    # Finds and executes the given command, filling in response
    response = None
    # This is where you start to implement more commands!
    if command.startswith(EXAMPLE_COMMAND):
        response = "Sure...write some more code then I can do that!"

    # Sends the response back to the channel
    slack_client.api_call(
        "chat.postMessage",
        channel=channel,
        text=response or default_response
    )

if __name__ == "__main__":
    if slack_client.rtm_connect(with_team_state=False):
        print("Starter Bot connected and running!")
        # Read bot's user ID by calling Web API method `auth.test`
        starterbot_id = slack_client.api_call("auth.test")["user_id"]
        while True:
            command, channel = parse_bot_commands(slack_client.rtm_read())
            if command:
                handle_command(command, channel)
            time.sleep(RTM_READ_DELAY)
    else:
        print("Connection failed. Exception traceback printed above.")

Now that all of our code is in place we can run our Starter Bot on the command line with the python starterbot.py command.

In Slack, create a new channel and invite Starter Bot or invite it to an existing channel.

Now start giving Starter Bot commands in your channel.

Additional Note: Currently there's an issue with the websocket package and the CA certificate it uses, so if you encounter an error like:

...
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)
...
slackclient.server.SlackConnectionError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)
Connection failed. Exception traceback printed above.

There are a couple of things that can be done: 1. Downgrading the websocket-client library to 0.47.0 2. Or, download the certificate (wget https://www.tbs-certificats.com/issuerdata/DigiCertGlobalRootCA.crt), then set the environment variable export WEBSOCKET_CLIENT_CA_BUNDLE=DigiCertGlobalRootCA.crt

Wrapping Up

Alright, now you've got a simple Starter Bot with a bunch of places in the code you can add whatever features you want to build.

There is a whole lot more that could be done using the Slack RTM API and Python. Check out these posts to learn what you could do:

Attach a persistent relational database or NoSQL back-end such as PostgreSQL, MySQL or SQLite to save and retrieve user data
Add another channel to interact with the bot via SMS or phone calls
Integrate other web APIs such as GitHub or Twilio
Explore other Slack Platform APIs and the reasons you might use one over another.
Build an onboarding bot using the Slack Events API

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Responding to SMS Text Messages with Python & Flask

Matt Makai — Wed, 05 Aug 2020 00:00:00 -0400

Short Message Service (SMS) text messages are easy to send from Python applications with a web application programming interface (API). Flask applications can also receive incoming text messages and respond back to the sender with just a few lines of Python code.

Tools We Need

This tutorial is fine for both Python 2 and 3. Make sure you have one of those two versions installed on your system.

Either Python 2 or 3
pip and virtualenv to handle application dependencies
The Flask micro web framework
A free Twilio account to use their SMS web API
Open source Twilio Python helper library
Ngrok for localhost tunneling to our Flask application while it's running on our local development environment

If you need assistance getting pip and virtualenv installed, take a look at the first few steps in the how to set up Python 3, Flask and Green Unicorn on Ubuntu 16.04 LTS guide, which shows how to install system packages for those tools.

Installing Our Dependencies

Our code will use a helper library to make it easier to respond to text messages from Python. The helper library dependency along with the Flask code library can be installed from PyPI into a virtualenv. In your terminal use the following command to generate a new virtualenv.

virtualenv respondsms

Activate the virtualenv.

source respondsms/bin/activate

The command prompt will change after we properly activate the virtualenv to something like this:

Install Flask and the Twilio Python helper library into the virtualenv with the pip command.

pip install flask twilio==5.7.0

The dependencies are installed so that we can use it with our Python code. Now we can write our Python application.

Building Our Flask Web App

Our Flask application will have two routes: one to make sure the web app is running and another that handles incoming HTTP POST requests. Create a new file named app.py in your home directory or where you choose to store your Python project files.

Within app.py write the following code. You can also see this code in a GitHub Gist if that's easier to copy and paste.

from flask import Flask, Response, request
from twilio import twiml


app = Flask(__name__)


@app.route("/")
def check_app():
    # returns a simple string stating the app is working
    return Response("It works!"), 200


@app.route("/twilio", methods=["POST"])
def inbound_sms():
    response = twiml.Response()
    # we get the SMS message from the request. we could also get the 
    # "To" and the "From" phone number as well
    inbound_message = request.form.get("Body")
    # we can now use the incoming message text in our Python application
    if inbound_message == "Hello":
        response.message("Hello back to you!")
    else:
        response.message("Hi! Not quite sure what you meant, but okay.")
    # we return back the mimetype because Twilio needs an XML response
    return Response(str(response), mimetype="application/xml"), 200


if __name__ == "__main__":
    app.run(debug=True)

The inline comments on the lines starting with # explain what the lines below them do. Flask applications define URL routes with the @app.route decorator. Our application needs two routes therefore we have two of those decorators defined.

Give the application a try by running it with python app.py. If you have trouble running the program, make sure your virtualenv is still active so that the application can use the Flask and Twilio code libraries we installed earlier.

Open a web browser and go to localhost:5000 (or 127.0.0.1:5000). We should see "It works!" on the screen.

There is one problem with our application running on our local development environment: there's no way for our server to receive HTTP POST requests unless we use a localhost tunnel.

Localhost Tunneling with Ngrok

Ngrok provides a localhost tunnel so that outside services can connect to a server running in your local development environment. Download and install Ngrok.

We can now run Ngrok locally and connect our Flask app running on port 5000. Within the directory where you extracted Ngrok, run this command.

./ngrok http 5000

Awesome, now we can use that Ngrok Forwarding URL to access our application from any machine that has an internet connection. Replace the URL in the web browser with your own Forwarding URL, like I did in this screenshot.

We just need a phone number that'll hit our application with a POST request to respond to text messages.

Obtaining Our Phone Number

We can use our Flask application's route to respond to incoming web API requests based on incoming SMS messages to a Twilio phone number. Go to the Twilio website and sign up for a free trial account to use their API. If you already have a Twilio account then sign into your existing account.

The Twilio trial account allows you to send and receive text messages to your own validated phone number. To send and respond to SMS to and from any phone number then you need to upgrade your account. Trial accounts are great for initial development before your application goes live.

When you sign up, you receive a free Twilio phone number. We can configure that phone number to forward the SMS information to our web application by setting up the response webhook.

Go to the manage phone numbers screen and click on the phone number you want to configure for responding to inbound text messages.

Scroll down to near the bottom of the page and look for the "Messaging" header. Modify the "A Message Comes in" text box so that it has your ngrok Forwarding URL plus the "/twilio" route, as shown in this screenshot.

Now press the red "Save" button at the bottom to make our changes take effect.

Our application is ready to go - time to give our phone number a try! Send "Hello" or whatever text you want to your phone number. Here is what the result looks like on my iPhone.

This simple Flask application is a good start to build more complicated responses such as adding natural language processing, building SMS Slack bots or coding SMS-powered NES Game Genies.

What's next?

Sweet, now our Flask web app automatically responds to incoming SMS text messages! It's pretty crazy to think that entire businesses such as SuperPhone and Remind are built off code that started out very similar to the code we just wrote.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

How to Install and Use MySQL on Ubuntu 16.04

Matt Makai — Fri, 22 Dec 2017 00:00:00 -0500

MySQL is a common open source relational database for creating, reading, updating and deleting data in Python web applications. Let's learn how to install MySQL on Ubuntu 16.04 and then run a few SQL queries within the command line client.

We will not go over connecting via Python applications using object-relational mappers (ORMs) but these steps can be used as a prerequisite to working with an ORM such as SQLAlchemy or Peewee.

Tools We Need

In this tutorial we'll use the following components:

Ubuntu 16.04.2 (this tutorial should also work on other Ubuntu versions)
MySQL

Install MySQL

We can install MySQL by using the apt package manager. First make sure your packages list are up to date. Open the terminal and run this apt command.

sudo apt-get update

We need to install the mysql-server package, which downloads the required files, configures the initial database set up and handles running MySQL as a system service. Run this apt command to get the process started.

sudo apt-get install mysql-server

Enter 'y' when prompted with whether or not you want to install the new package.

An administrative screen asking for a new root password will appear in the middle of the package installation process. Enter your chosen new password twice and the installation will continue.

In a moment the installation will finish and you'll be back at the command prompt.

MySQL is now installed with a root user. However, we do not want to have our applications connect to the database with that user, so next we will create a new non-root user.

Securing MySQL

MySQL is installed with a basic configuration meant for development and testing purposes. However, the configuration is not secure for production enviroments, therefore it comes with a utility to handle basic security. Run the following command and answer the questions based on your environment requirements.

sudo mysql_secure_installation

When you finish running the script you should see the following output and be back at the command prompt.

Our MySQL instance has basic security in place but we need to create a non-root user for applications to interact with the database.

Creating MySQL Users

To create a non-root user, connect to the MySQL instance with the mysql command line client.

mysql -u root -p

Now use the CREATE USER command to generate a new user. Make sure to change "mynewuser" and "goodPassword" with your own values.

CREATE USER 'mynewuser'@'localhost' IDENTIFIED BY 'goodPassword';

No output after the command is good - that means the command succeeded.

We need to apply privileges to the new user so it can handle basic database operations. Again, make sure to replace the default username in this command with your new username.

GRANT ALL PRIVILEGES ON * . * TO 'mynewuser'@'localhost';

It's a good idea to reload the privileges to make sure our new user's permissions are in place.

FLUSH PRIVILEGES;

Now that our permissions are reloaded we can connect with the new user.

New User Connection

We're set to connect to the database with our new user. Exit the MySQL client with "Ctrl-d". Reconnect using a slightly different command than we used earlier.

mysql -u mynewuser -p

Create a new database with the CREATE DATABASE command.

CREATE DATABASE fullstackpython;

Connect to the new database with the USE command.

use fullstackpython;

Create a simple new table with the CREATE TABLE command.

CREATE TABLE pages (name VARCHAR(50), url VARCHAR(1024));

Our table is ready to go - we can interact with it using the SELECT, INSERT, UPDATE and DELETE SQL commands.

What's next?

We now have our MySQL instance installed and ready for interaction. Take a look at the MySQL, relational databases and object-relational mappers (ORMs) pages for more tutorials.

Questions? Tweet @fullstackpython or post a message on the Full Stack Python Facebook page.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Setting up PostgreSQL with Python 3 and psycopg on Ubuntu 16.04

Matt Makai — Mon, 25 Dec 2017 00:00:00 -0500

PostgreSQL is a powerful open source relational database frequently used to create, read, update and delete Python web application data. Psycopg2 is a PostgreSQL database driver that serves as a Python client for access to the PostgreSQL server. This post explains how to install PostgreSQL on Ubuntu 16.04 and run a few basic SQL queries within a Python program.

We won't cover object-relational mappers (ORMs) in this tutorial but these steps can be used as a prerequisite to working with an ORM such as SQLAlchemy or Peewee.

Tools We Need

Our walkthrough should work with either Python 2 or 3 although all the steps were tested specifically with Python 3.5. Besides the Python interpreter, here are the other components we'll use:

Ubuntu 16.04.2 (these steps should also work fine with other Ubuntu versions)
pip and virtualenv to handle the psycopg2 application dependency
PostgreSQL

If you aren't sure how to install pip and virtualenv, review the first few steps of the how to set up Python 3, Bottle and Green Unicorn on Ubuntu 16.04 LTS guide.

Install PostgreSQL

We'll install PostgreSQL via the apt package manager. There are a few packages we need since we want to both run PostgreSQL and use the psycopg2 driver with our Python programs. PostgreSQL will also be installed as a system service so we can start, stop and reload its configuration when necessary with the service command. Open the terminal and run:

sudo apt-get install postgresql libpq-dev postgresql-client postgresql-client-common

Enter your sudo password when prompted and enter 'yes' when apt asks if you want to install the new packages.

After a few moments apt will finish downloading, installing and processing.

We now have PostgreSQL installed and the PostgreSQL service is running in the background. However, we need to create a user and a database instance to really start using it. Use the sudo command to switch to the new "postgres" account.

sudo -i -u postgres

Within the "postgres" account, create a user from the command line with the createuser command. PostgreSQL will prompt you with several questions. Answer "n" to superuser and "y" to the other questions.

createuser matt -P --interactive

Awesome, now we have a PostgreSQL user that matches our Ubuntu login account. Exit out of the postgres account by pressing the "Ctrl" key along with "d" into the shell. We're back in our own user account.

Create a new database we can use for testing. You can name it "testpython" or whatever you want for your application.

createdb testpython

Now we can interact with "testpython" via the PostgreSQL command line tool.

Interacting with PostgreSQL

The psql command line client is useful for connecting directly to our PostgreSQL server without any Python code. Try out psql by using this command at the prompt:

psql testpython

The PostgreSQL client will connect to the localhost server. The client is now ready for input:

Try out PostgreSQL's command prompt a try with commands such as \dt and \dd. We can also run SQL queries such as "SELECT * from testpython", although that won't give us back any data yet because we have not inserted any into the database. A full list of PostgreSQL commands can be found in the psql documentation.

Installing psycopg2

Now that PostgreSQL is installed and we have a non-superuser account, we can install the psycopg2 package. Let's figure out where our python3 executable is located, create a virtualenv with python3, activate the virtualenv and then install the psycopg2 package with pip. Find your python3 executable using the which command.

which python3

We will see output like what is in this screenshot.

Create a new virtualenv in either your home directory or wherever you store your Python virtualenvs. Specify the full path to your python3 installation.

# specify the system python3 installation
virtualenv --python=/usr/bin/python3 venvs/postgrestest

Activate the virtualenv.

source ~/venvs/postgrestest/bin/activate

Next we can install the psycopg2 Python package from PyPI using the pip command.

pip install psycopg2

Sweet, we've got our PostgreSQL driver installed in our virtualenv! We can now test out the installation by writing a few lines of Python code.

Using PostgreSQL from Python

Launch the Python REPL with the python or python3 command. You can also write the following code in a Python file such as "testpostgres.py" then execute it with python testpostgres.py. Make sure to replace the "user" and "password" values with your own.

import psycopg2

try:
    connect_str = "dbname='testpython' user='matt' host='localhost' " + \
                  "password='myOwnPassword'"
    # use our connection values to establish a connection
    conn = psycopg2.connect(connect_str)
    # create a psycopg2 cursor that can execute queries
    cursor = conn.cursor()
    # create a new table with a single column called "name"
    cursor.execute("""CREATE TABLE tutorials (name char(40));""")
    # run a SELECT statement - no data in there, but we can try it
    cursor.execute("""SELECT * from tutorials""")
    conn.commit() # <--- makes sure the change is shown in the database
    rows = cursor.fetchall()
    print(rows)
    cursor.close()
    conn.close()
except Exception as e:
    print("Uh oh, can't connect. Invalid dbname, user or password?")
    print(e)

When we run the above code we won't get anything fancy, just an empty list printed out. However, in those few lines of code we've ensured our connection to our new database works and we can create new tables in it as well as query them.

That's just enough of a hook to get started writing more complicated SQL queries using psycopg2 and PostgreSQL. Make sure to check out the PostgreSQL, relational databases and object-relational mappers (ORMs) pages for more tutorials.

Questions? Tweet @fullstackpython or post a message on the Full Stack Python Facebook page.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

How to Use Redis with Python 3 and redis-py on Ubuntu 16.04

Matt Makai — Fri, 28 Apr 2017 00:00:00 -0400

Redis is an in-memory key-value pair NoSQL data store often used for web application sessions, transient data and as a broker for task queues. redis-py is a common Python code library for interacting with Redis. Let's learn how to get Redis up and running on Ubuntu and then start using it in a simple Python application.

Tools We Need

This tutorial is tested with Python 3.5 but either Python 2 or 3 should work for everything written here. Just make sure one version is installed on your system by going to the terminal and typing python --version. Other than Python itself, here is the software we are going to use throughout the rest of this post:

Ubuntu 16.04 (these instructions should work fine with earlier Ubuntu versions as well)
pip and virtualenv to handle the redis-py application dependency
Redis
redis-py

If you aren't sure how how to install pip and virtualenv, review the first few steps of the how to set up Python 3, Flask and Green Unicorn on Ubuntu 16.04 LTS guide.

Install Redis

There are a few ways to install Redis, such as downloading and compiling from source. However, on Ubuntu we can install a system package through apt. The advantage of this method is that the apt process will take care of installing redis-server as a system service. Open the terminal and run the following command:

sudo apt-get install redis-server

Enter your sudo password and when you are prompted whether you want to install the new package enter 'yes'.

After a few moments the downloading and processing should be complete and you will be back at the prompt.

Redis is now installed and the Redis server is running in the background as a system service. Even though we installed the redis-server package, the installation also comes with the Redis command line client. The client is useful for connecting directly to the Redis server without any Python code. Give redis-cli a try by typing this into the command prompt:

redis-cli

The Redis client connects to the localhost server and gives a new prompt to show it's ready for commands:

Give the prompt a try by using Redis commands such as keys * or set a 1. The full list of Redis commands is provided in the project documentation.

Virtualenv and Install redis-py

We need to figure out our python3 location, then create a virtualenv, activate the virtualenv and then install redis-py with pip. Determine your python3 executable location with the which command.

which python3

You'll see some output like the following screenshot.

Create a new virtualenv either in your home directory or wherever you store your project virtualenvs. Specify the full path to your python3 installation.

# specify the system python3 installation
virtualenv --python=/usr/bin/python3 venvs/redistest

Activate the virtualenv.

source ~/venvs/redistest/bin/activate

Next we can install the redis-py Python package from PyPI using the pip command.

pip install redis

Alright, now it is installed in our virtualenv. Let's write some simple Python code to try out give redis-py!

Working with Redis from Python

Fire up the Python REPL with the python or python3 command. You can also write the following code in a Python file such as "testredis.py" then execute it with python testredis.py.

import redis
# create a connection to the localhost Redis server instance, by
# default it runs on port 6379
redis_db = redis.StrictRedis(host="localhost", port=6379, db=0)
# see what keys are in Redis
redis_db.keys()
# output for keys() should be an empty list "[]"
redis_db.set('full stack', 'python')
# output should be "True"
redis_db.keys()
# now we have one key so the output will be "[b'full stack']"
redis_db.get('full stack')
# output is "b'python'", the key and value still exist in Redis
redis_db.incr('twilio')
# output is "1", we just incremented even though the key did not
# previously exist
redis_db.get('twilio')
# output is "b'1'" again, since we just obtained the value from
# the existing key
redis_db.delete('twilio')
# output is "1" because the command was successful
redis_db.get('twilio')
# nothing is returned because the key and value no longer exist

That is a quick introduction to some commonly-used Redis commands invoked by their Python bindings through the redis-py library. Take a look at the redis-py official documentation to learn more about the extensive command list you can use to create, read, modify and delete keys and values in Redis.

Questions? Tweet @fullstackpython or post a message on the Full Stack Python Facebook page. See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

How to Send MMS Picture Messages with Python

Matt Makai — Wed, 28 Mar 2018 00:00:00 -0400

Multimedia Message Service (MMS) picture and video messages are a common extension to the Short Message Service (SMS) system for sending text messages. Using a web application programming interface (API) with Python makes it easy to send MMS messages from a web application or script. In this short tutorial we'll learn how to add MMS sending capability to a new or existing Python application.

Tools We Need

Either Python 2 or 3 works for the code in this tutorial. Just make sure you have one of those two versions installed on your system by going to the terminal and typing python --version. The other dependencies for this tutorial include:

Python version 2 or 3
pip and virtualenv to handle one application dependency
A free Twilio account to use their MMS web API
Twilio Python helper library, version 6.0.0 or later

If you are unsure of how to get pip and virtualenv installed, take a look at the first few steps of the how to set up Python 3, Flask and Green Unicorn on Ubuntu 16.04 LTS guide.

Twilio Web API

Our simple Python example application will use the Twilio web API to send picture messages. Go to the Twilio website sign up for a free trial account. If you already have a Twilio account (and you should because it makes it easy to add almost any type of communications to applications!) then sign into your existing account.

In trial mode Twilio can send MMS to a validated phone number associated with the account. When you're ready to send MMS messages to any phone in any country then you will have to upgrade your account.

After signing up for a Twilio account, you will receive your own phone number that'll be used to send messages. That phone number can send outbound MMS messages without any configuration. It can also receive messages but that requires modifying the Request URL webhook in the phone number details screen.

Installing Our Dependency

We'll use the twilio helper library as a dependency for our Python code. The helper library can be installed via the pip command, which pulls the code from PyPI into our local virtualenv. In this tutorial we'll call our virtualenv pymms but you can name it whatever you want for your application.

We have to create the virtualenv before using it. In your terminal enter:

virtualenv pymms

If you need to install virtualenv take a look at the how to set up Python 3, Django and Green Unicorn on Ubuntu 16.04 LTS guide.

Activate the virtualenv with the source command.

source pymms/bin/activate

The command prompt will change to look like this after it is activated:

Now install the Twilio Python helper library. Make sure you install the version 6.0.0 or later current version because the syntax for this code changed a bit from earlier helper library versions before 6.0.0.

pip install twilio>=6.0.0

Once the helper library installs we can use it in our Python code.

Sending MMS From Python

Launch the the Python interpreter by executing the python command in your terminal. You can also create a new file named send_mms.py if you want to re-use the code after we give it a try.

We need to grab our account credentials from the Twilio Console to connect our Python code to our Twilio account. Go to the Twilio Console and copy the Account SID and Authentication Token into your Python code.

Enter the following code into the new Python file, or copy it from this GitHub repository that contains all blog code examples.

# import the Twilio client from the dependency we just installed
from twilio.rest import Client

# the following line needs your Twilio Account SID and Auth Token
client = Client("ACxxxxxxxxxxxxxx", "zzzzzzzzzzzzz")

# this is the URL to an image file we're going to send in the MMS
media = "https://raw.githubusercontent.com/mattmakai/fullstackpython.com/master/static/img/logos/f.png"

# change the "from_" number to your Twilio number and the "to" number
# to the phone number you signed up for Twilio with, or upgrade your
# account to send MMS to any phone number that MMS is available
client.api.account.messages.create(to="+19732644152",
                                   from_="+12023351278",
                                   body="MMS via Python? Nice!",
                                   media_url=media)

All the lines above that start with # are comments to give you some context for what each line is doing. After entering that code into the interpreter or running the Python script with python send_mms.py Twilio will send your MMS.

In a few seconds you should see a message appear on your phone - note that MMS can take a little longer because your phone has to download the image. I use an iPhone so here is what the message looked like when I received it:

That is everything need to send MMS to a phone. Pretty awesome result for a few lines of Python code, right? This code can be added to any Python program to send outbound MMS.

One final note: keep your Twilio Auth Token secret otherwise anyone who gets it will be able to send and receive messages through your account.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

Configuring Python 3, Bottle and Gunicorn for Development on Ubuntu 16.04 LTS

Matt Makai — Fri, 28 Apr 2017 00:00:00 -0400

The Ubuntu 16.04 Long Term Support (LTS) Linux operating system was released in April 2016. This latest Ubuntu release is named "Xenial Xerus" and it is the first Ubuntu release to include Python 3, instead of Python 2.x, as the default Python installation.

We can quickly start a new Bottle web application project and run it with Green Unicorn (Gunicorn) on Ubuntu 16.04.

Tools We Need

Our setup requires the Ubuntu 16.04 release along with a few other code libraries. Don't install these tools just yet since we'll get to them as we go through the walkthrough. Our requirements and their current versions as of April 2017 are:

Ubuntu 16.04.2 LTS (Xenial Xerus)
Python version 3.5.1 (default in Ubuntu 16.04.2)
Bottle web framework version 0.13
Green Unicorn (Gunicorn) version 19.7.1

If you are developing on Mac OS X or Windows, make sure to use virtualization software such as Parallels or VirtualBox with the Ubuntu .iso file. Either the amd64 or i386 version of 16.04 is fine. I use the amd64 version for my own local development.

A desktop screen like this one appears when you boot up Ubuntu.

Open a terminal window to install the system packages.

System Packages

We can see the python3 system version Ubuntu comes with and where its executable is stored using these commands.

python3 --version
which python3

Our Ubuntu installation requires a few system packages. We will get prompted for the superuser password because restricted system access is needed to install packages through apt.

sudo apt-get install python3-pip python3-dev

Enter y to let the system package installation process do its job.

The packages we need are now installed. We can continue on to install our Python-specific dependencies.

Virtualenv

In the previous section, virtualenv and pip were installed to handle our application dependencies. We can now use them to download and install Bottle and Gunicorn.

Create a directory for the virtualenvs. Then create a new virtualenv.

# make sure pip and setuptools are the latest version
pip3 install --upgrade pip setuptools
# the tilde "~" specifies the user's home directory, like /home/matt
cd ~
mkdir venvs
# specify the system python3 installation
virtualenv --python=/usr/bin/python3 venvs/bottleproj
python3 -m venv venvs/bottleproj

Activate the virtualenv.

source ~/venvs/bottleproj/bin/activate

Our prompt will change after we properly activate the virtualenv.

Our virtualenv is now activated with Python 3. We can install whatever dependencies we want, in our case Bottle and Gunicorn.

Bottle and Gunicorn

We can now install Bottle and Green Unicorn via the pip command.

pip install bottle gunicorn

No errors like we see in the following screenshot is a good sign.

Use the mkdir command to create a new directory to keep our Bottle project then use the cd (change directory) command to move into the new folder.

mkdir ~/bottleproj
cd ~/bottleproj

Create a new file named app.py within our bottleproj directory so we can test to make sure Bottle is working properly. I prefer to use Vim but Emacs and other development environments work great as well.

Within the new app.py file write the following code.

import bottle
from bottle import route, run, Response

# a basic URL route to test whether Bottle is responding properly
@route('/')
def index():
    return Response("It works!")

# these two lines are only used for python app.py
if __name__ == '__main__':
    run(host='0.0.0.0', port=8000, debug=True, reloader=True)

# this is the hook for Gunicorn to run Bottle
app = bottle.default_app()

We could run our app with the Bottle development server using the python app.py command. Let's instead run our Bottle app with Gunicorn.

gunicorn -w 2 app:app

Sweet, we can bring up our shell Bottle app in the web browser at the localhost:8000 or 127.0.0.1:8000 address.

Time to develop a full-fledged web application with Bottle!

Ready for Development

Now you have a simple setup to develop Bottle web apps using Gunicorn as the WSGI server on Ubuntu 16.04. If you need a full step-by-step tutorial to deploy your Python web application to a production environment, check out the Full Stack Python Guide to Deployments book.

To decide what to do next with your Python project, check out the Full Stack Python table of contents page.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

How to Send SMS Text Messages with Python

Matt Makai — Wed, 05 Aug 2020 00:00:00 -0400

Short Message Service (SMS) text messages are ubiquitous for communication all over the world. It is easy to send SMS text messages from a Python application using a web application programming interface (API). Let's take a look at the tools we need to quickly add SMS capability to our Python apps.

Tools We Need

This guide works with both Python 2 and 3, so make sure you have one of those two versions installed.

Either Python 2 or 3
pip and virtualenv to handle application dependencies
A free Twilio account to use their SMS web API
Open source Twilio Python helper library, version 6.0.0 or later

If you need assistance getting pip and virtualenv installed, check out the first few steps of the how to set up Python 3, Flask and Green Unicorn on Ubuntu 16.04 LTS guide that'll show how to install system packages for those tools.

Using a Web API

We're going to use a web API to make sending SMS easier and more reliable. Head to the Twilio website and sign up for a free trial account awesome for more than just sending text messages!) then sign into your existing account.

The Twilio trial account allows you to send text messages to your own validated phone number. When you want to send SMS to any phone number in your country or other countries then you can upgrade your account to send messages for fractions of a cent.

After signing up, you will get a free phone number in your country. We can use that phone number without any configuration to send outbound text messsages. You can also receive text messages but that requires changing the Request URL webhook in the phone number configuration screen - we'll cover that in a future blog post.

Installing Our Dependency

Our code will use a helper library to make it easier to send text messages from Python. We are going to install the helper library from PyPI into a virtualenv. First we need to create the virtualenv. In your terminal use the following command to create a new virtualenv. If you need to install virtualenv take a look at the how to set up Python 3, Flask and Green Unicorn on Ubuntu 16.04 LTS guide.

virtualenv sendsms

Activate the virtualenv.

source sendsms/bin/activate

The command prompt will change after we properly activate the virtualenv to something like this:

Now install the Twilio Python helper library. We are using the 6.0.0 or above library version, which is important because the syntax in this post is backwards-incompatible with 5.x and previous Twilio helper library versions.

pip install twilio>=6.0.0

The helper library is now installed and we can use it with the Python code we create and execute.

Sending SMS From Python

Fire up the Python interpreter in the terminal using the python command, or create a new file named send_sms.py.

Enter the following code into the interpreter or into the new Python file. You can also copy and paste the code from the blog-code-examples Git repository in the Full Stack Python GitHub organization.

# we import the Twilio client from the dependency we just installed
from twilio.rest import Client

# the following line needs your Twilio Account SID and Auth Token
client = Client("ACxxxxxxxxxxxxxx", "zzzzzzzzzzzzz")

# change the "from_" number to your Twilio number and the "to" number
# to the phone number you signed up for Twilio with, or upgrade your
# account to send SMS to any phone number
client.messages.create(to="+19732644152", 
                       from_="+12023351278", 
                       body="Hello from Python!")

All the lines above that start with # are comments. Once you enter that code into the interpreter or run the Python script using python send_sms.py the SMS will be sent.

In a few seconds you should see a message appear on your phone. I'm on iOS so here's how the text message I received looked.

That's it! You can add this code to any Python code to send text messages. Just keep your Auth Token secret as it'll allow anyone that has it to use your account to send and receive messages.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

How to set up Python 3, Flask and Green Unicorn on Ubuntu 16.04 LTS

Matt Makai — Wed, 14 Jun 2017 00:00:00 -0400

Ubuntu's latest Long Term Support (LTS) operating system was released last year, in April 2016. The 16.04 update for Ubuntu is known as "Xenial Xerus" and it is the first Ubuntu release to include Python 3 as the default Python installation.

We can use the Ubuntu release along with Python version 3.5 to start a new Flask web application project and run it with Green Unicorn (Gunicorn).

Tools We'll Need

Our project will use the Ubuntu 16.04 release along with a few other libraries. You don't have to install these tools just yet, we will get to them as we progress through the walkthrough. Our requirements and their current versions as of April 2017 are:

Ubuntu 16.04.2 LTS (Xenial Xerus)
Python version 3.5.1 (default in Ubuntu 16.04.2)
Flask web framework version 0.12
Green Unicorn (Gunicorn) version 19.7.1

If you're running on Mac OS X or Windows, use virtualization software such as Parallels or VirtualBox with the Ubuntu .iso file. Either the amd64 or i386 version of 16.04 is fine. I'm using amd64 for development and testing in this tutorial.

Once you boot up Ubuntu, you should see a screen like this one.

Open up a terminal window to proceed with the setup.

System Packages

We can see the python3 system version Ubuntu comes with and where its executable is stored using these commands.

python3 --version
which python3

Our Ubuntu installation requires a few system packages. We will get prompted for the superuser password because restricted system access is needed to install packages through apt.

sudo apt-get install python3-dev python3-pip

Enter y to let the system package installation process do its job.

The packages we need are now installed. We can continue on to install our Python-specific dependencies.

Virtualenv

In the previous section, virtualenv and pip were installed to handle our application dependencies. We can now use them to download and install Flask and Gunicorn.

Create a directory for the virtualenvs. Then create a new virtualenv.

# make sure pip and setuptools are the latest version
pip3 install --upgrade pip setuptools
# the tilde "~" specifies the user's home directory, like /home/matt
cd ~
mkdir venvs
# specify the system python3 installation
python3 -m venv venvs/flaskproj

Activate the virtualenv.

source ~/venvs/flaskproj/bin/activate

Our prompt will change after we properly activate the virtualenv.

Our virtualenv is now activated with Python 3. We can install whatever dependencies we want, in our case Flask and Gunicorn.

Flask and Gunicorn

We can finally install Flask and Green Unicorn via the pip command.

pip install flask gunicorn

It is a good sign if we receive no errors like we see in the following screenshot.

Create a new directory under our home directory that will store our Flask project. Change directory into the new folder.

mkdir ~/flaskproj
cd ~/flaskproj

Create a new file named __init__.py within our flaskproj directory so we can test to make sure Flask is working properly. I prefer to use Vim but Emacs and other development environments work great as well.

Within __init__.py write the following code.

from flask import Flask, Response


app = Flask(__name__)

@app.route("/")
def index():
    return Response("It works!"), 200

if __name__ == "__main__":
    app.run(debug=True)

We could run our app with the Flask development server using the python __init__.py command. Instead run the Flask app with Gunicorn. Go to the directory above the flaskproj folder, in our case we can enter cd ~ then use the gunicorn command:

gunicorn flaskproj:app

Sweet, we can bring up our shell Flask app in the web browser at the localhost:8000 or 127.0.0.1:8000 address.

Now we're ready for some real Flask development!

Ready for Development

That's a simple setup for developing with Flask and Gunicorn on Ubuntu 16.04. If you need an in-depth step-by-step tutorial to deploy your WSGI-powered web application to a production environment, check out the Full Stack Python Guide to Deployments book.

To determine what to code next for your Python project, read the topics found on the table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

Something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Setting up Python 3, Django and Gunicorn on Ubuntu 16.04 LTS

Matt Makai — Fri, 28 Apr 2017 00:00:00 -0400

Ubuntu released the newest Long Term Support (LTS) version of its operating system in April 2016. The update brings Ubuntu to version 16.04 and its latest code name is "Xenial Xerus". 16.04 is the first Ubuntu release to include Python 3 as the default Python installation.

Let's use this newest Ubuntu release along with Python version 3.5 to start a new Django web application project and run it with Green Unicorn (Gunicorn).

Tools We Need

We will need a few tools to complete our project. Don't worry about installing these just yet as we'll get to them as we progress through the tutorial. The tools and their current versions as of April 2017 are:

Ubuntu 16.04.2 LTS (Xenial Xerus)
Python version 3.5.1 (default in Ubuntu 16.04.2). Python 3.6 has been released but Ubuntu 16.04.2 comes with 3.5.1 by default so we'll use the version 3.5.1 in this post.
Django web framework version 1.11
Green Unicorn (Gunicorn) version 19.7.1

If you are running Mac OS X or Windows, use virtualization software such as Parallels (this is what I use, but it's Mac OS X-only) or VirtualBox with the Ubuntu .iso file. Either the amd64 or i386 version of 16.04 is fine, but I use amd64 for development and testing in this blog post.

When we boot up for the first time, we should see a desktop screen like this one.

Open up terminal to proceed with the setup.

System Packages

We can see the python3 version Ubuntu comes with, as well as where its executable is stored.

python3 --version
which python3

Our Ubuntu installation first needs system packages for Python development. You'll be prompted for your superuser password because restricted system access is required to install packages through apt.

sudo apt-get install python3-pip python3-dev

Enter y and let the system package installation process run.

The basic system packages we need are now installed so we can proceed to our Python-specific dependencies.

Virtualenv

Virtualenv and pip for isolating and handling application dependencies were just installed via system packages so we can now use them to obtain Django and Gunicorn.

Create a directory to store virtualenvs then put a new virtualenv in it.

# make sure pip and setuptools are the latest version
pip3 install --upgrade pip setuptools
# the tilde "~" specifies the user's home directory, like /home/matt
cd ~
mkdir venvs
# specify the system python3 installation
python3 -m venv venvs/djproject

Activate the virtualenv.

source ~/venvs/djproject/bin/activate

We should see our prompt change so that we know the virtualenv is properly activated.

Our virtualenv with Python 3 is activated so we can install whatever dependencies we want, such as Django and Gunicorn.

Django and Gunicorn

Time to install Django and Green Unicorn into our virtualenv.

pip install django gunicorn

No errors is a good sign everything worked for us.

Create a new Django project named djproject, or whatever you want to name your project. Then change into the directory for the new project.

django-admin startproject djproject
cd djproject

We could run Django with the development server using the python manage.py runserver command. However, start Django up with Gunicorn instead.

gunicorn djproject.wsgi

Awesome, now we can bring up our shell project in the web browser at the localhost:8000 or 127.0.0.1:8000 address.

Ready for development!

Ready for Development

Those are the basics for starting development with Django and Gunicorn on Ubuntu 16.04. If you need an even more in-depth step-by-step tutorial to deploy your Python web application to a production environment, check out the Full Stack Python Guide to Deployments book.

To figure out what to do next for your Python project, read the topics found on the table of contents page.

Questions? Contact me via Twitter @fullstackpython or @mattmakai. I'm also on GitHub with the username mattmakai.

See something wrong in this post? Fork this page's source on GitHub and submit a pull request.

The Full Stack Python Blog

Matt Makai — Thu, 28 Dec 2017 00:00:00 -0500

Full Stack Python began way back in December 2012 when I started writing the initial deployment, server, operating system, web server and WSGI server pages. The site has has broadly expanded out into a many other subjects outside the deployment topics I originally started this site to explain.

However, I frequently wanted to write a Python walkthrough that was not a good fit for the page format I use for each topic. Many of those walkthroughs became Twilio blog posts but not all of them were quite the right fit on there. I'll still write more Twilio tutorials, but this Full Stack Python blog is the spot for technical posts that fall outside the Twilio domain.

Let me know what you think and what tutorials you'd like to see in the future.

Hit me up on Twitter @fullstackpython or @mattmakai.