- Redis with Python Redis-py is a solid Python client to use with Redis. Walrus is a higher-level Python wrapper for Redis with some caching, querying and data structure components build into. Writing Redis in Python with Asyncio shows a detailed example for how to use the new Asyncio standard.
- Amazon ElastiCache for Redis is a Redis-compatible in-memory data store service that can be used as a database, cache, message broker, and queue for real-time applications. It combines the speed, simplicity, and versatility of open-source Redis with manageability, security, and reliability from Amazon.
Redis-py is a Python interface to Azure Cache for Redis. Use the Python packages tool, pip, to install the redis-py package from a command prompt. The following example used pip3 for Python 3 to install redis-py on Windows 10 from an Administrator command prompt. Read and write to the cache.
RQ (Redis Queue) is a Python library that uses Redis for queueing jobs and processing them in the background with workers. It has a much lower barrier to entry and is simpler to work with than other libraries such as Celery.
RQ, and task queues in general, are great for executing functions that are lengthy or contain blocking code, such as networking requests.
Using RQ is as simple as creating a queue, and enqueueing the desired function along with the arguments you want to pass to it, according to the code in their “Hello World” example:
Let’s walk through how to use RQ to execute a function that grabs data from the Mars Rover API.
Setting up your environment and installing dependencies
Before moving on, you will need to make sure you have an up to date version of Python 3 and pip installed. Make sure you create and activate a virtual environment before installing any dependencies.
We will be using the requests library to get data from NASA’s Mars Rover API, and RQ for handling asynchronous tasks. With your virtual environment activated, run the following command in your terminal to install the necessary Python libraries:
In order for RQ to work, you'll need to install Redis on your machine. That can be done with the following commands using
Run Redis in a separate terminal window on the default port with the command
src/redis-server from the directory where it's installed.
Getting started with Redis Queue
Let’s start with an example function to hit the Mars Rover API and print out the URL to an image taken by the rover. Since this function includes making an HTTP request, that means the code in it is blocking, making it a great example of something that RQ would be useful for.
Create a file called
mars.py and add the following code to it:
Run this code by opening a Python shell and entering:
This will print a URL to a random image taken by the Mars Rover 1000 Martian solar days into its journey. Notice that it takes some time for the HTTP request to resolve and for the URL to print. While your code is waiting for the remote server to respond to your HTTP request, it can’t do anything else until it’s finished. Here is a photo that my code printed.
If you pass this function to RQ to be processed as an asynchronous task, then it will no longer be blocking the rest of your code. This can be done as easily as importing RQ, creating a queue, and enqueueing the function. Let’s do this by putting the following code in a file called
In order for this code to work, you will have to run an RQ worker in the background in another terminal window for processing tasks.
A worker is a Python process that typically runs in the background and exists solely as a work horse to perform lengthy or blocking tasks that you don’t want to perform inside web processes. RQ uses workers to perform tasks that are added to the queue.
To run the code we just wrote, run the command
rqworker in the same directory as your code in another terminal window, and then run your code with the following command:
You should see the URLs for pictures taken on Mars being printed one by one, along with other information about the task in the terminal window where the worker is running, but not in the window where your code is running.
To demonstrate the non-blocking nature of this, try adding print statements before and after the loop, and executing regular function calls instead of enqueueing them:
You will see each photo URL printed, and then the “After” message. If you revert to the original code, but keep the print statements in, you will see the messages both print pretty much immediately while the code executing the HTTP requests is processed in your other terminal window.
There is a lot you can do with RQ workers and it’s worth reading the documentation to find out. RQ also provides you with a host of utilities on getting insight into jobs, including a CLI that makes it easy to requeue failed jobs. Here’s an example of the latter:
To Infinity and Beyond
The code we used in this post are just short examples of what you can do with RQ, but after going through this post and the RQ docs, hopefully you find it easy to add RQ to your other projects. After being able to execute functions as tasks like we did above, adding RQ to your Flask or Django app means only writing a few lines of code wherever it's needed!
You can also use RQ Scheduler if you want to be able to schedule tasks. It is quick to add it to projects that already use RQ and has a simple API that allows you to execute Python functions at a given datetime.
For more examples of projects that use RQ, check out this post on how to receive text messages when the International Space Station flies above your location, or how to create a phone number that plays computer generated music that sounds like the soundtracks of old Nintendo games.
- Email: [email protected]
- Twitter: @Sagnewshreds
- GitHub: Sagnew
- Twitch (streaming live code): Sagnewshreds
Redis, standing for REmote DIctionary Server, is a type of key-value NoSQL server. Redis is designed to support disk storage for persistence, retaining data after power is shut off, and stores data so it can be cached and accessed quickly. Redis supports multiple data types, including strings, hashes, lists and sets. This tutorial will provide an explanation on how to create and set up a Redis example in a Python virtual environment.
While this tutorial will use a Python virtual environment for the Redis client, the Redis server must be properly installed and configured before beginning. If using macOS X, Homebrew can be used to install Redis with the
brew install rediscommand. Alternatively, a pre-compiled version of Redis can be used to run the server with Windows. In Ubuntu, install Redis (
redis-serverpackage) with APT-GET.
Python must be installed for the package manager to initialize the software packages that use the
pipcommand. Use the command
python3 -V) in a UNIX terminal to confirm what version of Python is currently installed. For Windows machines, just open the Python application IDLE (GUI) to obtain the currently installed version.
Install packages on Ubuntu
If needed for Debian-based distro of Linux, such as Linux Mint or Ubuntu, execute the following command to install the PIP package manager for Python 3:
sudo apt install python3-pip
The following command will install the virtual environment for Python:
Start the Redis server
If using a version of Linux that uses the Systemd management system, execute the following
systemctl command to start the Redis server:
sudo systemctl start redis
For macOS X, start the Homebrew Redis server with the following command:
Redis Virtual Environment
First, the virtual environment for executing commands inside of the Redis and Python project must be set up.
From the terminal, execute the following command to install the virtual environment using the
sudo pip3 install virtualenv
Now create a virtual environment folder inside of the project directory:
The above command should produce the following results:
Using base prefix '/usr'
New Python executable in/home/linux/venv/bin/Python3
Also creating executable in/home/linux/venv/bin/Python
Installing setuptools, pip, wheel...
Next, in a UNIX terminal, use the
ls command to list all of the files in the directory, or
dir in Windows command prompt. The
/venv directory should now be displayed.
Now execute the following
source command to activate and enter the virtual environment:
(venv) prefix should now be attached to the command prompt, indicating entrance into the virtual environment was successful.
Specify the Python version for the virtual env
If multiple versions of Python 3 are installed, use the following
which command in a UNIX terminal to obtain more information about the current version and installation location:
The results should resemble the following:
Now use the Python path, from above, to setup a virtual environment for that particular version of Python, as shown in the following example:
virtualenv --python=/usr/bin/python3 venv/redis_example
Install Redis for Python
Now install the
redis-py client for the Redis key-values in Python using PIP with the following
Once installation is completed, access to the client in the Python interpreter should be granted.
Run Redis in a Python virtual environment
A Python executable command can be ran in the console to import Redis and create an example using the Redis data types. Execute the following command to run the Python command-line console:
Access to the Python interactive interpreter should now be granted. Here a sample program to run a virtual environment using Redis can be created.
NOTE: Press CTRL+D to exit out of the interactive Python interpreter.
NOTE: The above instructions for setting up the Python virtual environment for Redis will work for both macOS and Linux systems.
Use the Redis client inside of Python
Once access to the Redis client has been granted, execute the following command to return its version string inside of Python:
The results should resembles the following, indicating a version of Python 3, or greater, is installed:
Access Redis in virtual env
Check that a Redis server and cli is installed in the system by executing the
redis-server --version command. This should confirm that the version is the same as the
Access the Redis server in Python
Python Redis Mq
The Redis database must be imported into Python. Execute the following command to create a connection to import the Redis-py client package:
Redis example in Python
Next, execute the following command to create a connection for the Redis server configuration:
redis_server = redis.Redis('localhost')
set() line of code should return a response of
Now execute the following command to get the set name for the Redis server:
The above command should return the server name in the form of a bytes string, as shown here:
This tutorial explained how to create and set up a Redis example in a Python virtual environment. The article specifically explained how to install packages on Ubuntu, start the Redis server, set up the Redis virtual environment and specify the Python version for the virtual environment. The tutorial also covered how to install Redis for Python, run Redis in a Python virtual environment, use the Redis client inside of python and access Redis in a virtual environment. Remember that the interactive Python interpreter can easily be exited by just pressing the CTRL + D keys.