64 |
65 |
66 |
67 |
72 |
77 |
82 |
83 |
84 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Using Celery with Flask for Cropping Images
2 |
3 | This explains how to configure Flask, Celery, RabbitMQ and Redis, together with Docker to build a web service that dynamically generates content and loads this contend when it is ready to be displayed. We'll focus mainly on Celery and the servies that surround it. Docker is a bit more straightforward.
4 |
5 | ## Project Structure
6 |
7 | The finished project structure will be as follows:
8 |
9 | ```
10 | .
11 | ├── Dockerfile
12 | ├── docker-compose.yml
13 | ├── README.md
14 | ├── app
15 | │ ├── app.py
16 | │ ├── tasks.py
17 | │ └── templates
18 | │ ├── download.html
19 | │ └── index.html
20 | ├── scripts
21 | │ ├── run_celery.sh
22 | │ └── run_web.sh
23 | └── requirements.txt
24 | ```
25 |
26 | ## Creating the Flask application
27 |
28 | First we create an folder for our app. For this example, our folder is simply called `app`. Within this folder, create an `app.py` file and an empty folder named `templates` where our HTML templates will be stored.
29 |
30 | For our app, we first include some basic Flask libraries and create an instance of the app:
31 |
32 | ```python
33 | from io import BytesIO
34 | from flask import Flask, request
35 | from flask import render_template, make_response
36 |
37 | APP = Flask(__name__)
38 | ```
39 |
40 | We define three routes for Flask to implement: a landing page, a secondary page that embeds and image, and a route for the image itself. Our image route crops an image dynamically. For this example, it crops an image using `pillow`, and some delays are also included so that the time taken to create the image is more apparent.
41 |
42 | ```python
43 | @APP.route('/')
44 | def index():
45 | return render_template('index.html')
46 | ```
47 |
48 | ```python
49 | @APP.route('/image_page')
50 | def image_page():
51 | job = tasks.get_data_from_strava.delay()
52 | return render_template('home.html')
53 | ```
54 |
55 | ```python
56 | @APP.route('/result.png')
57 | def image_demension(img):
58 | time.sleep(2)
59 | im = Image.open(img)
60 | width, height = im.size
61 | left = 4
62 | top = height / 5
63 | right = 154
64 | bottom = 3 * height / 5
65 |
66 | # Cropped image of above dimension \
67 | im1 = im.crop((left, top, right, bottom))
68 | newsize = (300, 300)
69 | im1 = im1.resize(newsize)
70 | width, height = im1.size
71 | location=os.path.join('static/worker-img','cropped_img.'+im.format.lower())
72 | im1.save(os.path.join('static/worker-img','cropped_img.'+im.format.lower()))
73 | print(width,height)
74 | print("pass")
75 |
76 | return location
77 | ```
78 |
79 | Next, we need to open our `templates` folder and create the following two templates:
80 |
81 | #### index.html
82 |
83 | ```html
84 |
85 | ```
86 |
87 | If we add the following code then run the script, we can load up our webpage and test the image generation.
88 |
89 | ```python
90 | if __name__ == '__main__':
91 | APP.run(host='0.0.0.0')
92 | ```
93 |
94 | We see that our page load takes a while to complete because the request to `result.png` doesn't return until the image generation has completed.
95 |
96 | ## Expanding our web app to use Celery
97 |
98 | In our `app` directory, create the `tasks.py` file that will contain our Celery tasks. We add the neccessary Celery includes:
99 |
100 | ```python
101 | from celery import Celery, current_task
102 | from celery.result import AsyncResult
103 | ```
104 |
105 | Assuming that our RabbitMQ service is on a host that we can reference by `rabbit` and our Redis service is on a host referred to by `redis` we can create an instance of Celery using the following:
106 |
107 | ```python
108 | REDIS_URL = 'redis://redis:6379/0'
109 | BROKER_URL = 'amqp://admin:mypass@rabbit//'
110 |
111 | CELERY = Celery('tasks',
112 | backend=REDIS_URL,
113 | broker=BROKER_URL)
114 | ```
115 |
116 | We then need to change the default serializer for results. Celery with versions 4.0 and above use JSON as a serializer, which doesn't support serialization of binary data. We can either switch back to the old default serializer (pickle) or use the newer MessagePack which supports binary data and is very efficient.
117 |
118 | Since we're changing the serializer, we also need to tell Celery to accept the results from a non-default serializer (as well as still accepting those from JSON).
119 |
120 | ```python
121 | CELERY.conf.accept_content = ['json', 'msgpack']
122 | CELERY.conf.result_serializer = 'msgpack'
123 | ```
124 |
125 | First, we'll implement a function that returns a jobs given an ID. This allows our app and the Celery tasks to talk to each other:
126 |
127 | ```python
128 | def get_job(job_id):
129 | return AsyncResult(job_id, app=CELERY)
130 | ```
131 |
132 | Next, we define the asynchronous function and move the image generation code from `app.py` and add the function decorator that allows the method to be queued for execution:
133 |
134 | ```python
135 | @CELERY.task()
136 | def image_demension(img):
137 | time.sleep(2)
138 | im = Image.open(img)
139 | width, height = im.size
140 | left = 4
141 | top = height / 5
142 | right = 154
143 | bottom = 3 * height / 5
144 |
145 | # Cropped image of above dimension \
146 | im1 = im.crop((left, top, right, bottom))
147 | newsize = (300, 300)
148 | im1 = im1.resize(newsize)
149 | width, height = im1.size
150 | location=os.path.join('static/worker-img','cropped_img.'+im.format.lower())
151 | im1.save(os.path.join('static/worker-img','cropped_img.'+im.format.lower()))
152 | print(width,height)
153 | print("pass")
154 |
155 | return location
156 | ```
157 |
158 | Instead of building a response, we return the binary image which will be stored on Redis. We also update the task at various points with a progress indicator that can be queried from the Flask app.
159 |
160 | We add a new route to `app.py` that checks the progress and returns the state as a JSON object so that we can write an ajax function that our client can query before loading the final image when it's ready.
161 |
162 | ```python
163 | @APP.route('/progress')
164 | def progress():
165 | jobid = request.values.get('jobid')
166 | if jobid:
167 | job = tasks.get_job(jobid)
168 | if job.state == 'PROGRESS':
169 | return json.dumps(dict(
170 | state=job.state,
171 | progress=job.result['current'],
172 | ))
173 | elif job.state == 'SUCCESS':
174 | return json.dumps(dict(
175 | state=job.state,
176 | progress=1.0,
177 | ))
178 | return '{}'
179 | ```
180 |
181 | Extend our `templates/download.html` with the following Javascript code:
182 |
183 | ```JavaScript
184 |
185 |
210 | ```
211 |
212 | The `poll` function repeatedly requires the `/progress` route of our web app and when it reports that the image has been generated, it replaces the HTML code within the placeholder with the URL of the image, which is then loaded dynamically from our modified `/result.png` route:
213 |
214 | ```python
215 | @APP.route('/result.png')
216 | def result():
217 | '''
218 | Pull our generated .png and return it
219 | '''
220 | jobid = request.values.get('jobid')
221 | if jobid:
222 | job = tasks.get_job(jobid)
223 | png_output = job.get()
224 | png_output="../"+png_output
225 | return png_output
226 | else:
227 | return 404
228 | ```
229 |
230 | At this stage we have a working web app with asynchronous image generation.
231 |
232 | ## Using Docker to package our application
233 |
234 | Our app requires 4 separate containers for each of our servies:
235 |
236 | - Flask
237 | - Celery
238 | - RabbitMQ
239 | - Redis
240 |
241 | Docker provides prebuilt containers for [RabbitMQ](https://hub.docker.com/_/rabbitmq/) and [Redis](https://hub.docker.com/_/redis/). These both work well and we'll use them as is.
242 |
243 | For Flask and Celery, we'll build two identical containers from a simple `Dockerfile`.
244 |
245 | ```bash
246 | # Pull the latest version of the Python container.
247 | FROM python:latest
248 |
249 | # Add the requirements.txt file to the image.
250 | ADD requirements.txt /app/requirements.txt
251 |
252 | # Set the working directory to /app/.
253 | WORKDIR /app/
254 |
255 | # Install Python dependencies.
256 | RUN pip install -r requirements.txt
257 |
258 | # Create an unprivileged user for running our Python code.
259 | RUN adduser --disabled-password --gecos '' app
260 | ```
261 |
262 | We pull all of this together with a Docker compose file, `docker-compose.yml`. While early versions of compose needed to expose ports for each service, we can link the services together using the `links` keyword. The `depends` keyword ensures that all of our services start in the correct order.
263 |
264 | To create and run the container, use:
265 |
266 | docker-compose build
267 | docker-compose up
268 |
269 | One of the major benefits of Docker is that we can run multiple instances of a container if required. To run multiple instances of our Celery consumers, do:
270 |
271 | docker-compose scale worker=N
272 |
273 | where N is the desired number of backend worker nodes.
274 |
275 | Visit http://localhost:5000 to view our complete application.
276 |
--------------------------------------------------------------------------------
