My Python boilerplate and a little Python-Fu

Let me show you my Python boilerplate and some little nifty standard features I’m working with. You can download the full template from github . This post will probably be extended in the future.

Summary

The boilerplate will help you with the following features:

Use virtual environments
manage requirements
notifications via Telegram
manage command line arguments
list comprehension
print progress in loops
measure time
play a sound after a loop is finished

Virtual environment

I like working with virtual environments. It helps me to work with individual clean development environments. How do you do that? Switch to your main development folder, create a subfolder for your new project and initiate it:

1cd /development/
2mkdir project1
3python3 -m venv project1
4cd project1
5source bin/activate

There you go. If you install packages, they are located in this folder only. Wait, packages? Right, let’s get to the next step:

This post comes without any images, that’s why I’m showing you this beautiful image that I took in Athens, a couple of years ago — a wonderful city!

Manage your libraries

As your Python scripts grows, by the amount of lines but also by the amount of features, the amount of used libraries will grow, too. But how to run the script on a different machine, in a differente environment? How to easily share you script? The answer is pipreqs:

1pip install pipreqs

After you installed this little tool, run it against your script (or a folder of scripts):

1pipreqs yourscript.py # one script only
2pipreqs . # all scripts in the folder

Pipreqs will scan your script for required libraries and list them in a file called requirements.txt (suprise!). You need to share this text file with your Python script. Everyone who wants to run the script needs to prepare the environment using this line of code:

1pip install -r requirements.txt

Et voilà. All requirements met.

Telegram logging

Please be adviced: Using a centralized messaging service like telegram, where your content is hosted on foreing computers, may be a risky plan. I’m fully aware of that, you should be too. That’s why I’m only sharing non-senstive information with telegram. It’s just my perfect notification service.

How does telegram help? You may ask. Well, imagine you have a script running hours or even days! Actually, you don’t really know. Wouldn’t it be nice to get a message, when the script is done, or about it’s progress.

First let’s create your own bot for Telegram. Open the app and look for “BotFather”. He will allow you to create a new bot. Name doesn’t matter, what does matter is the secret(!) token. Take it and start a chat with your bot. Then browse to this url (replace the token part with your secret(!) token):

1https://api.telegram.org/bot<bot token>/getUpdates

The page will return a JSON object with a couple of information, amongst it is your chat-id (it’s secret, too, everything is secret!). Grab it and put it with your secret (!!!) token into your telegram_notifier.py:

 1import requestsbot_token = 'secret token'
 2bot_chat_id = 'secret chat id'def send_text_to_telegram(bot_message):
 3   bot_token = bot_token
 4   
 5   send_text = '[https://api.telegram.org/bot'](https://api.telegram.org/bot') + bot_token + '/sendMessage?chat_id=' + bot_chat_id + '&parse_mode=Markdown&text=' + bot_message   response = requests.get(send_text)
 6 
 7   return response.json()def send_image_to_telegram(image):   bot_token = bot_token
 8 
 9   data = {'chat_id': bot_chat_id}
10   
11   url = '[https://api.telegram.org/bot'](https://api.telegram.org/bot') + bot_token + '/sendPhoto?chat_id=' + bot_chat_id + '&parse_mode=Markdown'
12 
13   with open(image, "rb") as image_file:
14 
15      response = requests.post(url, data=data, files={"photo": image_file})
16  
17   return response.json()

Please note, that there’s no error handling with this script.

Congratulations, you now have two functions to send text and images to your telegram bot. This will help you to get a notification from your Python scripts. Import the two functions with this line in every of your actual scripts:

1from telegram_notifier import *

In media res

Command line parameters

Enough prepping, let’s talk about the script itself. The first lines look like that:

1import time # for logging purposes
2import argparse # for command line argumentsparser=argparse.ArgumentParser()parser.add_argument('--limit', help='Source files in CSV format', required=True, type=int)
3parser.add_argument('--loop_length', help='Source files in CSV format', required=False, type=int)
4parser.add_argument('--message', help='Give me a message', required=False, default="Hello World", type=str)args = parser.parse_args()

You’ll find it helpful to pass arguments to your script from the command line. That’s what the argparse library is for. This library supports loads of other parameters and features, I’m just showing you some basic ones to read two parameters limit and loop_length. You can read those parameters using args.limit or args.loop_length. So easy. Example?

1loop_length = 1000000 if args.loop_length == None else args.loop_length
2limit = args.limit

That’s the shortend version of a if-condition. If the loop-length is set, take it. Otherwise use the default value of 1.000.000. The limit is always set to the one passed by the user — because it’s a mandatory parameter and therefore always present. Next, please.

Logging

Why is logging so important to me? Why sould it be important to you? You will get to that point, when a customer asks you:

How long will it take to process this dataset with 2 Mio. rows? How much will it costs?

~a customer, somewhen

Or if you want to compare two different methods. And so on. I’m a huge fan of (extensive) logging. It’s very helpful. Trust me. So, how do we start?

1feedback_frequency = 10                                               count = 1  
2start = time.time()

Frequency? Why? Imagine you have a script that crunches 1 Mio. rows. And depending on the surrounding parameters, it’s either pretty fast or pretty slow. When do you create a log message? With the feedback_frequency you define, how many times you want to see a log message. Set it to 10 to get a feedback every 1 Mio. / 10 rows. Increase it to get even more feedback.

Colors

If you are sending a lot of information to the console, you may want to use colored output. First let’s define a custom class for that. As you can see, this also allows you to set the font style to bold or underline it.

 1class bcolors:
 2    HEADER = '\033[95m'
 3    OKBLUE = '\033[94m'
 4    OKCYAN = '\033[96m'
 5    OKGREEN = '\033[92m'
 6    WARNING = '\033[93m'
 7    FAIL = '\033[91m'
 8    ENDC = '\033[0m'
 9    BOLD = '\033[1m'
10    UNDERLINE = '\033[4m'

Now you implement the class into your output like that:

1print(f'{bcolors.FAIL}{bcolors.BOLD}Bold red text{bcolors.ENDC}')

A simple loop — comprehension

The list or dict comprehension feature is a feature you either love or don’t understand. I’m not even sure if I really understand the feature. Nevertheless, here’s an example that hopefully explains it.

List comprehension works for dictonaries, too. It allows you to loop through data without writing a fully fledged loop. And its supposed to be faster than a common loop.

The following example goes through a range of numbers from 1 to loop_length=1.000.000 (second last line).

The last line in this comprehension says: Only process the current row, if index is greater than 10.

The first line just takes the index value. Work with it. Add it, round it, calculate it. That’s the value, that goes to our list “a_list”.

The second line is the second way to work with conditions: Depending on the acutal value of index, it defines an outcome. If index is less than 50, pass it on. Else pass on the value defined in default_value.

1default_value = 'x'
2a_list = [    index
3    if index < 50 else default_value
4    for index in range(1, loop_length)
5    if index > 10
6    ]

I know it’s tricky. You will get to it, as soon as you need it. ;P

Logging, actually

The next lines will send one message to telegram and to the standard output.

1duration = time.time() - start            message = '\r\nDone with list comprehension after %s seconds ' % (      '{:.2f}'.format(duration) # two decimal places
2)
3                                               send_text_to_telegram(message)                                               print(message, flush=True)

You should understand this one from the code, except the flush-Parameter? This one forces the Python-script to send the message to the output, without waiting for the script to end. That’s your real-time-update-guarantee!

The fully-fledged Loop

This is a full loop, that allows you to add as many conditions and calculations as you want.

 1for i in range(1, loop_length):   if i > limit - 1 and limit > 0: break
 2   
 3   count += 1   if i % (limit / feedback_frequency) == 0 or i % (round(loop_length / feedback_frequency, 0)) == 0:
 4        
 5      progress = i / limit if limit > 0 else i / loop_length      message = (
 6            '%s%%...' % 
 7            ('{:.1f}'.format(100 * progress))
 8        )      print(
 9            message,            
10            end = '', # no line break
11            flush=True # don't buffer output
12      )

The first lines are easy. Break the loop, if the limit is hit. The most exciting thing is the logging-feature. As mentioned above, it will only logg a message if it meets the frequency requirements. Also check out the end-Parameter of print. It removes the line break from the print-command.

Final message

Finally, after our script logic is finished, we send a message. This time to telegram, too.

1duration = time.time() - startmessage = '\r\nDone with processing %s iterations after %s seconds ' % (
2    '{:0,.0f}'.format(count), # formatting integers with thousand separator
3    '{:.2f}'.format(duration) # formatting decimals with two decimals
4    )send_text_to_telegram(message)print(message)

There’s nothing special happening here. The only interesting thing is, that Telegram also accepts images. This is good if you’re creating some sophisticated statistcial analysis and want to see the output. Again, a word of advise: Remember that Telegram is storing your messages on foreign servers. Keep that in mind.

Last words

I’m always learing, and I hope your learned something, too. I’m going to extend this post in the future. If you have further tricks for me, drop me a dove.