Future Maintainers

Hello, if you are reading this it should be because you are interested in or have been assigned to modify / maintain / improve Polo. Here, I will do my best to fill you in on Polo’s background and go through the problems, quirks and bugs that you may encounter while modifying Polo.

Lore

Polo was created (mostly) over the course of 10 weeks for the 2020 BioXFEL summer internship. Development focused on creating a working final product that could be improved on and extended in the future, which is probably why you are here. Polo was my first real adventure into GUI programming and the PyQt library so if you have significant experience with either there are definitely quirks to be found and many places for improvement. Enjoy.

Getting Started

Polo is written in Python (specifically Python 3.5) and uses the PyQt library as the GUI “engine”. To get going modifying Polo I would recommend taking a look at the Running From Source section as it goes over how to setup a virtual environment for Polo development. Make sure you follow these instructions or do something equivalent as Polo is dependent on some legacy versions of common python packages and mixing those in with a global environment could get messy.

I also highly, highly recommend installing PyQt Designer for your development machine. PyQt Designer will allow you to open the .ui files located in the pyqt_designer directory. The .ui files are then translated by the pyuic5 command line program into Python scripts which define the layout of the graphical interface.

I used this tutorial to install PyQt Designer and other devtools like pyuic5 on Ubuntu.

This will allow you to graphically modify interfaces and easily find the names of widgets so you can make sure buttons and other interfaces get connected to the correct functions (widget names are, admittedly, a bit of a mess). Additionally, you can modify the UI_helper.py script in the src directory with the path to your pyqt_designer directory to automatically generate the Python files which define each interface.

Directory Layout

Polo directory structure just for fun.

.
├── docs
│   ├── _images
│   ├── _modules
│   │   ├── polo
│   │   │   ├── crystallography
│   │   │   ├── marco
│   │   │   ├── threads
│   │   │   ├── utils
│   │   │   ├── widgets
│   │   │   └── windows
│   │   └── tests
│   ├── _sources
│   └── _static
│       ├── css
│       ├── fonts
│       │   ├── Lato
│       │   └── RobotoSlab
│       └── js
├── misc
│   └── inno_scripts
│       └── windows
├── pyqt_designer
├── sphinx
│   ├── _build
│   └── images
└── src
    ├── astor
    ├── data
    │   ├── cocktail_data
    │   ├── images
    │   │   ├── default_images
    │   │   ├── icons
    │   │   └── logos
    │   ├── savedmodel
    │   │   └── variables
    │   └── text
    ├── polo
    │   ├── cockatoo
    │   │   ├── cockatoo
    │   │   ├── data
    │   │   ├── docs
    │   │   │   ├── api
    │   │   │   ├── examples
    │   │   │   └── images
    │   │   ├── examples
    │   │   ├── screens
    │   │   │   ├── csv
    │   │   │   │   ├── hwi
    │   │   │   │   └── test-screens
    │   │   │   └── json
    │   │   │       ├── hwi
    │   │   │       └── test-screens
    │   │   └── tests
    │   ├── crystallography
    │   │
    │   ├── designer
    │   │
    │   ├── marco
    │   │
    │   ├── plots
    │   │
    │   ├── threads
    │   │
    │   ├── utils
    │   │
    │   ├── widgets
    │   │
    │   └── windows
    │
    ├
    ├── templates
    │   └── static
    ├── tests
    └── unrar
        ├── Darwin
        └── Windows
            ├── Win32
            └── Win64

src Directory

The src directory includes all Python scripts Polo needs to run plus any data required, such as the MARCO tensorflow model. Below are short descriptions of what you’ll find in this directory.

• Polo.py : Main script. Use to launch the application.

• UI_helper.py : Helper script to convert .ui files to .py files. See Getting Started.

• polo : The Polo package. Contains all scripts that define Polo behavior and interfaces.

• data : Contains all data required for Polo to run. Such as cocktail csv files, the MARCO model and icon images

• unrar : Contains unrar executables for Windows and Mac

• templates : Jinja2 html templates for creating html file exports

• astor : I kept getting an error from Tensorflow that this directory could not be found so I included it and the problem was solved

misc Directory

Contains odds and ends notes or other files that didn’t really have a place anywhere else. One of the most useful may be the polo.iss script. This is an inno setup script I used for created the Polo windows installer. Feel free to use it, but make sure to modify the filepaths in it before running.

pyqt_designer Directory

Holds all .ui files, which are really .xml files that PyQt Designer uses to serialize the interfaces you design in it. They are translated to Python files which then define the graphical interface for Polo. See the Getting Started section for a bit more detail on this.

sphinx Directory

Contains rst files and images which define Polo’s documentation. With sphinx installed via pip you can recreate Polo’s documentation with the command make html. For these files to be rendered on the GitHub pages site they need to be places into the docs directory. You will also need to install the read the docs theme for sphinx.

docs Directory

Holds documentation html files. Rendered as the Polo website using GitHub sites.

Polo Package Notes

All the scripts that make up the actual Polo program are located in src/Polo. In this section I will go over a few important aspects of the program that may be helpful and are not covered explicitly in the API documentation.

Notable Scripts

The __init__ Script

Most variables that are used across Polo scripts are defined in here and is worth taking a look at as the stuff in here comes up all over other scripts. The items below are some of the major functions of the __init__ : file.

1. Defines filepaths to the stuff in the data:

2. Defines the version of the program via the polo_version: variable

3. Defines the image classification and image spectrum keywords

4. Defines how MSO classification codes are translated into MARCO classifications

5. Determines which unrar executable to use based on the OS

6. Defines regular expressions used for parsing cocktail data

7. Defines urls to Polo documentation site pages

The windows/main_window Script

The windows/main_window.py script defines the MainWindow class from which all other widgets are staged within. If you are looking for how one widget communicates to another, how menu selections are handled or how runs are opened this would be the file to look in. Since the MainWindow class contains most other widgets in some way it is a good place to look when relationships between widgets are in question.

Subpackages

Crystallography Subpackage

Contains scripts relating to crystallography and high-throughput imaging. Most of the main data containing classes are defined here.

Designer Subpackage

Contains all the pyqt Designer generated UI scripts. These are used to define the graphical interfaces (buttons, knobs, etc.) that make up the widgets and dialogs defined in Polo. These scripts do not provide any functionality to the graphical components, they just define their layout and names.

Marco Subpackage

Holds scripts relating to running the MARCO model for image classification. There is really only one important function in here which classifies an image passed to it.

Plots Subpackage

Functions related to generating plots shown in the plots tab of the main window. This one is kind of a todo I didn’t get around to and there is a lot of clean up that could be done here. Especially since plots does not have its own widget like the slideshow inspector or plate inspectors.

Threads Subpackage

A lot of the operations Polo undertakes, like running the MARCO model, are CPU intense and cannot be run on the same thread that the GUI is run on. The threads package holds the thread.py script which defines QThread objects for running various tasks outside of the GUI thread. This prevents the freezing the GUI when preforming a large operation. Windows is particularly fast to recognize a frozen program so it is often necessary to put tasks that take more than a few seconds onto a thread.

Utils Subpackage

TODO

Widgets Subpackage

TODO

Windows Subpackage

TODO

Creating Exes for Distribution

Once you have made some modifications to your program you are going to want to create exe for potential users. I used Pyinstaller to create the exe files and then on Windows Inno setup to create an installer and will cover those topic in this section.

Pyinstaller Overview and Usage Guide

Pyinstaller is not included as a dependency in the requirements.txt file so will need to install it using Pip.

Polo includes a .spec file in the outermost directory. This is the file I used on Ubuntu, Mac and Windows to generate exe files. It should be noted that the exe will be specific to the operating system you create it on. A Polo exe created on Windows will only work on Windows machines. Despite it’s file extension, the .spec file is really a python script that passes information along to Pyinstaller.

Running Pyinstaller using the .spec file is actually very easy and can be done with the command pyinstaller Polo.spec. Additionally, I have made some changes to the .spec which allow you to generate one file exes by appending F onto the end of the pyinstaller command. However, you will likely need to make some modifications to the .spec before you run it.

Spec File Details and Modifications

Before running the .spec file you will need to modify a couple file paths based on the locations on your development machine(s).

First, you’ll notice the dictionary tensorflow_location. Much of the .spec is devoted to dealing with packing Tensorflow 1.14 as Pyinstaller misses the binary files in the library that are required for the everything to work correctly. Therefore these files need to be collected and passed to Pyinstaller explicitly to create a working exe. Strangely, I only encountered this problem on Linux and Mac. The tensorflow_location dictionary specifies the location of the Tensorflow package on your machine. You will need to modify these paths to the Tensorflow package being used by your virtual environment.

Next, you’ll need to modify the polo_locations dictionary values. This dictionary maps operating system types to the location of the Polo repository on that machine. When run, the .spec file recognizes the OS it is being run on and picks the file path corresponding to that OS.

After making these modifications you should be able to run the .spec file successfully. If something is not working correctly I recommend running the Pyinstaller command without F argument to create a directory instead of a single file. This will let you more easily see exactly what Pyinstaller has included in your distribution. If it isn’t in the directory distribution it won’t be found in the single file distribution.

Pyinstaller will generate two new directories build and dist. You can ignore build (I did and faced no wrath), the exe will be located in code dist. I have also found that if Pyinstaller is acting up it can help to delete both these directories and rerunning the spec file.

Editing and Extending this Documentation

Documentation Background

All Polo documentation is written using the RST (restructured text) markdown language. I find myself going back to this cheat sheet when I have syntax questions but overall it is very similar to GitHub markdown. The code (API) documentation and html is generated using Sphinx Python package which you can read more about at the Sphinx website.

Note

Everything below has been done on Ubuntu (Linux) OS. Results may vary on Windows or Mac.

Editing the Docs

You can edit the documentation in two main ways. First, directly editing the rst files located in the sphinx directory of the Polo repository. This is the best way to edit pages like the User Guide or Installation Guide that are not generated automatically. The second way is through the docstrings of Polo functions. Sphinx collects these docstrings and creates the code documentation from them. Either way, you will need to render your rst / Python files to HTML which actually forms the documentation website. To learn how, read on.

Creating HTML Files

1. If you have not done so already, install the Sphinx package and the Read the Docs theme. Follow the instructions in the Sphinx Installation Guide to install Sphinx and use the command pip install sphinx_rtd_theme to install the RTD theme. You should use whatever virtual environment you are using while working on Polo as all the dependencies required to run Polo will be required to run Sphinx.

2. Navigate to the sphinx directory of the Polo repository. Run the command make html. This should collect docstrings and render your RST files to html. They will be places in the sphinx/_build/html directory of the Polo repository.

3. Checkout the changes you made by opening the HTML files. If everything looks good move all files to the docs folder of the repository and commit your changes. If GitHub pages is still being used to host the documentation website, the changes should come online in a few minutes. If some other hosting solution is in use, I can guide you no further.

Debugging

A good place to start if things go weird in creating the documentation is the conf.py file located in the sphinx directory of the Polo repository. It defines filepaths and assets that are used when rendering so it is a good place to start looking for issues.