Future Maintainers ================================== Hello, if you are reading this it should be because you are interested in or have been assigned to modify / maintain / improve Polo. Here, I will do my best to fill you in on Polo's background and go through the problems, quirks and bugs that you may encounter while modifying Polo. Lore ####################################### Polo was created (mostly) over the course of 10 weeks for the 2020 BioXFEL summer internship. Development focused on creating a working final product that could be improved on and extended in the future, which is probably why you are here. Polo was my first real adventure into GUI programming and the PyQt library so if you have significant experience with either there are definitely quirks to be found and many places for improvement. Enjoy. Getting Started ####################################### Polo is written in Python (specifically Python 3.5) and uses the PyQt library as the GUI "engine". To get going modifying Polo I would recommend taking a look at the :ref:`Running From Source` section as it goes over how to setup a virtual environment for Polo development. Make sure you follow these instructions or do something equivalent as Polo is dependent on some legacy versions of common python packages and mixing those in with a global environment could get messy. I also **highly**, **highly** recommend installing PyQt Designer for your development machine. PyQt Designer will allow you to open the :code:`.ui` files located in the :code:`pyqt_designer` directory. The :code:`.ui` files are then translated by the :code:`pyuic5` command line program into Python scripts which define the layout of the graphical interface. I used `this tutorial `_ to install PyQt Designer and other devtools like pyuic5 on Ubuntu. This will allow you to graphically modify interfaces and easily find the names of widgets so you can make sure buttons and other interfaces get connected to the correct functions (widget names are, admittedly, a bit of a mess). Additionally, you can modify the :code:`UI_helper.py` script in the :code:`src` directory with the path to your :code:`pyqt_designer` directory to automatically generate the Python files which define each interface. Directory Layout ####################################### Polo directory structure just for fun. .. code-block:: text . ├── docs │   ├── _images │   ├── _modules │   │   ├── polo │   │   │   ├── crystallography │   │   │   ├── marco │   │   │   ├── threads │   │   │   ├── utils │   │   │   ├── widgets │   │   │   └── windows │   │   └── tests │   ├── _sources │   └── _static │   ├── css │   ├── fonts │   │   ├── Lato │   │   └── RobotoSlab │   └── js ├── misc │   └── inno_scripts │   └── windows ├── pyqt_designer ├── sphinx │   ├── _build │   └── images └── src ├── astor ├── data │   ├── cocktail_data │   ├── images │   │   ├── default_images │   │   ├── icons │   │   └── logos │   ├── savedmodel │   │   └── variables │   └── text ├── polo │   ├── cockatoo │   │   ├── cockatoo │   │   ├── data │   │   ├── docs │   │   │   ├── api │   │   │   ├── examples │   │   │   └── images │   │   ├── examples │   │   ├── screens │   │   │   ├── csv │   │   │   │   ├── hwi │   │   │   │   └── test-screens │   │   │   └── json │   │   │   ├── hwi │   │   │   └── test-screens │   │   └── tests │   ├── crystallography │   │   │   ├── designer │   │   │   ├── marco │   │   │   ├── plots │   │   │   ├── threads │   │   │   ├── utils │   │   │   ├── widgets │   │   │   └── windows │   ├ ├── templates │   └── static ├── tests └── unrar ├── Darwin └── Windows ├── Win32 └── Win64 src Directory ------------------------- The src directory includes all Python scripts Polo needs to run plus any data required, such as the MARCO tensorflow model. Below are short descriptions of what you'll find in this directory. - :code:`Polo.py` : Main script. Use to launch the application. - :code:`UI_helper.py` : Helper script to convert :code:`.ui` files to :code:`.py` files. See :ref:`Getting Started`. - :code:`polo` : The Polo package. Contains all scripts that define Polo behavior and interfaces. - :code:`data` : Contains all data required for Polo to run. Such as cocktail csv files, the MARCO model and icon images - :code:`unrar` : Contains unrar executables for Windows and Mac - :code:`templates` : Jinja2 html templates for creating html file exports - :code:`astor` : I kept getting an error from Tensorflow that this directory could not be found so I included it and the problem was solved misc Directory ------------------------ Contains odds and ends notes or other files that didn't really have a place anywhere else. One of the most useful may be the :code:`polo.iss` script. This is an inno setup script I used for created the Polo windows installer. Feel free to use it, but make sure to modify the filepaths in it before running. pyqt_designer Directory --------------------------- Holds all :code:`.ui` files, which are really :code:`.xml` files that PyQt Designer uses to serialize the interfaces you design in it. They are translated to Python files which then define the graphical interface for Polo. See the :ref:`Getting Started` section for a bit more detail on this. sphinx Directory ---------------------- Contains rst files and images which define Polo's documentation. With sphinx installed via pip you can recreate Polo's documentation with the command :code:`make html`. For these files to be rendered on the GitHub pages site they need to be places into the :code:`docs` directory. You will also need to install the read the docs theme for sphinx. docs Directory ------------------------ Holds documentation html files. Rendered as the Polo website using GitHub sites. Polo Package Notes ####################################### All the scripts that make up the actual Polo program are located in :code:`src/Polo`. In this section I will go over a few important aspects of the program that may be helpful and are not covered explicitly in the API documentation. Notable Scripts ---------------------------- The :code:`__init__` Script +++++++++++++++++++++++++++++ Most variables that are used across Polo scripts are defined in here and is worth taking a look at as the stuff in here comes up all over other scripts. The items below are some of the major functions of the :code:`__init__` : file. 1. Defines filepaths to the stuff in the :code:`data`: 2. Defines the version of the program via the :code:`polo_version`: variable 3. Defines the image classification and image spectrum keywords 4. Defines how MSO classification codes are translated into MARCO classifications 5. Determines which unrar executable to use based on the OS 6. Defines regular expressions used for parsing cocktail data 7. Defines urls to Polo documentation site pages The :code:`windows/main_window` Script +++++++++++++++++++++++++++++++++++++++++ The :code:`windows/main_window.py` script defines the :class:`MainWindow` class from which all other widgets are staged within. If you are looking for how one widget communicates to another, how menu selections are handled or how runs are opened this would be the file to look in. Since the :class:`MainWindow` class contains most other widgets in some way it is a good place to look when relationships between widgets are in question. Subpackages ####################################### Crystallography Subpackage ---------------------------- Contains scripts relating to crystallography and high-throughput imaging. Most of the main data containing classes are defined here. Designer Subpackage ---------------------------- Contains all the pyqt Designer generated UI scripts. These are used to define the graphical interfaces (buttons, knobs, etc.) that make up the widgets and dialogs defined in Polo. These scripts do not provide any functionality to the graphical components, they just define their layout and names. Marco Subpackage ---------------------------- Holds scripts relating to running the MARCO model for image classification. There is really only one important function in here which classifies an image passed to it. Plots Subpackage ---------------------------- Functions related to generating plots shown in the plots tab of the main window. This one is kind of a todo I didn't get around to and there is a lot of clean up that could be done here. Especially since plots does not have its own widget like the slideshow inspector or plate inspectors. Threads Subpackage ---------------------------- A lot of the operations Polo undertakes, like running the MARCO model, are CPU intense and cannot be run on the same thread that the GUI is run on. The threads package holds the :code:`thread.py` script which defines QThread objects for running various tasks outside of the GUI thread. This prevents the freezing the GUI when preforming a large operation. Windows is particularly fast to recognize a frozen program so it is often necessary to put tasks that take more than a few seconds onto a thread. Utils Subpackage ---------------------------- TODO Widgets Subpackage ---------------------- TODO Windows Subpackage ----------------------- TODO Creating Exes for Distribution ####################################### Once you have made some modifications to your program you are going to want to create exe for potential users. I used Pyinstaller to create the exe files and then on Windows Inno setup to create an installer and will cover those topic in this section. Pyinstaller Overview and Usage Guide ---------------------------------------- Pyinstaller is not included as a dependency in the :code:`requirements.txt` file so will need to install it using Pip. Polo includes a :code:`.spec` file in the outermost directory. This is the file I used on Ubuntu, Mac and Windows to generate exe files. It should be noted that the exe will be specific to the operating system you create it on. A Polo exe created on Windows will only work on Windows machines. Despite it's file extension, the :code:`.spec` file is really a python script that passes information along to Pyinstaller. Running Pyinstaller using the :code:`.spec` file is actually very easy and can be done with the command :code:`pyinstaller Polo.spec`. Additionally, I have made some changes to the :code:`.spec` which allow you to generate one file exes by appending :code:`F` onto the end of the pyinstaller command. However, you will likely need to make some modifications to the :code:`.spec` before you run it. Spec File Details and Modifications -------------------------------------- Before running the :code:`.spec` file you will need to modify a couple file paths based on the locations on your development machine(s). First, you'll notice the dictionary :code:`tensorflow_location`. Much of the :code:`.spec` is devoted to dealing with packing Tensorflow 1.14 as Pyinstaller misses the binary files in the library that are required for the everything to work correctly. Therefore these files need to be collected and passed to Pyinstaller explicitly to create a working exe. Strangely, I only encountered this problem on Linux and Mac. The :code:`tensorflow_location` dictionary specifies the location of the Tensorflow package on your machine. You will need to modify these paths to the Tensorflow package being used by your virtual environment. Next, you'll need to modify the :code:`polo_locations` dictionary values. This dictionary maps operating system types to the location of the Polo repository on that machine. When run, the :code:`.spec` file recognizes the OS it is being run on and picks the file path corresponding to that OS. After making these modifications you should be able to run the :code:`.spec` file successfully. If something is not working correctly I recommend running the Pyinstaller command without :code:`F` argument to create a directory instead of a single file. This will let you more easily see exactly what Pyinstaller has included in your distribution. If it isn't in the directory distribution it won't be found in the single file distribution. Pyinstaller will generate two new directories :code:`build` and :code:`dist`. You can ignore :code:`build` (I did and faced no wrath), the exe will be located in code :code:`dist`. I have also found that if Pyinstaller is acting up it can help to delete both these directories and rerunning the :code:`spec` file. Editing and Extending this Documentation ########################################### Documentation Background ------------------------------ All Polo documentation is written using the RST (restructured text) markdown language. I find myself going back to this `cheat sheet `_ when I have syntax questions but overall it is very similar to GitHub markdown. The code (API) documentation and html is generated using Sphinx Python package which you can read more about at the `Sphinx website `_. .. note:: Everything below has been done on Ubuntu (Linux) OS. Results may vary on Windows or Mac. Editing the Docs ----------------------------- You can edit the documentation in two main ways. First, directly editing the rst files located in the :code:`sphinx` directory of the Polo repository. This is the best way to edit pages like the :ref:`User Guide` or :ref:`Installation Guide` that are not generated automatically. The second way is through the docstrings of Polo functions. Sphinx collects these docstrings and creates the code documentation from them. Either way, you will need to render your rst / Python files to HTML which actually forms the documentation website. To learn how, read on. Creating HTML Files ------------------------------- 1. If you have not done so already, install the Sphinx package and the Read the Docs theme. Follow the instructions in the `Sphinx Installation Guide `_ to install Sphinx and use the command :code:`pip install sphinx_rtd_theme` to install the RTD theme. You should use whatever virtual environment you are using while working on Polo as all the dependencies required to run Polo will be required to run Sphinx. 2. Navigate to the :code:`sphinx` directory of the Polo repository. Run the command :code:`make html`. This should collect docstrings and render your RST files to html. They will be places in the :code:`sphinx/_build/html` directory of the Polo repository. 3. Checkout the changes you made by opening the HTML files. If everything looks good move all files to the :code:`docs` folder of the repository and commit your changes. If GitHub pages is still being used to host the documentation website, the changes should come online in a few minutes. If some other hosting solution is in use, I can guide you no further. Debugging --------------------------- A good place to start if things go weird in creating the documentation is the :code:`conf.py` file located in the :code:`sphinx` directory of the Polo repository. It defines filepaths and assets that are used when rendering so it is a good place to start looking for issues.