Development

The following sections are aimed at developers trying to contribute to MegaQC.

Please follow the Development guidelines to install a development ready version of MegaQC. You should also be aware of the detailed megaqc package and MegaQC API Reference. Next, some technical details regarding the front and backend are explained.

Backend

The Python backend is written in flask, which determines the structure of the project.

Database

MegaQC uses SQLAlchemy to handle database access, which means it can integrate with any SQL database that it supports. For development this will likely be SQLite.

Database models are located in model/models.py and user/models.py.

Database schema migrations

You need to generate a new migration whenever the database schema (ie any models class) changes. To generate a migration:

cd megaqc
export FLASK_APP=wsgi.py
flask db upgrade # Update to the latest migration
flask db migrate

API

MegaQC actually has two APIs. The first, older API is accessed at /api, and the code for this API is located in megaqc/api. This API is implemented using regular flask views.

However, all future development should be done on the newer REST API. This is accessed at /rest_api/v1, and the code for it is located in megaqc/rest_api. This API is composed of views, located in views.py. These view classes, which rely on flapison, each define an SQLAlchemy model that defines how to access the data, and a Marshmallow schema, which defines how to serialize and deserialize the data to JSON. The Marshmallow schemas themselves are defined in schemas.py. For more information, refer to the flapison documentation.

Views

Flask endpoints that return HTML (the non-API URLs) are defined in public/views.py, and user/views.py. All of these render a Jinja2 template, which all the other frontend CSS and JavaScript is connected to. This is further explained in the frontend docs.

Tests

Python tests are located in the python_tests folder. Please note that there are currently no Javascript tests. To run the Python tests, use pytest test. Every new URL should be tested, although since new pages are likely to rely on React and the REST API, testing can mostly be done inside test_api.py, which tests all REST API endpoints.

Frontend

As with the API, MegaQC is currently transitioning from using an HTML + CSS + JavaScript frontend to React. This can be confusing, as a number of technologies are mixed in the same project.

As explained in the backends section, each URL in MegaQC renders a Jinja2 template. The old-style endpoints have a lot of HTML code in their templates that defines the entire page layout, and load JavaScript and CSS located in the static directory.

However, the newer React pages instead render a special template, called react.html, which is a very simple page that acts as the entry point for the React code, which handles all layout and logic.

The source code for the React JavaScript can be found in the src directory, which currently has one root-level .js file for each React page that then imports other components to use.

Note that all new pages going forward should be written using React, to improve the maintainability of the frontend.

Documentation

MegaQC uses Sphinx to build the documentation and Github Pages to host it.

Building the documentation locally

The MegaQC documentation requires

  1. An installation of MegaQC to fetch the API endpoints and the Click commands.

  2. All dependencies specified in the docs requirements.txt. Install them by invoking: pip install -r docs/requirements.txt.

After having installed all requirements run make api-docs && make html in the docs directory. The generated html files are found in the docs/_build/html subfolder. Simply open a generated html file in your favorite browser to read the documentation.

Publishing the documentation

On pushes to the master branch, the documentation is automatically built and pushed to the gh-pages branch. The static html files on this branch are then deployed to Github Pages and displayed to the outside world. All of this is done with the Publish Docs Github Actions workflow <https://github.com/ewels/MegaQC/blob/master/.github/workflows/publish_docs.yaml>.