Continuous Integration and Continuous Deployment

Welcome to the CI/CD documentation of the Junqo-platform.
This documentation is intended for developers who want to contribute to the project.

Table of contents

Deployment

The deployment of the Junqo-platform is done using Docker and Docker Compose.

Prerequisites

To deploy the Junqo-platform, you need to have the following tools installed on your machine:

If you don’t have these tools installed, you can follow the installation instructions on the official websites of Git, Docker and Docker Compose.

Once you have installed these tools, you can proceed to the deployment of the Junqo-platform.

The first step to deploy the Junqo-platform is to clone or download the repository and move to the project directory:

# Clone the repository
git clone git@github.com:Junqo-org/junqo-platform.git

# Move to the project directory
cd junqo-platform

Deploy using Docker Compose

Once the project is cloned and you are in the project directory, you need to setup config files.

Config files

First, you need to create a db_password.conf file at the root of the project. You can use the db_password_example.conf file to create the new one. This file contain the password for the database user.

Then, you need to create the junqo_back/.env file to configure the backend. You can use the junqo_back/exemple.env file to create the new one.

For more informations, see the backend configuration documentation.

After creating the db_password.conf and the junqo_back/.env files, you can proceed to deploy the Junqo-platform using Docker Compose.

Development deployment

To deploy the Junqo-platform in development mode, you can use the following command:

# Deploy using Docker Compose
docker compose -f docker-compose.dev.yaml up --watch

# You can use the --build flag to rebuild the images
docker compose -f docker-compose.dev.yaml up --build --watch

This command will deploy the Junqo-platform in development mode and watch for changes in the source code. Allowing you to develop and test the application in real-time. For more information on the watch flag, you can refer to the Docker Compose documentation.

The development deployment has several differences from the production deployment:

  • The back server is running in development mode. And the watch flag is enabled.
  • The front server is running in development mode. And the watch flag is enabled.
  • The database adminer is deployed. Allowing you to access the database adminer at http://localhost:3000. (The port can be changed using the ADMINER_PORT environment variable)

Production deployment

To deploy the Junqo-platform in production mode, you first need to setup the ssl certificates.

For detailed instructions, visit the official Certbot documentation.

First, install Certbot (without the nginx plugin as we manage nginx config manually):

sudo apt update
sudo apt install certbot -y

Create the ACME challenge directory that nginx expects:

# Create the directory structure
sudo mkdir -p /var/www/letsencrypt/.well-known/acme-challenge

# Set correct permissions
# Since nginx runs in Docker, use root ownership with world-readable permissions
sudo chown -R root:root /var/www/letsencrypt
sudo chmod -R 755 /var/www/letsencrypt

Note: The directory is owned by root because the nginx reverse proxy runs inside a Docker container. The 755 permissions allow the containerized nginx to read the challenge files while Certbot (running as root on the host) can write to them.

Test that the challenge directory is accessible:

# Create a test file
echo "test" | sudo tee /var/www/letsencrypt/.well-known/acme-challenge/test.txt

# Test HTTP access (replace yourdomain.com with your actual domain)
curl http://yourdomain.com/.well-known/acme-challenge/test.txt

# Should return "test"
# Clean up test file
sudo rm /var/www/letsencrypt/.well-known/acme-challenge/test.txt

Then, obtain an SSL certificate using webroot mode:

# Replace yourdomain.com with your actual domain
sudo certbot certonly \
  --webroot \
  -w /var/www/letsencrypt \
  -d yourdomain.com \
  --email your-email@example.com \
  --agree-tos \
  --non-interactive

Certbot will:

  • Place challenge files in /var/www/letsencrypt/.well-known/acme-challenge/
  • Verify domain ownership via HTTP
  • Store certificates in /etc/letsencrypt/live/yourdomain.com/
  • Set up auto-renewal

Important: The nginx reverse proxy configuration (nginx.rproxy.conf) is already set up to serve these challenge files and use the certificates. No manual nginx configuration changes are needed.

Verify the certificate installation:

# Check certificate details
sudo certbot certificates

# Verify certificate files exist
sudo ls -la /etc/letsencrypt/live/yourdomain.com/

Verify the auto-renewal timer is active:

sudo systemctl status certbot.timer

Your SSL certificates will be automatically renewed before expiry.

To test the renewal process:

sudo certbot renew --dry-run

After setting up SSL, you can deploy the production environment:

# Deploy using Docker Compose
docker compose up -d

The production deployment has several differences from the development deployment:

  • All the requests are routed through the Nginx reverse proxy.
  • The back server is running in production mode.
  • The front server is running in production mode.
  • The database volume is set to the docker volume db_data.
  • The database adminer is available only through a ssh tunnel at localhost:3000.

Configuration

Environment Variables

The Junqo-platform uses environment variables to store the configuration of the project.

You can add a .env file at the root of the project to store the environment variables. If an environment variable is not found, the default value will be used.

Here is the list of environment variables used by the Junqo-platform:

  • FLUTTER_VERSION: The version of Flutter to use. Default value is 3.35.3.
  • BACK_PORT: The port of the back server. Default value is 4200.
  • DATABASE_SHM_SIZE: The size of the shared memory for the database container. Default value is 256MB.
  • DATABASE_USER: The user of the database. Default value is junqo.
  • DATABASE_NAME: The name of the database. Default value is junqo.
  • DATABASE_PASSWORD_FILE: The path to the file containing the password of the database user. Default value is ./db_password.conf.
  • GRAFANA_PASSWORD_FILE: The path to the file containing the Grafana admin password. Default value is ./grafana_password.conf.
  • API_URL: The URL of the back server used in the frontend. Default value is http://localhost:4200.
  • SSL_CERTS_PATH: The path to the folder containing the SSL certificates. Default value is /etc/letsencrypt.

The following are only available in development mode:

  • ADMINER_PORT: The port of the database adminer. Default value is 3000.
  • ADMINER_DESIGN: The design of the database adminer. Default value is pepa-linha-dark.

Configuration Files

The Junqo-platform uses configuration files to store the configuration of the project.

The nginx.conf file of the junqo front contains the configuration of the Nginx server. You can modify this file to fit your needs.

Secret Files

The Junqo-platform uses secret files to store sensitive information. These files must not be committed to the repository.

The secret files are:

  • db_password.conf: Contains the password for the database user. (The file path may defer depending on the environment variable DATABASE_PASSWORD_FILE)
  • grafana_password.conf: Contains the admin password for Grafana. (The file path may defer depending on the environment variable GRAFANA_PASSWORD_FILE)

Automatic tests

Automatic tests are run using the Github Actions pipeline.
These are defined in the .github/workflows folder.

Run github action locally

You can use the nektosact to run the tests locally.
First, you need to install nektosact by following the installation instructions.
Then run the following command to execute specific workflow tests:
Note: act requires root privileges to run Docker containers, hence the sudo requirement.

# Run the tests specified in the <specific file>
sudo act workflow_dispatch -W '.github/workflows/<specific_file>' -e tools/act-event-file.json -q
# Example
sudo act workflow_dispatch -W '.github/workflows/deployment-tests.yml' -e tools/act-event-file.json -q --concurrent-jobs 1

The tools/act-event-file.json file contains the event payload for the workflow. You can modify this file to change the event payload. The q flag is used to run the workflow in quiet mode. Remove it to see the logs. The concurrent-jobs flag is used to limit the number of concurrent jobs. Here it is set to 1 to avoid issues with same ports being used by multiple jobs.

Deployment tests

These tests are used to test the deployment of the Junqo-platform in development and production environments.

The deployment tests are run using the deployment-tests.yml workflow.
The workflow is triggered when the following conditions are met:

  • A pull request is opened.
  • The pull request merges into the main or dev branch.
  • The files inside the junqo_front or junqo_back folder are modified.

The workflow runs the following steps:

  1. Test the development environment deployment.
  2. Test the production environment deployment.

These tests are run using th deployment-test-template.yml workflow.
The workflow runs the following steps:

  1. Checkout the repository.
  2. Set up the environment variables.
  3. Generate SSL certificates.
  4. Start the docker compose services.
  5. Check if the services are up and running.
  6. Stop the docker compose services.

Front tests

These tests are used to test the front-end of the Junqo-platform.

The front tests are run using the front-tests.yml workflow. The workflow is triggered when the following conditions are met:

  • A pull request is opened.
  • The pull request merges into the main or dev branch.
  • The files inside the junqo_front folder are modified.

The workflow runs the following steps:

  1. Checkout the repository.
  2. Install Flutter.
  3. Install the dependencies.
  4. Run the front tests using flutter test.

Back tests

These tests are used to test the back-end of the Junqo-platform.

The back tests are run using the back-tests.yml workflow. The workflow is triggered when the following conditions are met:

  • A pull request is opened.
  • The pull request merges into the main or dev branch.
  • The files inside the junqo_back folder are modified.

The workflow runs the following steps:

  1. Checkout the repository.
  2. Install Node.js.
  3. Install the dependencies.
  4. Run the back tests using jest.

Documentation Generation

The documentation is available at http://doc.junqo.fr/.
The documentation is generated automatically when the project is deployed using Github Action.

The documentation is written in Markdown and is available in the docs directory.
It is automatically generated into a static website using Jekyll and is deployed using Github Pages.
If you want to generate the documentation manually, you can use the following command:

Install the dependencies by running the following command:

bundle install

Update the Gemfile.lock file by running the following command:

bundle update

To generate the documentation, run the following command:

bundle exec jekyll build

To serve the documentation locally, run the following command:

bundle exec jekyll serve --baseurl=""

For more information, see the the official github page documentation.

Automatic documentation deployment

The automatic documentation deployment generate the documentation using the markdown files in the docs folder.
Then it deploys the documentation at http://doc.junqo.fr using Github Pages.

The documentation deployment is done using the deploy-documentation.yml workflow.
The workflow is triggered when the following conditions are met:

  • A push is made to the main branch.
  • The files inside the docs folder are modified.
  • The deploy-documentation.yml file is modified.
  1. Checkout the repository.
  2. Install Ruby.
  3. Setup Github Pages.
  4. Upload the final documentation artifacts.

Finally in a different job:

  1. Fetch the final documentation artifacts.
  2. Deploy the documentation to Github Pages.

Mirroring

As the project is evaluated by the EPITECH school, the project is mirrored to a private evaluation repository.

Why is the project mirrored

The project is mirrored to allow the EPITECH school to evaluate the project.
It is developed on the Junqo-org as an open-source project.
We chose to create an open-source project to allow the community to contribute to the project.
Furthermore it allows the project to be more transparent and to be evaluated by the community.
We can also take advantage of the many advantageous plans offered by some organizations to open-source projects.
These organizations includes:

How is the project mirrored

The mirroring is done using the Github Actions pipeline.
The workflow is triggered when a push or delete event is triggered.
The workflow configuration can be found in the .github/workflows/eip-mirroring.yml file.