Proposal: Adding a development environment

[arejula27]

Description

This is a proposal to introduce an automated, isolated development environment for the Qbeast Spark project. The goal is to simplify the process of setting up the project with all necessary dependencies and configurations, making it easier for:

• New developers: To quickly get started and contribute to the project.
• Current contributors: To work within a consistent, declaratively defined environment, reducing bugs and errors caused by dependency version mismatches.

The PR adds a nix-shell file to define the development environment and updates the CONTRIBUTING.md file with clear instructions on how to use it.

Motivation

Setting up the Qbeast Spark project currently requires manually installing multiple dependencies (Java, Scala, Spark), which can be time-consuming and error-prone. For example, during setup, I encountered an issue where sbt defaulted to my system's Java version (21), which was incompatible with the project and caused most tests to fail. This motivated me to create a declarative development environment to streamline the process and avoid such issues.

A declarative environment brings several benefits to the project:

• Consistency 🛠️: Ensures all developers use the same environment, reducing bugs caused by mismatched dependency versions. It can also be integrated into CI/CD pipelines to guarantee the project is built and tested in a consistent environment, eliminating reliance on external container images or GitHub Actions maintainers.
• Isolation 🗃️: Keeps the development environment separate from the host system, preventing conflicts with other projects or dependencies. This also makes cleanup simple, avoiding any impact on the developer’s host system.
• Newcomer-friendly 🌱: Simplifies onboarding for new contributors by automating the setup process, eliminating the need to manually install or configure dependencies. This lowers the barrier to entry and increases the likelihood of attracting new contributors. It will also be a impactful feature during meetups or workshops, where participants can quickly set up the project.
• Cross-platform support 🖥️: The nix-shell file works on Windows (via WSL2), Linux, and macOS, making it easy for developers to contribute regardless of their platform.
• Simplification 🤗: Consolidates all dependencies and configurations into a single file, making the setup process easier to understand, maintain, and modify when needed.

By adopting this declarative approach, we can significantly improve the development experience and project reliability while making it more accessible to a broader audience.

Why nix-shell?

Currently, one of the most common ways to set up a development environment is by using Docker. While Docker provides an isolated environment, it comes with several limitations and drawbacks:

• Resource-intensive ⚙️: Running a container consumes significant system resources and can be slow, especially on less powerful machines.
• Complexity with multiple components 🔗: When a project requires various components (e.g., databases, message brokers), managing them all within containers can become a nightmare for developers, especially for everyday coding tasks.
• Not a personal environment 🖌️: Docker essentially works as a separate operating system, with no configuration except what is explicitly defined in the Dockerfile, which can make customization and integration with the host system more challenging.

In contrast, Nix offers a better solution for defining development environments. It is a package manager that allows us to declaratively specify all dependencies, configurations, and tools required for a project in a single file (shell.nix). Using this approach with nix-shell offers several advantages:

• Lightweight and fast 🚀: Only the required dependencies and tools are installed, avoiding the overhead of running a container.
• Host OS integration 👨‍🔧: Developers can work directly on their host OS, avoiding the need for containerized environments while still maintaining isolation.

Nix has been around for a long time but is now becoming more popular because it is secure and reliable. Big companies like Google use it for their products, like the cloud IDE service IDX