PyCOMPSs Distributed Data Set (DDS)

Website • Documentation • Releases • Slack

Table of content:

• Description
• Requirements
• Installation
• The Most Useful Methods
• Examples
• Availability
• Acknowledgements
• License

Description

DDS is a lightweight library for PyCOMPSs developers which contains some basic and widely used data processing methods such as map, filter, reduce, etc. The main purpose of this library is to avoid implementations of simple 'task' functions by developers. DDS is trustful and it processes the data in the most adequate way in terms of parallelism.

Requirements

• Python >= 3.6
• pycompss >= 3.3

Installation

This package can be installed:

• From source: ./install.sh
• From Pypi: python3 -m pip install dds

The Most Useful Methods

map
filter
reduce
count
max
min
sum
foreach
map_values
combine_by_key
reduce_by_key
collect
collect_as_dict

Please feel free to suggest more methods!

Examples

Please see examples or notebooks folders.

Availability

Currently, the following supercomputers have already PyCOMPSs and DDS installed and ready to use. If you need help configuring your own cluster or supercomputer, drop us an email and we will be pleased to help.

• Marenostrum 5 - Barcelona Supercomputing Center (BSC)

Acknowledgements

This work has received funding from the European Commission's through the Horizon Europe CyclOps project (Contract 101135513)

License

Apache License Version 2.0, see LICENSE