Pharo DataFrame

DataFrame is a tabular data structure for data analysis in Pharo. It organizes and represents data in a tabular format, resembling a spreadsheet or database table. It is designed to handle structured data and offer various functionalities for data manipulation and analysis. DataFrames are used as visualization tools for Machine Learning and Data Science related tasks.

Installation

To install the latest stable version of DataFrame (pre-v3), go to the Playground (Ctrl+OW) in your Pharo image and execute the following Metacello script (select it and press Do-it button or Ctrl+D):

EpMonitor disableDuring: [
    Metacello new
      baseline: 'DataFrame';
      repository: 'github://PolyMathOrg/DataFrame:pre-v3/src';
      load ].

Use this script if you want the latest version of DataFrame:

EpMonitor disableDuring: [
    Metacello new
      baseline: 'DataFrame';
      repository: 'github://PolyMathOrg/DataFrame/src';
      load ].

If you'd be interested in SQLite support, use load: 'sqlite' at the end:

EpMonitor disableDuring: [
    Metacello new
      baseline: 'DataFrame';
      repository: 'github://PolyMathOrg/DataFrame/src';
      load: 'sqlite' ].

Note: EpMonitor serves to deactive Epicea, a Pharo code recovering mechanism, during the installation of DataFrame.

How to depend on it?

If you want to add a dependency on DataFrame to your project, include the following lines into your baseline method:

spec
  baseline: 'DataFrame'
  with: [ spec repository: 'github://PolyMathOrg/DataFrame/src' ].

If you are new to baselines and Metacello, check out the Baselines tutorial on Pharo Wiki.

What are data frames?

Data frames are the one of the essential parts of the data science toolkit. They are the specialized data structures for tabular data sets that provide us with a simple and powerful API for summarizing, cleaning, and manipulating a wealth of data sources that are currently cumbersome to use.

A data frame is like a database inside a variable. It is an object which can be created, modified, copied, serialized, debugged, inspected, and garbage collected. It allows you to communicate with your data quickly and effortlessly, using just a few lines of code. DataFrame project is similar to pandas library in Python or built-in data.frame class in R.

Very simple example

In this section I show a very simple example of creating and manipulating a little data frame. For more advanced examples, please check the DataFrame Booklet.

Creating a data frame

weather := DataFrame withRows: #(
  (2.4 true rain)
  (0.5 true rain)
  (-1.2 true snow)
  (-2.3 false -)
  (3.2 true rain)).

	1	2	3
1	2.4	true	rain
2	0.5	true	rain
3	-1.2	true	snow
4	-2.3	false	-
5	3.2	true	rain

Removing the third row of the data frame

weather removeRowAt: 3.

	1	2	3
1	2.4	true	rain
2	0.5	true	rain
4	-2.3	false	-
5	3.2	true	rain

Adding a row to the data frame

weather addRow: #(-1.2 true snow) named: 6.

	1	2	3
1	2.4	true	rain
2	0.5	true	rain
4	-2.3	false	-
5	3.2	true	rain
6	-1.2	true	snow

Replacing the data in the first row and third column with 'snow'

weather at:1 at:3 put:#snow.

	1	2	3
1	2.4	true	snow
2	0.5	true	rain
4	-2.3	false	-
5	3.2	true	rain
6	-1.2	true	snow

Transpose of the data frame

weather transposed.

	1	2	4	5	6
1	2.4	0.5	-2.3	3.2	-1.2
2	true	true	false	true	true
3	snow	rain	-	rain	snow

SQLite examples

Following examples expect valid/connected SQLite connection in a variable conn

Load data from SQLite query:

df := DataFrame readFromSqliteCursor: (conn execute: 'SELECT * FROM table').

Write data to SQLite table (DataFrame column names <=> table column names):

df writeToSqlite: conn tableName: 'table'.

Write to differently named colums (provide names for ALL DataFrame columns!)

df writeToSqlite: conn tableName: 'table' columnNames: #('col1' 'col2' 'col3').

Mapping (selecting / renaming dataframe columns):

Let's assume:

• CREATE TABLE tbl (a,b,c)
• DataFrame with columns (a,x,c,d)
• We want to write:
  • a to a
  • x to b
  • c to c
  • ignore d
• NB: no mention of column d, order is irrelevant

df writeToSqlite: conn tableName: 'table' columnMappings: { #c. #x -> #b. #a }.

Documentation and Literature

1. Data Analysis Made Simple with Pharo DataFrame - a booklet that serves as the main source of documentation for the DataFrame project. It describes the complete API of DataFrame and DataSeries data structures, and provides examples for each method.

2. Zaytsev Oleksandr, Nick Papoulias and Serge Stinckwich. Towards Exploratory Data Analysis for Pharo In Proceedings of the 12th edition of the International Workshop on Smalltalk Technologies, pp. 1-6. 2017.