Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CI/CD in Github Actions #451

Merged
merged 13 commits into from
Feb 8, 2022
Merged
31 changes: 31 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Release Python package

on:
workflow_run:
workflows: [tests]
branches: [main]
types: [completed]

jobs:
release:
needs: test
runs-on: ubuntu-latest
name: Release package to PyPi
if: github.event_name == 'push' && contains(github.ref, 'refs/tags/')
steps:
- uses: actions/checkout@master
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox tox-gh-actions
- name: Build and release 📦 to PyPI
with:
TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }}
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
run: |
tox -e release

18 changes: 12 additions & 6 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ on:
- push

jobs:
build:
test:
runs-on: ubuntu-latest
name: Docker | GDAL=${{ matrix.gdal-version }} | python=${{ matrix.python-version }}
container: osgeo/gdal:ubuntu-full-${{ matrix.gdal-version }}
container: osgeo/gdal:ubuntu-small-${{ matrix.gdal-version }}
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9"]
python-version: ["3.8", "3.9", "3.10"]
gdal-version: ["3.4.0", "3.3.3"]
include:
- python-version: "3.8"
Expand All @@ -38,10 +38,16 @@ jobs:
python${{ matrix.python-version }}-dev \
python${{ matrix.python-version }}-venv \
python3-pip \
g++
g++ \
libgl1
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox tox-gh-actions
python -m pip install tox
- name: Lint with tox
run: tox -e lint
- name: Format with tox
run: tox -e format
rodrigoalmeida94 marked this conversation as resolved.
Show resolved Hide resolved
- name: Test with tox
run: tox
run: tox -e test

14 changes: 14 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
repos:
- repo: https://github.com/pycqa/isort
rev: 5.10.1
hooks:
- id: isort
- repo: https://github.com/ambv/black
rev: 22.1.0
hooks:
- id: black
language_version: python3.9
- repo: https://gitlab.com/pycqa/flake8
rev: 3.9.2
hooks:
- id: flake8
4 changes: 2 additions & 2 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
import sys

sys.path.insert(0, os.path.abspath(".."))
import numpy as np
import numpy as np # noqa: F401
import sphinx_bootstrap_theme

# -- Project information -----------------------------------------------------
Expand All @@ -25,7 +25,7 @@
license = "MIT"
import time

copyright = u"2018-{}, CosmiQ Works: an IQT Lab".format(time.strftime("%Y"))
copyright = "2018-{}, CosmiQ Works: an IQT Lab".format(time.strftime("%Y"))

# The full version, including alpha/beta/rc tags
release = "0.0.1"
Expand Down
27 changes: 27 additions & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
[flake8]
extend-ignore =
E203, # whitespace before ':'
E501, # line too long
C901, # is too complex
E402, # module level import not at top of file (for docs)
E741, # ambiguous variable name

exclude =
.git,
__pycache__,
docs/source/conf.py,
old,
build,
dist,
.tox
max-line-length = 88
max-complexity = 10
count=true

[isort]
profile = black

[black]
line-length = 88
target-version = ['py37', 'py38', 'py39', 'py310']
experimental_string_processing = true
3 changes: 2 additions & 1 deletion setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ def check_output(cmd):
"pyproj>=2.1",
"PyYAML>=5.4",
"rasterio>=1.0.23",
"rio-cogeo>=3.0.2",
"requests==2.22.0",
"rtree>=0.9.3",
"scikit-image>=0.16.2",
Expand Down Expand Up @@ -79,7 +80,7 @@ def check_output(cmd):
"Programming Language :: Python :: 3.10",
"Topic :: Scientific/Engineering :: GIS",
],
author=u"Ryan Avery",
author="Ryan Avery",
author_email="[email protected]",
url="https://github.com/CosmiQ/solaris",
license="MIT",
Expand Down
2 changes: 1 addition & 1 deletion solaris/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from . import data, utils, vector
from . import data, utils, vector # noqa: F401

# data, eval, preproc, raster, tile, have gdal in them need to replace with rasterio
__version__ = "0.0.1"
2 changes: 1 addition & 1 deletion solaris/eval/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
from . import base, iou, pixel, vector
from . import base, iou, pixel, vector # noqa: F401
2 changes: 0 additions & 2 deletions solaris/eval/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,8 +111,6 @@ def eval_iou_spacenet_csv(
scoring_dict_list = []
self.ground_truth_GDF[iou_field] = 0.0
iou_index = self.ground_truth_GDF.columns.get_loc(iou_field)
id_cols = 2
ground_truth_ids = self.ground_truth_GDF.iloc[:, :id_cols]

for imageID in tqdm(imageIDList):
self.ground_truth_GDF_Edit = self.ground_truth_GDF[
Expand Down
2 changes: 1 addition & 1 deletion solaris/preproc/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
from . import image, label, optical, pipesegment, sar
from . import image, label, optical, pipesegment, sar # noqa: F401
2 changes: 1 addition & 1 deletion solaris/preproc/image.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import pandas as pd
from osgeo import gdal_array

from .pipesegment import LoadSegment, MergeSegment, PipeSegment
from .pipesegment import LoadSegment, PipeSegment


class Image:
Expand Down
2 changes: 1 addition & 1 deletion solaris/preproc/label.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import shapely.wkt

from ..vector.polygon import convert_poly_coords
from .pipesegment import LoadSegment, MergeSegment, PipeSegment
from .pipesegment import LoadSegment, PipeSegment


class LoadString(LoadSegment):
Expand Down
3 changes: 1 addition & 2 deletions solaris/preproc/optical.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,8 @@

import numpy as np

from . import image
from .image import Image
from .pipesegment import LoadSegment, MergeSegment, PipeSegment
from .pipesegment import PipeSegment


class RGBToHSL(PipeSegment):
Expand Down
2 changes: 1 addition & 1 deletion solaris/preproc/pipesegment.py
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ def PipeFunction(inner_class=PipeSegment, pin=(), *args, **kwargs):
and *args and **kwargs are sent to the PipeSegment's constructor.
"""
psobject = inner_class(*args, **kwargs)
if issubclass(self.inner_class, LoadSegment):
if issubclass(self.inner_class, LoadSegment): # noqa: F821
return psobject()
else:
return (pin * psobject)()
7 changes: 3 additions & 4 deletions solaris/preproc/sar.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import json
import math
import os
import uuid
import warnings
import xml.etree.ElementTree as ET
Expand All @@ -12,7 +11,7 @@

from . import image
from .image import Image
from .pipesegment import LoadSegment, MergeSegment, PipeSegment
from .pipesegment import PipeSegment


class BandMath(PipeSegment):
Expand Down Expand Up @@ -668,8 +667,8 @@ def fineoffset(self, latgrid, longrid, lattarget, lontarget, uidx, vidx):
ulon = longrid[uidx + 1, vidx] - longrid[uidx, vidx]
vlat = latgrid[uidx, vidx + 1] - latgrid[uidx, vidx]
vlon = longrid[uidx, vidx + 1] - longrid[uidx, vidx]
uoffset = (mlat * ulat + mlon * ulon) / (ulat ** 2 + ulon ** 2)
voffset = (mlat * vlat + mlon * vlon) / (vlat ** 2 + vlon ** 2)
uoffset = (mlat * ulat + mlon * ulon) / (ulat**2 + ulon**2)
voffset = (mlat * vlat + mlon * vlon) / (vlat**2 + vlon**2)
return uoffset, voffset


Expand Down
2 changes: 1 addition & 1 deletion solaris/raster/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
from . import image
from . import image # noqa: F401
2 changes: 0 additions & 2 deletions solaris/raster/image.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import os

import numpy as np
import rasterio

Expand Down
2 changes: 1 addition & 1 deletion solaris/tile/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
from . import raster_tile, vector_tile
from . import raster_tile, vector_tile # noqa: F401
2 changes: 1 addition & 1 deletion solaris/tile/raster_tile.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@
from rasterio.mask import mask as rasterio_mask
from rasterio.vrt import WarpedVRT
from rasterio.warp import Resampling, calculate_default_transform
from rio_cogeo.cogeo import cog_translate # ,cog_validate
from shapely.geometry import box
from tqdm.auto import tqdm

# from rio_cogeo.cogeo import cog_validate, cog_translate
from ..utils.core import _check_crs, _check_rasterio_im_load

# removing the following until COG functionality is implemented
Expand Down
2 changes: 1 addition & 1 deletion solaris/utils/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
from . import cli, core, data, geo, io, tile
from . import cli, core, data, geo, io, tile # noqa: F401
6 changes: 0 additions & 6 deletions solaris/utils/tile.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,5 @@
import json

import geopandas as gpd
from affine import Affine
from rasterio.enums import Resampling
from rasterio.vrt import WarpedVRT
from rasterio.windows import Window

from .core import _check_crs

# temporarily removing the below until I can get COG functionality implemented
Expand Down
2 changes: 1 addition & 1 deletion solaris/vector/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
from . import graph, mask, polygon
from . import graph, mask, polygon # noqa: F401
2 changes: 0 additions & 2 deletions tests/test_cli/test_cli.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
import os
import pickle
import subprocess

import networkx as nx
import numpy as np
import skimage.io

Expand Down
4 changes: 2 additions & 2 deletions tests/test_data/test_coco.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ def test_multiclass_single_geojson(self):
expected_dict = json.load(f)
with open(os.path.join(data_dir, "tmp_coco.json"), "r") as f:
saved_result = json.load(f)
## Simplified test due to rounding errors- JSS
# Simplified test due to rounding errors- JSS
assert (
coco_dict["annotations"][0]["bbox"]
== expected_dict["annotations"][0]["bbox"]
Expand Down Expand Up @@ -62,7 +62,7 @@ def test_singleclass_multi_geojson(self):

with open(os.path.join(data_dir, "coco_sample_1.json"), "r") as f:
expected_dict = json.load(f)
## Simplified test due to rounding errors- JSS
# Simplified test due to rounding errors- JSS
assert (
expected_dict["annotations"][0]["bbox"]
== coco_dict["annotations"][0]["bbox"]
Expand Down
1 change: 0 additions & 1 deletion tests/test_eval/evaluator_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,6 @@ def test_iou_by_building(self):
path_truth = os.path.join(data_folder, "SN2_sample_truth.csv")
path_pred = os.path.join(data_folder, "SN2_sample_preds.csv")
path_ious = os.path.join(data_folder, "SN2_sample_iou_by_building.csv")
path_temp = "./temp.pd"
eb = Evaluator(path_truth)
eb.load_proposal(path_pred, conf_field_list=["Confidence"], proposalCSV=True)
eb.eval_iou_spacenet_csv(miniou=0.5, imageIDField="ImageId", min_area=20)
Expand Down
1 change: 1 addition & 0 deletions tests/test_imports.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# flake8: noqa: F401
class TestImports(object):
def test_imports(self):
import solaris
Expand Down
5 changes: 1 addition & 4 deletions tests/test_raster/test_image.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
import os

import numpy as np
import skimage.io
from affine import Affine

import solaris as sol
from solaris.data import sample_load_rasterio
from solaris.raster.image import get_geo_transform, stitch_images
from solaris.raster.image import get_geo_transform

data_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), "../data/"))

Expand Down
47 changes: 17 additions & 30 deletions tox.ini
Original file line number Diff line number Diff line change
@@ -1,44 +1,31 @@
[tox]
envlist = py37,py38,py39,py310
envlist =
lint,
format,
test

[gh-actions]
python =
3.7: py37
3.8: py38
3.9: py39
3.10: py310

[testenv]
[testenv:test]
extras = test
commands=
commands =
python -m pytest --cov --cov-report xml --cov-report term-missing --ignore=venv

# Lint
[flake8]
exclude = .git,__pycache__,docs/source/conf.py,old,build,dist
max-line-length = 90

[mypy]
no_strict_optional = True
ignore_missing_imports = True

[tool:isort]
include_trailing_comma = True
multi_line_output = 3
line_length = 90
known_first_party = solaris
default_section = THIRDPARTY
[testenv:lint]
basepython = python3
skip_install = true
deps = flake8
commands =
flake8 .

# Autoformatter
[testenv:black]
[testenv:format]
basepython = python3
skip_install = true
deps =
deps =
isort
black
commands =
black
isort . -c
black . --check

# Release tooling
[testenv:build]
basepython = python3
skip_install = true
Expand Down