Skip to content
This repository has been archived by the owner on Dec 17, 2023. It is now read-only.
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: piraces/rsslay
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: planetary-social/rsslay
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Can’t automatically merge. Don’t worry, you can still create the pull request.

Commits on Apr 26, 2023

  1. Copy the full SHA
    b53874d View commit details

Commits on May 31, 2023

  1. Always use long form content

    Not all substack links use their domain, quite a lot of them have custom
    domains actually.
    boreq committed May 31, 2023
    Copy the full SHA
    a2df5ee View commit details

Commits on Jun 22, 2023

  1. Copy the full SHA
    6fdc6df View commit details
  2. Merge pull request #4 from planetary-social/publish-pipeline

    Adjust build step for our repo
    cooldracula authored Jun 22, 2023
    Copy the full SHA
    b9df88b View commit details

Commits on Jun 23, 2023

  1. push to cooldracula docker hub too

    experiment until correct visibility in place for ghcr package
    cooldracula committed Jun 23, 2023
    Copy the full SHA
    c8b8863 View commit details
  2. Merge pull request #5 from planetary-social/publish-pipeline

    push to cooldracula docker hub too
    cooldracula authored Jun 23, 2023
    Copy the full SHA
    0bf7f88 View commit details
  3. Copy the full SHA
    1459b21 View commit details
  4. Merge pull request #6 from planetary-social/publish-pipeline

    only push to accessible docker
    cooldracula authored Jun 23, 2023
    Copy the full SHA
    4989b01 View commit details
  5. Copy the full SHA
    a3d1b62 View commit details
  6. Copy the full SHA
    75723ff View commit details
  7. log into docker

    cooldracula committed Jun 23, 2023
    Copy the full SHA
    7be6e30 View commit details

Commits on Jun 29, 2023

  1. Copy the full SHA
    87a89fb View commit details

Commits on Jul 5, 2023

  1. Fix filtering of events in REQ requests

    Previously only notes were supported so querying for long form content
    was impossible. Instead we were incorrectly getting it when querying for
    notes.
    boreq committed Jul 5, 2023
    Copy the full SHA
    f4fa3c5 View commit details

Commits on Jul 6, 2023

  1. Copy the full SHA
    48e89de View commit details
  2. Copy the full SHA
    1fdb478 View commit details

Commits on Jul 8, 2023

  1. Copy the full SHA
    59b552d View commit details
  2. Merge pull request #11 from planetary-social/pipeline-fix

    only push to ghcr using correct image name
    cooldracula authored Jul 8, 2023
    Copy the full SHA
    9335747 View commit details
  3. fix templating typo

    cooldracula committed Jul 8, 2023
    Copy the full SHA
    e86bc9e View commit details
  4. Merge pull request #12 from planetary-social/pipeline-fix

    fix templating typo
    cooldracula authored Jul 8, 2023
    Copy the full SHA
    cd6bee9 View commit details

Commits on Jul 10, 2023

  1. Merge pull request #13 from planetary-social/substack

    Merge substack into main.
    boreq authored Jul 10, 2023
    Copy the full SHA
    a9459b8 View commit details

Commits on Jul 11, 2023

  1. Copy the full SHA
    ef3ff6e View commit details
  2. Merge pull request #14 from planetary-social/fix-long-form-content

    Fix a merge error which caused item content to be removed
    boreq authored Jul 11, 2023
    Copy the full SHA
    29eec6b View commit details

Commits on Aug 30, 2023

  1. Update from upstream

    boreq committed Aug 30, 2023
    Copy the full SHA
    23cc7b0 View commit details
  2. Copy the full SHA
    a7817dd View commit details

Commits on Aug 31, 2023

  1. Add causes support to rsslay

    This approach is less than ideal unfortunately and makes us perform ~90
    requests to get articles.
    boreq committed Aug 31, 2023
    Copy the full SHA
    2627b30 View commit details
  2. Merge pull request #20 from planetary-social/causes

    Add causes support to rsslay
    boreq authored Aug 31, 2023
    Copy the full SHA
    f9c763c View commit details

Commits on Sep 18, 2023

  1. Refactor rsslay to proactively retrieve feeds (#21)

    Goals:
    
    - Download data in the background to always be able to return data quickly.
      - Done.
    - Make the project more composable e.g. separate out markdown and html processing from constructing nostr events.
      - Created clear adapter/app/domain layers. Didn't fix everything to keep the pull request small.
    - Make testing easier.
      - Testing the domain layer is easy now.
    - Eliminate the bug which causes the feed to not be updated under some circumstances.
      - Unclear. Maybe we need to remove caching altogether, it is not really useful right now.
    
    What has been done:
    - http handlers no longer contain core domain logic
    - http handlers no longer have direct access to the database
    - http handlers no longer misuse the same types for database queries and template rendering
    - http handlers no longer have access to secrets
    - created an adapters layer which encapsulates database access
    - created an domain layer for data validation, more core logic needs to be moved there in the future
    - created an application layer to glue everything together
    - cleaned up duplicated calls to libraries to convert keys to the nip19 format in http handlers
    - cleaned up error handling logic in http handlers
    - saving feed definitions will no longer fail quietly in case of database errors
    - fixed nonsensical error handling when saving feed definitions
    - events are properly retrieved in a single place in the application layer and not from multiple places
    - removed duplicate code
    boreq authored Sep 18, 2023
    Copy the full SHA
    2d07a44 View commit details

Commits on Jan 4, 2024

  1. Add deployment link to README

    This adds a short link to the larger notion doc
    cooldracula committed Jan 4, 2024
    Copy the full SHA
    e28ba9a View commit details

Commits on Jan 10, 2024

  1. Merge pull request #23 from planetary-social/deployment-readme

    Add deployment link to README
    cooldracula authored Jan 10, 2024
    Copy the full SHA
    78cd3af View commit details
Showing with 2,303 additions and 653 deletions.
  1. +1 −1 .env.sample
  2. +13 −16 .github/workflows/docker-publish.yml
  3. +52 −0 .github/workflows/docker-stable-publish.yml
  4. +1 −1 .github/workflows/goreleaser-snapshot.yml
  5. +1 −1 .github/workflows/goreleaser.yml
  6. +1 −1 .github/workflows/main.yml
  7. +2 −2 Dockerfile
  8. +2 −2 Dockerfile.fly
  9. +2 −2 Dockerfile.railwayapp
  10. +12 −1 README.md
  11. +72 −142 cmd/rsslay/main.go
  12. +51 −0 cmd/rsslay/store.go
  13. +1 −1 fly.toml
  14. +17 −11 go.mod
  15. +53 −26 go.sum
  16. +80 −129 internal/handlers/handlers.go
  17. +67 −32 pkg/converter/rules.go
  18. +3 −2 pkg/custom_cache/cache.go
  19. +4 −3 pkg/events/events.go
  20. +2 −1 pkg/events/events_test.go
  21. +216 −0 pkg/feed/converter.go
  22. +317 −0 pkg/feed/converter_test.go
  23. +38 −0 pkg/feed/downloader.go
  24. +196 −88 pkg/feed/feed.go
  25. +3 −181 pkg/feed/feed_test.go
  26. +1 −0 pkg/feed/translator.go
  27. +2 −1 pkg/feed/translator_test.go
  28. +2 −1 pkg/helpers/helpers.go
  29. +2 −1 pkg/helpers/helpers_test.go
  30. +4 −4 pkg/metrics/registries.go
  31. +54 −0 pkg/new/adapters/event_storage.go
  32. +147 −0 pkg/new/adapters/feed_definition_storage.go
  33. +25 −0 pkg/new/adapters/pubsub/event_created_pubsub.go
  34. +67 −0 pkg/new/adapters/pubsub/pubsub.go
  35. +39 −0 pkg/new/app/app.go
  36. +77 −0 pkg/new/app/handler_create_feed_definition.go
  37. +17 −0 pkg/new/app/handler_get_events.go
  38. +23 −0 pkg/new/app/handler_get_random_feeds.go
  39. +15 −0 pkg/new/app/handler_get_total_event_count.go
  40. +30 −0 pkg/new/app/handler_on_new_event_created.go
  41. +219 −0 pkg/new/app/handler_update_feeds.go
  42. +23 −0 pkg/new/app/search_feeds.go
  43. +19 −0 pkg/new/domain/domain.go
  44. +59 −0 pkg/new/domain/feed/feed.go
  45. +156 −0 pkg/new/domain/nostr/nostr.go
  46. +41 −0 pkg/new/domain/nostr/nostr_test.go
  47. +36 −0 pkg/new/ports/pubsub/received_events.go
  48. +34 −0 pkg/new/ports/timer_update_feeds.go
  49. +4 −3 pkg/replayer/replayer.go
2 changes: 1 addition & 1 deletion .env.sample
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@ PORT=8080
DB_DIR="/db/rsslay.sqlite"
DEFAULT_PROFILE_PICTURE_URL="https://i.imgur.com/MaceU96.png"
SECRET="CHANGE_ME"
VERSION=0.5.3
VERSION=0.5.4
REPLAY_TO_RELAYS=false
RELAYS_TO_PUBLISH_TO=""
NITTER_INSTANCES=""
29 changes: 13 additions & 16 deletions .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
@@ -2,13 +2,15 @@ name: Publish Docker image

on:
push:
tags:
- '*'
workflow_dispatch:
branches: ['main']

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
push_to_registries:
name: Push Docker image to multiple registries
name: Push Docker image to ghcr registry
runs-on: ubuntu-latest
permissions:
packages: write
@@ -23,13 +25,7 @@ jobs:
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}

- name: Log in to the Container registry
- name: Log in to the Github Container registry
uses: docker/login-action@v2
with:
registry: ghcr.io
@@ -38,17 +34,18 @@ jobs:

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v4
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: |
piraces/rsslay
ghcr.io/piraces/rsslay
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch,suffix=_{{sha}}
type=raw,value=latest,enable={{is_default_branch}}
- name: Build and push Docker images
uses: docker/build-push-action@v4
with:
context: .
push: true
platforms: linux/amd64,linux/arm64/v8
platforms: linux/amd64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
52 changes: 52 additions & 0 deletions .github/workflows/docker-stable-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Publish Stable Docker image with tag

on:
push:
tags:
- 2**-**-**

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
push_to_registries:
name: Push stable image to gchr registry
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- name: Check out the repo
uses: actions/checkout@v3

- name: Set up QEMU
uses: docker/setup-qemu-action@v2

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Log in to the Github Container registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=tag
type=raw,event=tag,value=stable
- name: Build and push Docker images
uses: docker/build-push-action@v4
with:
context: .
push: true
platforms: linux/amd64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
2 changes: 1 addition & 1 deletion .github/workflows/goreleaser-snapshot.yml
Original file line number Diff line number Diff line change
@@ -18,7 +18,7 @@ jobs:
- run: git fetch --force --tags
- uses: actions/setup-go@v4
with:
go-version: '>=1.20.0'
go-version: '>=1.21.0'
cache: true
- uses: goreleaser/goreleaser-action@v4
with:
2 changes: 1 addition & 1 deletion .github/workflows/goreleaser.yml
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ jobs:
- run: git fetch --force --tags
- uses: actions/setup-go@v4
with:
go-version: '>=1.20.0'
go-version: '>=1.21.0'
cache: true
- uses: goreleaser/goreleaser-action@v4
with:
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: 1.20.0
go-version: 1.21.0

- name: Build
run: go build -v ./...
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM golang:1.20-alpine as build
FROM golang:1.21-alpine as build

WORKDIR /app

@@ -28,7 +28,7 @@ ENV PORT="8080"
ENV DB_DIR="/db/rsslay.sqlite"
ENV DEFAULT_PROFILE_PICTURE_URL="https://i.imgur.com/MaceU96.png"
ENV SECRET="CHANGE_ME"
ENV VERSION=0.5.3
ENV VERSION=0.5.4
ENV REPLAY_TO_RELAYS=false
ENV RELAYS_TO_PUBLISH_TO=""
ENV NITTER_INSTANCES=""
4 changes: 2 additions & 2 deletions Dockerfile.fly
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# syntax=docker/dockerfile:1
FROM flyio/litefs:0.4 AS litefs
FROM golang:1.20-alpine as build
FROM golang:1.21-alpine as build

WORKDIR /app

@@ -30,7 +30,7 @@ ENV PORT="8080"
ENV DB_DIR="/db/rsslay.sqlite"
ENV DEFAULT_PROFILE_PICTURE_URL="https://i.imgur.com/MaceU96.png"
ENV SECRET="CHANGE_ME"
ENV VERSION=0.5.3
ENV VERSION=0.5.4
ENV REPLAY_TO_RELAYS=false
ENV RELAYS_TO_PUBLISH_TO=""
ENV NITTER_INSTANCES=""
4 changes: 2 additions & 2 deletions Dockerfile.railwayapp
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM golang:1.20-alpine as build
FROM golang:1.21-alpine as build

ARG PORT
ARG DB_DIR
@@ -69,7 +69,7 @@ ENV PORT=$PORT
ENV DB_DIR=$DB_DIR
ENV DEFAULT_PROFILE_PICTURE_URL=$DEFAULT_PROFILE_PICTURE_URL
ENV SECRET=$SECRET
ENV VERSION=0.5.3
ENV VERSION=0.5.4
ENV REPLAY_TO_RELAYS=false
ENV RELAYS_TO_PUBLISH_TO=""
ENV NITTER_INSTANCES=""
13 changes: 12 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@
**Working relay: `wss://rsslay.nostr.moe`. Frontend available in [rsslay.nostr.moe](https://rsslay.nostr.moe).**


[![Deploy on Railway](https://railway.app/button.svg)](https://railway.app/new/template/UDf6vC?referralCode=Zbo_gO)
[![Deploy on Railway](https://railway.app/button.svg)](https://railway.app/template/UDf6vC?referralCode=Zbo_gO)

![Screenshot of main page](screenshot.png)

@@ -70,6 +70,17 @@ In the case of the main instance `rsslay.nostr.moe`, Redis is used in HA mode to
Since version v0.5.1, rsslay uses [Prometheus](https://prometheus.io/) instrumenting with metrics exposed on `/metrics` path.
So with this you can mount your own [Graphana](https://grafana.com/) dashboards and look into rsslay insights!

# Related projects

- **atomstr** by [@psic4t](https://github.com/psic4t): atomstr is a RSS/Atom gateway to Nostr. It fetches all sorts of RSS or Atom feeds, generates Nostr profiles for each and posts new entries to given Nostr relay(s). If you have one of these relays in your profile, you can find and subscribe to the feeds.
- Source code: https://sr.ht/~psic4t/atomstr/
- Demo instance: https://atomstr.data.haus/

## Deployment

We deploy this service using [ansible](https://github.com/planetary-social/ansible-scripts). The steps for deployment can be read in our [internal notion page](https://www.notion.so/nossocial/Deploying-services-via-Ansible-2b8d01a60d2441e197d4e03d3274c9ef?pvs=4).


# Contributing

Feel free to [open an issue](https://github.com/piraces/rsslay/issues/new), provide feedback in [discussions](https://github.com/piraces/rsslay/discussions), or fork the repo and open a PR with your contribution!
Loading