Skip to content

Add benchmarks #491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 99 commits into
base: main
Choose a base branch
from
Open

Add benchmarks #491

wants to merge 99 commits into from

Conversation

Hrovatin
Copy link
Collaborator

@Hrovatin Hrovatin commented Feb 19, 2025

EDIT:

  • Added 2 chemical data TL benchmarks and 3 synthetic ones.
  • Lines:
    • For chemical ones and Hartmann: TL model with 0-10% of data (e.g. 0) and one with non-TL model using only task data (nonTL).
    • For Michaelewicz and Easom: TL model with single source data size(TL), TL with no source data (TL-noSource), and non-TL model with only target data (non-TL).

OLD:
Started by adding direct arylation TL campaign with Temp as task - adapted from the paper.

Will continue adding more. If someone could already check if this goes in the right direction that would be great so that I do not repeat the mistakes for the others.

Things to discuss:

  • Where to read data from?
  • How should TL benchmarks be set up? E.g. should we always include comparison to no TL and different source data proportions? - We didnt do this before on some datasets as we mainly focused on comparing different TL methods.

@Hrovatin Hrovatin changed the title Add direct arylation benchmark for TL with temperature as a task Add benchmarks Feb 19, 2025
@Scienfitz
Copy link
Collaborator

@Hrovatin
if there are discussion items before we start the review, can you open one thread per item so we can collect thoughts? if needed we can also have meetings

@Hrovatin Hrovatin force-pushed the feature/benchmark_cases branch from 0ea9be1 to 87d9deb Compare March 3, 2025 12:42
Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First batch of comments, only regarding one benchmark file, but probably apply to the others as well

@AVHopp AVHopp force-pushed the feature/benchmark_cases branch 4 times, most recently from e29bc15 to 31fa3c0 Compare March 12, 2025 16:15
@AVHopp AVHopp force-pushed the feature/benchmark_cases branch 5 times, most recently from 0c76238 to a5c47bc Compare March 18, 2025 11:16
Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @AVHopp: went throught the aryl halides folder. Here some comment first, before I go trhough the other folders

@AVHopp AVHopp force-pushed the feature/benchmark_cases branch from c99d45f to 482114c Compare March 19, 2025 15:52
@AVHopp AVHopp force-pushed the feature/benchmark_cases branch from f8c0ec3 to 7d16c72 Compare April 9, 2025 15:30
@AVHopp AVHopp force-pushed the feature/benchmark_cases branch from acd959d to 7f86fc6 Compare April 9, 2025 17:59
synthetic_2C1D_1C:
description: "synthetic_2C1D_1C benchmark"
description: "synthetic_2C1D_1C"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment for line 1:
Did you change the name of this for or something? Doesnt render properly for me in the actions tab:
image

I suggest to simply call this

name: Benchmark

no need for additional words

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants