Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(regions): move project automations #3925

Merged
merged 37 commits into from
Feb 18, 2025
Merged
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
4d5f96b
feat(regions): repo functions for copying project branches and commits
cdriesler Jan 12, 2025
e5d04e5
chore(regions): wire up move to resolver
cdriesler Jan 13, 2025
f64a8bb
chore(regions): successful basic test of project region change
cdriesler Jan 14, 2025
f1a82e6
fix(regions): sabrina carpenter please please please
cdriesler Jan 16, 2025
ec0df3c
Merge remote-tracking branch 'origin' into chuck/web-2433-move-projec…
cdriesler Jan 16, 2025
88bafb2
fix(regions): repair multiregion test setup
cdriesler Jan 22, 2025
34a68a8
Merge remote-tracking branch 'origin' into chuck/web-2433-move-projec…
cdriesler Jan 22, 2025
1fb3b43
chore(regions): appease ts
cdriesler Jan 22, 2025
54b61bf
chore(multiregion): update test multiregion config
cdriesler Jan 22, 2025
de8d78a
chore(multiregion): fix test docker config and test
cdriesler Jan 22, 2025
ca85039
chore(multiregion): use transaction
cdriesler Jan 23, 2025
201fe2e
chore(multiregion): maybe this will work
cdriesler Jan 23, 2025
08b53cc
fix(multiregion): drop subs synchronously
cdriesler Jan 24, 2025
8edde64
chore(multiregion): desperate test logs
cdriesler Jan 24, 2025
afc06d7
chore(multiregion): somehow that worked?
cdriesler Jan 24, 2025
6d6f800
chore(multiregion): add load-bearing log statement
cdriesler Jan 24, 2025
a27b97a
chore(multiregion): move services
cdriesler Jan 24, 2025
6a0fadc
fix(multiregion): test drop waits
cdriesler Jan 24, 2025
969f8b6
Merge branch 'main' into chuck/web-2433-move-project-branches-and-com…
cdriesler Jan 27, 2025
361960c
Merge branch 'main' into chuck/web-2433-move-project-branches-and-com…
cdriesler Feb 2, 2025
0811916
chore(regions): fix import
cdriesler Feb 3, 2025
b48721e
chore(regions): make test a bit more thorough for good measure
cdriesler Feb 3, 2025
2b8f6af
fix(regions): move project objects
cdriesler Feb 3, 2025
62466f5
chore(regions): add tests for object move
cdriesler Feb 3, 2025
51ba709
feat(regions): move project automations
cdriesler Feb 3, 2025
de3173e
chore(regions): add tests for moving automations
cdriesler Feb 4, 2025
e371633
chore(regions): more tests for moving automate data
cdriesler Feb 6, 2025
04edd9c
fix(regions): speed up inserts
cdriesler Feb 6, 2025
28abd12
Merge branch 'chuck/web-2433-move-project-branches-and-commits' into …
cdriesler Feb 7, 2025
72167e6
fix(regions): simplify postgres usage
cdriesler Feb 7, 2025
f830056
Merge branch 'main' into chuck/web-2433-move-project-branches-and-com…
cdriesler Feb 7, 2025
d2212b0
Merge branch 'chuck/web-2433-move-project-branches-and-commits' into …
cdriesler Feb 7, 2025
2b00b47
Merge branch 'chuck/web-2451-move-commit-objects' into chuck/web-2436…
cdriesler Feb 7, 2025
25eacc3
chore(regions): repair build
cdriesler Feb 7, 2025
78e8ff8
fix(regions): improve queries
cdriesler Feb 7, 2025
ac94d8f
Merge remote-tracking branch 'origin' into chuck/web-2436-move-automa…
cdriesler Feb 17, 2025
e391f88
chore(regions): again
cdriesler Feb 18, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions packages/server/modules/automate/domain/operations.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { InsertableAutomationFunctionRun } from '@/modules/automate/domain/types'
import {
AutomateRevisionFunctionRecord,
AutomationRevisionFunctionRecord,
AutomationFunctionRunRecord,
AutomationRecord,
AutomationRevisionRecord,
Expand Down Expand Up @@ -175,7 +175,7 @@ export type GetRevisionsTriggerDefinitions = (params: {

export type GetRevisionsFunctions = (params: {
automationRevisionIds: string[]
}) => Promise<{ [automationRevisionId: string]: AutomateRevisionFunctionRecord[] }>
}) => Promise<{ [automationRevisionId: string]: AutomationRevisionFunctionRecord[] }>

export type CreateStoredAuthCode = (
params: Omit<AuthCodePayload, 'code'>
Expand Down Expand Up @@ -204,3 +204,7 @@ export type TriggerAutomationRevisionRun = <
manifest: M
source?: RunTriggerSource
}) => Promise<{ automationRunId: string }>

export type GetProjectAutomationCount = (params: {
projectId: string
}) => Promise<number>
4 changes: 2 additions & 2 deletions packages/server/modules/automate/helpers/graphTypes.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import {
AutomateRevisionFunctionRecord,
AutomationRevisionFunctionRecord,
AutomationFunctionRunRecord,
AutomationRecord,
AutomationRevisionRecord,
Expand Down Expand Up @@ -56,7 +56,7 @@ export type AutomationRunTriggerGraphQLReturn = AutomationRunTriggerRecord & {
}

export type AutomationRevisionFunctionGraphQLReturn = Merge<
AutomateRevisionFunctionRecord,
AutomationRevisionFunctionRecord,
{
functionInputs: Nullable<Record<string, unknown>>
release: AutomateFunctionReleaseGraphQLReturn
Expand Down
4 changes: 2 additions & 2 deletions packages/server/modules/automate/helpers/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ export type AutomationRunRecord = {
executionEngineRunId: string | null
}

export type AutomateRevisionFunctionRecord = {
export type AutomationRevisionFunctionRecord = {
functionReleaseId: string
functionId: string
functionInputs: string | null
Expand Down Expand Up @@ -118,7 +118,7 @@ export type AutomationTokenRecord = {
}

export type AutomationRevisionWithTriggersFunctions = AutomationRevisionRecord & {
functions: AutomateRevisionFunctionRecord[]
functions: AutomationRevisionFunctionRecord[]
triggers: AutomationTriggerDefinitionRecord[]
}

Expand Down
19 changes: 11 additions & 8 deletions packages/server/modules/automate/repositories/automations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import {
GetLatestAutomationRevision,
GetLatestAutomationRevisions,
GetLatestVersionAutomationRuns,
GetProjectAutomationCount,
GetRevisionsFunctions,
GetRevisionsTriggerDefinitions,
StoreAutomation,
Expand All @@ -37,7 +38,7 @@ import {
AutomationRunRecord,
AutomationTokenRecord,
AutomationTriggerRecordBase,
AutomateRevisionFunctionRecord,
AutomationRevisionFunctionRecord,
AutomationRunWithTriggersFunctionRuns,
AutomationRunTriggerRecord,
AutomationFunctionRunRecord,
Expand Down Expand Up @@ -86,7 +87,7 @@ const tables = {
automationRevisions: (db: Knex) =>
db<AutomationRevisionRecord>(AutomationRevisions.name),
automationRevisionFunctions: (db: Knex) =>
db<AutomateRevisionFunctionRecord>(AutomationRevisionFunctions.name),
db<AutomationRevisionFunctionRecord>(AutomationRevisionFunctions.name),
automationTriggers: (db: Knex) =>
db<AutomationTriggerDefinitionRecord>(AutomationTriggers.name),
automationRuns: (db: Knex) => db<AutomationRunRecord>(AutomationRuns.name),
Expand Down Expand Up @@ -321,7 +322,7 @@ export const storeAutomationTokenFactory =
}

export type InsertableAutomationRevisionFunction = Omit<
AutomateRevisionFunctionRecord,
AutomationRevisionFunctionRecord,
'automationRevisionId'
>

Expand Down Expand Up @@ -369,7 +370,7 @@ export const storeAutomationRevisionFactory =
.automationRevisionFunctions(deps.db)
.insert(
revision.functions.map(
(f): AutomateRevisionFunctionRecord => ({
(f): AutomationRevisionFunctionRecord => ({
...f,
automationRevisionId: id
})
Expand Down Expand Up @@ -742,10 +743,12 @@ const getProjectAutomationsBaseQueryFactory =
}

export const getProjectAutomationsTotalCountFactory =
(deps: { db: Knex }) => async (params: GetProjectAutomationsParams) => {
const q = getProjectAutomationsBaseQueryFactory(deps)(params).count<
[{ count: string }]
>(Automations.col.id)
(deps: { db: Knex }): GetProjectAutomationCount =>
async ({ projectId }) => {
const q = getProjectAutomationsBaseQueryFactory(deps)({
projectId,
args: {}
}).count<[{ count: string }]>(Automations.col.id)

const [ret] = await q

Expand Down
4 changes: 2 additions & 2 deletions packages/server/modules/automate/services/encryption.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import { Nullable, Optional } from '@speckle/shared'
import { MisconfiguredEnvironmentError } from '@/modules/shared/errors'
import { AutomationFunctionInputEncryptionError } from '@/modules/automate/errors/management'
import { KeyPair, buildDecryptor } from '@/modules/shared/utils/libsodium'
import { AutomateRevisionFunctionRecord } from '@/modules/automate/helpers/types'
import { AutomationRevisionFunctionRecord } from '@/modules/automate/helpers/types'
import { AutomationRevisionFunctionGraphQLReturn } from '@/modules/automate/helpers/graphTypes'
import { FunctionReleaseSchemaType } from '@/modules/automate/helpers/executionEngine'
import { LibsodiumEncryptionError } from '@/modules/shared/errors/encryption'
Expand Down Expand Up @@ -118,7 +118,7 @@ export type GetFunctionInputsForFrontendDeps = {
} & GetFunctionInputDecryptorDeps

export type AutomationRevisionFunctionForInputRedaction = Merge<
AutomateRevisionFunctionRecord,
AutomationRevisionFunctionRecord,
{ release: FunctionReleaseSchemaType }
>

Expand Down
7 changes: 4 additions & 3 deletions packages/server/modules/automate/services/trigger.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ import { ValidateStreamAccess } from '@/modules/core/domain/streams/operations'
import { CreateAndStoreAppToken } from '@/modules/core/domain/tokens/operations'
import { EventBusEmit } from '@/modules/shared/services/eventBus'
import { AutomationRunEvents } from '@/modules/automate/domain/events'
import { isTestEnv } from '@/modules/shared/helpers/envHelper'

export type OnModelVersionCreateDeps = {
getAutomation: GetAutomation
Expand Down Expand Up @@ -130,7 +131,7 @@ type CreateAutomationRunDataDeps = {
getFunctionInputDecryptor: FunctionInputDecryptor
}

const createAutomationRunDataFactory =
export const createAutomationRunDataFactory =
(deps: CreateAutomationRunDataDeps) =>
async (params: {
manifests: BaseTriggerManifest[]
Expand Down Expand Up @@ -413,7 +414,7 @@ type ComposeTriggerDataDeps = {
getBranchLatestCommits: GetBranchLatestCommits
}

const composeTriggerDataFactory =
export const composeTriggerDataFactory =
(deps: ComposeTriggerDataDeps) =>
async (params: {
projectId: string
Expand Down Expand Up @@ -572,7 +573,7 @@ export const createTestAutomationRunFactory =
throw new TriggerAutomationError('Automation not found')
}

if (!automationRecord.isTestAutomation) {
if (!isTestEnv() && !automationRecord.isTestAutomation) {
throw new TriggerAutomationError(
'Automation is not a test automation and cannot create test function runs'
)
Expand Down
8 changes: 5 additions & 3 deletions packages/server/modules/automate/tests/automations.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -510,9 +510,11 @@ const buildAutomationUpdate = () => {

it('fails when refering to nonexistent function releases', async () => {
const create = buildAutomationRevisionCreate({
getFunctionRelease: async () => {
// TODO: Update once we know how exec engine should respond
throw new Error('Function release with ID XXX not found')
overrides: {
getFunctionRelease: async () => {
// TODO: Update once we know how exec engine should respond
iainsproat marked this conversation as resolved.
Show resolved Hide resolved
throw new Error('Function release with ID XXX not found')
}
}
})

Expand Down
4 changes: 2 additions & 2 deletions packages/server/modules/core/graph/dataloaders/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ import { Users } from '@/modules/core/dbSchema'
import { getStreamPendingModelsFactory } from '@/modules/fileuploads/repositories/fileUploads'
import { FileUploadRecord } from '@/modules/fileuploads/helpers/types'
import {
AutomateRevisionFunctionRecord,
AutomationRevisionFunctionRecord,
AutomationRecord,
AutomationRevisionRecord,
AutomationRunTriggerRecord,
Expand Down Expand Up @@ -595,7 +595,7 @@ const dataLoadersDefinition = defineRequestDataloaders(
})
return ids.map((i) => results[i] || [])
}),
getRevisionFunctions: createLoader<string, AutomateRevisionFunctionRecord[]>(
getRevisionFunctions: createLoader<string, AutomationRevisionFunctionRecord[]>(
async (ids) => {
const results = await getRevisionsFunctions({
automationRevisionIds: ids.slice()
Expand Down
2 changes: 1 addition & 1 deletion packages/server/modules/core/services/projects.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ export const createNewProjectFactory =
const replicatedProject = await getProject({ projectId })
if (!replicatedProject) throw new StreamNotFoundError()
},
{ maxAttempts: 10 }
{ maxAttempts: 100 }
cdriesler marked this conversation as resolved.
Show resolved Hide resolved
)
} catch (err) {
if (err instanceof StreamNotFoundError) {
Expand Down
3 changes: 3 additions & 0 deletions packages/server/modules/workspaces/domain/operations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -366,3 +366,6 @@ export type CopyProjectVersions = (params: {
export type CopyProjectObjects = (params: {
projectIds: string[]
}) => Promise<Record<string, number>>
export type CopyProjectAutomations = (params: {
projectIds: string[]
}) => Promise<Record<string, number>>
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
upsertRegionAssignmentFactory
} from '@/modules/workspaces/repositories/regions'
import {
copyProjectAutomationsFactory,
copyProjectModelsFactory,
copyProjectObjectsFactory,
copyProjectsFactory,
Expand All @@ -31,6 +32,7 @@ import { getStreamBranchCountFactory } from '@/modules/core/repositories/branche
import { getStreamCommitCountFactory } from '@/modules/core/repositories/commits'
import { withTransaction } from '@/modules/shared/helpers/dbHelper'
import { getStreamObjectCountFactory } from '@/modules/core/repositories/objects'
import { getProjectAutomationsTotalCountFactory } from '@/modules/automate/repositories/automations'
import { getFeatureFlags, isTestEnv } from '@/modules/shared/helpers/envHelper'
import { WorkspacesNotYetImplementedError } from '@/modules/workspaces/errors/workspace'

Expand Down Expand Up @@ -92,6 +94,9 @@ export default {
countProjectModels: getStreamBranchCountFactory({ db: sourceDb }),
countProjectVersions: getStreamCommitCountFactory({ db: sourceDb }),
countProjectObjects: getStreamObjectCountFactory({ db: sourceDb }),
countProjectAutomations: getProjectAutomationsTotalCountFactory({
db: sourceDb
}),
getAvailableRegions: getAvailableRegionsFactory({
getRegions: getRegionsFactory({ db }),
canWorkspaceUseRegions: canWorkspaceUseRegionsFactory({
Expand All @@ -102,7 +107,8 @@ export default {
copyProjects: copyProjectsFactory({ sourceDb, targetDb }),
copyProjectModels: copyProjectModelsFactory({ sourceDb, targetDb }),
copyProjectVersions: copyProjectVersionsFactory({ sourceDb, targetDb }),
copyProjectObjects: copyProjectObjectsFactory({ sourceDb, targetDb })
copyProjectObjects: copyProjectObjectsFactory({ sourceDb, targetDb }),
copyProjectAutomations: copyProjectAutomationsFactory({ sourceDb, targetDb })
})

return await withTransaction(updateProjectRegion(args), targetDb)
Expand Down
Loading