bug: index out of range [0] with length 0 in generateOverlapQuery during subset

[chris-downs]

Version: 0.2.8
OS: osx arm64

Database: AWS Aurora Postgres v16.6

I am trying to subset a database using the following config:

....
dump:
  pg_dump_options:
    dbname: <redacted>
    host:  <redacted>
    username: read-only
    schema: public
    jobs: 10
    pgzip: true
    no-owner: true
    no-privileges: true
  transformation:
    - schema: public
      name: posts
      subset_conds:
       - "public.posts.created_at > (CURRENT_DATE - INTERVAL '30' DAY)"

But I get the following error:

2025-03-21T12:58:04-07:00 DBG github.com/greenmaskio/greenmask/internal/db/postgres/cmd/dump.go:167 > performing snapshot export pid=96240
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/greenmaskio/greenmask/internal/db/postgres/subset.generateOverlapQuery(0x0, {0x1400015e270, 0x26}, {0x14000696360, 0x1, 0x1}, {0x0, 0x0, 0x14000049458?}, 0x14000049458, ...)
        github.com/greenmaskio/greenmask/internal/db/postgres/subset/graph.go:846 +0x191c
github.com/greenmaskio/greenmask/internal/db/postgres/subset.(*Graph).generateRecursiveQueriesForCycle(0x140006a21c8?, 0x14000049748, 0x0, {0x140000777a8, 0x1, 0x1049bec30?}, {0x0, 0x0, 0x14000680008?}, {0x0, ...}, ...)
        github.com/greenmaskio/greenmask/internal/db/postgres/subset/graph.go:465 +0x5a0
github.com/greenmaskio/greenmask/internal/db/postgres/subset.(*Graph).generateQueryForScc(0x14000758360, 0x14000049748, 0x0, 0x14000691440, 0x0)
        github.com/greenmaskio/greenmask/internal/db/postgres/subset/graph.go:412 +0x2f0
github.com/greenmaskio/greenmask/internal/db/postgres/subset.(*Graph).generateQueriesSccDfs(0x14000758360, 0x14000049748, 0x14000691440, 0x0)
        github.com/greenmaskio/greenmask/internal/db/postgres/subset/graph.go:390 +0x88
github.com/greenmaskio/greenmask/internal/db/postgres/subset.(*Graph).generateAndSetQueryForScc(0x14000758360?, 0x140006912c0?)
        github.com/greenmaskio/greenmask/internal/db/postgres/subset/graph.go:374 +0x5c
github.com/greenmaskio/greenmask/internal/db/postgres/subset.SetSubsetQueries(0x14000758360)
        github.com/greenmaskio/greenmask/internal/db/postgres/subset/set_queries.go:7 +0xa4
github.com/greenmaskio/greenmask/internal/db/postgres/context.NewRuntimeContext({0x105869cd8?, 0x140000300f0?}, {0x1058709f8, 0x140000fe840}, 0x14000457268, 0x10612b500, {0x0, 0x0, 0x0}, 0x27106)
        github.com/greenmaskio/greenmask/internal/db/postgres/context/context.go:118 +0x670
github.com/greenmaskio/greenmask/internal/db/postgres/cmd.(*Dump).buildContextAndValidate(0x140000d61c0, {0x105869cd8?, 0x140000300f0?}, {0x1058709f8?, 0x140000fe840?})
        github.com/greenmaskio/greenmask/internal/db/postgres/cmd/dump.go:190 +0x64
github.com/greenmaskio/greenmask/internal/db/postgres/cmd.(*Dump).Run(0x140000d61c0, {0x105869cd8, 0x140000300f0})
        github.com/greenmaskio/greenmask/internal/db/postgres/cmd/dump.go:517 +0x300
github.com/greenmaskio/greenmask/cmd/greenmask/cmd/dump.init.func1(0x1400045c500?, {0x10534008e?, 0x4?, 0x1053400f6?})
        github.com/greenmaskio/greenmask/cmd/greenmask/cmd/dump/dump.go:57 +0x1d8
github.com/spf13/cobra.(*Command).execute(0x1060efbe0, {0x140000f0c30, 0x3, 0x3})
        github.com/spf13/[email protected]/command.go:989 +0x818
github.com/spf13/cobra.(*Command).ExecuteC(0x1060ef620)
        github.com/spf13/[email protected]/command.go:1117 +0x344
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:1041
github.com/greenmaskio/greenmask/cmd/greenmask/cmd.Execute(...)
        github.com/greenmaskio/greenmask/cmd/greenmask/cmd/root.go:60
main.main()
        github.com/greenmaskio/greenmask/cmd/greenmask/main.go:24 +0x24

Here is the schema of the posts table:

create table public.posts
(
    id              char(32)                    not null
        primary key,
    name            varchar(512),
    created_at      timestamp                   not null,
    updated_at      timestamp                   not null,
    group_id        char(32)                    not null,
    user_id         char(32)                    not null
        references public.users
            on update restrict on delete restrict,
    post_type       integer                     not null,
    parent_id       char(32)
        references public.posts
            on update restrict on delete restrict,
    root_id         char(32)
        references public.posts
            on update restrict on delete restrict,
    deleted         boolean default false       not null,
     <redacted> double precision,
     <redacted> double precision,
     <redacted> double precision,
    metadata        jsonb   default '{}'::jsonb not null,
    no_comment      boolean default false       not null,
    label_id        char(32),
    <redacted>      char(32)[]
);

The same query works on a different table, however:

  transformation:
    - schema: public
      name: messages
      subset_conds:
        - "public.messages.created_at > (CURRENT_DATE - INTERVAL '30' DAY)"

Here is the schema for the messages table:

create table public.messages
(
    id         char(32)                    not null
        primary key,
    created_at timestamp                   not null,
    updated_at timestamp                   not null,
    dst        char(32)                    not null,
    metadata   jsonb   default '{}'::jsonb not null,
    unread     boolean default false       not null,
    group_id   char(32)
        references public.groups
            on update restrict on delete restrict
);

Any idea as to why that subset works for the messages table and not the posts table?

[wwoytenko]

Hi! Thanks for reporting this — it’s definitely a bug. We're currently working on fixing issues related to subset queries with circular dependencies (it you case as well).

In your case, there's a circular dependency in the public.posts table, which references itself twice via posts.parent_id and posts.root_id.

We’ll keep you posted on the progress.