Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feedback] Use binary format and compression for table copy #2983

Open
wrobell opened this issue Feb 6, 2024 · 1 comment
Open

[Feedback] Use binary format and compression for table copy #2983

wrobell opened this issue Feb 6, 2024 · 1 comment
Assignees
@billy-the-fish
Labels
community Raised by a community member documentation Improvements or additions to documentation feedback Submissions from the docs site feedback form

Comments

@wrobell
Copy link

wrobell commented Feb 6, 2024

Is it easy to find the information you need?

None

Are the instructions clear?

Yes

How could we improve the Timescale documentation site?

The documentation at

https://docs.timescale.com/self-hosted/latest/migration/schema-then-data/

suggests to use CSV format when copying a table.

IMHO, a better option could be use of binary format with compression (if possible)

https://www.postgresql.org/docs/current/sql-copy.html

For example

\copy (select * from position) to program 'zstd > position.dump.zst' binary

results in 14 GB file as opposed to estimated[1] 120 GB.

[1] I killed it at 10% when using CSV file format.
@wrobell wrobell added community Raised by a community member documentation Improvements or additions to documentation feedback Submissions from the docs site feedback form labels Feb 6, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 6, 2024

Thank you for the report. We welcome documentation contributions!

  • For information about how to propose a change, see the contributing guide in our GitHub repository.
  • For information on style and word usage, see the style guide

@billy-the-fish billy-the-fish self-assigned this Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@billy-the-fish billy-the-fish

Labels
community Raised by a community member documentation Improvements or additions to documentation feedback Submissions from the docs site feedback form
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wrobell @billy-the-fish