Skip to content
This repository has been archived by the owner on Sep 18, 2020. It is now read-only.

Profile csv-mapper Memory Issue #1

Open
nkuehn opened this issue Jun 19, 2015 · 0 comments
Open

Profile csv-mapper Memory Issue #1

nkuehn opened this issue Jun 19, 2015 · 0 comments

Comments

@nkuehn
Copy link
Owner

nkuehn commented Jun 19, 2015

The csv-mapper fails on the full dataset (that's why there is currenty a filter on notebooks set).

we should use https://github.com/felixge/node-memory-leak-tutorial or some other technique to find out the source of the issue.

Hopefully it's just a leak and not a general issue.
If there is a large amount of duplicate Strings that are not a leak I have made good experiences (in java) with de-duplicating the strings in memory by using Weak Sets or a corresponding pair of weak maps (you pay in CPU when loading data, but on the long run the time saved in the garbage collector is higher because garbage collection is nonlinear to the amount of memory in use). Later versions of Node already support Weak Sets via ES6 / experimental flag (io.js already defaults).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant