Here is the start of a small and good discussion on how to read large data
sets into many. They appears frequently on the list, but this one seemed to
touch upon a lot of the recommended approaches:
- ColByCol
- Mentions true cost of loading a 1 GiB file with
read.table
, which is much
more than 1 GiB
- Mentions true cost of loading a 1 GiB file with
- R.filesets
- Large memory and out-of-memory data in High-Performance and Parallel Computing with R
- Use a real database with SQLite or sqldf