Read compressed elements without loading into memory #34

2025-04-05T10:45:18Z

Rob-Hague commented

2025-04-05 10:45:18 +00:00

(Migrated from github.com)

This change removes the load of compressed variables into a MemoryStream before reading them, instead reading out of the original stream.

In testing on some files of sizes up to 70MB the allocated memory when reading was reduced by 50-70%

I wrote the first commit as part of the change but it stands alone as a fix for reading unseekable streams (where stream.Position throws e.g. DeflateStream.Position) or streams where the mat file does not start at 0 (so stream.Position does not work for padding adjustments). It basically just runs all the tests using different factories for different cases

This change removes the load of compressed variables into a MemoryStream before reading them, instead reading out of the original stream. In testing on some files of sizes up to 70MB the allocated memory when reading was reduced by 50-70% I wrote the first commit as part of the change but it stands alone as a fix for reading unseekable streams (where stream.Position throws e.g. DeflateStream.Position) or streams where the mat file does not start at 0 (so stream.Position does not work for padding adjustments). It basically just runs all the tests using different factories for different cases

mahalex commented

2025-04-06 10:27:01 +00:00

(Migrated from github.com)

This is great! Thanks for your contribution!

Rob-Hague commented

2025-04-06 10:48:46 +00:00

(Migrated from github.com)

Thanks!

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mahalex/MatFileHandler#34