decodedsource

decodedsource turns whole-stream compressed inputs into either sequential decoded readers or materialized, random-access decoded sources.

It does not parse the decoded bytes. Format-specific parsing, validation, and writing belong in their own layers.

Supported envelopes

• Plain
• Gzip
• Zstandard
• Bzip2
• XZ
• LZ4 frame streams

These are treated as whole-stream storage envelopes. decodedsource does not infer or promise application-level record boundaries.

Sequential decoding

decoded, err := decodedsource.NewReader(input, decodedsource.CompressionAuto)
if err != nil {
	return err
}
defer decoded.Close()

scanner, err := unwarc.NewScanner(decoded, unwarc.ScannerOptions{
	Compression: unwarc.CompressionPlain,
})

The returned reader does not close input; the caller retains ownership of the compressed input.

Materialized random access

Materialization decodes the input once into a seekable file. The returned value implements a small Open/OpenRange/OpenAt/Size method set, which is also accepted by readers such as unwarc.

materialized, err := decodedsource.Materialize(ctx, input, decodedsource.MaterializeOptions{
	Compression: decodedsource.CompressionAuto,
	TempDir:     "/var/tmp",
})
if err != nil {
	return err
}
defer materialized.Close()

scanner, err := unwarc.NewScannerFromSource(
	materialized.Source(),
	unwarc.ScannerOptions{Compression: unwarc.CompressionPlain},
)

Temporary materializations are deleted by Close. To transfer ownership to the caller and keep the file, persist it before opening scanners:

if err := materialized.Persist("/archive/crawl.warc", decodedsource.PersistOptions{}); err != nil {
	return err
}

Alternatively, set MaterializeOptions.Destination to create a persistent file directly. Output is written to a temporary file first and installed only after successful decoding and syncing.

License

CC0-1.0