Convert some panics that happen on invalid parquet files to error results #6738

jp0317 · 2024-11-16T06:09:04Z

Which issue does this PR close?

This solves some of #6737.

Rationale for this change

Some code changes to replace some panics with proper errors

What changes are included in this PR?

Some codes that lead to panic are converted to returning error results.

Are there any user-facing changes?

Behavior change from panics to errors.

tustvold

I'm not sure about this, this seems to add a number of untested additional checks, some to very hot codepaths.

I suggest rather than just looking for things that might panic, instead going from a failing test to a fix. This would also better capture the more problematic cases where the reader gets stuck on malformed input, a panic is a good outcome IMO...

tustvold · 2024-11-16T08:08:35Z

parquet/src/file/metadata/mod.rs

@@ -959,17 +959,18 @@ impl ColumnChunkMetaData {
    }

    /// Returns the offset and length in bytes of the column chunk within the file
-    pub fn byte_range(&self) -> (u64, u64) {
+    pub fn byte_range(&self) -> Result<(u64, u64)> {


This is a breaking API change

yeah..is it fine given that it just wraps the return value within Result? The behavior change is just "panic --> error".

No this is a breaking change

removed this change. btw i saw that the planned 54.0.0 will have "major api change", how is an api change considered as acceptable breaking change??

parquet/src/format.rs

tustvold · 2024-11-16T10:09:06Z

parquet/src/schema/types.rs

@@ -1227,6 +1231,10 @@ fn from_thrift_helper(elements: &[SchemaElement], index: usize) -> Result<(usize
                if !is_root_node {
                    builder = builder.with_repetition(rep);
                }
+            } else if !is_root_node {


Do we need this check?

this is based on the comment at line 1230 which says All other types must have one, and the assert at line 1066: assert!(tp.get_basic_info.()has_repetitio())

My concern with this check is unless it is necessary for correctness, there is potential adding it breaks something for someone. Parquet is a very broad ecosystem, lots of writers have interesting interpretations of the specification

it breaks something for someone

if it happened then someone must have already experienced panics. But i get your point, removed this change. thx

tustvold · 2024-11-16T10:09:42Z

parquet/src/thrift.rs

@@ -67,7 +67,17 @@ impl<'a> TCompactSliceInputProtocol<'a> {
        let mut shift = 0;
        loop {
            let byte = self.read_byte()?;
-            in_progress |= ((byte & 0x7F) as u64) << shift;
+            let val = (byte & 0x7F) as u64;
+            let val = val.checked_shl(shift).map_or_else(


This is a very performance critical code path, this probably should use wrapping_shl

i'm afraid wrapping might cause correctness issues...besides, would the checked_shl really make a noticeable performance difference here?

It shouldn't cause correctness issues, and yes it will matter. There are benchmarks that will likely show this

okay, changed to wrapping_shl, thx!

tustvold · 2024-11-16T10:10:34Z

parquet/src/thrift.rs

 impl TInputProtocol for TCompactSliceInputProtocol<'_> {
    fn read_message_begin(&mut self) -> thrift::Result<TMessageIdentifier> {
-        unimplemented!()
+        thrift_unimplemented!()


This should be unreachable, a panic is the correct thing to do here

iiuc it "should be unreachable" unless the input file is malformed? I guess this goes back to the discussion on how to handle invalid inputs:

No it is actually genuinely unreachable, we don't use thrift messages

changed back to unimplemented

jp0317 · 2024-11-19T03:28:54Z

Thanks @tustvold for the review!

this seems to add a number of untested additional checks...

I think the changes are more about converting panics to errors, rather than actual code logic.

looking for things that might panic, instead going from a failing test to a fix

these panics were triggered in my own fuzzing test with invalid parquet files. Nevertheless, i think it's a similar topic of "how to handle invalid inputs" as discussed in #5323. Reading this doc, imho errors better than panics unless it's really something unrecoverable.

jp0317 · 2024-11-20T05:06:53Z

Hi @tustvold, I removed some changes based on your comment. PTAL, thanks!

etseidl

Thanks! Just a few nits. However, I wonder if perhaps this should be multiple PRs with rationales given for each change.

etseidl · 2024-11-20T19:05:23Z

parquet/src/file/metadata/reader.rs

+        let column_orders =
+            Self::parse_column_orders(t_file_metadata.column_orders, &schema_descr)?;


I realize this would currently panic, but would one ever prefer to just set column_orders to None and continue? The only impact AFAIK would be statistics being unusable, which would only matter if predicates were in use.

good point! i agree with the setting to None idea. but i guess this worths a separate issue to discuss and fix.

etseidl · 2024-11-20T19:05:44Z

parquet/src/file/statistics.rs

+                if let Some(min) = min {
+                    if min.len() < len {
+                        return Err(ParquetError::General(
+                            "Insufficient bytes to parse max statistic".to_string(),


Suggested change

"Insufficient bytes to parse max statistic".to_string(),

"Insufficient bytes to parse min statistic".to_string(),

thanks for catching this.

jp0317 · 2024-11-21T00:30:05Z

Thanks! Just a few nits. However, I wonder if perhaps this should be multiple PRs with rationales given for each change.

Thanks for the review! imho these changes share the same rationale in that they just convert panics to errors

etseidl

This all looks correct to me. I personally would prefer errors to panics when processing multiple files.

jp0317 · 2024-11-26T17:37:57Z

Hi @tustvold could you please take another look at this one? Thanks!

github-actions bot added the parquet Changes to the parquet crate label Nov 16, 2024

jp0317 force-pushed the panic branch from 18c494b to 8235bf3 Compare November 16, 2024 06:25

Reduce panics

f481dff

jp0317 force-pushed the panic branch from 8235bf3 to f481dff Compare November 16, 2024 06:29

tustvold requested changes Nov 16, 2024

View reviewed changes

t pushmove integer logical type from format.rs to schema type.rs

a4f8286

jp0317 force-pushed the panic branch from 42f0223 to a4f8286 Compare November 19, 2024 03:30

jp0317 requested a review from tustvold November 19, 2024 03:32

jp0317 added 2 commits November 20, 2024 04:45

remove some changes as per reviews

a88bc81

use wrapping_shl

3c97bbc

etseidl reviewed Nov 20, 2024

View reviewed changes

fix typo in error message

a7db494

jp0317 requested a review from etseidl November 21, 2024 00:43

etseidl mentioned this pull request Nov 22, 2024

Read nested Parquet 2-level lists correctly #6757

Merged

etseidl approved these changes Nov 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert some panics that happen on invalid parquet files to error results #6738

Convert some panics that happen on invalid parquet files to error results #6738

jp0317 commented Nov 16, 2024

tustvold left a comment

tustvold Nov 16, 2024

jp0317 Nov 19, 2024

tustvold Nov 19, 2024

jp0317 Nov 20, 2024

tustvold Nov 16, 2024

jp0317 Nov 19, 2024

tustvold Nov 19, 2024

jp0317 Nov 20, 2024

tustvold Nov 16, 2024

jp0317 Nov 19, 2024

tustvold Nov 19, 2024

jp0317 Nov 20, 2024

tustvold Nov 16, 2024

jp0317 Nov 19, 2024

tustvold Nov 19, 2024

jp0317 Nov 20, 2024

jp0317 commented Nov 19, 2024

jp0317 commented Nov 20, 2024

etseidl left a comment

etseidl Nov 20, 2024

jp0317 Nov 21, 2024

etseidl Nov 20, 2024

jp0317 Nov 21, 2024

jp0317 commented Nov 21, 2024

etseidl left a comment

jp0317 commented Nov 26, 2024

		let column_orders =
		Self::parse_column_orders(t_file_metadata.column_orders, &schema_descr)?;

	"Insufficient bytes to parse max statistic".to_string(),
	"Insufficient bytes to parse min statistic".to_string(),

Convert some panics that happen on invalid parquet files to error results #6738

Are you sure you want to change the base?

Convert some panics that happen on invalid parquet files to error results #6738

Conversation

jp0317 commented Nov 16, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

tustvold left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jp0317 commented Nov 19, 2024

jp0317 commented Nov 20, 2024

etseidl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jp0317 commented Nov 21, 2024

etseidl left a comment

Choose a reason for hiding this comment

jp0317 commented Nov 26, 2024