HIVE-20889: Support timestamp-micros in AvroSerDe #5779

armitage420 · 2025-04-21T16:43:31Z

What changes were proposed in this pull request?

Timestamps for avro file format in Hive supports timestamp-millis. This PR brings in the feature where Hive can now support Timestamp-micros logicaltype for avro too.
Some details on Logical Types of Avro here: https://gist.github.com/armitage420/fa1432f4dbda56f67bdb08c252415623
More details on Jira description is already present: HIVE-20889

Why are the changes needed?

These changes will increase the reading of timestamps precision for users.

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Added new qfile and other existing tests

SourabhBadhya

@armitage420 This is nice contribution!
However, you might have to cover with some tests for a variety of cases to avoid any timestamp issues. I would highly advise to extend existing tests for timestamp-millis to timestamp-micros.

serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java

serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java

SourabhBadhya · 2025-05-02T14:39:27Z

serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java

+            (AvroSerDe.TIMESTAMP_TYPE_NAME_MILLIS.equals(schema.getProp(AvroSerDe.AVRO_PROP_LOGICAL_TYPE)) ||
+                    AvroSerDe.TIMESTAMP_TYPE_NAME_MICROS.equals(schema.getProp(AvroSerDe.AVRO_PROP_LOGICAL_TYPE)))) {


Consider creating a function to handle all these timestamp cases. Currently its only 2 cases, but looking at the extension itself, in the future it can be more.

@armitage420 I am already seeing a Timestamp-nanos now in Avro specification.
https://avro.apache.org/docs/1.12.0/specification/#timestamps

SourabhBadhya · 2025-05-02T14:40:56Z

serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java

@@ -159,7 +159,7 @@ private Schema createAvroPrimitive(TypeInfo typeInfo) {
      case TIMESTAMP:
        schema = AvroSerdeUtils.getSchemaFor("{" +
          "\"type\":\"" + AvroSerDe.AVRO_LONG_TYPE_NAME + "\"," +
-          "\"logicalType\":\"" + AvroSerDe.TIMESTAMP_TYPE_NAME + "\"}");
+          "\"logicalType\":\"" + AvroSerDe.TIMESTAMP_TYPE_NAME_MICROS + "\"}");


Why this change? Does this have the effect of making the type name as timestamp-micros as default?

Yes, it does make parsing avro timestamps in microsecond precision in Hive as default. Before it was parsed in millisecond precision by default.

Consider keeping it as millis to make it backward compatible. We dont want to surprise users with this change.

I wonder how will users differentiate between millis and micros in their table schema. Is it only possible by providing a custom avro schema in avro.schema.url which explicitly mentions the timestamp column as timestamp-micros?

SourabhBadhya · 2025-05-02T15:19:50Z

serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java

      long millis = defaultProleptic ? timestamp.toEpochMilli() :
-          CalendarUtils.convertTimeToHybrid(timestamp.toEpochMilli());
+              CalendarUtils.convertTimeToHybrid(timestamp.toEpochMilli());


nit: Remove unnecessary whitespace changes to avoid rewriting git history

SourabhBadhya · 2025-05-02T15:24:51Z

storage-api/src/java/org/apache/hadoop/hive/common/type/CalendarUtils.java

+  public static long convertTimeToHybridMicros(long prolepticMicros) {
+    long hybridMicros = prolepticMicros;
+    long prolepticMillis = prolepticMicros / 1_000L; // Convert micros to millis
+
+    if (prolepticMillis < SWITCHOVER_MILLIS) {
+      String dateStr = PROLEPTIC_TIME_FORMAT.get().format(new Date(prolepticMillis));
+      try {
+        hybridMicros = HYBRID_TIME_FORMAT.get().parse(dateStr).getTime() * 1_000L; // Convert millis back to micros
+      } catch (ParseException e) {
+        throw new IllegalArgumentException("Can't parse " + dateStr, e);
+      }
+    }
+    return hybridMicros;
+  }


Same as earlier, consider keeping it as micros due to loss in precision while changing to millis.

SourabhBadhya · 2025-05-02T15:31:51Z

common/src/java/org/apache/hadoop/hive/common/type/Timestamp.java

@@ -162,6 +162,11 @@ public long toEpochMilli() {
    return localDateTime.toInstant(ZoneOffset.UTC).toEpochMilli();
  }

+  public long toEpochMicro() {
+    return localDateTime.toEpochSecond(ZoneOffset.UTC) * 1_000_000


Can we use similar semantics as toEpochMilli()?

return localDateTime.toInstant(ZoneOffset.UTC).toEpochMilli() * 1000 + localDateTime.getNano() / 1000;

Doing so might lead to loss of precision due to integer overflow. Hence we are parsing the value to seconds here.

localDateTime.toInstant(ZoneOffset.UTC).toEpochMilli() returns a long value and due to addition of all these values the implicit datatype is long hence there should be no chance of integer overflow.

SourabhBadhya · 2025-05-02T15:32:20Z

common/src/java/org/apache/hadoop/hive/common/type/Timestamp.java

+    return localDateTime.toEpochSecond(ZoneOffset.UTC) * 1_000_000
+            + localDateTime.getNano() / 1000;
+  }
+
  public long toEpochMilli(ZoneId id) {


Isn't zone conversion also applicable for micros?

I could not find any code changes related to zone, and hence did not add extra unnecessary changes. Although, adding a similar method for future does make sense.

SourabhBadhya · 2025-05-02T15:33:08Z

ql/src/test/queries/clientpositive/avro_timestamp_micros.q

@@ -0,0 +1,3 @@
+CREATE EXTERNAL TABLE hive_test(`dt` timestamp) STORED AS AVRO;


This needs to have more test cases. Consider atleast referring to existing test case from millis and extend it for micros.
We are handling a lot of cases, such as proleptic calendar, zone based conversions etc.

Understandable, adding equivalent test cases to avro in milliseconds precision, and some more test classes as required

SourabhBadhya · 2025-05-02T15:36:49Z

serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java

+      }
+      timestamp = TimestampTZUtil.convertTimestampToZone(
+              timestamp, ZoneOffset.UTC, convertToTimeZone, legacyConversion);
+      if (!skipProlepticConversion && logicalType != null && logicalType.getName().equals(AvroSerDe.TIMESTAMP_TYPE_NAME_MICROS)) {


This can be simplified into a nested if block -
if (!skipProlepticConversion) {
// Add other if checks
....
}

Also use equalsIgnoreCase

armitage420 · 2025-05-05T15:54:37Z

@SourabhBadhya Thank you so much for the review! I will fix the required changes as soon as possible :D

… for zoneId method

armitage420 · 2025-05-12T16:31:17Z

@SourabhBadhya I have made the required changes, I hope you could have a look at it, when you have time!
P.P.S- The test failures are unrelated to my changes and everyone recently running the pipeline having the same failures. I will re-trigger for green build, post this issue gets fixed :D

sonarqubecloud · 2025-05-13T05:50:49Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

armitage420 · 2025-05-17T17:18:18Z

@SourabhBadhya The build is green! Please look at the changes when you have the time! Thank you ✨

SourabhBadhya · 2025-05-18T04:15:12Z

serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java

+      LogicalType logicalType = schema.getLogicalType();
+      if (logicalType != null && logicalType.getName().equalsIgnoreCase(AvroSerDe.TIMESTAMP_TYPE_NAME_MICROS)) {
+        long micros = defaultProleptic ? timestamp.toEpochMicro() :
+                CalendarUtils.convertTimeToHybridMicros(timestamp.toEpochMicro());
+        timestamp = TimestampTZUtil.convertTimestampToZone(
+                Timestamp.ofEpochMicro(micros), TimeZone.getDefault().toZoneId(), ZoneOffset.UTC, legacyConversion);
+        return timestamp.toEpochMicro();
+      }
      long millis = defaultProleptic ? timestamp.toEpochMilli() :
          CalendarUtils.convertTimeToHybrid(timestamp.toEpochMilli());
      timestamp = TimestampTZUtil.convertTimestampToZone(


Consider creating a function which would pass the required fields and return timestamp object.

This is because of enhancing extensibility for timestamp-nanos.

SourabhBadhya · 2025-05-18T04:19:21Z

serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java

+    if (logicalType == null) {
+      return false;
+    }
+
+    // Supported timestamp logical types (extend this set as needed)
+    return logicalType.equalsIgnoreCase(AvroSerDe.TIMESTAMP_TYPE_NAME_MILLIS) ||
+            logicalType.equalsIgnoreCase(AvroSerDe.TIMESTAMP_TYPE_NAME_MICROS);


Its better to replace this code with -
AvroSerDe.TIMESTAMP_TYPE_NAME_MILLIS.equalsIgnoreCase(logicalType) || AvroSerDe.TIMESTAMP_TYPE_NAME_MICROS.equalsIgnoreCase(logicalType);

SourabhBadhya · 2025-05-18T04:19:49Z

serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java

@@ -227,7 +227,7 @@ public class TestAvroObjectInspectorGenerator {
      "  \"fields\" : [\n" +
      "    {\"name\":\"timestampField\", " +
      "     \"type\":\"" + AvroSerDe.AVRO_LONG_TYPE_NAME + "\", " +
-      "         \"logicalType\":\"" + AvroSerDe.TIMESTAMP_TYPE_NAME + "\"}" +
+      "         \"logicalType\":\"" + AvroSerDe.TIMESTAMP_TYPE_NAME_MILLIS + "\"}" +


Remove this and add explicit test cases for micros.

SourabhBadhya · 2025-05-18T04:23:04Z

serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroSerializer.java

@@ -125,7 +125,7 @@ public void canSerializeDoubles() throws SerDeException, IOException {
  public void canSerializeTimestamps() throws SerDeException, IOException {
    singleFieldTest("timestamp1", Timestamp.valueOf("2011-01-01 00:00:00").toEpochMilli(),
        "\"" + AvroSerDe.AVRO_LONG_TYPE_NAME + "\"," +
-        "\"logicalType\":\"" + AvroSerDe.TIMESTAMP_TYPE_NAME + "\"");
+        "\"logicalType\":\"" + AvroSerDe.TIMESTAMP_TYPE_NAME_MILLIS + "\"");


Remove this and add explicit test cases for micros.

SourabhBadhya · 2025-05-18T04:24:06Z

serde/src/test/org/apache/hadoop/hive/serde2/avro/TestTypeInfoToSchema.java

@@ -261,7 +261,7 @@ public void createAvroDateSchema() {
  public void createAvroTimestampSchema() {
    final String specificSchema = "{" +
      "\"type\":\"long\"," +
-      "\"logicalType\":\"timestamp-millis\"}";
+      "\"logicalType\":\"timestamp-micros\"}";


Remove this and add explicit test cases for micros.

SourabhBadhya · 2025-05-18T06:02:15Z

ql/src/test/queries/clientpositive/avro_hybrid_mixed_timestamp.q

 ('0947-02-11 07:15:11.123'),
-('0200-02-11 07:15:11.123');
+('0200-02-11 07:15:11.1234');


Always refrain from modifying existing tests. Considering adding new tests.

SourabhBadhya · 2025-05-18T06:03:17Z

ql/src/test/queries/clientpositive/avro_timestamp_micros.q

+('2012-02-21 07:08:09.123'),
+('1014-02-11 07:15:11.12345');
+
+SELECT * FROM micros_table;


Consider adding a test with timestamp-micros as a column and timestamp-millis as a column.

SourabhBadhya · 2025-05-18T06:04:34Z

ql/src/test/queries/clientpositive/avro_proleptic_mixed_timestamp.q

-('0947-02-11 07:15:11.123'),
-('0200-02-11 07:15:11.123');
+('0947-02-11 07:15:11.12345'),
+('0200-02-11 07:15:11.123456');


Refrain from changing existing tests. Add a new test for timestamp-micros explicitly mentioning its column type.

asf-ci-hive added tests pending tests passed and removed tests pending labels Apr 21, 2025

SourabhBadhya requested changes May 2, 2025

View reviewed changes

asf-ci-hive added tests pending tests failed and removed tests passed tests pending tests failed labels May 11, 2025

araika added 4 commits May 12, 2025 12:35

HIVE-20889: Support timestamp-micros in AvroSerDe

ce3758f

HIVE-20889: Add microsecond precision methods

3602cfc

HIVE-20889: Decouple avro schema timestamp check and add toEpochMicro…

8fdeb5a

… for zoneId method

HIVE-20889: extend avro test cases

578f94a

armitage420 force-pushed the HIVE-20889 branch from 3bc2895 to 578f94a Compare May 12, 2025 07:06

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels May 12, 2025

dba1d0d

asf-ci-hive added tests pending and removed tests unstable labels May 13, 2025

asf-ci-hive added tests passed and removed tests pending labels May 13, 2025

SourabhBadhya requested changes May 18, 2025

View reviewed changes

		(AvroSerDe.TIMESTAMP_TYPE_NAME_MILLIS.equals(schema.getProp(AvroSerDe.AVRO_PROP_LOGICAL_TYPE)) \|\|
		AvroSerDe.TIMESTAMP_TYPE_NAME_MICROS.equals(schema.getProp(AvroSerDe.AVRO_PROP_LOGICAL_TYPE)))) {

		@@ -0,0 +1,3 @@
		CREATE EXTERNAL TABLE hive_test(`dt` timestamp) STORED AS AVRO;

HIVE-20889: Support timestamp-micros in AvroSerDe #5779

Are you sure you want to change the base?

HIVE-20889: Support timestamp-micros in AvroSerDe #5779

Uh oh!

Conversation

armitage420 commented Apr 21, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SourabhBadhya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SourabhBadhya May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

armitage420 commented May 5, 2025

Uh oh!

armitage420 commented May 12, 2025

Uh oh!

sonarqubecloud bot commented May 13, 2025

Quality Gate passed

Uh oh!

armitage420 commented May 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SourabhBadhya May 2, 2025 •

edited

Loading