HAWQ 1078. Implement hawqsync-falcon DR utility.

[kdunn926]

This is the initial commit for a Python utility to orchestrate a DR syncronization for HAWQ, based on Falcon HDFS replication and a cold backup of the active HAWQ master's MASTER_DATA_DIRECTORY.

A code review would be greatly appreciated, when someone has cycles. Active testing is currently underway in a production deployment.

[kdunn-pivotal]

@vVineet How can we get this prioritized for the next release? Also, anyone that can put eyes on it for a code review would be helpful.

[vVineet]

@kdunn-pivotal : I propose a discussion including @ictmalili as this ties with HAWQ Register feature. I'd love to see the contribution make it in HAWQ.

[ictmalili]

Quite a lot of work here. Could you describe a step-by-step instruction that users can do DR? Thanks a lot!

[ictmalili]

@kdunn926 , may I understand till here all the files are copied to new HAWQ master data directory, but HAWQ catalog information has not been changed. We can leverage hawq register to register the files into HAWQ?

[kdunn-pivotal]

@ictmalili - At this point in the process several things have happened on the source cluster:

1. the source HAWQ Master Data Directory (MDD) has been tarballed
2. the tarball has been copied to HDFS in the /hawq_default directory on the source
3. The /hawq_default directory has been recursively copied to the DR cluster via Apache Falcon (distcp)
4. A checksum generated by a recursive listing of files+sizes has been generated for each cluster (source and DR) and successfully validated

Basically, by the time this point in the code is reached, the source HAWQ system (data & metadata) is completely archived & verified in the DR site.

[ictmalili]

Which specific directory does this map to?

[ictmalili]

Is it HAWQ master's MASTER_DATA_DIRECTORY?

[kdunn-pivotal]

Yes, the MDD archive is the MASTER_DATA_DIRECTORY from the source HAWQ cluster.

[kdunn-pivotal]

HAWQSYNC partial-sync recovery runbook:

1. Copy "last known good state" tarball from hdfs://hawq_default/hawqExtract-*.tar.bz2

2. Re-run hawqsync-extract to establish "current state".

3. Perform diff's for every table file, determine tables with inconsistencies.

4. For each inconsistent table:
   Re-register faultyTable using "last known good" YAML - (updates the EOF field only).
   a. hawq register --force -f faultyTable.yaml faultyTable

   Store the valid records in a temporary table
   b. CREATE TABLE newTemp AS SELECT * FROM faultyTable

   Truncate the faulty table, to allow the catalog and HDFS file sizes to be consistent again
   c. TRUNCATE faultyTable

   Re-populate the table with valid records
   d. INSERT INTO faultyTable SELECT * FROM newTemp

   Purge the temporary table
   e. DROP TABLE newTemp

This process, overall, ensures our catalog EOF marker and actual HDFS file size are properly aligned for every table. This is especially important when ETL needs to resume on tables that may have previously had "inconsistent bytes" appended, as would be the case for a partial sync.