Tdl 20357 add missing tap tester tests #53

NevilParikh14 · 2022-08-29T05:27:55Z

Description of change

Added all the missing integration test

Automatic fields test
All fields test
Discovery test
Parent-Child Independent test

Added all the missing assertions

Bookmarks test
Start date test

Separated Bookmark test for "tickets" and "contacts stream to handle their separate bookmarks and RECORDS collected with the different filter params.

Note: Build will pass after discovery mode is implemented.

Manual QA steps

Risks

Rollback steps

revert this branch

savan-chovatiya · 2022-08-29T09:33:43Z

tests/base.py


+    @staticmethod
+    def get_selected_fields_from_metadata(metadata):


Add function doc string.

savan-chovatiya · 2022-08-29T09:34:10Z

tests/test_freshdesk_all_fields.py

+
+from base import FreshdeskBaseTest
+
+# As we are not able to generate following fields by Freshdesk UI, so removed it form expectation list.


Suggested change

# As we are not able to generate following fields by Freshdesk UI, so removed it form expectation list.

# As we are not able to generate following fields by Freshdesk UI, so removed it from expectation list.

savan-chovatiya · 2022-08-29T09:38:06Z

tests/test_freshdesk_bookmarks.py

+        # Tickets and Contacts stream also collectes some deleted data on the basis of filter param.
+        # Written seprate bookmark test case for them in test_freshdesk_bookmarks_stream_with_fillter_param.py
+        # To collect "time_entries", "satisfaction_ratings" pro account is needed. Skipping them for now.


Suggested change

# Tickets and Contacts stream also collectes some deleted data on the basis of filter param.

# Written seprate bookmark test case for them in test_freshdesk_bookmarks_stream_with_fillter_param.py

# To collect "time_entries", "satisfaction_ratings" pro account is needed. Skipping them for now.

# Tickets and Contacts stream also collect some deleted data on the basis of filter param.

# Written separate bookmark test case for them in test_freshdesk_bookmarks_stream_with_fillter_param.py

# To collect "time_entries", "satisfaction_ratings" pro account is needed. Skipping them for now.

savan-chovatiya · 2022-08-29T09:41:24Z

tests/test_freshdesk_bookmarks_stream_with_fillter_param.py

+    """Test tap sets a separate bookmark for PUBLISHED and SCHEDULED tweets and 
+    respects it for the next sync of a tweets stream"""


Please update the above comment, It looks incorrect for this test.

savan-chovatiya · 2022-08-29T09:50:51Z

tests/test_freshdesk_pagination.py

+                # Not able to generate more data as roles stream requires pro account.
+                # So, updating page_sie according to data available.
+                if stream == "roles":
+                    page_size = 2


Does the tap have a default page_size of 2 for the roles stream? If not, then are we passing it using config?

Updated test case to use configurable 'page_size' param instead of if condition.

namrata270998 · 2022-08-29T09:58:41Z

tests/test_freshdesk_start_date.py

@@ -1,170 +1,148 @@
 import os
-
-from tap_tester import connections, runner
+import requests


Remove the unused import

namrata270998 · 2022-08-29T10:02:33Z

tests/test_freshdesk_start_date.py


 from base import FreshdeskBaseTest
+from datetime import datetime, timedelta


Remove this import if not used anywhere

namrata270998 · 2022-08-29T12:40:33Z

tests/test_freshdesk_start_date.py

-        # instantiate connection
-        conn_id_1 = connections.ensure_connection(self)
+        # Instantiate connection
+        conn_id_1 = connections.ensure_connection(self, original_properties=False)


We are using the original start_date here, so can we remove the original_properties=False from the ensure_connection?

tests/test_freshdesk_pagination.py

tests/test_freshdesk_start_date.py

dbshah1212 · 2022-09-08T10:53:28Z

tests/base.py

        return {table: properties.get(self.REPLICATION_METHOD, None)
                for table, properties
                in self.expected_metadata().items()}

-    def expected_streams(self):
+    def expected_streams(self, all_streams: bool = False):
        """A set of expected stream names"""


Update this comment

dbshah1212 · 2022-09-08T10:58:49Z

tests/test_freshdesk_parent_child_independent.py

+        """
+
+        # Test for the case of child is selected and parent is not selected
+        # To collect "time_entries", "satisfaction_ratings" pro account is needed. Skipping them for now.


Remove this comment and add comments for the conversations stream.

Updated the comment.

dbshah1212 · 2022-09-08T10:59:49Z

tests/test_freshdesk_pagination.py


-                sync_messages = sync_records.get(stream, {'messages': []}).get('messages')
+        # To collect "time_entries", "satisfaction_ratings" pro account is needed. Skipping them for now.
+        expected_streams = streams_to_test


Update comment in all tests. As not excluding here, no need to mention it.

dbshah1212 · 2022-09-12T07:47:18Z

tests/base.py

-    def expected_streams(self):
-        """A set of expected stream names"""
-        return set(self.expected_metadata().keys())
+    def expected_streams(self, all_streams: bool = False):


Instead of all_streams use only_trial_account_stream, take default value False

tests/test_freshdesk_parent_child_sync.py

dbshah1212 · 2022-09-19T05:05:49Z

tests/test_freshdesk_parent_child_sync.py

@@ -105,7 +105,8 @@ def run_test(self, new_state, stream_to_test, minimum_bookmark, maximum_bookmark
                    # Verify that the minimum bookmark is used for selected parent-child stream.
                    replication_key_value = self.dt_to_ts(
                        record.get(replication_key), self.BOOKMARK_FORMAT)
-
+
+                    # Verify that the records replicated for the selected streams are >= given bookmark.


Suggested change

# Verify that the records replicated for the selected streams are >= given bookmark.

# Verify that the records replicated for the selected streams are greater than or equal to the given bookmark.

Updated comment as per suggestion.

sgandhi1311 · 2022-09-22T11:50:18Z

tests/base.py

+            calculated_state_as_datetime = state_as_datetime - timedelta(days=days, hours=hours, minutes=minutes)
+
+            state_format = self.BOOKMARK_FORMAT
+            calculated_state_formatted = datetime.datetime.strftime(calculated_state_as_datetime, state_format)


Instead of datetime.datetime use dt.

Replaced datetime.datetime with dt.

sgandhi1311 · 2022-09-22T11:52:13Z

tests/base.py

@@ -1,11 +1,12 @@
 import os
 import unittest
+import dateutil.parser
+import datetime


Remove unused datetime package.

Removed datetime import.

sgandhi1311 · 2022-09-23T11:41:55Z

tests/base.py

@@ -17,17 +17,16 @@ class FreshdeskBaseTest(unittest.TestCase):
    INCREMENTAL = "INCREMENTAL"
    FULL = "FULL_TABLE"

-    START_DATE_FORMAT = "%Y-%m-%dT00:00:00Z" # %H:%M:%SZ
-    BOOKMARK_FORMAT = "%Y-%m-%dT%H:%M:%SZ"
+    start_date = ""


Is start_date variable with blank string really required?

Yes, in the start_date test case we override the start date using this variable.

sgandhi1311 · 2022-09-23T12:11:06Z

tests/test_freshdesk_discovery.py

+        return "tap_tester_freshdesk_discovery"
+
+    def test_run(self):
+        """


Can we test below scenarios?

Verify there are no duplicate/conflicting metadata entries.

Verify replication key(s) match expectations.

Verify the actual replication matches our expected replication method.

Verify that if there is a replication key we are doing INCREMENTAL otherwise FULL.

Verify all streams have inclusion of automatic

All the assertions mentioned above are already added to this file.

sgandhi1311 · 2022-09-23T12:18:02Z

tests/test_freshdesk_pagination.py

-        # Run check mode
-        # Check mode has no catalog discovery for freshdesk
-        check_job_name = self.run_and_verify_check_mode(conn_id)
+    def run_test(self, streams_to_test, page_size):


Rename the function argument from streams_to_test to expected_streams. And remove expected_streams = streams_to_test from below.

Updated code as per the above suggestion.

sgandhi1311 · 2022-09-23T12:38:08Z

tests/test_freshdesk_start_date.py

    start_date_1 = ""
    start_date_2 = ""


Remove these 2 lines as those are not needed.

This was already present as an existing code but updated as per your suggestion.

NevilParikh14 added 8 commits August 24, 2022 18:17

initial commit

25aabda

Updated test cases

3beaf78

added contacts stream in base.py

4d98589

updated bookmark test cases

55e9e53

Removed unused line

378ad79

updated bookmark with filter param test case

2286d1f

updated all fileds test case

e1aafd9

Updated to LOGGER from print

ba6395b

NevilParikh14 requested review from karanpanchal-crest, dbshah1212, namrata270998, savan-chovatiya and prijendev August 29, 2022 05:27

NevilParikh14 added 2 commits August 29, 2022 11:01

Updated assertion message

010217d

added back commented print statement

f2313b8

savan-chovatiya reviewed Aug 29, 2022

View reviewed changes

namrata270998 reviewed Aug 29, 2022

View reviewed changes

NevilParikh14 added 3 commits August 29, 2022 17:25

resolved review comments

5fc93e0

Updated pagination test case

3ecf64b

uncommented LOGGERS

5c3ccd5

namrata270998 reviewed Aug 29, 2022

View reviewed changes

umangagrawal-crest reviewed Aug 30, 2022

View reviewed changes

tests/test_freshdesk_pagination.py Outdated Show resolved Hide resolved

umangagrawal-crest reviewed Aug 30, 2022

View reviewed changes

tests/test_freshdesk_start_date.py Outdated Show resolved Hide resolved

resolved review comments

0c1ba1f

NevilParikh14 requested a review from namrata270998 September 2, 2022 06:51

dbshah1212 reviewed Sep 8, 2022

View reviewed changes

resolved review comments

6dc4ee8

NevilParikh14 requested a review from dbshah1212 September 8, 2022 12:12

dbshah1212 reviewed Sep 12, 2022

View reviewed changes

updated param name of expected_streams

73afe0c

NevilParikh14 requested a review from dbshah1212 September 12, 2022 08:17

savan-chovatiya approved these changes Sep 12, 2022

View reviewed changes

dbshah1212 approved these changes Sep 12, 2022

View reviewed changes

NevilParikh14 and others added 2 commits September 15, 2022 16:23

removed unused import

35e3d16

added tap-tester test for parent-child syncing

422ec12

dbshah1212 reviewed Sep 16, 2022

View reviewed changes

tests/test_freshdesk_parent_child_sync.py Show resolved Hide resolved

dbshah1212 requested review from kspeer825 and RushiT0122 September 16, 2022 14:00

updated parent-child sync file

957e4b9

dbshah1212 reviewed Sep 19, 2022

View reviewed changes

NevilParikh14 added 2 commits September 19, 2022 18:38

updated pagination test case

33180fb

updated bookmark with filter param test

2d777e5

sgandhi1311 reviewed Sep 22, 2022

View reviewed changes

resolved review comments

013772e

NevilParikh14 requested a review from sgandhi1311 September 23, 2022 05:00

sgandhi1311 reviewed Sep 23, 2022

View reviewed changes

resolved review comments

a215fb0

NevilParikh14 requested a review from sgandhi1311 September 27, 2022 05:33

sgandhi1311 approved these changes Oct 3, 2022

View reviewed changes


		@staticmethod
		def get_selected_fields_from_metadata(metadata):


		from base import FreshdeskBaseTest

		# As we are not able to generate following fields by Freshdesk UI, so removed it form expectation list.

		"""Test tap sets a separate bookmark for PUBLISHED and SCHEDULED tweets and
		respects it for the next sync of a tweets stream"""


		from base import FreshdeskBaseTest
		from datetime import datetime, timedelta

	# Verify that the records replicated for the selected streams are >= given bookmark.
	# Verify that the records replicated for the selected streams are greater than or equal to the given bookmark.

Tdl 20357 add missing tap tester tests #53

Are you sure you want to change the base?

Tdl 20357 add missing tap tester tests #53

Conversation

NevilParikh14 commented Aug 29, 2022

Description of change

Manual QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

namrata270998 Aug 29, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgandhi1311 Sep 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

namrata270998 Aug 29, 2022 •

edited

Loading

sgandhi1311 Sep 23, 2022 •

edited

Loading