Skip to content

Commit 9fcc646

Browse files
priyabhatnagar25priyabhatnagar25
committed
CloudSql My Sql e2e tests
1 parent 0bd19ae commit 9fcc646

17 files changed

Lines changed: 646 additions & 461 deletions

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlDesignTime.feature‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
# Copyright © 2023 Cask Data, Inc.
2+
##
3+
## Licensed under the Apache License, Version 2.0 (the "License"); you may not
4+
## use this file except in compliance with the License. You may obtain a copy of
5+
## the License at
6+
##
7+
## http://www.apache.org/licenses/LICENSE-2.0
8+
##
9+
## Unless required by applicable law or agreed to in writing, software
10+
## distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
## WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
## License for the specific language governing permissions and limitations under
13+
# the License..
14+
115
@CloudMySql
216
Feature: CloudMySql sink- Verify CloudMySql sink plugin design time scenarios
317

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTime.feature‎

Lines changed: 54 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,47 +1,64 @@
1-
Feature: CloudMySql - Verify data transfer from BigQuery source to CloudMySql sink
1+
# Copyright © 2023 Cask Data, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
4+
# use this file except in compliance with the License. You may obtain a copy of
5+
# the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
# License for the specific language governing permissions and limitations under
13+
# the License.
214

15+
@CloudMySql
16+
Feature: CloudMySql Sink - Run time scenarios
17+
18+
@BQ_SOURCE_TEST @CLOUDMYSQL_TEST_TABLE
319
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully
420
Given Open Datafusion Project to configure pipeline
521
When Expand Plugin group in the LHS plugins list: "Source"
622
When Select plugin: "BigQuery" from the plugins list as: "Source"
23+
And Navigate to the properties page of plugin: "BigQuery"
24+
And Enter input plugin property: "referenceName" with value: "Reference"
25+
And Replace input plugin property: "project" with value: "projectId"
26+
And Enter input plugin property: "datasetProject" with value: "datasetprojectId"
27+
And Enter input plugin property: "dataset" with value: "dataset"
28+
And Enter input plugin property: "table" with value: "bqSourceTable"
29+
Then Click on the Get Schema button
30+
Then Verify the Output Schema matches the Expected Schema: "bqOutputMultipleDatatypesSchema"
31+
Then Validate "BigQuery" plugin properties
32+
And Close the Plugin Properties page
733
When Expand Plugin group in the LHS plugins list: "Sink"
834
When Select plugin: "CloudSQL MySQL" from the plugins list as: "Sink"
935
Then Connect plugins: "BigQuery" and "CloudSQL MySQL" to establish connection
10-
Then Navigate to the properties page of plugin: "BigQuery"
11-
Then Replace input plugin property: "project" with value: "projectId"
12-
Then Enter input plugin property: "datasetProject" with value: "projectId"
13-
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
14-
Then Enter input plugin property: "dataset" with value: "dataset"
15-
Then Enter input plugin property: "table" with value: "bqTargetTable"
16-
# Then Click on the Get Schema button
17-
# Then Validate "BigQuery" plugin properties
18-
Then Close the Plugin Properties page
1936
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
2037
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
2138
Then Select radio button plugin property: "instanceType" with value: "public"
2239
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
2340
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
2441
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
2542
Then Enter input plugin property: "referenceName" with value: "RefName"
26-
Then Enter input plugin property: "database" with value: "TestDatabase"
27-
Then Enter input plugin property: "tableName" with value: "mytable"
28-
# Then Validate "CloudSQL MySQL" plugin properties
43+
Then Enter input plugin property: "database" with value: "DatabaseName"
44+
Then Replace input plugin property: "tableName" with value: "targetTable"
45+
Then Validate "CloudSQL MySQL" plugin properties
2946
Then Close the Plugin Properties page
3047
Then Save the pipeline
3148
Then Preview and run the pipeline
3249
Then Wait till pipeline preview is in running state
3350
Then Open and capture pipeline preview logs
34-
# Then Verify the preview run status of pipeline in the logs is "succeeded"
35-
# Then Close the pipeline logs
36-
# Then Close the preview
37-
# Then Deploy the pipeline
38-
# Then Run the Pipeline in Runtime
39-
# Then Wait till pipeline is in running state
40-
# Then Open and capture logs
41-
# Then Verify the pipeline status is "Succeeded"
42-
# Then Close the pipeline logs
43-
# Then Validate OUT record count is equal to records transferred to target BigQuery table
51+
Then Verify the preview run status of pipeline in the logs is "succeeded"
52+
Then Close the pipeline logs
53+
Then Close the preview
54+
Then Deploy the pipeline
55+
Then Run the Pipeline in Runtime
56+
Then Wait till pipeline is in running state
57+
Then Open and capture logs
58+
Then Verify the pipeline status is "Succeeded"
59+
Then Close the pipeline logs
4460

61+
@BQ_SOURCE_TEST @CLOUDMYSQL_SOURCE_TEST
4562
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink successfully when connection arguments are set
4663
Given Open Datafusion Project to configure pipeline
4764
When Expand Plugin group in the LHS plugins list: "Source"
@@ -55,8 +72,8 @@ Feature: CloudMySql - Verify data transfer from BigQuery source to CloudMySql si
5572
Then Enter input plugin property: "referenceName" with value: "BQReferenceName"
5673
Then Enter input plugin property: "dataset" with value: "dataset"
5774
Then Enter input plugin property: "table" with value: "bqTargetTable"
58-
# Then Click on the Get Schema button
59-
# Then Validate "BigQuery" plugin properties
75+
Then Click on the Get Schema button
76+
Then Validate "BigQuery" plugin properties
6077
Then Close the Plugin Properties page
6178
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
6279
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
@@ -66,24 +83,23 @@ Feature: CloudMySql - Verify data transfer from BigQuery source to CloudMySql si
6683
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
6784
Then Enter input plugin property: "referenceName" with value: "RefName"
6885
Then Enter input plugin property: "database" with value: "TestDatabase"
69-
Then Enter input plugin property: "tableName" with value: "mytable"
86+
Then Replace input plugin property: "tableName" with value: "targetTable"
7087
Then Enter key value pairs for plugin property: "connectionArguments" with values from json: "connectionArgumentsList"
71-
# Then Validate "CloudSQL MySQL" plugin properties
88+
Then Validate "CloudSQL MySQL" plugin properties
7289
Then Close the Plugin Properties page
7390
Then Save the pipeline
7491
Then Preview and run the pipeline
7592
Then Wait till pipeline preview is in running state
76-
# Then Open and capture pipeline preview logs
77-
# Then Verify the preview run status of pipeline in the logs is "succeeded"
78-
# Then Close the pipeline logs
79-
# Then Close the preview
80-
# Then Deploy the pipeline
81-
# Then Run the Pipeline in Runtime
82-
# Then Wait till pipeline is in running state
83-
# Then Open and capture logs
84-
# Then Verify the pipeline status is "Succeeded"
85-
# Then Close the pipeline logs
86-
# Then Validate OUT record count is equal to records transferred to target BigQuery table
93+
Then Open and capture pipeline preview logs
94+
Then Verify the preview run status of pipeline in the logs is "succeeded"
95+
Then Close the pipeline logs
96+
Then Close the preview
97+
Then Deploy the pipeline
98+
Then Run the Pipeline in Runtime
99+
Then Wait till pipeline is in running state
100+
Then Open and capture logs
101+
Then Verify the pipeline status is "Succeeded"
102+
Then Close the pipeline logs
87103

88104

89105

‎cloudsql-mysql-plugin/src/e2e-test/features/sink/CloudMySqlRunTimeMacro.feature‎

Lines changed: 40 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,21 @@
1-
Feature: CloudMySql - Verify data transfer to CloudMySql sink with macro arguments
1+
# Copyright © 2023 Cask Data, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
4+
# use this file except in compliance with the License. You may obtain a copy of
5+
# the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
# License for the specific language governing permissions and limitations under
13+
# the License.
214

3-
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink using macro arguments
15+
16+
Feature: CloudMySql Sink - Run time scenarios (macro)
17+
18+
Scenario: To verify data is getting transferred from BigQuery source to CloudMySql sink using macro arguments in connection section
419
Given Open Datafusion Project to configure pipeline
520
When Expand Plugin group in the LHS plugins list: "Source"
621
When Select plugin: "BigQuery" from the plugins list as: "Source"
@@ -13,8 +28,8 @@ Feature: CloudMySql - Verify data transfer to CloudMySql sink with macro argumen
1328
Then Click on the Macro button of Property: "datasetProject" and set the value to: "bqDatasetProjectId"
1429
Then Click on the Macro button of Property: "dataset" and set the value to: "bqDataset"
1530
Then Click on the Macro button of Property: "table" and set the value to: "bqTargetTable"
16-
# Then Click on the Get Schema button
17-
# Then Validate "BigQuery" plugin properties
31+
Then Click on the Get Schema button
32+
Then Validate "BigQuery" plugin properties
1833
Then Close the Plugin Properties page
1934
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
2035
Then Click on the Macro button of Property: "jdbcPluginName" and set the value to: "cloudsql-mysql"
@@ -25,7 +40,7 @@ Feature: CloudMySql - Verify data transfer to CloudMySql sink with macro argumen
2540
Then Enter input plugin property: "referenceName" with value: "RefName"
2641
Then Enter input plugin property: "database" with value: "TestDatabase"
2742
Then Click on the Macro button of Property: "tableName" and set the value to: "mytable"
28-
# Then Validate "CloudSQL MySQL" plugin properties
43+
Then Validate "CloudSQL MySQL" plugin properties
2944
Then Close the Plugin Properties page
3045
Then Save the pipeline
3146
Then Preview and run the pipeline
@@ -39,24 +54,24 @@ Feature: CloudMySql - Verify data transfer to CloudMySql sink with macro argumen
3954
Then Enter runtime argument value "table" for key "mytable"
4055
Then Run the preview of pipeline with runtime arguments
4156
Then Wait till pipeline preview is in running state
42-
# Then Open and capture pipeline preview logs
43-
# Then Verify the preview run status of pipeline in the logs is "succeeded"
44-
# Then Close the pipeline logs
45-
# Then Close the preview
46-
# Then Deploy the pipeline
47-
# Then Run the Pipeline in Runtime
48-
# Then Enter runtime argument value "projectId" for key "projectId"
49-
# Then Enter runtime argument value "bqDatasetId" for key "bqDatasetProjectId"
50-
# Then Enter runtime argument value "dataset" for key "bqDataset"
51-
# Then Enter runtime argument value "bqSourceTable" for key "bqTargetTable"
52-
# Then Enter runtime argument value "driver" for key "cloudsql-mysql"
53-
# Then Enter runtime argument value from environment variable "name" for key "username"
54-
# Then Enter runtime argument value from environment variable "pass" for key "password"
55-
# Then Click on the Macro button of Property: "dataset" and set the value to: "bqDataset"
56-
# Then Enter runtime argument value "table" for key "mytable"
57-
# Then Run the Pipeline in Runtime with runtime arguments
58-
# Then Wait till pipeline is in running state
59-
# Then Open and capture logs
60-
# Then Verify the pipeline status is "Succeeded"
61-
# Then Close the pipeline logs
57+
Then Open and capture pipeline preview logs
58+
Then Verify the preview run status of pipeline in the logs is "succeeded"
59+
Then Close the pipeline logs
60+
Then Close the preview
61+
Then Deploy the pipeline
62+
Then Run the Pipeline in Runtime
63+
Then Enter runtime argument value "projectId" for key "projectId"
64+
Then Enter runtime argument value "bqDatasetId" for key "bqDatasetProjectId"
65+
Then Enter runtime argument value "dataset" for key "bqDataset"
66+
Then Enter runtime argument value "bqSourceTable" for key "bqTargetTable"
67+
Then Enter runtime argument value "driver" for key "cloudsql-mysql"
68+
Then Enter runtime argument value from environment variable "name" for key "username"
69+
Then Enter runtime argument value from environment variable "pass" for key "password"
70+
Then Click on the Macro button of Property: "dataset" and set the value to: "bqDataset"
71+
Then Enter runtime argument value "table" for key "mytable"
72+
Then Run the Pipeline in Runtime with runtime arguments
73+
Then Wait till pipeline is in running state
74+
Then Open and capture logs
75+
Then Verify the pipeline status is "Succeeded"
76+
Then Close the pipeline logs
6277

‎cloudsql-mysql-plugin/src/e2e-test/features/source/CloudMySqlDesignTime.feature‎

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
# Copyright © 2023 Cask Data, Inc.
2+
##
3+
## Licensed under the Apache License, Version 2.0 (the "License"); you may not
4+
## use this file except in compliance with the License. You may obtain a copy of
5+
## the License at
6+
##
7+
## http://www.apache.org/licenses/LICENSE-2.0
8+
##
9+
## Unless required by applicable law or agreed to in writing, software
10+
## distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
11+
## WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
12+
## License for the specific language governing permissions and limitations under
13+
# the License..
14+
115
@CloudMySql
216
Feature: CloudMySql source- Verify CloudMySql source plugin design time scenarios
317

@@ -7,12 +21,12 @@ Feature: CloudMySql source- Verify CloudMySql source plugin design time scenario
721
When Select plugin: "CloudSQL MySQL" from the plugins list as: "Source"
822
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
923
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
10-
Then Click on the Macro button of Property: "user" and set the value to: "username"
11-
Then Click on the Macro button of Property: "password" and set the value to: "password"
24+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
25+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
1226
Then Select radio button plugin property: "instanceType" with value: "public"
1327
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
1428
Then Enter input plugin property: "referenceName" with value: "RefName"
15-
Then Enter input plugin property: "database" with value: "TestDatabase"
29+
Then Enter input plugin property: "database" with value: "DatabaseName"
1630
Then Enter textarea plugin property: "importQuery" with value: "insertQuery"
1731
Then Click on the Get Schema button
1832
Then Verify the Output Schema matches the Expected Schema: "outputSchema"
@@ -26,13 +40,12 @@ Feature: CloudMySql source- Verify CloudMySql source plugin design time scenario
2640
Then Navigate to the properties page of plugin: "CloudSQL MySQL"
2741
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
2842
Then Select radio button plugin property: "instanceType" with value: "public"
29-
Then Enter input plugin property: "connectionName" with value: "connection"
43+
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
3044
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
3145
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
32-
Then Enter key value pairs for plugin property: "connectionArguments" with values from json: "connectionArgumentsList"
33-
Then Enter input plugin property: "referenceName" with value: "refName"
34-
Then Enter input plugin property: "database" with value: "TestDatabase"
35-
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
46+
Then Enter input plugin property: "referenceName" with value: "RefName"
47+
Then Enter input plugin property: "database" with value: "DatabaseName"
48+
Then Enter textarea plugin property: "importQuery" with value: "insertQuery"
3649
Then Click on the Get Schema button
3750
Then Verify the Output Schema matches the Expected Schema: "outputSchema"
3851
Then Validate "CloudSQL MySQL" plugin properties
@@ -46,10 +59,12 @@ Feature: CloudMySql source- Verify CloudMySql source plugin design time scenario
4659
Then Select dropdown plugin property: "select-jdbcPluginName" with option value: "cloudsql-mysql"
4760
Then Select radio button plugin property: "instanceType" with value: "public"
4861
Then Enter input plugin property: "connectionName" with value: "ConnectionName"
62+
Then Replace input plugin property: "user" with value: "username" for Credentials and Authorization related fields
63+
Then Replace input plugin property: "password" with value: "password" for Credentials and Authorization related fields
4964
Then Enter key value pairs for plugin property: "connectionArguments" with values from json: "connectionArgumentsList"
5065
Then Enter input plugin property: "referenceName" with value: "referencename"
51-
Then Enter input plugin property: "database" with value: "TestDatabase"
52-
Then Enter textarea plugin property: "importQuery" with value: "selectQuery"
66+
Then Enter input plugin property: "database" with value: "DatabaseName"
67+
Then Enter textarea plugin property: "importQuery" with value: "insertQuery"
5368
Then Click on the Get Schema button
5469
Then Verify the Output Schema matches the Expected Schema: "outputSchema"
5570
Then Validate "CloudSQL MySQL" plugin properties

0 commit comments

Comments
 (0)