[Release 3.0] Planned Breaking Changes for 3.0 in Plugin

[peterzhuamazon]

[Release 3.0] Planned Breaking Changes for 3.0 in Plugin

As mentioned in this META Issue, we want to track and increase transparency around the breaking changes planned for the 3.0 release across participating plugins.

Please kindly document all planned breaking changes related to your plugin for 3.0 here, by sharing detailed information about the changes, expected impact, and any guidance for users or other teams to adapt to the changes.

If needed, please also prepare PRs on documentation-website changes as part of the 3.0 release process.

Overall, this would help us deliver a smooth and seamless upgrade experience.

Thanks.

Breaking behavior in 3.0+

• Deprecated OpenSearch DSL format. #3280
• Remove DELETE statement support in SQL and remove plugins-sql-delete-enabled #3281
• Remove deprecated opendistro.xxx settings. https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#breaking-change #3282
• Deprecated opendistro endpoints. #3283
• Remove plugins.sql.pagination.api, default to PIT, deprecated scroll API. #3284
• Remove PromQL supports for prometheus connector. #3285
• Remove SparkSQL supports for spark connector. #3286

[penghuo]

IMO, OpenSearch DSL format support could be consided deperated in 3.0. The reasons are

• The response is not structured and does not align with other supported formats, such as JDBC or CSV.
• Query support is limited; only queries that can be fully translated to OpenSearch DSL are supported.

[dai-chen]

DELETE statement: https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#plugins-sql-delete-enabled

[penghuo]

SQL Plugin Breaking Change Tracker

As part of the OpenSearch 3.0 release plan, we intend to introduce breaking changes. To align with the release timeline, we have defined the following review and decision process:

1. Breaking Change Proposal Collection

We are collecting breaking change proposals. Please add your comments to this thread.
The collection phase will conclude on January 24, 2025.

2. Review and Finalization

A meeting will be scheduled during the last week of January 2025 to review the proposed changes with the maintainers.
The final list of breaking changes will be finalized by January 31, 2025.

Criteria for Breaking Change Candidates

Features that meet any of the following criteria are considered strong candidates for breaking changes:

• Low Usage or Value: Features that are rarely used or provide minimal benefit to users.
• Technical Debt: Features that increase system complexity or hinder maintainability.
• Misalignment with Future Goals: Features that conflict with the product’s long-term vision or updated design principles.

[penghuo]

Settings

• opendistro.xxx setttings could be deprecated. https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#breaking-change

PPL

• The AD and KMeans commands could be deprecated. The ML command includes the functionality of both AD and KMeans, making the existing AD and KMeans commands redundant.

Connector

The Prometheus connector and Spark connector are considered experimental features, as they have limited adoption and do not align with our long-term goals. We could consider deprecating them.

[vamsimanohar]

In 3.0, we should deprecate the setting to move to PIT from scroll. We can make PIT the default one.
https://github.com/opensearch-project/sql/blob/main/common/src/main/java/org/opensearch/sql/common/setting/Settings.java#L26

[penghuo]

Breaking behavior in 3.0+

• Deprecated OpenSearch DSL format.
• Remove DELETE statement support in SQL and remove plugins-sql-delete-enabled
• Remove deprecated opendistro.xxx settings. https://github.com/opensearch-project/sql/blob/main/docs/user/admin/settings.rst#breaking-change
• Deprecated opendistro endpoints.
• Remove plugins.sql.pagination.api, default to PIT, deprecated scroll API.
• Remove PromQL supports for prometheus connector.
• Remove SparkSQL supports for spark connector.

Reference

• [META] Breaking changes in 3.0 OpenSearch#3351

[ps48]

@penghuo While we were planning for 3.0 breaking changes and support on Dashboards plugins. We saw the SQL team is removing PROMQL support in PPL. While investigating the implications of these changes we saw that metrics page is using this pretty heavily for visualizations support from prometheus. While we don’t have a roadmap for prometheus support on metrics page yet, we also don’t have plan to deprecate this feature either. Here are some examples of prometheus queries:

1. Get Prometheus datasource

show datasources | where CONNECTOR_TYPE="PROMETHEUS" | fields DATASOURCE_NAME

2. Get metrics

source = my_prometheus.information_schema.tables

3. Get labels

describe my_prometheus.go_gc_duration_seconds | fields COLUMN_NAME

4. Fetch metric data

source = my_prometheus.query_range('avg (go_gc_duration_seconds)', 1738797632, 1738884032, '1h')

Questions:

1. Is our understanding correct that the removal of PromQL supports for prometheus connector will break metrics page based on the above queries?
2. If this break’s the metrics page, Is there a work around for these queries to be queried with PPL?
3. Is it possible to not deprecate this feature until the roadmap is clear for the metrics page?

[y3n11]

Hi OpenSearch team,

I would like to better understand the implications of the removal of the SparkSQL connector. We are currently using OpenSearch along with PySpark in a production application, querying data with the library "opensearch-spark-30_2.12-1.3.0.jar" and spark.read.

Can we still use this approach with OpenSearch 3.0.0, or will we need to make changes?

Thank you in advance! @peterzhuamazon

[peterzhuamazon]

Hi OpenSearch team,

I would like to better understand the implications of the removal of the SparkSQL connector. We are currently using OpenSearch along with PySpark in a production application, querying data with the library "opensearch-spark-30_2.12-1.3.0.jar" and spark.read.

Can we still use this approach with OpenSearch 3.0.0, or will we need to make changes?

Thank you in advance! @peterzhuamazon

That would be a question for @penghuo @noCharger @ps48 .

Thanks.