Optic lint reports errors for valid examples

[klaude]

Describe the bug
Howdy, Optic folks. 👋

optic lint reports invalid examples in our services. The services are OAS compliant, and we've run them through spectral to be sure. We've noticed this in versions 0.47 and 0.48 of the CLI utility. The require_request_examples, require_response_examples, and require_parameter_examples rules report these false errors

We've triple checked our specs and tried different combinations of defining or not defining examples, but haven't found a fix. We've also tried linting bundled and de-referenced versions of our specs (using optic bundle and optic dereference) but run into the same issue. Bundled and de-referenced specifications look correct too.

We noticed similar issues around OAS examples with spectral. They ended up ignoring superfluous errors from ajv to fix those issues (stoplightio/spectral#2408). I wonder if something similar is happening here too.

Thanks for your time and for looking! Please let me know if there's any other information I can provide.

To Reproduce
Steps to reproduce the behavior:

The following OpenAPI spec is a slimmed down version of one of our service APIs. It uses schemas defined in-line in the operation's response, defined in components/schemas, and defined by remote URL. Please let me know if you'd like our organization or API IDs to help troubleshoot.

openapi: 3.0.3
x-optic-url: "https://app.useoptic.com/organizations/<snip>/apis/<snip>"
x-optic-standard: "@<snip>/<snip>"
info:
  title: Foo service
  version: '1.0'
  description: A friendly foo service
  contact:
    name: The Foo Admins
    email: [email protected]
    url: 'https://example.org/foo'
    x-slack: '#foo'
servers:
  - url: https://example.org
tags:
  - name: foo
paths:
  '/foo':
    get:
      summary: Get a foo
      operationId: get-a-foo
      description: Get the foo!
      tags: [foo]
      x-tier: 3
      x-category: system
      responses:
        '200':
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  category:
                    $ref: '#/components/schemas/Category'
                  otherCategories:
                    type: array
                    items:
                      $ref: '#/components/schemas/Category'
                  links:
                    type: array
                    items: 
                      $ref: 'https://stoplight.io/api/v1/projects/backmarket/api-models/nodes/models/Link.yaml'
                  content:
                    type: object
                    required:
                      - suggestions
                      - metadata
                    properties:
                      suggestions:
                        type: array
                        items:
                          $ref: '#/components/schemas/Suggestion'
                      metadata:
                        $ref: '#/components/schemas/Metadata'
                required:
                  - category
                  - content
              example:
                category:
                  name: hello Cleveland!
                otherCategories:
                  - name: hello Ohio!
                links:
                  - type: external
                    href: 'https://example.org/'
                content:
                  suggestions:
                    - name: A suggestion
                  metadata:
                    metaTitle: A title
                    metaDescription: A description
components:
  schemas:
    Category:
      description: A foo category
      type: object
      example:
        name: hello Cleveland!
      properties:
        name:
          type: string
          example: hello Cleveland!
      required:
        - name
    Suggestion:
      description: A nice suggestion
      type: object
      properties:
        name:
          type: string
          example: Well hello
      example:
        name: Well hello
    Metadata:
      description: Metadata configuration
      title: HeadMetadata
      type: object
      example:
        metaTitle: Some title
        metaDescription: Some description
      properties:
        metaTitle:
          type: string
          example: lorem ipsum
        metaDescription:
          type: string
          example: lorem ipsum
      required:
        - metaTitle
        - metaDescription

Our Optic standard's contents are:

ruleset:
  - breaking-changes
  - naming:
      properties: camelCase
      required_on: added
      pathComponents: param-case
      requestHeaders: Capital-Param-Case
      queryParameters: camelCase
      responseHeaders: Capital-Param-Case
      cookieParameters: Capital-Param-Case
  - examples:
      required_on: addedOrChanged
      require_request_examples: true
      require_response_examples: true
      require_parameter_examples: true
  - spectral:
      added:
        - https://unpkg.com/@backmarket-api/[email protected]

1. Run optic lint against the following file
2. Note errors in the console related to invalid examples and the rules that generated those errors

Expected behavior

I expect optic lint to pass

Screenshots

I don't have screenshots, but here's console output:

My system:

$ sw_vers
ProductName:		macOS
ProductVersion:		13.5
BuildVersion:		22G74
$ uname -a
Darwin tome.local 22.6.0 Darwin Kernel Version 22.6.0: Wed Jul  5 22:22:52 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T8103 arm64
$ node --version
v20.3.0
$ optic --version
0.48.1-0
$ spectral --version
6.8.0

Linting with spectral passes:

$ spectral lint \
  --ruleset https://unpkg.com/@backmarket-api/[email protected] \
  --fail-severity info \
  ./test.yaml
No results with a severity of 'info' or higher found!

But linting with optic produces errors:

$ optic lint --severity=info ./test.yaml
Linting spec ./test.yaml...
✔ OpenAPI is valid

Running Optic Checks

   FAIL  GET /foo
    requirement rule [error]: require property examples match schemas
      x 'category' example does not match the schema.
  - example  must NOT have additional property 'name'
      at paths > /foo > get > responses > 200 > content > application/json > schema > properties > category (/<snip>/test.yaml:75:2294)

    requirement rule [error]: require property examples match schemas
      x 'metadata' example does not match the schema.
  - example  must NOT have additional property 'metaTitle'
      at paths > /foo > get > responses > 200 > content > application/json > schema > properties > content > properties > metadata (/<snip>/test.yaml:95:2726)

    requirement rule [error]: response body examples must match schemas
      x the example does not match the schema.
  - example /links/0 must NOT have additional property 'type'
      at paths > /foo > get > responses > 200 (/<snip>/test.yaml:27:650)




18 passed
3 errors

x Check failures detected with your OpenAPI spec.

Details (please complete the following information):

• OS and version: macOS 13.5, running on an M1
• Optic version: 0.48.1-0
• NodeJS version: 20.3.0

Additional context

None, but thanks again for looking. ❤️

[niclim]

Hey @klaude - thanks for the detailed bug report - I've got a PR open for a bug fix for the - example must NOT have additional property 'name' errors #2163

As for the - example /links/0 must NOT have additional property 'type' I did some digging and I think it's because of the Link key here https://stoplight.io/api/v1/projects/backmarket/api-models/nodes/models/Link.yaml on L6 it's specifying that it's type: object and a oneOf which I think is invalid (since if you have a top level type: object and a oneOf that includes a type: string root level there isn't a way to reconcile that / it's invalid).

title: Link
description: |
  backmarket common Link model

  For more info check this [link](https://github.com/BackMarket/RFC/blob/master/api/models/API-Model-Link.md)(internal)
type: object # <- here's the line
oneOf:
  - $ref: ./LinkInternal.yaml
  - $ref: ./LinkExternal.yaml
x-internal: false

My suggestion for the link/0 error is to update your schema to remove the type: object key on Link. In this case, Optic is treating the Link schema as type: object with no properties. Removing the type: object line in the schema fixes this issue. We could probably be more strict with detecting this (similar to this issue #1804) and fail earlier, since this is pretty hard to trace

I'll report back here once the above PR is merged in and I've published a new version

[klaude]

Thanks for the fast response, @niclim. You make a great point about our link schema. I'll try that out and let you know how it goes. The link schema has been a thorn in our side for a while, and I bet this would fix a lot of stuff.

[niclim]

Hi @klaude I just released 0.48.2 which should fix the other issues with the example rulesets - if you run into anything else please feel free to reopen this issue or to open a different issue - thanks!

[klaude]

@niclim I can confirm that 0.48.2 indeed fixes the issue. Thanks for the help and fast turn around!