Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chassis][nokia][202405.17]: Chassisd crashes when updating moduledb #21131

Closed
liamkearney-msft opened this issue Dec 11, 2024 · 2 comments · Fixed by sonic-net/sonic-platform-daemons#573
Closed

[chassis][nokia][202405.17]: Chassisd crashes when updating moduledb #21131

liamkearney-msft opened this issue Dec 11, 2024 · 2 comments · Fixed by sonic-net/sonic-platform-daemons#573

Comments

@liamkearney-msft
Copy link
Contributor

Description

Chassisd fails on supervisor on Nokia 7250, leading to syncd/swss not starting up. Backtrace seen on the syslog is as follows:

2024 Dec 10 01:57:04.137945 svcstr-7250-sup-1 INFO pmon#supervisord 2024-12-10 01:57:04,137 INFO success: chassisd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

2024 Dec 10 01:57:09.544727 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd Traceback (most recent call last):

2024 Dec 10 01:57:09.544727 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd   File "/usr/local/bin/chassisd", line 710, in <module>

2024 Dec 10 01:57:09.545539 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd     main()

2024 Dec 10 01:57:09.545753 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd   File "/usr/local/bin/chassisd", line 705, in main

2024 Dec 10 01:57:09.545938 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd     chassisd.run()

2024 Dec 10 01:57:09.545949 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd   File "/usr/local/bin/chassisd", line 683, in run

2024 Dec 10 01:57:09.546348 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd     self.module_updater.module_db_update()

2024 Dec 10 01:57:09.546558 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd   File "/usr/local/bin/chassisd", line 279, in module_db_update

2024 Dec 10 01:57:09.546731 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd     if self.my_slot == int(module_info_dict['slot']):

2024 Dec 10 01:57:09.546891 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024 Dec 10 01:57:09.546902 svcstr-7250-sup-1 INFO pmon#supervisord: chassisd ValueError: invalid literal for int() with base 10: 'A'

This is due to this PR : sonic-net/sonic-platform-daemons#560 , and the fact that nokia uses characters (eg. A, B, C etc.) for its slot name.

Steps to reproduce the issue:

  1. Start a nokia 7250 on 202405.17
  2. Note that syncd / swss does not start

Describe the results you received:

No syncd containers running

Describe the results you expected:

syncd starts as expected

Output of show version:

20240510.17

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):
@liamkearney-msft liamkearney-msft mentioned this issue Dec 11, 2024
Merged
@arlakshm
Copy link
Contributor

@mlok-nokia for viz...

@Javier-Tan
Copy link

Javier-Tan commented Dec 11, 2024 •
edited
Loading

Just for documentation purposes I believe this causes iBGP neighbours not to come up so tests generally fail (when .17 is on sup card), whole host of other issues as well

@mlok-nokia mlok-nokia mentioned this issue Dec 12, 2024
Merged
10 tasks
mssonicbld pushed a commit to mssonicbld/sonic-platform-daemons that referenced this issue Dec 20, 2024
@mlok-nokia @mssonicbld
[chassisd] Address the chassisd crash issue and add UT for it (sonic-…
7a9daac 
…net#573)

Description
On Nokia platform, slot name of Supervisor is string "A" instead of a number. Using "int" to convert it could cause issue backtrace. We should use slot value to any checking without any conversion. This will fixes sonic-net/sonic-buildimage#21131

Motivation and Context
Modify the _get_module_info not to convert "slot" to a string value. And also modify the code not to convert slot value to an to do any checking. Just directly use the returned value of get_slot(). Also add UT test_moduleupdater_check_slot_string() to valid it.

How Has This Been Tested?
Tested on 202405 branch


Signed-off-by: mlok <[email protected]>
@mssonicbld mssonicbld mentioned this issue Dec 20, 2024
Merged
mssonicbld pushed a commit to sonic-net/sonic-platform-daemons that referenced this issue Dec 20, 2024
@mlok-nokia @mssonicbld
[chassisd] Address the chassisd crash issue and add UT for it (#573)
ef80d32 
Description
On Nokia platform, slot name of Supervisor is string "A" instead of a number. Using "int" to convert it could cause issue backtrace. We should use slot value to any checking without any conversion. This will fixes sonic-net/sonic-buildimage#21131

Motivation and Context
Modify the _get_module_info not to convert "slot" to a string value. And also modify the code not to convert slot value to an to do any checking. Just directly use the returned value of get_slot(). Also add UT test_moduleupdater_check_slot_string() to valid it.

How Has This Been Tested?
Tested on 202405 branch


Signed-off-by: mlok <[email protected]>
vvolam pushed a commit to vvolam/sonic-platform-daemons that referenced this issue Jan 3, 2025
@mlok-nokia @vvolam
[chassisd] Address the chassisd crash issue and add UT for it (sonic-…
564ecd0 
…net#573)

Description
On Nokia platform, slot name of Supervisor is string "A" instead of a number. Using "int" to convert it could cause issue backtrace. We should use slot value to any checking without any conversion. This will fixes sonic-net/sonic-buildimage#21131

Motivation and Context
Modify the _get_module_info not to convert "slot" to a string value. And also modify the code not to convert slot value to an to do any checking. Just directly use the returned value of get_slot(). Also add UT test_moduleupdater_check_slot_string() to valid it.

How Has This Been Tested?
Tested on 202405 branch


Signed-off-by: mlok <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[chassisd] Address the chassisd crash issue and add UT for it mlok-nokia/sonic-platform-daemons
3 participants
@Javier-Tan @arlakshm @liamkearney-msft