From 74b7c583d6e4165f55b692ceacf69ff30c42296d Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Tue, 17 Jun 2025 13:43:40 +0100 Subject: [PATCH 1/4] Mention that MOT_ENC_SYNC_TOL is now set everywhere --- doc/specific_iocs/motors/Motors-Trouble-Shooting.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/doc/specific_iocs/motors/Motors-Trouble-Shooting.md b/doc/specific_iocs/motors/Motors-Trouble-Shooting.md index a0529c62a..958d98efa 100644 --- a/doc/specific_iocs/motors/Motors-Trouble-Shooting.md +++ b/doc/specific_iocs/motors/Motors-Trouble-Shooting.md @@ -100,7 +100,14 @@ There is a Galil specific PV called `MTRXXXX_AUTOONOFF_CMD` which controls wheth This can mean it has hit a hard limit switch (look at the limit switch status). If it has hit a soft limit it means that the motor steps have hit the hi/low soft limit set in the motor record. This may be the case or it maybe that the motor/encoder steps are out of sync - inside the galil controller limits are applied to motor steps not encoder position. You can see how big this motor/encoder difference is by looking at the `motor drift` on the motor details page, or looking at for e.g. `MTR0101` the PV `MTR0101_MTRENC_DRIFT`. This drift is not a live value, it is updated at end of a move - you can see a live value during a move in `MTR0101_MTRENC_DIFF`. Rehoming the axis will fix it. (If you can't home it, talk with the instrument scientist and make sure they get a homing routine set up. For the moment you can set the raw motor steps on the detailed motor panel by clicking `set` then entering raw motor steps to be what you want, then clicking `use`).
-If you would like the motor to resync back to the encoder automatically then you can set e.g. for `MTR0101` the PV `MTR0101_MOT_ENC_SYNC_TOL_SP` to a non-zero value which is the max EGU they are allowed to differ by, this resync is done at the start of each move. This scheme assumes the encoder is correct, if it is an incremental encoder and there has been a power outage then this will definitely not be the case for example. So this has been described here as "dangerous" as you do not know if it's the encoder or the motor that's out. But if there has been a power outage the motor step counter will probably be wrong too, and both incremental encoders and motors can lose steps (whenever a motor stalls the step count will go up but things do not move, hence the motor step count is now inaccurate in an absolute positioning sense, hence later issue with soft limits). Setting `MTR0101_MOT_ENC_SYNC_TOL_SP` may hide a slowly developing mechanical issue, but if there is a known issue on an axis (it keeps stalling/slipping/losing steps) then enabling it could avoid lost beam time. +If you would like the motor to resync back to the encoder automatically then you can set e.g. for `MTR0101` the PV `MTR0101_MOT_ENC_SYNC_TOL_SP` to a non-zero value which is the max EGU they are allowed to differ by, this resync is done at the start of each move. This scheme assumes the encoder is correct, if it is an incremental encoder and there has been a power outage then this will definitely not be the case for example. So this has been described here as "dangerous" as you do not know if it's the encoder or the motor that's out. But if there has been a power outage the motor step counter will probably be wrong too, and both incremental encoders and motors can lose steps (whenever a motor stalls the step count will go up but things do not move, hence the motor step count is now inaccurate in an absolute positioning sense, hence later issue with soft limits). Setting `MTR0101_MOT_ENC_SYNC_TOL_SP` may hide a slowly developing mechanical issue, but if there is a known issue on an axis (it keeps stalling/slipping/losing steps) then enabling it could avoid lost beam time. + +:::{note} +In June 2025, in [ticket 5220](https://github.com/ISISComputingGroup/IBEX/issues/5220), we set `MOT_ENC_SYNC_TOL` to +be equal to `ERES` (encoder resolution), i.e. a resync of motor & encoder position will be done if they differ by more +than one encoder step. [A test](https://github.com/ISISComputingGroup/InstrumentChecker) ensures that all axes have a +defined, non-zero, resync tolerance. +::: #### Galil won't move after stalling and does not send motor pulses - soft limits To show the soft limits on the controller use `MG _BLx` to show reverse limit and `MG _FLx` to show forward limit in the engineering view OPI. From d48cbca76581f289f3aec698efe7e2693b51bc5a Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Tue, 17 Jun 2025 14:23:27 +0100 Subject: [PATCH 2/4] Expand explanation of motor-encoder resync --- doc/specific_iocs/motors/Galil.md | 39 ++++++++++++++++++- .../motors/Motors-Trouble-Shooting.md | 13 ------- 2 files changed, 38 insertions(+), 14 deletions(-) diff --git a/doc/specific_iocs/motors/Galil.md b/doc/specific_iocs/motors/Galil.md index 4e1c92cf6..ded509d62 100644 --- a/doc/specific_iocs/motors/Galil.md +++ b/doc/specific_iocs/motors/Galil.md @@ -127,7 +127,44 @@ EN ## Syncing encoder and motor steps -It can be useful to sync the motor steps to the encoder steps before each move. This is especially true with an absolute encoder where a power cycle of a Galil controller can change the motor steps to 0 but not the encoder steps because this makes the soft limits stop the motion at strange places. To do this the PV `_MOT_ENC_SYNC_TOL_SP` should be set to a non zero value, when the difference differs by more than this tolerance the motor steps will be resynced. If the encoder is not absolute you should be cautious when doing this, the encoder and motor steps should not get out of sync so don't do it without recording the reason somewhere. +The Galil keeps two counters: a motor step counter, and an encoder readback. The motor step counter corresponds to +how far a given axis should have moved, given the number of motor pulses sent to it. The encoder readback is an +independent position readback from the encoder. + +These two numbers can get out of sync with each other for various reasons, for example: +- The motor may mechanically stall (motor steps will be sent; some of the sent steps will not cause corresponding motion) +- Motion may be disabled by a safety system (motor steps will be sent; no physical motion will occur) +- The encoder may be broken (motor steps will be sent, motion will occur, but will not be registered by the encoder) + +In June 2025, in [ticket 5220](https://github.com/ISISComputingGroup/IBEX/issues/5220), it was decided that the encoder +would be the "source of truth" number used by IBEX (wherever an encoder is present). This means that +IBEX has been set up to resync the motor position and the encoder readback before every move. + +This is configured using PVs of the form `MTR0101_MOT_ENC_SYNC_TOL_SP`. If the drift between the motor and encoder +exceeds the limit defined by this PV, then the motor position will be resynced to the encoder readback just before a +move. The drift is available in PVs of the form `MTR0101_MTRENC_DRIFT`. +For most axes, a setting equal to `ERES` is appropriate - this resyncs motor to encoder if they differ by more than one +encoder pulse (which is the smallest increment accurately measurable). + +A [config checker test](https://github.com/ISISComputingGroup/InstrumentChecker) checks that the resync tolerance has +been set. If you need to disable the resync mechanism for a particular axis, the resync tolerance should be set to an +explicit large value (e.g. much larger than the total range of travel for the axis). + +A side effect of enabling the resync logic on every axis is that if an encoder fails, and is switched to open loop, it +will need to be rehomed or rescanned to have an accurate position. This is because it will have done a resync to the +broken encoder. +Even without the resync logic, the open-loop axis may have lost its absolute position, for example due to +motor record retries. + +On axes where the resync logic has been disabled, by setting `MTR0101_MOT_ENC_SYNC_TOL_SP` to a large value, you may +see errors of the form: + +``` +move begin failure axis X after XXX seconds: ... [Decelerating or stopped by FWD limit switch or soft limit FL]" +``` + +Even when the encoder readback is not close to a limit. This is because the IBEX Galil driver sets limits in the +Galil based on the motor position - this means that if the motor and encoder are out of sync with each other. ## Further Information diff --git a/doc/specific_iocs/motors/Motors-Trouble-Shooting.md b/doc/specific_iocs/motors/Motors-Trouble-Shooting.md index 958d98efa..3720477cc 100644 --- a/doc/specific_iocs/motors/Motors-Trouble-Shooting.md +++ b/doc/specific_iocs/motors/Motors-Trouble-Shooting.md @@ -96,19 +96,6 @@ in the logs. To fix this you must ask electronics to confirm the jumper location There is a Galil specific PV called `MTRXXXX_AUTOONOFF_CMD` which controls whether an axis automatically powers up when given a move (found at the bottom of the motor details OPI). This should normally be `On` so that the motor energises before a move and switches off after a move. In some cases, the motor axis needs to be energised for the whole time, when the axis is under great load or the motor is old. In this case, the auto de-energise should be `No` the energised state should be `energise`. The "motor off deadband" should also be set to a negative number, -1, so that the motor does not turn itself off after a move, this is found in the engineering screen, `Motor off deadband`. If you don't set this to -ve it may not deenergise the motor but this will be unreliable, we think when you don't have an encoder it won't deenergise the motor. There is a warning that reads `Motor off deadband is not small enough, motor may deenergise after move.` which will show in this later case. -### The axis will not move, a message gets put in the log of "move begin failure axis X after XXX seconds: ... [Decelerating or stopped by FWD limit switch or soft limit FL]" - -This can mean it has hit a hard limit switch (look at the limit switch status). If it has hit a soft limit it means that the motor steps have hit the hi/low soft limit set in the motor record. This may be the case or it maybe that the motor/encoder steps are out of sync - inside the galil controller limits are applied to motor steps not encoder position. You can see how big this motor/encoder difference is by looking at the `motor drift` on the motor details page, or looking at for e.g. `MTR0101` the PV `MTR0101_MTRENC_DRIFT`. This drift is not a live value, it is updated at end of a move - you can see a live value during a move in `MTR0101_MTRENC_DIFF`. Rehoming the axis will fix it. (If you can't home it, talk with the instrument scientist and make sure they get a homing routine set up. For the moment you can set the raw motor steps on the detailed motor panel by clicking `set` then entering raw motor steps to be what you want, then clicking `use`).
- -If you would like the motor to resync back to the encoder automatically then you can set e.g. for `MTR0101` the PV `MTR0101_MOT_ENC_SYNC_TOL_SP` to a non-zero value which is the max EGU they are allowed to differ by, this resync is done at the start of each move. This scheme assumes the encoder is correct, if it is an incremental encoder and there has been a power outage then this will definitely not be the case for example. So this has been described here as "dangerous" as you do not know if it's the encoder or the motor that's out. But if there has been a power outage the motor step counter will probably be wrong too, and both incremental encoders and motors can lose steps (whenever a motor stalls the step count will go up but things do not move, hence the motor step count is now inaccurate in an absolute positioning sense, hence later issue with soft limits). Setting `MTR0101_MOT_ENC_SYNC_TOL_SP` may hide a slowly developing mechanical issue, but if there is a known issue on an axis (it keeps stalling/slipping/losing steps) then enabling it could avoid lost beam time. - -:::{note} -In June 2025, in [ticket 5220](https://github.com/ISISComputingGroup/IBEX/issues/5220), we set `MOT_ENC_SYNC_TOL` to -be equal to `ERES` (encoder resolution), i.e. a resync of motor & encoder position will be done if they differ by more -than one encoder step. [A test](https://github.com/ISISComputingGroup/InstrumentChecker) ensures that all axes have a -defined, non-zero, resync tolerance. -::: - #### Galil won't move after stalling and does not send motor pulses - soft limits To show the soft limits on the controller use `MG _BLx` to show reverse limit and `MG _FLx` to show forward limit in the engineering view OPI. It may be that the soft limit is applying a `FL` or `BL` value from the motor record - homing should 0 the motor pulses which will fix this. You can also raise the soft limit from the IBEX table of motors which will raise the `FL/BL` value and enable moves again, provided it does not stall again. From 61cded12004eddfe197b654f5717e5703013bdf0 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Tue, 17 Jun 2025 14:29:02 +0100 Subject: [PATCH 3/4] Spelling --- doc/specific_iocs/motors/Galil.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/specific_iocs/motors/Galil.md b/doc/specific_iocs/motors/Galil.md index ded509d62..182bed0fe 100644 --- a/doc/specific_iocs/motors/Galil.md +++ b/doc/specific_iocs/motors/Galil.md @@ -143,7 +143,7 @@ IBEX has been set up to resync the motor position and the encoder readback befor This is configured using PVs of the form `MTR0101_MOT_ENC_SYNC_TOL_SP`. If the drift between the motor and encoder exceeds the limit defined by this PV, then the motor position will be resynced to the encoder readback just before a move. The drift is available in PVs of the form `MTR0101_MTRENC_DRIFT`. -For most axes, a setting equal to `ERES` is appropriate - this resyncs motor to encoder if they differ by more than one +For most axes, a setting equal to `ERES` is appropriate - this re-syncs motor to encoder if they differ by more than one encoder pulse (which is the smallest increment accurately measurable). A [config checker test](https://github.com/ISISComputingGroup/InstrumentChecker) checks that the resync tolerance has @@ -151,7 +151,7 @@ been set. If you need to disable the resync mechanism for a particular axis, the explicit large value (e.g. much larger than the total range of travel for the axis). A side effect of enabling the resync logic on every axis is that if an encoder fails, and is switched to open loop, it -will need to be rehomed or rescanned to have an accurate position. This is because it will have done a resync to the +will need to be re-homed or re-scanned to have an accurate position. This is because it will have done a resync to the broken encoder. Even without the resync logic, the open-loop axis may have lost its absolute position, for example due to motor record retries. From eb4e87877b798ddcab344ef60dc038822892f098 Mon Sep 17 00:00:00 2001 From: Tom Willemsen Date: Wed, 25 Jun 2025 08:17:52 +0100 Subject: [PATCH 4/4] Add more about limits --- doc/specific_iocs/motors/Galil.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/doc/specific_iocs/motors/Galil.md b/doc/specific_iocs/motors/Galil.md index 182bed0fe..2b1065d95 100644 --- a/doc/specific_iocs/motors/Galil.md +++ b/doc/specific_iocs/motors/Galil.md @@ -125,7 +125,8 @@ EN 5) Power cycle and the Galil should be available on the network. 6) Please note : do not overwrite the permanent program resident in the Galil. It can be overwritten as long as it is not made permanent. Other programs, such as homing routines can be downloaded into the device but should not be burnt in. -## Syncing encoder and motor steps +{#galil_mot_enc_sync} +### Syncing encoder and motor steps The Galil keeps two counters: a motor step counter, and an encoder readback. The motor step counter corresponds to how far a given axis should have moved, given the number of motor pulses sent to it. The encoder readback is an @@ -156,15 +157,20 @@ broken encoder. Even without the resync logic, the open-loop axis may have lost its absolute position, for example due to motor record retries. -On axes where the resync logic has been disabled, by setting `MTR0101_MOT_ENC_SYNC_TOL_SP` to a large value, you may -see errors of the form: +### Limits +If the motor is on limits and a move is attempted, you will see a message in the log like: ``` move begin failure axis X after XXX seconds: ... [Decelerating or stopped by FWD limit switch or soft limit FL]" ``` -Even when the encoder readback is not close to a limit. This is because the IBEX Galil driver sets limits in the -Galil based on the motor position - this means that if the motor and encoder are out of sync with each other. +This means that the axis is on a limit, which can be either a hard limit or a soft limit. If it is a hard limit, the +hard limit indicators in IBEX (`.LLS` and `.HLS` PVs will be active) - this means that a mechanical limit switch is +engaged, or that an external system has disabled motion (safety systems commonly cause both limits to show engaged). + +If it is a soft limit, these are set in IBEX. For axes where the [motor and encoder resync](#galil_mot_enc_sync), +the internal limits in the galil should match closely with the limits set in IBEX. However, if the motor-encoder resync +tolerance is set very high, it is possible for the internal galil limits to differ from those configured in IBEX. ## Further Information