Troubleshooting
Fixing SCADA coil oscillation through a Modbus gateway
Short answer
The oscillation is a translation fault, not a plant fault: when an upstream read fails, the gateway most likely answers the master with null data instead of a Modbus exception, so coils at 1 appear to flip every poll. Order the gateway timeouts strictly descending, pace the master's polling, and route to the SLC 5/05 over CSPv4 — the documented fault interval went from under two weeks to two months.
On this page
- Why do SCADA coils oscillate when the SLC 5/05 data is stable?
- What timeout order does the gateway need?
- How do the timeouts behave in one failed poll?
- Why move the SLC 5/05 route to AB/Ethernet on port 2222?
- How fast should the Modbus master poll?
- How do you capture the fault before rebooting the gateway?
- What result should you expect from these changes?
- Questions asked
The fault is in the translation path, not in the plant. The SLC 5/05 data table is steady while the coils at the SCADA master flip once per transmission, and rebooting the gateway restores correct values — that combination points at the translator handing back null data instead of a Modbus exception when an upstream read fails. Two changes hold the fault off in practice: order the gateway timeouts strictly descending — master 2 s > messagetimeout 600 ms > slavetimeout 500 ms > chartimeout 100 ms — and move the upstream route off EtherNet/IP onto the SLC 5/05's native AB/Ethernet. In the documented installation the fault-free interval went from under two weeks to two months. Nothing here proves the root cause, so capture a trace before the next reboot.
Why do SCADA coils oscillate when the SLC 5/05 data is stable? #
Because the failure sits between the master and the PLC: on a failed upstream read the translator most likely returns null data instead of a Modbus exception, so a coil at 1 reads 1, 0, 1, 0 once per poll — every 1 s in the documented case — until a reboot.
Four observations identify this fault:
- The data in the SLC 5/05 is steady; only the SCADA view moves.
- One or many coils change state on every transmission, not at random times.
- Rebooting the translator restores correct reporting immediately.
- The interval between recurrences is measured in days to weeks, so a short test proves nothing.
The null-data mechanism stays a hypothesis until a trace shows it, but it is the one that fits: an exception would raise a comm alarm at the master, while empty data silently overwrites real values with zeros. One safety note: while this fault is live, any alarm or operator decision keyed to those coils is reading fiction — verify critical states in the PLC until the path is fixed.
What timeout order does the gateway need? #
Strictly descending, each layer longer than the one below it: upstream master timeout > messagetimeout > slavetimeout > chartimeout, at the values listed in the introduction.
The gateway supervises three timers, and the master supervises the gateway. When two timers expire together — or the wrong one expires first — a retry lands on a transaction that is still alive, the race condition the translator documentation warns about. The installation that oscillated started with messagetimeout 5 s, slavetimeout 5 s and chartimeout 1 s, under a master timeout of 1 s: two equal timers at the top and a master that gave up 4 s before the gateway did. An intermediate attempt fixed the magnitudes but left the message and slave timers equal, so both could still fire on the same response. The correction separates every layer:
- upstream master timeout: the longest
- messagetimeout: next below it
- slavetimeout: below that
- chartimeout: the shortest
Those parameter names and the ordering rule belong to this translator family. Another vendor's gateway carries different names and defaults, but the logic transfers: the supervising layer must always be able to outwait the layer below it.
How do the timeouts behave in one failed poll? #
With the ladder in place, one dead SLC 5/05 costs the master at most 600 ms of its window — 30 percent — so retries never stack on a sick link.
Walk one failed poll, with the SLC 5/05 going silent on a single request:
- chartimeout expires first: no bytes are arriving.
- slavetimeout caps the wait for the PLC response at five character-timeout periods.
- The gateway settles its answer to the master inside messagetimeout, leaving exactly one chartimeout of margin.
- The master holds the request open up to 2000 ms — 1400 ms of margin beyond the 600 ms latest answer.
- The master timeout is 4.0 times the slavetimeout, above the 2.5× rule of thumb that lets a second master join gracefully.
Now the broken configuration, for contrast: a 1 s master timeout under a 5 s messagetimeout has the master retrying 4 s before the gateway finishes the first request, and the equal message and slave timers of the intermediate attempt left no separation at all. Both arrangements let a null answer and a live transaction collide. Set the ladder above.
Why move the SLC 5/05 route to AB/Ethernet on port 2222? #
Because CSPv4 — the SLC 5/05's native protocol, offered as AB/Ethernet on TCP port 2222 — is a plain poll-response exchange with no CIP connection to time out, and it moves less than half the network bytes of the EtherNet/IP route.
EtherNet/IP was not designed for supervised poll-response traffic. It closes connections on its own clock — comparable gateway documentation shows a 120 s connection timeout as a default — and the ODVA rules leave an ambiguity for connected class 3 messaging when the encapsulation session times out while the TCP socket is still open: the peer is not allowed to answer with an error, so the pair can sit in a broken state that only a reboot clears. That is exactly the observed symptom shape.
AB/Ethernet does less work on both ends, carries fewer bytes, and sidesteps the connection machinery entirely. If EtherNet/IP must stay because another device on that route needs it, configure class=ucmm, which sends unconnected messages and cuts out the CIP connection-timeout wait, then verify with traces. The argument is the same one behind why Modbus polling is not the same as remote I/O: a master's scheduled read is not a maintained connection, and the route you pick should not force one.
How fast should the Modbus master poll? #
Fix the interval rather than letting the master poll as fast as responses arrive: response-driven polling raises the request rate exactly when the link starts failing, which is when it can least carry the load. Set the poll interval equal to the master timeout.
The documented master originally ran a 1 s timeout and issued a new request the moment the previous response landed. When the translator is answering null data, those answers come back fast, so a response-driven master turns a fault into a flood. Do not confuse the two numbers: the response timeout bounds one transaction, the poll interval paces the whole scan, and both were set equal in the final configuration. Pacing a shared path is the same discipline that explains why Modbus RTU drive polling on one serial port is slow — the budget is request count times per-request time, and you decide it before the fault does.
How do you capture the fault before rebooting the gateway? #
Start the translator's trace from a telnet session with set trace state=on mask=ia:+i+w+d while the oscillation is live, capture both the Modbus side and the PLC side, and only then reboot for a comparison capture.
Work the checklist in order:
- Confirm in the SLC 5/05 programming software that the suspect bits are steady while SCADA shows them flipping.
- Record which points oscillate and what their true states are — the pattern separates a translation fault from real field chatter.
- Enable the trace command above and keep the capture short, because the trace can drop lines under heavy output.
- Correlate each Modbus request with its PLC-side request, the PLC response, and what actually came back: data, exception, or nothing.
- Reboot, capture again, and diff the two.
The documented test set was Modbus coil addresses 1-56 and 4001-4112, plus registers 40001-40005 and 43001-43004. Verification passes when every one of those tracks the SLC 5/05 across repeated polls with no restart.
What result should you expect from these changes? #
A longer fault-free interval, not a proven cure: the documented installation went from oscillation after 13 days — under two weeks — to two months of correct reporting after the route change and the timeout correction, and the fault still returned once at the two-month mark.
That recurrence is what triggered the final adjustments: the master timeout doubled from 1 s, the poll interval fixed to match it, and messagetimeout separated to 600 ms. No further recurrence is on record. Commit to that configuration as the standing setup — timeout ladder first, because it costs minutes and changes nothing the master sees, then the route to AB/Ethernet on TCP port 2222 as a separate dated change — and keep the trace procedure documented at the cabinet so the next occurrence is recorded instead of rebooted away.
Questions asked
Why do only the coils that are actually on appear to oscillate?
Null data reads as all zeros. A bit whose true state is 1 alternates 1, 0, 1, 0 as good and empty responses replace each other, while a bit already at 0 looks steady whether the response carried data or nothing. Which points oscillate is therefore a free test of the mechanism before you set up any trace.
What if the SLC 5/05 needs longer than the slave timeout to answer?
Scale the whole ladder, never one timer. If the controller needs 1 s, set slavetimeout to 1 s, messagetimeout to 1.1 s — one character-timeout of separation — and the master timeout at 2.5 × 1 s = 2.5 s minimum. Breaking the order to speed up one layer is how the equal-timer race comes back.
Why put the master timeout at 2.5 times the slavetimeout?
The margin lets a second master — a maintenance laptop or an engineering tool — join the same gateway without its session and the first master's timers expiring together. At the documented slave timeout the rule gives 1250 ms; the documented master setting clears it with 750 ms to spare.
Did the firmware upgrade stop the oscillation?
No. The documented unit went from firmware E to F and the fault returned after 13 days, unchanged. Keep the translator's firmware current as hygiene, but the route change and the timeout ordering are the two corrections that moved the fault interval — not the firmware.
How long until the gateway counts as fixed?
Beat the documented bars: past two weeks quickly, then past two months before you trust it, and even then call it no recurrence on record, not proven. Leave the trace command and capture procedure at the cabinet so a future episode is recorded before a reboot destroys the evidence.