Troubleshooting
Why HMI values show question marks: PLC comms timeouts
Short answer
Yes — a running PLC program can cause intermittent HMI timeouts by consuming nearly all controller execution time, and clean Ethernet counters do not clear it: utilization measures bytes, not the cost of servicing requests. Work controller load first: Task Monitor above 90% on the dropping units against 7–15% on healthy ones, then message pacing, packet-rebuild address forms, read grouping, and power last.
On this page
- What each message is telling you
- Why clean Ethernet counters do not clear the controller
- Measure controller load where it counts
- Find the logic that starves communications
- Remove the documented packet-rebuild triggers
- Regroup reads by update rate, not by plant area
- Rule power and grounding in or out by timestamp
- The order of attack
- Questions asked
Yes — the program already running in the controller can cause these dropouts, and your own measurements point there rather than at the network. Values that turn to ??? for about 10 seconds and recover unattended, on different controllers at different times, with under 1% link utilization and zero Ethernet errors, is the signature of request starvation or packet rebuilds on the controller side — not of a saturated or failing link. Work it in this order: measure controller load, fix the logic that creates load, remove documented packet-rebuild triggers, regroup reads by update rate, and settle the power question last, by timestamp.
What each message is telling you #
Four messages, four different layers, and none of them names the failed component by itself:
- "ControlLogix Optimized packet being reinitialized" — the driver discarded and rebuilt an optimized request packet. Triggers include a changed item list, an incompatible address, a connection state change, or a failed request. Start with the item the event names, F07_Air_Alarm_Reset in your log, and everything sharing its packet, because a packet-level failure carries all of its items down together.
- "CIP Messaging Error: a message timed out waiting for a response" — the request went out and the response did not come back before the timeout expired. The message does not say where the delay sat: controller, communications module, backplane, network, or client.
- "EE00036h : Control Net Service cannot be performed while Object is in current state" — the target object was mid-transition and refused the service. Repeated hits mean looking at what immediately precedes them: a reconnect, a screen change, a packet rebuild.
- "A Timeout occurred where the engine didn't generate a timeout in the time required" — the client's own engine ran late. Host scheduling and driver queueing are now in scope even though the wire is clean.
The @IsPresent item in your log is a driver health probe, not a process value: its failure says the whole path to that controller stalled, not that one tag went bad.
Why clean Ethernet counters do not clear the controller #
Utilization measures bytes on the wire. It does not measure the controller-side cost of parsing thousands of small requests, managing connections, moving data across the chassis backplane, rebuilding optimized packets, or looking up scattered tags. Your findings — under 1% utilization on every machine, no Ethernet I/O, scan classes of 1 s or longer — rule out sustained link saturation. They say nothing about transaction load. Commit here: replacing the managed switch is not the next step, and the half-duplex mismatch you already found sat on a port outside the PLC circuit.
The recovery pattern agrees. A module or cable that fails does not heal itself in 10 seconds; something queued, timed out, and then drained once the load passed.
Measure controller load where it counts #
Use Task Monitor in the Logix programming software, not the controller's web page. The basis is a Studio 5000 Task Monitor release note: the high-usage warning was reworded to say that very high CPU usage can appear as intermittent communication issues. That is the vendor naming your exact symptom as a load symptom.
Your numbers, read correctly:
- Most dropping controllers showed about 94% in Task Monitor while their web pages showed about 5%. Those are different measurements over different windows; trust the task accounting, captured with identical settings on every controller.
- Two controllers showed 7–15%. Those are your healthy control group — count drops per controller per day and compare against them.
- After moving the time slice to 50% and enabling the communications-priority option, displayed load fell to roughly 60% with dips near 30%. Idle capacity went from 100 − 94 = 6% to 100 − 60 = 40%.
- The null task is the true idle indicator: execution opportunity no task consumed. A controller with a near-zero null task has nothing left to answer requests with, whatever the headline percentage says.
Set one sampling rate, capture the same time window everywhere, and know which tasks are continuous and which are periodic before comparing two controllers. The downward spike that coincided with a dropout is a lead, not proof — it can be a task-rate change or the same disturbance seen twice.
Safety note: buying headroom by stretching task periods or widening a watchdog changes when outputs actually update, so confirm interlock and motion timing still hold before any such change stays in.
If task structure and scan behavior are unfamiliar ground, the base mechanics are laid out in how to learn PLC, SCADA and DAQ from scratch.
Find the logic that starves communications #
A program can delay HMI responses while generating no external Ethernet traffic at all — the case you suspected. Two mechanisms, and both fit your symptoms:
- Execution starvation: a continuous task running flat out, a periodic task firing far faster than the process needs, unbounded loops, or repeated array moves leave no time to service external requests.
- Hidden messaging: message instructions between processors or modules ride the backplane, not the Ethernet port your sniffer watched, and they consume connection and buffer resources all the same.
Audit in this order:
- Message instructions retriggered every scan. Check the enable, done, error, and retry sequencing: a failed message re-enabled without pacing retries at scan rate and delivers nothing — the classic self-inflicted load.
- Periodic tasks whose rates outrun the process they serve.
- Continuous-task logic that leaves the null task near zero.
Remove the documented packet-rebuild triggers #
The RSLinx Classic help topic for this exact message documents one trigger worth a full search: an HMI request for an individual bit within a floating-point address, in the forms [TopicName]FloatAddress.1 and [TopicName]FloatAddress[0].1. One such reference disrupts the packet and ripples into neighboring addresses. Search every topic database and screen export for a bit suffix on a floating-point item. Your search found none, so close this branch — and re-run it after any screen edit, because one keystroke reopens it.
The second trigger is an item list that changes. One reported field case had a driver rebuilding its requests around whichever screen was visible, so the controller never saw a stable, repeatable transaction to optimize. Test it cheaply: timestamp every packet-reinitialized event and compare against screen-change records from the same shift.
Regroup reads by update rate, not by plant area #
Every transaction carries request, response, connection, and scheduling overhead, which is why 10 reads of 100 tags cost the controller less than 100 reads of 10 tags at the same update rate. Your groups are organized by plant area, which fragments compatible data into many small transactions; the installation's estimated worst case was about 1,000 tag updates per second.
Illustrative worked example on those numbers — 1,000 tags on a 1 s scan class, assuming the area-based groups average 10 tags per read and regrouped packets hold 100:
- As grouped today: 1,000 tags ÷ 10 tags per read = 100 reads per update; 100 reads × 1 update/s = 100 transactions per second; 100 × 3,600 s = 360,000 transactions per hour.
- Regrouped by update rate and address proximity: 1,000 tags ÷ 100 tags per packet = 10 packets; 10 × 1 update/s = 10 transactions per second; 10 × 3,600 s = 36,000 transactions per hour.
- Same 1,000 tags, same 1 s freshness: 100 ÷ 10 = 10 times fewer transactions to service.
Decision: regroup every driver's tags by required update rate first, address proximity second. Count drops before and after so the gain is proven, not assumed.
Rule power and grounding in or out by timestamp #
A supply disturbance can interrupt a controller or communications module without leaving one Ethernet error. But your drops already occur at night and on weekends, when the construction crews and their generators are gone — that parks this branch behind the load work. Settle it with data anyway: log every dropout with date, time, and controller for seven straight days. Clustering inside working hours points at large loads sharing feeds with the panels — inspect supply quality and bonding at those hours. An even spread across 2 a.m. Sundays keeps this branch parked for good.
The order of attack #
- Re-measure every controller in Task Monitor with identical sampling settings and time slice; record the null-task percentage. Controllers reading above about 90% while the healthy pair reads 7–15% carry the fault until proven otherwise.
- Fix the load: pace failed message instructions, bound the loops, slow periodic tasks that outrun their process, and verify watchdog and output timing after every change. Expected result: drop counts fall on the high-load controllers and stay flat on the healthy pair.
- Search all topics for bit-in-float address forms; remove any found; re-search after every screen edit.
- Regroup tags by update rate and contiguity — the worked example's 100 to 10 transactions per second is the scale of gain available.
- Log drop timestamps for seven days against site activity: working-hours clustering sends you to supply and bonding; an even spread keeps you on load and transactions.
- Keep the packet captures on file. They are the proof the wire is clean the next time someone proposes replacing the switch.
Questions asked
Can the controller be at fault when the packet capture shows the network is clean?
Yes. A capture proves wire delivery, not controller response: a request can arrive intact and still time out because the controller had no execution time left to service it, and recovery in about 10 seconds fits a transient overload draining, not a failed module. Controller-to-controller messaging over the backplane never appears in an Ethernet capture either, so hidden traffic is invisible to the sniffer.
What does an optimized packet actually contain?
A set of tags the driver plans to request from one controller, built once and reused so the controller can serve the same transaction efficiently. It gets rebuilt when the item list changes, an address is incompatible, or a request fails — and a rebuild restarts service for every item in that packet, which is why one bad address degrades its neighbors too.
Why did Task Monitor read 94% when the controller web page read about 5%?
They are different measurements taken over different windows, and the web page figure is not task accounting. Trust Task Monitor with one sampling rate set identically on every controller, captured over the same interval, and read the null task as the true idle indicator before comparing any two controllers. The tool's own high-usage warning names intermittent communication issues as the symptom.
Is regrouping tags by update rate enough to fix the drops on its own?
Often not alone, but it is the cheapest permanent gain: the same 1,000 tags at a 1 s scan can drop from 100 transactions per second to 10. Pair it with load work — pacing retriggered message instructions and slowing periodic tasks that outrun the process — and count drops per controller per day so the fix that worked is identified by data, not by feel.