Intermittent connection issues with CGTP

[JTremblayAGT]

I started seeing some intermittent issues where the CGTP connection times out when attempting to connect to a controller. In one of the instances everything worked fine on my end which makes us suspect potential port exhaustion on the client side. However it could also be an issue on the controller.

Does the client perform a proper cleanup when being garbage collected or reinstantiated? It does not implement IDisposable so it is hard to tell whether it behaves as one would expect.

[Rufus31415]

Good question..

For the CGTP client, it does not hold any persistent TCP connection between calls. Every HTTP request open a new connection and close it right after in a .NET HttpWebRequest, so there is nothing to "dispose" in the traditional sense. This is why it does not implement IDisposable - it is by design, not a bug. Garbage collection will not cause any leak of open sockets.

But your port exhaustion theory is actually very plausible. When you close a TCP connection it does not disappear immediately, the OS keep it in TIME_WAIT state for maybe 60 to 120 seconds depending on the system. On Windows the default range of ephemeral ports is quite small, around 16000 ports. So if you make many requests quickly, you can run out of available ports and new connections will fail. This match well with your "intermittent" observation.

Few things you can try: first, if you are reading many variables in a loop, try to group them in a single batch call instead of one call per variable, this reduce a lot the number of connections. Second, on Windows you can increase the port range and reduce the TIME_WAIT delay in the registry, there are parameters called MaxUserPort and TcpTimedWaitDelay under the TCP/IP parameters key.

Also it is possible the issue is on controller side, the FANUC web server has a limit of concurrent connections it can handle. If you can run netstat at the moment of the timeout you will see quickly if the problem is client side (many TIME_WAIT) or controller side.

[JTremblayAGT]

The specific case that's failing is when trying to read a bunch of TP programs' comments where we write their revisions for program updates. As far as I know there isn't a batch operation for this. I will look into diagnosing the connection to pinpoint where the issue comes from.

[JTremblayAGT]

We've been investigating this issue and our favourite networking expert Claude has given us the following information after analyzing Wireshark captures:

What's happening

The controller always closes connections (Connection: close)

Every single HTTP/1.1 request the Underautomation library sends includes Connection: close. The Fanuc R30iB honors it by sending a TCP FIN immediately after each response. This means one new TCP connection is opened per request (or per 1–2 pipelined requests).

The library leaks sockets in CLOSE_WAIT

This is the real bug. After the server sends FIN, a proper client should call Close() on the socket to complete the 4-way teardown. The library doesn't. Instead:

1. Server FINs at e.g. t+0ms
2. Client ACKs the FIN → socket enters CLOSE_WAIT (OS is waiting for the app to close)
3. Library immediately opens a new socket for the next request
4. The old CLOSE_WAIT socket is abandoned — it sits there sending TCP keepalive probes every ~5 seconds
5. After ~60 seconds, the controller gets tired and sends RST to forcibly kill it

This is visible in both captures:

Capture | Close behavior | CLOSE_WAIT duration | Port burn rate -- | -- | -- | -- LOCAL (working) | Server FIN → client keepalives → controller RST at ~62s | ~62 s | ~7 new ports/sec during bulk write CGTP (failing) | Same — controller RST not reached in the 131s capture window | >131 s | Same rate

The pipelining retry doubles the problem

The library pipelines: it sends a 2nd request ~2ms after receiving the 1st response, before the server FINs. When the server FINs mid-pipeline, the 2nd request is dropped. The library retries it on a new connection. So per pair of requests: 2 sockets opened, 2 leaked into CLOSE_WAIT.

Steady-state port math

During a bulk write ($UALRM_MSG[1]…[N]):

• ~7 new connections/second
• Each lingers in CLOSE_WAIT for ~60 seconds (until controller RSTs it)
• Steady-state: 7 × 60 ≈ 420 ports occupied

On your machine (ephemeral range 49152–65535 = ~16K ports) → 2.6% usage, no problem.

On users' machines that fail:

• Restricted ephemeral port range (old Windows default was 1024–5000 = ~4K; some enterprise GPO policies still set this)
• Or they run multiple robot connections simultaneously, multiplying the rate
• Or their TcpTimedWaitDelay / keepalive timeout is longer

Here is what it suggested is the root cause, keep in mind this is from a LLM so it might be wrong:

the root cause is that their HttpClient/WebRequest usage isn't disposing response objects, which prevents the underlying socket from completing the TCP close. They should either remove Connection: close (and let HTTP/1.1 keep-alive work) or properly call Dispose() on the response.

[Rufus31415]

We looked at the code and confirmed your analysis is correct.

Every HTTP request the library sends includes a Connection: close header. The controller honors it and sends a TCP FIN right after each response. That means one TCP connection is opened and closed per request, every time.

After the controller sends its FIN, the socket on your side enters CLOSE_WAIT. It stays there for roughly 60 seconds until the controller eventually sends a RST to force-close it. During that window, the port is occupied. Your Wireshark captures match this exactly.

Based on your description, you are most likely calling GetProgramComment() in a loop over a large list of programs. That method does one HTTP request per call. With, say, 100 programs at 7 requests per second, you build up around 420 sockets stuck in CLOSE_WAIT at any given moment. On a machine with a restricted ephemeral port range, or when you run several robot connections at the same time, that is enough to exhaust the available ports and cause the timeouts you see.

The main fix is to remove the Connection: close behavior. Without it, .NET handles the socket cleanup properly through its connection pool, so sockets are closed cleanly instead of being left in CLOSE_WAIT.

We also found a secondary issue: when downloading binary files from the controller, the HTTP response object is not disposed after use. That is a separate socket leak, less likely to be the cause of your problem since you are reading comments and not downloading files... but we will fix it at the same time