Automatic Retries

7 min read

7 min read

7 min read

7 min read

7 min read

7 min read

7 min read

7 min read

7 min read

An invocation can take several attempts: when an attempt fails, Ice can automatically make another one. By default, Ice retries once, immediately; Ice.RetryIntervals sets how many times Ice retries and after which delays. Ice retries only when it is safe to do so, as described below.

An attempt fails when Ice cannot establish a connection or send the request, when the connection closes before the reply arrives, or when the server reports a dispatch exception. An attempt that completes with a user exception is not a failed attempt, and Ice does not retry it.

Ice answers two questions to decide whether to retry a failed attempt.

Ice retries an attempt that fails with a local exception, including a dispatch exception such as UnknownException, except when:

  • the exception is MarshalException, FacetNotExistException, or OperationNotExistException
  • the exception is ObjectNotExistException and the proxy is not a well-known proxy
  • the invocation timeout expired
  • the invocation was canceled, including by interrupting its thread
  • the application closed the connection

Ice does not retry a batched invocation or an invocation on a fixed proxy.

When an attempt fails before Ice has written the entire request to the client's local transport buffers, the server cannot have received the request, so retrying is safe. Once the request is written, Ice assumes that the server may have dispatched it.

When a server closes a connection gracefully, for example because it is shutting down, it completes the dispatches in progress and does not dispatch the requests it still receives on this connection. Ice retries these requests: the server did not process them, so retrying them is safe.

Ice also retries an attempt on a well-known proxy that fails with ObjectNotExistException, which allows an Ice object to migrate to another object adapter: Ice removes the proxy's entry from the locator cache, so the retry asks the locator for the object's current location.

When Ice does not retry a failed attempt, or when the last retry fails, the invocation fails with the exception of its last attempt.

By default, Ice provides at-most-once semantics: the server dispatches a request at most once, so Ice does not retry an attempt when the server may have dispatched its request. Declare an operation idempotent when it does not need this guarantee: Ice then retries a failed attempt even when the server may have already dispatched the request. For example:

Slice
interface Account
{
long withdraw(long amount);
idempotent long getBalance();
idempotent void changeAddress(string newAddress);
}

Although withdraw clearly requires the stricter treatment, there is no harm in automatically retrying the getBalance operation even if the server executes the same request more than once.

In general, "read-only" operations are good candidates for the idempotent keyword whereas many mutating operations are not. However, the risk of duplicate requests is acceptable even for some kinds of mutating operations: changeAddress is idempotent because executing the request twice has the same effect as executing it only once.

The benefit of the idempotent keyword and the associated relaxation of retry semantics is that an invocation that otherwise might have thrown an exception has at least one more chance to succeed. Furthermore, the application does not need to initiate the retry, and in fact the retry activities are completely transparent: if a subsequent retry succeeds, the application receives its results as if nothing went wrong. The invocation only throws an exception once Ice has reached its configured retry limits.

The Ice.RetryIntervals property configures the retry behavior for a communicator and affects invocations on every proxy created by that communicator. The value of this property is a list of integers, separated by commas or whitespace. The number of integers determines how many retry attempts Ice makes, and the value of each entry represents a delay in milliseconds. If this property is not defined, the default behavior is to retry once immediately after the first failure, which is equivalent to the following property definition:

Properties
Ice.RetryIntervals=0

You may want a more elaborate configuration for your application, such as a gradual increase in the delay between retries:

Properties
Ice.RetryIntervals=0 100 500 1000

With this setting, Ice retries immediately as in the default case. If the first retry attempt also fails, Ice waits 100 milliseconds before trying again, then 500 milliseconds, and finally tries one more time after waiting one second.

In some situations you may need to disable retries completely. For example, an application might implement its own retry logic and therefore require immediate notification when a failure occurs. A first value of -1 disables retries:

Properties
Ice.RetryIntervals=-1

Set Ice.Trace.Retry to 1 to trace retries:

Properties
Ice.Trace.Retry=1

Ice then logs each retry, with the exception that caused it, and logs when an invocation reaches the retry limit. Set Ice.Trace.Retry to 2 to also trace the failed connection attempts to endpoints from the locator cache, after which Ice asks the locator for the endpoints again.