Restarting, Resubmitting, and Retrying Failed Jobs

Follow

 

Restarting, Resubmitting, and Retrying Failed Jobs

When a Job or Setup fails in JAMS, it can be resubmitted or restarted manually, configured to recover automatically, or even ignore failures to continue with preset recurrence schedules.

In this article:

 

Manually Restart or Resubmit Failed Jobs

Failed Jobs in JAMS can be manually Restarted or manually Resubmitted.

Restarting a Job will release an existing Job instance to run again, and will preserve existing Parameters associated with that Job instance. Most users will wish to Restart their failed Jobs to preserve Parameters. 

Resubmitting a Job will create a new Job instance (with a new Entry ID number), and will not preserve the Parameters of the failed Job. This method would be useful in cases where a Job failed due to invalid Parameters, as it allows the user to set new Parameters on the Resubmit Job.

Manually Restart a failed Job:

  1. Right Click on the failed Job in the Monitor.
  2. Select Release from the menu that appears.
  3. Ensure Release to run again is checked, and set other options as needed.
  4. Click OK. The existing Job instance will be restarted, preserving any existing Parameters.

Manually Restart a Failed Job within a Setup

Jobs within Setups may halt their Setup upon failure.

If the released Job exists within a Setup, ensure the Setup is also released from its halted state.

  1. Right Click on the failed Job in the Monitor.
  2. Select Release from the menu that appears.
    SetupJobSelect.png
  3. Ensure Release to run again is checked, and set other options as needed.
    SetupJobReleaseDialog.png
  4. Click OK.
    The released Setup Job may now enter a halted state. Release the Setup from its halted state to allow the Job to begin executing.
    SetupJobReleasedHalted.png
  5. Right Click on the halted Setup in the Monitor.
  6. Select Release from the menu that appears.
    SetupJobReleaseSetup.png
  7. Ensure Release halted Setup is checked, and set other options as needed.
    SetupJobReleaseSetupDialog.png
  8. Click OK. With the Setup released, the previously released Job will begin executing.

 

 

Manually Resubmit a failed Job:

  1. Select the Job from the Monitor or History view.
  2. Click the Submit button on the ribbon bar.
  3. Enter all applicable information and set Parameters as necessary in the Submit dialog.
  4. Click Submit Run Request. The new Job instance will be created. 

Return to Top

 

 

Recovery Options for Jobs

Using JAMS Recovery options, a Job can be configured to automatically retry on a failure. Like restarting a Job, a Job Recovery will re-run an existing Job instance, preserving any existing Parameters on the Job.

If a Job with Recovery options inside of a Setup or Workflow fails, the Job will automatically execute its retry options within the Setup or Workflow.

Settings to know

The Minimum Completion Severity is the level at which users want to take action on the Job.

Retry Count is the maximum number of retries if the Job fails. For instance, a Job with a Retry Count of 3 would attempt to execute a total of 4 times before ultimately failing.

Retry Interval is the wait time interval between attempts to run the Job. This is configured as dd.HH:MM:SS. Note that the interval is measured between the end time of the previous failure and the Scheduled time of the retry attempt.

Then Submit This Job allows users to respond to a failure by running a secondary Job. For some, this may be a cleanup job to deal with the failure event. Note that users should NEVER attempt to configure a Job as its own recovery Job, as this could cause an endless loop of failed Jobs.

Recovery Instructions contains the instructions that are included in the notification email sent on a Job failure. 

Configure Retry Options at the Job Level

  1. Open the Job Properties dialog for the Job in question, then select the Properties tab.
  2. In the Recovery section, adjust the Minimum Completion SeverityRetry Count, and Retry Interval options as desired. In the example below, the Job is configured to Retry 3 times after a failure, with an interval of one minute between retries.
  3. Save and Close the Job. The Job will now retry in the event of a failure.

Return to Top

 

 

Recovery Options for Setups

Setup level recovery options mimic those at the Job level, with options for Minimum Completion Severity, Retry Count, and Retry Interval. Jobs within a Setup will automatically execute their recovery options, but a failed Job within a Setup will not cause the Setup itself to fail unless properly configured. To cause a Setup to fail, users must configure Job step settings within the Setup. 

Settings to Know

Minimum Completion Severity is the level at which users want to take action on the Job step.

Setup Action if This Job Fails defines what the Setup will do if the Job fails. Continue will advance the Setup to the next step, even when a Job fails. Halt will cause the Setup to halt, requiring manual intervention to continue. Fail will cause the entire Setup to fail. Note that failing a Setup allows the Setup to begin executing its Setup-level recovery options. 

Wait for Completion before advancing step allows users to determine whether or not the Setup will wait for the Job to complete before advancing to the next step. Checking this box will cause the Setup to wait for the Job step to complete all of its retry options before attempting to move to the next step in the Setup. Leaving this box unchecked will result in the Setup moving on to the next step immediately, even if the Job in question is still executing, or attempting recovery options. 

Configure Retry Options on a Setup

  1. Open the Setup Properties dialog, then select the Properties tab.
  2. In the Recovery section, adjust the Retry Count and Retry Interval options for the Setup as desired.
  3. Select the Jobs tab, then click on the Properties button to open the Job Step Properties dialog.
  4. To force a Setup to wait while Job-level retries are performed, ensure Wait for completion before advancing step is checked on each relevant Job step.
  5. For each Job step where a Job failure should cause the entire Setup to fail, set the Setup Action if This Job Fails to Fail.
  6. Save and Close the Setup Job Properties dialog.
  7. Save and Close the Setup.

Return to Top

 

 

Recovery Options for Workflows

A Workflow Job's Recovery options are the same as other JAMS Jobs, with Minimum Completion Severity, Retry Count, and Retry Interval. Like Setups, Workflows will not fail unless they are configured to do so. 

Jobs and Setups inside the Workflow

Jobs and Setups inside of a Workflow exist as SubmitEntry activities. As with Setups, the Workflow can be configured to wait for a Job activity to complete, to wait after a Job activity fails, or to continue on in the Workflow, even before a Job completes.

To configure Job activity options, users can select the Job in question and use the Wait and WaitAfterFailure properties. Setting a Job to Wait will not allow the Workflow to continue past the Job activity that is waiting, until the Job activity initially completes. WaitAfterFailure, if checked, will cause the Workflow to halt on the failure of the Job activity. Halting the Workflow using WaitAfterFailure is a useful way to allow a Job or Setup activity to attempt its own recovery options before the Workflow continues.

Causing a Workflow to Fail

A Workflow will not fail based on the failure of an individual Job activity unless the Completion Action for that Job activity has either a TerminateWorkflow or Throw activity configured. TerminateWorkflow and Throw activities are not limited to Completion Actions; they can also be configured in line with any other activity or logic that should cause a Workflow to fail. 

Configure Retry Options for Workflow Jobs

  1. Open the Job Properties dialog and select the Properties tab.
  2. In the Recovery section, adjust the Minimum Completion SeverityRetry Count, and Retry Interval options as desired.
  3. Select the Source tab to view the Workflow Editor.
  4. Ensure that Wait and WaitAfterFailure properties have been configured as needed on Job and Setup activities.
  5. Ensure that TerminateWorkflow or Throw activities have been configured for situations that should cause the entire Workflow to fail. In the image below, the Workflow will terminate if the Sleep60 Job fails.
  6. Save and Close the Workflow Job. 

Return to Top

 

 

Failed JAMS Jobs with Recurrences

JAMS Jobs with recurrences will resubmit (creating a new Job instance with a unique Entry ID number) on configured intervals, until a set end time. When a recurring Job fails, it will either cease to create any further recurrences or continue normally depending on the Job's Resubmit even after failure setting.

Configure Resubmit Options on a Recurring Job

  1. Open the Job Properties dialog and select the Properties tab.
  2. In the Schedule Options section, expand and set the Recurrence options. In the example below, the Job is configured to Resubmit every 5 minutes after 8AM until 5PM.
  3. Set Resubmit even after failure as desired. If a recurrence fails with this box unchecked, there will be no further recurrent submissions of the Job. Checking the box will cause the recurrent submissions to continue after a failure is observed.
  4. Save and Close the Job.

Return to Top 

Have more questions? Submit a request

Comments