Multi-queue processing allows you to split CIR410 ? Cycle-end Processing across multiple queues, with each queue dedicated to a different range of customer numbers. For very large databases, this can help speed the total wall-time processing required for cycle-end. Multi-queue processing only works if you are also running DBR.
You can increase the throughput for this process by dividing the input file into separate queues (a maximum of nine queues is allowed), and running the CIR4MQ ? Multi-queue Cycle-end process instead. CIR4MQ is identical to CIR410, except that it is designed to split processing for the publication across multiple queues.
Setting up the Queues
This is controlled at the publication level. To enable multi-queue streaming for a pub, go to the Cycle End Queues tab when inquiring on a pub at CIRPUB.
Use the Add button to specify a queue and starting customer number. You can use the Customize button to add other available columns to the view.
The columns available in the view are:
- Queue: queue number for the customer range in question; selected by the user from the available defined queues)
- Start customer: starting customer number for the queue; user-assigned. If the lowest defined customer number (across all queues) is not zero, the system resets it to zero to ensure that no customers are missed.
- Issue date: date of the issue for which the cycle-end is being performed. Populated by the system as the queue completes. Used (with last cycle-end number) to recognize that a queue has completed, and the order of queue completion.
- Last cycle end number: system-assigned number for each queue as it finishes processing. Advantage uses this, plus the issue date, to determine the order of queue completion. The last queue to complete also performs process-wide wrap-up tasks.
- Request number: system-assigned number that uniquely identifies an Advantage process request. Used for internal purposes and troubleshooting.
- Update date: last date a queue definition was added or changed at CIRPUB.
- Update user: user ID of the person to last perform a queue
definition add or update.
Queues are defined by publication (not issue date), so that the multi-queue information present at CIRPUB is the information that CIR4MQ uses when it initiates the cycle-end(s).
If a queue fails, all of its updates are rolled back to the beginning, as if that queue had never started. If the queue determines that it is the last to finish (based on the other CEQ records) then it completes the global wrap-up. If an error occurred after that point, the CEQ update is also rolled back, along with any global updates. In an error situation either before or after the wrap-up section, all that is necessary is to correct the problem and restart that queue.
When CIR4MQ is initiated, it checks the queue information for the publication involved. The process then initiates a separate cycle-end process for each queue (and corresponding starting customer number). These processes can run concurrently.
As the queues finish, the Issue Date field is updated to reflect successful completion of that queue. Also, the Cycle End Last Number is incremented on a queue-by-queue basis. These two fields---Issue Date and Cycle End Last Number---allow the process to track the order in which the queues are completing, and also which of the queues is the very last to complete. The last queue-based process to finish conducts final reporting for the issue.
Multi-queuing may be used with any of the CIR4MQ processing modes: report, trial, update, mail, etc.
CIRSMQ (Multi-queue Setup Processing)
An alternative to using the Cycle End Queue tab at CIRPUB is to run the CIRSMQ process. The process includes a run-time setting for the number of cycle-end queues. The process then evenly divides the customer base across the queues---in effect providing an automated shortcut to the ADD function on the Cycle End Queue tab.