ADOP Explained (Online Patching tool for Oracle EBS 12.2)

ADOP Explained (Online Patching tool for Oracle EBS 12.2)

This section describes the online patching cycle from beginning to end, illustrating the actions taken in the different phases and putting into context the more detailed description of online patching in the following sections. It is designed to be read in conjunction with the important background material provided in the "Patching and Management Tools" chapter of Oracle E-Business Suite Concepts.

The online patching cycle consists of a number of high level phases:

prepare

apply

finalize

cutover

cleanup

A high level overview of an online patching cycle would, programmatically, look like this:

# Prepare for patching:

$ adop phase=prepare

# Apply patches:

$ adop phase=apply patches=<patch number>

# Finalize patch application:

$ adop phase=finalize

# Perform cutover:

$ adop phase=cutover

$ . <EBS_ROOT>/EBSapps.env run

# Perform user acceptance testing via application UI

# Perform cleanup:

$ adop phase=cleanup

Important Additional Points

After an online patching cycle is started, you should not perform any configuration changes in the run edition file system. Any change made will not be propagated, and will therefore be lost after cutover is complete.

The prepare, apply, and fs_clone phases all require at least 10GB of free disk space. All other phases require 1GB of free space. A warning message will be displayed if less than the needed amount is available.

The directories where you extracted the patches applied in a given patching cycle must be retained, in the same location and with the same contents, until the next prepare phase completes. This is also a requirement for patches applied in hotpatch or downtime mode.

Any customizations must be applied to the patch edition during the apply phase, normally after any Oracle E-Business Suite patches have been applied.

Special Phases

Two additional phases are provided for specialized use. Neither can be run in conjunction with any other phase. Further details of these phases are described in later sections.

The abort phase is used to terminate a patching cycle before it is complete, and roll back any changes that have been made. It can also be run in conjunction with a full cleanup operation.

The fs_clone phase is a command (not related to adcfgclone.pl) that is used to synchronize the patch file system with the run file system. Normally, the fs_clone phase should only be run when mentioned as part of a specific documented procedure.

Important: You may perform a procedure that as a final step instructs you to run fs_clone. You do not have to do this immediately: the key requirement is to run fs_clone before you start the next patching cycle. And if you are performing multiple procedures, each of which requires fs_clone to be run at the end, you only need to run it once before the start of the next patching cycle.

The Online Patching Cycle

 

adop will automatically set its environment as required, but it is the user's responsibility to set the environment correctly for any other commands that may be run. Set the run edition environment whenever executing commands that you intend to affect the run edition.

For example:

$ . <EBS_ROOT>/EBSapps.env run

$ adstrtal.sh

Set the patch edition environment whenever you intend to execute commands that affect the patch edition.

For example:

$ . <EBS_ROOT>/EBSapps.env patch

$ sqlplus apps/apps @my_custom_patch_script.sql

The adop tool executes non-interactively, executing the specified phase or phases in order. In a multi-node deployment, adop is only executed by the user on the master node: internally, adop will use ssh remote execution to run required actions on all secondary nodes automatically. In addition, adop can be used to generate reports about patching operations in the environment.

adop is typically run as follows:

$ adop phase=<phase_name>

The phase parameter accepts the following phase names. These names can be specified individually, or (except where otherwise noted) with other phase names in a comma-separated list:

prepare - Prepares the environment for patching.

apply - Applies the specified patches to the environment.

finalize - Performs any final steps required to make the system ready for cutover.

cutover - Shuts down application tier services, makes the patch edition the new run edition, and then restarts application tier services. This is the only phase where downtime requires.

cleanup - Removes obsolete code and data from old editions.

abort - Aborts the online patching cycle by dropping the database patch edition. This phase cannot be specified with any other phase.

fs_clone - Recreates the patch edition file system as an exact copy of the run edition file system. This phase cannot be specified with any other phase. Use of fs_clone is normally not required. Situations that do require fs_clone are will explicitly document that requirement. If running this phase, ensure that your current working directory is not within the patch edition file system.

Online Patching Cycle Steps - Prepare Phase

This section describes the principles of adop operation in the prepare phase, followed by the steps you take to run this phase.

Note: The exact actions taken during the prepare phase are context-dependent: for example, the first time it is ever run on a system; when it is run after an apply phase has been aborted; and when it has been run after cutover.

Principal adop Actions

During the prepare phase, adop performs the following steps.

Checks whether to perform a cleanup, which will be needed if the user failed to invoke cleanup after the cutover phase of a previous online patching cycle.

Validates system configuration to ensure that the system is ready to start an online patching cycle.

Checks to see if the database is prepared for online patching:

Checks if the database user is edition-enabled. If not, adop immediately exits with an error.

Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. This service is created automatically, but its continued existence is validated on each prepare.

Checks to see if logon trigger exists and is enabled. If the logon trigger is missing or the patch service has not been created, adop will automatically try to fix the issue so that it can proceed. If it cannot do so, it will exit with an error message.

Checks the integrity of the database data dictionary. If any corruption is found, adop will exit with an error. For information on how to resolve data dictionary corruptions, refer to My Oracle Knowledge Document 1531121.1, Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2.

Checks that the E-Business Suite Technology Codelevel Checker (ETCC) has has been run, to verify that all required patches have been applied to the database.

Checks system configuration on each application tier node. A number of critical settings are validated to ensure that each application tier node is correctly registered, configured, and ready for patching.

Checks for the existence of the "Online Patching In Progress" (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists).

The flow of control is as follows.

If the ADZDPATCH program has not yet been requested to run, a request is submitted.

The status of ADZDPATCH is determined. If it is pending, it may be waiting for an incompatible program to finish. At that point, its status will change to running, and it will allow the prepare phase to proceed. A message to this effect is displayed to the user.

The next stage depends on whether the concurrent managers are running:

If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.

If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.

If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running (that is to say, as noted in Step 2, no incompatible programs are found). The prepare phase then continues.

Note: ADZDPATCH is cancelled when the cutover phase is complete.

Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.

Checks the database for the existence of a patch edition, and creates one if it does not find one.

Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working.

If any of these checks fail, adop will exit with an error.

Optional User Checks

Before you run the prepare phase to start a new patching cycle, you may wish to perform a couple of optional checks.

The first check is to validate your system for patching, by running the command:

$ adop -validate

Note: If you run this command while a patching cycle is in progress, validation will take place for the cutover phase.

If you run this command from the primary (master) node, adop will perform the validation checks on all the available nodes available in the system. In contrast, if you run it from a secondary (slave) node, it will run only on that node.

The second check is to confirm there is adequate free space on your system to support a patching cycle:

SYSTEM tablespace - minimum of 25 GB free

APPS_TS_SEED tablespace - minimum of 5 GB free

You can do this by running the $AD_TOP/sql/ADZDSHOWTS.sql script:

Note: For instructions on how to increase the size of a tablespace, refer to the "Altering and Maintaining Tablespaces" section in the "Managing Tablespaces" chapter of the Oracle Database Administrator's Guide.

Required User Actions

You perform the following steps in the prepare phase.

Set the environment by executing (sourcing) the run file system environment file:

 $ source <EBS install base>/EBSapps.env run

For more information, see Setting the Environment in Running AD Utilities

You can confirm that the environment is properly set by examining the relevant environment variables:

$ echo $FILE_EDITION

run

$ echo $TWO_TASK

dbSID

If you had sourced the incorrect environment file (i.e. from the patch file system), the environment variables would show as:

$ echo $FILE_EDITION

patch

$ echo $TWO_TASK

dbSID_patch

Download patches to be applied and place them in the $PATCH_TOP directory of your system. This directory is pre-created by the install in the non-editioned file system (fs_ne) and should not be changed.

Important: On a multi-node system with non-shared file systems, you must copy the patch files to each separate $PATCH_TOP directory, so that the patch files are available from the same location on all nodes.

Unzip the patch:

$ unzip p99999999.zip

Prepare the system for patching by running the following command to start a new patching cycle:

$ adop phase=prepare

Synchronization Options

In this step, the patch file system application tier is synchronized with the run file system application tier. This can be done by any one of three methods, the first of which is the default and therefore does not need to be specified in the adop command:

Method 1 - Identify the patches that were applied to the run APPL_TOP and apply them to the patch APPL_TOP. The following steps are performed automatically:

The patches that need to be applied to the patch APPL_TOP are identified from the database.

The merged patches are applied by the adop utility.

The adop utility identifies the patches to be applied, and applies them silently to the current patch APPL_TOP. As this procedure only requires the application of previously unapplied patches, it requires less time and disk space compared to Method 2 (below).

Using the skipsyncerror parameter

In some circumstances, this synchronization method may fail when applying a series of patches to the patch edition. This can happen if the previous patching cycle included patches that failed to apply correctly, and was followed by subsequent patches that corrected the issue.

The skipsyncerror parameter enables you to specify that you expect any synchronization errors in the prepare phase to be fixed automatically in the synchronization that takes place with subsequent patches.

If the value of the parameter is passed as 'yes', the first patch to be synchronized will be done with the 'autoskip' flag set.

Important: It is your responsibility to check the log files and correct any errors in the subsequent apply phase, or to confirm that synchronization with subsequent patches resolved the issue.

An example of using this parameter would be as follows.

You run adop phase=prepare.

The phase fails with an error when trying to synchronize the run and patch file systems. That is, the attempt to synchronize a patch fails, but it is known that a subsequent patch will correct the problem.

You examine the log files and conclude that the synchronization errors will be fixed automatically in the synchronization that takes place with subsequent patches.

You run the command adop phase=prepare skipsyncerror=yes to restart the prepare phase. This time, application of the patch that failed in the previous prepare will be retried with the 'autoskip' flag set.

Alternatively, if you are not confident that the error will be fixed (for example, you cannot identify the cause from examination of the log files), you should:

Run the command adop phase=abort

Run the command adop phase=fs_clone

Rerun the command adop phase=prepare

Method 2 - Create a new patch file system by cloning the run file system using the fs_clone command.

This method is useful if the APPL_TOPs have become very unsynchronized (meaning that there would be a large number of delta patches to apply). It is a heavyweight process, taking a backup of the entire current patch APPL_TOP and then cloning the run APPL_TOP to create a new patch APPL_TOP. As this method requires more time and disk space, it should only be used when the state of the patch file system is unknown.

The fs_clone command is run as a special adop phase:

$ adop phase=fs_clone

This command must be invoked from the run file system, before the next prepare phase is run.

Note: The patch file system requires at least 25 GB of free disk space to be available for adop operations, including fs_clone. If there is insufficient free space, the adop operation will fail.

If desired, you can change the temporary file system location used by fs_clone, by setting the T2P_JAVA_OPTIONS environment variable to point to a temporary location of your choice:

$ T2P_JAVA_OPTIONS="-Djava.io.tmpdir=/home/t2p/temp"

$ export T2P_JAVA_OPTIONS

Note: You cannot change the fs_clone temporary location by changing the value of the $TMP environment variable.

If an fs_clone operation fails, you can rerun it with the option force=yes to restart it from the beginning (with the same session ID), or force=no to restart it from the point where it failed.

Method 3 - Much faster than the other two methods, this delta synchronization method uses your choice of third-party utility to synchronize the file systems by copying files as applicable from the source directory to the destination directory, optionally ignoring any files and directories you may decide to specify in an exclusion file.

To use this method, specify the parameter/value pair sync_mode=delta on the adop command line:

$ adop phase=prepare sync_mode=delta

Example implementations are provided using rsync on UNIX and RoboCopy on Windows, but you are free to use another utility of your choice.

The delta_sync_drv.txt file includes examples for setting up synchronization using rsync on UNIX or RoboCopy on Windows.

Windows users should refer to their platform-specific release notes for restrictions that currently apply when running fs_clone.

Synchronizing Customizations

The default method of file system synchronization handles official patches but will not synchronize any manually applied customizations.

Additional Information: The synchronization modes and their associated actions are as follows:

Patch synchronization - apply missing patches

Delta (file) synchronization - copy file changes

fs_clone synchronization - clone entire file system

Examples of patching actions that are not synchronized by default include:

Compiling user-defined JSPs

Copying some third-party libraries

Copying and compiling user-defined concurrent programs

Copying and generating user-defined forms

To include custom patching actions in the default file system synchronization, you must include the required commands in the Custom Synchronization Driver, $APPL_TOP_NE/ad/custom/adop_sync.drv. You will add your customizations to the following section of the file:

#Begin Customization

...

#End Customization

All the actions defined in this file will be performed by adop automatically during the prepare phase. Be aware that there are two categories of custom command in adop_sync.drv: those that are run one time only, and those that are run at each file system synchronization (during the adop prepare phase).

Important: The adop_sync.drv file is not currently reset to its template file at any point. Consequently, after cutover (and before the next prepare phase), you should review the contents of adop_sync.drv and ensure the requirementns for your custom commands continue to be met.

This is only an outline of the steps you need to take to preserve customizations. For full details, refer to Oracle E-Business Suite Developer's Guide

Prepare Phase in Multi-Node Environments

In a multi-node environment, one application tier node will be designated as the primary node. This is the node where the Admin Server is located, and will usually also be the node that runs Oracle HTTP Server. All other application tier nodes are designated as secondary nodes.

In a multi-node environment, you must enable ssh from the primary node to all secondary nodes to permit adop remote invocation. For ssh setup steps, refer to Set Up Secure Shell on Application Tier Nodes. In a multi-node environment, adop commands are always run from the primary node only. adop executes required patching actions on the secondary nodes automatically via remote invocation.

If a node unexpectedly becomes inaccessible via ssh, it will be abandoned by adop, and the appropriate further actions taken. Consider a scenario where the adop phase=prepare command is run in a system with ten application tier nodes. The command is successful on nine nodes, but fails on the tenth. In such a case, adop will identify the services enabled on nodes 1-9. If they are sufficient for Oracle E-Business Suite to continue to run normally, adop will mark node 10 as abandoned and then proceed with its patching actions. If they are not sufficient, adop will proceed no further.

Online Patching Cycle Steps - Apply Phase

This section describes the principles of adop operation in the apply phase, followed by the steps you take to run this phase.

Principles

In the apply phase, adop applies the specified patches to the system. In an online patching cycle, patches are applied to the patch edition of the database and file system.

Steps

In this phase, you will apply the patches that you designated for inclusion in this patching cycle. You can apply as many patches as you want per patching cycle. By default, a list of patches is applied one at a time, in the specified order. If you specify the merge option "merge=yes", the listed patches will automatically be merged and the resulting merged patch will be applied.

The following example will illustrate the options.

$ adop phase=apply input_file=<inputfile.txt>

This uses the input_file that was mentioned earlier in this section.

An example input_filemight liook like this:

workers=<number of workers>

patches=<patch number 1>:<driver file 1>.drv, <patch number 2>:<driver file 2>.drv ...

Reports under the $APPL_TOP/admin/<SID>/out directory can help you identify and diagnose problems that may occur in the online patching cycle. These reports list the proposed changes to database objects, both new and modified.

The key files to examine are:

$APPL_TOP/admin/<SID>/out/adzdcmped.out

$APPL_TOP/admin/<SID>/log/u<patch_number>.log

Note: For merged patches, the log file name will be derived from the timestamp when merging was performed.

Using the analytics parameter in apply

If you want to use the analytics parameter (see adop Parameters) with the apply phase, enter the command:

$ adop phase=apply analytics=yes

Specifying this option will cause adop to run the following scripts and generate the associated output files (reports):

ADZDCMPED.sql - This script is used to display the differences between the run and patch editions, including new and changed objects. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>/adzdcmped.out.

ADZDSHOWED.sql - This script is used to display the editions in the system. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowed.out.

ADZDSHOWOBJS.sql - This script is used to display the summary of editioned objects per edition. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowobjs.out

ADZDSHOWSM.sql - This script is used to display the status report for the seed data manager. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowsm.out

Note: The analytics parameter should only be used when required, because of the extra processing needed.

Online Patching Cycle Steps - Finalize Phase

The finalize phase is used to perform any remaining processing that is needed to ensure the system is ready for the fastest possible cutover.

The key actions of the finalize phase are:

Pre-compute DDL that needs to be run at cutover.

Compile all invalid objects.

Validate that the system is ready for cutover.

If finalize_mode=full, compute statistics for key data dictionary tables for improved performance.

Run the finalize phase as follows:

     $ adop phase=finalize

Online Patching Cycle Steps - Cutover Phase

This section describes the principles of adop operation in the cutover phase, followed by the manual steps you can optionally execute to run this phase.

Important: No users should remain on the system during cutover, as there will be a short downtime period while the application tier services are restarted. Also, any third-party processes connected to the old run edition of the database should be shut down, or they will be terminated automatically. If desired, you can defer running cutover until a time which will cause minimal disruption to users.

Principles

The key actions performed in the cutover phase are:

Shut down internal concurrent manager: The adop utility signals the internal concurrent manager to shut down, but will wait for any existing concurrent requests to finish before it proceeds with cutover actions. The system is still available to users during this waiting period.

If you do not wish to wait indefinitely for concurrent requests to finish, specify the option cm_wait=<maximum_minutes_to_wait> with a number of minutes that reflects your operational needs.

When deciding whether to use this option, Oracle recommends:

On production systems, do not specify cm_wait, but monitor progress of concurrent tasks and take manual action on them if needed.

On non-production systems, specify cm_wait to limit the waiting time before cutover proceeds.

Shut down application tier services: All application tier services are brought down. During this period, the system is unavailable to users.

Cutover database: Promote patch database edition to become the new run database edition, using adzdpmgr.pl script.

Cutover file system: Promote patch file system to become the new run file system, switching the $FILE_EDITION values in the patch and run enviroments. The current patch APPL_TOP becomes the new run APPL_TOP, and the current run APPL_TOP becomes the new patch APPL_TOP.

Terminate old database sessions: Terminate any database connections to the old run edition of the database.

Start application tier services: Application tier services are restarted, on the new run edition. The system is now available again to users.

Note: The adop utility invokes the TXK script txkADOPCutOverPhaseCtrlScript.pl to perform tasks 1, 2, 3, 5, and 6. Task 4 is performed by AutoConfig.

Before running the cutover command, ensure you are ready to commit to application of the selected patches. Once cutover is complete, it is not possible to revert to the previous edition.

Note: Cutover will take longer if it has to wait for long-running concurrent processes to complete. In such a case, you can expect to see an informational message of the form:

[STATEMENT] [END 2013/10/28 23:47:16] Waiting for ICM to go down

If you do not want to wait for in-progress concurrent requests to finish normally, you can terminate the internal concurrent manager by executing the adcmctl.sh abort command from a different shell.

In most cases (but see below for the important exception of analytics), you then proceed to execute cutover with the command:

$ adop phase=cutover

This will promote the patch edition to be the new run edition, as well as switching the patch and run labels on the file systems (and thereby, as noted above, changing the patch file system to be the new run file system and the run file system to be the new patch file system).

Important: In the event of problems with the cutover phase, refer to My Oracle Support Knowledge Document 1584097.1, Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover.

Deferring Application Tier Restart at Cutover

In some cases, you may need to perform additional manual steps after cutover but before restarting the application tier services. If this is the case, you can supply an additional parameter to the cutover command that causes the application services to remain shut down:

$ adop phase=cutover mtrestart=no

With this parameter, cutover will complete without restarting the application tier services. You can perform any additional steps that require the services to be shut down, and then start the application tier services manually using the adstrtal.sh script.

You must then also run the steps in the following section, Patching the Database Tier:

Patching the Database Tier

These steps are performed post-cutover.

On the application tier, as the applmgr user:

Change directory to the run file system $APPL_TOP and source your environment file.

Run the following command:

$ perl <AD_TOP>/bin/admkappsutil.pl

This will create the appsutil.zip file in <INST_TOP>/admin/out.

On the database tier, as the oracle user:

Copy or ftp the appsutil.zip file to the RDBMS_ORACLE_HOME, then run the following commands:

$ cd <RDBMS_ORACLE_HOME>

$ unzip -o appsutil.zip

Run AutoConfig on the database tier.

Run AutoConfig on the run file system of each application tier node.

Start the application tier services.

JAR Files and Cutover

In an online patching cycle, the requisite JAR files are initially stored in the $APPL_TOP/admin/<SID>/out directory, and then uploaded into the database during the cutover phase. Therefore, the out directory must not be deleted at least until cutover is complete.

Online Patching Cycle Steps - Cleanup Phase

This section describes the principles of adop operation in the cleanup phase, followed by the steps performed in this phase.

Important: If you fail to run the cleanup phase explicitly, it will be run automatically on the next prepare cycle, but this will cause a delay in starting your next online patching cycle.

Principles

Various actions are performed during cleanup, including dropping (removing) obsolete:

Crossedition triggers

Seed data

Editioned code objects (covered objects)

Indexes

Columns

Editions

Steps

Cleanup is performed with the command:

$ adop phase=cleanup

The adop parameter cleanup_mode provides control of cleanup processing:

cleanup_mode=quick - Performs minimum cleanup, including removal of obsolete crossedition triggers and seed data.

cleanup_mode=standard - Does the same as quick mode, and also drops (removes) obsolete editioned code objects (covered objects). This is the default mode , so does not need to be specified.

cleanup_mode=full - Performs maximum cleanup, which drops all obsolete code and data from earlier editions

Choosing the Cleanup Mode

Generally, you can use standard mode (the default). You might want to use the other modes in the following circumstances:

Use quick cleanup when you need to start the next patching cycle as soon as possible. For example, if you want to start a new patching cycle right away, but have not yet run cleanup from the previous patching cycle, you can use quick cleanup mode to complete the essential cleanup tasks as fast as possible.

Use full cleanup when you want to recover the maximum amount of space in the database. If you have run a large number of patching cycles, or applied a very large patch such as a rollup, significant space may be consumed by obsolete table columns and recovered by running a full cleanup. A full cleanup should only be performed when there is no immediate need to start a new patching cycle.

Note: Prior to AD-TXK Delta 8, if a table is patched and the definition of an existing column changed, the original column is marked as unused on a subsequent full cleanup. From AD-TXK Delta 8, a lower-versioned column is not marked as unused. If an abort is carried out, a higher-versioned column is marked as unused, as such a newly-added column is not in use.

From

Using the analytics parameter in cleanup

If you want to use the analytics parameter (see adop Parameters) with the cleanup phase, enter the command:

$ adop phase=cleanup analytics=yes

Specifying this option will cause adop to run the following script and generate the associated output file (report):

ADZDCLEANUPRP.sql - This script is used to display the display the cleanup status. The output file location is: $NE_BASE/EBSapps/log/adop/<adop_sessionID>/<cleanup_directory>/<context_name>/adzdcleanuprp.out.

Note: The analytics parameter should only be used when required, because of the extra processing needed.

Online Patching Cycle Steps - Abort Phase

If for some reason either the prepare or apply phase failed or gave problems, you can abort the patching cycle at either of these points by running a special phase with the command:

$ adop phase=abort

In the abort phase, adop does the following:

Confirms that there is an in-progress online patching cycle, so the abort call is therefore valid.

Checks for the existence of a patch edition and drops one if it exists.

Cancels the ADZDPATCH concurrent program, if it is running.

Deletes the rows inserted for the pending session ID from the ad_adop_sessions and ad_adop_session_patches tables.

Be aware of the following important points:

After running abort, a full cleanup must be performed. The cleanup command is:adop phase=cleanup cleanup_mode=full). This will remove any columns that were added by the patch but are no longer needed because of the abort. If they are not removed, they may cause problems in a later patching cycle.

Alternatively, you can run a combined command to abort the patching cycle and perform a full cleanup:

$ adop phase=abort,cleanup cleanup_mode=full

If any attempt was made to apply patches to the patch edition, after abort you must run the fs_clone phase (adop phase=fs_clone) to recreate the patch file system.

This section covers a variety of tasks that may apply either to individual online patching operations, or to your system setup as a whole. Diagnostic, troubleshooting, and reporting features are also described.

Manual Post-Patch Installation Tasks

Traditionally, some patches have associated post-patch installation tasks, including recompilation of invalid packages, regenerating JAR files, and running AutoConfig. In an online patching environment such as Release 12.2 such tasks will normally be performed automatically in the apply phase.

If a post-installation patch step mentions any tasks that need to be performed explicitly, where they are run from depends on the type of patching:

In a normal online patching cycle, the steps should be executed from the patch file system after the apply phase.

If the patch is being applied in hotpatch mode or downtime mode, the steps should be executed from the run file system after the apply phase.

Dropping Old Editions With the actualize_all Phase

As each online patching cycle is completed, the database will accumulate an additional old database edition. If the number of these grows too large, system performance will start to be affected. When the number of old database editions reaches 25 or more, you should consider dropping all old database editions by running the adop actualize_all phase and then performing a full cleanup.

Important: This procedure will take a large amount of time (significantly longer than a normal patching cycle), and should only be performed when there is no immediate need to start a new patching cycle.

Before starting, you should ensure that the system has the recommended database patches and latest AD-TXK code level installed.

To proceed, run the following commands in the order shown:

$ adop phase=prepare

$ adop phase=actualize_all

$ adop phase=finalize finalize_mode=full

$ adop phase=cutover

$ adop phase=cleanup cleanup_mode=full

You have now completed removal of the old database editions.

Context Variable Requirements Across Nodes and File Systems

The following context variables must have same value across all nodes, and also across the run and patch file systems:

s_dbport

s_java_object_cache_port

s_cmanport

s_apps_jdbc_connect_descriptor

The following context variables must have same value across the run and patch file systems of a given node:

s_http_listen_parameter

s_https_listen_parameter

s_rpcport

s_webssl_port

s_webport

s_active_webport

s_fnd_cache_port_range

s_external_url

s_login_page

s_endUserMonitoringURL

s_dbport

s_java_object_cache_port

s_cmanport

s_apps_jdbc_connect_descriptor

Note: Following the application of one-off Patch 18942757:R12.TXK.C, the following MWA-related ports will have the same values on the run and patch file systems:

s_mwaPortNo

s_mwaTelnetPortNo

s_mwaDispatcherPort

Configuration Management and Patching

The following guidelines apply to making configuration changes to Oracle E-Business Suite in the context of online patching. They particularly apply to the technology stack and application components that reside in the file system.

Note: For specific instructions on how to patch technology stack components, refer to My Oracle Support Knowledge Document 1355068.1, Oracle E-Business Suite 12.2 Patching Technology Components Guide.

The two basic scenarios are online and offline configuration changes. Each will be considered in turn.

Online configuration changes are performed within the context of an online patching cycle. This is the recommended strategy.

First, you prepare your system by running the adop phase=prepare command. You then make the desired configuration changes to the patch file system. They may include:

Oracle WebLogic Server configuration changes

HTTP Server configuration changes

File system changes performed by the AD utilities

After making the configuration changes, you must run the command adop phase=cutover to promote them.

You must also run the command adop phase=fs_clone to propagate the configuration changes to the secondary file system.

Offline configuration changes are applied directly to the run file system, outside an online patching cycle.You can use the adop -status command to verify that no patching cycle is currently active. After making the desired configuration changes, you must explicitly run the adop phase=fs_clone command to propagate the changes to the patch file system.

Important: This offline scenario will require a period of downtime for users.

Support for Single File System Development Environments

A normal Release 12.2 online patching environment requires two application tier file systems, one for the run edition and another for the patch edition. This dual file system architecture is fundamental to patching of Oracle E-Business Suite Release 12.2, and is necessary both for production environments and test environments that are intended to be representative of production. This feature makes it possible to create a development environment with a single file system, where custom code can be built and tested. The code should then always be tested in a standard dual file system test environment before being applied to production.

You can set up a single file system development environment by installing Oracle E-Business Suite Release 12.2 in the normal way, and then deleting the $PATCH_BASE directory with the command:

$ rm -rf $PATCH_BASE

A limited set of adop phases and modes are available to support patching of a single file system development environment. These are:

apply phase in downtime mode

cleanup phase

Specification of any other phase or mode will cause adop to exit with an error.

The following important restrictions apply to using a single file system environment:

You can only use a single file system environment for development purposes.

A single file system environment must have a single-node application tier: multi-node application tiers are not supported.

A single file system environment can only be created by conversion from an existing dual file system environment: you cannot directly create a single file system environment via Rapid Install or cloning.

You cannot use online patching in a single file system environment.

You cannot convert a single file system environment back to using a dual file system.

You cannot clone from a single file system environment.

Restrictions on Applying Patches in hotpatch Mode

Applying patches in hotpatch mode is only supported for use with patches that have been designed and tested to be applied in this way. This is because hotpatch mode applies changes to the run edition while this edition is in active use, which may result in one or more of the following issues for patches not designed to be applied as hotpatches:

Runtime transactions may fail due to invalid objects.

Runtime transactions may fail due to loss of PL/SQL package state.

Application code and database objects may be temporarily inconsistent.

Seed data may change, and may be temporarily inconsistent.

Tables that are patched will be temporarily inconsistent.

Code and data cached in application tier server memory may be inconsistent with changes made by the hotpatch.

Runtime processing may hold long-term locks on code or data, leading to execution failures in the hotpatch.

Also, when you use hotpatch mode to apply a patch that contains a downloadable resource (such as a Forms-related client JAR file), that resource will only become available after you restart the Oracle WebLogic Server Managed Servers. Until you perform the restart, you may receive an error on the client or server about the integrity of the resource file.

Therefore, you should not attempt to apply a patch in hotpatch mode unless explicitly directed to do so by the patch readme.

Emergency Application of Patches

In an emergency, you can deploy patches directly to the run file system and run edition, with the following important restrictions being strictly adhered to:

The patch application should be performed using adop downtime mode.

No patching cycle can be in progress. Even though adop will prevent you from applying patches in downtime mode while a patching cycle is under way, it is important to keep this in mind for planning purposes.

To prevent massive invalidation in the database, code objects deployed to the database should not include parent objects that could cause extensive invalidation on dependent objects. If this does happen, a significant amount of time will be required for code recompilation.

Directories that contain code deployed to the application tier during emergency patch deployment must be registered with the custom synchronization driver, to ensure successful automatic file system synchronization by the adop synchronization process.

Deployment of grants to base objects results in code actualization and recompilation, and the subsequent risk of code invalidation. For guidance on how to perform such grant operations safely, refer to My Oracle Support Knowledge Document:1987947.1, Granting Privileges On An Object May Cause Invalidations.

Warning: Most Oracle E-Business suite patches are not tested in downtime or hotpatch mode. It is therefore important that this type of deployment is only used in an emergency, and not incorporated into standard maintenance practices.

Considerations When Re-Applying Patches

As mentioned in the "adop Options" section of Chapter 2, if you try to apply a patch that has already been applied, and you do not specify options=forceapply, adop will display an error such as:

[WARNING] Skipping the application of patch 14125999_AR since it has been already applied

[WARNING] Hint: Patches can be applied again by specifying options=forceapply when invoking adop

There are two more scenarios that may occur in this type of situation:

When a failed patch session is restarted with abandon=no, restart=yes, the patches applied in current adop session will not be applied even if options=forceapply is specified. For example, you run the command adop phase=apply options=forceapply patches=1111,2222 , and application of patch 1111 is successful but patch 2222 fails. After correcting the problem, you try to rerun adop with the command adop phase=apply options=forceapply patches=1111,2222 abandon=no, restart=yes. In this example, patch 1111 would be skipped as it had successfully been applied, and application of patch 2222 would resume. If you wanted to apply patch 1111 again, you would need to specify abandon=yes, restart=no.

If you apply multiple patches with merge=yes , and you do not specify options=forceapply, the patches will be applied only if at least one of the patches has not been successfully applied before.

Note: This check will be performed for AD and non-AD patches separately, as adop applies these two categories of patch in different sessions.

Using the Database Recycle Bin With Online Patching

Section 6 of My Oracle Support Knowledge Document 396009.1, Database Initialization Parameters for Oracle E-Business Suite Release 12, states:

#########

#

# recyclebin parameter

#

# The database recyclebin must be turned off to allow

# the cleanup phase of the online patching cycle to be

# performed without having to connect as SYS.

#

# This feature may still be used at other times.

#

#########

recyclebin=off

You can use the database recycling bin by following these steps:

Set the 'recyclebin' database initialization parameter to 'on'

Run the command adop phase=prepare

Purge the dba_recyclebin table

Run the command adop phase=actualize_all

Run the command adop phase=finalize,cutover

Run the command adop phase=cleanup cleanup_mode=full

For more information about the database recycle bin, refer to the 'Using Flashback Drop and Managing the Recycle Bin' section in Oracle Database Administrator's Guide 11g Release 2 (11.2).

Customizing Patch File System Backup Count

A new AutoConfig context variable, s_fs_backup_count, is used to specify the 'Patch File System Backup Count'. This variable denotes the number of backups of the patch file system that are to be preserved by adop. The variable is used during the fs_clone phase, where the existing patch file system is backed up before it is recreated from the run file system.

Valid values for the s_fs_backup_count variable are 0-9. A value of 0 (the default) will not preserve any patch file system backups. A value of 1 will preserve the latest patch file system backup, a value of 2 will preserve the latest two backups, and so on. You can normally leave the setting at the default value of 0 (no backup), as the patch edition file system can be fully recreated from the run edition file system by fs_clone.

Note: For more information about AutoConfig, see the "Technical Configuration" chapter of the Oracle E-Business Suite Setup Guide.

Requirements When Running Oracle HTTP Server on a Privileged Port

On a UNIX system, the TCP/IP port numbers below 1024 are special in that only processes with root privileges are allowed to listen on those ports. If you have configured Oracle HTTP Server to run on a privileged port, you must perform the following additional steps when running an online patching cycle. These steps are required for both SSL and non-SSL privileged ports.

Before running the prepare phase or the fs_clone phase, you must run the following commands as the root user on both the run file system and the patch file system:

chown root $FMW_HOME/webtier/ohs/bin/.apachectl

chmod 6750 $FMW_HOME/webtier/ohs/bin/.apachectl

After running the prepare phase or the fs_clone phase, you must run the following commands as the root user on the patch file system:

chown root $FMW_HOME/webtier/ohs/bin/.apachectl

chmod 6750 $FMW_HOME/webtier/ohs/bin/.apachectl

For more information, see: Starting Oracle HTTP Server on a Privileged Port, Oracle Fusion Middleware Administrator's Guide for Oracle HTTP Server and Running Oracle HTTP Server on a Privileged Port, Oracle E-Business Suite Setup Guide.

Integrating Your Custom Tasks Into the Online Patching Cycle

You may have business-specific tasks specific that need to be performed before, during or after a patching cycle. Support for such tasks is currently provided by callout points at the begining and end of the cutover phase of the online patching cycle. This support will be extended in future releases of Oracle E-Business Suite.