Known Issues And Bugs Using the Scheduler

Oracle Database - Enterprise Edition - Known Issues And Bugs Using the Scheduler

Oracle Database - Enterprise Edition - Known Issues And Bugs Using the Scheduler

Titleimage

Posted by Patrick Hamou on 2017:09:12 16:56:07

Oracle Database - Enterprise Edition - Version 9.0.1.0 to 11.2.0.3 [Release 9.0.1 to 11.2]

Information in this document applies to any platform.

PURPOSE

The purpose of this document is to assist with issues using the Oracle Scheduler system. 
With the usage of the Oracle Scheduler system, a very flexible application specific environment can arise, which makes it difficult to analyze unexpected behaviour.
Therefore, it is important to break down the issue into the different aspects for a detailed analysis, to finally find the cause of an unexpected behaviour.
There are many notes for problems with the Scheduler system, however, it is not always easy to find an immediate solution. This guide is an attempt to list some tips and tricks, relevant notes, known bugs and issues related to Scheduler usage and try to provide the first steps for analyzing, and listing the relevant notes dependent on the database version and specific scheduler part.

NOTE:
Any feedback and comments are greatly appreciated. This note will be continually reviewed and updated, adding new tricks, notes, bugs and additional information in an attempt to keep it up to date.

TROUBLESHOOTING STEPS

--- Symptoms ---

1. Generic Issues setting up Scheduler Jobs and the Job Definition

Depending on the expectation to realize a more or less complex scheduler system it might sometimes be difficult to set up the scheduler as needed. Once some unexpected behaviour or errors are encountered, it is often useful to go back, step by step, and try a more simplified testcode.

Suggestions:

Check for any errors receiving when executing procedures from the DBMS_SCHEDULER package.

Use simplified testcode to build a simple scheduler job and try to see if the issue reproduces.
Help for setting up sample code can be found in the following notes:

Note 807159.1 - Sample Code and Select Statements to build Simple Testcases for Scheduler / Jobs
Note 727499.1 - Sample how to create a job running chains
Note 1431664.1 - How to build up an event based job - an example
Note 1300444.1 - How to Create a Event Based Chained Job
Note 470764.1 - How to Control Resources used by a Statistics Job using the Resource Manager
Note 1300313.1 - How to use an own Maintenance Window for the Statistics Collection in 11g
Note 1074141.1 - How to install and setup the Scheduler Agent (schagent) so can run Remote External Jobs

2. Scheduled Job doesn't run automatically or using run_job

Under some circumstances it may happen that some jobs don't run anymore at their scheduled time;
The job definitions haven't been changed.
No scheduler attributes have been changed.

Known Notes and Bugs:

Bug 9197805 - Infrequent delay in execution DBMS_SCHEDULER or DBMS_JOB jobs (Doc ID 9197805.8)

==> fixed in 12.1.0.1, 11.2.0.2

Suggestions:

Check for the CJQ process and possible job slaves on the OS level.

Verify if any database patches have been installed recently.

Oracle version 11.2:
The usage of the parameter job_queue_processes has been changed.
While in earlier versions (<11.2) there was no real supported method to stop the cjq process and it's job slaves, it is now possible from 11.2 onwards to stop these processes by setting job_queue_processes = 0. 
Therefore, the actual setting of job_queue_processes can prevent the automatic run of jobs.
If a job won't run automatically on version 11.2, please check the current setting of this parameter using :

SQL> show parameter job_queue_processes

and make sure to set this value higher than zero.

Oracle version <= 11.2:
Check if the CJQ process was somehow disabled using :

SQL> select * from dba_scheduler_global_attribute;

==> check the value of the following for 

MAX_JOB_SLAVE_PROCESSES: Null per default, but once set then check if accurate to a sufficient value

SCHEDULER_DISABLED : Often used in version < 11.2 to stop the cjq.
As mentionned, in Oracle version 11.2, the parameter job_queue_processes has been changed to implement this.
CURRENT_OPEN_WINDOW:  Check if a window is open, and that this is the correct one depending on the actual time

Check the columns ADDITIONAL_INFO and ERROR# in the view DBA_SCHEDULER_JOB_RUN_DETAILS

It might be the case that the job started but then failed. These columns will give a hint regarding the reason for it failing.

Check the alert.log, the job slave - and cjq trace files.
Have a look at the alert.log for any messages and research possible errors which may be shown in the trace files.

Check if a manual run of the job using dbms_scheduler.run_job executes without problems. 
If this fails then check what kind of JOB_TYPE the job is based on.
For jobs that you have not created, check the view DBA_SCHEDULER_JOBS and the value of the columns JOB_TYPE and JOB_ACTION, for example:

PLSQL_BLOCK:
  Execute this block with sqlplus and check for a possible error.
STORED_PROCEDURE :
  Execute the procedure directly in sqlplus and test if this executes without any problems. 
CHAIN :
  For chains it is important to know that chains are built with chain steps running specific programs.
  Please go to the section for chains.
EXTERNAL:
  There are a lot of possibilities why external jobs wouldn't run.
  Please go to the section for external jobs.

Check if a non-default value for the scheduler attribute job_weight is used :
In a loaded system with limited job slave processes, the job_weight will prefer to run jobs with a high priority. 
See Note 1310649.1 - Scheduler Jobs Refuse To Restart At The Scheduled Time

Does the problem happen after an OS clock change (winter <=> summer time) ?
In this case a DST issue might have happened. Please see the section about DST issues.

Was the database cloned?
Check for valid services
Note 1267379.1 - Oracle Scheduler doesn't execute any job in a cloned database

Does the job not run automatically after a database upgrade?
In this case it might be an issue with the Scheduler Windows in which jobs can be scheduled.
Note 731678.1 - Scheduled jobs do not run as expected after 10.2 upgrades

 Statistics not running:
Check for the correct open window:
Note 1320246.1 Why Auto Optimizer Statistics Collection May Appear to be "Stuck"?

3. Scheduled Job doesn't run automatically, but a manual run will succeed

Suggestions:  

Database in restricted mode: Check if database is in restricted mode. If yes, then issue ALTER SYSTEM DISABLE RESTRICTED SESSION; 
Check if jobs are picked up automatically after that.

Check the alert.log, job slave trace files and cjq.. trace file for possible errors.

Check on the OS level for the existence of CJQ and job slaves

Check the scheduler log views (in particular the additional_info, next_run_date and last_run_date columns)

When the next_run_date is in the past , then try:

alter system set events '27402 trace name context forever,  level 65535';';
execute DBMS_SCHEDULER.run_job (job_name => 'Test_Manual_Job', use_current_session => FALSE);
exec dbms_lock.sleep(300);
alter system set events '27402 trace name context off'; 

==> upload the generated trace files to Oracle Support

Check the load on the system. Are resources available to run the jobs? An AWR report or systemstate dump might help.

Version 11.2.0.3:
Check for any settings of event 10195. When this is set to a non zero value then Bug 13062518 may be relevant:

Known Notes and Bugs:

Note 461304.1 How to activate tracing for the scheduler

Bug 13062518: DBMS_JOB IS NOT PICKED UP AUTOMATICALLY WHEN SUBMIT FROM APPLICATION
==> Base Bug 10103086 , fixed in 12.1.0.1, 11.2.0.3

Bug 13324090 - CJQ PROCESS CONSUMING HIGH CPU
Suspended, Req'd Info not Avail

Bug 13906790 - SYS$SERVICE_METRICS IS GROWING ON 2NODE RAC
Suspended, Req'd Info not Avail

Bug 14015067: MESSAGES ON SYS$SERVICE_METRICS ARE INCREASING CAUSING DELAY ON DBMS_SCHEDULER
dup of Bug 13855490 - CJQ CONSUME PGA MEMORY IN RAC 11.2
==> fixed in 12.1.0.1, 11.2.0.4, 11.2.0.3.7 PSU

 

4. Hanging Jobs cannot be stopped or dropped

Dropping using the force option will result in an error like:
  ORA-27475: "username.jobname" must be a job 
or
  ORA-27478: job "username.jobname" is active

Even a restart of the database sometimes won't prevent the restart of the job.
DBA_SCHEDULER_RUNNING_JOBS will show an entry for the job.


Possible causes:

* stop_job will only work on background job runs, not on foreground job runs 
   using run_job(use_current_session=>true). So the foreground run_job run will 
   continue unaffected by the stop_job. 
* The job slave process of this job might have been killed, for example with a hard kill on the OS level.
  This will lead to a situation where the job is kept in a running state, but no slave process is available.
   There are some bugs listed on this situation.


The following needs to be checked:

1. Check the job attribute restartable, and set it to no

2. Use the following steps in the same order:
    - Disable the job using force=>true
    - Stop the job using force=>true
    - Drop the job using force=>true

3. As a last resort (this will affect the database and/or other jobs): 
    3a. Restart the database when possible or 
    3b. Disable the cjq process and its processes with:
       for version 11.2 and higher :
       - set job_queue_processes = 0
       - wait and check on the OS level that the processes have ended
       - disable/drop the job
       for versions < 11.2:
       - set job_queue_processes = 0
       - disable the cjq process with
         exec dbms_scheduler.set_scheduler_attribute('scheduler_disabled','true');
       - wait and check on the OS level that the processes have ended
       - disable/drop the job
       - enable the cjq with 
         exec dbms_scheduler.set_scheduler_attribute('scheduler_disabled','false');
         and set job_queue_processes back to the original value

4. Gather the following for the job which is hanging in run state and having a SID(connect to thread/instance in question):

    select pid from v$process, v$session where addr=paddr and sid=&sid;
    oradebug setorapid &pid
    oradebug event immediate trace name errorstack level 3;
    oradebug event 10046 trace name context forever, level 12;
    exec dbms_lock.sleep(300);
    oradebug event 10046 trace name context off;
    oradebug tracefile_name

     ==> upload these trace files to Oracle Support
   

Known Notes and Bugs:

Note 1290836.1 - Can't Delete Scheduled Job : ORA-27475 or ORA-27478

Bug 7606518 - ORA-23766 HIT AT STOP_JOB / ORA-27478 HIT AT DROP_JOB
=> Fixed in 11.2.0.1
Unpublished Bug 7258928 - SCHEDULER JOBS THAT ARE HANGING, CANNOT BE STOPPED
   => Fixed in  11.2.0.1, 11.1.0.7.2 PSU
Unpublished Bug 10363801 - JOB RECOVERY AFTER DATABASE BOUNCE FAILS
   ==> fixed in 12.1.0.1 ,11.2.0.3
Bug 11696169-  SCHEDULER JOB CANNOT BE STOPPED / DROPPED ON EXADATA MACHINE
   ==> fixed in 12.1.0.1 ,11.2.0.3

 

5. DST issues: Scheduled Jobs don't run after a clock change

In some cases a clock change related to DST will prevent the jobs from running automatically.
The cause is mainly due to incorrect settings in the timezones which are not DST compliant.

The SCHEDULER can have two different timezones:
a. one is the timezone for the SESSION creating the SCHEDULED JOB
b. one is the TZ as read by the ORACLE BINARY when STARTED (this can be different for the client, the database, the listener, the EM Agent, etc. as each may have their own ENV settings that may or may not match anything else.)

First Analysis can be done with:

Check the start_date and it's DST relation to the job that is not running and the actual Timezone settings

Check  the following output for timezone settings:

-- start a sqlplus session

connect / as sysdba
spool DSToutput.txt
alter session set  NLS_TIMESTAMP_TZ_FORMAT ='DD/MM/YYYY HH24:MI:SS TZR TZD';
select dbms_scheduler.stime from dual;
select sessiontimezone from dual;
select DBMS_SCHEDULER.GET_SYS_TIME_ZONE_NAME from DUAL;
select name, value$ from props$ where name = 'DBTIMEZONE';
select distinct tzname from v$timezone_names;

! echo $TZ -- for unix systems

-- check of runtimes of job:

set pagesize 9999
set markup html on
spool dba_scheduler_job_log.html
select * from dba_scheduler_job_log where log_date > systimestamp -3 and job_name=upper('&name_of_job') order by log_date desc;
spool off
set markup html off

-- ==> upload dba_scheduler_job_log.html

set pagesize 9999
set markup html on
spool dba_scheduler_job_run_details.html
select * from dba_scheduler_job_run_details where log_date > systimestamp -3 and job_name=upper('&name_of_job') order by log_date desc;
spool off
set markup html off

-- ==> upload dba_scheduler_job_run_details.html

set pagesize 9999
set markup html on
spool dba_scheduler_jobs.html
select * from dba_scheduler_jobs where
job_name=upper('&name_of_job') order by start_date desc;
spool off
set markup html off

-- ==> upload dba_scheduler_jobs.html

set pagesize 9999
set markup html on
spool dba_scheduler_running_job.html
select * from DBA_SCHEDULER_RUNNING_JOBS;
select * from dba_scheduler_global_attribute;
spool off
set markup html off

-- ==> upload dba_scheduler_running_job.html

 

NOTE: 
On Windows platforms:
Make sure the clock's time zone setting is correct. Go to "Control Panel" and choose "Date and Time Properties" . Then click on "Time Zone". When the specified time zone has a DST period, the "Automatically adjust clock for daylight savings changes" checkbox needs to be enabled. Make sure the clock is set to the correct time of day for the specified time zone.

What Timezone was used when the job was created?
Was the session altered that CREATED the JOB for example to the correct timezone?
For example:
ALTER SESSION SET TIME_ZONE = 'Europe/London';

Please be aware that the session Timezone used during the creation of the JOB sets rules for the JOB. So it might be necessary to set the time_zone first to an expected value and then recreate the scheduler object/job. An alternative would be to explicitly set the time zone using the procedure set_attribute.

For more information refer to:

Note 340512.1 - Timestamps & time zones - Frequently Asked Questions
Note 467722.1 - DBMS_SCHEDULER or DBMS_JOB And DST / Timezones Explained
Note 1077446.1 - All Scheduled Jobs Run One Hour Before or Ahead of Their Scheduled Times After Daylight Saving Time 
Note 357056.1 - Impact of changes to daylight saving time (DST) rules on the Oracle database 
Note 402614.1 - Time Zone Data in the Data Dictionary (SYS) and the Effects of a Time Zone File Update

Known bugs:

Bug 6336914 - DBMS_SCHEDULER.GET_SYS_TIME_ZONE_NAME returns NULL for GMT time zone
==> Fixed in 11.2.0.1, 11.1.0.7, 10.2.0.5
Bug 12325266 - JOBS FAIL WITH ORA-01878 AFTER SWITCH TO DST
dup of Bug 11900845 
BUG:11900845 - AFTER DST CHANGE LOT OF DBMS_JOBS STARTED FAILING WITH ORA-1878
==> fixed in 12.1.0.1, 11.2.0.3

6. Issues related to the Job Log Scheduler views

The Scheduler provides the ability to monitor the execution of jobs or opening/closing of windows and additional information in the following scheduler log views: 
DBA_SCHEDULER_JOB_RUN_DETAILS
DBA_SCHEDULER_JOB_LOG
DBA_SCHEDULER_WINDOW_LOG

These views will provide important information about the actual states of scheduler objects.

In some cases it happens that the log information has been cleared out. This can be done manually be executing the package dbms_scheduler.purge_log or by setting an attribute LOG_HISTORY which implies that the purge will run after a specific amount of time.
The attribute can be set either at the global level or for a job class. 
Please note that every job has an associated Job Class. If not explicitly specified then the job is located in the DEFAULT_JOB_CLASS.
The default value job LOG_HISTORY is set to 30, so 30 days after the job ran, the information of it's run will be cleared out.

 

In other situations, it can sometimes happen that these Job Log Scheduler views grow to an unexpected high volume and this can lead to a situation where jobs are delayed.
AWR outputs show for example:

enq: JS - queue lock
enq: JS - q mem clnup lck
 

as top events.

This can happen when

- the log views haven't been purged for a long time (check log_history , or check if the purge job has been disabled) 
- when a large amount of jobs are created per day/hour filling up the log tables very fast to a high volume

Suggestions:

- Check the actual value of 'log_history' by selecting the global attributes and the setting for this attribute within a Job Class:

connect / as sysdba
select  ATTRIBUTE_NAME, VALUE 
from   DBA_SCHEDULER_GLOBAL_ATTRIBUTE;
select  * 
from   dba_scheduler_job_classes 
where job_class_name in (select job_class 
                                       from   dba_scheduler_jobs 
                                       where job_name=upper('&jobname'));

select count(*) from DBA_SCHEDULER_JOB_RUN_DETAILS;
select count(*) from DBA_SCHEDULER_JOB_LOG;
select JOB_NAME, STATE from DBA_SCHEDULER_JOBS where job_name='PURGE_LOG';

Note: The setting of log_history for a Job Class will override the setting for the global attribute.
For example, for the Job Class 'A' the log_history is set to a value of 10, but within the view DBA_SCHEDULER_GLOBAL_ATTRIBUTE the default value of 30 is shown. The Log information for all jobs within the Jobclass 'A' will be purged out after 10 days.

 

Known Notes and Bugs:

 Jobs not Automatically start, High waits on enq: JS - queue lock (Doc ID 1928894.1)
Note 1270854.1 - PURGE_LOG Job will purge all entries and ignore the setting of LOG_HISTORY
Note 749440.1 - Dbms_scheduler.Purge Not Removing Entries from dba_scheduler_job_run_details
Bug 9930455 - PURGE_LOG JOB WILL PURGE ALL ENTRIES OF SOME JOBS AND DISREGARD THE SETTING OF L
==> fixed in 12.1.0.1, 11.2.0.3
Bug 5687223 - DBMS_SCHEDULER.PURGE_LOG() DOES NOT CLEAR THE ENTRIES IN:SCHEDULER$_EVENT_LOG
 closed, Could Not Reproduce

 

7. External Job Issues

Issues related to the usage of external jobs also need to be analyzed at the OS level. In most cases, the reason the external job doesn't run is related to incorrect permissions/settings on the scheduler files/executables.

Suggestions for UNIX Platforms:

Check that the following permissions have been set:

Version 10.2.:
- rdbms/admin/externaljob.ora : 644 (rw-r--r--) and owned by root:oraclegroup 
   Jobs without credentials and not owned by SYS run as the user specified in the file.
   (run_user and run_group)
 - bin/extjob : 4750 (-rwsr-x---) and owned by root:oraclegroup 
 - bin/extjobo : 755 (rwxr-xr-x)  and be owned by oracle:oraclegroup

Additionally in version 11g (due to credentials) :
-  bin/jssu : 4750 (-rwsr-x---) and owned by root:oraclegroup
   External jobs which have a credential defined run as the user
   specified in the credential.

Check the oracle user in /etc/group and /etc/passwd

Check if a simple testjob does work. For example use the following:

--  create a shell script /tmp/test.sh on OS with following contents:

#!/usr/bin/ksh
date >> /tmp/tmp.out

-- change permissions and test it on OS level:

$>  chmod 777 test.sh
$>  ./test.sh
$>  cat /tmp/tmp.out

-- create testuser and job in sqlplus:

   connect / as sysdba
   drop user crc cascade;
   grant connect, dba to crc identified by crc;
   connect crc/crc

   begin
   dbms_scheduler.create_job(
                            job_name => 'MYTEST',
                            job_type => 'EXECUTABLE',
                            job_action => '/tmp/test.sh',
                            auto_drop => FALSE,
                            enabled => true,
                            comments => 'Run shell-script');
    end;
   /

--  Check if the job did run successfully in the scheduler log tables.

Use the trace utilities on the OS level to perform OS tracing:
Some possibilities depending on their availability on the OS:

- TRUSS:
   Create a truss file which will show the background process as well:
   (ensure to connect as the Oracle Software owner)

$> truss -feao /tmp/truss.log sqlplus /nolog connect crc/crc
   SQL> exec dbms_scheduler.run_job(job_name=>'MYTEST', use_current_session=>TRUE);
   SQL> exit

- STRACE example:

 Use two sessions:

 session1: 
 =======
 login to Sqlplus 

 session2:
 =======
 login as user root on the OS level and grep for the Sqlplus pid from session1 :
 For example:
  $>  ps -ef | grep sqlplus
      oracle    8478  8137  0 15:46 pts/9    00:00:00 sqlplus

  $> ps -ef | grep 8478
    oracle    8478  8137  0 15:46 pts/9    00:00:00 sqlplus
    oracle    8479  8478  0 15:46 ?        00:00:00 oracleGSADEVC -   (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
    oracle    8489  7976  0 15:47 pts/7    00:00:00 grep 8478

  ==> perform stracing for pid 8479

  $> strace -fo output1 -p8479

  back to session1:
  =============
   SQL> exec dbms_scheduler.run_job('MY_JOB');
   SQL> exit

Check the generated log file for errors

Check if PAM is correctly installed especially on Solaris and Linux platforms.
Jssu authenticates using PAM, so PAM must be configured correctly.
Check that the "su" to the username without any errors.
If PAM is configured to disallow "su" then the customer must configure PAM to explicitly allow the jssu service. The PAM documentation for the OS release should say how to do this.
See also:
Note 976049.1 - DBMS_SCHEDULER fails with ORA-27369 and STANDARD_ERROR="Launching external job failed: Invalid username or password"

When credentials are used, run jssu manually passing an OS username and password on STDIN
For example, to run "/bin/echo hi there" using jssu using OS user ostestusr with password ostestpwd you can use:
 

jssu
  ostestusr
  ostestpwd
  1
  /bin/echo
  2
  hi
  there

==> The output should look like "  !@#--!@#7#@!--#@! " . Please provide this output to Oracle Support.
 

Suggestions for Windows Platforms:

The OracleJobScheduler Windows Service must be started before external jobs will run. 
  (except for jobs in the SYS schema and jobs with credentials).

The user that the OracleJobScheduler Windows Service runs as must have the "Log on as batch job" Windows privilege.

A batch file (ending in .bat) cannot be called directly by the Scheduler. Instead a cmd.exe must be used and the name of the batch file passed in as an argument. 

For example:

begin
dbms_scheduler.create_job('myjob',
job_action=>'C:\WINDOWS\SYSTEM32\CMD.EXE',
number_of_arguments=>3,
job_type=>'executable', enabled=>false);
dbms_scheduler.set_job_argument_value('myjob',1,'/q');
dbms_scheduler.set_job_argument_value('myjob',2,'/c');
dbms_scheduler.set_job_argument_value('myjob',3,'c:\temp\test.bat');
dbms_scheduler.enable('myjob');
end;
/

For more information refer to:

OTN Forum - Guide to External Jobs on 10g with dbms_scheduler e.g. scripts,batch files

Note 389685.1 - Items to Check when Problems Running Executables through the Scheduler
Note 976049.1 - DBMS_SCHEDULER fails with ORA-27369 and STANDARD_ERROR="Launching external job failed: Invalid username or password"
Note 1300215.1 - ORA-27369: job of type EXECUTABLE failed with exit code: 274667

Note 11814763.8 - Execution of external job via DBMS_SCHEDULER fails with ORA-12012 / ORA-27369
Note 1494843.1 External Job fails with ORA-27369: job of type EXECUTABLE failed with exit code: 255 
Note 961019.1 DBMS_SCHEDULER Extjob Fails With "Login Executable Not Setuid-Root" 
Note 787802.1 DBMS_SCHEDULER fails with ORA-27476 ORA-27475 
Note 277399.1 DBMS_SCHEDULER Fails To Execute Jobs With Program_type EXECUTABLE On HP-UX
Note 1383236.1 DBMS_SCHEDULER fails with ORA-27369: job of type EXECUTABLE failed with exit code: Arg list too long

Known Bugs:

Bug 9689988 - EXTERNAL JOB FAILS WHEN RUN AS SYS USER
Closed, Not a Bug
Bug 8780871 - EXTERNAL JOB CANNOT BE STOPPED 
==>Fixed 12.1.0.1 , 11.2.0.2
Unpublished Bug 3887932 - SCHEDULER: STOP_JOB WITH FORCE ON EXECUTABLE JOBS FAIL
==>Fixed 10.2.0.1
Bug 7497013 - RELINKING EXTJOB RESETS PERMISSIONS AND OWNERSHIP OF $ORACLE_HOME/BIN/EXTJOB
dup of Bug 7361549 - CPUJUL2008 WILL CHANGE PERMISSIONS OF ROOT OWNED EXECUTABLES
only applicable to 10202,10203,10204 and 11106 old CPU patches

Bug 11814763 - Execution of external job via DBMS_SCHEDULER fails with ORA-12012 / ORA-27369
==> fixed in 12.1.0.1, 11.2.0.3

8. Version 11G: Remote External Jobs and the Scheduler Agent

Starting with version 11g, the Scheduler can run external jobs on a remote host.
As a minimum requirement for the remote host, the Scheduler agent must be installed.

Relevant log files to check are:
-  $ORACLE_HOME\scheduler\execution_agent\data\agent.log   (scheduler agent log file)
-  $ORACLE_HOME\scheduler\execution_agent\data\log\   directory containing the stdout for each job 

To trace the Scheduler Agent, set the LOGGING_LEVEL=ALL in the schagent.conf file.

For an example see
Note 1074141.1 - How to install and setup the Scheduler Agent (schagent) so can run Remote External Jobs

Known Notes and Bugs:

Note 1074141.1 - How to install and setup the Scheduler Agent (schagent) so can run Remote External Jobs 
Note 789320.1 - ORA-29024: Certificate validation failure
Note 1360434.1 - 11.2 set up schagent -registgerdatabase error: java.net.SocketException 

Bug 12413355 - REMOTE EXTJOB IN WINDOWS CORES IN SCHEDULER AGENT JSSU.EXE
dup of Bug 12886827 - GET/PUT_FILE FAILURES FOR WINDOWS SERVER
==> fixed in 12.1.0.1, 11.2.0.4. windows only issue

Bug 14038023 - TT12.1TOOLS: JOB SCHEDULER ERROR ORA-06512: AT "SYS.DBMS_ISCHED", LINE 8423
==> ORA-1422 when more than one remote execution agent shares the same port
Unpublished Bug 12886827 - GET/PUT_FILE FAILURES FOR WINDOWS SERVER
==> fixed in 12.1.0.1, 11.2.0.4. windows only issue
Unpublished Bug 6435087 - VARIOUS ISSUES WITH SCHEDULER AGENT
==> Fixed in 11.2.0.1, 11.1.0.7

9. Event based Job Issues

The Scheduler can use this feature to send or recieve messages from a process or an application to indicate that some action or occurrence has been detected. 
For example, the Scheduler enqueues a message onto a default event queue. Applications subscribe to this queue, dequeue event messages, and take the appropriate action.
It can also be possible so that when an application enqueues a message onto a queue the Scheduler will be notified to start a job. Thus the sender or the receiver can be the Scheduler.

This mechanism is based on the Oracle Streams Advanced Queuing mechanism to raise and/or consume events, for which the Qmon process is relevant..
Qmon is responsible for removing PROCESSED messages from a queue table and updating the dequeue IOT as necessary. In combining the usage of Scheduler with Advanced Queuing it is highly recommended to let the AQ_TM_PROCESSES parameter be unspecified (default value = 1) and let the system autotune.


An example to setup an event based job can be found in:
Note 1300444.1 - How to create a event based chained job
Note 1431664.1 - How to build up an event based job - an example

Known Bugs:

Unpublished Bug 9673944 - DBMS_SCHEDULER.RUN_JOB DOES NOT WORK ON EVENT JOBS.
==> fixed in 12.1.0.1
Unpublished Bug 12423122 - ORA-600 [KWQITNMPHE:LTBAGI] WHEN SCHEDULER USES AQ
==> fixed in 12.1.0.1, 11.2.0.3
Unpublished Bug 9865045 -  SCHE:HANG WHILE DROPPING EVENT RECEIVE JOBS.
==> fixed in 12.1.0.1, 11.2.0.3
Bug 12899768 - PROCESSED MESSAGES REMAIN IN QUEUE CAUSING SPACE ISSUES
==> fixed in 12.1.0.1, 11.2.0.4 , 11.2.0.3.8 PSU
Bug 13038485 - JOB_QUEUE_PROCESS NOT SPAWNED BY SCHEDULER AQ PROPAGATION JOB IN 11.2.0.2
Suspended, Req'd Info not Avail
Bug 9946875 - Scheduler event-based job does not fire when there is a subscriber on AQ queue
==> fixed in 12.1.0.1, 11.2.0.4

10. Chained Job Issues

A chain is a grouping of programs linked together for a single, combined objective.
Job chains are used or created to implement workflows where each chain step represents an individual task.
The chain rules represent dependencies among those tasks and can take the form of condition => action

Examples:
a) Serial chain:
     rule1: TRUE => START S1
     rule2: S1 completed => START S2
     rule3: S2 completed => START S3
     rule4: S3 completed => END

b) Branched chain:
    rule1: TRUE => START S1
    rule2: S1 completed => START S2, S3
    rule3: S2 completed AND S3 completed => START S4
    rule4: S4 succeeded => START S5
    rule5: S4 failed => START S6
    rule6: S5 completed OR S6 completed => END

Known Notes and Bugs:

Note 727499.1  - Sample how to create a job running chains
Note 1272728.1 - Is It Possible To Assign Chain Parameters To A Scheduler Job?
Note 1300444.1 -  How to create a event based chained job

Bug 12986415 - DBMS_SCHEDULER.RUN_CHAIN WITH STEP_STATE_LIST NOT WORKING DUE TO A MISSING JOIN
==> fixed in 12.1.0.1, 11.2.0.4

11. Scheduler Job Notifications Failure

You can configure a job to send e-mail notifications when it changes state. E-mail notifications can be sent to multiple recipients, and can be triggered by any event in a list of job state events that you specify. You can also provide a filter condition, and only job state events that match the filter condition generate notifications. You can include variables like job owner, job name, event type, error code, and error message in both the subject and body of the message. The Scheduler automatically sets values for these variables before sending the e-mail notification.

For example, you can configure a job to send an e-mail to both the principle DBA and one of the senior DBAs whenever the job fails with error code 600 or 700. You can also configure the same job to send a notification to only the principle DBA if the job fails to start at its scheduled time.

Scheduler Job Email Notifications will stop working for every job,even after restarting the database, for any user.

Known Notes and Bugs:

Note 16623661.8 - Bug 16623661 - No email notification is sent by scheduler 

Return to Blog