Troubleshooting ORA-27300 ORA-27301 ORA-27302 Errors
Troubleshooting ORA-27300 ORA-27301 ORA-27302 Errors
Titleimage
Posted by Patrick Hamou on 2017:09:22 18:33:02
APPLIES TO: Oracle Database - Enterprise Edition
Oracle Database - Enterprise Edition - Version 11.2.0.4 to 11.2.0.4 [Release 11.2]
Oracle Database - Enterprise Edition - Version 9.2.0.1 to 11.2.0.3 [Release 9.2 to 11.2]
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Information in this document applies to any platform.
PURPOSE: ORA-27300, ORA-27301, ORA-27302
RENAPS Offers Oracle Database & Middleware Managed Services. Learn More about how we can help your business
This note helps identify the possible causes of the ORA-2730x errors that have already been reported and provides solution where available. The errors are usually caused by OS system call error or OS configuration issue and are in the form of:
ORA-27300: OS system dependent operation:%s failed with status: %s
ORA-27301: OS failure message: %s
ORA-27302: failure occurred at: %s
You can refer the respective OS manual for the background information on how Unix OS errors work. e.g.,
http://www.ibm.com/developerworks/aix/library/au-errnovariable/
Please note only a handful of errors are standard across all Unix-based platforms. Documentation is available at the OS level for some of the dependent operation or OS calls such as fork() or semget(). The 'status' reported in the ORA-27300 error usually corresponds to a Unix error code defined in the "errno.h" header file on the OS, returned by the OS call.
For example,
ORA-27300: OS system dependent operation:fork failed with status: 11
A "man fork" shows:
The fork() function shall fail if:
EAGAIN (status 11) : The system lacked the necessary resources to create another process, or the system-imposed limit on the total number of processes under execution system-wide or by a single user {CHILD_MAX} would be exceeded. EAGAIN corresponds to status 11.
The fork() function may fail if:
ENOMEM (status 12) : Insufficient storage space is available.
TROUBLESHOOTING STEPS
The note is broken down by the status code.
STATUS 0
ORA-27300: OS system dependent operation:invalid_process_id failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgpalive1
See Note 356640.1 ORA-27300, ORA-27301, ORA-27302 Failure Occurred At Skgpalive1 During 'Shutdown' - UnPublished Bug 4923667.
In 10.2.0.5 and up, see Note 1252265.1 - ORA-27300 ORA-27301 ORA-27302: failure occurred at: skgpalive1 - bug 10010310.
ORA-27300: OS system dependent operation:invalid_process failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgpstime3
ORA-27144: attempt to kill process failed
See Note 458442.1 > />10.2.0.3 PMON CRASHES ON STARTUP ON AIX 5L 5.3 ML05 -- WORKS on ML06> />
STATUS 1 - EPERM Not owner
An attempt was made to modify a file that doesn't belong to the user, or the user was forbidden to modify that file by its owner or super-user.
ORA-27300: OS system dependent operation:invalid_egid failed with status: 1
ORA-27301: OS failure message: Not owner
ORA-27302: failure occurred at: skgpwinit6
ORA-27303: additional information: startup egid = 202 (?), current egid = 101 (dba)
See Note 453959.1 Cannot Connect As "/ as sysdba" ORA-27140 ORA-27300 ORA-27301
ORA-27512: IPC error posting a process
ORA-27300: OS system dependent operation:kill failed failed with status: 1
ORA-27301: OS failure message: Not owner
ORA-27302: failure occurred at: skgxppost1
See Note 3411021.8 - Bug 3411021 - Oracle process may die due to ORA-27300/ORA-27301/ORA-27302 at skgxppost1
STATUS 2 - ENOENT No such file or directory
This error occurs when trying to access a file or subdirectory that does not exist.
ORA-12012: error on auto execute of job 71010
ORA-27370: job slave failed to launch a job of type EXECUTABLE
ORA-27300: OS system dependent operation:accessing execution agent failed with status: 2
ORA-27301: OS failure message: No such file or directory
ORA-27302: failure occurred at: sjsec 3
See Note 277399.1 DBMS_SCHEDULER Fails To Execute Jobs With Program_type
EXECUTABLE On HP-UX
ORA-27300: OS system dependent operation:open failed with status: 2 (Doc ID 1664332.1)
ORA-27300: OS system dependent operation:open failed with status: 2
ORA-27301: OS failure message: No such file or directory
ORA-27302: failure occurred at: sskgmsmr_7
STATUS 3 - ESRCH No such process
The process number passed to the signal handling procedure belongs to
a process that doesn't exist, or it is already dead.
In a RAC environment, errors of the form below may be seen causing process failures:
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sskgxp_select failed with status: 3
ORA-27301: OS failure message: No such process
ORA-27302: failure occurred at: skgxpvfymmtu
ORA-27303: additional information: MTU could not be verified. Did not receive valid message.
See Note 746888.1 ORA-27302: Failure Occurred at: skgxpvfymmtu Signalled in the Alert.log
STATUS 6 - ENXIO No such device or address
I/O error on a component of a device that is missing.
Windows specific:
ORA-27300: OS system dependent operation:SuspendThread failed with status: 6
ORA-27301: OS failure message: The handle is invalid.
ORA-27302: failure occurred at: sssxcpttcs6
See BUG: 6991131 - Status: 32,Not a Bug
This is an issue during shutdown. We send a shutdown message to MMON and we wait for 30 sec to check if it has gone away. If it still exists then we collect the diagnostic information and try to suspend the target thread ("MMON" in this case). However we find that target thread handle is no longer valid since the thread has exited or about to exit. This causes the error to be reported in the alert.log. This error may be ignored during the shutdown process as long as it doesn't hang since the target thread was going away at the time we tried to collect the diag information.
======
ORA-27300: OS system dependent operation:GetThreadTimes failed with status: 6
ORA-27301: OS failure message: The handle is invalid.
ORA-27302: failure occurred at: skgpalive
See unpublished BUG: 6455284 - Status 80 : DNFS NT: ORA-27300,ORA-27301,ORA-27302: FAILURE OCCURRED AT: SKGPALIVE
Fixed in 11.1.0.7 and 11.2.
STATUS 8 - ENOEXEC Exec format error
Attempt was made to execute a file that does not have an executable format.
ORA-27300: OS system dependent operation:CreateThread failed with status: 8
ORA-27301: OS failure message: Not enough storage is available to process this command.
ORA-27302: failure occurred at: ssthrddcr.
See Note 466370.1 ORA-7445 [ACCESS_VIOLATION] [unable_to_trans_pc] [UNABLE_TO_WRITE] ORA-27301 OS failure message Not enough storage ORA-27300 ORA-27302> />
For Windows, see Note 225349.1 Implementing Address Windowing Extensions (AWE) or VLM on Windows Platforms
STATUS 11 - EAGAIN No more processes
Executing a fork and the system's process table is full, or the user is not allowed to create more process.
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
See Note 392006.1 Ora-27300 OS system dependent operation fork failed with status 11
and
Note 1546393.1 ORA-27300: OS System Dependent Operation:fork Failed With Status: 11
Maximum number of PROCESSES allowed per user may be too low (for example: maxuproc on AIX, maxuprc on HP and Solaris. On Solaris 10, check project.max-lwps)
@See Bug 7620133 - project.max-lwps was changed from the default which is 2 billion.
ORA-27300: OS system dependent operation:socket failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: sskgxpcre1
See Note 364353.1 ORA-00603 ORA-27504 ORA-27300 ORA-27504 in the Alert Log
(HP port specific)
1. Set MAX_ASYNC_PORTS to a value high enough
2. Deactivate async io at os level.
ORA-27300: OS system dependent operation:ioctl failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgxpvaddr1
See Note.6629265.8 Bug 6629265 - Intermittent ORA-27504 / ORA-27300 ioctl error 11
When system resources are overloaded, an intermittent failure to create a socket can cause new processes to fail to start with above errors.
See Unpublished Bug 17783533 : ORA-27300: OS SYSTEM DEPENDENT OPERATION:FORK FAILED WITH STATUS: 11
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
See Unpublished Bug 17783533 : ORA-27300: OS SYSTEM DEPENDENT OPERATION:FORK FAILED WITH STATUS: 11
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
This bug is fixed in 12.2
STATUS 12 - ENOMEM Not enough core / memory
During an exec or a break, the program asked for more memory than the one available by the system. This error also occurs when there are too many segmentation registers which are required for the arrangement of text data or stack segments.
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
See Note 465002.1 Database Crash With Error ORA-00490
Note 580552.1 Ora-04030 Ora-07445 Ora-27300 Ora-27301 Ora-27302 Crashed Database
Note 560309.1> /> Database Cannot Start Due to Lack of Memory
** All related to swap space being depleted. Check the OS system logs for more information. On some Operating Systems increasing nproc or maxuprc is required **> />
STATUS 13 - EACCES Permission denied
An attempt was made to access a forbidden file by the protection system.
ORA-27300: OS system dependent operation:accessing execution agent failed with status: 13
ORA-27301: OS failure message: Permission denied
ORA-27302: failure occurred at: sjsec 3
See Note 557153.1 ORA-27370, ORA-27301(Permission denied) when running job of type EXECUTABLE .
Restart the Oracle database using an appropriate user account (often the "oracle" user).
STATUS 17 - EEXIST File exists
An existing file was entered as a parameter of a command that will modify it.
ORA-27300: OS system dependent operation:semget failed with status: 17
ORA-27301: OS failure message: File exists
ORA-27302: failure occurred at: sskgpcreate
STATUS 22 - EINVAL Invalid argument
Invalid argument passed.
For example: Passing an unknown signal in the signal function.
ORA-27300: OS system dependent operation:semctl failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgpwrm1
ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 36
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
See Note 438205.1 > />ORA-27300 ORA-27301 ORA-27302 ORA-27157 Database Crash Semaphores/shared memory might have been removed> />
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
This error was signaled because we were waiting to acquire a latch and it appears that the underlying identifier was removed during the wait which was unexpected and caused the ora-600 to be signaled.
If the similar error is occuring while instance startup,then the below bug might be a cause.
(Unpublished) Bug 17638002 - ORA-600 [KSLGESS_3] [] [] [] [] [] [] [] [] [] [] []
ORA-27300: OS system dependent operation:mpctl_ldomspus failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: skgsnnprocs
See BUG: 7232946 - ORA-600[KSKRECONFIGNUMA2] CAUSES INSTANCE CRASH
10.2.0.4.0
It looks like this is all resulting of NUMA issues because of NUMA feature is enabled by default starting with 10.2.0.4. To disable NUMA you need to see the following db parameters:
_db_block_numa=1
_enable_NUMA_optimization=FALSE
ORA-27300: OS system dependent operation:pw_postv failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: sskgpwvp3
ORA-27303: additional information: ukid, errno, post_count = 0x7a9d93eb191f0b4e, 22, 3
See Note 6441119.8 - unpublished Bug 6441119 - Instance crash due to ORA-27300 / ORA-27152
An ORA-27300 error from sskgpwpost or sskgpwvectorpost can lead to an instance crash.
ORA-00600: internal error code, arguments: [ksbmoveme4], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:run on node failed with status: 22
ORA-27301: OS failure message: Invalid argument
ORA-27302: failure occurred at: skgpmoveme:1
See Note 848387.1 - With NUMA Enabled, Database Fails To Open With ORA-600 [ksbmoveme4], ORA-27300, ORA-27301, ORA-27302 Errors Reported
STATUS 28 - ENOSPC No space left on device
When writing to a file, the device runs out of space.
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occured at: sskgpcreates
See Note 314179.1 Instance Startup Fails With Error ORA-27154,ORA-27300,ORA-27301,ORA-27302
Oracle 10.2 Installation Guide (for Solaris 8 and 9):
semsys:seminfo_semmns 1024
semsys:seminfo_semmsl 256
For Linux:
check semaphores settings:
cat /proc/sys/kernel/sem
STATUS 34
ORA-27303: additional information: old_semid post_count 0x278031 6
ORA-27152: attempt to post process failed
ORA-27300: OS system dependent operation:semop failed with status: 34
ORA-27301: OS failure message: Numerical result out of range
ORA-27302: failure occurred at: sskgpwvp2
ORA-27303: additional information: old_semid post_count 0x278031 6
Modify system sem parameter to avoid the kernel error:
eg:
from
kernel.sem = 250 32000 100 128
to:
kernel.sem = 1024 60000 1024 256
STATUS 59
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 59
ORA-27301: OS failure message: Message too long
ORA-27302: failure occurred at: sskgxpsnd1
See RAC: Note 300956.1 Ora-27302: sskgxpsnd1 - Starting Instance
Note 295832.1 - Unable To Create Database Ora-00603, ORA-27300
STATUS 63
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:ioctl failed with status: 63
ORA-27301: OS failure message: Out of stream resources
ORA-27302: failure occurred at: skgxpvaddr1
Specifically error code 63 is ENOSR, "out of streams resources", as reported in the ora-27301 message.
STATUS 227
ORA-27300: OS system dependent operation:bind failed with status: 227
ORA-27301: OS failure message: Can't assign requested address
ORA-27302: failure occurred at: sskgxpcre3
See BUG: 9059053 - Status 92 - Not a bug.
Solution:
Increase the udp port range, recommended setting is:
udp_smallest_anon_port=9000
udp_largest_anon_port=65535
STATUS 997
ORA-27300: OS system dependent operation:spcdr:9261:4200 failed with status: 997
ORA-27301: OS failure message: Overlapped I/O operation is in progress.
ORA-27302: failure occurred at: skgpspawn
See Note 371074.1 ORA-27300 ORA-27301 ORA-27302 in alert log. Cannot connect to database.
Windows specific OS resource issue:
The maximum amount of addressable memory by a process ('oracle.exe' for example) running in a
32-Bit Windows environment has been reached.
STATUS 10054
ORA-27508 : IPC error sending a message
ORA-27300 : OS system dependent operation:IPCSOCK_Send failed with status: 10054
ORA-27301 : OS failure message: An existing connection was forcibly closed by the remote host.
ORA-27302 : failure occurred at: send_3
See Note 297498.1 Resolving Instance Evictions on Windows Platforms
Also on Windows:
The following observation was made:
These errors were received regardelss of the DOS command or BATCH file executed:
ORA-27370: job slave failed to launch a job of type EXECUTABLE
ORA-27300: OS system dependent operation:accessing execution agent failed with status: 2
ORA-27301: OS failure message: Le fichier spécifié est introuvable.
ORA-27302: failure occurred at: sjsec 6a
ORA-27303: additional information: Le fichier spécifié est introuvable.
It was discovered that the OracleJobScheduler Service was stopped.
Therefore, please check the OracleJobScheduler service if you encounter this problem.
For IBM AIX on POWER Systems (64-bit)
please see the following note:
Ora-27300: Os System Dependent Operation:Fork Failed With Status: 12 <>
If you see either of these function codes Skgpspawn skgpspawn5
please refer to <1125188.1> , <812115.1> & <435787.1> for details.
If the solution searched for is not on this "master" list, then one search tip to narrow down the search findings would be to try specifically searching with the keywords:
ORA-27300 OS-call-raised-error(first %s of ORA-27300) Oracle-call-where-OS-failure-occurred(%s for ORA-27302)
So for example try searching on: ORA-27300 fork skgpspawn3
Posted by Patrick Hamou on 2017:09:22 18:33:02