Sunday 12 January 2014

Troubleshoot concurrent program

A host based custom concurrent program was ending in ERROR and in the logs, we see 
ERROR:
ORA-03114: not connected to ORACLE

Found that there was no issue with connection as it is tested successfully when running manual.

Enabled the trace for the concurrent program.
(Check "Enable Trace" in Program -> Define -> Your Concurrent proram )

(How To Trace a Concurrent Request And Generate TKPROF File [ID 453527.1]
How to Generate a SQLTrace Including Binds and Waits for a Concurrent Program for 11.5.10 and R12 [ID 301372.1] )

Found nothing abnormal in the trace files but observed that core dump was being generated when ever this program error out.

In core dump files, observed that the error was being thwon at UPDATE statement which is executed after connecting to oracle. 
Further in the core dump file, we got this error:
ORA-07445: exception encountered: core dump [qervwRestoreViewBufPtrs()+204] [SIGSEGV] [ADDR:0xC00000010] [PC:0x108DAFA6C] [Address not mapped to object] []

Looked up the code in the oracle Notes and found that this is bug.

ORA-600/ORA-7445/ORA-700 Error Look-up Tool (Doc ID 153788.1)
Bug 9185829 - ORA-7445 under evaopn3 / qervwRestoreViewBufPtrs / kkqcscpopn_Int with GROUP BY query (Doc ID 9185829.8) 

Workaround proposed is to 
Set "_optimizer_group_by_placement" = false;
or
Set "_simple_view_merging"=false; 

Both these parameters were set to TRUE in our env.

How to check Hidden parameters:
SELECT a.ksppinm "Parameter",b.ksppstvl "Session Value",c.ksppstvl "Instance Value" FROM   x$ksppi a, x$ksppcv b, x$ksppsv c WHERE  a.indx = b.indx
AND    a.indx = c.indx AND    a.ksppinm LIKE '/_%' escape '/'
and a.ksppinm  like '%_simple_view_merging%'
--'%optimizer_group_by_placement%'

After setting these parameters to TRUE, the program completed successfully.