Well this was one of the few things in his presentation I was fit to comment on so I tried to attract Julian’s attention by waving my arm, sadly my timid nature meant I went unnoticed. So this blog post is me going public with my attempted interruption.

First an example of how the log location for SCAN listeners typically differs to that of Database listeners. I personally do not like this behaviour as I like my Grid Infrastructure home to have resonably stable size, which it can have once the log directory’s rotation is mature, however the ADR home for listeners is not cleared down automatically so you can end up with a directory growing unchecked.

[grid@n01 n01]$ $GRID_HOME/bin/lsnrctl status LISTENER|grep Log
Listener Log File         /u01/app/oracle/diag/tnslsnr/n01/listener/alert/log.xml
[grid@n01 n01]$ $GRID_HOME/bin/lsnrctl status LISTENER_SCAN3|grep Log
Listener Log File         /u01/app/grid/11203/ghome_1/log/diag/tnslsnr/n01/listener_scan3/alert/log.xml

And we know from the Oracle documentation that ADR_BASE is defined as:

Purpose
To specify the base directory into which tracing and logging incidents are stored when ADR is enabled.

Default
The default on the server side is ORACLE_BASE, or ORACLE_HOME/log, if ORACLE_BASE is not defined.

So… let’s check ORACLE_BASE for CRS resources:

[grid@n01 ~]$ $GRID_HOME/bin/crsctl stat res ora.LISTENER.lsnr -p|grep -i base
USR_ORA_ENV=ORACLE_BASE=/u01/app/oracle
[grid@n01 ~]$ $GRID_HOME/bin/crsctl stat res ora.LISTENER_SCAN3.lsnr -p|grep -i base
[grid@n01 ~]$

Notice above that the Database listener has $ORACLE_BASE set and the Scan listener does not. Hence the log location.

This can be controlled for Database listeners by using the SETENV switch for SRVCTL but this does not work for Scan listeners.

[grid@n01 ~]$ $GRID_HOME/bin/srvctl setenv listener -l listener_scan1 -t "ORACLE_BASE=/u01/app/oracle"
PRKO-3162 : The actions setenv, getenv, and unsetenv are not suppported for Single Client Access Name listener listener_scan1
PRCN-2066 : Failed to retrieve cluster listener for listener_scan1 because it has the type of Single Client Access Name Listener

There is another way of setting environment variables for cluster resources and that is via crsctl modify resource.

[grid@n01 ~]$ $GRID_HOME/bin/crsctl modify resource ora.LISTENER_SCAN3.lsnr -attr USR_ORA_ENV=ORACLE_BASE=/u01/app/oracle
[grid@n01 ~]$ $GRID_HOME/bin/crsctl stat res ora.LISTENER_SCAN3.lsnr -p|grep -i base
USR_ORA_ENV=ORACLE_BASE=/u01/app/oracle
[grid@n01 ~]$ $GRID_HOME/bin/srvctl stop scan_listener -i 3
[grid@n01 ~]$ $GRID_HOME/bin/srvctl start scan_listener -i 3
[grid@n01 ~]$ lsnrctl status  listener_scan3 | grep Log
Listener Log File         /u01/app/oracle/diag/tnslsnr/n01/listener_scan3/alert/log.xml

To my mind the whole thing would seem a little cleaner if there was an “SRVCTL SETENV SCAN_LISTENER” command.

After writing this post I thought I’d check around for other information to support this post and, lo and behold, there is a My Oracle Support article covering almost exactly what I’ve written above. The article also talks about using the LISTENER.ORA attribute ADR_BASE_LISTENER_SCANn to control this.

Diagnostic Destination For Listeners Is Under Grid_home/Log Instead Of Oracle_base/Diag [ID 1269109.1]

Lets create list base partitioned table transactions

And create two indexes TRANSACTIONS_PK (SERNO, PARTITIONKEY) and local partitioned non-prefixed index TRANSACTIONSI (SGENERAL).

Let’s start from the most common statement

EXPLAIN PLAN FOR
 SELECT SGENERAL, count(1)
 FROM transactions
 WHERE PARTITIONKEY='JUL2012' AND SGENERAL is not null
 GROUP BY SGENERAL;

PLAN_TABLE_OUTPUT
———————————————————————————————————————————————-
Plan hash value: 2338610280

Predicate Information (identified by operation id):
—————————————————
3 – filter(“STGENERAL” IS NOT NULL)

Does it look like non-optimal? Optimizer has chosen to do FULL SCAN even if it has nice looking index TRANSACTIONSI. The problem is that partition pruning works only for prefixed indexes, that means you have to pay for support extra column in your index even if you do not need it because you clearly can get the partition name based from the identical rules on a table.

But luckily we have extended syntaxes. Uses it we can manually define the partition that should be used. In our case the partition name is equal to the partition key. Getting the partition name is very simple for list partitions but can be received in all other cases too.

EXPLAIN PLAN FOR
 SELECT SGENERAL, count(1)
 FROM TRANSACTIONS PARTITION ("JUL2012")
 WHERE SGENERAL is not null
 GROUP BY SGENERAL;

PLAN_TABLE_OUTPUT
———————————————————————————————————————————————-
Plan hash value: 198159339

Predicate Information (identified by operation id):

—————————————————
3 – filter(“SGENERAL” IS NOT NULL)

This plan looks much better. According to optimizer estimations we can save 14 minutes on it. But there is even more attractive syntaxes for those who do not want to spend their time calculating the partition name

 EXPLAIN PLAN FOR
 SELECT SGENERAL, count(1)
 FROM TRANSACTIONS PARTITION FOR ('JUL2012')
 WHERE SGENERAL is not null
 GROUP BY SGENERAL;

PLAN_TABLE_OUTPUT
———————————————————————————————————————————————
Plan hash value: 198159339

Predicate Information (identified by operation id):
—————————————————
3 – filter(“SGENERAL” IS NOT NULL)

The only problem with last syntax is that it is supported only from 11.2 version. The syntax was developed to support interval partitions but can be handy for all other types.

Does it looks that with new syntax we came to the kingdom of wealth and prosperity, where we can easily avoid prefixes on local indexes and still use the effective partition pruning? But it is not. The hidden rock is that the partition keys in PARTITION FOR clause could not be defined throw the variables like in WHERE clause. Below there are examples of such attempt

Firstly traditional approach

 DECLARE
 v_part char(7):='JUL2012';
 BEGIN
 FOR REC IN (SELECT SGENERAL, count(1) FROM TRANSACTIONS
 WHERE PARTITIONKEY=v_part AND SGENERAL is not null
 GROUP BY SGENERAL)
 LOOP
 NULL;
 END LOOP;
 END;
 /
 Elapsed: 00:12:42.05

It works but does not looks like very performance effective. Let’s go and try 10g approach in PL/SQL

DECLARE
 v_part varchar2(10):="JUL2012";
 BEGIN
 FOR REC IN (SELECT SGENERAL, count(1) FROM TRANSACTIONS PARTITION (v_part)
 WHERE SGENERAL is not null
 GROUP BY SGENERAL)
 LOOP
 NULL;
 END LOOP;
 END;
 /

ERROR at line 2:
 ORA-06550: line 2, column 25:
 PLS-00201: identifier "JUL2012" must be declared
 ORA-06550: line 2, column 10:
 PL/SQL: Item ignored

It looks like kind of expected behavior. You could not use table name as variable, why you should be able to use partition name. Personally I put a lot of hope on the last test with PARTITIONKEY reference

DECLARE<span style="color: #000000;">
v_part char(7):='JUL2012';
 BEGIN
 FOR REC IN (SELECT SGENERAL, count(1) FROM TRANSACTIONS
 PARTITION FOR (v_part)
 WHERE SGENERAL is not null
 GROUP BY SGENERAL)
 LOOP
 NULL;
 END LOOP;
 END;
 /

ERROR at line 1:
 ORA-14763: Unable to resolve FOR VALUES clause to a partition number
 ORA-06512: at line 4

It was the first time I see the error thus have a look into the documentation

ORA-14763: Unable to resolve FOR VALUES clause to a partition number

Cause: Could not determine the partition corresponding to the FOR VALUES clause.

Action: Remove bind variables and dependencies on session parameters from the values specified in the FOR VALUES clause.

Ha-Ha-Ha nothing change we still could not pass partition key values into query.

Summarizing oracle offer the mechanism that allows to use partition pruning on non-prefixed local indexes but have not support the syntaxes in pl/sql using binds. The usage of this syntaxes have sense only in case of big queries (e.g. reports) running on a table, when gains from effective execution plan prevail over the loses of dynamic sql.

awrsql.sql:

prompt enter start and end times in format DD-MON-YYYY [HH24:MI]
 
column sample_end format a21
select to_char(min(s.end_interval_time),'DD-MON-YYYY DY HH24:MI') sample_end
, q.sql_id
, q.plan_hash_value
, sum(q.EXECUTIONS_DELTA) executions
, round(sum(DISK_READS_delta)/greatest(sum(executions_delta),1),1) pio_per_exec
, round(sum(BUFFER_GETS_delta)/greatest(sum(executions_delta),1),1) lio_per_exec
, round((sum(ELAPSED_TIME_delta)/greatest(sum(executions_delta),1)/1000),1) msec_exec
from dba_hist_sqlstat q, dba_hist_snapshot s
where q.SQL_ID=trim('&sqlid.')
and s.snap_id = q.snap_id
and s.dbid = q.dbid
and s.instance_number = q.instance_number
and s.end_interval_time >= to_date(trim('&start_time.'),'dd-mon-yyyy hh24:mi')
and s.begin_interval_time <= to_date(trim('&end_time.'),'dd-mon-yyyy hh24:mi')
and substr(to_char(s.end_interval_time,'DD-MON-YYYY DY HH24:MI'),13,2) like '%&hr24_filter.%'
group by s.snap_id
, q.sql_id
, q.plan_hash_value
order by s.snap_id, q.sql_id, q.plan_hash_value
/

Nothing ground breaking and I’m sure many will have a similar script.

Below I have example output showing one of the scripts many successful outings, quite a dramatic plan change I’m sure you’ll agree.

SQL> @awrsql
enter start and end times in format DD-MON-YYYY [HH24:MI]
Enter value for sqlid: 1jjpo2i4b313g
Enter value for start_time: 15-NOV-2011
Enter value for end_time: 21-NOV-2011 13:00
Enter value for hr24_filter:

SAMPLE_END            SQL_ID        PLAN_HASH_VALUE EXECUTIONS PIO_PER_EXEC LIO_PER_EXEC  MSEC_EXEC
--------------------- ------------- --------------- ---------- ------------ ------------ ----------
15-nov-2011 TUE 08:00 1jjpo2i4b313g      3133159894     129629            0            5          0
16-nov-2011 WED 08:01 1jjpo2i4b313g      3133159894     115003            0            5         .1
17-nov-2011 THU 08:01 1jjpo2i4b313g      3133159894     115741            0            5          0
18-nov-2011 FRI 07:00 1jjpo2i4b313g      3133159894      30997            0            5         .1
18-nov-2011 FRI 08:00 1jjpo2i4b313g      3133159894      81034            0            5          0
21-nov-2011 MON 00:00 1jjpo2i4b313g       790865878         16     323091.6     323128.3    36905.8
21-nov-2011 MON 01:00 1jjpo2i4b313g       790865878         29     349676.2     349713.7    48387.2
21-nov-2011 MON 02:00 1jjpo2i4b313g       790865878         35     339474.6     339509.2    34057.7
21-nov-2011 MON 03:00 1jjpo2i4b313g       790865878         37     340934.6     340970.2    35899.4
21-nov-2011 MON 04:01 1jjpo2i4b313g       790865878         38     333469.1     333503.9    35450.8
21-nov-2011 MON 05:00 1jjpo2i4b313g       790865878         35     347559.3     347595.2    35231.8
21-nov-2011 MON 06:00 1jjpo2i4b313g       790865878         32     340224.8       340260    35208.3

I also like to use the query to track number of executions or LIO per execution over a longer time frame to see if either the frequency or individual impact of the SQL is changing over time. I can use the “hr24_filter” variable to do this, for example showing me all snapshots for hour “13″ over a whole month.

create role readonly_dba;
grant create session to readonly_dba;
grant select any dictionary to readonly_dba;
grant select any table to readonly_dba;
-- and maybe a few other bits and pieces such as below (site permitting)
grant execute on dbms_monitor to readonly_dba;

With such a role a user can figure out most things including the DDL that defines an object. However with such a user DBMS_METADATA.GET_DDL on an object in a different schema returns an error (output below from 10.2 but 11.2 is similar):

SQL> create user mydba identified by mydba;
SQL> grant readonly_dba to mydba;
SQL> conn mydba/mydba

SQL> -- picked a table at random - DBSNMP.MGMT_SNAPSHOT
SQL> SELECT DBMS_METADATA.GET_DDL('TABLE','MGMT_SNAPSHOT','DBSNMP') FROM dual;
ERROR:
ORA-31603: object "MGMT_SNAPSHOT" of type TABLE not found in schema "DBSNMP"
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.DBMS_METADATA", line 2805
ORA-06512: at "SYS.DBMS_METADATA", line 4333
ORA-06512: at line 1

The solution to this is documented as below but I must admit I only discovered this once I’d figured things out the hard way. A classic case of something being obvious if you already know the answer.

The DBMS_METADATA package considers a privileged user to be one who is connected as user SYS or who has the SELECT_CATALOG_ROLE role.

So now anyone Googling “ORA-31603″ has a potential answer I’ll show you how this is implemented.

When faced with the error above I enabled SQL_TRACE and tried again.

SQL> exec dbms_monitor.session_trace_enable;
SQL> SELECT DBMS_METADATA.GET_DDL('TABLE','MGMT_SNAPSHOT','DBSNMP') FROM dual;
ERROR:
ORA-31603: object "MGMT_SNAPSHOT" of type TABLE not found in schema "DBSNMP"
ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.DBMS_METADATA", line 2805
ORA-06512: at "SYS.DBMS_METADATA", line 4333
ORA-06512: at line 1



no rows selected

The penultimate fetch in the resulting trace file was for the statement below (reformatted for readability).

PARSING IN CURSOR #7 len=283 dep=1 uid=39 oct=3 lid=39 tim=2007863993 hv=2568162639 ad='af4fed2c'
SELECT /*+rule*/ SYS_XMLGEN(VALUE(KU$)
, XMLFORMAT.createFormat2('TABLE_T', '7'))
, KU$.OBJ_NUM 
FROM SYS.KU$_HTABLE_VIEW KU$ 
WHERE NOT (BITAND (KU$.PROPERTY,8192)=8192) 
AND  NOT BITAND(KU$.SCHEMA_OBJ.FLAGS,128)!=0 
AND  KU$.SCHEMA_OBJ.NAME=:NAME1 
AND  KU$.SCHEMA_OBJ.OWNER_NAME=:SCHEMA2
END OF STMT
PARSE #7:c=0,e=721,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=3,tim=2007863991
EXEC #7:c=0,e=674,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=3,tim=2007864789
FETCH #7:c=0,e=120,p=0,cr=14,cu=0,mis=0,r=0,dep=1,og=3,tim=2007866558

The FETCH line shows no rows were returned (“r=0″). Next I checked the definition of the view – using DBA_VIEWS as DBMS_METADATA isn’t working for other schemas

SQL> set pages 100
SQL> select text from dba_views where view_name = 'KU$_HTABLE_VIEW';

TEXT
--------------------------------------------------------------------------------
... SNIP ...
AND (SYS_CONTEXT('USERENV','CURRENT_USERID') IN (o.owner_num, 0) OR
                EXISTS ( SELECT * FROM session_roles
                        WHERE role='SELECT_CATALOG_ROLE' ))

The role is hardcoded in the view.

After granting SELECT_CATALOG_ROLE to the READONLY_DBA role DBMS_METADATA.GET_DDL returns the desired DDL.

SQL> SELECT DBMS_METADATA.GET_DDL('TABLE','MGMT_SNAPSHOT','DBSNMP') FROM dual;

DBMS_METADATA.GET_DDL('TABLE','MGMT_SNAPSHOT','DBSNMP')
--------------------------------------------------------------------------------

  CREATE TABLE "DBSNMP"."MGMT_SNAPSHOT"
   (    "INSTANCE_NUMBER" NUMBER NOT NULL

Success.

Lessons learned:

1) Once again – read the documentation.
2) Shying away from the 1600+ privileges (2100+ in 11.2) in SELECT_CATALOG_ROLE is not always ideal.
3) Even Oracle aren’t averse to a bit of hardcoding

The following privileges required for the user who run the script

-- GRANT CREATE PROEDURE
-- GRANT EXECUTE ON DBMS_CRYPTO
-- GRANT EXECUTE ON DBMS_METADATE
-- GRANT SELECT ON DBA_SOURCE
-- GRANT SELECT ON DBA_SYS_PRIVS
-- GRANT SELECT ON DBA_TAB_PRIVS
-- GRANT SELECT ON DBA_ROLE_PRIVS

The following script take one parameter – the application schema name and provide as output list of database objects with corresponding hash values.

drop function verify_source
/


Hope you will find it usefull.

The output from this script can be compared as simple flat file using any favorite comparison tool, e.g. WinMerge.

Parameter     Description
------------  ------------------------------
files         Number of files to be written
blocks        Number of blocks to be written
requests      Number of I/O requests
------------  ------------------------------

File is obvious, the number of log file members in the current redo log group, but for the other two parameters I wasn’t sure if it is the number of blocks/request per file or in total. I assumed total, but wanted to be sure so did a quick bit of searching. This led me to OraDBPedia where I read,

P2=The number of redo blocks to be written to each log member.

and

P3=Total number of I/O requests needed to satisfy the write load (i.e. P2).

For whatever reason, I didn’t feel 100% happy to accept this so did a quick test in 11.2.0.2 XE as shown below.

Test Details:

1. Enable tracing of LGWR
2. Run “alter system switch logfile” to cause LGWR to flush the redo buffer (not shown in output below)
3. Run small insert statement
4. Commit
5. Examine trace file

Starting off with the default in XE of 1 member per group:

14:01:19 SQL> insert into t (c1, c2, c3) values (1,'DELETE', sysdate);

1 row created.

14:01:23 SQL> commit;

Commit complete.

14:01:27 SQL>

Looking at the 10046 trace reveals:

*** 2011-09-29 14:01:27.315
WAIT #0: nam='log file parallel write' ela= 374 files=1 blocks=2 requests=1 obj#=-1 tim=1481335308118

Adding another member to each of the 2 groups giving 2 members per group:

14:01:27 SQL> insert into t (c1, c2, c3) values (1,'DELETE', sysdate);

1 row created.

14:04:48 SQL> commit;

Commit complete.

14:04:53 SQL>

Looking at the 10046 trace reveals:

*** 2011-09-29 14:04:52.477
WAIT #0: nam='rdbms ipc message' ela= 2969787 timeout=300 p2=0 p3=0 obj#=-1 tim=1481538543689
WAIT #0: nam='rdbms ipc message' ela= 30043 timeout=3 p2=0 p3=0 obj#=-1 tim=1481538574024
WAIT #0: nam='rdbms ipc message' ela= 567670 timeout=300 p2=0 p3=0 obj#=-1 tim=1481539141991
WAIT #0: nam='log file parallel write' ela= 550 files=2 blocks=4 requests=2 obj#=-1 tim=1481539142779

Adding another member to each of the 2 groups giving 3 members per group:

14:04:53 SQL> insert into t (c1, c2, c3) values (1,'DELETE', sysdate);

1 row created.

14:07:51 SQL> commit;

Commit complete.

14:07:53 SQL>

Looking at the 10046 trace reveals:

*** 2011-09-29 14:07:53.980
WAIT #0: nam='rdbms ipc message' ela= 2560082 timeout=300 p2=0 p3=0 obj#=-1 tim=1481718344261
WAIT #0: nam='log file parallel write' ela= 1658 files=3 blocks=6 requests=3 obj#=-1 tim=1481718346216
WAIT #0: nam='rdbms ipc message' ela= 438069 timeout=44 p2=0 p3=0 obj#=-1 tim=1481718784548

Adding another member to each of the 2 groups giving 4 members per group:

14:07:53 SQL> insert into t (c1, c2, c3) values (1,'DELETE', sysdate);

1 row created.

14:09:48 SQL> commit;

Commit complete.

14:09:51 SQL>

Looking at the 10046 trace reveals:

*** 2011-09-29 14:09:51.687
WAIT #0: nam='rdbms ipc message' ela= 1445500 timeout=300 p2=0 p3=0 obj#=-1 tim=1481834980859
WAIT #0: nam='log file parallel write' ela= 1910 files=4 blocks=8 requests=4 obj#=-1 tim=1481834983045

I see the above a conclusive evidence that both blocks (p2) and requests (p3) are the total rather than “per file”.

Maybe things have changed since the OraDBPedia articles was originally written, but I’ve contacted the guys behind the site and expect the page to be updated shortly.

A few comments about my last post before I get into this one:

1. DTrace support in Linux was deliberately missed off the list as it was conveyed in the Oracle Linux 6 session I attended that it was not production ready.
2. I’m an Oracle database geek. The other sessions I attended were not bad, but as someone that spends a lot of time trying to learn as much as possible about the Oracle database server, when I attend a presentation on a subject I’m much less familiar, but also very keen on, I tend to get a lot out of it – The point is, don’t get the impression that the other sessions were not good.

This was my 3rd OpenWorld and the best experience for me by far. There are a number of factors contributing to that including an apparent increase in the number of technical sessions (necessary as Unconference was not run this year). I highly recommend attending OpenWorld. It is a different experience from the only other Oracle focused conference I’ve attended (UKOUG Conference), but they are both great from my point of view, which is why I plan to continue attending both… And hopefully getting to some more

Reasons to consider attending OpenWorld:

1. The speakers are absolutely top notch – There are too many names to “drop” here, but you really do have the best independent and Oracle employed experts
2. The evening events provide an opportunity to meet like-minded people, and “talk tech”
3. You are there to hear the latest hardware and software releases as they happen
4. It’s in San Francisco!

Anyone that follows me on Twitter may have noticed that I was very impressed by what I saw of Oracle Enterprise Manager 12c “Cloud Control”. While watching the presentation I planned to get it installed on a VM as soon as I got back to the UK. Martin Bach was clearly as keen as I am and he’s posted details of his install here. Boy, that man’s fast. 11.2.0.3 installed about as fast as humanly possible after the release and pretty much the same with this – Nice work Martin

Boarding call!

Without a doubt the session that got me most excited so far was “Overview: New Features in Oracle Linux 6″. I’m a DBA, but take a very keen interest in Linux. I’ve been choosing Oracle Enterprise Linux (note that it has actually been renamed as Oracle Linux now) as my OS of choice for pretty much everything I do in my personal lab environment, not just for database servers. This is something that a couple of my Linux geek friends have questioned. They have extolled the joys of Debian and criticised Oracle for being lazy with Linux – “Just taking Red Hat and re-branding it.” I really wished they’d been in this session. I came away with a very strong impression that Oracle really do care about Linux, they do invest in Linux and they have some very smart guys onboard who really know what they are doing. It’s hard for me to cover everything that was discussed during the session here, but here’s an attempt to summarise:

1. Oracle Linux is not a desktop distribution – Obvious, but worth stating
2. Focus is on the kernel – Hence Unbreakable Enterprise Kernel
3. Aim to stay close to the mainline as possible – Aim to release around once a year
4. 9 month grace before expecting the new kernel to be used
5. Network optimisation such as receive packet steering (RPS) and transmit packet steering (XPS) sound very significant
6. Oracle Linux 6.2 is beta now or very very soon – This is wrong (see comment from Lenz below)
7. Transcendent Memory
8. Btrfs – “Build around snapshots” (Chris Mason)
9. Task Control Groups (Cgroups) – Offer fine grained control of resources and particularly useful for NUMA systems
10. Linux Containers – Yes, that’s right
11. Ksplice – Patch your kernel with zero downtime… Oh, and you can rollback with no downtime too

If you do nothing else after reading this post then checkout Ksplice! Read the official site and this on wikipedia

I should stress that there was the usual disclaimer about the information provided and there is always the possibility that I’ve misrepresented what was said, so do additional research before making big decisions based on the above.

Also mentioned was Open vSwitch which sounds like something I should look into.

While writing this I have been prompted to think about what matter to me with respect to Linux and I’ve concluded that exactly how much money (by paying kernel developers) Oracle have put into the Linux kernel is something that I don’t have the time or inclination to work out. What matters to me is that Linux is developing, Oracle are contributing and Oracle provide me with a completely free version that I can use in my lab for database servers and anything else I want to do.

MAXQUERYLEN NUMBER Identifies the length of the longest query (in seconds) executed in the instance during the period. You can use this statistic to estimate the proper setting of the UNDO_RETENTION initialization parameter. The length of a query is measured from the cursor open time to the last fetch/execute time of the cursor. Only the length of those cursors that have been fetched/executed during the period are reflected in the view.

And here is a quote from My Oracle Support note “V$UNDOSTAT MAXQUERYLEN Suddenly Appears With a High Value For a Simple Query [ID 1307600.1]“

the MAXQUERYLEN value is defined as “the length of a query measured from the cursor open time to the last fetch/execute time of the cursor”. Hence if the cursor is suspended (e.g. dbms_lock.sleep) or the session is switched out (session switching) then the MAXQUERYLEN time will continue to increase but will not be show as no current execution/fetch is occurring on the cursor.

…

This is expected behaviour

Nothing wrong with any of this but it is all relevant to the test that follows. In order to set up the test I change my local database to have a small undo tablespace and generate some workload. I’ve collapsed the code in case you want to jump straight to the story, in summary it generates some transactions and random queries with a duration between 1 and 10 minutes.

alter system set undo_tablespace = undo scope=both sid='*';

create undo tablespace undo_small 
datafile 'C:\ORACLEXE\ORADATA\XE\UNDO_SMALL.DBF' size 30m reuse autoextend on maxsize 500m;

alter system set undo_tablespace = undo_small scope=both sid='*';

create or replace function slow_query(i_seconds_to_wait in number)
	return number
as
	l_n number;
	l_pad2 bigtab.pad2%TYPE;

	cursor c_bigtab is
		select pad2 from bigtab; 
begin
	open c_bigtab;
	fetch c_bigtab into l_pad2;

	sys.dbms_lock.sleep(i_seconds_to_wait);
	
	fetch c_bigtab into l_pad2;
	close c_bigtab;

	return i_seconds_to_wait;
end slow_query;
/

-- session #1 - generate trxns

declare
	L_rno number;
	L_i number;
	L_pad1 bigtab.pad1%TYPE;
begin
	while true loop
		L_i := L_i + 1;
		L_rno := round(dbms_random.value(1,1000000));

		update bigtab
		set pad1 = pad1 where rno = L_rno;

		commit;

		dbms_lock.sleep(0.01);
	end loop;
end;
/

-- session #2 - generate queries of random length betwen 1 min and 10 mins

declare
	l_sleep number;
	l_slept number;
begin
	while true loop
		l_sleep := round(dbms_random.value(60,600));

		select slow_query(l_sleep) into l_slept from dual;
	end loop;
end;
/

Output from V$UNDOSTAT after the background workload has been running for a while is below along with the current size of the undo tablespace.

END_TIME_CHAR          TXNCOUNT MAXQUERYLEN MAXQUERYID    ACTIVEBLKS UNEXPIREDBLKS EXPIREDBLKS TUNED_UNDORETENTION
-------------------- ---------- ----------- ------------- ---------- ------------- ----------- -------------------
28-SEP-2011 19:45:50      38118         584 200k2np23z57q        160           912         496             1425
28-SEP-2011 19:55:50      38404         452 200k2np23z57q        160          1120        1280             1292
28-SEP-2011 20:05:50      38507         479 200k2np23z57q        160          1704        1464             1320
28-SEP-2011 20:15:50      38340         383 200k2np23z57q        160          2656         496             1224

   FILE_ID FILE_NAME                                             MB
---------- --------------------------------------------- ----------
         5 C:\ORACLEXE\ORADATA\XE\UNDO_SMALL.DBF                 30

Now a simple query is executed from SQL Developer and the first set of rows is fetched. V$SESSION output below.

SQL> select username,program,status,state,event,sql_id,seconds_in_wait siw
  2  from v$session
  3  where username = 'ADHOC';

USERNAME PROGRAM         STATUS   STATE               EVENT                       SQL_ID               SIW
-------- --------------- -------- ------------------- --------------------------- ------------- ----------
ADHOC    SQL Developer   INACTIVE WAITING             SQL*Net message from client 6fwqzurbc8y7k       7

SQL> select sql_text from v$sql where sql_id = '6fwqzurbc8y7k';

SQL_TEXT
-----------------------------------------------------------------------
select * from bigtab

Another quote from the Oracle Documentation

SQL_ID VARCHAR2(13) SQL identifier of the SQL statement that is currently being executed

You can see from the V$SESSION output above that we have a SQL ID and therefore must be executing some SQL. But we also have a session status of “INACTIVE” and are waiting on event “SQL*Net message from client”. That is a nice way of spotting sessions that are similar to the focus of this post.

Around 15 minutes later the session is still waiting on the user to act and V$UNDOSTAT looks consistent with my steady state.

USERNAME PROGRAM         STATUS   STATE               EVENT                       SQL_ID               SIW
-------- --------------- -------- ------------------- --------------------------- ------------- ----------
ADHOC    SQL Developer   INACTIVE WAITING             SQL*Net message from client 6fwqzurbc8y7k        912

END_TIME_CHAR          TXNCOUNT MAXQUERYLEN MAXQUERYID    ACTIVEBLKS UNEXPIREDBLKS EXPIREDBLKS TUNED_UNDORETENTION
-------------------- ---------- ----------- ------------- ---------- ------------- ----------- -------------------
28-SEP-2011 19:45:50      38118         584 200k2np23z57q        160           912         496             1425
28-SEP-2011 19:55:50      38404         452 200k2np23z57q        160          1120        1280             1292
28-SEP-2011 20:05:50      38507         479 200k2np23z57q        160          1704        1464             1320
28-SEP-2011 20:15:50      38340         383 200k2np23z57q        160          2656         496             1224
28-SEP-2011 20:25:50      38557         985 200k2np23z57q        160          2592         816             1825
28-SEP-2011 20:31:28      20492         866 200k2np23z57q        160          2592         816             1706

We now fetch the next set of records and check V$SESSION and V$UNDOSTAT again.

USERNAME PROGRAM         STATUS   STATE               EVENT                       SQL_ID               SIW
-------- --------------- -------- ------------------- --------------------------- ------------- ----------
ADHOC    SQL Developer   INACTIVE WAITING             SQL*Net message from client 6fwqzurbc8y7k      35

END_TIME_CHAR          TXNCOUNT MAXQUERYLEN MAXQUERYID    ACTIVEBLKS UNEXPIREDBLKS EXPIREDBLKS TUNED_UNDORETENTION
-------------------- ---------- ----------- ------------- ---------- ------------- ----------- -------------------
28-SEP-2011 19:45:50      38118         584 200k2np23z57q        160           912         496             1425
28-SEP-2011 19:55:50      38404         452 200k2np23z57q        160          1120        1280             1292
28-SEP-2011 20:05:50      38507         479 200k2np23z57q        160          1704        1464             1320
28-SEP-2011 20:15:50      38340         383 200k2np23z57q        160          2656         496             1224
28-SEP-2011 20:25:50      38557         985 200k2np23z57q        160          2592         816             1825
28-SEP-2011 20:32:49      26370         992 6fwqzurbc8y7k        160          2592         816             1766

V$UNDOSTAT.MAXQUERYID is now that of the SQL Developer session. What if an extra 10 minutes had elapsed?

10 minutes later…

END_TIME_CHAR          TXNCOUNT MAXQUERYLEN MAXQUERYID    ACTIVEBLKS UNEXPIREDBLKS EXPIREDBLKS TUNED_UNDORETENTION
-------------------- ---------- ----------- ------------- ---------- ------------- ----------- -------------------
28-SEP-2011 19:45:50      38118         584 200k2np23z57q        160           912         496             1425
28-SEP-2011 19:55:50      38404         452 200k2np23z57q        160          1120        1280             1292
28-SEP-2011 20:05:50      38507         479 200k2np23z57q        160          1704        1464             1320
28-SEP-2011 20:15:50      38340         383 200k2np23z57q        160          2656         496             1224
28-SEP-2011 20:25:50      38557         985 200k2np23z57q        160          2592         816             1825
28-SEP-2011 20:35:50      41787         992 6fwqzurbc8y7k        160          3440         688             1833
28-SEP-2011 20:41:31      20088        1293 6fwqzurbc8y7k        160          3440         688             2134

The TUNED_UNDORETENTION is starting to climb but the number of expired blocks is unchanged because our UNDO tablespace is extending – it is now at 42M.

   FILE_ID FILE_NAME                                             MB
---------- --------------------------------------------- ----------
         5 C:\ORACLEXE\ORADATA\XE\UNDO_SMALL.DBF                 42

So let’s prevent it from extending and wait another 10 minutes before fetching the next set of rows.

alter database datafile 5 autoextend off;

END_TIME_CHAR          TXNCOUNT MAXQUERYLEN MAXQUERYID    ACTIVEBLKS UNEXPIREDBLKS EXPIREDBLKS TUNED_UNDORETENTION
-------------------- ---------- ----------- ------------- ---------- ------------- ----------- -------------------
28-SEP-2011 20:05:50      38507         479 200k2np23z57q        160          1704        1464             1320
28-SEP-2011 20:15:50      38340         383 200k2np23z57q        160          2656         496             1224
28-SEP-2011 20:25:50      38557         985 200k2np23z57q        160          2592         816             1825
28-SEP-2011 20:35:50      41787         992 6fwqzurbc8y7k        160          3440         688             1833
28-SEP-2011 20:45:50      41534        1594 6fwqzurbc8y7k        160          4608         576             2752
28-SEP-2011 20:55:50      44669        2196 6fwqzurbc8y7k        160          5200         264             2766

We can see that the number of expired blocks is falling and the number of un-expired blocks is rising. 10 minutes later I click to fetch the next set of records and…

ORA-01555: snapshot too old: rollback segment number 17 with name "_SYSSMU17$" too small
01555. 00000 -  "snapshot too old: rollback segment number %s with name \"%s\" too small"
*Cause:    rollback records needed by a reader for consistent read are
           overwritten by other writers
*Action:   If in Automatic Undo Management mode, increase undo_retention
           setting. Otherwise, use larger rollback segments

That was only after 35-40 minutes – well within reach of a normal trip to the canteen (admittedly my undo tablespace is sized modestly in order to help me prove a point).

So, here, in reverse order, are my top 3 undo related threats when giving read only access via a tool such as SQL Developer or Toad to application support staff (even though they have the best of intentions).

#3 Undo tablespace pressure.
Unbeknownst to the ad-hoc user he can quite easily cause pressure on the undo tablespace causing an AUTOEXTEND file to grow purely to satisfy their next page of data.

#2 Undo tablespace alerts
For undo tablespaces with no AUTOEXTEND files Grid Control or some other monitoring tool will alert as the number of unexpired blocks starts to fall causing unnecessary (albeit low priority) work for your first line DBAs.

#1 ORA-1555
For undo tablespaces with no AUTOEXTEND files and either a sufficiently high transaction rate or sufficiently high gap between page 1 and page X of our ad-hoc user’s query we may find that the undo required to construct page X has been overwritten. The user gets ORA-1555, thinks “no harm done” and re-starts the query. However the alert log now contains something like below and an alert is fired shortly afterwards.

Wed Sep 28 21:14:48 2011
ORA-01555 caused by SQL statement below (SQL ID: 6fwqzurbc8y7k, Query Duration=3538 sec, SCN: 0x0000.001aad4e):

If the hour is late and the company is pretty twitchy then this could involve someone getting out of bed to investigate a potentially failed batch run.

Have I missed any undo related threats?

Changing the SDU size is something to consider if you want to get the most out of your network. The situations where it will offer a notable benefit are covered in MOS ID 99715.1^[1].

The 11.2 Net Services Administrator’s Guide and MOS ID 67983.1 (Oracle Net Performance Tuning, which applies to “Oracle Net Services – Version: 8.1.7.4.0 to 11.2.0.1 – Release: 8.1.7 to 11.2″), both list two ways of setting SDU size. One method involves setting SDU in listener.ora (for a statically registered listeners) and the other setting it in sqlnet.ora.

The documentation also states:

If you have configured the listener with a list of targets in the listener.ora file, then the value for SDU in the SID_LIST element overrides the current setting in the sqlnet.ora file when using dedicated server processes.

There is a caveat in the “Purpose” section of MOS ID 67983.1 which seems worth drawing attention to:

Note: This document contains references to obsolete parameters.

If this caveat is intended to alert the reader to what I cover below then I think it would be reasonable to be more explicit. MOS ID 44694.1 also provides examples on setting SDU, but applies to “Oracle Net Services – Version: 8.0.5.0.0 to 11.1.0.6.0″.

It was the subtlties of the applicable versions in the MOS notes listed above and spotting something different in MOS ID 1292915.1 (Scan Listener, Queuesize, SDU, Ports) that led me to get a change in SDU to be effective.

When it comes to establishing the negotiated SDU I have relied on MOS ID 304235.1 and this page from Jonathan Lewis. The latter relates specifically to Oracle 7.3.4.1, but appears to still be relevant.

The system I was originally attempting to enable an increased SDU size on is an 11.2.0.2 Exadata environment, but when things didn’t go as planned I moved to XE 11.2.0.2 in order to experiment. The 11.2.0.2 testing below is from XE and the same approach has been confirmed as effective in the 11.2.0.2 Exadata environment (Enterprise Edition).

I have also carried out tests on 11.2.0.1, 11.1.0.6 and 10.2.0.5 (all Enterprise Edition) and reference to these versions is made where I have observed differences in behaviour.

Testing

1. Validate SDU size used by default (“out of the box”)

1.1. Set “TRACE_LEVEL_LISTENER = ADMIN” in listener.ora (only change to standard configuration)
1.2. Connect via listener^[2]
1.3. Searching for “sdu=” in the trace file reveals the following

2011-09-21 09:11:05.759068 : nsconneg:vsn=314, lov=300, opt=0x41, sdu=8192, tdu=65535, ntc=0xc60e
2011-09-21 09:11:05.759125 : nsconneg:vsn=314, gbl=0x1, sdu=8192, tdu=65535

Which, based the methods referenced above, shows that an SDU of 8192 bytes is used.

2. Setting SDU size in sqlnet.ora

2.1. Setting “DEFAULT_SDU_SIZE = 16384″ in sqlnet.ora (both client and server)
2.2. Connect via listener^[2]
2.3. Searching for “sdu=” in the trace file reveals the following

2011-09-21 09:22:41.897852 : nsconneg:vsn=314, lov=300, opt=0x41, sdu=8192, tdu=65535, ntc=0xc60e
2011-09-21 09:22:41.897910 : nsconneg:vsn=314, gbl=0x1, sdu=8192, tdu=65535

… Showing that an SDU of 8192 bytes is used.

2.4. “DEFAULT_SDU_SIZE = 16384″ removed from sqlnet.ora

Note: One observation during this test on 11.2.0.1, and earlier, is that when sqlnet.ora is has DEFAULT_SDU_SIZE set the client will attempt to negotiate this SDU size.

3. Setting SDU size in tnsnames.ora

3.1. tnsnames.ora updated to (DESCRIPTION = (SDU = 16384) <–snip–>)
3.2. Connect via listener^[2]
3.3. Searching for “sdu=” in the trace file reveals the following

2011-09-21 09:30:21.498865 : nsconneg:vsn=314, lov=300, opt=0x41, sdu=16384, tdu=65535, ntc=0xc60e
2011-09-21 09:30:21.498925 : nsconneg:vsn=314, gbl=0x1, sdu=8192, tdu=65535

… Showing that SDU of 16384 was requested by the client, but the server would only go to 8192.

4. Setting SDU size in listener.ora (static listener entry)

4.1. listener.ora updated to include (SID_DESC = (SDU = 16384) <–snip–>)
4.2. Connect via listener^[2]
4.3. Searching for “sdu=” in the trace file reveals the following

2011-09-21 09:54:28.324832 : nsconneg:vsn=314, lov=300, opt=0x41, sdu=16384, tdu=65535, ntc=0xc60e
2011-09-21 09:54:28.324890 : nsconneg:vsn=314, gbl=0x1, sdu=8192, tdu=65535

… Showing that SDU of 16384 was requested by the client, but the server would only go to 8192.

4.4. Searching for SDU= in the trace file reveals a number of lines, including the line below, showing the SDU parameter

2011-09-21 09:54:28.339728 : snlpcss:Spawn Oracle completed oracle   (DESCRIPTION=(LOCAL=NO)(SDU=16384)) XE.

I’m taking this as showing that the syntax is not flagged as invalid, but as shown by the previous output, it is not being honored.

4.5. SDU parameter removed from this position in listener.ora

5. Setting SDU size in listener.ora (Listener DESCRIPTION)

5.1. listener.ora updated to include (DESCRIPTION_LIST = (DESCRIPTION = (SDU = 16384) <–snip–>))
5.2. Connect via listener^[2]
5.3. Searching for “sdu=” in the trace file reveals the following

2011-09-21 10:07:08.008144 : nsconneg:vsn=314, lov=300, opt=0x41, sdu=16384, tdu=65535, ntc=0xc60e
2011-09-21 10:07:08.008203 : nsconneg:vsn=314, gbl=0x1, sdu=16384, tdu=65535

… Showing that SDU of 16384 was requested by the client, and the server could match it.

5.4. Searching for SDU= reveals a number of lines, including the line below, showing the SDU parameter

2011-09-21 10:06:39.910083 : nsgllsn:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))(SDU=16384))

I went on to experiment with combining the placement of SDU within listener.ora, but could not find a situation where placement within SID_DESC was honored.

After working through testing above, and finding my experiences did not match the documentation, I ran further tests on 10.2.0.5, which threw up some more interesting discoveries.

Simply enabling tracing and making a connection gave:

nsconneg: vsn=313, lov=300, opt=0x81, sdu=2048, tdu=32767, ntc=0x7308
nsconneg: vsn=313, gbl=0x81, sdu=2048, tdu=32767

This is expected as the default SDU in 10.2 is 2048 bytes.

Setting DEFAULT_SDU_SIZE = 8192 in sqlnet.ora gave:

nsconneg: vsn=313, lov=300, opt=0xc01, sdu=8192, tdu=32767, ntc=0x7f08
nsconneg: vsn=313, gbl=0xc01, sdu=8192, tdu=32767

OK! So now it’s working as documented and I didn’t even need to restart the listener

Setting DEFAULT_SDU_SIZE = 16384 in sqlnet.ora gave:

nsconneg: vsn=313, lov=300, opt=0xc01, sdu=16384, tdu=32767, ntc=0x7f08
nsconneg: vsn=313, gbl=0xc01, sdu=8192, tdu=32767

… and just for good measure setting DEFAULT_SDU_SIZE = 8193 in sqlnet.ora gave:

nsconneg: vsn=313, lov=300, opt=0xc01, sdu=8193, tdu=32767, ntc=0x7f08
nsconneg: vsn=313, gbl=0xc01, sdu=8192, tdu=32767

From this I’m concluding that it is not possible to increase the SDU size above 8K via the sqlnet.ora method alone. Values greater than 8192 are picked up the client, but not by the listener (at least until 11.2.0.2, where they are no longer picked up by the client either).

The next thing I wanted to test was setting the SDU size in listener.ora at 10.2.0.5. Try as I might, I could not get adding SDU to listener.ora using the syntax given in the 10.2 documentation to work, i.e.:

SID_LIST_listener_name=
  (SID_LIST=
    (SID_DESC=
     (SDU=16384)
     (SID_NAME=sales)))

However, if used the approach I’d established on 11.2, as below, the SDU size change was picked up by the listener.

LISTENER_listener_name =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (SDU = 16384)
      (ADDRESS = (PROTOCOL = TCP)(HOST = hostname)(PORT = 1521))
    )
  )

Key Points (it appears that):

1. SDU size can’t be increased above 8192 in sqlnet.ora and be honored by the listener
2. From 11.2.0.2 changing DEFAULT_SDU_SIZE in sqlnet.ora does not affect the SDU size used by the client or the listener
3. The documented location of the SDU parameter within the listener.ora file in all locations apart from MOS ID 1292915.1 (Scan Listener, Queuesize, SDU, Ports) is not effective

I am not ruling out that I’ve missed something in my testing, so if you read this blog post and have different experiences, particularly with respect to the location of SDU within listener.ora, then I would be very happy to hear from you.
________________

1. The activity I was attempting to speed up by increasing the SDU did not show an improvement even when the increased SDU was successfully implemented (assuming the method used for establishing the SDU, detail above, is appropriate in 11.2.0.2).
   
2. The connection testing to produce trace output for analysis was done using a SYSDBA connection to a database that was shutdown in order to guarantee that my connection was going via the static listener registration.