LB and DS FD overflow

Load Balancer will use tcp_connect to ping downline systems it is protecting, and since this is not a protocol understood by the app, you get that error. It also winds up leaving an orphaned FD *each* time.

If you do find a LB is the source, the solution is to reset the LB to use an acceptable protocol (http for web, ldap for DS, etc.). Most all the LB's have ready-to-use scripted options in their config-options.

Although its not a true fix, another option is to reset the idletimeout onthe DS to soemthing like 900/1800s so it can recoup the wasted FD's periodically. This is a fine temporary fix, sometimes cu's leave that situation in that state for years. But the true fix is to re-config the LB to use the proper protocol .

cron script from open solaris

For every command in the script?

http://cvs.opensolaris.org/source/xref/jds/spec-files/trunk/cron-script.sh
See if a single command experiment working first
 
> Where does the redirection go? Inside the quoted command or for zlogin?
> zlogin -l gbuild big-zone "the_command > zlogin -l gbuild big-zone "the_command" 

DS DB index out of sync-poor performance

DS is observed slower search response time on Solaris 10U3 compared to Windows. The cache size is the same on the two boxes, even though it should be bigger on solaris (windows is 32bit, solaris is 64).

This search for example shows different etimes consistently :

On Windows the search took 0.079000 seconds :

[19/Oct/2007:15:33:41 +0200] conn=2062529 op=1 msgId=2 - SRCH base="ou=qp_na02,ou=apps,ou=internal,o=ericsson" scope=2 filter="(&(objectClass=groupOfUniqueNames)(uniqueMember=UID=XBMVIVI,OU=Partners,OU=External,O=ericsson))" attrs="cn"

[19/Oct/2007:15:33:42 +0200] conn=2062529 op=1 msgId=2 - RESULT err=0 tag=101 nentries=114 etime=0.079000

however this same search took 0.320350 seconds on solaris :

[19/Oct/2007:17:02:30 +0200] conn=460 op=1 msgId=2 - SRCH base="ou=qp_na02,ou=apps,ou=internal,o=ericsson" scope=2 filter="(&(objectClass=groupOfUniqueNames)(uniqueMember=UID=XBMVIVI,OU=Partners,OU=External,O=ericsson))" attrs="cn"

[19/Oct/2007:17:02:30 +0200] conn=460 op=1 msgId=2 - RESULT err=0 tag=101 nentries=114 etime=0.320350

In the solaris errors logs, i see lines like this "candidate not found" :

[19/Oct/2007:17:02:30 +0200] - INFORMATION - conn=-1 op=-1 msgId=-1 - SRCH base="ou=qp_na02,ou=apps,ou=internal,o=ericsson" scope=2 deref=0 sizelimit=0 timelimit=0 attrsonly=0 filter="(&(objectClass=groupOfUniqueNames)(uniqueMember=UID=XBMVIVI,OU=Partners,OU=External,O=ericsson))" attrs="cn"

[19/Oct/2007:17:02:30 +0200] - INFORMATION - conn=-1 op=-1 msgId=-1 - mapping tree selected backend : unix

[19/Oct/2007:17:02:30 +0200] - INFORMATION - conn=-1 op=-1 msgId=-1 - mapping tree release backend : unix

[19/Oct/2007:17:02:30 +0200] - INFORMATION - conn=-1 op=-1 msgId=-1 - mapping tree selected backend : userRoot

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - be: 'o=ericsson' indextype: "eq" indexmask: 0x2

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - be: 'o=ericsson' indextype: "eq" indexmask: 0x2

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - be: 'o=ericsson' indextype: "eq" indexmask: 0x2

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - be: 'o=ericsson' indextype: "eq" indexmask: 0x2

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 306256 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 307389 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309528 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309532 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309538 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309542 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309726 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309866 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309877 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309878 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309881 not found

[19/Oct/2007:17:02:30 +0200] - DEBUG - conn=-1 op=-1 msgId=-1 - candidate 309898 not found

[19/Oct/2007:17:02:30 +0200] - INFORMATION - conn=-1 op=-1 msgId=-1 - mapping tree release backend : userRoot

it looks like the index is out of sync.

Zone attaching to shared storage

Using "attach -F" (ie capital F) you can attach a shared storage
to a different zone even it is not de-attached with it's original
zone, if you previously created the zone configuration via "zonecfg -z 
..." like on the original host.

Of course you must make sure that only one node is really booting the
zone at a time.

Scheme Setup

(1) install Petite Chez Scheme
(2) install eopl source code
(3) change dir

(cd "/home/test/eopl/intercept")
(3) init code

(load "chez-init.scm")

DB cache size for large DB on DS

The best results with a maximum of about 50 GB in database cache because of checkpoints. That is a concern here. But response times is over 1 second. It is caused by check pointing, reducing the db cache size may help. A very large db, and not a lot of memory.

(1) Set both the entry cache and db cache to 100 mb each offered the best performance - allowing the fs cache to do everything. This was on Solaris 10 with the latest updates for ZFS.
(2) modified the file system partitions to force direct io on the ones we didn't want being cached - this way we reserved more of the fs cache for LDAP.
(3) Like if your logs are on a separate FS then you can exclude that mount, etc. For the partition/disk that holds the database, set: "noatime" & "logging", and if "/data" is the disk that holds the database, set: tunefs -e 2097152 /data

Host name vs Security Compliance

(1) update with latest security patches and fixes for OS, Tools, dependencies
(2) ensure no hostname is embeded in any UI component
(3) no host name in URL or CLI

News Analysis

I'm doing a Master's Thesis project based on scanning web-based news
topics and analyzing their frequency. I have a table containing rows that
represent groups of news stories about a particular topic

Adding Missing Values for Random Variables

Missing values for real-valued attributes could be implemented via Laplacian smoothing, by adding one initial default value to every attribute, either the mean of available observations for the attribute, or the mean of observations for some other similar subset of attributes, including possibly the set of all attributes.

How to define a set of values

BNF is the way to define a set of values. This set of rule calls grammar ---- BNF grammar defines a set of values.

::= ()
::= (.)

ZFS and DS

MODS/WRITES with ZFS compression
It would seem that using ZFS with compression could theoretically yield some amazing write performance but I haven't done any testing myself yet. Any figures or observations here?

CACHE COMPRESSION?
I'm not sure what they meant there because I have seen that ZFS uses ARC for relinquishing memory to cache However, the memory relinquishing algorithm was less than desirable for database apps . But cache compression was to minimize the memory footprint of the DS entries within the FS cache which sounds great, if it were possible, for large databases.

Lamda Calculus

lambda calculus, also λ-calculus, is a formal system designed to investigate (1)function definition, (2)function application (3) recursion.

Lambda calculus can be used to define (1) what a computable function is. The question of whether two lambda calculus expressions are equivalent cannot be solved by a general algorithm. This was the first question, even before the (2) halting problem for which undecidability could be proved.

Lambda calculus can be called the smallest universal programming language. It consists of a single

(1)transformation rule -----variable substitution

(2) function definition scheme.

Lambda calculus is universal in the sense that any computable function can be expressed and evaluated using this formalism. It is thus equivalent to the Turning Machine formalism.

Lambda calculus emphasizes the use of transformation rules(variable substitution) and does not care about the actual machine implementing them. It is an approach more related to software than to hardware.

Smba wit Solaris Zone

touch /etc/krb5.keytab

the smb.conf file was not configured to use kerberos keytab but it will try to.

Storage Data Classification & Storage Policies

Archive solution is designed for tape and disk storage devices. NAS package provides the file level representation of data. Storage Management System requires data classification and policies.

Data classification is done based on the file system and a few other attributes. The attributes include file name, type, ownership, size, access rights and age. Classification can done automatically using customer defined categories.

The customer defined data classification categories are assigned to storage management policies. Policies control the placement of the files onto the various types of physical storage, replication, deletion, and access.

DS replication and search

Enabling or disabling the replication should not impact the search

Oracle Database and Solaris Zone

The Oracle data base product line comes in two significantly different forms:

1) Oracle has a single machine version of their data base product. It supports zone deployment today.

2) Oracle has version of their data base product that works cooperatively and concurrently on multiple nodes. This product is called Oracle RAC. Oracle RAC does not run in zones now. But it
will do so soon

POSIX Thread, Select & Poll

The model of simply using one thread per connection, while simpler to code and support,
is viable for only a few to several hundred (possibly a few thousand) threads.

From a performance perspective, select(3C) has stated in the man page (from circa Solaris 2.6 days IIRC) "poll(2) function is preferred over this function." select(3C) in Solaris is very inefficient as it has to translate in user space from the bitfields and calls poll() to do the kernel work anyway. Not to mention that recompiling an application that uses select(3C) as a 64 bit binary can result in a 500% performance drop. This occurs if they do not modify FD_SETSIZE, as this goes from 1024 in 32 bit compiles to 65536 in 64 bit compiles. So avoid select like the plague if performance is at an issue for your application. Select has been implemented as a direct syscall in several other Unix versions so code calling select(3C) will likely underperform on Solaris compared to other OSs.

Even poll(2) suffers a lot of unnecessary performance degredation issues so, if dealing with thousands of descriptors, the far more scalable /dev/poll "man -s 7d poll" or Event Completion
Framework should be used. Here's an article wrote back in 2002 comparing scalability of
poll(2) and /dev/poll :

http://developers.sun.com/solaris/articles/polling_efficient.html

And it seems /dev/poll has been considered a bit too complicated for a lot of real world uses, and also suffers some performance issues in cases of quickly changing lists of file descriptors to
monitor, so the even newer Event Completion Framework/Ports have been created to deal with both issues :

http://developers.sun.com/solaris/articles/event_completion.html
http://partneradvantage.sun.com/protected/solaris10/adoptionkit/tech/tecf.html

Both /dev/poll and Event Ports have been ported to Linux, so, whether they are POSIX standards, extentions, or just de facto standards since they are the only tools to do the job if the state of many thousands of connections must be polled, they must be considered.

SAN Disk/ Tap backup with Solaris Container

Several systems are set up as SAN media servers so that backup to tapes occurs from SAN-disk to SAN-tape rather than across a network.

The simplest approach is for each non-global zone to send its backup stream to the media server. It avoids potential security issues which must be addressed if you backup the zones' file systems from the GZ.

DS database mini cache size heristrics

dbase cache at least enough to hold the indexes. It is true that during search DS first read the indexes then the entries, but I would not make any assumption about what it means for the database cache. Usually when coding a cache algorithm. It is true for the entry cache and the changelog cache, but I do not know if it is also true for the DB cache.

When there is not enough space to add some data, the data that was not used since the oldest time are removed and the new one replace it. LRU, MRU.

So if you have not enough data for both kind of data, you will rather ends up with the most frequently used indexes and entries pages staying in the cache while the others go in and out
IMHO, There is probably no "priority" about indexes versus entries ...

In modify things are much more complex: dn2entryid index is read first then entry then all the indexes are read.

System Call vs. POSIX Thread on Select & Poll

Solaris System calls select() (BSD) poll() (AT&T) are _not_ POSIX. BSD and AT&T unix used to have their own ways on this. POSIX threads is recommended than relying on select() or poll() if you really want to write POSIX compliant code. With POSIX threads you should be able to evade the need for using these two system calls. Alternatively you could supply an extra .c file which contains the call to the poll() system call; this file must be translated with the _POSIX_C_SOURCE macro undefined. (You will be calling poll() even if you call select(). As Solaris 2.x is derived from AT&T, poll(2) is used to implement select(3C)).