« Laid Low by the Lowly Apostrophe - A Javascript Story | Main| DC Lotus User Group Meeting Thursday 6:30PM »

Can a Notes database "really" be excluded from the Catalog?

Category

I put this out in the Notes 6/7 Forum but figured I'd shake the trees here as well...

Regarding the database/domain catalog (catalog.nsf), I know that even databases marked "do not list in catalog" are still listed anyway (just hidden by view selection formulas). What I haven't been able to figure out is whether a database that is locked down in some way so the server can't "see" it will still show up in the catalog. Maybe if the ACL locks out the server or the database is encrypted, the catalog task will have to skip over it.

In short, are there any circumstances where a notes database (.nsf, .ntf, .ns*) on a server might not show up in the Catalog database?

The reason I ask is that I need to audit ALL the databases on the server and would like to rely in the Catalog for obvious reasons. If I can't, I will need to recursively crawl the file system with my own agent in order to flag anything the catalog missed.

Thanks!

Kevin

Update: Richard Schwartz pointed out here that if a database is encrypted with any id other than that of the server on which it resides, the database might as well not be there at all. Therefore, if the catalog task overlooks such a database, no biggie since it can't be used anyway. Meanwhile, Paul Mooney and I tested the scenario where the ACL of a database is specifically set to deny the server access, and the Enforce Consistent ACL flag is set, to see if the catalog task would pick it up anyway, and it did. I don't know how much more locked down you can make an ACL, so my conclusion is that YES, the Catalog database (at least with respect to the databases on the local server) can be relied on to include ALL databases without (meaningful) exception. If you're dealing with cross-server/Domain Catalog situations this may not always be true, but in this case I'm not.

Update 2: Jack Dausman has pointed out an interesting scenario (see the comments) that probably wouldn't affect very many actual production environments, but nevertheless should probably become a standard item on any Domino infrastructure audit checklist. I will probably look into running a separate agent specifically for identifying Notes databases with non-standard suffixes.

Comments

1 - Hi Jack,

I didn't see your comment until I posted my update, so I guess I'll have to amend it...

I guess the possibility exists that there may be a database out there with a non-standard suffix, but very unlikely on our production servers.

Nevertheless, if I had to locate such a database, I guess I would simply loop through the file system recursively and test each file that doesn't look like a database to see if I can put it into a NotesDatabase object, and if so, flag it, otherwise move on. Since there are lots of non-database files (.gifs etc.) under the Domino Data directory, there would be a lot of noise to sift through, but I figure it wouldn't take too long.

2 - Hi Philip,

Thanks very much for the additional insight. I had not been clear before digging into this issue that the catalog was so important for cross-database access via replica id. I happen to be one developer that generally avoids such links in favor of configurable pointers to other databases by filepath. I could be wrong, as it may only apply to the first time a database is accessed on a server, but I've always thought looking up a database by replica id was slower. I would certainly welcome clarification on that point.

As for the whole file suffix issue, I'm sure Jack and I both agree with you that there is no legitimate reason to use non-standard file suffixes. There is certainly no conscious attempt that I am aware of to achieve "security by obscurity" as you say...the issue here is ensuring that any inadvertent goofs in the filename of a "real" database are picked up by any sort of database audit.

The approach I outlined in @2 above should probably do the job of finding such exceptions. I would of course need to "seed the waters" with some actual malformed databases to ensure it picked them up successfully. The one potential "gotcha" that concerns me is how to account for directory and database links (e.g. *.dir files that appear as directories in the open database dialog, but are actually text files containing an entirely different pathname where databases are stored). I suppose in the (presumably) rare instances where these are encountered, a manual check of that directory's contents would do the trick, since all "normal" databases should be picked up in the catalog already.

3 - Kev, technically, it is possible to have a Notes database not picked up by the catalog tasks.

If I choose a non-standard suffix (talk.xyz) then, the client will still connect to "talk.xyz" (no http) but it won't be recognized by the catalog task.

Also, if I remove the suffix ("talk") and then access it as "talk." (note the period), then the client can access a Notes database without any suffix.

So, what's the likelyhood that you have databases without a correct suffix or even missing a suffix?

4 -

5 - I feel I should probably point out that non-{NSF|NTF} extensions are pretty much a very daft idea, except for mailboxes. (Which, you'll note, don't show up in catalog.nsf either.)

The main reason for this is that developers often use the replica ID to open other databases. For obvious reasons, this is a smart thing to do - it means that the app can be moved from one server to another, for instance, and yet not care about file paths.

Replica IDs are translated to database paths by following theses steps:
1. The server reads the database cache and looks for the replica ID.
2. The server checks the catalog.nsf
3. If it's not found it yet, te server gives up and returns an error.

(I first tested this with R4, and it's notchanged in any version I've tested since - although I've not tested R7 yet... Hmm.)

That's why sometimes you create a database on a server, restart the server and then can't see the database when using Replica IDs. Running catalog solves the problem, as does opening the database manually by specifying a path\filename in the File -> Database -> Open dialogue box, because this puts it in the server's database cache.

(I think this is also why the Notes client always uses the filenames to open a user's mail database, rather than the replica ID. Otherwise, for the first day of its life a new Notes account couldn't read its own mail! And the router always uses the filenames too, for similar reasons. )

Obviosuly, neither technique is perfect - filenames in an application survive an unexpected server restart before catalog has run, and replica IDs survive file locations moving. But most developers and admins I've known prefer replica IDs, as they're a little more robust and flexible - we work around the problems either by running catalog manually, or creating replicas on new servers a day ahead of schedule to make sure that the Db is serviceable the next day.

For this reason, I believe that any non-{NSF|NTF} dbs are probably broken anyway - except for mailboxes, that is! If you find anything that isn't a mailbox, then you should seriously consider deleting the damned thing - or beating whoever put it there with a big stick!

Security by obscurity like this is no security at all, and will only complicate things when you need to move hardware or try to implement clustering. (What would this do to clustering, anyway? Ooh. Nasty. I bet that's not a supported configuration!)

Still, good luck in finding them all!

Your Host

KevinPettitt.jpg
Kevin Pettitt View Kevin Pettitt's profile on LinkedIn

Tools I Use

Idea Jam

Subscribe to This Blog

 Full Posts  Comments

MyYahoo
netvibes Add to Netvibes

Contact

Hosted by

OpenNTF

Disclaimer

This site is in no way affiliated, endorsed, sanctioned, supported, nor blessed by Lotus Software nor IBM Corporation, nor any of my past or future clients (although they are welcome to do so). The opinions, theories, facts, etc. presented here are my own and in no way represent any official pronouncement by me on behalf of any other entity.

© 2005-2017 Kevin Pettitt - all rights reserved as listed below.

Creative Commons License
Unless otherwise labeled by its originating author, the content found on this site is made available under the terms of an Attribution / NonCommercial / ShareAlike