Re: [coldsync-hackers] ColdSync 1.5.0 snapshot (and RFC)

[John-David Smith <jdsmith@astro.cornell.edu>]

Andrew Arensburger wrote:
> 
> On Sat, 11 Nov 2000, J.D. Smith wrote:
> > >         - Added John-David Smith's Install conduits.
> >
> > Thanks for including this, Andrew.  I've been using it with success for
> > over a month (though I've yet to test if your main() redesign might have
> > broken anything).
> 
>         I guess I should point out that I tentatively deleted your install
> mode, when I realized that it was essentially the same as restore mode.
>         Or would there be a point in having a mode that installs files,
> but runs pre-Install conduits on them first? (but doesn't perform a full
> Sync). Perhaps a sysadmin could use this to initialize a user's Palm?

I thought restore mode was to restore everything from a directory, as in the
case of a hard reset.  My install mode was for when you've just downloaded a new
program, and you want to install it directly, without doing any checking,
conduit running, syncing, etc.  Or you want to install a version of a database
with no processing etc.  This is equivalent to pilot-xfer -i.  Currently,
coldsync takes much longer because of all the "User Info" it downloads in the
first stage of any connection.  Any optimization here would be worth it.  I
switched to ColdSync from pilot-link primarily because it supports fast syncs,
which the latter does not.  The time saved is often offset by the length of time
downloading user info (some of which, I know, actually allows fast sync to
operate!).  

> 
>         One other new feature in 1.5 that I forgot to mention earlier is
> that ColdSync reads a site-wide /etc/coldsync.conf file. This file is in
> the same format as .coldsyncrc . It is intended to allow a sysadmin to
> provide hints and impose some policy (e.g., it can prevent any
> user-defined conduit from running).
> 
> > >         Question: Let's say that your site administrator has just
> > >   dropped a copy of the corporate directory in your
> > >   ~/.palm/install/AddressDB.pdb . However, you prefer to use the
> > >   SuperAddressBook application, so you write an Install conduit to
> > >   convert ~/.palm/install/AddressDB.pdb to
> > >   ~/.palm/install/SuperAB.pdb. After ColdSync runs the Install
> > >   conduits for AddressDB.pdb, should it also run the Install conduits
> > >   for SuperAB.pdb ? Or should it assume that if you just created it,
> > >   then you know what you're doing?
> >
> > The latter, absolutely.  Otherwise you could end up with loops.
> 
>         Depends on the implementation (see below). If install conduits are
> run on newly-created databases, then this can trigger all sorts of complex
> and wonderful behavior (and I definitely believe in giving people enough
> rope to shoot themselves, though it shouldn't be easy to do so
> accidentally).
> 
> > >         Question: currently, if an Install conduit deletes its input
> > >   database, that means that it has taken care of installing it (or
> > >   whatever). What if you have two conduits:
> > >         conduit install {
> > >                 type: Foox/Barx;
> > >                 path: /conduits/first;
> > >         }
> > >         conduit install {
> > >                 type: Foox/Barx;
> > >                 path: /conduits/second;
> > >         }
> > >   If /conduits/first deletes its input database, should
> > >   /conduits/second still run?
> >
> > Maybe try:
> >
> > this-round=all install conduits
> > LOOP UNTIL this-round is empty, or no files remain:
> > Clear "next-round".
> > Loop over install/ files (generate the list anew each time):
> > 1) if install conduit from "this-round" matches file, run it
> > 2) if another install conduit from "this-round" matches, don't run it,
> > but instead save it to a list of further conduits to run (next-round).
> > Set this-round to next-round.
> > END LOOP
> 
>         If I'm reading this correctly, it contradicts what you said
> earlier: if an install conduit creates a new file, then install conduits
> will be run on the new file.

Yes, I suppose you are correct here.  Without shenanigans, it would be hard to
implement.  I was just trying to think of how to allow multiple install conduits
to fit in this picture, but see below.

>         Secondly, the logic here is backwards from what is easily
> implementable: it's easier to read a file, then apply each matching
> conduit, than it is to look at a conduit and see which files match. But I
> understand that you weren't talking about implementation details.
>         Also, from a purely pragmatic standpoint, if the two conduits are
> ordinary Perl scripts that use ColdSync.pm, then the second one will barf
> because its input file doesn't exist.
> 
> > Another possibility: let the conduit writers deal with this.  Of course,
> > the user may just chain together two install conduits he shouldn't
> > have... it's somewhat of a tough call.
> 
>         I think this is the sanest approach: "Here's a file. Do something
> with it. If there's anything left when you're done, I'll install it." The
> conduit mechanism is flexible enough to do all sorts of reasonable (and
> unreasonable) things. So I guess the logic becomes:
> 
>         all-files = all files in ~/.palm/install;
>         for each F in all-files, do:
>                 for each conduit C, do:
>                         if C applies to F, then:
>                                 run C with input F;
>                         fi
>                 done;
>         done;
> 
>         all-files = all files in ~/.palm/install;
>         for each F in all-files, do:
>                 install F;
>         done;

This is essentially exactly how I had it coded against the older version of
ColdSync, and how I presume it operates now.  Obviously, since it's unchanged,
it doesn't address any of the concerns with the present method you brought up. 
I think we've decided that some of these concerns are unfounded, though.  One
additional technique that might help is to specifically check if multiple
install (or maybe sync also?) conduits are defined for a given id pair, and to
warn the user only one will be used, or even generate an error (probably
unecessary).  If we are encouraging users to stack fetch and dump conduits to
sync up with as many different databases as they like, we'll have to be careful
to counsel them *not* to extend this method to sync and install conduits.  Maybe
we're worrying too much.

> 
> The list of files must be generated before any install conduits are
> run. If you use the obvious implementation
>         while (readdir() != NULL)
> then you run the risk of running install conduits on databases that didn't
> exist before. But not always, and inconsistency would be bad.
> 

Even if readdir has the directory open it will be updated?  I think I tried that
and it works fine (as I recall I used this exact test in ReadNextFile).  I'm not
sure if this is guaranteed behavior, but I found it was not a problem.  Once
open, the directory remained intact.  Building the list beforehand is no big
problem though.

JD

-- 
This message was sent through the coldsync-hackers mailing list.  To remove
yourself from this mailing list, send a message to majordomo@thedotin.net
with the words "unsubscribe coldsync-hackers" in the message body.  For more
information on Coldsync, send mail to coldsync-hackers-owner@thedotin.net.