[Cook] (ingredients-)fingerprint question.
Aryeh M. Friedman
aryeh.friedman at gmail.com
Mon Nov 16 07:24:35 EST 2009
Pieter den Hertog wrote:
> I was trying to understand the effects of (ingredients-)fingerprint
> and constructed a simple
> example, hence the use of [glob *.txt]. The example entails an empty
> dir with files appearing
> one by one... so something like: (1.txt, 2.txt, ... n.txt) -> summary.
> On each file appearing
> I wanted to (re)build the summary.
Do they appear frequently (i.e. as soon one cook is done there is an
other file ready)? If not I suspect cook will be run from cron or some
other script (that is likely run by cron).... that being the case the
case of files appearing after you make your manifest is quiet unlikely
and thus there is very little reason to use non-native pattern matching
(i.e. glob or some other OS based method)... the main reason for
wanting to do it this way instead of the glob way is that cook patterns
are reversible (see the tutorial and the chapter on pattern matching in
the manual)... additionally it lets you match on specific components of
the pattern instead of all or nothing... for example even though not
specifically stated in the tutorial or the manual it is possible to have
some very powerful recipes that take advantage of this... for example
here is the default java recipe for the main project I use cook for:
obj/%0%.class: src/%0%.java
set mkdir
{
[setColor red];
echo JAVA':'Compiling [target];
[setColor white];
javac -d obj -cp [classPath] -sourcepath src src/%0%.java;
}
or here is one we use for building web pages from their components (the
company's main product is a widget based framework for making web
frameworks [with a few vertical market examples thrown in]):
site/ROOT/%0%.html: src/frontend/pages/%0%/definition
set mkdir
{
[setColor green];
echo HTML':'Creating [target];
[setColor white];
[infiniteGlue] src/frontend/pages/%0%/definition>[target];
}
(set color is a little function I wrote that sets the text color on a
ansi/vt100 term)... basically it says for all sources in our source tree
(some 1000+ classes) we do exactly the same thing....
"[infiniteGlue]" is a language we wrote for "gluing" together files from
components (think cpp on steroids).
The reason for making the recipes more general then needed is we have a
single master cook book that is shared among many projects (all
maintained by aegis as different projects)... therefor a client's site
cookbook is usually no more then:
(includes not shown):
classPath=[getClassPath backend infinteGlue isuParse thistest];
all: [html] [css] [js] db
set default
{
echo [project] built;
}
install: all
set nodefault
{
echo [project] installed to personal instance;
rm -rf site/ROOT/fragments;
touch [home]/isu2/servlet/WEB-INF/web.xml;
touch [home]/isu2/ROOT/WEB-INF/web.xml;
cp -Rv site/'*' [home]/isu2;
}
[aegis supplies the [project] and [home] vars and all the other vars are
generated by the includes not shown] (some of this needs to be
refactored and made more general [form example [home]/isu2 is rarely the
right destination dir but some other tools assume all sites live there
so we are stuck with it for now)
>
> That's when I noticed the rebuild doesn't happen if 2.txt is added,
> but after adding 3.txt
> everything works exactly as expected.
ingredients-fingerprint if I understand it correctly does not tell cook
to generate finger prints for the ingredients, it only tells it to
expect the ingredients, to have fingerprints vs. time stamps (if you
have "set fingerprint" globally this is redundant anyways)
>
> Using your suggestion, I can work around the special case (i.e. the
> second run for the
> summary target) with the following recipe:
>
> inp = [glob "*.txt"];
> summ : [inp]
> set fingerprint ingredients-fingerprint
> ["if" [in [count [inp]] 2] "then" forced]
> {
> cat [need] > [target];
> }
That gate really shouldn't be needed (in 7 years of using cook I have
only written one recipe that needed a gate and even on that one my gut
says I am doing something wrong if I need the gate at all)... Peter (or
Walt) if you're reading this would a :: work instead (i.e. tell it to
examine all the ingredients [recursivally] even if the immediate ones
are up to date)?
Namely if this is all your doing shouldn't the following cook book do it:
set fingerprint;
manifest=[match_mask %0%.txt [collect find . -name]];
summ: [manifest]
{
cat [manifest] > [target];
}
>
> So that works and my problem is solved, but it still has me puzzled
> about the
> ingredients-fingerprint: its purpose is to track a changes to the list
> of ingredients
> for which it would be necessary to record the very first
> ingredients-list used to
> build the original first incarnation of the target... right?
I don't think so... as explained above
>
> The background to all this is that I'm trying to automate part of a
> reporting process
> where cisco syslog files (after duplicate removals and some name
> mangling which totally
> destroys their timestamps) are shoved into a directory where I can
> read them.
>
> The logs are daily files and the summary should reflect a (running)
> monthly totals on
> some specific events in the network. I wanted to be selective about
> rebuild the totals
> since they take a looong time to complete.
>
> I'll have a look at aegis as well to see if that's a better fit for my
> particular challenge
> (though cook seemed a pretty straightforward tool for such a job).
>
Aegis is overkill unless you need to "rewind" the logs to any given past
state... a better use of aegis in your case would be to use it to keep
track of router configs (and any other config) as they change over
time... read Peter's article "Aegis Is Only For Software, Isn't It?"
(http://aegis.sourceforge.net/auug96.pdf)
>
>
> Thanks for the feedback!
>
>
> Aryeh M. Friedman wrote:
>> I forgot to ask last time (and I guess this time) why your using
>> [glob *.txt] instead of %.txt or %0%.txt as your ingredient list (it
>> shouldn't matter in this case but it is non-standard)
>>
>> Aryeh M. Friedman wrote:
>>> Does set force do the job?
>>>
>>> Also unless I am not sure of what your doing this seems more like a
>>> job for aegis or an other cms then for cook (I am assuming that
>>> XXX.txt are infrequently updated in the real world)
>>>
>>> pdh wrote:
>>>> In this case I'm just working with text files where the actual
>>>> content and presence
>>>> of the input files are more important than timestamps... (it is not
>>>> about building a
>>>> program)... timestamps are messed up by other tools beyond my
>>>> control...
>>>>
>>>> The recipe actually works, except for the second run ... and as far
>>>> as I can tell,
>>>> that's because the ingredients-list is not fingerprinted on the
>>>> first run.
>>>>
>>>> Using only the ingredients-fingerprint feature, I still have
>>>> strange results:
>>>>
>>>> recipe:
>>>>
>>>> summ : [glob "*.txt"]
>>>> set ingredients-fingerprint
>>>> {
>>>> cat [need] > [target];
>>>> }
>>>> then
>>>> {
>>>> function print "Ingredients-list:" [need];
>>>> }
>>>>
>>>> Starting with an empty directory with only this recipe present:
>>>>
>>>> $ echo 1 > 1.txt
>>>> $ cook
>>>> /* .cookrc.list, /home/pdh/dev/buildtools/cook, Sun Nov 15 15:06
>>>> 2009 */
>>>> cook: cat 1.txt > summ
>>>> cook: Ingredients-list: 1.txt
>>>> $
>>>> $ echo 2 > 2.txt
>>>> $ touch summ
>>>> $ cook
>>>> /* .cookrc.list, /home/pdh/dev/buildtools/cook, Sun Nov 15 15:06
>>>> 2009 */
>>>> cook: Ingredients-list: 1.txt 2.txt
>>>> cook: the "summ" file is already up to date, no action required
>>>> $
>>>>
>>>> From the documentation I expected cook to rebuild the target
>>>> (regardless
>>>> of its timestamp) because it's ingredients-list changed but that's
>>>> not the
>>>> case.
>>>>
>>>> So, it that correct behavior then? I am not a programmer so this could
>>>> very well be desired behaviour for cook as a build tool for programs.
>>>> (If so, I need to solve my problem through other means).
>>>>
>>>> Thanks
>>>>
>>>>
>>>> Aryeh M. Friedman wrote:
>>>>> Fingerprints are based on file content not on time stamp (i.e. I
>>>>> can "touch" the file and cook will still see it as the same)...
>>>>> the only way to fix this is to use time stamps not finger print on
>>>>> the particular file (I don't have the manual handy so you will
>>>>> have to look the syntex up)... but you really should be sure if
>>>>> this is what you *REALLY* want because fingerprint using actual
>>>>> file contents is 99.9999...% of the time correct (namely every
>>>>> time I think I have a problem like this it turns out to be
>>>>> something else)
>>>>>
>>>>> dwark wrote:
>>>>>> Hi List,
>>>>>>
>>>>>> I am new to cook and I'm trying to create a recipe that will
>>>>>> create a summary out of a changing set of files. The summary
>>>>>> should be rebuilt whenever a file is added/removed or updated
>>>>>> (i.e. contents changed).
>>>>>>
>>>>>> The combination of fingerprints and ingredients-fingerprint
>>>>>> looks to be a perfect match and indeed works largely as advertised.
>>>>>> Except for when the set of input files grows from 1 to 2 files,
>>>>>> for the first time.
>>>>>>
>>>>>> It looks like the ingredients-fingerprint is not taken when
>>>>>> the target is being built for the first time. The second
>>>>>> time around, cook can't see if the list has changed.
>>>>>>
>>>>>> Another thing is that fingerprinting the target itself, seems
>>>>>> to rejuvenate the target file to an age younger than it should
>>>>>> be (in the example below, summ should not be younger than 2.txt
>>>>>> during the second run).
>>>>>>
>>>>>> Am I missing something, like an additional flag or something? The
>>>>>> current workaround is to force the recipe when there are only 2
>>>>>> input files.
>>>>>>
>>>>>> Any hints/pointers would be appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Cook-users mailing list
>> Cook-users at lists.auug.org.au
>> http://lists.auug.org.au/listinfo/cook-users
>
>
More information about the Cook-users
mailing list