atime doesn't quite work because it makes a read operation into a write. Mostly that's ok, because you're just overwriting the same timestamp field over and over again. But it gets strange with filesystem snapshots or anything else that might be copy on write. Now your reads end up using disk space as it has to make a copy of each directory to update the atime - if it is in fact doing that. Traditionally, reading a snapshot just didn't update the atime (but then does the snapshot keep the atime as it was at the time of the snapshot, and reading files in the live filesystem means making a copy?). This means, as was pointed out, snapshots aren't POSIX compliant.
atime isn't updated on every access. That "relatime" mount option that's almost universally used? It updates atime when the file is written (so mtime and ctime are updated anyway, no extra cost to bumping atime), or after a day if no writes happen but reads.
> atime doesn't quite work because it makes a read operation into a write.
I remember discovering this little fact a lot of years ago. It really shook my confidence in the existence of a rational world. Then I learned about noatime and never looked back.
This really isn't that weird. Imagine you had auditing requirements for a SQL database that you must know when the last time each row was accessed. But in this universe the SQL database is your only persistent datastore (because machines might only have one partition and your system must still function when there is only one partition) so you have to store access times in the rows.
But your SQL server only has two modes, the whole database is read-only or the whole database is read-write. You really wish there was a soft read-only where only the auditing data is allowed to be updated but it doesn't exit. So now you're stuck with read-only replicas of your database being noncomplaint.
It's a problem that really has no good solution outside of the filesystem driver supporting this mode.
Off-topic: The font of this blog looks distractingly weird on Firefox + Ubuntu. For some reason the e and o letters are slightly bigger than the rest: https://i.imgur.com/W1JaiWQ.png
My suspicion is that the font rendering is slightly off for whatever font is being selected.
Right click the offending text, select "Inspect Element". The dev tools should popup. On the right hand side there should be a "Fonts" tab. If you click this it'll show you the font being used.
I'd be interested in comparing a fast CRC checksum versus their approach of stat() plus pulling a sequence number. I'm sure it's slower, but I'm curious how much slower for tens of thousands of files.
Far and away the fastest CRC on my local box is Adler's checksum that uses the Intel hardware CRC32 instruction (introduced a long time ago, almost universally supported at this date). It only needs 40ms CPU time per GB, about 50x faster than md5sum.
If you have a very new CPU you might have hardware SHA support.
It's not a contradiction to state something is not a problem most of the time. (Though the next point, calculating checksums prior to build, is much more significant.)
Related example: on a cloud instance with SSD drive, I have scripts that generate a ~10GiB file, then immediately after (while it might still be in cache), calculating an md5sum still takes tens of seconds (maybe it wasn't in cache? I've not investigated deeply). It's just an example of a case that falls outside of that "Mostly" category.
On my local system md5sum takes 1.9 user CPU-seconds per GB.
$ time dd if=/dev/zero bs=1048576 count=1024 | md5sum
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.104 s, 510 MB/s
cd573cfaace07e7949bc0c46028904ff -
real 0m2.108s
user 0m1.891s
sys 0m0.628s
If you want to try measuring something more relevant, my redo implementation comes with a cubehash tool that uses the same hash as redo-ifchange et al. do.
It's annoying, we all ran into it, but usually if you get weird results you just make clean and make again and move on. Deployments should always build from scratch.
All those fancy checks plus database could be added to make too if you really wanted. It still wouldn't support all the fancy features of redo, but improve your experience with all those existing projects based on make.
You don't have to build from scratch if you have high confidence that an incremental build is correct. You get this from modern build systems like Bazel.
Well, getting "weird results" is one particularly strong hint that the incremental build is not, in fact, correct. Weird linker errors, or better yet, weird resulting binaries. "I've just changed this routine and built the app, but when I run it the behaviour is the same?.. <attaches gdb> WTF, it has the old version of the routine! kill, q, make clean && make && gdb run, ah, now it works. Apparently, computers are haunted by the angry ghosts".
I've done this for long enough to not have that confidence. It's not just whether the build system is correct, but also all the external things you do with it, random files laying around that aren't referenced by the build system but make it into the release bundle and mess up things at runtime, moon phase, whatnot.
Even if Bazel is the ultimate, final, perfect build system, I don't trust everyone, including myself, to use it properly.
I’ve used Bazel for long enough that I’ve regained that confidence. I don’t think it’s some “ultimate, final, perfect build system”, but it is a build system that doesn't have insidious failure modes when used improperly. Execution of build rules is sandboxed by default so you generally don’t run into the problem of implicit dependencies screwing up your builds—not that you can’t, but you generally don’t.
If I were running a one man show I might actually try, but debugging build problems when you're not the (only) author of the Makefile (equivalent) just isn't fun.
You have to go far out of your way to make bazel not have implicit system deps, don't you? The default toolchain is to just call whatever cc is in the path, or whatever is in CC environment variable. Working on the toolchain rules / crosstool is very little practiced art and severely under-documented.
It is underdocumented, but you don’t have to go out of your way because you in most cases you can use prepackaged toolchains that other people have made.
I have successfully written my own toolchain rules in Bazel. Yes, it is a pain. But it’s usually only necessary if you have some kind of unusual target—usually, someone else has written the toolchain rules for you.
I've even witnessed a Bazel build system that made working DOS binaries. The entire set of rules was smaller than you might think.
It looked to me like putting toolchain and bespoke tool dependencies into Bazel rules was about as manual as any other build system (e.g., next to impossible in a cross-platform manner, and error-prone at best).
The correct response for a compiler upgrade or a modification to a bespoke tool that generates a product is almost certainly a full rebuild unless you have very high confidence that you've identified ALL the dependencies. I believe this is not easily discoverable (e.g., you might have a dependency on "/usr/bin/cc", but not know enough about 'cc's internals to realize that you also have to have a dependency on "/usr/local/lib/cc1-arm-bfhsh-37.so" for some specific compiler option).
I'd like to believe you, but I have scars from doing build systems and discovering all the wacky and wonderful things that tools do.
Bazel sandboxes the build rules. If you need to discover dependencies, you can flip a switch and have all the missing dependencies be hard errors. Once you flip the switch and fix the errors, you leave it flipped on and you have confidence that you are not missing any dependencies. The entire C toolchain is normally an input to a Bazel build. You make a package (.tar.gz) containing the toolchain, put it somewhere accessible (http), and check a SHA-256 hash of the toolchain into your WORKSPACE file. When it changes, everything built with it is rebuilt.
Getting complicated cross-platform toolchains working on Bazel is going to be hard, sure. But knowing whether you have hidden dependencies is not hard, because of the sandboxing.
It used to be the case that file hashing wasn't coordinated well with a concurrently edited file, so changing source while building corrupts your cache.
It is customary to have various optimizations in modify/save/rebuild/execute/verify loop.
I, for one, don't even restart the app for most changes and use REPL to modify application at run time. I only rebuild it from scratch and restart from time to time to verify it all still works as intended.
Because when you rebuild and redeploy you loose the state that you have in your application and then you need to put it in the correct state for you to test if your change worked correctly.
Here is Shaun Mahood explaining this while building an application to do the presentation without even having to reload the page: https://www.youtube.com/watch?v=cDzjlx6otCU
Production builds don't have to be a full build if you have high confidence that an incremental build is correct. You can achieve this in practice with Bazel and other similar build systems. I would have agreed with you five years ago.
Depending on how long it takes to do a full build versus an incremental build, doing incremental builds for releases can make a massive improvement in developer experience. Just speaking from my own experiences here. Large C++ codebases in particular suffer from long build times.
With the disclaimer that I don't know the Bazel and I have started over 20 years ago, I still think incremental builds are much more important for developer efficiency especially in environments and languages where they have to rebuild entire binary.
Methods that can help you achieve that goal faster are more valuable than ones that are completely correct but require more time to execute.
Most massive improvement in developer experience is if you can get feedback immediately without lapse in your concentration.
Production build for me is something that happens in the background after I have pushed my code and moved to work on something else and so it has very little effect on my productivity. It is nice having it complete in seconds but it does not hurt my productivity significantly even if it takes a whole day, as long as I can continue getting feedback on my current coding task (ie. I have working local environment where I can fully test with little overhead0.
At the most extreme example, I have worked on a credit card terminal where you needed a whole week to get the application built and deployed on a real device because it had to be built by a special compiler on another host and then once it builds it had to get approvals and get signed by external company.
It of course did not affect my productivity in the long run as I found ways to quickly compile and run the functionality outside the real device.
Bazel, Buck, et al are for large monorepo code bases, where recompiling and retesting the entire code base for every changes is prohibitively expensive, even for CI.
> Annoyingly, when you update the content of a file, the mtime of its containing directory is not changed. [...] And anyway, purists might argue that the "content" of a directory doesn't change when the files it points to change; the content is merely a list of filenames and inode numbers, after all, and those stay the same, no matter what happens inside those inodes. Purists make me sad.
I don’t think it’s just about purism. For some it might be. But to me, ever since I learned some of the details about how for example the FreeBSD VFS is implemented, from reading the book The Design and Implementation of the FreeBSD Operating System by M. K. McKusick, it makes sense. Prior to that my own mental model of how file systems work on for example FreeBSD was very fuzzy. But reading about the technical details, like what kinds of structs are being used in the VFS and what fields those structs have, I realized why a lot of things are the way that they are.
In the same paragraph that I quoted above, in the part I omitted for relevance, he also said:
> All sorts of very convenient tree traversals would be possible if the directory mtime were updated (recursively to the root) when contained files changed, but no. This is probably because of hardlinks: since the kernel doesn't, in general, know all the filenames of an open file, it literally cannot update all the containing directories because it doesn't know what they are either.
On this I have two thoughts.
1. Doing this means adding more overhead every time a file is written to. The overhead might be negligible, or it might be significant. We’d have to run a lot of benchmarks to really know. (For me the answer is that with the mental model of the file system that I have as mentioned above, I don’t desire this anyways, so I am not personally even going to do that type of benchmarking.)
2. Consider that one or more directories in the path between the root and your current working directory may be mounted from a file system different from the root file system. They might not even be local to the system! For example, perhaps your home directory is mounted over NFS or SMB. That’s certainly common in a lot of deployments at universities and big companies. From a philosophical point of view, does it even make sense to modify the mtime of every directory from the root down to your current working directory when you modified a file that is being written to a storage medium that isn’t even local to your system? And from a practical point of view, what good does it do that the desktop computer on desk A in floor 2 had all of its mtimes in its path modified when you head on up to floor 3 and sit down at the computer on desk K? Any tools you have that were relying on being able to start at the root file system and scan mtimes to find modified files would end up sometimes picking up changes (because someone else had recently modified another file in their respective home directory while logged onto computer 3K) and sometimes not (because at other times no one had been changing any files in their home directories between the time that you last did so on computer 3K and when you came back to it after having worked on your files from computer 2A).
As a workaround you might say, okay so let’s stop at file system boundaries. And that might work out, but it’s a bit hairy.
And still it could be that a longer component of the remote path is sometimes mounted and sometimes a shorter component. Then again, if the remote system is also doing this recursive mtime modification then that by itself is not a problem.
Furthermore, as a consequence of this, in order to make things workable, you should also be changing mtimes of all parent directories when mounting any file system to the most recent mtime of the file system you are mounting, if it is more recent than the mtime of the directory you are mounting it on in order to make the change visible. And if that makes sense then it means that mounting and unmounting file systems are con...
Yeah, overall this article had a lot of great points and insights, but I think they oversimplified on this. (Not a big deal since it is a side issue.)
It's easy to look at a piece of software and think that, because it doesn't do what you want in a particular situation, it's a bad design. That's not the way you determine whether something is a good design; instead, you have to look at the big picture.
Usually, each different design option will have its own pluses and minuses and will make certain things easier and certain other things harder. The only look at your own situation, you're sort of cherry picking the data.
You could solve item #2 by simply having the kernel lie about the recursive parents of each mount point. Basically, when you mount a filesystem, anyone asking for the mtime of a parent (even across filesystems) will return max(real_mtime, root_of_mountpoint_mtime). When you unmount it, the parent mtimes might jump backwards, but that's fine, since we've established that mtime sometimes goes backwards anyway. Any change in mtime is sufficient to trigger a change detection according to the rules in the post.
> Random side note: on MacOS, the kernel does know all the filenames of a hardlink, because hardlinks are secretly implemented as fancy symlink-like data structures.
Note that the link is outdated and refers to HFS+. HFS+ originated in classic Mac OS and originally didn't support hard links at all, so they had to be added after the fact [1] using a hacky format based on a hidden directory. APFS has a cleaner design, but it still tracks the list of hard links to an inode. [2]
> when you update the content of a file, the mtime of its containing directory is not changed ... if the directory mtime were updated (recursively to the root) when contained files changed ... This is probably because of hardlinks
It might be partly due to hardlinks, but I suspect it is mostly an efficiency issue. Modifying a file could mean updating quite a few directories every time if you have a deep tree. For some uses this could be quite a performance hit, and for some media would potentially reduce the life of your hardware too.
IIRC even when updating atime for every access was the usual default for many filesystems, only the current object had its time updated not all its chain of parents.
Yes imagine recording a video stream, you'd have to update every directory above a file every few mS when files are written, and because file metadata usually gets priority on a spinning disk the heads would go crazy and throughput would go way down - probably couldn't keep up with the incoming stream
You're assuming the inodes of all the parent directories would be flushed to disk every time they change. This is already not the case. It would be just a matter of touching the data structure in memory and eventually flushing it out.
You are assuming a modern filesystem and/or cacheing architecture in a machine with enough memory to spare to run it effectively (though in fairness the post you replied to directly did mention a fairly modern use case). Back when the behaviours being discussed became the norm the landscape was pretty different.
A point that neither this article nor the recent ninja retrospective (http://neugierig.org/software/blog/2020/05/ninja.html) touch on though is that hashing of files is an important step if you'd like to keep caches locally, but especially if you'd like to keep caches remotely.
A key reason that Bazel-alikes (including http://www.pantsbuild.org, which I work on) hash things is for use in cache keys. And many of those systems use file watching facilities (notify, fsevents) to determine whether files need to be rehashed. If done properly (i.e.: no false negatives), this is significantly faster than checking file mtimes, and it allows for "early cutoff", where if after hashing the digest of a file has not actually changed, you can skip rebuilding its dependents.
I agree. I would estimate 90% of all Linux packages (over all distros) are based on make. Well, for the distro it does not matter, because packages are typically built from scratch and not incrementally. But everybody hacking on such a package will be affected by the mtime issues. I would claim it hardly ever happens. Of course if it happens it might cause great frustration and waste of half a day.
The article describes a third way which is neither mtime nor hashes. It's basically a database of (mtime, size, inode number, file mode, owner, group) tuples.
So it still requires keeping a database. But because all of that would come out of inodes, it avoids all the I/O of reading the files' contents.
Compared to mtime, it's harder to implement, but more resilient and at a similar cost. Compared to hashes, it's almost as resilient but much faster.
bitbake (openembedded and Yocto) does not use timestamps, but hashes all inputs. Well, it's a bit different scope. It builds packages and each package use make (or some replacement depending on package maintainer's choice) internally. So the number of files to checksum is small even so if their size can be big, typically mostly the source tarball of the package.
Sun used to have a version of make that WOULD keep track of commandlines, and would rebuild targets if they changed.
Meaning with the command:
foo.o: foo.c
cc -o foo.o $(CFLAGS) foo.c
make would save away "cc -o foo.o -g -DFOO foo.c" as part of the rebuild check.
Also, I was under the (mistaken?) impression that some make actions came from a dependency tree, not a timestamp. Otherwise everything would be breaking during any invocation.
One other point about ATIME -- turn it off if you value your SSD.
63 comments
[ 2.3 ms ] story [ 138 ms ] threadIsn't POSIX fun?
I remember discovering this little fact a lot of years ago. It really shook my confidence in the existence of a rational world. Then I learned about noatime and never looked back.
But your SQL server only has two modes, the whole database is read-only or the whole database is read-write. You really wish there was a soft read-only where only the auditing data is allowed to be updated but it doesn't exit. So now you're stuck with read-only replicas of your database being noncomplaint.
It's a problem that really has no good solution outside of the filesystem driver supporting this mode.
* https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1...
Turning every read into a write......there are very few cases that would be preferrable for.
Does anyone have an idea what's going on there?
Right click the offending text, select "Inspect Element". The dev tools should popup. On the right hand side there should be a "Fonts" tab. If you click this it'll show you the font being used.
That's a much easier way to know something's changed if you're going to the trouble of keeping state.
"Checksumming every output after building it is somewhat slow."
"Mostly this is not too serious: the file is probably already in disk cache (since you just wrote it a moment ago!)"
> Checksumming every input file before building is very slow
If you have a very new CPU you might have hardware SHA support.
Related example: on a cloud instance with SSD drive, I have scripts that generate a ~10GiB file, then immediately after (while it might still be in cache), calculating an md5sum still takes tens of seconds (maybe it wasn't in cache? I've not investigated deeply). It's just an example of a case that falls outside of that "Mostly" category.
Edit: "sum -s" is faster for me, but the man page doesn't give much info on what the algorithm is.
* http://jdebp.uk./Softwares/redo/
All those fancy checks plus database could be added to make too if you really wanted. It still wouldn't support all the fancy features of redo, but improve your experience with all those existing projects based on make.
Even if Bazel is the ultimate, final, perfect build system, I don't trust everyone, including myself, to use it properly.
I have successfully written my own toolchain rules in Bazel. Yes, it is a pain. But it’s usually only necessary if you have some kind of unusual target—usually, someone else has written the toolchain rules for you.
I've even witnessed a Bazel build system that made working DOS binaries. The entire set of rules was smaller than you might think.
The correct response for a compiler upgrade or a modification to a bespoke tool that generates a product is almost certainly a full rebuild unless you have very high confidence that you've identified ALL the dependencies. I believe this is not easily discoverable (e.g., you might have a dependency on "/usr/bin/cc", but not know enough about 'cc's internals to realize that you also have to have a dependency on "/usr/local/lib/cc1-arm-bfhsh-37.so" for some specific compiler option).
I'd like to believe you, but I have scars from doing build systems and discovering all the wacky and wonderful things that tools do.
Getting complicated cross-platform toolchains working on Bazel is going to be hard, sure. But knowing whether you have hidden dependencies is not hard, because of the sandboxing.
Like https://github.com/bazelbuild/bazel/issues/3360 .
It used to be the case that file hashing wasn't coordinated well with a concurrently edited file, so changing source while building corrupts your cache.
Production builds should be a full build.
It is customary to have various optimizations in modify/save/rebuild/execute/verify loop.
I, for one, don't even restart the app for most changes and use REPL to modify application at run time. I only rebuild it from scratch and restart from time to time to verify it all still works as intended.
Here is Shaun Mahood explaining this while building an application to do the presentation without even having to reload the page: https://www.youtube.com/watch?v=cDzjlx6otCU
Depending on how long it takes to do a full build versus an incremental build, doing incremental builds for releases can make a massive improvement in developer experience. Just speaking from my own experiences here. Large C++ codebases in particular suffer from long build times.
Methods that can help you achieve that goal faster are more valuable than ones that are completely correct but require more time to execute.
Most massive improvement in developer experience is if you can get feedback immediately without lapse in your concentration.
Production build for me is something that happens in the background after I have pushed my code and moved to work on something else and so it has very little effect on my productivity. It is nice having it complete in seconds but it does not hurt my productivity significantly even if it takes a whole day, as long as I can continue getting feedback on my current coding task (ie. I have working local environment where I can fully test with little overhead0.
At the most extreme example, I have worked on a credit card terminal where you needed a whole week to get the application built and deployed on a real device because it had to be built by a special compiler on another host and then once it builds it had to get approvals and get signed by external company.
It of course did not affect my productivity in the long run as I found ways to quickly compile and run the functionality outside the real device.
I don’t think it’s just about purism. For some it might be. But to me, ever since I learned some of the details about how for example the FreeBSD VFS is implemented, from reading the book The Design and Implementation of the FreeBSD Operating System by M. K. McKusick, it makes sense. Prior to that my own mental model of how file systems work on for example FreeBSD was very fuzzy. But reading about the technical details, like what kinds of structs are being used in the VFS and what fields those structs have, I realized why a lot of things are the way that they are.
In the same paragraph that I quoted above, in the part I omitted for relevance, he also said:
> All sorts of very convenient tree traversals would be possible if the directory mtime were updated (recursively to the root) when contained files changed, but no. This is probably because of hardlinks: since the kernel doesn't, in general, know all the filenames of an open file, it literally cannot update all the containing directories because it doesn't know what they are either.
On this I have two thoughts.
1. Doing this means adding more overhead every time a file is written to. The overhead might be negligible, or it might be significant. We’d have to run a lot of benchmarks to really know. (For me the answer is that with the mental model of the file system that I have as mentioned above, I don’t desire this anyways, so I am not personally even going to do that type of benchmarking.)
2. Consider that one or more directories in the path between the root and your current working directory may be mounted from a file system different from the root file system. They might not even be local to the system! For example, perhaps your home directory is mounted over NFS or SMB. That’s certainly common in a lot of deployments at universities and big companies. From a philosophical point of view, does it even make sense to modify the mtime of every directory from the root down to your current working directory when you modified a file that is being written to a storage medium that isn’t even local to your system? And from a practical point of view, what good does it do that the desktop computer on desk A in floor 2 had all of its mtimes in its path modified when you head on up to floor 3 and sit down at the computer on desk K? Any tools you have that were relying on being able to start at the root file system and scan mtimes to find modified files would end up sometimes picking up changes (because someone else had recently modified another file in their respective home directory while logged onto computer 3K) and sometimes not (because at other times no one had been changing any files in their home directories between the time that you last did so on computer 3K and when you came back to it after having worked on your files from computer 2A).
As a workaround you might say, okay so let’s stop at file system boundaries. And that might work out, but it’s a bit hairy.
And still it could be that a longer component of the remote path is sometimes mounted and sometimes a shorter component. Then again, if the remote system is also doing this recursive mtime modification then that by itself is not a problem.
Furthermore, as a consequence of this, in order to make things workable, you should also be changing mtimes of all parent directories when mounting any file system to the most recent mtime of the file system you are mounting, if it is more recent than the mtime of the directory you are mounting it on in order to make the change visible. And if that makes sense then it means that mounting and unmounting file systems are con...
Yeah, overall this article had a lot of great points and insights, but I think they oversimplified on this. (Not a big deal since it is a side issue.)
It's easy to look at a piece of software and think that, because it doesn't do what you want in a particular situation, it's a bad design. That's not the way you determine whether something is a good design; instead, you have to look at the big picture.
Usually, each different design option will have its own pluses and minuses and will make certain things easier and certain other things harder. The only look at your own situation, you're sort of cherry picking the data.
Note that the link is outdated and refers to HFS+. HFS+ originated in classic Mac OS and originally didn't support hard links at all, so they had to be added after the fact [1] using a hacky format based on a hidden directory. APFS has a cleaner design, but it still tracks the list of hard links to an inode. [2]
[1] http://www.wsanchez.net/papers/USENIX_2000/
[2] https://developer.apple.com/support/downloads/Apple-File-Sys..., search for "Siblings"
It might be partly due to hardlinks, but I suspect it is mostly an efficiency issue. Modifying a file could mean updating quite a few directories every time if you have a deep tree. For some uses this could be quite a performance hit, and for some media would potentially reduce the life of your hardware too.
IIRC even when updating atime for every access was the usual default for many filesystems, only the current object had its time updated not all its chain of parents.
A point that neither this article nor the recent ninja retrospective (http://neugierig.org/software/blog/2020/05/ninja.html) touch on though is that hashing of files is an important step if you'd like to keep caches locally, but especially if you'd like to keep caches remotely.
A key reason that Bazel-alikes (including http://www.pantsbuild.org, which I work on) hash things is for use in cache keys. And many of those systems use file watching facilities (notify, fsevents) to determine whether files need to be rehashed. If done properly (i.e.: no false negatives), this is significantly faster than checking file mtimes, and it allows for "early cutoff", where if after hashing the digest of a file has not actually changed, you can skip rebuilding its dependents.
So it still requires keeping a database. But because all of that would come out of inodes, it avoids all the I/O of reading the files' contents.
Compared to mtime, it's harder to implement, but more resilient and at a similar cost. Compared to hashes, it's almost as resilient but much faster.
Meaning with the command:
make would save away "cc -o foo.o -g -DFOO foo.c" as part of the rebuild check.Also, I was under the (mistaken?) impression that some make actions came from a dependency tree, not a timestamp. Otherwise everything would be breaking during any invocation.
One other point about ATIME -- turn it off if you value your SSD.
1. The target doesn't exist
2. Any children are dirty
3. The mtime of any children is more recent than the mtime of the target.
All recursively propagated from leaves up.