tools/licenses/lib/README - mirrors/engine - Git at Google

 This license collection script is, fundamentally, one giant pile of
 special cases. As such, while there is an attempt to model the rules
 that apply to licenses and apply some sort of order to the process,
 the code is less than clear. This file attempts to provide an overview.

 main.dart is the core of the operation. It first walks the entire
 directory tree starting from the root of the repository (which is to
 be specified on the command line as the only argument), creating an
 in-memory representation of the project (make sure to run this only
 after you've run gclient sync, so that all dependencies are on disk).
 This is the step that is labeled "Preparing data structures".

 Then, it walks this in-memory representation, attempting to assign to
 each file one or more licenses. This is the step labeled "Collecting
 licenses", which takes a long time.

 Finally, it prints out these licenses.

 The in-memory representation is a tree of RepositoryEntry objects.
 There's three important types of these objects: RepositoryDirectory
 objects, which represent directories; RepositoryLicensedFile, which
 represents source files and resources that might end up in the binary,
 and RepositoryLicenseFile, which represents license files that do not
 themselves end up in the binary other than as a side-effect of this
 script.

 RepositoryDirectory objects contain three lists, the list of
 RepositoryDirectory subdirectories, the list of RepositoryLicensedFile
 children, and the list of RepositoryLicenseFile children.

 RepositoryDirectory objects are the objects that crawl the filesystem.

 While the script is pretty conservative (including probably more
 licenses than strictly necessary), it tries to avoid including
 material that isn't actually used. To do this, RepositoryDirectory
 objects only crawl directories and files for which shouldRecurse
 returns true. For example, shouldRecurse returns false for ".git"
 files.

 Some directories and files require special handling, and have specific
 subclasses of the above classes. To create the appropriate objects,
 RepositoryDirectory calls createSubdirectory and createFile to create
 the nodes of the tree.


 The low-level handling of files is done by classes in filesystem.dart.
 This code supports transparently crawling into archives (e.g. .jar
 files), as well as handling UTF-8 vs latin1. It contains much magic
 and hard-coded file names and so on to handle distinguishing binary
 files from text files, and so forth.

 This code uses the cache described in cache.dart to try to avoid
 having to repeatedly reopen the same file many times in a row.


 In the case of a binary file, the license is found by crawling around
 the directory structure looking for a "default" license file. In the
 case of text files, though, it's often the case that the file itself
 mentions the license and therefore the file itself is inspected
 looking for copyright or license text. This scanning is done by
 determineLicensesFor() in licenses.dart.

 This function uses patterns that are themselves in patterns.dart. In
 this file we find all manner of long complicated and somewhat crazy
 regular expressions. This is where you see quite how absurd this work
 can actually be. It is left as an exercise to the reader to look for
 the implications of many of the regular expressions; as one example,
 though, consider the case of the pattern that matches the AFL/LGPL
 dual license statement: there is one file in which the ZIP code for
 the Free Software Foundation is off by one, for no clear reason,
 leading to the pattern ending with "MA 0211[01]-1307, USA".


 The license.dart file also contains the License object, the currently
 simplistic normalizer (_reformat) for license text (which mostly just
 removes comment syntax), the code that attempts to determine what
 copyrights apply to which licenses, and the code that attempts to
 identify the licenses themselves (at a high level), to make sure that
 appropriate clauses are followed (e.g. including the copyright with a
 BSD notice).
	This license collection script is, fundamentally, one giant pile of
	special cases. As such, while there is an attempt to model the rules
	that apply to licenses and apply some sort of order to the process,
	the code is less than clear. This file attempts to provide an overview.

	main.dart is the core of the operation. It first walks the entire
	directory tree starting from the root of the repository (which is to
	be specified on the command line as the only argument), creating an
	in-memory representation of the project (make sure to run this only
	after you've run gclient sync, so that all dependencies are on disk).
	This is the step that is labeled "Preparing data structures".

	Then, it walks this in-memory representation, attempting to assign to
	each file one or more licenses. This is the step labeled "Collecting
	licenses", which takes a long time.

	Finally, it prints out these licenses.

	The in-memory representation is a tree of RepositoryEntry objects.
	There's three important types of these objects: RepositoryDirectory
	objects, which represent directories; RepositoryLicensedFile, which
	represents source files and resources that might end up in the binary,
	and RepositoryLicenseFile, which represents license files that do not
	themselves end up in the binary other than as a side-effect of this
	script.

	RepositoryDirectory objects contain three lists, the list of
	RepositoryDirectory subdirectories, the list of RepositoryLicensedFile
	children, and the list of RepositoryLicenseFile children.

	RepositoryDirectory objects are the objects that crawl the filesystem.

	While the script is pretty conservative (including probably more
	licenses than strictly necessary), it tries to avoid including
	material that isn't actually used. To do this, RepositoryDirectory
	objects only crawl directories and files for which shouldRecurse
	returns true. For example, shouldRecurse returns false for ".git"
	files.

	Some directories and files require special handling, and have specific
	subclasses of the above classes. To create the appropriate objects,
	RepositoryDirectory calls createSubdirectory and createFile to create
	the nodes of the tree.


	The low-level handling of files is done by classes in filesystem.dart.
	This code supports transparently crawling into archives (e.g. .jar
	files), as well as handling UTF-8 vs latin1. It contains much magic
	and hard-coded file names and so on to handle distinguishing binary
	files from text files, and so forth.

	This code uses the cache described in cache.dart to try to avoid
	having to repeatedly reopen the same file many times in a row.


	In the case of a binary file, the license is found by crawling around
	the directory structure looking for a "default" license file. In the
	case of text files, though, it's often the case that the file itself
	mentions the license and therefore the file itself is inspected
	looking for copyright or license text. This scanning is done by
	determineLicensesFor() in licenses.dart.

	This function uses patterns that are themselves in patterns.dart. In
	this file we find all manner of long complicated and somewhat crazy
	regular expressions. This is where you see quite how absurd this work
	can actually be. It is left as an exercise to the reader to look for
	the implications of many of the regular expressions; as one example,
	though, consider the case of the pattern that matches the AFL/LGPL
	dual license statement: there is one file in which the ZIP code for
	the Free Software Foundation is off by one, for no clear reason,
	leading to the pattern ending with "MA 0211[01]-1307, USA".


	The license.dart file also contains the License object, the currently
	simplistic normalizer (_reformat) for license text (which mostly just
	removes comment syntax), the code that attempts to determine what
	copyrights apply to which licenses, and the code that attempts to
	identify the licenses themselves (at a high level), to make sure that
	appropriate clauses are followed (e.g. including the copyright with a
	BSD notice).