dependencies - Collecting the Symbols in a File

Question

As the individual files of a program are compiled and loaded in sequence, is there a way to find out which symbols are referenced (ie, interned or referred to) in a particular file? (Assuming a single package for simplicity.)

For example, if the first two files of a program have been successfully loaded, how can the newly interned symbols from the second file, along with the second file's symbols that depend on definitions in the first file, be collected into a list?

Could the function do-symbols somehow be used to extract the symbols created at each step of file loading? (Or alternatively, could the Common Lisp reader be used independently to acquire the symbols from each form in a file?)

(PS: The wider objective is to determine if there are dependencies between any two files.)

question from:https://stackoverflow.com/questions/65910170/collecting-the-symbols-in-a-file

Answer 1

You can't easily find out what preexisting symbols are simply referred to without doing complicated things in the reader (you'd have to implement your own, compatible, symbol-reader) (but see below).

You can find out what new symbols are created.

stash-symbols makes a stash of symbols, and new-symbols returns a list of symbols not in the stash:

(defun stash-symbols (&key (packages (list-all-packages))
                           (into (make-hash-table))
                           (external-only nil))
  (dolist (p packages into)
    (if external-only
        (do-external-symbols (s p)
          (setf (gethash s into) s))
      (do-symbols (s p)
        (setf (gethash s into) s)))))

(defun new-symbols (stash &key (packages (list-all-packages))
                          (external-only nil))
  (let ((news '()))
    (dolist (p packages)
      (if external-only
          (do-external-symbols (s p)
            (unless (eq (gethash s stash) s)
              (push s news)))
          (do-symbols (s p)
            (unless (eq (gethash s stash) s)
              (push s news)))))
    news))

Now load/comparing makes a stash, loads a file, and reports the new symbols. Note that loading files can create packages (and in fact a smarter version of this would report a list of new symbols in preexisting packages together with a list of new packages: that would be easy to do but I'm too lazy now.

(defun load/comparing (file &key (packages nil packages-p)
                            (external-only nil))
  ;; Note the list of packages can easily change during LOAD!
  (let ((stash (stash-symbols :packages (if packages-p
                                            packages
                                          (list-all-packages))
                              :external-only external-only)))
    (values (load file)
            (new-symbols stash
                         :packages (if packages-p
                                       packages
                                     (list-all-packages))
                         :external-only external-only))))

One way of trying to find out all the symbols in a file which is loaded is to try and write a function which merely pretends to load a file but in fact just reads it. This is significantly hard to get right. Here is a function which very definitely does not get it right, but it does at least 'hear' in-package forms, so it may work in many useful cases (but beware of eval-when, and also (defpackage ...) ... (in-package ...) will doom it, which can perhaps be worked around by first really loading the file to create the package):

(defun try-to-pretend-to-load-file (file)
  ;; Attempt to read a form doing what LOAD would do.  This very
  ;; definitely will not always do the right thing.
  (let ((*package* *package*))
    (with-open-file (in file)
      (loop for form = (read in nil in)
            while (not (eq form in))
            when (and (consp form)
                      (eq (first form) 'in-package))
            do (setf *package* (find-package (second form)))
            collect form))))

And now

(defun extract-interesting-symbols (forms &key (into (make-hash-table))
                                          (filter nil filterp))
  ;; Find interesting symbols in FORMS.  NIL is never interesting
  ;; (sorry).
  (labels ((extract (thing)
             (typecase thing
               (null nil)
               (symbol
                (when (or (not filterp)
                          (funcall filter thing))
                  (incf (gethash thing into 0))))
               (cons
                (extract (car thing))
                (extract (cdr thing))))))
    (extract forms)
    into))

So, now (using collecting):

> (collecting (maphash (lambda (s c) (collect (cons s c)))
                       (extract-interesting-symbols
                        (try-to-pretend-to-load-file 
                         "binding.lisp"))))
((org.tfeb.hax.binding::what . 2)
 (:compile-toplevel . 1)
 (org.tfeb.hax.binding::expanded . 4)
 (labels . 6)
 (org.tfeb.hax.binding::form/expansion . 2)
 (:load-toplevel . 1)
 (:test . 2)
 (org.tfeb.hax.binding::a . 16)
 (ecase . 1)
 (&rest . 4)
 (:org.tfeb.hax.collecting . 2)
 (org.tfeb.hax.binding::b . 20)
 (quote . 31)
 (equal . 1)
 (collecting . 1)
 (org.tfeb.hax.binding::f . 4)
 (consp . 1)
 (first . 14)
 ...)

You can see from the results that there is an eval-when in that file (and in fact there's a defpackage as well), but it doesn't matter as I've already loaded it so its work is done.