Repository navigation
Conversation
pydeps statically follows every import from each UI section's module, including into numpy, scipy, torch, sympy and the rest of the environment: over 10,000 modules per section, of which ~640 are napari. The `--only` filter that keeps just the napari modules is only applied afterwards, so each of the 7 sections spent 50-150 s in that analysis. Pass every top-level module that can't lead back to napari as an exclude, so the module finder never enters it. Packages that import napari modules (napari_*, and npe2.io_utils) are still analyzed, and names that napari also uses for its own modules (e.g. a local `signal.py`) are never excluded, because pydeps matches the bare name of relative imports against the excludes. The UI sections step goes from 482 s to 12 s, and all 7 generated pages are byte-identical to before.
github-actions
Bot
added
the
documentation
Member
Author
|
@napari-bot make html |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
References and relevant issues
Part of #1209, found by profiling with #1204
Description
prep_docs.pygenerates the UI architecture pages with pydeps and this step has been known to be superrr slow (part of @psobolewskiPhD's motivation for the slim builds). In CI, this step takes 350s of the whole 940s build docs step (it takes 480s for me locally).For each of the 7 sections, pydeps walked every import from the section's module, recursively, through the whole environment.
(via LLM: For the status bar section alone that is 10,834 modules: torch (2,192), sympy, scipy, pandas, numba, networkx, … and only about 640 napari modules.)
The
--only napari...filter that keeps just the napari modules is applied to the finished graph, so its wasted work.The pydeps module finder accepts
excludesto stop it from investigating an import. The4 script now adds every top-level module that can't lead back to napari as an exclude. The output stays exactly the same because:napari_plugin_managerandnpe2.io_utilsimport napari modules, so napari modules can be reached through them (napari._qt.qthreadingin the Application menus page comes in that way). Anything namednapari*andnpe2, is kept, but we should probably use this as an opportunity to investigate some likely problematic import patterns too.signal. This results in dropping the vendoredqt_jsonschema_form.widgets -> qt_jsonschema_form.signaledge in the Dialogs page. The script has extra logic to keep any names that napari uses for its own modules.Results
before, locally: 480s
AFTER: TWELVE (12!) SECONDS!!!!!!!!!!!!!!!
So, hopefully CI also drops by ~340s (ALMOST 6 MINUTES!!!!!!).
A 30% decrease in total build time!!!!!!!!!