===============================================================================
 LinuxNoteText — a text and source editor
===============================================================================

 Stage:       0.2, early version
 Programmer:  Stan Switaj
 From:        LinuxDoors.com
 Bugs to:     bug-reports-linuxnotetext@linuxdoors.com

 The window, the menus, the tabs and the file handling are Python and Qt
 (PySide6). The text area is Scintilla — the same editing engine Notepad++ is
 built on — compiled from source in src/ — the official 5.6.6 release — together with
 Lexilla 5.5.3, which colours the syntax of 139 languages.

 It is one process with one Qt in it. The C++ canvas is handed to PySide6 with
 shiboken6.wrapInstance() and from Qt's point of view it is an ordinary
 QWidget, so it goes in a layout, takes focus and gets styled like anything
 else.


-------------------------------------------------------------------------------
 Building
-------------------------------------------------------------------------------

     sudo apt install build-essential cmake pkg-config qt6-base-dev
     pip3 install PySide6

     ./build.sh

 Nothing is installed system-wide and nothing is downloaded — Scintilla and
 Lexilla are already in src/. The build ends by running the checks.

     ./LinuxNoteText.py [file ...]
     ./LinuxNoteText.py --check-contrast
     ./LinuxNoteText.py --version


-------------------------------------------------------------------------------
 How it is laid out
-------------------------------------------------------------------------------

 LinuxNoteText.py         the program; command line, then hands over
 build.sh                 builds the canvas and runs the checks

 linuxnotetext/
   canvas.py              the join to C++ — loads the library, adopts the
                          widget, and exposes send(). Knows nothing about any
                          particular language.
   languages.py           53 languages: what each is called, which Lexilla
                          lexer colours it, its file extensions, its comment
                          marker, and its keyword lists. Also the content
                          sniffing that tells an Apache .conf from an nginx
                          one, since the name cannot.
   theme.py               both palettes, the contrast check, the Scintilla
                          styling and the Qt stylesheet
   find.py                find and replace: the search requests, the walk
                          from match to match, and the panel
   document_list.py       the list of open files down the left, and the
                          drop target for files dragged in from elsewhere
   document_map.py        the file in miniature: a second Scintilla view on
                          the same document, with an overlay on top
   styles.py              what each kind of text looks like, and where that
                          is kept
   style_dialog.py        the style configurator
   preferences.py         the preferences dialog
   projects.py            named sessions -- a piece of work and a note
   project_dialog.py      the session manager
   search_results.py      every match, as a tree
   commandline.py         the command line, following Notepad++
   report.py              --what-works
   plugins.py             finding, describing, loading and surviving plugins
   lua_runtime.py         running Lua plugins, and the three escapes it shuts
   plugin_manager.py      the plugin manager window
   window.py              the window: tabs, menus, status bar, open and save
   settings.py            what is remembered between runs
   sci_constants.py       generated — do not edit

 native/
   SciCodec.h/.cpp        a stand-in for QTextCodec (see below)
   shim_include/QTextCodec  puts it on the include path under that name
   lnt_shim.cpp           the C entry points Python calls
   CMakeLists.txt         builds Scintilla + Lexilla + the shim
   make_constants.py      generates sci_constants.py from Scintilla's headers
   prove_canvas.py        the canvas checks
   prove_find.py          the find and replace checks
   prove_plugins.py       the plugin checks, including the safety claims
   prove_map.py           the document map checks
   prove_configs.py       the Apache and nginx checks, against /etc
   prove_drop.py          dragging files in, with real drop events
   prove_lua.py           the Lua plugin checks, including the escapes
   shoot.py               photographs the window in both themes

 plugins/                 the plugins that ship with the editor
 plugins/lua/             the Lua ones, and how to write your own
 vendor/lupa              Lua 5.5, for the Lua plugins
 vendor/PROVENANCE.txt    where it came from, and what was taken out of it
 src/PROVENANCE.txt       where the two libraries came from, with checksums
 src/scintilla            Scintilla 566, official release, unmodified
 src/lexilla              Lexilla 553, official repository, unmodified
 lib/liblntcanvas.so      the built canvas


-------------------------------------------------------------------------------
 Two decisions worth knowing about
-------------------------------------------------------------------------------

 QTEXTCODEC
     Scintilla's Qt layer uses QTextCodec, which Qt 6 moved into the Qt5Compat
     module. PySide6 does not ship Qt5Compat, so linking against the system
     copy would put a second, differently-versioned Qt into a process that
     already has one — which does not work. So native/SciCodec.cpp provides
     the four calls Scintilla actually uses, backed by iconv from glibc, and
     shim_include/QTextCodec puts it on the include path under that name.
     Scintilla's own sources are therefore completely unmodified, and updating
     src/ is a matter of replacing the folder.

 THE EDITOR IS A CONTAINER
     shiboken will not adopt a foreign C++ pointer into a Python subclass — a
     subclass has to be constructed by Python from the start. So Editor is a
     real QWidget holding the adopted canvas as its only child under a
     zero-margin layout. Both halves stay intact, and there is somewhere to
     put the things that will sit beside the text later.


-------------------------------------------------------------------------------
 Light and dark
-------------------------------------------------------------------------------

 Both, switched with Ctrl+D. Every colour pair in both themes is measured
 against the WCAG 2.1 contrast standard — 4.5:1 for text, 3:1 for a control
 outline — rather than judged by eye, and the program will report on itself:

     ./LinuxNoteText.py --check-contrast

 58 pairs, all passing. The check runs as part of ./build.sh, so a colour
 cannot quietly be changed to something unreadable.


-------------------------------------------------------------------------------
 What is in it so far
-------------------------------------------------------------------------------

 Tabs, opening and saving, unlimited undo and redo, syntax colouring for 53
 languages with real keyword lists, a Language menu to override the guess,
 line numbers, a folding margin, indent guides, whitespace marks, word wrap,
 current-line highlight, bracket matching, multiple selections, zoom, comment
 and uncomment, go to line, and settings that persist in
 ~/.config/LinuxNoteText/LinuxNoteText.save.

 FIND AND REPLACE
     Ctrl+F to find, Ctrl+H to replace, F3 and Shift+F3 to step through,
     Ctrl+F3 to search for whatever is selected, Escape to close. Match case,
     whole word, wrap around, and regular expressions. Every match is boxed
     as you type and the panel says which one you are on — "3 of 114".
     Replace All is a single undo step, and can be held to the selection.

     The matching is not done in Python. Scintilla does it in C++: a range
     becomes the target, flags say how to match, and SCI_SEARCHINTARGET runs
     either Scintilla's own small engine or std::regex over it. The
     "Regular expression" option asks for std::regex in ECMAScript mode, so
     alternation, {n,m}, lookahead and \1 backreferences all work — the
     built-in engine has none of them.

     Python decides what to search, walks match to match, wraps, replaces
     and draws the panel. That division was measured, not assumed: walking
     all 200,000 matches in a 10.8 MB file from Python takes 0.9 seconds and
     a single find is too fast to time, so moving the walk into C++ would
     buy nothing a person could notice.

 PLUGINS
     Plugins -> Manage Plugins. Nine ship with the editor. Four are
     Python — Word Count, Sort Lines, Tidy On Save and Insert Date — and
     five are Lua, in plugins/lua: Hello Lua, Change Case, Sum Numbers,
     Align Equals and Stamp Modified. A plugin is a folder holding a
     plugin.ini and either a plugin.py or a plugin.lua, and goes in:

         ~/.config/LinuxNoteText/plugins/

     plugins/lua/README.txt is the guide to writing one in Lua: the whole
     of what a plugin is given, what it is not given, and the two things
     about Lua specifically that will catch you out.

     WHAT "SAFE" MEANS, EXACTLY

     A plugin is Python running inside the editor's own process. Nothing
     short of a separate process with a real sandbox can stop such code
     doing whatever the person running the editor could do, and this is not
     that. Rather than imply otherwise, here is what is actually true:

       - Finding a plugin never runs it. Discovery reads plugin.ini, which
         is plain data. Dropping a folder into the plugins directory
         executes nothing at all — there is a check for exactly this, using
         a plugin that writes a file the moment it is imported.

       - Nothing runs until it is switched on by hand, and new plugins
         arrive switched off. A plugin that has never been enabled has
         never been imported.

       - What a plugin may do through the editor is declared in its
         manifest and enforced at the API boundary. A plugin that did not
         ask for change_text gets a refusal naming the missing capability,
         not an edited document — including through the raw send() route.
         An unknown capability makes the plugin unusable rather than being
         ignored, and an entry file naming a path rather than a filename is
         refused, so a manifest cannot point the loader outside its folder.

       - This is a real limit on the supported way of working and an honest
         disclosure of intent. It is not a wall. A plugin that imports os
         and writes to disk has gone round the API, and the manager says so
         next to any capability the editor cannot police.

       - A plugin that fails cannot take the editor down. Every call into
         plugin code is guarded; one that keeps failing is switched off and
         what went wrong is shown in the manager.

       - Edits made through the API are one undo step.

     AND WHY A LUA PLUGIN IS DIFFERENT

     All of the above is about Python plugins. A plugin whose entry file
     ends in .lua is not Python and does not run in this process's Python:
     it runs in a Lua interpreter whose entire global environment is built
     by the editor. There is no import to go round the API with, because
     there is no import — and no io, no os.execute, no require and no load.
     What is left of os is os.date, os.time, os.clock and os.difftime,
     which cannot reach anything.

     So for a Lua plugin the capability list is enforced rather than
     declared, and the plugin manager says so instead of showing the
     "cannot enforce this one" note. Three genuine escapes had to be shut
     to be able to say that, and each is tried again by prove_lua.py every
     time the build runs:

       - lupa puts a `python` table into Lua by default, and python.eval
         will evaluate any Python expression at all.

       - Lua's own standard library reaches the machine through os.execute,
         io.open, require and load.

       - The one that is easy to miss. Any Python function handed to Lua
         can be asked for its attributes, and
         f.__globals__["__builtins__"].__import__ is a complete escape from
         an otherwise clean environment — a plain closure gives it up just
         as a bound method does. This is why the editor is given to Lua as
         a table of closures behind an attribute filter that refuses every
         attribute lookup on every Python object: calling still works,
         reaching inside does not.

     What it is still not: a security boundary of the sort a separate
     process would give. A Lua plugin can loop forever and hang the editor,
     and it can use everything it was granted.

 APACHE AND NGINX
     Both are coloured, and both had to be told apart first. They share
     Lexilla's `conf` lexer — which is written for Apache but is really a
     lexer for "a directive, then its values, with # comments" — and they
     are separated by their keyword lists rather than by two lexers, which
     keeps src/ unmodified upstream as promised above.

     Telling which is which is the interesting part. Both live in .conf
     files, and the ones that matter most on a real server often have no
     extension at all — /etc/nginx/sites-enabled/default is the usual
     example. So a name is tried first, and when the name leaves the
     question open the first 8 KB of the file is read and the marks of each
     are counted: <VirtualHost and DocumentRoot against server { and
     proxy_pass, with the directory it sits in weighing in. Two clear hits
     are needed, so prose that mentions a directive once is still prose,
     and a .py file is never re-read at all.

     Checked against every file under /etc/nginx and /etc/apache2 on the
     machine it was written on — 172 of them, all correct.

 DOCUMENT MAP
     Ctrl+M, or View, Document Map. The whole file in miniature down the
     right, still syntax coloured, with the part you are looking at lit and
     the rest dimmed. Click or drag anywhere on it to go there; the wheel
     works over it too. When the file is taller than the map, the map
     scrolls as well, so the lit part travels the height of the map exactly
     as the editor travels the length of the file.

     It is a second Scintilla view onto the SAME document, which is how
     Notepad++ does it. SCI_GETDOCPOINTER hands the document out and
     SCI_SETDOCPOINTER points the other view at it; from then on both show
     the same text and the same colouring with nothing to keep in step, and
     an edit or an undo appears in both at once. The alternative — painting
     a miniature by hand — would have meant a second renderer and a second
     lexer path, and a standing risk of the two disagreeing about a file
     being edited.

     What it costs instead is knowing which settings belong to a DOCUMENT
     and which to a VIEW. Read-only, the lexer, the tab width and the undo
     history belong to the document and are shared; the fonts, styles,
     margins and caret belong to the view, and are what make the map small.
     Setting read-only on the map — which looks like exactly the right way
     to stop it being typed into — would have made the real editor
     read-only as well. So the map is not read-only. It is covered by an
     overlay that takes every mouse event and is never given focus, so
     nothing reaches it to type. There is a check for that specifically.

 Still to come: a session that reopens what was last open.
