Currently just a collection of notes, about a program I’ve thought about on-and-off over a few years. It’s log, it’s log, it’s better than bad, it’s good. That’s probably the worst possible name to search for…

  • Capture timestamps concurrently with the smallest-possible unit of input data. For instance, each read of a serial port should be associated with a timestamp.
  • Support storing data from any number of sources together in one file, also exporting data from individual streams like the traditional approach.
  • Where it makes sense, like serial ports, support bidirectional data flow (also logged).

Prior Art

lnav has a nice terminal UI, filtering. Interesting database usage.

Architecture

  • Uses a client-server architecture along the lines of tmux.
  • GUI provided via web server
  • Server closes when last connection to streamer and GUI closes.
  • Single executable
  • Configurable default for storing files, by default keep session info in a /tmp file.
  • Handle crashes of the host machine as nicely as possible.

Filtering in GUI or server?

  • Probably most responsive at runtime in browser context
  • Doing it in server would be more resilient over time, less code subject to changing browsers
  • Presumably, filters will condense information on average, so filtering in server would require less bandwidth with GUI
$ good /dev/ttyUSB1
Good server not running... started
Initialised new session: default on http://localhost:1234
Capturing serial port /dev/ttyUSB1
Ctrl+C to exit

$ dmesg -w | good
Adding stream to session: default
Ctrl+C to exit
Hello from the keyboard, via Good's GUI!

$ good --stream notes --message "Pressed green button"
Atomic log to new stream: notes session: default
$

$ good --stream notes --message "Pressed red button"
Atomic log to existing stream: notes session: default
$

This starts a new session on a stack with the same streams/configuration/etc as the existing one. Consider approaches taken by Wireshark (save/discard/cancel modal - ick) and Saleae (discard - without confirmation IIRC - also ick)

$ good --restart
Restarted session: default

$ good /dev/ttyUSB2
Initialised new session: default-2 on http://localhost:1235
Capturing serial port /dev/ttyUSB2
Ctrl+C to exit
$ good --restart
Multiple sessions are active, which should this apply to?
-> default
   default-2
   <None>

Streams

The basic philosophy is to get the most data possible in to good, so the streams should do the absolute minimum filtering. Need to support multiple channels within each stream, so that a serial port stream for instance could track changes to the flow control lines, and so the log makes it clear what ended a particular stream in a session.

Sources:

  • Serial port
  • Pipes
  • Existence/contents of files and directories (maybe pipes are good enough for this)
  • good CLI
  • pcap
  • gpio (via a standalone program, through a pipe)
  • audio
  • video

Handling streams:

  • Nameable
  • Handling of control characters should be done by filters in the GUI

Filtering

  • Configurable to match on data streams, to do things like hide irrelevant data, highlight important data, graphically represent data (sequence diagrams, graphs), transform data.
  • Stored in hierarchal structure, so that a set of relevant filters could be committed with a project they fit with, and an individual user might have their own personal ones too.
  • Stateful, for instance a periodic log of temperature might update a graph of temperature over time, generated by a filter.
  • Run code, maybe using Python interpreter?
  • Filters should be able to apply to sets of streams (generate new streams too? Could be complicated, might not want to store filtered output so how to handle these derived streams?)

GUI

The gdbgui approach looks pretty nice, especially given the Rust GUI story so far. Presumably this leaves room for a more traditional GUI app as well.

GUI for manipulating filters has blocks, arrows can be dragged between them, terminals are just an output block.

Clearly show the time-relationship between data sources. Eg, should be able to select a block of text in one stream, and easily understand what was going on in other streams at the same time.

Indicate current log file size, provide for saving/loading/discarding/etc

Columns could be added to the log output, which indicate state based on filtering logs or whatever. For instance, if USB cable connectivity can be determined from the log items (if there are connection/disconnection events), there could be a column that shows whether cable is connected for each log item - 1/0/?.