Got basic timeseries with linestack working. I've also got some code snippets for interpolating to display heatmap with non-uniformly sampled timeseries data. I should be able to have this fully working with time series very soon :D

Got heatmap to display timeseries working. It should also work with non-uniformly sampled data by interpolating, need to test.

Also need to implementing switching between heatmap and line representations, need to delete the graphic when switching.

So timeseries can be represented with arrays of one of the following shapes (let's ignore x-axis values for now).

If we have:

l: number of timeseries
p: number of datapoints in a timeseries

We can have the following shapes:

p: only y-values
p, 2: yz vals
l, p: l-timeseries with y values
l, p 2: l-timeseries with yz values

Extended to n-dimensional arrays (for example, trajectories projected onto principal components?). If each non-timeseries dim is $d_1, d_2, ... d_n$, then the above becomes:

I don't think we can auto-detect if l is present or not and the user should specify, something like:

multi_timeseries: bool = True

Scatters can be similar to some cases of nd-lines 🤔 , but x values would be directly specified and the current index is parametric (example with time indicating position in a low dim space). This would actually be true for lines as well if representing trajectories.

So for nd-line maybe we have two versions, parametric (y and z are not functions of x, but x, y, z are a function of some other dim) and non-parametric (simple timeseries lines where y and z are functions of x).

I made a more generalist NDPositions which can map data that is:

[s1, s2, ... sn, l, p, 2 | 3]

where:
s1, s2, .... sn are slider dims
l is number of lines or number of scatters, this dimension is optional and the user must specify whether or not it exists
2 | 3 is the last dim, indicating xy or xyz positions.

It can map arrays of these dims to a line, line collection, line stack, scatter, or list of scatters (similar to multi-line).

I think this is a much more elegant way to deal with things, and NDTimeSeries is not necessary. The user can provide a slider mapping (to map from reference units to array index) for the p dimension which is the same as the "x-axis" for time series data!

Example if we have data that is [n_timepoints, 2], and the x-positions here (in the last dim) are in seconds. The NDWidget reference units for the slider can also be in seconds, and we can provide a mapping function that goes from the slider reference units to the n_timepoints index.

I think we can also use this for heatmaps and interpolation. Use the reference units to determine a uniform x-range for the current display window, and we can interpolate using [n_timepoints, 2] data.

EDIT: I think that the NDPositions will also work for PolygonGraphic ! Can think about meshes in general later.

For positions graphics, I should actually do [n_datapoints, n_lines, 2 | 3] so everything before the last 1 or 2 dims is always sliders.

Some more ideas:

Allow any 2-3 dims to be used as the graphic dimensions and specify the slider dims. This would also allow using named dims (such as those used in xarray).
API like:

add_nd_<scatter|lines|heatmap>(
  data=<array>, # array like, defaults to last 2-3 dims are graphical, first (n - 2 | 3) dims are sliders and in order
  x=<int | str | None>  # optionally specify x graphical dim using the integer dim index or dim name
  y = ... # same for other graphical dims
  slider_dims=<None | tuple[int | str]> # specify slider dims, None is auto first (n - 2 | 3), or specify a tuple of slider dims as int or str, examples: (0, 1, 2, 3), ("time", "plane")
  ... # other args
)

We interpret the given order of the slider_dims passed as $$s_1, s_2, ... s_n$$, regardless of their order within the actual array. This will make it clear to users which dims they are syncing when they use the sliders.

EDIT: A limitation of the above is that a user can't collapse multiple "graphic/display dimensions" into "final graphic/display dimensions" if they're hard-coded this like. So something like:

add_nd_<...>(
  data=<array>,
  display_dims=<tuple[int | str]>, # specify display dims in [x, y, z] order, OR display dims that collapse to xyz after the finalizer function
...
)

An example for images would be collapsing [z, m, n] to display a projection over z as slider dims are moved. But we could also have > 3 dims that are used, and then collapsed to 3 or fewer dims for the "final graphic/display dims".

We can use LineCollections to display multiple lines, like behavior tracks of keypoints, with shape [n_keypoints, n_timepoints, 2]. This works well with NDPositions and display windows.

I was thinking of what's the best way to show a scatter for each keypoint, and I think I should make a ScatterCollection that behaves like a LineCollection so the same array with the same shape can be given, the only difference is the graphical representation would be a scatter instead of lines. For typical behavior keypoint viz, the display_window would usually be just 1, but it can be greater than 1 for any viz that needs to show a window of scatter points.

ok I think stuff is working

I think I need to make a PolygonCollection too 🤔 . Would be very similar to the ScatterCollection.

so the basics are all working 🥳

LineStack and heatmap representations are swappable:

Scatter and line collection to show behavior trajectories:

A set of imgui UIs that allow controlling some aspects of the "nd graphics" could be useful, such as:

• graphical representation, dropdown menu to choose scatter, line, heatmap
• display_window
• multi bool

Stuff I should finish before implementing the orchestrator:

• merge auto-replace buffers #974 , so I can implement changing the display_window
• implement logic to allow the display_window to be one of:
  • centered at the index, "center" mode
  • start at the index, "start" mode
  • end at the index, "end" mode
• adding a linear selector, it's "default" positions and its position when the main slider for the n_datapoints dimension changes should correspond whether the display_window is in "center", "start" or "end" mode. Moving the linear selector should probably change the index in all other NDobjects but not its "parent" object. So that the visual representation of the parent object doesn't change, ex: if the linear selector is moved we don't want a linestack to also "move"
• spatial function
• implement mapping from a slider reference index with units (such as time) to array index.
• figure out how to implement stuff like colors, cmaps given that we're essentially performing out of core rendering with these graphics
• auto display_window? Zoom in/out and set graphic data at different subsample levels

Things that make me "uncomfortable" that I need to settle:

dim shapes

Dim shape for nd-positions is [s1, s2, ... sn, n, p, 2 | 3] where:

n: number of lines, scatters, or heatmap rows (optional, can be 1)
p: number of datapoints, often corresponds to a time index.

p would be a slider dim, but there's this n dim that's in between p and all the other slider dims. We could instead use a shape of [s1, ... sn, p, n, 2 | 3] but that feels weird and we'd have to do a transpose to get the array for the graphical representation, i.e. [p, n, 2 | 3] -> [n, p, 2 | 3] is required for the graphical representation.

Do we just document this well, that the n dimension is not a slider dim but p is?

When using nd-positions data in conjunction with nd-image data, we'd have something like this:

# nd-positions
[s1, ... sn, n, p, 2 | 3]

# nd-image
[s1, ... sn, p, r, c, 1 | 3 | 4]

Where r: rows, cols: columns. 1 | 3 | 4 denotes grayscale or RGB(A).

The p dimension in the nd-image array above would correspond to the p in the nd-positions. nd-images don't have any n dimension, it doesn't make sense there.

Working on "implement mapping from a slider reference index with units (such as time) to array index.", which requires proper implementation of slider_dims and n_slider_dims properties. Need to figure out how to properly separate the p n_datapoints dimension from other slider dims, and also apply the window funcs on the p dim. Will do tomorrow.

Window funcs working for p (n_datapoints) dim and all graphical representations

I need to restore the current ImageWidget in main so we can deprecate and remove it in a future release.

List of thoughts:

• allow multiple histograms in a subplot, name them, correspond to individual images or heatmaps
• reference ranges object instead of just a list of tuples
• Relatedly, indices object that has a nice repr, keeps tracks of the reference range objects etc. I think we should be able to push and pop dimensions on this object. If a reference range isn't specified we should just use the max for each slider dim for all ND graphics in the NDWidget. Each ndgraphic could also have this index class but only with info for the dims that graphic has. Make a nice repr for ndgraphics which also has info of the processor.
• auto-setting the x-range on timeseries data. Can probably make subplot properties like x_range etc.
• the position of the heatmap seems wrong, should check if LineStack is the same.
• tidy up missing stuff in NDProcessors like window functions and spatial function things, clean things up so window computing is done in a private method in the base class etc.
• linear selector stuff for time series
• edge cases: how to deal with when NDProcessor returns empty array

I think I can make the linear selector not respond to index changes when it the selector itself changes the index by using pause_graphics on the event handler that sets the linear selector index within the NDGraphic

basic orchestration of ndgraphics works 😭

@Vipitis you do some point tracking work too right?

# start, stop, step range for time
 reference_range = [(vid_indexing[0], vid_indexing[-1], 0.025, "time (s)")]
 
 # Create an ND Widget with the reference dimensions
 ndw = fpl.NDWidget(ref_ranges=reference_range, shape=(1, 2))
 
 # add video as an nd image
 ndw[0, 0].add_nd_image(
     vid, 
     index_mappings=(vid_second_to_index,), 
     processor_kwargs={"compute_histogram": False, "rgb": True}
 )
 
 # add behavior data as an nd scatter
 nd_scatter = ndw[0, 0].add_nd_scatter(
     df_tracks,  # tracks dataframe
     keypoints_cols[:, :-1],  # provide the columns to take the tracks data from in the form [(keypoint_n_x), (keypoint_n_y), ...], 
     likelihood_cols,  # show likelihood as tooltips
     processor=ndp_extras.NDPP_Pandas, # use the processor for pandas dataframes
     display_window=5,
     index_mappings=(tracks_second_to_index,)  # time (s) -> array index mapping
 )

For NDPositions for other graphic features like colors, sizes, markers, cmap_transform, etc, maybe the way to do it is to just pass an additional array of the same shape as the data (i.e. s1,... sq, n, p, _) but with the corresponding feature values instead. And then we can use the same index and display window to get the values for those features at the given indices.

Could also take a mapping function that takes the current indices and current data_slice array to create the feature values on the fly.

I just noticed these weird glitches on my desktop on both my AMD and Nvidia GPU. Never noticed it before. I wonder if it's related to replacing the buffer when the number of datapoints changes but I'm not sure. The line turns red whenever the buffer has been replaced. One or more lines is rendered according to some previous positions data and not the current one.

I checked the values in the actual pygfx.Buffer and they are identical between the lines so IDK why they are glitching in the wrong locations. Hard to reproduce, seems sporadic. The only difference between the lines is the y-offset position (i.e. WorldObject.world.y)

@almarklein any guesses?

EDIT: I posted on pygfx

linear selector stuff working 😄

@clewis7 very clever for suggesting I should just keep the same linear selector logic and set the display_window to the entire data to get the way we usually use linear selectors on an entire timeseries 🥳

I think github is broken, there's a merge conflict which I solved locally and then pushed. Those changes are in the branch (can see on github if I go to that branch), but it's not showing up on the commits list here. And when I try to resolve the conflicts here on the PR it keeps saying "this page is out of date" and I make the changes 🤦‍♂️

seems like github is working again, the commits have shown up in the PR 🤷

I just realized I think we can do subplot.frame.rect to get the number of pixels in x across the subplot and use that to figure out a good auto number of data points to display for OOC rendering and LOD of timeseries data.

EDIT: this needs more thought, easy to compute if it's a single line across x but more compicated for real data lol
EDIT2: rendering is just sampling so I think this is a typical sampling problem that can be solved

We can have something like an index for even the l dim (number of lines, rows, scatters, etc) and p (n datapoints) dim, (context: last 3 dims are l x p x 2 | 3).
This would basically be a selection vector! Like neuron or trace or keypoint or specific timepoint. And we could allow multiple such "global selection vectors" in a NDWidget to represent selection of neurons, keypoints, or anything that has a graphical representation! This is gonna abstract away even more event code 😄

When you plot contours on a movie and traces, that's l x p x 2. l is the same for both, it's number of components. We can use a selection vector to represent interaction with them. And if you have behavior, you can use another selection vector for keypoints. When you add an NDgraphic, you just specify which selection vector it's on. If looking at calcium and behavior you give the calcium arrays all the same component/neuron selection vector, and you give the behavior related arrays keypoint or whatever relevant selection vector that corresponds to the l dim

 This way we don't event have to write event code for even this interactivity, it gets even more declarative.

Can also think of how to use the selector tools (rect and polygon) to auto-update the selection vectors.

• checkbox for show/hide histograms
• ndwidget control tab UI with list of ndgraphics, hover graphic and highlight corresponding graphic
• options and sliders to change display window, no display window (see full data), graphic features
• show active ndgraphic with new graphic that has slightly different opacity or something like this
• toggle between display window as a slider vs. with panning & zooming

ok I need to do the following:

NDGraphics need eventing for the indices. Need to allow adding event handlers to get when the indices change. Right now this is necessary so that NDWidget knows when a graphic internally changes its indices for any reason so that the global index can be changed too.

Better idea: GlobalIndexVector object. It also has the references ranges instances.

Each NDGraphic gets the instance of GlobalIndexVector. When the graphic has to change the index, it changes it via the global_index_vector instance only. The global_index_vector then updates all graphics.

Need to draw this out properly and take linear selectors into consideration as well.