---
title: "Pathing and Table Structure"
author: "Gabriel Becker"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Pathing and Table Structure}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options:
  chunk_output_type: console
---

# Introduction

`rtables` models both the row- and column-structure of a table as trees. These trees collectively reflect the layout instructions used to declare the table's structure. We can use this to describe locations within the row-, column-, or cell-space of a table in a semantically self-describing way. We call these semantically meaningful locations *paths*.

A *path* is an ordered set of names which declares both a path for traversing the tree structure for the relevant dimensions, and consequently a corresponding subset of the table in that dimension. *Column* paths may contain only split names and names of facets generated from those splits. *Row* paths, can additionally contain names of tables corresponding to `analysis` calls, the `"@content"` directive which steps from a facet into the table generated by `summarize_row_groups` containing its marginal summary row(s), and names of individual rows.The location of individual cells or rectangular groups of cells is then defined by a row-path column-path pair.

As of `rtables` version `0.6.13`, any structural element of a successfully built[^1] table is guaranteed to correspond to a unique row path, column path, or combination thereof.

## An Illustrative Example

Consider a the table with non-trivial structure in both the column and row dimensions:
```{r}
library(rtables)

keep_rc <- c("ASIAN", "WHITE") ## chosen for brevity

afun <- function(x) {
  list(
    Mean = rcell(mean(x), format = "xx.x"),
    Median = rcell(median(x), format = "xx.x")
  )
}

lyt <- basic_table() |>
  split_cols_by("ARM", split_fun = keep_split_levels(c("A: Drug X", "C: Combination"))) |>
  split_cols_by("SEX", split_fun = keep_split_levels(c("F", "M"))) |>
  add_overall_col("All") |>
  split_rows_by("RACE", split_fun = keep_split_levels(keep_rc)) |>
  summarize_row_groups() |>
  split_rows_by("STRATA1") |>
  summarize_row_groups() |>
  analyze("AGE", afun = afun) |>
  analyze("BMRKR1", nested = FALSE, show_labels = "visible")

tbl <- build_table(lyt, DM)
tbl
```

We can get a first look at the row- and column-structure of our table (if in different formats) with `table_structure` and `col_info`.


```{r}
table_structure(tbl)
```

```{r}
col_info(tbl)
```

We can use paths to declare intuitive substructures of our table. We illustrate
this using `[` which interpret character vector indices as individual paths in
the respective dimension.

Note that while we will use these paths to subset our table for illustrative purposes,
they are more often used to specify where something should happen within the larger
table, which we discuss in the following section.

### Row Paths

Row paths, in isolation, describe horizontal slices of our table. We can see all the valid row paths (including an optional "root" beginning value which is technically correct but not necessary to include) via `row_paths_summary`.

```{r}
rpsummry <- row_paths_summary(tbl)
```

In addition to displaying a nicely formatted summary, this returns a data.frame
containing the same information in a programmatically accessible form. In particular
`path` is a list-valued column whose values can be used directly as row paths:

```{r}
head(rpsummry)

tbl[rpsummry$path[[6]], ]
```

The `c("RACE", "ASIAN")` row path refers to the horizontal slice of our table
containing all rows that represent analysis on Asian patients. We see that we get the expected subtable:


```{r}
tbl[c("RACE", "ASIAN"), ]
```

Similarly we can get the groups summary and row for strata B of Caucasian patients via


```{r}
tbl[c("RACE", "WHITE", "STRATA1", "B"), ]
```

Notice this is a strict subtable in the structural sense, which means we do not get
the ethnicity-level group summary here. We can see this because our structure and our path
now starts with `"B"`:


```{r}
table_structure(tbl[c("RACE", "WHITE", "STRATA1", "B"), ])
```

As mentioned above, to "path into" a group summary we use the `"@content"` directive:

```{r}
tbl[c("RACE", "ASIAN", "@content"), ]
```

We can path down to analysis tables and then individual rows via their name, which unlike other structures
tends to be identical to their label:

```{r}
tbl[c("RACE", "WHITE", "STRATA1", "B", "AGE"), ]
tbl[c("RACE", "WHITE", "STRATA1", "B", "AGE", "Median"), ]
```


### Column Paths

Similar to row paths, we can get information about column paths via `col_paths_summary`:

```{r}
col_paths_summary(tbl)
```

We can then describe vertical slices of our table via these paths (we use `head`, which
subsets via absolute position to limit the amount of output here):


```{r}
head(tbl[, c("ARM", "A: Drug X")])
head(tbl[, c("ARM", "C: Combination", "SEX", "M")])
head(tbl[, c("All", "All")])
```

Note that despite being displayed next to each-other, the last two columns of
our table have fundamentally different paths. This is due to `add_overall_col` 
adding a **non-nested** additional split rather than adding an additional implicit
combination level to the `ARM` split.

## Uniqueness of Paths

As of `rtables` `0.6.13`, rtables enforces uniqueness of names within groups
of direct sibling structures in both row and column[^1] space, thus guaranteeing unique paths to every substructure in the table.

In row space, it does this by appending `[k]` to 
the names of elements which would otherwise have an identical name to a 
previous sibling, where k is a sequence of integers such that all siblings have unique names. This affects the paths to these substructures[^2], as we see from the informative messages below:



```{r}
lytdup <- basic_table() |>
  analyze("STRATA1") |>
  split_rows_by("STRATA1") |>
  analyze("AGE")

tbldup <- build_table(lytdup, DM)
tbldup
```

```{r}
row_paths_summary(tbldup)
```

 This allows us to path to all elements of the row structure, which was not possible
in previous (`<0.6.13`) `rtables` versions:


```{r}
tbldup[c("STRATA1", "A"), ]

tbldup[c("STRATA1[2]", "A"), ]
```

## Wildcards In Paths

Many, though not all, `rtables` functions which accept a row or column paths
support the `"*"` path wildcard. Where supported, the wild-card will match
any *name* present at that step in the table structure, leading to (potentially)
multiple matches. Note `"*"` will never behave as the `"@content"` directive, which
must always be used explicitly.

```{r}
tbl[c("RACE", "*", "STRATA1", "B", "AGE", "Median"), ]
```


Multiple wildcards can appear in a path, with each wildcard applied recursively
within the full combined set of matches from all wildcards earlier in the path.


```{r}
tbl[c("RACE", "*", "STRATA1", "*", "AGE", "Median"), ]
```


Note that while the `[` method does support wildcards, we are only
using that to illustrate the behavior, as the tables resulting from 
using wildcard paths with `[` are generally not going to be very useful.



For (currently only) row paths, we can resolve a path with one or more 
wildcards into a set of fully specified paths that match the path in our table
using the `tt_normalize_row_path` utility function

```{r}
tt_normalize_row_path(tbl, c("RACE", "*", "STRATA1", "*", "AGE", "Median"))
```

We can also test whether a row path (including those containing wildcards) exists
in our table with `tt_row_path_exists`

```{r}
tt_row_path_exists(tbl, c("RACE", "*", "STRATA1", "*", "AGE", "Median"))
```

```{r}
tt_row_path_exists(tbl, c("RACE", "*", "STRATA1", "*", "FAKEFAKEFAKE", "Median"))
```


Note also that each `"*"` wildcard will only match *a single step*, there is
not currently a directive that searches for a match anywhere in the relevant
(sub)structure.

Thus we get
```{r}
tt_normalize_row_path(tbl, c("*", "Mean"))
```

Despite there being other `"Mean"` elements elsewhere in our row structure.


Though the above utilities don't currently exist for column paths (which are implemented
differently in ways not relevant to end users), generally those mechanisms which support
wildcards in row space and also accept a column path support wildcards for column paths
as well:


```{r}
tbl[, c("ARM", "*", "SEX", "F")]
```

## Operating On Tables With Paths

In addition to subsetting via paths, which as we mentioned is likely to be of limited utility, many aspects of a table can be selectively inspected or changed using paths.

We will explore some of these throughout this section

### Column Counts

We can set (though, currently not get, an oversight that will likely be remedied in a future
version) the visibility on *a set of sibling facets*.
```{r}
tbl2 <- head(tbl)
facet_colcounts_visible(tbl2, c("ARM", "A: Drug X", "SEX")) <- TRUE
tbl2
```

NB: unlike virtually all functions which accept paths, `facet_colcounts_visible` accepts the path to ***the parent of the facets you'd like to change the colcount visibility for***. This is because direct siblings cannot have different column count visibilities, so pathing
to individual facets would lead to an invalid table.

We can also get or  modify the value of any particular column count (note no s here):

```{r}
facet_colcount(tbl2, c("ARM", "A: Drug X", "SEX", "M")) <- 5
tbl2
```

If we need to mix visibilty and non-visibilty of counts within a direct sibling group
the best we can do is setting one to NA, which will leave a blank space there:

```{r}
facet_colcount(tbl2, c("ARM", "A: Drug X", "SEX", "F")) <- NA_integer_
tbl2
```

### Section Dividers

Section dividers are character(s) that are printed in a line after a
particular row or subtable during rendering to differentiate sections
of a table (they are most often, and by default, " " to create a blank line).

```{r}
tbl3 <- tbl

section_div_at_path(tbl3, c("RACE", "*")) <- "*"
section_div_at_path(tbl3, c("RACE", "*", "STRATA1", "B")) <- "+"
tbl3
```

Section dividers have a *least specific to most specific* order of precedence, 
with only the least specific applicable section divider displayed after any
given row. See `?section_div` for more details.

In addition to the path specific section dividers, there is also the more general
section divider function, which can be used to query the divider after each row:

```{r}
section_div(tbl3)
```

Together with the corresponding replacement function, this can be used to insert a divider
after specific rows:

```{r}
section_div(tbl3)[1] <- "-"
tbl3
```

Although here again, the path-specific section divider could come in more handy, e.g.:

```{r}
section_div_at_path(tbl3, c("RACE", "*", "@content", "*")) <- "-"
tbl3
```

The relevant paths can be looked up via `section_div_info`:

```{r}
section_div_info(tbl3)
```

### Sorting Within An `rtables` Table

Sorting rows in a table occurs in a path-specific way. See the sorting section
in the [pruning and sorting vignette](sorting_pruning.html#Sorting) for a 
detailed discussion of this.

### Extracting And Modifying Values

We saw that the `[` method interprets character indicies as paths. Beyond that, 
the `value_at` and `cell_values` getters and setters accept paths as well.
See the [subsetting vignette](subsetting_tables.html).

### Footnotes

Pathing can also be used to add referential footnotes to rows, columns,
or cells. This is discussed in the [title and footer](title_footer.html)
and [subsetting](subsetting_tables.html) vignettes.

## Table Structure And Technical Details

Here we will go into a bit more detail of how layouts, table structure, and pathing
are related. This is largely for informational purposes and most of it will
not be directly relevant to end-users who are simply creating tables.

### Row Structure

`rtables` is row-dominant (as opposed to R's `data.frame`s which are column
dominant). This means that tables are modelled as a (generalized) collection
of rows, rather than columns. More accurately, a table is modeled as a collection
of children, which can have children, etc until ultimately all of the "leaf" children
in the defined tree-graph are individual rows. 

We can see this using the `tree_children` function. The table we've been 
working  with throughout this vignette has two direct children, one containing
all of the structure generated underneath the initial `"RACE"` split, and one
containing the unnested analysis of `"BMRKR1"`:


```{r}
tree_children(tbl)
```

For convenience we will define a `multi_step_children` function which
recursively retrieves children from the table, and then from those children,
etc. For information purposes, we will print the "path step" taken each time,
thus building up our path as we descend using the class structure.

```{r}
multi_step_children <- function(tbl, indices) {
  print(obj_name(tbl))
  ret <- tree_children(tbl)
  for (i in indices) {
    print(obj_name(ret[[i]]))
    ret <- tree_children(ret[[i]])
  }
  ret
}
```


Thus we can see that the first of our table's children has the path `c("root", "RACE")`
and has children for each ethnicity in our table (recall the "root" path element is 
correct but optional):

```{r}
multi_step_children(tbl, 1)
```

Each of these children under `"RACE"` is a subtable. 

The children under our `BMRKR1` analysis, on the other hand, are rows (in this
case only one row, in fact):

```{r}
multi_step_children(tbl, 2)
```

Within each race subtable, we see a table corresponding to the `STRATA1` split:


```{r}
multi_step_children(tbl, c(1, 1))
```

```{r}
multi_step_children(tbl, c(1, 1, 1))
```


And finally within each strata facet is a table representing the analysis of `AGE`

```{r}
multi_step_children(tbl, c(1, 1, 1, 2))
```

And within each of those `AGE` analysis tables, like our `BMRKR1` top level
analysis table, we have a collection of rows:

```{r}
multi_step_children(tbl, c(1, 1, 1, 2, 1))
```

Thus we see that `analyze` calls create tables (called `ElementaryTable`s) containing
individual rows as children, while `split_rows_by` (and siblings) calls create a
subtable with children that are a table for each facet declared by the split operation:

```{r}
## child is AGE analysis table within RACE->WHITE->STRATA1->A
multi_step_children(tbl, c(1, 2, 1, 1))
```

```{r}
## children are individual rows of that AGE table
multi_step_children(tbl, c(1, 2, 1, 1, 1))
```


### Label And Group Summary Rows

For technical and historical reasons, label rows and so-called "content rows"
(which are essentially marginal analyses at a non-leaf point in the tree
graph defined by the parent-child relationships discussed above) are 
modeled separately.

Given a (sub)table, the content table (containing the content rows) and label
can be retrieved by `content_table` and `obj_label`, respectively. Note that
`obj_label` returns a string, not a row, as the label row is an internal detail
not currently exposed.

Recall that our `multi_step_children` function returns *the set of children
at a location*, so we must subset one additional time to arrive at a single child:


```{r}
tb <- multi_step_children(tbl, c(1, 1, 1))[[2]] ## second ie B strata
tb

content_table(tb)
```

Typically (ie by default) label rows for (sub)tables that have a non-empty
content table are not visible when rendering, but they do still exist:

```{r}
obj_label(tb)
```

Thus we see that:

- `split_rows_by*` layout instructions create a single subtable with the split name, which contains a facet for each value of the split;
- individual `analyze` instructions create a subtable containing individual rows defined by the afun used;
- repeated or multi-var `analyze` instructions create a parent subtable with a child for each individual analyzed var, as above; and 
- `summarize_row_groups` create content tables on *the children of* the table for that split


### Column Structure

For largely historical reasons, and due to the fact that the `rtables`
object model is row-dominant, the exact way that column structure is
modeled is an arcane implementation detail not useful to end users
(much more so than the row structure explored above). Thus we will
largely gloss over it here.

For our purposes here it suffices to say that the analog of the
subtables representing split instructions are implicit in column
space after the first split, as opposed to explicit as we saw
them to be in row space. That said, the relationship between layout
instructions and resulting paths in the table remains valid and useful.

We can see this by looking again at our column paths summary:

```{r}
col_paths_summary(tbl)
```

Column paths have a more rigid structure than their row-based counterparts.
Because column space has no analog to analyze layout instructions, 
All paths corresponding to facets or individual columns come 
in the form of one or more pairs of the form (split name, split value).

Virtually all useful column paths will be of the form above. The only exception
to this is when setting column count visibility to a set of facets via
`facet_colcounts_visible<-`, for which we path to the implicit parent
structure whose children are the facets we are interested in. We saw
this in action in the previous section.

To summarize, as for row space, the relationship between layout
instructions and column paths as follows: column split instructions
create structures pathable via (split_name, split value/facet name)
pairs. Because there is no `analyze` analog for column splitting,
this paradigm is sufficient to understand and predict all column paths.


[^1]: In `rtables` `0.6.13` table layouts which would result in non-unique paths in column space will fail to build. This will likely change to be more in line with the behavior in row space in a future release.

[^2]:  the result-data.frame / ARD machinery knows to remove these uniquification artifacts, so these modifications to the names will not be reflected there. 
