`display-tex-shell-buffer' (defined in `window') is marked obsolete with
the message "use a `(category . tex-shell)' condition in
`display-buffer-alist'."
The default value of `display-tex-shell-buffer' contains
`(inhibit-same-window . t)'. I feel that doing
(add-to-list 'display-buffer-alist
(cons t (cons #'display-buffer-same-window nil)))
should have the desired effect of displaying a TeX shell buffer in the
same window (when compiling TeX files in latex-mode), without having to
alter the value of the obsolete variable `display-tex-shell-buffer'.
But `(inhibit-same-window . t)' in the default value prevents that from
happening. Instead another frame is popped up or reused.
You can override `inhibit-same-window' by the corresponding category:
(add-to-list 'display-buffer-alist
'((category . tex-shell)
(display-buffer-same-window)
(inhibit-same-window . nil)))