AI infrastructure has become a thermal and energy-planning issue due to the increased power requirements of the computational processes necessary to perform AI tasks. Analyses by Lawrence Berkeley National Laboratory, the International Energy Agency, and the Electric Power Research Institute all report that data-center electricity requirements are becoming a material concern for the nation’s infrastructure [1]-[3]. Previous analyses of data-center power requirements suggest that power estimates for such facilities require detailed, bottom-up analyses of the components to be deployed within them [18]. Thus, the literature on these topics helps establish the “why now” factor that supports the significance of the research presented in this article.

Furthermore, the recommendations regarding the hardware components of AI systems also indicate the importance of this topic. For instance, both the specifications of current data-center GPUs and the published recommendations for cooling such GPUs indicate that AI systems will generate heat at high rates within data centers [4][19]-[21]. While these publications motivate the study and its relevance to AI infrastructure, they do not provide evidence for the results to be published from this study. For instance, the rating of hardware components indicates their design and operational limits but does not indicate the distribution of heat generated within GPUs during AI tasks. Thus, while related, the existing reports from studies on the hardware and its cooling requirements do not provide evidence for the result that this study will publish.

Data-center operators require aggregate metrics. Metrics such as PUE, WUE, rack power, temperature, and facility capacity are all important for data-center operators [5][6][22]. Furthermore, data on power provisioning helps explain why these aggregate metrics cannot be applied to the various populations of GPUs housed in data centers [7]. These metrics are useful to data center operators and owners for planning energy, capacity, resilience, and efficiency within those data centers.

The limitation of these metrics, though, is not the metrics themselves. It is that they do not answer the question of the temperature difference between the hottest and coolest GPUs within a given server node. While it is possible for a server to have an average temperature acceptable for that data center, some GPUs within that node may reach temperatures detrimental to their lifespan and reliability. Thus, these aggregate metrics are inherently different from, and not a substitute for, measuring each GPU within a node to determine whether such a temperature difference exists.

Workloads within AI data centers differ in their power draw, performance characteristics, and the types of useful output they produce. Work on deep learning (LLM) power management, power capping, inference efficiency, and energy benchmarking has shown that the characteristics of these AI workloads affect the power, performance, and energy of the GPUs executing them [12][13][15][23]-[25]. Furthermore, work on benchmarking AI models and workloads indicates that the useful output of those models should be used to evaluate their performance, rather than the power they draw to produce that output [24].

Work on GPUs and GPU telemetry also indicates that each GPU’s power draw and performance characteristics must be determined [26]-[30]. Furthermore, work on sensors built into GPUs suggests that the telemetry systems used to measure GPU power draw may be of low quality [31]. Thus, while the literature on GPUs supports the idea that each GPU within a node should be measured for performance and power characteristics, the work done on those GPUs to measure such characteristics warrants caution and should not be accepted unquestioningly.

The behavior of heat in electronic systems is local, layered, and time-dependent. Work on modeling heat in electronic systems indicates that heat originates at various components of a system, travels through it due to its physical properties, and is emitted by the system itself [8]-[11][32]. Furthermore, work on the transient behavior of heat within electronic systems and the compact models that can be used to model that heat indicates that the heat models of electronic systems are additionally influenced by factors such as the time that elapses within the system, the impedance within the system, and the location within the system at which the heat is generated or emitted [33]-[37].

These different types of work on heat in electronic systems provide the foundation for investigating locality within a node’s GPUs. Such work does not, however, indicate that the GPUs on a node are being measured for parameters as fine-grained as the die-level heat flux of GPUs in public H100 and B200 accelerator models. Instead, this study intends to measure the temperature difference between the hottest and coolest of the eight GPUs in each node. Furthermore, such an investigation of the locality of heat within the GPUs of each node is supported by the existing literature on GPUs that helps to explain the reasons for investigating such a factor as the difference between the hottest and coolest GPUs and why it may be important to measure such a factor within AI data-center nodes.

That literature on GPU nodes also supports the idea that the local heat within those nodes affects GPU performance. For instance, research on the thermal characteristics of distributed deep learning training indicates that factors such as thermal hotspots, power capping, and frequency throttling in GPUs are common in AI distributed training models [14]. Additionally, work on the positional thermal characteristics of GPUs within multi-GPU systems indicates that, for instance, GPUs positioned at the rear of a server may reach higher temperatures than front-facing GPUs on the same node [16]. Work on individual GPUs indicates that GPU temperature can affect GPU efficiency [38][39]. Furthermore, because the rate at which AI distributed training workloads can train AI models is limited by the rate at which the slowest of the GPUs within that distributed node completes its work, local differences in temperature that impact the performance of those GPUs can have an impact on the efficiency with which the entire training node can complete its work [40]-[42]. These mechanisms for the impact of local heat within nodes indicate that the performance characteristics of each GPU within a node can be affected by local thermal conditions, even if these conditions are not directly measured in this investigation.

Several response categories for localizing thermal characteristics in data-center GPUs have been established in the existing literature. For instance, work on workload placement within data centers indicates that workloads can be placed based on the thermal characteristics of the hosts on which they will execute [43]. Furthermore, work on thermal- and power-aware scheduling in data centers indicates that scheduling algorithms exist for data-center GPU clusters and for deep learning (LLM) inference data centers [44][45]. Thus, the scheduling of workloads to GPUs is one of the response categories to consider in light of each GPU’s localized thermal characteristics.

Furthermore, work within cooling architectures of data centers indicates that architectural considerations for data-center cooling systems, cold plates, liquid cooling standards for AI systems (OAI/OAM), the resiliency of cooling systems within the data centers, and research into alternative cooling systems for those data centers help to establish that temperature management is one of the factors concerning data centers that is actively managed by data-center engineers [21][46]-[48]. Work on near-source thermal management and buffering also helps to establish that these response categories exist within the data-center field. For instance, work on heterogeneous integration within electronic systems indicates that thermal management at the die, package, module, and system levels is a consideration for data-center and AI-chip architects [11]. Work on phase-change buffering in power-electronic modules highlights the need to consider methods for heat buffering in electronic systems [49]. While the literature on each of these response categories supports their consideration, it does not provide evidence, using any method, of a reduction in the temperature spread between the hottest and coolest GPUs within a node, or of an improvement in the useful output of the H100 and B200 GPU models.

Thus, while the existing literature on the locality of heat within nodes indicates each of the response categories to that locality, it also highlights the need to investigate each of those response categories and to justify their consideration over alternative responses to the issues of local thermal characteristics within data-center GPUs.

As established by the literature on locality and performance of GPUs in AI data centers, several facts are known about GPUs and their applications in those data centers. First, AI data centers tend to generate significant amounts of energy and heat. Furthermore, while metrics on data-center power, energy, and efficiency are helpful to data-center operators and owners, they are often aggregate measures of GPU performance within the data center [5][6][22]. Additionally, work on deep learning (LLM) power management, power capping, inference efficiency, energy benchmarking, and even benchmarking of AI models and workloads indicates that the power, performance, and energy of GPUs are affected by the workloads executed on them [12][13][15][23]-[25]. Furthermore, work on heat in electronic systems indicates that the heat generated by these systems is local, transient, and often affects their performance [8]-[11][32]. Previous research also indicates that within GPU nodes, some GPUs may experience higher temperatures than others, and these localized temperature differences can affect their overall performance [14]. Finally, work on GPU performance also indicates that each of these performance issues has response categories related to their management and correction [21][43]-[49].

As such, since each aspect covered in the literature is well known in the field, the knowledge gap to be addressed by the present study concerns the differences among current GPU generations in the temperature spread between the hottest and coolest GPUs within those nodes. Furthermore, within each GPU generation, one model may exhibit a greater spread between the hottest and coolest GPUs than another model within the same generation. Thus, one of the contributions of the present study is to establish that current GPUs exhibit these spread differences between the hottest and coolest GPUs on each node. The response to this established issue is not a suggestion to address scheduling, cooling, near-source management, or GPU buffering within those data centers. Instead, this established need to measure the hottest and coolest GPUs within a GPU node indicates that decisions regarding cooling, scheduling, near-source management, and buffering should also be informed by the workload executed on those GPUs.

The main purpose of this research is to investigate whether intra-node temperature divergence, measured during public GPU node sessions, varies across workload-hardware cells. The study’s methodology is designed to fulfill that purpose. Specifically, each node contains eight GPUs, and the temperature divergence among them can be measured. These temperature divergences are aggregated to produce a temperature divergence value for each GPU node session, and it is to be determined whether that value exceeds a certain measurement floor, which varies by workload-hardware cells.

Two main hypotheses are tested in this study. The first hypothesis (H1) is tested to determine whether the intra-node temperature divergence within each public GPU node session exceeds a specified low measurement floor, given the GPUs’ physical characteristics. The second hypothesis (H2) is tested to determine whether intra-node temperature divergence within each session varies with the workload-hardware cell to which the session belongs. Each of these hypotheses is tested using session-level data.

The data collected and used in this study comprise 32 public GPU node sessions recorded by Elsayed, Al-Obaidi, and Farag [17] on NVIDIA’s H100 and B200 AI GPUs. Each of these sessions was recorded on a full GPU node (eight GPUs per node) and contained either image-generation or LLM/text-generation workloads. Thus, there are four workload-hardware cells within the dataset: H100 image generation, B200 image generation, H100 LLM/text generation, and B200 LLM/text generation.

The results table is created by calculating each session’s divergence and using that value to test the hypotheses. The table used for the descriptive summaries and figures of the data is aggregated by each workload-hardware cell. A detailed overview of the collected data is presented in Section 4.

Although the data consist of 45,000 individual timestamp records per session, the unit of analysis is the session itself. Each session contains timestamp records for the same run of workloads on the same GPU node; thus, treating these timestamp records as independent observations would overstate the effective sample size. However, the timestamps within each session are used to calculate various parameters for that session, yielding 32 observations of intra-node temperature divergence for testing the hypotheses.

Let T s , g , t denote the temperature of the GPU g in session s at timestamp t , recorded in degrees Celsius for the eight GPUs in the node. The intra-node thermal spread at a timestamp is the difference between the hottest and coolest GPU at that instant:

where D s , t is the intra-node spread and G s is the set of GPUs in session s . In other words, this calculation subtracts the coolest GPU from the hottest GPU inside the same node at the same timestamp.

Each session is then summarized by its mean spread:

where D ¯ s is the session-level mean spread and n s is the number of timestamp samples in the session. The session-level 95th percentile (P95) is also computed as an upper-tail descriptor. P95 is reported descriptively in Section 5, but it is not treated as direct evidence of sustained operational harm.

All formal tests use a significance level of α = 0.05 . H1 is tested with a single-variable and one-sided t-test of the session-level mean spread against the practical measurement-resolution floor, δ floor = 1 ˚C, defined in Section 4.2:

where D ¯ is the mean of the 32 session-level spreads, s D is the standard deviation of those spreads, and N is the number of sessions. The test asks whether the observed spread exceeds a conservative measurement-resolution floor. It does not ask whether the spread harms performance, and the floor is not a throttling threshold.

H2 is tested using a two-factor ordinary least squares model of session-level mean spread that accounts for each hardware indicator, the workload-family indicator, and their interaction term, to test whether the direction of the spread varies across workload-hardware cells. The same effect may also be expressed as a double-difference contrast statistic. The model, its coefficient table, and its double-difference equation are presented in Section 6, where the results are applied and interpreted.

In addition to the main model, a series of robustness tests is performed on the H2 interaction term: using HC3 heteroskedasticity-robust standard errors, performing a permutation test with 20,000 permutations and seed 20260625, performing a cell-stratified bootstrap with 20,000 resamples and the same seed, performing a leave-one-session-out model refit, and performing a power-covariate model in which the mean total GPU power for each session is included as a Z-scored covariate. Each of these models evaluates the robustness of the interaction effect in the collected data; none of them addresses the confounding effect of the data collection date, as described in Section 4.4.

The study is reproducible using the public data collection source and the session- and group-level summary tables published in Section 4. The analysis and figure-producing software utilizes the Python programming language and its standard scientific packages, pandas, NumPy, and SciPy. The figure-producing software utilizes session- and grouped-level tables to create both a figure of workloads and hardware that exhibit a reversal of the expected spread across nodes, and a figure of the distribution of session-level spreads.

To fully reproduce the study, others must ensure they use the same unit of analysis as the original study. Additionally, they must report the criteria for inclusion in the study, the means of de-duplication, the labeling of the hardware models, workload families, dates of collection, timestamps of data collection, the definition of spread, and the fields for missing telemetry data.

The method of this study can only support the claim that the H100 and B200 GPUs exhibited thermal spread within each graphics processing unit node during the analyzed sessions, and that the direction of that spread did not remain consistent across workloads and hardware models. Any additional claims made outside of this narrow parameter, such as that the sessions were thermally throttled, that the GPUs lost useful computing power during those sessions, that the workloads were the cause of the reversal of spread within the nodes, or that any intervention in cooling, scheduling of workloads, near-source thermal management, or thermal buffering of the GPUs was performed during those sessions would require a proof-grade data collection methodology that co-measured those fields; currently, those fields are all represented in the study in their missing form (as presented in Section 4).

Within the narrowed dataset to be used, there are four cells in the analysis: each of the H100 and B200 servers will be analyzed for image generation and LLM/text generation tasks. Each of the image-generation tasks was performed during 7 sessions of each type of GPU, and each of the LLM and text-generation tasks was performed during 9 sessions of each type of GPU. Each session contained approximately 45,000 rows of timestamped data. Thus, the image-generation tasks contained 315,000 rows on each of the H100 and B200 GPUs, and the LLM and text-generation tasks contained 405,000 rows on each GPU type. Each session represented running the eight GPUs on a server. These sessions were not randomly sampled from the available servers for each GPU type. Instead, they are public sessions used to determine whether the thermal spread within each node is within an expected measurement floor and whether any differences exist among the various workloads and GPU types.

The main variable to be analyzed for each server is the temperature spread within the node, defined as the difference between the hottest GPU’s temperature at the same timestamp and the coolest GPU’s temperature on that server. This metric will be calculated for each session, and the mean spread within each session will be used to test the hypotheses formulated in this paper. Additional data to be published in this paper include the mean P95 within each session. The session-level P95 spread is reported as an upper-tail descriptor of intra-node spread; it is not a direct identifier of the hottest GPU. However, that statistic should not be used to assess the performance of the GPUs under analysis under normal operating conditions.

The practical measurement floor for the temperature-spread test is 1˚C, a conservative estimate given the integer resolution of public temperature measurements. This floor is not a threshold for system performance, nor is it a threshold for throttling or sensor calibration. Finally, this threshold is not a performance threshold in the context of this study; it is only a floor for performance measurements.

The dataset is strong for one task: measuring the thermal divergence of GPUs within a single node under the observed workload-hardware cells. However, it is not proof-grade for establishing causation across the entire chain from workload to useful-output loss. Such a proof-grade study would have required data for each of the following fields for each session in the dataset: workload, per-GPU power, per-GPU temperature, clock frequency for each GPU, throttle violation for each GPU, cooling condition for each GPU, and useful output for each GPU.

Due to a lack of data on clock frequency and throttle violations for each GPU, this paper cannot make claims about thermally throttled sessions. Because it lacks data on the useful output of each GPU, this current research cannot claim any loss of useful output due to heat. Finally, because we lack data on cooling conditions for each GPU, we cannot quantify the impact of cooling-related factors on thermal divergence within the node. Each of these factors, however, remains a candidate for investigation in the future in baseline-versus-intervention studies.

Finally, in this analysis, the maximum temperature across all GPUs on the node was 77˚C in the grouped data. This value represents the ceiling temperature of the GPUs in this dataset, and its presence in the report does not indicate that any of the work sessions were thermally limited.

Perhaps the most important limitation of the dataset is the confusion about the dates on which the data were collected. The four different workload-hardware cells were each collected on four dates: the H100 image-generation sessions on 2025-09-16; the B200 image-generation sessions on 2025-09-17; the H100 LLM/text-generation sessions on 2025-09-20; and the B200 LLM/text-generation sessions on 2025-09-22. This is a mix of midweek, weekend, and early workweek, with unknown times of day. Thus, each workload-hardware cell is aliased to the date on which it was collected, as well as to any confounding factor that differed across these collection dates. Consequently, while the interaction effect can be analyzed to determine whether there is a statistically supported relationship between the workload-hardware cells, no conclusions can be drawn about the cause of the reversal among those workloads.

Beyond this important confound, other biases are introduced by the use of this publicly available secondary dataset. For instance, the sessions are a publicly available sample of those conducted in the establishment, rather than a sample collected in a way that minimizes selection and availability biases. Additionally, while the dataset contains two accelerator generations and two workload families, any conclusions drawn from the data should not be generalized to other GPU types, workloads, node designs, rack designs, or cooling architectures. Furthermore, since the 7 or 9 sessions within each workload-hardware cell were collected on the same day, across four different days overall, and may share common environmental conditions, it is appropriate to generalize within each workload-hardware cell. However, any generalization beyond those categories would be invalid. Additionally, while a slot within the node labels the GPUs, those labels do not indicate either the cooling condition of each GPU or the airflow from the node to each GPU. Thus, while it is possible to determine the temperature of each GPU within the node, it is not possible to determine its cooling conditions or airflow.

Finally, an issue that arises from using these different aggregated measurements is that the data for nodes, facilities, and GPUs can inherently differ. For instance, a data center has GPUs with acceptable average temperatures, yet still has a temperature difference between the hottest and coolest GPU within a single node. Additionally, while higher total power consumption by a data center may indicate it is hotter than other data centers with lower total power, this does not imply a difference between the hottest and coolest GPU within each data center. For these reasons, while the power of each data center can be measured, the study directly measures the temperature differences between GPUs within each node, treats total power as a context variable, and tests the robustness of the findings.

Table 2 summarizes the four workload-hardware cells. Each cell includes either 7 image-generation sessions or 9 LLM/text-generation sessions, for a total of 32 sessions. Be advised that the effective number of independent collection environments may be closer to 4 collection blocks than to 32 fully independent sessions. Thus, the reported p-values may reflect small-sample effects and should be interpreted with caution.

Table 2.Session-level descriptive statistics by workload family and hardware (Values are means across sessions within each workload-hardware cell unless otherwise noted.).

The mean session P95 spread is the average of each session’s 95th percentile. This metric does not represent the system’s typical conditions.

Across all 32 sessions, the mean session-level intra-node spread is 13.58˚C (SD = 5.03˚C). Individual sessions range from 5.90˚C to 23.89˚C, and all 32 sessions have mean spreads that exceed the 1˚C floor for practical resolution of temperature measurements (Section 4). The smallest cell mean (8.72˚C) is approximately nine times this floor. These measurements are within-node temperature differences; Figure 1 presents the distribution of these session-level measurements. A statistical test to determine whether these differences are significantly above the measurement floor is presented in H1 in Section 6.

Figure 1.Distribution of session-level intra-node thermal spread across 32 H100/B200 node sessions. Each point represents the mean of the hottest-minus-coolest GPU temperature spread for that session. Each session exceeds the 1˚C floor for practical measurement of temperature differences, with an overall mean of 13.58˚C and a 95 percent confidence interval of 11.77˚C to 15.39˚C. The figure uses sessions rather than individual rows in the data table to reflect that measurements were taken over time within each session.

The mean spread across nodes of the H100 and B200 hardware components is higher for image-generation workloads than for LLM/text-generation workloads (Figure 2). For the image-generation workloads, the H100 has a mean intra-node spread of 18.45˚C compared to the B200’s 8.72˚C (a difference of 9.74˚C using the unrounded session-level spread measurements; the rounded means differ by 9.73˚C). For the LLM/text-generation workloads, the spread within the B200 is higher than within the H100; the B200’s mean spread is 17.09˚C compared to the H100’s 10.06˚C (a difference of 7.03˚C). Thus, the difference between the H100 and B200 within the image-generation workloads is reversed relative to the LLM/text-generation workloads; the double difference between the hardware components is 16.77˚C. Because the sessions within each workload family were collected on separate dates (as noted in Section 4), these differences cannot be attributed solely to within-family variation.

Figure 2. Workload x hardware reversal in mean intra-node thermal spread. The figure plots the mean hottest-minus-coolest GPU temperature spread within each group of sessions held on each eight-GPU node. For image-generation sessions, the H100 GPUs exhibit a greater spread than the B200 GPUs. For LLM/text-generation sessions, however, the B200 GPUs exhibit a greater spread than the H100 GPUs. The difference between these amounts is 16.77˚C. This contrast does not indicate that session type caused the reversal of the trend in GPU temperature spreads, as each workload-hardware combination is aliased with the date on which the GPUs were measured.

Two additional aggregate perspectives of the same sessions do not reproduce the spread ordering. First, the node-average temperature does not reproduce the ordering of spread within the nodes. For instance, the cell with the highest mean GPU temperature was LLM/H100 at 51.90˚C, with a mean spread of 10.06˚C across its GPUs, one of the lowest spreads within the cluster. By contrast, the cell with the highest spread (image/H100) had a mean temperature of 42.19˚C. This relationship between spread and temperature is consistent with the literature on the thermal spread of GPUs [8][10]. The temperature measurements were performed at the node level, not the die level, meaning that the spread was calculated for the eight GPUs within each node.

Second, total GPU power does not have a monotonic relationship with spread within the cluster. The cell with the highest spread (image/H100) had the lowest mean total GPU power among the four compute cells in the cluster (1995 W). Both B200 models had higher total GPU power than the H100 models for both workload families: image and LLM/text generation. B200 used 3165W and 4712W compared to 1995W and 3717W, respectively. B200, however, had a wider spread within the LLM/text-generation workload. Section 6 presents a model that predicts spread within compute cells, reversing the relationship between total GPU power and spread. Thus, the spread among GPUs on a compute node cannot be predicted from the average temperature or total power of the GPUs in the node.

The maximum temperature of each GPU in each of the four compute cells within the data center was measured. Each GPU reached a maximum of 73, 61, 70, or 77˚C (Table 2). The highest maximum temperature among GPUs was 77˚C, achieved by GPUs in the LLM/B200 compute cell. Thus, 77˚C is the maximum temperature of any GPU in the cluster. It is important to reiterate, however, that this temperature does not indicate that the GPUs were at the thermal throttling threshold. GPU clock frequency, the reason for any potential throttling, and any metrics related to the usefulness of the computations performed by the GPUs were not available for these measurements. Determining whether thermal throttling occurred on any GPU is therefore outside the scope of this paper (see Section 4).

H1 asks whether the mean session-level spread exceeds the 1˚C practical measurement-resolution floor, δ floor , defined in Section 4. Because the per-GPU temperatures are recorded at integer resolution, δ floor = 1 ˚C corresponds to one quantization step; the test therefore asks whether the observed spread exceeds a conservative measurement-resolution floor, not whether it reaches any particular operationally consequential magnitude.

H1₀: μ D ≤ 1 ˚CH1_A: μ D > 1 ˚C

A one-sample, one-sided t test against the floor (Equation (3)) yields a mean spread of 13.58˚C (SD = 5.03˚C, SE = 0.89˚C), t(31) = 14.16, p < 0.001; the two-sided 95% confidence interval for the mean spread within nodes is [11.77, 15.39]˚C. The null hypothesis H1₀ is rejected. These results do not rely on the pooled test result alone: individual session means range from 5.90˚C to 23.89˚C, and each of the 32 session means exceeds 1˚C. In addition, each of the four cell means for workload and hardware configurations exceeds 1˚C (the smallest mean is 8.72˚C). Thus, the within-node spread exceeds the resolution of our measuring instruments in all 32 analyzed sessions. Since H1 does not claim that any measured spread of node temperatures within a system is harmful, the confidence interval refers to the 32 analyzed sessions rather than some general population of system sessions.

H2 tests whether the direction of intra-node spread depends on the workload-hardware cell. Using the two-factor model,

with B200 = 1 for B200 sessions and LLM = 1 for LLM/text-generation sessions (baseline = H100 image generation), the hypotheses are:

H2₀: β 3 = 0 H2_A: β 3 ≠ 0

The fitted coefficients are reported in Table 3.

Table 3. Two-factor OLS coefficients for session-level intra-node thermal spread (N = 32; baseline = H100 image generation).

The interaction term is β 3 = 16.77 ˚C, F(1, 28) = 67.88, p = 5.75 × 10 − 9 , R 2 = 0.71 , adjusted R 2 = 0.68 , and partial η 2 = 0.71 . H2₀ is rejected. The interaction estimate matches the descriptive double difference reported in Section 5:

These cell means are rounded to two decimal places. Thus, the subtraction that defines the contrast within the cell yields 16.76˚C; the exact value based on the session means is 16.77˚C, which matches the interaction coefficient. Furthermore, the gap between the H100 and B200 for image generation is reported as the rounded value of 9.73˚C rather than the unrounded contrast coefficient of 9.74˚C (Section 5).

The effect is carried almost entirely by the interaction effect. A Type-II analysis of variance decomposition indicates that the sum of squares for the main effect of the type of hardware was 0.74, that for the main effect of workload was 0.001, and that for the interaction effect was 553.65 (with a residual sum of squares of 228.37). Thus, each of the main effects is very small compared with the interaction effect. The reason for these small main effects is that the effect of the hardware type varies with the workload: B200 processors are 9.74˚C less spread out in their performance in image generation than on H100 servers, yet are 7.03˚C more spread out in their performance in LLM/text generation. Furthermore, in H100 systems, image generation exhibits an 8.40˚C spread difference compared to text generation. In contrast, within the B200 systems, text generation exhibits an 8.37˚C difference in spread relative to image generation. These two sets of differences cancel each other out, an effect that is accounted for by the interaction effect. Thus, the coefficients that are statistically significant for B200 and LLM in Table 3 are each conditional on the H100 baseline for image generation, rather than on main effects generalizable to all workloads.

Lastly, it should be noted that this study investigates only the association between workload type, hardware type, and intra-node thermal spread within each system; no claims can be made regarding causation. Moreover, because each workload was tested on separate dates, any performance differences may be attributable to the testing date. Thus, no conclusions can be drawn from this analysis regarding whether the type of workload caused the differences in hardware performance.

The following four checks are used to examine whether the interaction estimate is sensitive to distributional assumptions or to individual sessions:

Heteroskedasticity-robuststandarderror(HC3): SE = 2.20, t = 7.62 (p = 2.70 × 10⁻⁸), accommodating unequal cell variances.Permutationtest (20,000 permutations, seed 20260625; workload labels shuffled within each hardware group, preserving group sizes): the observed β₃ exceeds every permuted value in both tails (permuted range −13.84˚C to 12.08˚C), so the plus-one-corrected two-sided permutation p is 1 / 20 , 001 ≈ 5.0 × 10 − 5 .Stratifiedbootstrap (20,000 resamples stratified by cell): 95% CI for β₃ = [13.08, 20.54]˚C, excluding zero.Leave-one-session-out: refitting after dropping each session in turn yields β 3 ∈ [ 15.86 , 17.47 ] ˚C, positive in all 32 refits.

These checks help ensure that the interaction effect is preserved across the different types of data. Each workload-hardware cell has only 7 - 9 data sessions. Therefore, these distribution-free tests serve as an alternative to the normality and homogeneity-of-variance assumptions underlying parametric tests. Neither of these tests addresses the confounding effect of the date on which the sessions were collected (see sections 4 and 10).

In Section 5, using descriptive statistics, we established that the cell with the highest spread in GPU temperatures also had the lowest total power delivered to those GPUs. A statistical model was created, with the mean total GPU power per session as a covariate, to test whether the interaction between the workloads and the hardware specifications was an artifact of total GPU power. The interaction term remained statistically significant in the model (β₃ = 17.21˚C, t = 13.04, p < 0.001). However, the total GPU power was also significantly related to session-level spread (3.24˚C per standard deviation in total power, t = 6.31). Thus, the total power of the GPUs does not account for the interaction between workloads and hardware types. Furthermore, as with the other tests in this section, the run-day, node, and environment confound are not accounted for by this test, nor is there any available covariate to address them. Thus, this test served only as a descriptive check for the alternative explanation of the interaction with total power, rather than as a means of causal control over that variable.

These tests provided evidence for both hypotheses: H1 and H2 were found to be true across the 32 GPU sessions. However, each of these statistical analyses only helps to prove that the thermal spread within the nodes of the GPUs was experienced above the thermal floor of those GPUs (H1) and that there was a statistically significant interaction between the workloads that were assigned to each GPU and the type of GPU hardware that was utilized (H2). Neither hypothesis, however, establishes a link between the workloads and the temperature spread within the GPUs, that the GPUs were experiencing thermal throttling as a result, or that their processing power was reduced. Furthermore, while these analyses used N = 32 session-level observations, the interaction between those workloads and GPUs was only established across the four blocks of dates on which the sessions were collected (Section 10). Thus, these hypotheses are supported but limited to the interpretations that can be made about these two variables at the current level of analysis. The implications of these results are discussed in sections 7 and 8.

The within-node temperature spread exceeds the 1˚C floor in all 32 observed sessions. The mean spread across all sessions is 13.58˚C within each node, and each of the sessions individually exhibits within-node spreads above the 1˚C floor (with the lowest mean within any workload/hardware cell measuring 8.72˚C; Figure 1; Section 6.1: t(31) = 14.16, p < 0.001, 95% CI [11.77, 15.39]˚C). Thus, in support of Question 1, these results show that each observed session includes within-node thermal divergence exceeding the 1˚C measurement resolution floor. The smallest mean within each workload/node cell is 8.72˚C; however, these results apply only to the analyzed sessions and do not generalize to GPU systems beyond the H100 and B200 evaluated.

The hardware that exhibits the largest temperature spread varies with the type of workload the servers perform. For instance, during image-generation workloads, the H100 GPUs exhibit a wider temperature range than the B200 GPUs (18.45˚C vs. 8.72˚C). However, for LLM/text-generation workloads, the B200 GPUs exhibit a larger spread in temperature than the H100 GPUs (17.09˚C vs. 10.06˚C) (Figure 2; Table 2). This workload × hardware interaction is statistically significant and is driven mainly by the interaction term: β₃ = 16.77˚C, F(1, 28) = 67.88, p = 5.75 × 10⁻⁹, partial η² = 0.71 (Section 6.2; Table 3). The finding is robust to adjustments for heteroskedasticity-robust standard errors, permutation tests, stratified bootstrap sampling (95% CI [13.08, 20.54]˚C), leave-one-session-out model refitting, and adjustment for total GPU power (sections 6.3 and 6.4). These results support question 2: there is an association between divergence and workload-hardware cells. However, because each cell contains samples collected on a single date (Section 4), it is not possible to conclude that the workload affected the observed reversal in the direction and magnitude of the divergence.

Since the hardware with the higher intra-node spread within a given workload family is the hardware that reverses within workload families, and since neither the main effect of the hardware nor the main effect of the workload explains the spread within a given hardware system, the effect is best characterized as a crossover interaction (Section 6.2). Thus, within these sessions, the GPU identity alone does not indicate which configuration is likely to exhibit greater intra-node spread. Across the four workload-hardware cells, node-average GPU temperature and mean total GPU power do not reproduce the ordering of intra-node spread; notably, the cell with the highest mean spread had the lowest mean total GPU power (Sections 5.4 and 6.4).

In response to the main research question, the observed data indicate that intra-node spread is not explained by GPU hardware identity alone; instead, it differs across workload-hardware cells. The spread among GPUs is explained not solely by their manufacturers but also by the characteristics of the workloads they are assigned to execute. This finding does not invalidate the use of the aggregate metrics for their intended applications. The implications of this finding are discussed in Section 8.

The reversal of the association between hardware and intra-node thermal locality implies that the intra-node locality of observed sessions cannot be understood from the identity of the accelerator hardware alone. The same hardware displays larger intra-node spreads under some workloads than under others and exhibits varying degrees of intra-node locality across different hardware brands. It is this crossover phenomenon that is observed in this study.

It is important to provide a view of the data aggregated across the entire facility and at individual nodes within it. Metrics at the level of the entire facility help in understanding its efficiency and data center management [5][6]. Metrics at the level of individual nodes help facility operators to understand their individual nodes’ power and thermal capabilities [7]. The infrastructure’s ability to integrate design and operations components ensures that thermal and power planning are coordinated in the design phase [4].

These metrics are, by construction, not capable of providing insight into how temperature is distributed across GPUs within a single node. While the analyzed sessions have provided insight into differences in average GPU temperatures within a node and the spread of those temperatures (Section 5.4), these metrics remain insufficient to understand the thermal locality of GPUs within a node. Thus, while these aggregate metrics are useful for their intended use cases, they remain incomplete as metrics for assessing the thermal locality of GPUs within a node.

If the thermal locality of a GPU can be understood in part through the hardware on which it runs and the workload it executes, then it can be best understood when co-measurements are made of those variables. Such a dataset would include measurements of the workload executed by each GPU, each GPU’s power, and each GPU’s temperature at the same time intervals.

While the available dataset contains the power and temperature of each GPU, it does not include variables for measuring each GPU’s clock frequency, whether it is being throttled, the cooling condition of each GPU, or the amount of useful output. To measure these variables would require the inclusion of instrumentation to measure the clock frequency of each GPU [29], whether it is being throttled or not [30], the accuracy of the power sensors for each GPU [31], the cooling condition of each GPU [15], and the ability to measure the amount of useful output of each GPU [25].

Thus, co-measurement is proposed as a means of making decisions about thermal management and cooling. Regarding research question 3, cooling, scheduling, and thermal management decisions should be based on co-measurement of these variables rather than on the identity of the hardware or the average temperature of the GPU nodes.

There are several potential approaches to addressing intra-node thermal locality, based on the recognition that workloads can characterize this locality. One set of approaches involves the scheduling of workloads. Workload-aware scheduling has a long-standing tradition in the literature [43]. Thermal-aware and power-aware scheduling approaches have recently been proposed for data centers that deploy GPUs and LLM inference clusters for machine learning tasks [44][45].

Additional approaches to managing thermal locality involve the data-center facility’s cooling system. Approaches to cooling that introduce liquid-cooling architectures and enhance the resiliency of data-center cooling systems have been suggested [21][50]. Additionally, public cooling research programs are considering methods of effectively removing heat from chips and GPUs in data centers [48].

A third set of approaches relates to the thermal management of the chips themselves. Methods for managing the heat generated by chips have been proposed at both the chip package level (encompassing all dies within a given package) and within the chips themselves (e.g., within processor nodes) [11][37]. Techniques involving phase-change materials to buffer heat in power electronic circuits have been discussed in the literature, though with goals different from those for managing heat in data-center nodes housing GPUs [49]. Thus, approaches to thermal management of data-center nodes and their GPUs (especially if GPU chips include techniques for thermal buffering of their nodes) represent one potential way to address the problem of providing thermal locality for GPUs within each data-center node. Unfortunately, no studies were conducted to determine whether these thermal-buffering methods would reduce thermal spread to GPUs within each data-center node or increase their useful output.

None of these candidate responses to intra-node thermal locality of GPUs within data centers are supported by the findings of this study. The dataset did not contain a field for measuring interventions in the cooling or operation of the GPUs. Thus, it was not possible to determine the effect of any of these candidate responses.

Any response to the problem of intra-node locality needs to be evaluated through a baseline-versus-intervention study that measures the same variables under the same workloads and on the same GPUs. Ideally, such a study would utilize the same data-center nodes to test these responses (Section 10). This study’s contribution is the suggestion that the intra-node thermal locality of GPUs should be treated as a variable understood through its workload-aware, co-measured nature prior to making any cooling, scheduling, or thermal-management decisions. Thus, while each of these response categories is a potential means of addressing the problem, this study does not provide support for any particular candidate response.

The study was conducted to determine whether the thermal locality within GPU nodes of AI systems depended on the workload performed by those GPUs rather than being an inherent feature of the GPUs themselves. In each of the 32 observed public sessions on eight-GPU nodes, the higher-spread GPU hardware within each node reversed depending on the workload performed by that node.

Two results support the argument that thermal locality within GPU nodes is not an inherent feature of the GPUs. First, the within-node spread of GPU temperatures often exceeded the 1˚C floor used to measure temperatures within each node; the mean spread within each node was 13.58˚C across sessions, and the smallest mean spread within any workload-hardware cell was 8.72˚C (H1). Second, the direction of temperature spread within each node differed across hardware nodes; the H100 GPU models exhibited greater spread in workloads that performed image generation, while the B200 GPU models exhibited greater spread in workloads that performed LLM and text-generation tasks. Furthermore, the workload-by-hardware was statistically significant, with a crossover magnitude of 16.77˚C (H2). Each of these workloads-hardware cells occurred on the same dates as the data were collected, supporting only the notion that there is some relationship between the two variables, rather than providing evidence of a relationship that developed over time as a result of the performance of those workloads.

The findings of this research indicate that the thermal locality within AI GPU nodes depends on the workloads they perform. Thus, thermal locality within these nodes should be evaluated on a per-GPU basis rather than across the entire node. However, the research did not determine whether there was any loss of the GPUs’ computational usefulness due to differences in thermal locality across nodes, or whether any cooling or thermal-buffering solutions were effective in mitigating these problems. Future research ought to examine these performance effects and potential thermal-management solutions.

1) Co-measure before deciding. Before committing to any scheduling, cooling, management, or buffering effort, characterize the workloads and GPUs that are to be used to collect the co-measured data described in Section 8.3.

2) Use intra-node spread as a standing signal. Per-GPU temperature spread should be tracked in addition to within-data-center and within-node metrics (which can be derived from the within-node spread).

3) Test the target workload on the actual hardware. Because workloads differ in their thermal characteristics, only by testing the target workload on the actual hardware can thermal characterization and procurement decisions be made.

4) Consider the standard of baseline versus intervention. Any response to the problems described in this paper should be applied only in the instances in which measured baseline data indicate the response will reduce the spread between GPUs within a node and increase the amount of useful output from those GPUs, best determined through testing the workloads on the same node in interleaved jobs (Section 10).

Observationaldesignandthecollection-dateconfound. This is the most important limitation of the study. Each of the four cells containing combinations of workloads and hardware GPUs was collected on a single date: H100 image generation on 2025-09-16; B200 image generation on 2025-09-17; H100 LLM/text generation on 2025-09-20; and B200 LLM/text generation on 2025-09-22. Thus, each workload-hardware cell is confounded with the collection date for that workload. Consequently, it is not possible to separate any effect of the workload from the effect of the date on which it was collected. Thus, the reversal is supported by the workload-by-hardware interaction, but the interaction term does not remove the collection-date confound; the robustness checks all conditions on the observed cells and therefore cannot resolve it.

Samplesize,clustering,andunbalancedcells. The study is based on 32 session-level observations. Such a sample size is appropriate for providing evidence regarding the workloads and hardware types in general. However, the public layout of the sessions provides only four distinct collection dates for the workloads and hardware types; each workload-hardware combination was collected on a single date. Furthermore, the sessions for each workload-hardware combination were collected sequentially on the same date. Thus, the sessions within each of the four cells are not independently collected. Additionally, there are seven sessions of image-generation workloads but nine LLM/text-generation workloads, indicating an unbalanced study. Finally, the sessions have varied configurations: batch size, image size, model size, sequence length, and DeepSpeed settings vary across sessions and are not independently separated from the categories of workloads and hardware.

Descriptivestatisticslimits. The descriptive statistics in Section 5 provide only a summary of the 32 observed sessions. They are not to be interpreted as population estimates, hardware benchmarks, or indicators of any general difference between GPU generations. Thus, the reversal stated in Section 5 applies only to the observations of workloads and hardware types; the descriptive statistics motivate the statistical test, but they are not the test itself.

Singlepublicsecondarydataset. The study was performed on a public dataset collected by authors different from those who conducted this study [17]. Thus, the findings of this study are based on the public and separate datasets.

Missingmeasurementfields. The temperature measurements did not include clock frequency, throttling reasons, cooling loop state, coolant temperature, or the GPUs’ useful output.

Observedthermalregime. The maximum temperature of any GPU across all workload-hardware combinations was 77˚C. This is not the threshold at which the GPUs are designed to be throttled. Because the reasons for and clock frequency measurements of the GPUs are unavailable, no conclusions can be drawn about whether the GPUs were throttled during the observed sessions.

Scopeofgeneralizationandmeasurementscale. These results apply only to the two accelerator generations of GPUs tested and to the two types of workloads performed on them. These results cannot be generalized to other hardware, other types of workloads, or to any other data-center cooling architectures. Furthermore, temperature-difference measurements were taken only within each node of the eight GPUs; the results do not apply to within-die or within-package temperature differences or to heat-flux measurements.

Nointerventionvalidated. This study found no evidence of interventions related to the GPUs or the data center. For instance, no interventions related to thermal buffering, cooling, scheduling workloads on GPUs or any other vendor- or product-specific interventions were validated by this study.

The recommended next step is a study that addresses the confounding factors in this public dataset. Such a study would run both workload families on the same node or on nodes matched for cooling and time window. This design would then be repeated on both H100 and B200 nodes. Within each node, workloads would be executed in an interleaved, randomized order. In such a study, it would be necessary to measure the workload state, GPU power draw, GPU temperature, GPU utilization, memory behavior, clock frequency, throttle state, cooling condition, and the useful output of each GPU on each node. Each of these metrics is necessary to determine whether the node is experiencing thermal spread issues.

Future intervention studies would employ a baseline-versus-intervention study design. Each intervention candidate (workload-aware scheduling, cooling, thermal management, buffering) would be tested under matched workload, hardware, cooling, and time-window conditions. Each intervention would be credited only for reducing the thermal metric and for increasing (or at least not decreasing) the nodes’ useful output. Thus, future work should perform a study that measures the workload, the hardware components of the nodes, the local thermal spread of each node, the cooling condition of each node, and the useful output of each node. This study identifies the problem of insufficient measurement of these variables. Furthermore, it demonstrates the impact of workload and hardware on thermal locality within nodes. However, it does not provide solutions for establishing a causal relationship among workload, heat generation, throttling, and loss of useful output. That relationship is the problem for the next study on the topic.