01signal.com

Clock domains, related and unrelated clocks

This page is the first of three in a series on clock domains.

Introduction

Except for very trivial FPGA designs, more than one clock is used for clocking its synchronous elements (flip-flops, block RAMs, shift registers etc). To understand why this is an issue, consider the following drawing, showing the signal path from one flip-flop to another, through a LUT:

 

Crossing clock domains: Two flip-flops and a LUT between them

The LUT (Look-Up Table) in this drawing represents any set of combinatoric logic: Any change in I1, I2, I3 or I4 causes an immediate change in the output O, with some propagation delay.

Hence the signal path goes like this: The left flip-flop's Q output, labeled @foo, changes following a rising edge on its input clock @clk1, which possibly changes the LUT's output O after a brief delay. This goes into the right flip-flop's data input, D. After a rising edge on @clk2, the logic level is copied into Q, labeled as @bar.

Ah, but it's not that simple. All flip-flops have timing requirements: The data input (D) must be stable tsu (the setup time) before the clock's rising edge, and remain stable thold (the hold time) after this edge.

The vast majority of signal paths in an FPGA involve a single clock, so @clk1 and @clk2 are exactly the same clock signal (clock skew disregarded). For such paths, it's possible to calculate the time from the source flip-flop's output Q to the destination flip-flop's input D, through any combinatoric logic it went through, and verify that the timing requirements are met for all paths.

This is what the tools actually do: Recall that the design tools are fed with timing constraints, which inform them on the frequency of the clock. Based upon this information, the tools ensure that the tsu and thold timing requirements are met, or if that's impossible, they report the failure. The tools rely on the fact that it's the same clock that causes the data signal update on the source flip-flop, as the one used to sample the data signal at the destination flip-flop. With the knowledge of the time period between the clock's rising edges (i.e. the clock's frequency), the tools can make these timing calculations.

But what if two different clocks are used, one for the source clock and another for the destination? What if @clk1 isn't the same as @clk2? Is it still possible to make a timing calculation and ensure the destination flip-flop's tsu and thold timing? Well, it depends. That's what the rest of this page is about.

But first, a technical remark: I shall focus this discussion on positive edge triggered flip-flop, i.e. flip-flops that react to positive edges of their clock signals. There are of course other synchronous elements, i.e. logic elements that sample and produce signals based upon a clock: Shift registers, block RAMs and a whole lot of other functional elements. Some of these elements may react on the falling edge of the clock, or even on both. I disregard all this for the sake of simplicity, as it doesn't change anything essential.

I'll also use the expression "X to be clocked by Y", meaning that the synchronous element X has its clock input signal connected to the clock signal Y. Hence it samples its input(s) with this clock, and also updates its outputs with it.

Clock domains and clock domain crossing

A clock domain consists of all synchronous elements (i.e. flip-flops and such) that are clocked by a certain clock signal.

Consider this simple Verilog code snippet:

reg foo, bar;

always @(posedge clk1)
  foo <= !foo;

always @(posedge clk2)
  bar <= foo;

In this example, @foo is clocked by @clk1, and @bar is clocked by @clk2. So clearly, @foo and @bar belong to different clock domains (of @clk1 and @clk2, respectively).

There's nothing to worry about @foo: It depends only on itself. However @bar samples @foo. The former is clocked by @clk2, but the latter by @clk1. So can @bar be used as a legit register? Can we assume that @bar always samples @foo with legal timing, so its behavior is known and repeatable?

Before attempting to answer this question, let's give a name to what just happened here. It's a clock domain crossing: @foo is implemented by a flip-flop which is clocked by @clk1, and there's a signal path from this flip-flop's data output to the data input that implements @bar, which is clocked by @clk2. So the path from @foo to @bar crosses clock domains.

More generally, clock domain crossing refers to when the output of a synchronous element of one clock domain drives the input of a synchronous element of another clock domain, possibly through combinatoric logic.

So even if @bar would have been defined as

always @(posedge clk2)
  bar <= !foo || !bar;

there's still a clock domain crossing path. In this case, it would have started at @foo, gone through the LUT that implements the logic function, and ended at @bar, just as shown in the drawing above.

Related vs. unrelated clocks

Let's get back to the original question: Can @bar in the example above be used as a legit register? Or more generally speaking, if a flip-flop is the destination of a path that crosses clock domains, can we reliably use its output just like any flip-flop's output? Which is the same as asking whether it's possible to ensure the setup and hold times of this destination flip-flop, given that at least one path reaching its data input is clocked by another clock.

The answer is simple and short: If the clocks on both sides (@clk1 and @clk2 in the example) are related, then the destination flip-flop's output is perfectly fine, and can be used like any register, given that the proper timing constraint for this path is met. Otherwise, the timing of the destination can't be ensured, and resynchronization logic must be added to tackle that, as explained in the following pages in this series.

The term related clocks refers to clocks that are derived from the same reference clock in a way that ensures predictable time differences between their rising and falling edges (give or take tolerances and jitter, of course).

A common example of a related clock is when a single FPGA PLL is used to generate several clocks, with known frequency relations between them. In this scenario, FPGA tools usually arrange the clock buffers so that the clock edges are aligned according to these clock frequency relations.

For example, when a reference clock is multiplied by 2 and 3:

Simple clock diagram with clock multiplication

In this example, the clocks x1, x2 and x3 are mutually related, as the timing between each pair of these clocks is predictable.

In the general case, the reference clock is unrelated to any of these, since the timing between its edges with relation to the others may change with temperature and other factors. However if the PLL is wired and configured to ensure a predictable alignment with the outputs it generates, even the reference clock is related.

Knowing the time relations between clocks allows timing calculations for paths going between their clock domains. For example, a path between the domains of the x1 and x2 clocks' rising edges is constrained with the same delay requirements as a path between the x2 clock and itself, because the worst-case time difference between these clocks corresponds to a full cycle of the x2 clock.

Likewise, a path between the x2 and x3 clocks can be constrained, but the shortest time difference is just a sixth of the x1 clock's period. Therefore, the worst case timing requirement corresponds to that of an imagined x6 clock. This meets the case of the second rising edge of the x2 clock vs. the third rising edge of the x3 clock.

So the point with this example is that when the clocks are related, the paths between them can be timed and constrained just like paths within the same clock domain, but with different propagation delay limits — which can be very short in some cases. What makes this possible is that the FPGA tools intentionally make sure that the clock edges are aligned.

Note however that the fact that two clocks are derived from the same reference clock doesn't necessarily make them related. In particular, if the skew between them is uncontrolled or otherwise unknown (which is the case unless the design tools have explicitly used clock distribution resources with equal delays), they should be treated as unrelated.

And even the x1 clk isn't necessarily related to its reference clock, even though they have exactly the same frequency. Unless they are mutually aligned deliberately, their phase relation is unknown.

Because of this phase difference issue, I wrote "a certain clock signal" in the definition for a clock domain above, and not just "a certain clock". The latter could mean, for example, the same clock on the board, that is connected to two different input pins of the FPGA. A certain clock signal, on the other hand, refers to a wire in Verilog or in the design's netlist that carries a clock, like @clk1 and @clk2 in the example above. It's distributed in the Verilog design by virtue of named port connection of instantiations or with plain "assign" statements.

The fact that the clock is exactly the same signal in Verilog means that the tools will make sure to use resources that ensure a low clock skew on hardware. It also has implications on whether the design tools consider the clocks related (more on this below).

Related and unrelated: In what sense?

There are in fact three different aspects to the assertion that two clocks are related in an FPGA design:

So the first aspect is the physical reality, the second is what happens in Verilog code, and the third has to do with timing constraints and reports. This leaves room for choice and interpretation in second and third aspects.

First and foremost, if a pair of clocks are unrelated in reality, the logic design must treat them as such as well, or else the clock domain crossings are unsafe. However if the timing machinery treats them as related, it's harmless in the sense that it won't impact the design's reliability. It will nevertheless cause unnecessary constraining of the clock domain crossing paths.

When a pair of clocks are related in reality, the logic design may treat them as unrelated nevertheless. In other words, it may insert resynchronization logic even when crossing between the clock domains could have been done safely without it.

As for the timing machinery, when a pair of clocks are related in reality, the timing constraints must ensure that the paths that cross these clock domains are constrained. But there's an exception: Paths that are treated by the logic as unrelated clock domains, i.e. with resynchronization logic, are allowed to be unconstrained.

This table summarizes these rules. The table's rows are on whether there's resynchronization logic at the end of the path, and its columns are on whether the clocks are related or not in reality. The middle of the table says whether the path needs to be subject to timing constraints or not.

  In reality, clocks are
related unrelated

Resynch-
ronization logic:

Not applied Timing constraints are required on path This is wrong
applied Timing constraints are not required on path

 

It's easy to mess up

In a complex design, where signals are wired across different modules, it's easy to overlook what signal is clocked by which clock, and cross clock domains unknowingly.

There are two options for harmful mistakes: The first one is crossing clock domains of unrelated clocks without any resynchronization logic. This means that the timing is possibly violated occasionally at the destination flip-flops. Like all timing violation failures, they can be very misleading. The tools may add to the confusion if they deduce from the timing constraints that the two clocks are related, and hence unnecessarily apply constraints to the paths (which don't ensure proper timing, of course). This can confuse a human reader of the timing reports into thinking that the clocks are really related.

The second harmful mistake is when two clocks are related and treated as such by the logic, but the tools don't consider them as such. As a result, the paths between them have no timing constraints applied to them, so there's no guarantee that the timing requirements are met at the destination. Once again, this can lead to timing failures. It's however possible to spot a mistake of this sort in a timing validation checkup of the design.

The only situation where failing to notice a clock domain crossing is really harmless is when the clocks are related, and the timing machinery also considers them as such (and hence applies timing constraints to the related paths). There can still be functional bugs if the clocks have different frequencies, and the logic doesn't take this into account, but this is just a plain bug in the logic.

Treating related clocks as unrelated

Quite often, a single PLL is used to create clocks with different frequencies that are intended for use by different functional modules. From the designer's point of view, they are unrelated, but in reality they are related, and the tools usually consider them as such.

For example, A PLL might generate a 90 MHz clock and a 100 MHz from a 10 MHz reference clock, with the intention of using them as unrelated clocks. Now suppose that some flip-flop that is clocked by the first clock that needs to pass its output to a flip-flop clocked by the second.

Because the period time of the former is 11.11 ns and the latter 10 ns, the worst-case time difference between the rising edges is 1.11 ns (taking all possible phase differences into account). That is equivalent to the clock period of a 900 MHz clock. Hence the timing of the path between these two flip-flops is tight, and even if it's possible to meet, it makes things harder for the tools.

A common pitfall is to use a dual-clock FIFO to interface between clock domains, without paying attention to whether the clocks are related or not. There's no functional problem with this possible mistake, but since dual-clock FIFOs always have resynchronization logic in place, it's pointless to force the tools into constraining these paths. The tools may struggle meeting timing in vain.

Hence it's important to pay attention to clocks that are related but not used as such, in particular those coming out from the same PLL: Odds are that the paths crossing the clock domains between them are unnecessarily constrained, possibly making it harder to meet timing of the entire design.

To solve this, add timing constraints that declare the clocks are unrelated, or define false paths.

Treating unrelated clocks as related

Treating unrelated clocks as related by the logic itself is of course a serious mistake, as this means not putting synchronization logic where it should be. So that's out of discussion.

A less serious mistake is when the tools consider unrelated clocks as related, and apply timing constraints to them as if they were such. As with the other scenarios mentioned above for unnecessary constraining, this is often harmless, but can sometimes make meeting timing harder or even impossible.

Once again, it may be required to add timing constraints that define the clocks as unrelated, or define false paths.

Be sure the tools get it right

For all reasons mentioned above, it's crucial that the FPGA design tools have the desired view of which clocks are related, and which aren't. Or more precisely, that the really unrelated clocks are treated as such, and that the really related clocks are treated as the logic interprets their relations: With or without resynchronization is clock domain crossings.

Making sure that the tools have the same view as the logic design can be tricky however: Each toolset has it's own way to make automatic assumptions on clock relations. The only thing that all tools appear to have in common, is that if the tools' dedicated clocking IP is used to generate two or more clocks with the same PLL, the clocks will be assumed to be related, and paths between them will be timed and constrained, usually based upon the timing constraint given for the reference clock. But don't take even that for granted.

Other than that, each tool has its own way to deduce the relation between clocks, and even different tools from the same vendor may decide differently in certain situations. Most notably, Xilinx' Vivado is more inclined towards assuming that clocks are related, compared to Xilinx' previous flagship toolset, ISE.

So there's no catch-all principle for all FPGA design tools, and even a specific tool can make surprising decisions. The only way to really tackle this issue is checking the timing reports, and produce timing reports for specific path groups to ensure that the timing requirements are correct where they are necessary, and nonexistent where they are not. This is a daunting task, but worth its while.

This concludes the first page in this series. The next page goes through the basics of crossing clock domains.

Copyright © 2021-2022. All rights reserved. (42e6e8c4)