An increasing number of substation security and control applications are starting to use Ethernet as the primary communication channel. It is therefore reasonable to assume that Ethernet will become the main communication medium for future substations, not only at the transmission level, but associated services such as precise time synchronisation.

Digital stations require a time-optimised network architecture that fully integrates all elements of the IEC 61850 automation system. Time domain optimisation is understood to include, the criticality of latency and packet fluctuations, but also, and perhaps most importantly, the ability of the network to carry precise time synchronisation. Many IEC 61850 applications require high accuracy time synchronisation and fast, if not lossless, communication redundancy to operate correctly and achieve the level of reliability required in the power industry.

Given the cost and reliability of measurement systems based on distributed Global Navigation Satellite Systems (GNSS) receivers, it is advisable for distribution units to use fewer distributed GNSS receivers and opt for the implementation of more reliable and robust centralised time distribution systems with different input data sources and validation algorithms. This will address issues of time synchronisation quality, synchronisation network maintenance and resistance to jamming or spoofing. Assuming a precise and centralised station time source is available, reliable and accurate time distribution will be essential for the correct operation of critical control and measurement applications requiring timing accuracy of no worse than ±1 μs.

In terms of cost, complexity and reliability compared to the various time synchronisation methods available, the Precision Time Protocol (PTP) defined in IEEE1588 v2.1 is a good candidate for solving synchronisation problems at substations.
This article provides an introduction to IEEE 1588 v2.1 and provides general knowledge of precision time synchronisation application issues. General requirements for precision time in substation applications are also addressed.

IEEE 1588 - Introduction

The current standard IEEE 1588 v2.1: 2019 (previous version IEEE1588 v.2.0:2008) defines the second generation of Precision Time Protocol also known as PTPv2.1 or 1588v2.1. The PTP standard provides the ability to achieve highly accurate time synchronisation with Ethernet devices (typically switches) through hardware-software processing of accurate time and PTP messages. Hardware processing with software support, can compensate for the uncertainty introduced by real-time operating systems and other delays created during the synchronisation process in the synchronisation distribution devices as well as in the devices that are synchronised. The great advantage of the second version of PTP is that it does not affect the operation of other protocols running on the Ethernet network and reduces the CPU load. As a result, it is possible to distribute it with other time-sensitive station protocols such as Generic Object Oriented Substation Event (GOOSE), Sampled Values (SV), Phasor Measurement Units (PMU) and other station automation protocols. It is important when building stations to equip Ethernet switches with full hardware support for PTP together with the required profiles, i.e. at least Power Profile - IEEE C37.238 and IEC 61850-9-3 available only on the highest class of switches.

PTP supports multiple Master Clock (MC) support, allowing devices to select the best Grand Master Clock (GMC) through the Best Master Clock Algorithm (BMCA). In the event of a degradation in the quality of the clock selected as master, it is possible to automatically select another clock in real time to act as the new GMC for the receiver with better accuracy than the previous one until it regains its synchronisation capability at the previous level.

One of the main features of PTP is its flexibility, as it can be used for many time synchronisation applications with less than 1 μs accuracy.

The objectives of PTP are to achieve:

• nanosecond accuracy of time distribution,
• minimising CPU resources at the expense of hardware support,
• implementation of precise synchronisation in data networks,
• support of clocks with different capabilities such as precision, accuracy and stability.

IEEE 1588 PTP is used in many areas such as telecommunications, industrial automation, station automation, audio and video networks, stock exchange and specific IoT projects, among others. One key advantage is that IEEE 1588 can be distributed over wired, but also over radio Ethernet: it does not require an additional time distribution network and avoids the need to install dozens or hundreds of GNSS receivers associated with units requiring precision time. At the same time, PTP is more accurate than Network Time Protocol (NTP), as PTP can provide nanosecond accuracy with hardware time-stamping support. It is therefore ideally suited for station-based applications. Table 1 summarises the characteristics of the different synchronisation methods currently available for substations.

Table 1: Comparison of different substation time synchronisation methods

Clock types

Four main device types are defined for the PTP standard, namely Ordinary Clock (OC), Transparent Clock (TC) and Boundary Clock (BC), as well as the aforementioned Master Clock. These clocks work together to distribute highly accurate synchronisation messages throughout the time structure.

The Ordinary Slave Clock (OSC) is a PTP precision time synchronised device. It maintains a time scale in the PTP domain and synchronises to it for further synchronisation of application-dependent internal circuits. For applications, it can offer a time scale in the form of a marker and/or Pulse Per Second (PPS) signal. The inaccuracy of synchronisation that the OSC can offer will depend on the internal circuits and the class of local oscillator. It is possible to use Synchronous Ethernet to improve the synchronisation performance of the local oscillator if such functionality is supported.

Transparent Clock (TC) PTP messages in communication systems are sent through switches and routers with some delay. The role of the TC is to accurately measure the switching delay and add this information to the PTP message.

TC switches can be configured in two ways: TC End-to-End (E2E) or TC Peer-to-Peer (P2P) depending on the delay measurement mechanism used. If the timer operates in E2E mode, the dwell time is added to the correction field of the target PTP message. The P2P TC, on the other hand, also communicates with the upstream device (on the MC side) to which it sends PTP Peer messages in order to obtain information about the delay of the connection, which is added to the correction fields together with the dwell time of the PTP packets. P2P delay calculation mode is required for station applications in the required profiles. It is not possible to operate smart station equipment with E2E mode, which is defined in telecommunications applications e.g. ITU-T G.8275.1 Telecom Profile. When selecting hardware dedicated to station solutions, the PTP capabilities of the device should therefore be carefully checked. Subsequent correction of the delay calculation mechanism by software replacement may prove cumbersome or impossible without replacing the hardware.

Boundary clock (BC) consists of OSC and MC separated by internal time and frequency synchronisation circuits (local oscillator circuits). The whole implemented in the network switch allows the side of the master clocks to be automatically detected and, through the BMCA algorithm, the best source to be selected. On the switch port designated by the BMCA algorithm, the OSC function is automatically set and the rest of the ports automatically configure MC mode. As a result, BCs divide the synchronisation area into different segments or subdomains, each of which may have different configuration settings to some extent. As with TCs, delay calculation can take place in E2E and P2P mode. For station applications, P2P delay calculation mode is required. When configuring BC and station applications, it should be mentioned that support of specific profiles, e.g. such as IEEE C37.238 and IEC 61850-9-3, is required.

Figure 1: Example PTP synchronisation network based on Bitstream solutions

As shown in Figure 1, the time synchronisation network can support any connection topology. A best clock selection mechanism will ensure that the synchronisation route is correctly selected and that the path is switched to the best source. The structure can be multi-level and include heterogeneous transmission links (twisted-pair, optical fibre or WiFi radio). In the absence of direct PTP protocol support in the end devices, they can be synchronised from intermediary devices in the PTP chain equipped with appropriate outputs, e.g. PPS - synchronisation and 10MHz clock - syntonisation. Such a node must play the role of OSC or BC to be able to issue the appropriate signals. An important role in such an extensive network is played by a supervision system or monitoring the quality of synchronisation. Therefore, if time servers and node elements come from one manufacturer, they will be equipped with a consistent configuration and supervision system, which will facilitate the maintenance of such an infrastructure.

Figure 2 shows the delay calculation method for P2P configurations required in power generation, industrial automation or audio video networks. The message exchange shown relates to One Step mode, which will be addressed later in this article.

Figure 2: Example calculation of latency in P2P mode (One Step)

As depicted in Figure 2, a Sync message carrying a Precision Timestamp is distributed through two nodes (switches) operating in TC mode. Each of them, based on the exchange of Pdelay_Req (Peer Delay Request) and Pdelay_Res (Peer Delay Response) messages between the nodes, has the task of determining the Link Delay. TC devices also just determine the propagation delay Residence Delay. Each node includes the calculated delays in the SynC message. At the end of the process, the receiver takes into account the last track delay.

Receiver and network synchronisation IEEE 1588v2

Describing the process of network and device synchronisation in general through the IEEE 1588 v2.1 protocol, it consists of several steps as follows:

1. Establishment of the Master-Slave hierarchy: deciding the role and state of each port of all Boundary Clocks and Ordinary Clocks by using the BMCA mechanism.

2. Synchronisation: the Grandmaster Clock starts to synchronise the slave clocks.

3. delay measurement for network nodes configured in P2P mode or determination of delay between master and slave in E2E mode.

4. synchronisation of Boundary Clocks and Ordinary Slave Clocks nodes.

Synchronisation of OSC and BC nodes will be done in stages. Typically in such devices, automatic control circuits are built in to tune the internal timing. The regulator typically attempts to tune the frequency and in the next stage deals with fine-tuning the phase (timing). Depending on the settings of the controller's inertial members, the phase-tuning time can reach several minutes. Phase tuning will be sensitive to changes in the frequency of the local oscillator hence it is an important part of the OSC or BC device. We can improve PTP timing synchronisation by synchronising the frequency circuits using Synchronous Ethernet (SyncE) specified via ITU-T G.8261, ITU-T G.8262 and ITU-T G.8264, as shown in Figure 3. In this case, frequency synchronisation is done using the Ethernet physical layer and the PTP takes care of OSC phase tuning. The combination of these two methods is inherent in telecommunications networks while, for unknown reasons, less common in automation networks.

Figure 3: Using Synchronous Ethernet to improve the performance of BC devices with PTP.

Best Master Clock Algorithm

BMCA is a decision-making algorithm applied to all nodes that support the clock synchronisation function, i.e. have an OSC. A port on the Ordinary Slave Clock has two possible basic states: i.e. Slave and for BC Slave and Master. There are, of course, many more states that an OSC or BC device port can take. These then define transient or emergency states. Nevertheless, depending on the configuration settings and the decision of the BMCA in the case of the availability of the Master Clock, a final decision is made about the state of the port as a receiving Slave type or as a synchronising Master type. TC devices do not synchronise internal circuits and only transmit PTP messages and process timestamps for selected messages, so they do not use the BMCA algorithm and there are no defined port states as mentioned above.

The Master Clock with the timing defined for the profile sends out information about the clock properties to the network via Announce messages. Changing the Master and initialising the BMCA can be triggered by the absence of Announce messages from the existing Grandmaster for a certain period of time. The process can also be triggered automatically when the active Master degrades (no GNSS signal) or another node with a better clock is connected to the network. BMCA uses the data set from Announce to decide which Master has the best accuracy or the user has given it the highest priority to be selected as Grandmaster. The decision data is listed below, in order of priority:

1. Priority 1: this is a user-defined setting that can be configured from 0 to 255. Lower values have priority. It is designed to be highest in the list to bypass the rest of the BMCA comparisons and speed up execution, as well as to give users freedom in terms of clock settings.

2 ClockClass (the state of synchronisation to the atomic clock): ClockClass value, which indicates the ability to track time or frequency or, in other words, the state of the clock. A value of 6 indicates synchronisation to atomic time. The higher the value, the worse the class. A ClockClass value of255 is used for slave clocks that do not synchronise in the PTP chain of downstream devices.

3. Clock accuracy: this is an enumerated value estimated by the clock based on the time source attribute and the ability of the clock itself to support it.

4. Clock deviation (offsetScaledLogVariance): This is a logarithmic scale statistical value representing the accuracy of the timestamp when it is not synchronised by PTP based on the algorithm specified in PTPv2 (BC).

5. Priority 2: This is another priority setting by the user, similar to Priority 1. If there are two identical Grandmaster clocks this setting can be used to select the preferred Master.

6. Clock Identification (GrandMasterID): the modified value of the clock's MAC address, which is a unique value for each clock on the LAN.
It may happen that messages from the Grand Master Clock take different routes to the Ordinary Clock device. In this case, the BMCA algorithm enters synchronisation path determination mode and analyses a range of other information to decide on the port state and error message. Figure 4 shows a simplified BMCA algorithm.

Figure 4: Method for selecting a better BMCA clock

Two Step and One Step

Two timestamp transmission methods have been defined for each type of PTP node, i.e. One Step and Two Step. The definition of two methods for transmitting precise time stamps resulted from the development of hardware capabilities of network devices in supporting PTP messages. In the first solutions, the function of retrieving real time from internal circuitry in the Master Clock device during the generation of Sync frames carrying the timestamp proved difficult to achieve. It was therefore decided to add time announcement via a Two Step mechanism. When sending a Sync frame carrying a timestamp, the actual time is not appended, but only latched into the device register. Then, with the Follow Up message, the previously stored timestamp is retrieved and sent already in the normal broadcast frame mode. In the course of device development, One Step mode is already supported in most cases and can be considered the preferred and target mode. There is interoperability when devices operate in both modes. Figure 5 shows how messages are exchanged in One Step and Two Step mode. For simplicity, this is presented in E2E delay calculation mode. For P2P mode also, a Follow Up message will be sent after the Sync frame, there will be no other messages, which will be replaced by Peer messages.

Figure 5: Calculation of One Step and Two Step delays in E2E mode

IEEE 1588 Power Profile

In the PTPv2 standard, the introduced profiles allow a number of options when it comes to their configuration. Profiles define certain functions, indicating their specific use.
Firstly, the IEEE Std C37.238-2011 Power Profile was created for power applications (mainly PMU applications), which, thanks to optimised parameters and minimal configuration on the user side, allows synchronisation accuracy of less than 1 µs to be achieved with network topologies typical of substation automation systems. This was followed by a profile referred to as the Utility Profile described in IEC61850-9-3:2016. Immediately afterwards, the Power Profile was simplified and the document IEEE Std C37.238-2017 was created. The profiles specify PTP configurations, how delays are determined and add specific TLV fields. The absence of these fields in PTP messages means that PTPs are not interoperable with station automation equipment.

The Management Information Base (MIB) for the Simple Network Management Protocol (SNMP) for PTP Power Profile is also defined in the profile standardisation documents and allows key device parameters to be monitored using standard network management tools.
The Power Profile defines requirements for Ethernet switches that can introduce an inaccuracy of no more than 50ns in TC mode and 200ns in BC mode. An inaccuracy value of no more than 200ns is assumed for OSC. The MC device should offer an accuracy of better than 200ns.
Profiles used in the power industry require that switches in PTP Peer-to-Peer mode are used to transport all PTP messages on the Ethernet network, and that all messages are transmitted using Ethernet broadcast frames at Layer 2 of the OSI/ISO model. The assumption made about the method of determining delay carries the following advantages:

• The traffic and load that is directed to the master clock on the network does not increase as more OSC devices are added. The Grand Master only communicates with the Ethernet switch to which it is connected.
• The PTP system automatically takes into account the delays calculated for each connection , when the primary network connection fails and an alternative path is used. Path delays are measured on all network links even those blocked to normal traffic e.g. by SpanningTree protocols, ITU-T G8032, etc.
• Ability to track on devices (even TCs) the status of links and the delays involved.

Advantages of using PTP

Given the technology outline presented, the use of PTP to distribute time from accurate and centralised GMC time servers located at substations over distributed local GNSS receiver applications has the following advantages:

1. Data distribution and time synchronisation are carried out on the same Ethernet network.
2. The accuracy of synchronisation is not affected by Ethernet traffic. Only if the network is overloaded or reconfigured, PTP messages may be lost. This situation can be avoided by using Ethernet switches with 10Gbit/s interfaces in the architecture design. This saves budget and a network with precise synchronisation can be used to transmit time-sensitive propagation data from the IEC 61850 process bus such as PMU, GOOSE or SV.
3. The PTP offers the possibility of using redundant Grandmaster clocks with automatic failover.
4. The network can be expanded without unnecessary load on the Grandmaster thanks to the P2P configuration.
5. Propagation delays due to long cable runs are automatically compensated. Calibration of nodes and devices in the field does not have to be done manually.
6. In PTP Power Profile, the message rate ( approximately every 1s) has been optimised to meet the requirements of achieving better than 1 µs accuracy, without also causing excessive traffic in the shared network.
7. There are no configuration issues regarding Universal Time Coordinated (UTC) or local time. A single time reference is used so that all OSC devices on and off station use International Atomic Time (TAI), avoiding, among other things, problems with time change.
8. PTPv2 can be transmitted over both fibre and twisted-pair Ethernet, or even radio networks.
9. PTP transmits a local time offset, so there is no need to configure a local time zone on the security relays.
10. any changes to daylight saving time dates only need to be made to the Grandmaster and not to every device on the network. The mechanism used is defined in IEEE C37.238-2011/2017 as well as the Utility Profile.
11. protocols to enable redundant Ethernet connections such as RSTP, ITU-TG.8032 Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy (HSR) can be used to increase the reliability of network connections between PTP devices.

Krzysztof Nowacki, M.Sc.
Research and Development Director
Bitsream S.A
www.bitstream.pl

Literature:

1. IEEE 1588 v2.1:2019 - IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems
2. IEEE Std C37.238™-2017, IEEE Standard Profile for Use of IEEE 1588™ Precision Time Protocol in Power System Applications
3. IEC TR 61850-90-12 :2020 Communication networks and systems for power utility automation - Part 90-12: Wide area network engineering guidelines
4. IEC TR 61850-90-5:2012 Communication networks and systems for power utility automation - Part 90-5: Use of IEC 61850 to transmit synchrophasor information according to IEEE C37.118
5. Practical aspects of IEC 61850-9-2 implementation in microprocessor-based protection and control IEDs
6. IEC 61850-9-2 LE (Lite Edition). Implementation Guideline for Digital Interface to Instrument Transformers using IEC 61850-9-2.
7. IEC 62439-3 (International Standard). Industrial Communication Networks – High Availability Automation Networks – Part 3: Parallel Redundancy Protocol (PRP) and High-availability Seamless Redundancy (HSR).
8. A. Carta, N. Locci, C. Muscas, F. Pinna, and S. Sulis, "GPS and 1588 synchronization for the measurement of synchrophasors in electric power systems," Computer Standards & Interfaces, vol. 33, no. 2, pp. 176-181, February 2011.
9. K. Behrendt and K. Fodero, "The Perfect Time: An Examination of Time Synchronization Techniques," SEL Inc., TP6226-01,