Runbook: uupl-eng-ws

Entry

Via SSH on port 22 as engineer. The password is in the 2019 network map buried in the backup archive on hex-legacy-1, or in the engineering notes on any lateral pivot that has already visited the operational zone.

ssh engineer@10.10.2.30

Password: spanner99 (from C:\UUPL\NETWORK.TXT on hex-legacy-1, or from backups\PLC_Backup_2019.tar.gz on this workstation once you are in).

wizzards-retreat has an authorised SSH key for this workstation. If you already have a shell there, the key pivot works without a password.

Identity and network

The first question on a dual-homed host is always: what does it touch?

PS C:\Users\engineer> whoami

Returns ot.local\engineer. Standard domain account.

PS C:\Users\engineer> hostname

Returns ENG-WS01.

PS C:\Users\engineer> ipconfig

Two NICs. Ethernet0 at 10.10.3.100 (control zone). Ethernet1 at 10.10.2.30 (operational zone). This box has a direct path to every PLC, relay, and HMI on the control network. That is the find.

PS C:\Users\engineer> route print

Default gateway via 10.10.2.202. The control-zone subnet (10.10.3.0/24) is directly attached on Ethernet0, no routing hop required. The Interface List at the top of the output shows the real MAC addresses for both NICs.

PS C:\Users\engineer> netstat -ano

Two persistent MQTT connections are always present: one from 10.10.3.100 to 10.10.3.60:1883 (subscribing from the control-zone broker) and one from 10.10.2.30 to 10.10.5.12:1883 (publishing to the DMZ broker). These belong to mqtt_bridge.py and stay ESTABLISHED for the lifetime of the workstation. The cron-driven Modbus and HTTP poll connects, reads, and closes in under a second, so those are gone before a manual netstat can catch them.

Credential hunting

PS C:\Users\engineer> cat config\plc-access.conf

The entire OT device inventory: IP, port, protocol, unit ID, and operational notes for every PLC, relay, actuator, HMI, and breaker. Written by Ponder Stibbons in 2001 and updated ever since. The relay web credentials (admin/relay1234) are in the notes column.

PS C:\Users\engineer> cat Documents\engineering_notes.txt

Consolidates credentials across multiple systems: historian DB password (Historian2015), SCADA web login (admin/admin) and SSH (scada_admin / W1nd0ws@2016), historian SSH (hist_admin / Historian2015), and the HMI web at http://10.10.3.10:1881/ which takes no login. The author’s note “it’s fine” appears next to the historian password.

PS C:\Users\engineer> cat Documents\mqtt_topics.txt

Explains every MQTT topic on the control-zone broker, including the payload schema for uupl/turbine/telemetry and the fact that freq_x10 is frequency times ten. Confirms the DMZ bridge republishes all topics to clacks-relay.

PS C:\Users\engineer> cat Documents\telemetry_sample_2024-01-20.log

Twenty-four lines of steady-state turbine telemetry captured from the broker. Useful as a baseline: typical RPM sits in the 2930-2965 range, voltages around 224-229 V, freq_x10 around 499-500.

PS C:\Users\engineer> cat Documents\snmp_plc_2024-03-15.txt

Saved snmpwalk output from the turbine PLC at 10.10.3.21. Confirms firmware version (4.1.2), sysContact (Ponder Stibbons), and that rwcommunity private is active alongside the read-only public community.

PS C:\Users\engineer> cat Documents\grafana_turbine_panel.json

Grafana panel export for the turbine overview dashboard. The __note field carries the historian credentials (historian / Historian2015) inline.

PS C:\Users\engineer> cat Documents\alarm_history_2024-Q1.csv

Alarm events for January through March 2024. Includes an overspeed alarm in January (REL-200a, 3312 RPM), a temperature exceedance in March, and an emergency stop event on 3 February. The ack_by column confirms that pstibbons is the only account acknowledging alarms.

PS C:\Users\engineer> cat Projects\Firmware\README.txt

PLC admin credentials in plain text: admin / turbineadmin. Listed under “Prerequisites”. The firmware update script on the Desktop echoes the same credentials.

PS C:\Users\engineer> cat Desktop\update_plc_firmware.ps1

PS C:\Users\engineer> cat Tools\send_alarm.ps1

The alarm relay script carries the SMTP password (plantmail123) in plain text. The same credential is in C:\SCADA\Config\scada.ini on the SCADA server and in the SCADA /config endpoint.

PS C:\Users\engineer> cat Tools\poll_and_ingest.py

The historian ingest credentials (hist_read / history2017) are hardcoded here. This is the account used to POST readings; it also works against the /report endpoint for read access.

Backup archives

PS C:\Users\engineer> dir backups\

Two files. PLC_Backup_2019.tar.gz is the older one: inside is plc-access-2019.conf with the 2019 pre-audit credential set, and network_map_2019.txt, the most complete device map in the lab. The map names every operational and control zone host with its IP, username, and password.

PS C:\Users\engineer> Expand-Archive .\backup_2022_final_v3.zip -DestinationPath .\backup_2022_expanded
PS C:\Users\engineer> dir .\backup_2022_expanded

backup_2022_final_v3.zip is the 2022 annual service snapshot. Contains two files: plc-access-2022.conf (all OT credentials as at 2022, including historian / Historian2015 and admin / admin for the SCADA web) and setpoints_2022.txt, which documents all writable Modbus holding registers on the PLC and relay IEDs with their values and a note that no authentication protects any of them.

PLC project files

PS C:\Users\engineer> cat Projects\PLC\turbine_controller.project

The exported PLC project file. Contains the full register map (coil 0 is the emergency stop), the admin password (turbineadmin), and the firmware version. Also documents that Modbus TCP has no authentication: “The network IS the access control.”

PS C:\Users\engineer> cat Projects\RelayConfigs\relay_a_2019.txt

Relay A protection thresholds in the Modbus holding register map. HR[0] is undervoltage threshold, HR[1] is overspeed, HR[2] is overcurrent. All are writable via Modbus with no authentication. Reducing HR[1] allows overspeed conditions to persist without a trip.

PS C:\Users\engineer> cat Projects\RelayConfigs\trip_history_2024.txt

Trip event log for both relay IEDs through 2024. Notable: REL-200b had an overspeed trip in October 2024 (RPM 3347, threshold 3300). Cross-reference with the threshold override record to check whether the overspeed calibration ticket ever got resolved.

PS C:\Users\engineer> cat Projects\RelayConfigs\threshold_override_2023-09.txt

Record of a temporary HR[2] increase on REL-200b during a 2023 load test. The note at the bottom mentions chasing the sorting-office gateway about a password rotation before the annual audit. That is the DMZ Neuron gateway at 10.10.5.11, which still runs on default credentials (admin / uupl2015).

PS C:\Users\engineer> cat Projects\RelayConfigs\relay_maintenance_log.txt

Full commissioning and service history, 2019 to 2024. The 2023 entry confirms the OC threshold was restored, but the overspeed recalibration check remains outstanding.

PSReadLine history

PS C:\Users\engineer> cat AppData\Roaming\Microsoft\Windows\PowerShell\PSReadLine\ConsoleHost_history.txt

Shows recent sessions: Modbus reads against the turbine PLC, historian and SCADA queries via curl, an SSH session to the SCADA server, and an nmap of 10.10.3.0/24. The command history confirms which hosts were recently active and what the engineer was doing.

Modbus access

PS C:\Users\engineer> python Tools\modbus_read.py 10.10.3.21 502 input 0 5

Live turbine input registers: RPM, temperature, pressure, voltage, current.

PS C:\Users\engineer> python Tools\modbus_read.py 10.10.3.21 502 holding 0 4

Governor setpoint, fuel valve command, cooling pump speed, overcurrent threshold.

PS C:\Users\engineer> python Tools\modbus_read.py 10.10.3.31 502 holding 0 3

Relay A protection thresholds. Write HR[1]=0 to zero the overspeed threshold and prevent the relay from acting on an overspeed condition.

PS C:\Users\engineer> python Tools\modbus_write.py 10.10.3.21 502 coil 0 1

Emergency stop. Writes coil 0 high. The runbook note says: “DO NOT write coil 0 without coordination with the duty engineer.” There is no other access control.

MQTT telemetry tools

The workstation carries three Python scripts in Tools\ related to process telemetry.

PS C:\Users\engineer> python Tools\mqtt_check.py

Subscribes to uupl/turbine/telemetry on the internal broker at 10.10.3.60. Prints a live JSON stream: rpm, temp_c, pressure, voltage_a, voltage_b, current_a, current_b, freq_x10, power_kw, and estop as the PLC publishes them every five seconds. No credentials required; the broker allows anonymous connections.

PS C:\Users\engineer> python Tools\mqtt_bridge.py

Bridges uupl/turbine/telemetry and uupl/relay/# from the internal broker (10.10.3.60) to clacks-relay in the DMZ (10.10.5.12:1883), republishing verbatim under the same topic names. The script starts automatically at workstation startup and runs as a background daemon. Its output goes to mqtt_bridge.log in the profile directory. Killing or replacing this process affects the only live feed of control-zone telemetry into the DMZ.

PS C:\Users\engineer> python Tools\rtu_updater.py

Polls relay IEDs (10.10.3.31, 10.10.3.32) and the turbine PLC (10.10.3.21) every ten seconds and POSTs live values to the substation RTU management API at 10.10.5.14:8080. Updates all six datapoints: feeder voltages, load current, frequency, and breaker states. Starts at workstation startup alongside the MQTT bridge. Its output goes to rtu_updater.log. Values injected into the RTU via the REST API get overwritten within one polling cycle.

Lateral movement

The SSH known_hosts file lists every host the workstation has connected to.

PS C:\Users\engineer> cat .ssh\known_hosts

Three entries: the turbine PLC at 10.10.3.21, the process historian at 10.10.2.10, and the SCADA server at 10.10.2.20. A comment notes that the relay IEDs were never added (ticket still open). SSH reaches both operational-network and control-network hosts directly from here:

PS C:\Users\engineer> ssh hist_admin@10.10.2.10

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

*******************************************************************************
*                                                                             *
*   Unseen University Power & Light Co.                                       *
*   HIST-SRV01, Process Historian Server (Windows Server 2019)              *
*                                                                             *
*   This system stores all plant time-series data since 1997.                *
*   Authorised personnel only. Contact: Ponder Stibbons (ext 201).           *
*                                                                             *
*******************************************************************************

PS C:\Users\hist_admin> exit

HIST-SRV01 answers. Windows Server 2019 banner, store open since 1997.

PS C:\Users\engineer> ssh scada_admin@10.10.2.20

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

*******************************************************************************
*                                                                             *
*   Unseen University Power & Light Co.                                       *
*   SCADA-SRV01, Distribution SCADA Server (Windows Server 2016)            *
*                                                                             *
*   Authorised UU P&L personnel only. Usage is monitored and logged.         *
*   Contact: Ponder Stibbons (ext 201) for access requests.                  *
*                                                                             *
*******************************************************************************

PS C:\Users\scada_admin> exit

SCADA-SRV01 answers. Windows Server 2016. The SSH key in .ssh\id_rsa may also work on control-network hosts that were provisioned at commissioning.

Poll log

PS C:\Users\engineer> cat plc_poll.log

poll_and_ingest: rpm=2997 temp=144 press=84 volt_a=229 curr_a=75 [ok, 9/9 ingested]
poll_and_ingest: rpm=2934 temp=178 press=83 volt_a=226 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: rpm=2946 temp=175 press=84 volt_a=227 curr_a=74 [ok, 9/9 ingested]
poll_and_ingest: rpm=2947 temp=171 press=84 volt_a=227 curr_a=74 [ok, 9/9 ingested]
poll_and_ingest: rpm=2949 temp=176 press=84 volt_a=226 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: skipped cycle
poll_and_ingest: rpm=2950 temp=177 press=83 volt_a=223 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: rpm=2939 temp=174 press=84 volt_a=225 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: rpm=2940 temp=174 press=84 volt_a=225 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: rpm=2958 temp=176 press=83 volt_a=228 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: skipped cycle
poll_and_ingest: rpm=2925 temp=174 press=83 volt_a=222 curr_a=72 [ok, 9/9 ingested]
[snip]
poll_and_ingest: rpm=2939 temp=173 press=84 volt_a=225 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: rpm=2971 temp=174 press=84 volt_a=230 curr_a=74 [ok, 9/9 ingested]
poll_and_ingest: skipped cycle
poll_and_ingest: rpm=2950 temp=181 press=84 volt_a=226 curr_a=73 [ok, 9/9 ingested]
poll_and_ingest: rpm=0 temp=20 press=0 volt_a=0 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=1 temp=19 press=0 volt_a=0 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=16 temp=22 press=0 volt_a=0 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: skipped cycle
poll_and_ingest: rpm=7 temp=19 press=0 volt_a=1 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=0 temp=18 press=0 volt_a=0 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=19 temp=18 press=0 volt_a=0 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=23 temp=18 press=1 volt_a=0 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=6 temp=19 press=0 volt_a=1 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=0 temp=18 press=0 volt_a=0 curr_a=0 [ok, 9/9 ingested]
poll_and_ingest: rpm=17 temp=21 press=0 volt_a=0 curr_a=0 [ok, 9/9 ingested]

Each line is one poll cycle: RPM, temperature, pressure, voltage, and current, plus the ingest count. skipped cycle entries appear roughly one in twenty as jitter. A long run of steady readings followed by rpm=0 is the emergency stop written earlier in this session.

PS C:\Users\engineer> schtasks /query

TaskName                         Schedule 
--------                         -------- 
PLC-Poll                         Per Minute

Lists the scheduled task registered to the engineer account. PLC-Poll appears here; it is the cron entry driving poll_and_ingest.py.

What you can know now

Access:

  • Shell on ENG-WS01 as engineer at 10.10.2.30

  • SSH to HIST-SRV01 as hist_admin at 10.10.2.10

  • SSH to SCADA-SRV01 as scada_admin at 10.10.2.20

Network:

  • 10.10.3.0/24 (control network) is directly attached on Ethernet0 (10.10.3.100), no gateway hop

  • PLC at 10.10.3.21, relays A/B at 10.10.3.31/32, HMI at 10.10.3.10, actuators at 10.10.3.51-54, MQTT broker at 10.10.3.60

  • Default gateway 10.10.2.202

Credentials:

  • engineer / spanner99 (ENG-WS01 SSH)

  • hist_read / history2017 (historian ingest and report API)

  • hist_admin / Historian2015 (historian SSH)

  • historian / Historian2015 (historian DB)

  • admin / admin (SCADA web)

  • scada_admin / W1nd0ws@2016 (SCADA SSH)

  • admin / turbineadmin (PLC admin)

  • admin / relay1234 (relay web interfaces, both feeders)

  • alarms@uupl.am / plantmail123 (SMTP alarm relay)

  • HMI at http://10.10.3.10:1881/, no login required

OT reach:

  • Modbus TCP write to any device on 10.10.3.0/24, no authentication

  • PLC emergency stop: write coil 0 = 1 on 10.10.3.21

  • Relay trip thresholds writable: HR[0-2] on 10.10.3.31 and 10.10.3.32

  • MQTT broker at 10.10.3.60 allows anonymous subscribe and publish

  • Live telemetry bridgeable to the DMZ broker at 10.10.5.12 via mqtt_bridge.py