Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
20822d000e | ||
|
|
7dba1750ee | ||
|
|
abe88df3a4 | ||
|
|
aa7341af06 | ||
|
|
390748173c | ||
|
|
64eba37f81 | ||
|
|
0631de74c7 | ||
|
|
0921e82c54 | ||
|
|
7b80633088 | ||
|
|
379e3be365 | ||
|
|
6b03fe7b44 | ||
|
|
3834546f38 |
No files matched your search
@@ -19,11 +19,13 @@ A Proxmox/Ceph cluster lives or dies by its network, and network problems are sn
|
|||||||
|
|
||||||
## Gallery
|
## Gallery
|
||||||
|
|
||||||
**Dashboard**: alert queue with linked tickets, a live topology, per-host interface status and the UniFi device inventory.
|
**Dashboard**: alert queue with linked tickets, per-host interface status and the UniFi device inventory.
|
||||||
|
|
||||||

|

|
||||||
|
|
||||||

|
**Topology**: every device drawn from its real faceplate or chassis, with each cable leaving the port it is plugged into, styled by medium and speed, with live traffic.
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||

|

|
||||||
|
|
||||||
@@ -92,7 +94,7 @@ Two processes share a MariaDB database:
|
|||||||
|---|---|
|
|---|---|
|
||||||
| large1 | 10.10.10.2:9100 |
|
| large1 | 10.10.10.2:9100 |
|
||||||
| compute-storage-01 | 10.10.10.4:9100 |
|
| compute-storage-01 | 10.10.10.4:9100 |
|
||||||
| micro1 | 10.10.10.8:9100 |
|
| monitor-01 | 10.10.10.8:9100 |
|
||||||
| monitor-02 | 10.10.10.9:9100 |
|
| monitor-02 | 10.10.10.9:9100 |
|
||||||
| compute-storage-gpu-01 | 10.10.10.10:9100 |
|
| compute-storage-gpu-01 | 10.10.10.10:9100 |
|
||||||
| storage-01 | 10.10.10.11:9100 |
|
| storage-01 | 10.10.10.11:9100 |
|
||||||
@@ -105,12 +107,14 @@ Ping-only (no node_exporter): **pbs** (10.10.10.3)
|
|||||||
|
|
||||||
### Dashboard (`/`)
|
### Dashboard (`/`)
|
||||||
- Device search (see Device pages below)
|
- Device search (see Device pages below)
|
||||||
- Active alerts first, then the WAN / Fiber panel, hosts, UniFi devices, topology, and recently resolved alerts
|
- Active alerts first, then the WAN / Fiber panel, hosts, UniFi devices, and recently resolved alerts
|
||||||
- Real-time host status grid with per-NIC link state (UP / DOWN / degraded)
|
- Real-time host status grid with per-NIC link state (UP / DOWN / degraded)
|
||||||
- Network topology diagram (Internet → Modem → Gateway → Switches → Hosts)
|
|
||||||
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
|
- WAN / Fiber panel: WAN and PON state, Rx/Tx power, optics temperature, throughput, errors, gateway uptime and last restart, 7-day Rx trend, recent WAN outages
|
||||||
- UniFi device table (switches, APs, gateway)
|
- UniFi device table (switches, APs, gateway)
|
||||||
- Active alerts table with severity, target, consecutive failures, ticket link
|
- Active alerts table with severity, target, consecutive failures, ticket link
|
||||||
|
- Follow-up flags (see Follow-up below): a ⏳ badge on alerts active for days, a "ticket closed" badge when the
|
||||||
|
ticket was closed but the alert is still active, and a Needs Follow-up panel of tickets left open after their
|
||||||
|
alert cleared. A "TO FOLLOW UP" chip in the status bar counts them all.
|
||||||
- Quick-suppress modal: apply timed or manual suppression from any alert row
|
- Quick-suppress modal: apply timed or manual suppression from any alert row
|
||||||
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
|
- Auto-refreshes every 30 seconds via `/api/status` + `/api/network`
|
||||||
|
|
||||||
@@ -177,6 +181,47 @@ The rack front, drawn from the rack units set on cutsheet items, top first.
|
|||||||
claim the same top unit, the one recorded first is drawn. Items without a rack unit are listed below.
|
claim the same top unit, the one recorded first is drawn. Items without a rack unit are listed below.
|
||||||
- The rack keeps its own dark "hardware" palette in both themes.
|
- The rack keeps its own dark "hardware" palette in both themes.
|
||||||
|
|
||||||
|
### Topology (`/topology`)
|
||||||
|
The whole network as the switches see it now, top down: the internet, the modem, the gateway, then
|
||||||
|
each switch by how many hops it is from the gateway, the gear and unknown devices below them, and the
|
||||||
|
servers along the bottom. Power gear (PDU, UPS, power strip) sits in a rail on the right.
|
||||||
|
- **What needs a look** is listed above the diagram: devices down, degraded or with active alerts,
|
||||||
|
cables down or last seen elsewhere, ports named after the wrong device, and cables with errors.
|
||||||
|
Click one to jump to it. With nothing to report it says all clear. Problem cables are drawn in full
|
||||||
|
colour with a glow and always shown, whatever the filters; healthy cables are drawn quietly.
|
||||||
|
- **A card per device**, all the same height so the whole network reads at a glance: name, state
|
||||||
|
(coloured edge and dot), and a picture: UniFi gear shows its real faceplate (the same drawings as the
|
||||||
|
port inspector and rack page), cropped to the ports; servers show the **Front drawing** of their
|
||||||
|
cutsheet item and a socket per physical NIC, so every cable lands on the NIC it is plugged into
|
||||||
|
(NICs marked unused are dimmed). The internet, modem and gateway sit side by side along the top.
|
||||||
|
- **Cables run like cable management**: each leaves the card edge right below its port (a short tail
|
||||||
|
shows the port), drops into a tray above the row it goes to, runs along its own lane (10G on the
|
||||||
|
upper lanes), and drops straight into its socket, so cables never lie on top of each other and only cross at right angles. Devices are placed clear of
|
||||||
|
the cables passing down through their row, and each sits under its fastest cable.
|
||||||
|
- **One line per physical cable**, leaving the exact port: a switch-to-switch cable is reported by both
|
||||||
|
switches and drawn once. A server with two links (10G and 1G) has two cables. Colour is the cable
|
||||||
|
type (DAC, multimode fibre, the provider's fibre, an SFP+ module that doesn't say which, copper) and
|
||||||
|
thickness is the speed. A cable that is down is dashed red.
|
||||||
|
- **Last seen**: a device with no live cable is drawn on the port it was last seen on, dotted red.
|
||||||
|
- **Unknown devices**: something plugged into a switch port that Gandalf can't name is a grey stub,
|
||||||
|
labelled with the port name (or its address), linking to that port. By default each switch's unknown
|
||||||
|
devices are one block with one bundled cable ("10 unknown devices"); click it to show them one by
|
||||||
|
one, or pick "one by one" or "hidden" in the toolbar.
|
||||||
|
- **Port names**: a ⚠ marks a port named after one device with a different one plugged in (a port
|
||||||
|
called "Large1 Management" with the AP on it). Generic names ("Port 4", "SFP+ 2", "Uplink") and
|
||||||
|
"Downlink to X" with X on the other end never flag.
|
||||||
|
- **Alerts**: a device with active alerts (not suppressed) gets a ring and a count in its worst
|
||||||
|
severity colour, and the NIC it is about is marked.
|
||||||
|
- Click a device for its page, or a port on a faceplate for that port. Click a cable for its details:
|
||||||
|
both ends, speed, cable type and module, traffic each way and how busy it is, errors and drops, and
|
||||||
|
its cutsheet record. Hovering a device traces its path to the internet.
|
||||||
|
- Drag to pan, scroll or pinch to zoom, **Fit** (or `0`) to see everything, `+` and `-` to zoom. Show
|
||||||
|
unknown devices grouped, one by one or not at all. **Server 1G links** (off by default) adds each
|
||||||
|
server's slower second cable, its management link; **Power gear** (off by default) adds the PDU, UPS
|
||||||
|
and power strip. Traffic labels and dimming everything without an alert can be switched too. The
|
||||||
|
filters are remembered per browser. Refreshes every 30 seconds without moving the view.
|
||||||
|
- API: `GET /api/topology` returns `{nodes, links, warnings, updated}` (see `topology.py`).
|
||||||
|
|
||||||
### Health check (`/health`)
|
### Health check (`/health`)
|
||||||
No login needed. Returns 200 when the database answers and the monitor has completed a check recently,
|
No login needed. Returns 200 when the database answers and the monitor has completed a check recently,
|
||||||
otherwise 503 with the failing check: `{"status": "degraded", "checks": {"db": "ok", "monitor": "stale",
|
otherwise 503 with the failing check: `{"status": "degraded", "checks": {"db": "ok", "monitor": "stale",
|
||||||
@@ -309,6 +354,12 @@ can view it; only the `admin` group can edit, and every change is logged with wh
|
|||||||
- Suppression history (last 50)
|
- Suppression history (last 50)
|
||||||
- Available targets reference grid (all known hosts + interfaces)
|
- Available targets reference grid (all known hosts + interfaces)
|
||||||
|
|
||||||
|
**Maintenance**: admins get a **Maintenance** button in the header. It starts a global suppression for
|
||||||
|
30 min, 1, 2 or 4 hours with a reason (by default "Maintenance"). While it runs, every page shows a banner
|
||||||
|
saying when tickets resume, with **End now** for admins. Gandalf keeps checking and recording alerts, and
|
||||||
|
anything still failing when maintenance ends gets a ticket as usual. Use it before a UniFi update, a switch
|
||||||
|
swap or recabling.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Alert Logic
|
## Alert Logic
|
||||||
@@ -352,6 +403,18 @@ whatever Gandalf knows around the problem:
|
|||||||
|
|
||||||
Titles keep the `[target][auto][production]...` tag format, which Tinker uses to spot duplicates.
|
Titles keep the `[target][auto][production]...` tag format, which Tinker uses to spot duplicates.
|
||||||
|
|
||||||
|
### Unused interfaces
|
||||||
|
|
||||||
|
An admin can mark a host interface as unused on its device page (**Mark unused** on a down interface,
|
||||||
|
with an optional reason; **In use again** to undo). An unused interface:
|
||||||
|
|
||||||
|
- never raises a link-down alert, and can't count toward the cluster-wide alert;
|
||||||
|
- doesn't count toward the host's status, so a host whose only down ports are unused shows as up;
|
||||||
|
- shows as "unused" (greyed, dashed dot) on the dashboard, the device page and the rack drawings;
|
||||||
|
- has any open link-down alert closed quietly when it is marked (no recovery note: nothing recovered).
|
||||||
|
|
||||||
|
Marks are stored in `unused_interfaces` (host, interface, reason, who, when).
|
||||||
|
|
||||||
### Recovery notes
|
### Recovery notes
|
||||||
|
|
||||||
When an alert that opened a ticket clears, the monitor adds a comment to that ticket: when it cleared, how
|
When an alert that opened a ticket clears, the monitor adds a comment to that ticket: when it cleared, how
|
||||||
@@ -363,6 +426,20 @@ A note that fails to post is retried every cycle for 24 hours. Posting needs a T
|
|||||||
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
|
`read_write` scope; if Tinker answers 401 or 403, the monitor logs one error and stops trying until it is
|
||||||
restarted.
|
restarted.
|
||||||
|
|
||||||
|
### Follow-up
|
||||||
|
|
||||||
|
Gandalf only comments on tickets, so a ticket can stay open long after its problem is fixed, and an alert can
|
||||||
|
sit active for days once everyone has stopped looking. The dashboard flags both:
|
||||||
|
|
||||||
|
- **Stale alert**: active for `monitor.stale_alert_days` (default 3) or longer. Shown as a ⏳ badge with its age.
|
||||||
|
- **Ticket closed**: the alert is still active but its ticket was closed in Tinker, so nobody is tracking it.
|
||||||
|
- **Ticket left open**: every alert on a ticket cleared at least `monitor.stale_ticket_hours` (default 24) ago,
|
||||||
|
but the ticket is still Open, Pending or In Progress. Listed in the Needs Follow-up panel.
|
||||||
|
|
||||||
|
The monitor reads the list of open tickets from Tinker's `api/tickets_api.php` every 15 minutes (one request,
|
||||||
|
read scope only) and saves it. If that list is over 2 hours old, because Tinker or the key stopped working,
|
||||||
|
the ticket flags are left off rather than guessed. Stale alerts are still flagged.
|
||||||
|
|
||||||
### WAN modem notes
|
### WAN modem notes
|
||||||
|
|
||||||
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
|
The alarm and warning limits are read from the gateway on every poll, so they are never hard-coded here.
|
||||||
@@ -374,7 +451,23 @@ Outages can be over between polls: two real ones lasted about two minutes agains
|
|||||||
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
|
So WAN outages and modem restarts also come from the UniFi system log, and a drop in the modem's uptime
|
||||||
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
|
records a restart even if nothing logged it. A single short outage only appears in the panel's history.
|
||||||
|
|
||||||
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`.
|
**One ticket per incident.** The modem and WAN reports from one check cycle open a single ticket, led by
|
||||||
|
the most urgent (a P2 "WAN modem unreachable" leads, and an RPDU power-cycle seen in the same cycle is listed
|
||||||
|
under "Also seen in this incident"). A report that comes while the modem has an open alert, or one that
|
||||||
|
ended under 30 minutes ago, becomes a comment on that alert's ticket instead of a new ticket. If the
|
||||||
|
comment can't be posted, the report gets its own ticket as before.
|
||||||
|
|
||||||
|
**UniFi updates.** UniFi logs `NETWORK_UPDATED` (Network app) and `FIRMWARE_UPDATED_V2` (device firmware)
|
||||||
|
once an update has finished. Updating the Network app can drop the UDM's WAN, and Smart Power may then
|
||||||
|
power-cycle a modem that was fine. When Gandalf reads one of these entries it:
|
||||||
|
|
||||||
|
- comments on modem and WAN tickets first seen from 20 minutes before the update to 15 minutes after,
|
||||||
|
naming the update (for example "UniFi Network 11.0.97 to 11.0.99") as the likely cause;
|
||||||
|
- opens no new modem or WAN tickets for 15 minutes after the update. A problem still there after that gets
|
||||||
|
its ticket as usual; a power-cycle report during the hold only goes into the event history.
|
||||||
|
|
||||||
|
To silence modem alerts during maintenance, create a `host` suppression with the modem's `name`, or use
|
||||||
|
**Maintenance** (below) to pause every ticket.
|
||||||
|
|
||||||
### Baseline Tracking
|
### Baseline Tracking
|
||||||
|
|
||||||
@@ -437,7 +530,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
|||||||
"hosts": [
|
"hosts": [
|
||||||
{ "name": "large1", "prometheus_instance": "10.10.10.2:9100" },
|
{ "name": "large1", "prometheus_instance": "10.10.10.2:9100" },
|
||||||
{ "name": "compute-storage-01", "prometheus_instance": "10.10.10.4:9100" },
|
{ "name": "compute-storage-01", "prometheus_instance": "10.10.10.4:9100" },
|
||||||
{ "name": "micro1", "prometheus_instance": "10.10.10.8:9100" },
|
{ "name": "monitor-01", "prometheus_instance": "10.10.10.8:9100" },
|
||||||
{ "name": "monitor-02", "prometheus_instance": "10.10.10.9:9100" },
|
{ "name": "monitor-02", "prometheus_instance": "10.10.10.9:9100" },
|
||||||
{ "name": "compute-storage-gpu-01", "prometheus_instance": "10.10.10.10:9100" },
|
{ "name": "compute-storage-gpu-01", "prometheus_instance": "10.10.10.10:9100" },
|
||||||
{ "name": "storage-01", "prometheus_instance": "10.10.10.11:9100" }
|
{ "name": "storage-01", "prometheus_instance": "10.10.10.11:9100" }
|
||||||
@@ -474,6 +567,7 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
|||||||
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
|
| `ticket_api.url` | Tinker Tickets `create_ticket_api.php` URL |
|
||||||
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
|
| `ticket_api.api_key` | Tinker Tickets bearer token. Needs `read_write` scope to add recovery notes |
|
||||||
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
|
| `ticket_api.comment_url` | Tinker comment endpoint (default: `api/ticket_comment_api.php` beside `ticket_api.url`) |
|
||||||
|
| `ticket_api.read_url` | Tinker ticket read endpoint, used to find tickets left open (default: `api/tickets_api.php` beside `ticket_api.url`) |
|
||||||
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
|
| `gandalf.public_url` | Gandalf's own address, for links in tickets (default: `https://gandalf.lotusguild.org`) |
|
||||||
| `rack.units`, `rack.name` | The single rack's height and name on `/rack` (defaults: 42, "Rack 1") |
|
| `rack.units`, `rack.name` | The single rack's height and name on `/rack` (defaults: 42, "Rack 1") |
|
||||||
| `racks` | Several racks: `[{"name": "...", "units": 42}, ...]`, in tab order; overrides `rack` |
|
| `racks` | Several racks: `[{"name": "...", "units": 42}, ...]`, in tab order; overrides `rack` |
|
||||||
@@ -491,6 +585,8 @@ Shared by both processes. Located in the working directory (`/var/www/html/prod/
|
|||||||
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold`: UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
|
| `monitor.unifi_failure_threshold` | Consecutive failures before ticketing a UniFi device as offline (default: same as `failure_threshold`). Set higher than `failure_threshold`: UniFi switches/APs can take several minutes to rejoin after a firmware-update reboot, so a low threshold cuts tickets for momentary maintenance reboots instead of real failures. |
|
||||||
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
|
| `monitor.cluster_threshold` | Hosts with failures to trigger cluster-wide P1 (default: 3) |
|
||||||
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
|
| `monitor.ping_hosts` | Hosts checked only by ping (no node_exporter) |
|
||||||
|
| `monitor.stale_alert_days` | Days an alert can stay active before the dashboard flags it as stale (default: 3) |
|
||||||
|
| `monitor.stale_ticket_hours` | Hours a ticket can stay open after all its alerts cleared before it is listed under Needs Follow-up (default: 24) |
|
||||||
| `cutsheet.mismatch_ticket_hours` | Hours a cutsheet mismatch must last before it opens a P4 ticket (default: 24; `0` never tickets). The mismatch shows on the dashboard and the switch's page straight away, so recabling never pages anyone. Suppressing the switch or its port also stops the ticket. |
|
| `cutsheet.mismatch_ticket_hours` | Hours a cutsheet mismatch must last before it opens a P4 ticket (default: 24; `0` never tickets). The mismatch shows on the dashboard and the switch's page straight away, so recabling never pages anyone. Suppressing the switch or its port also stops the ticket. |
|
||||||
| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. |
|
| `modem.url` | AT&T gateway base URL. Omit the `modem` block to disable modem monitoring. Must be reachable from the monitor host. |
|
||||||
| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) |
|
| `modem.name` | Name used for the modem's alerts, tickets, and suppressions (default: `att-modem`) |
|
||||||
@@ -684,6 +780,7 @@ gandalf/
|
|||||||
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
|
├── monitor.py # Background daemon (Prometheus, UniFi, Pulse, alert logic)
|
||||||
├── modem.py # Read-only AT&T gateway status page client and parser
|
├── modem.py # Read-only AT&T gateway status page client and parser
|
||||||
├── devices.py # Device inventory for search and device pages
|
├── devices.py # Device inventory for search and device pages
|
||||||
|
├── topology.py # The topology graph: devices, cables, depth, last-seen links
|
||||||
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
|
├── cutsheet.py # Cutsheet validation, path tracing, plan-vs-actual checks, CSV
|
||||||
├── ticket_text.py # Markdown for tickets and recovery notes
|
├── ticket_text.py # Markdown for tickets and recovery notes
|
||||||
├── db.py # Database operations (MariaDB via pymysql, thread-local conn reuse)
|
├── db.py # Database operations (MariaDB via pymysql, thread-local conn reuse)
|
||||||
@@ -697,6 +794,8 @@ gandalf/
|
|||||||
│ ├── cutsheet.js # Cutsheet page
|
│ ├── cutsheet.js # Cutsheet page
|
||||||
│ ├── events.js # Alert history page
|
│ ├── events.js # Alert history page
|
||||||
│ ├── rack.js # Rack drawing
|
│ ├── rack.js # Rack drawing
|
||||||
|
│ ├── topology.js # Topology page: layout, cables, pan and zoom
|
||||||
|
│ ├── faceplates.js # UniFi faceplates drawn from each model's real layout
|
||||||
│ ├── fronts.js # Drawn fronts for gear without a live faceplate (NVR, server cases)
|
│ ├── fronts.js # Drawn fronts for gear without a live faceplate (NVR, server cases)
|
||||||
│ └── port-inspector.js # Switch front panel, port panel, path check, diagnostics
|
│ └── port-inspector.js # Switch front panel, port panel, path check, diagnostics
|
||||||
└── templates/
|
└── templates/
|
||||||
@@ -706,6 +805,7 @@ gandalf/
|
|||||||
├── cutsheet.html # Cutsheet page and its editors
|
├── cutsheet.html # Cutsheet page and its editors
|
||||||
├── events.html # Alert history page
|
├── events.html # Alert history page
|
||||||
├── rack.html # Rack drawing
|
├── rack.html # Rack drawing
|
||||||
|
├── topology.html # Topology page
|
||||||
├── _modem_panel.html # WAN / Fiber panel, shared by the dashboard and modem page
|
├── _modem_panel.html # WAN / Fiber panel, shared by the dashboard and modem page
|
||||||
└── suppressions.html # Suppression management page
|
└── suppressions.html # Suppression management page
|
||||||
```
|
```
|
||||||
@@ -753,6 +853,7 @@ Take LED colors from the model's quick start guide. See the comment at the top o
|
|||||||
|
|
||||||
- `cutsheet_items` gained `rack`, `shelf`, `device`, `mount`, `chassis`, `facing` and `slot` columns; they are added automatically on first use
|
- `cutsheet_items` gained `rack`, `shelf`, `device`, `mount`, `chassis`, `facing` and `slot` columns; they are added automatically on first use
|
||||||
- `cutsheet_racks`: racks saved on the rack page (name, height, notes)
|
- `cutsheet_racks`: racks saved on the rack page (name, height, notes)
|
||||||
|
- `unused_interfaces`: host interfaces marked unused (no alerts, not counted in host status)
|
||||||
- `device_links`: each device's inventory URL, by device name; created with the cutsheet tables
|
- `device_links`: each device's inventory URL, by device name; created with the cutsheet tables
|
||||||
- `network_events`: one row per active event; `resolved_at` is set when recovered, and
|
- `network_events`: one row per active event; `resolved_at` is set when recovered, and
|
||||||
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
|
`recovery_notified` once the recovery note is on its ticket (column added automatically at monitor startup)
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ import cutsheet
|
|||||||
import db
|
import db
|
||||||
import devices
|
import devices
|
||||||
import diagnose
|
import diagnose
|
||||||
|
import topology
|
||||||
from monitor import PulseClient, UnifiClient
|
from monitor import PulseClient, UnifiClient
|
||||||
|
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
@@ -60,6 +61,19 @@ def inject_config():
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@app.context_processor
|
||||||
|
def inject_maintenance():
|
||||||
|
"""The active global suppression (maintenance), shown as a banner on every page."""
|
||||||
|
try:
|
||||||
|
active = [s for s in db.get_active_suppressions() if s.get('target_type') == 'all']
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Reading maintenance state failed: {e}')
|
||||||
|
active = []
|
||||||
|
# Of several, the one that ends last decides when tickets resume; no expiry beats any
|
||||||
|
active.sort(key=lambda s: (s.get('expires_at') is None, s.get('expires_at') or ''))
|
||||||
|
return {'maintenance': active[-1] if active else None}
|
||||||
|
|
||||||
|
|
||||||
# In-memory diagnostic job store { job_id: { status, result, created_at } }
|
# In-memory diagnostic job store { job_id: { status, result, created_at } }
|
||||||
_diag_jobs: dict = {}
|
_diag_jobs: dict = {}
|
||||||
_diag_lock = threading.Lock()
|
_diag_lock = threading.Lock()
|
||||||
@@ -261,6 +275,72 @@ def _annotate_suppressions(events: list, suppressions: list) -> None:
|
|||||||
ev['is_suppressed'] = db.check_suppressed(suppressions, 'host', name, detail)
|
ev['is_suppressed'] = db.check_suppressed(suppressions, 'host', name, detail)
|
||||||
|
|
||||||
|
|
||||||
|
# The monitor refreshes the open ticket list every 15 minutes; older than this and
|
||||||
|
# it has stopped working, so nothing is flagged rather than flagging on stale data.
|
||||||
|
_OPEN_TICKETS_MAX_AGE_S = 2 * 3600
|
||||||
|
|
||||||
|
|
||||||
|
def _stale_settings() -> tuple:
|
||||||
|
"""(days an alert can stay active before it counts as stale,
|
||||||
|
hours a ticket can stay open after its alert cleared)."""
|
||||||
|
mon = _config().get('monitor', {})
|
||||||
|
try:
|
||||||
|
days = float(mon.get('stale_alert_days', 3))
|
||||||
|
hours = int(mon.get('stale_ticket_hours', 24))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
days, hours = 3.0, 24
|
||||||
|
return days, hours
|
||||||
|
|
||||||
|
|
||||||
|
def _open_tickets() -> Optional[dict]:
|
||||||
|
"""{ticket_id: status} of open Tinker tickets, or None when the list is missing or out of date."""
|
||||||
|
try:
|
||||||
|
raw = db.get_state('open_tickets')
|
||||||
|
data = json.loads(raw) if raw else None
|
||||||
|
if not data or time.time() - float(data.get('checked_at', 0)) > _OPEN_TICKETS_MAX_AGE_S:
|
||||||
|
return None
|
||||||
|
return {str(k): v for k, v in (data.get('tickets') or {}).items()}
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Reading open ticket list failed: {e}')
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _annotate_staleness(events: list, open_tickets: Optional[dict]) -> None:
|
||||||
|
"""Add stale_days (set when an alert has been active too long) and ticket_closed
|
||||||
|
(its ticket was closed in Tinker while the alert is still active) to each event."""
|
||||||
|
days, _ = _stale_settings()
|
||||||
|
now = datetime.now(timezone.utc)
|
||||||
|
for ev in events:
|
||||||
|
ev['stale_days'] = None
|
||||||
|
ev['ticket_closed'] = False
|
||||||
|
try:
|
||||||
|
first = datetime.fromisoformat(str(ev.get('first_seen', '')).replace('Z', '+00:00'))
|
||||||
|
if first.tzinfo is None:
|
||||||
|
first = first.replace(tzinfo=timezone.utc)
|
||||||
|
age = (now - first).total_seconds() / 86400
|
||||||
|
if age >= days:
|
||||||
|
ev['stale_days'] = int(age)
|
||||||
|
except ValueError:
|
||||||
|
pass
|
||||||
|
tid = ev.get('ticket_id')
|
||||||
|
if tid and open_tickets is not None and str(tid) not in open_tickets:
|
||||||
|
ev['ticket_closed'] = True
|
||||||
|
|
||||||
|
|
||||||
|
def _stale_view(events: list) -> list:
|
||||||
|
"""Flag stale active alerts in place; return tickets left open after their alerts cleared."""
|
||||||
|
open_tickets = _open_tickets()
|
||||||
|
_annotate_staleness(events, open_tickets)
|
||||||
|
if open_tickets is None:
|
||||||
|
return []
|
||||||
|
_, hours = _stale_settings()
|
||||||
|
try:
|
||||||
|
return db.get_cleared_tickets(list(open_tickets), older_than_hours=hours)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Finding tickets left open failed: {e}')
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
def _state_json(key: str) -> dict:
|
def _state_json(key: str) -> dict:
|
||||||
"""Return a monitor_state JSON blob as a dict, or {} if missing or unreadable."""
|
"""Return a monitor_state JSON blob as a dict, or {} if missing or unreadable."""
|
||||||
raw = db.get_state(key)
|
raw = db.get_state(key)
|
||||||
@@ -294,6 +374,7 @@ def index():
|
|||||||
snapshot = {}
|
snapshot = {}
|
||||||
suppressions = db.get_active_suppressions()
|
suppressions = db.get_active_suppressions()
|
||||||
_annotate_suppressions(events, suppressions)
|
_annotate_suppressions(events, suppressions)
|
||||||
|
stale_tickets = _stale_view(events)
|
||||||
recent_resolved = db.get_recent_resolved(hours=24, limit=10)
|
recent_resolved = db.get_recent_resolved(hours=24, limit=10)
|
||||||
return render_template(
|
return render_template(
|
||||||
'index.html',
|
'index.html',
|
||||||
@@ -305,6 +386,8 @@ def index():
|
|||||||
last_check=last_check,
|
last_check=last_check,
|
||||||
suppressions=suppressions,
|
suppressions=suppressions,
|
||||||
recent_resolved=recent_resolved,
|
recent_resolved=recent_resolved,
|
||||||
|
stale_tickets=stale_tickets,
|
||||||
|
stale_alerts=sum(1 for e in events if e.get('stale_days') is not None or e.get('ticket_closed')),
|
||||||
daemon_ok=_daemon_ok(last_check),
|
daemon_ok=_daemon_ok(last_check),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -406,11 +489,13 @@ def api_status():
|
|||||||
active = db.get_active_events(limit=_PAGE_LIMIT)
|
active = db.get_active_events(limit=_PAGE_LIMIT)
|
||||||
suppressions = db.get_active_suppressions()
|
suppressions = db.get_active_suppressions()
|
||||||
_annotate_suppressions(active, suppressions)
|
_annotate_suppressions(active, suppressions)
|
||||||
|
stale_tickets = _stale_view(active)
|
||||||
last_check = db.get_state('last_check', 'Never')
|
last_check = db.get_state('last_check', 'Never')
|
||||||
return jsonify({
|
return jsonify({
|
||||||
'summary': db.get_status_summary(),
|
'summary': db.get_status_summary(),
|
||||||
'last_check': last_check,
|
'last_check': last_check,
|
||||||
'events': active,
|
'events': active,
|
||||||
|
'stale_tickets': stale_tickets,
|
||||||
'total_active': db.count_active_events(),
|
'total_active': db.count_active_events(),
|
||||||
'daemon_ok': _daemon_ok(last_check),
|
'daemon_ok': _daemon_ok(last_check),
|
||||||
})
|
})
|
||||||
@@ -528,6 +613,12 @@ def api_device(name: str):
|
|||||||
logger.error(f'Cutsheet unavailable for {name}: {e}')
|
logger.error(f'Cutsheet unavailable for {name}: {e}')
|
||||||
detail['cutsheet'] = []
|
detail['cutsheet'] = []
|
||||||
detail['inventory'] = _inventory_for(name, detail.get('model') or '')
|
detail['inventory'] = _inventory_for(name, detail.get('model') or '')
|
||||||
|
try:
|
||||||
|
_ensure_cutsheet_schema()
|
||||||
|
detail['unused_interfaces'] = db.get_unused_interfaces().get(name, {})
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Unused interfaces unavailable for {name}: {e}')
|
||||||
|
detail['unused_interfaces'] = {}
|
||||||
return jsonify(detail)
|
return jsonify(detail)
|
||||||
|
|
||||||
|
|
||||||
@@ -1105,17 +1196,7 @@ def _racks(items: list) -> list:
|
|||||||
return racks
|
return racks
|
||||||
|
|
||||||
|
|
||||||
_VIRTUAL_IFACE = ('lo', 'vmbr', 'bond', 'docker', 'veth', 'tap', 'fwbr', 'fwpr', 'fwln', 'br-', 'virbr', 'wg', 'tun')
|
_physical_interfaces = devices.physical_interfaces
|
||||||
|
|
||||||
|
|
||||||
def _physical_interfaces(name: str, snapshot: dict) -> list:
|
|
||||||
"""A host's physical NICs in name order, with their state: [{'name', 'up'}].
|
|
||||||
|
|
||||||
Drawn fronts light one RJ45 per NIC in this order (first NIC on the leftmost jack).
|
|
||||||
"""
|
|
||||||
ifaces = ((snapshot.get('hosts') or {}).get(name) or {}).get('interfaces') or {}
|
|
||||||
return [{'name': n, 'up': str(state).lower() == 'up'} for n, state in sorted(ifaces.items())
|
|
||||||
if not n.startswith(_VIRTUAL_IFACE)]
|
|
||||||
|
|
||||||
|
|
||||||
def _device_status(name: str, snapshot: dict) -> str:
|
def _device_status(name: str, snapshot: dict) -> str:
|
||||||
@@ -1224,6 +1305,35 @@ def api_rack():
|
|||||||
})
|
})
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Topology
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@app.route('/topology')
|
||||||
|
@require_auth
|
||||||
|
def topology_page():
|
||||||
|
return render_template('topology.html', user=_get_user())
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/api/topology')
|
||||||
|
@require_auth
|
||||||
|
def api_topology():
|
||||||
|
"""Every device and physical cable, for the topology page (see topology.build_graph)."""
|
||||||
|
events = db.get_active_events(limit=_PAGE_LIMIT)
|
||||||
|
_annotate_suppressions(events, db.get_active_suppressions())
|
||||||
|
try:
|
||||||
|
_ensure_cutsheet_schema()
|
||||||
|
sheet = db.cutsheet_load()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Cutsheet unavailable for topology: {e}')
|
||||||
|
sheet = {'items': [], 'cables': []}
|
||||||
|
graph = topology.build_graph(
|
||||||
|
_state_json('network_snapshot'), _state_json('link_stats'), _state_json('modem_stats'),
|
||||||
|
events, sheet['items'], sheet['cables'], _state_json('known_uplinks'),
|
||||||
|
)
|
||||||
|
return jsonify(graph)
|
||||||
|
|
||||||
|
|
||||||
@app.route('/api/cutsheet/racks', methods=['POST'])
|
@app.route('/api/cutsheet/racks', methods=['POST'])
|
||||||
@require_auth
|
@require_auth
|
||||||
@require_admin
|
@require_admin
|
||||||
@@ -1257,6 +1367,36 @@ def api_cutsheet_delete_rack(rack_id: int):
|
|||||||
return jsonify({'success': True})
|
return jsonify({'success': True})
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Unused interfaces
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_IFACE_RE = re.compile(r'^[A-Za-z0-9_.:@-]{1,64}$')
|
||||||
|
|
||||||
|
|
||||||
|
@app.route('/api/device/<path:name>/interfaces/<iface>/unused', methods=['PUT', 'DELETE'])
|
||||||
|
@require_auth
|
||||||
|
@require_admin
|
||||||
|
@_cutsheet_write
|
||||||
|
def api_interface_unused(name: str, iface: str):
|
||||||
|
"""Mark a host interface unused (PUT {reason}) or in use again (DELETE).
|
||||||
|
|
||||||
|
Unused interfaces raise no link-down alerts and don't count toward the host's status;
|
||||||
|
marking one quietly closes any link-down alert it has open.
|
||||||
|
"""
|
||||||
|
if len(name) > 255 or not _IFACE_RE.match(iface):
|
||||||
|
raise cutsheet.ValidationError('Not a valid host or interface name')
|
||||||
|
user = _get_user()['username']
|
||||||
|
if request.method == 'DELETE':
|
||||||
|
found = db.clear_unused_interface(name, iface)
|
||||||
|
logger.info(f'{user} marked {name} {iface} in use again')
|
||||||
|
return jsonify({'success': True, 'changed': found})
|
||||||
|
reason = str(_json_body().get('reason') or '').strip()[:255]
|
||||||
|
db.set_unused_interface(name, iface, reason, user)
|
||||||
|
logger.info(f'{user} marked {name} {iface} unused: {reason or "(no reason)"}')
|
||||||
|
return jsonify({'success': True})
|
||||||
|
|
||||||
|
|
||||||
@app.route('/api/cutsheet/cables', methods=['POST'])
|
@app.route('/api/cutsheet/cables', methods=['POST'])
|
||||||
@require_auth
|
@require_auth
|
||||||
@require_admin
|
@require_admin
|
||||||
|
|||||||
@@ -4,7 +4,7 @@ import logging
|
|||||||
import threading
|
import threading
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
from typing import Optional
|
from typing import List, Optional
|
||||||
|
|
||||||
import pymysql
|
import pymysql
|
||||||
import pymysql.cursors
|
import pymysql.cursors
|
||||||
@@ -198,6 +198,35 @@ def set_ticket_id(event_id: int, ticket_id: str) -> None:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_incident_ticket(target_name: str, within_s: int) -> Optional[str]:
|
||||||
|
"""The ticket of *target_name*'s newest event that is still open or ended within *within_s*."""
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"""SELECT ticket_id FROM network_events
|
||||||
|
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
|
||||||
|
AND (resolved_at IS NULL OR resolved_at >= NOW() - INTERVAL %s SECOND)
|
||||||
|
ORDER BY first_seen DESC, id DESC LIMIT 1""",
|
||||||
|
(target_name, int(within_s)),
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
return row['ticket_id'] if row else None
|
||||||
|
|
||||||
|
|
||||||
|
def get_ticket_ids_between(target_name: str, start: datetime, end: datetime) -> List[str]:
|
||||||
|
"""Tickets of *target_name*'s events first seen between *start* and *end* (UTC), oldest first."""
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"""SELECT ticket_id, MIN(first_seen) AS first FROM network_events
|
||||||
|
WHERE target_name=%s AND ticket_id IS NOT NULL AND ticket_id <> ''
|
||||||
|
AND first_seen BETWEEN %s AND %s
|
||||||
|
GROUP BY ticket_id ORDER BY first""",
|
||||||
|
(target_name, start, end),
|
||||||
|
)
|
||||||
|
return [r['ticket_id'] for r in cur.fetchall()]
|
||||||
|
|
||||||
|
|
||||||
def get_active_events(limit: int = 200, offset: int = 0) -> list:
|
def get_active_events(limit: int = 200, offset: int = 0) -> list:
|
||||||
with get_conn() as conn:
|
with get_conn() as conn:
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
@@ -218,6 +247,49 @@ def get_active_events(limit: int = 200, offset: int = 0) -> list:
|
|||||||
return rows
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def get_cleared_tickets(open_ids: List[str], older_than_hours: int = 24) -> list:
|
||||||
|
"""Of the given open tickets, those whose every alert cleared at least `older_than_hours` ago.
|
||||||
|
|
||||||
|
One row per ticket, describing its most recent alert, newest clearance first:
|
||||||
|
{ticket_id, event_type, target_name, target_detail, severity, first_seen, cleared_at, events}
|
||||||
|
"""
|
||||||
|
ids = sorted({str(i) for i in open_ids if i})
|
||||||
|
if not ids:
|
||||||
|
return []
|
||||||
|
placeholders = ','.join(['%s'] * len(ids))
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
f"""SELECT ticket_id, event_type, target_name, target_detail, severity,
|
||||||
|
first_seen, resolved_at
|
||||||
|
FROM network_events WHERE ticket_id IN ({placeholders})
|
||||||
|
ORDER BY first_seen""", # nosec B608: placeholders are only %s markers
|
||||||
|
ids,
|
||||||
|
)
|
||||||
|
rows = cur.fetchall()
|
||||||
|
cur.execute('SELECT NOW() AS now')
|
||||||
|
now = cur.fetchone()['now']
|
||||||
|
by_ticket: dict = {}
|
||||||
|
for r in rows:
|
||||||
|
by_ticket.setdefault(str(r['ticket_id']), []).append(r)
|
||||||
|
out = []
|
||||||
|
for tid, events in by_ticket.items():
|
||||||
|
if any(e['resolved_at'] is None for e in events):
|
||||||
|
continue
|
||||||
|
cleared = max(e['resolved_at'] for e in events)
|
||||||
|
if (now - cleared).total_seconds() < older_than_hours * 3600:
|
||||||
|
continue
|
||||||
|
last = events[-1]
|
||||||
|
out.append({
|
||||||
|
'ticket_id': tid, 'event_type': last['event_type'], 'target_name': last['target_name'],
|
||||||
|
'target_detail': last['target_detail'], 'severity': last['severity'],
|
||||||
|
'first_seen': events[0]['first_seen'].isoformat() + 'Z',
|
||||||
|
'cleared_at': cleared.isoformat() + 'Z', 'events': len(events),
|
||||||
|
})
|
||||||
|
out.sort(key=lambda r: r['cleared_at'], reverse=True)
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
def count_active_events() -> int:
|
def count_active_events() -> int:
|
||||||
"""Return count of all unresolved events (for pagination)."""
|
"""Return count of all unresolved events (for pagination)."""
|
||||||
with get_conn() as conn:
|
with get_conn() as conn:
|
||||||
@@ -738,6 +810,15 @@ _CUTSHEET_SCHEMA = (
|
|||||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||||
UNIQUE KEY uq_rack_name (name)
|
UNIQUE KEY uq_rack_name (name)
|
||||||
) ENGINE=InnoDB""",
|
) ENGINE=InnoDB""",
|
||||||
|
# Host interfaces an admin has marked as not in use: no alerts, not counted in host status
|
||||||
|
"""CREATE TABLE IF NOT EXISTS unused_interfaces (
|
||||||
|
host VARCHAR(255) NOT NULL,
|
||||||
|
iface VARCHAR(64) NOT NULL,
|
||||||
|
reason VARCHAR(255) NOT NULL DEFAULT '',
|
||||||
|
set_by VARCHAR(255) NOT NULL,
|
||||||
|
set_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
PRIMARY KEY (host, iface)
|
||||||
|
) ENGINE=InnoDB""",
|
||||||
# Where a device lives in the asset inventory (HomeBox), set by hand on its Gandalf page
|
# Where a device lives in the asset inventory (HomeBox), set by hand on its Gandalf page
|
||||||
"""CREATE TABLE IF NOT EXISTS device_links (
|
"""CREATE TABLE IF NOT EXISTS device_links (
|
||||||
name VARCHAR(255) NOT NULL PRIMARY KEY,
|
name VARCHAR(255) NOT NULL PRIMARY KEY,
|
||||||
@@ -1107,3 +1188,47 @@ def cutsheet_delete_rack(rack_id: int, user: str) -> bool:
|
|||||||
cur.execute('DELETE FROM cutsheet_racks WHERE id=%s', (rack_id,))
|
cur.execute('DELETE FROM cutsheet_racks WHERE id=%s', (rack_id,))
|
||||||
_cs_log(cur, user, 'rack', rack_id, 'delete', f'Removed rack {before["name"]}', _cs_jsonable(before), None)
|
_cs_log(cur, user, 'rack', rack_id, 'delete', f'Removed rack {before["name"]}', _cs_jsonable(before), None)
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Unused interfaces: ports an admin says aren't meant to be up
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def get_unused_interfaces() -> dict:
|
||||||
|
"""{host: {iface: {'reason', 'set_by', 'set_at'}}}"""
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('SELECT host, iface, reason, set_by, set_at FROM unused_interfaces')
|
||||||
|
out: dict = {}
|
||||||
|
for r in cur.fetchall():
|
||||||
|
at = r['set_at'].isoformat() + 'Z' if hasattr(r['set_at'], 'isoformat') else r['set_at']
|
||||||
|
out.setdefault(r['host'], {})[r['iface']] = {'reason': r['reason'], 'set_by': r['set_by'],
|
||||||
|
'set_at': at}
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def set_unused_interface(host: str, iface: str, reason: str, user: str) -> None:
|
||||||
|
"""Mark an interface unused and quietly close any link-down alert it has open.
|
||||||
|
|
||||||
|
The alert is resolved without a recovery note: nothing recovered, the port just isn't used.
|
||||||
|
"""
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"""INSERT INTO unused_interfaces (host, iface, reason, set_by) VALUES (%s, %s, %s, %s)
|
||||||
|
ON DUPLICATE KEY UPDATE reason=VALUES(reason), set_by=VALUES(set_by)""",
|
||||||
|
(host, iface, reason, user),
|
||||||
|
)
|
||||||
|
cur.execute(
|
||||||
|
"""UPDATE network_events SET resolved_at=NOW(), recovery_notified=1, last_seen=last_seen
|
||||||
|
WHERE event_type='interface_down' AND target_name=%s AND target_detail=%s
|
||||||
|
AND resolved_at IS NULL""",
|
||||||
|
(host, iface),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def clear_unused_interface(host: str, iface: str) -> bool:
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute('DELETE FROM unused_interfaces WHERE host=%s AND iface=%s', (host, iface))
|
||||||
|
return cur.rowcount > 0
|
||||||
+13
@@ -14,6 +14,19 @@ _KIND_BY_TYPE = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
_VIRTUAL_IFACE = ('lo', 'vmbr', 'bond', 'docker', 'veth', 'tap', 'fwbr', 'fwpr', 'fwln', 'br-', 'virbr', 'wg', 'tun')
|
||||||
|
|
||||||
|
|
||||||
|
def physical_interfaces(name: str, snapshot: dict) -> List[dict]:
|
||||||
|
"""A host's physical NICs in name order, with their state: [{'name', 'up', 'unused'}].
|
||||||
|
|
||||||
|
Drawn fronts light one RJ45 per NIC in this order (first NIC on the leftmost jack).
|
||||||
|
"""
|
||||||
|
ifaces = (((snapshot or {}).get('hosts') or {}).get(name) or {}).get('interfaces') or {}
|
||||||
|
return [{'name': n, 'up': str(state).lower() == 'up', 'unused': str(state).lower() == 'unused'}
|
||||||
|
for n, state in sorted(ifaces.items()) if not n.startswith(_VIRTUAL_IFACE)]
|
||||||
|
|
||||||
|
|
||||||
def unifi_kind(unifi_type: str, model: str) -> str:
|
def unifi_kind(unifi_type: str, model: str) -> str:
|
||||||
if model in _PDU_MODELS:
|
if model in _PDU_MODELS:
|
||||||
return 'pdu'
|
return 'pdu'
|
||||||
|
|||||||
Binary file not shown.
|
Before Width: | Height: | Size: 90 KiB After Width: | Height: | Size: 163 KiB |
@@ -242,6 +242,31 @@ LOG_WAN_RESTORED = 'NETWORK_WAN_RESTORED_2'
|
|||||||
LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2'
|
LOG_WAN_UNSTABLE = 'NETWORK_WAN_FAILED_MULTIPLE_TIMES_2'
|
||||||
LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED'
|
LOG_MODEM_RESTARTED = 'SMART_POWER_MODEM_RESTARTED'
|
||||||
WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED)
|
WAN_LOG_KEYS = (LOG_WAN_FAILED, LOG_WAN_RESTORED, LOG_WAN_UNSTABLE, LOG_MODEM_RESTARTED)
|
||||||
|
# An app or firmware update finishing. The UDM often drops its WAN while the Network app
|
||||||
|
# updates, and Smart Power may then power-cycle a modem that was fine.
|
||||||
|
LOG_NETWORK_UPDATED = 'NETWORK_UPDATED'
|
||||||
|
LOG_FIRMWARE_UPDATED = 'FIRMWARE_UPDATED_V2'
|
||||||
|
_UPDATE_KEY_RE = re.compile(r'[A-Z_]+_(?:UPDATED|UPGRADED)(?:_V\d+)?')
|
||||||
|
|
||||||
|
|
||||||
|
def is_update_key(key: str) -> bool:
|
||||||
|
"""True for log keys that mean an update finished (other app names follow the same pattern)."""
|
||||||
|
return key in (LOG_NETWORK_UPDATED, LOG_FIRMWARE_UPDATED) or (
|
||||||
|
bool(_UPDATE_KEY_RE.fullmatch(key or '')) and 'AVAILABLE' not in key
|
||||||
|
and not key.startswith('MADE_CHANGES'))
|
||||||
|
|
||||||
|
|
||||||
|
def wanted_log_key(key: str) -> bool:
|
||||||
|
return key in WAN_LOG_KEYS or is_update_key(key)
|
||||||
|
|
||||||
|
|
||||||
|
def update_label(event: dict) -> str:
|
||||||
|
"""'UniFi Network 11.0.97 to 11.0.99' from an update log event."""
|
||||||
|
what = event.get('app') or event.get('device') or 'UniFi'
|
||||||
|
before, after = event.get('previous') or '', event.get('version') or ''
|
||||||
|
if before and after:
|
||||||
|
return f'{what} {before} to {after}'
|
||||||
|
return f'{what} {after}'.strip()
|
||||||
|
|
||||||
_HISTORY_LIMIT = 50
|
_HISTORY_LIMIT = 50
|
||||||
# The RPDU's log entry and the gateway's calculated boot time can differ by this much.
|
# The RPDU's log entry and the gateway's calculated boot time can differ by this much.
|
||||||
|
|||||||
+227
-31
@@ -13,7 +13,8 @@ import re
|
|||||||
import shlex
|
import shlex
|
||||||
import statistics
|
import statistics
|
||||||
import time
|
import time
|
||||||
from datetime import datetime, timezone
|
from contextlib import contextmanager
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
from typing import Dict, List, Optional
|
from typing import Dict, List, Optional
|
||||||
from urllib.parse import urljoin, urlparse
|
from urllib.parse import urljoin, urlparse
|
||||||
|
|
||||||
@@ -192,8 +193,12 @@ class UnifiClient:
|
|||||||
logger.error(f'UniFi client list error: {e}')
|
logger.error(f'UniFi client list error: {e}')
|
||||||
return []
|
return []
|
||||||
|
|
||||||
def get_system_events(self, since_ms: int, keys: tuple) -> Optional[List[dict]]:
|
def get_system_events(self, since_ms: int, keys) -> Optional[List[dict]]:
|
||||||
"""Return system log entries with the given keys, or None if the log is unavailable."""
|
"""Return system log entries with the given keys, or None if the log is unavailable.
|
||||||
|
|
||||||
|
keys: a tuple of log keys, or a function that takes a key and says whether to keep it.
|
||||||
|
"""
|
||||||
|
wanted = keys if callable(keys) else keys.__contains__
|
||||||
url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/system-log/all'
|
url = f'{self.base_url}/proxy/network/v2/api/site/{self.site_id}/system-log/all'
|
||||||
found: List[dict] = []
|
found: List[dict] = []
|
||||||
try:
|
try:
|
||||||
@@ -210,19 +215,29 @@ class UnifiClient:
|
|||||||
if not items:
|
if not items:
|
||||||
break
|
break
|
||||||
for e in items:
|
for e in items:
|
||||||
if e.get('key') in keys and e.get('timestamp', 0) >= since_ms:
|
if wanted(e.get('key') or '') and e.get('timestamp', 0) >= since_ms:
|
||||||
params = e.get('parameters') or {}
|
params = e.get('parameters') or {}
|
||||||
found.append({
|
found.append({
|
||||||
'key': e['key'], 'ts': e['timestamp'] / 1000,
|
'key': e['key'], 'ts': e['timestamp'] / 1000,
|
||||||
'device': (params.get('DEVICE') or {}).get('name', ''),
|
'device': (params.get('DEVICE') or {}).get('name', ''),
|
||||||
'wan': (params.get('WAN_ID') or {}).get('name', '')
|
'wan': (params.get('WAN_ID') or {}).get('name', '')
|
||||||
or str((params.get('WAN_ID') or {}).get('id', '')),
|
or str((params.get('WAN_ID') or {}).get('id', '')),
|
||||||
|
**self._update_fields(params),
|
||||||
})
|
})
|
||||||
return found
|
return found
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f'UniFi system log error: {e}')
|
logger.error(f'UniFi system log error: {e}')
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _update_fields(params: dict) -> dict:
|
||||||
|
"""The app and versions of an update entry; nothing for other entries."""
|
||||||
|
def text(key):
|
||||||
|
value = params.get(key)
|
||||||
|
return str(value.get('name') or value.get('id') or '') if isinstance(value, dict) else str(value or '')
|
||||||
|
fields = {'app': text('APPLICATION'), 'version': text('VERSION'), 'previous': text('PREVIOUS_VERSION')}
|
||||||
|
return {k: v for k, v in fields.items() if v}
|
||||||
|
|
||||||
def get_devices(self) -> Optional[List[dict]]:
|
def get_devices(self) -> Optional[List[dict]]:
|
||||||
"""Return list of UniFi devices, or None if the controller is unreachable."""
|
"""Return list of UniFi devices, or None if the controller is unreachable."""
|
||||||
try:
|
try:
|
||||||
@@ -393,6 +408,8 @@ class TicketClient:
|
|||||||
self.comment_url = cfg.get('comment_url') or (
|
self.comment_url = cfg.get('comment_url') or (
|
||||||
urljoin(self.url, 'api/ticket_comment_api.php') if self.url else '')
|
urljoin(self.url, 'api/ticket_comment_api.php') if self.url else '')
|
||||||
self.comments_denied = False
|
self.comments_denied = False
|
||||||
|
self.read_url = cfg.get('read_url') or (
|
||||||
|
urljoin(self.url, 'api/tickets_api.php') if self.url else '')
|
||||||
# Titles whose create matched a Closed ticket Tinker couldn't reopen: {title: (ticket_id, when)}
|
# Titles whose create matched a Closed ticket Tinker couldn't reopen: {title: (ticket_id, when)}
|
||||||
self._closed_matches: Dict[str, tuple] = {}
|
self._closed_matches: Dict[str, tuple] = {}
|
||||||
|
|
||||||
@@ -427,6 +444,41 @@ class TicketClient:
|
|||||||
logger.error(f'Ticket comment failed for #{ticket_id}: {e}')
|
logger.error(f'Ticket comment failed for #{ticket_id}: {e}')
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
# Statuses that mean someone still has a ticket on their list
|
||||||
|
OPEN_STATUSES = 'Open,Pending,In Progress'
|
||||||
|
_OPEN_TICKETS_MAX_PAGES = 20
|
||||||
|
|
||||||
|
def open_tickets(self) -> Optional[Dict[str, str]]:
|
||||||
|
"""{ticket_id: status} for every ticket not yet closed, or None if Tinker can't say.
|
||||||
|
|
||||||
|
Read only: this needs the key's read scope and never changes a ticket.
|
||||||
|
"""
|
||||||
|
if not self.api_key or not self.read_url:
|
||||||
|
return None
|
||||||
|
found: Dict[str, str] = {}
|
||||||
|
try:
|
||||||
|
for page in range(1, self._OPEN_TICKETS_MAX_PAGES + 1):
|
||||||
|
resp = requests.get(
|
||||||
|
self.read_url,
|
||||||
|
params={'status': self.OPEN_STATUSES, 'limit': 100, 'page': page},
|
||||||
|
headers={'Authorization': f'Bearer {self.api_key}'},
|
||||||
|
timeout=15,
|
||||||
|
)
|
||||||
|
resp.raise_for_status()
|
||||||
|
data = resp.json()
|
||||||
|
if not data.get('success'):
|
||||||
|
logger.warning(f'Unexpected ticket list response: {str(data)[:200]}')
|
||||||
|
return None
|
||||||
|
for t in data.get('tickets') or []:
|
||||||
|
found[str(t.get('ticket_id'))] = t.get('status') or ''
|
||||||
|
if page >= int(data.get('pages') or 1):
|
||||||
|
return found
|
||||||
|
logger.warning('Open ticket list is longer than expected; using the first pages only')
|
||||||
|
return found
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Reading open tickets failed: {e}')
|
||||||
|
return None
|
||||||
|
|
||||||
# How long to wait before asking again after a create matched an unreopenable Closed ticket.
|
# How long to wait before asking again after a create matched an unreopenable Closed ticket.
|
||||||
# Each ask adds a "recurred" comment to that ticket, so once a poll would pile them up.
|
# Each ask adds a "recurred" comment to that ticket, so once a poll would pile them up.
|
||||||
CLOSED_MATCH_RETRY_S = 24 * 3600
|
CLOSED_MATCH_RETRY_S = 24 * 3600
|
||||||
@@ -877,17 +929,21 @@ class LinkStatsCollector:
|
|||||||
logger.warning(f'UniFi switch port collection failed: {e}')
|
logger.warning(f'UniFi switch port collection failed: {e}')
|
||||||
|
|
||||||
unifi_devices = (self.unifi.get_device_details() or {}) if self.unifi else {}
|
unifi_devices = (self.unifi.get_device_details() or {}) if self.unifi else {}
|
||||||
self._resolve_neighbors(unifi_switches, unifi_devices, instance_map)
|
host_macs = self._resolve_neighbors(unifi_switches, unifi_devices, instance_map)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
'hosts': result_hosts,
|
'hosts': result_hosts,
|
||||||
'unifi_switches': unifi_switches,
|
'unifi_switches': unifi_switches,
|
||||||
'unifi_devices': unifi_devices,
|
'unifi_devices': unifi_devices,
|
||||||
|
# Which host NIC owns each MAC, for matching outlets and neighbours named by MAC
|
||||||
|
'host_macs': {mac: list(owner) for mac, owner in host_macs.items()},
|
||||||
'updated': datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC'),
|
'updated': datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S UTC'),
|
||||||
}
|
}
|
||||||
|
|
||||||
def _resolve_neighbors(self, switches: dict, unifi_devices: dict, instance_map: Dict[str, str]) -> None:
|
def _resolve_neighbors(self, switches: dict, unifi_devices: dict, instance_map: Dict[str, str]) -> dict:
|
||||||
"""Name port neighbors: UniFi's LLDP data has only MACs, and many servers send none."""
|
"""Name port neighbors: UniFi's LLDP data has only MACs, and many servers send none.
|
||||||
|
|
||||||
|
Returns {mac: (host, interface)} for every physical host NIC Prometheus reports."""
|
||||||
host_macs: Dict[str, tuple] = {}
|
host_macs: Dict[str, tuple] = {}
|
||||||
host_ips: Dict[str, str] = {}
|
host_ips: Dict[str, str] = {}
|
||||||
# Bonds share their MAC with their member NICs, and LLDP runs on the NIC.
|
# Bonds share their MAC with their member NICs, and LLDP runs on the NIC.
|
||||||
@@ -910,6 +966,7 @@ class LinkStatsCollector:
|
|||||||
}
|
}
|
||||||
devices.resolve_lldp_neighbors(switches, host_macs, device_macs, host_ips)
|
devices.resolve_lldp_neighbors(switches, host_macs, device_macs, host_ips)
|
||||||
devices.apply_client_locations(switches, self.unifi.get_wired_clients(), host_macs, device_macs)
|
devices.apply_client_locations(switches, self.unifi.get_wired_clients(), host_macs, device_macs)
|
||||||
|
return host_macs
|
||||||
|
|
||||||
def _compute_unifi_rates(self, raw: Dict[str, dict], now: float) -> Dict[str, dict]:
|
def _compute_unifi_rates(self, raw: Dict[str, dict], now: float) -> Dict[str, dict]:
|
||||||
"""Compute per-port byte/error rates from delta against previous snapshot."""
|
"""Compute per-port byte/error rates from delta against previous snapshot."""
|
||||||
@@ -965,12 +1022,25 @@ CLUSTER_NAME = 'proxmox-cluster'
|
|||||||
DEFAULT_PUBLIC_URL = 'https://gandalf.lotusguild.org'
|
DEFAULT_PUBLIC_URL = 'https://gandalf.lotusguild.org'
|
||||||
# Reports of something already over: the ticket is the whole story, no recovery note
|
# Reports of something already over: the ticket is the whole story, no recovery note
|
||||||
ONE_SHOT_EVENTS = ('modem_power_cycled', 'wan_unstable')
|
ONE_SHOT_EVENTS = ('modem_power_cycled', 'wan_unstable')
|
||||||
|
# Modem and WAN reports this close together are one incident, so they share one ticket
|
||||||
|
WAN_INCIDENT_WINDOW_S = 1800
|
||||||
|
# UniFi logs an update when it finishes, after any WAN trouble it caused. So tickets opened
|
||||||
|
# this long before the entry get a note, and new modem/WAN tickets wait this long after it.
|
||||||
|
UPDATE_LOOKBACK_S = 1200
|
||||||
|
UPDATE_HOLD_S = 900
|
||||||
|
UNIFI_UPDATE_STATE = 'unifi_update'
|
||||||
# Ports a host or device was last seen on, kept so a ticket for a dead host
|
# Ports a host or device was last seen on, kept so a ticket for a dead host
|
||||||
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
|
# (whose link, and so its LLDP entry, is gone) can still say where it plugs in
|
||||||
KNOWN_UPLINKS_STATE = 'known_uplinks'
|
KNOWN_UPLINKS_STATE = 'known_uplinks'
|
||||||
|
# Which tickets are still open in Tinker, so the dashboard can flag ones left open
|
||||||
|
# after their alert cleared. Refreshed this often; one list request per refresh.
|
||||||
|
OPEN_TICKETS_STATE = 'open_tickets'
|
||||||
|
OPEN_TICKETS_SYNC_S = 900
|
||||||
|
|
||||||
|
|
||||||
class NetworkMonitor:
|
class NetworkMonitor:
|
||||||
|
_open_tickets_synced = 0.0
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
with open('config.json') as f:
|
with open('config.json') as f:
|
||||||
self.cfg = json.load(f)
|
self.cfg = json.load(f)
|
||||||
@@ -1045,6 +1115,16 @@ class NetworkMonitor:
|
|||||||
def _hostname(self, instance: str) -> str:
|
def _hostname(self, instance: str) -> str:
|
||||||
return self._instance_map.get(instance, instance.split(':')[0])
|
return self._instance_map.get(instance, instance.split(':')[0])
|
||||||
|
|
||||||
|
def _unused_for(self, host: str) -> dict:
|
||||||
|
return (getattr(self, '_unused', None) or {}).get(host) or {}
|
||||||
|
|
||||||
|
def _load_unused(self) -> None:
|
||||||
|
"""Reload the interfaces marked unused (once a cycle; keeps the last list on error)."""
|
||||||
|
try:
|
||||||
|
self._unused = db.get_unused_interfaces()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Could not load unused interfaces: {e}')
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Tickets: context, and notes when an alert clears
|
# Tickets: context, and notes when an alert clears
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -1213,6 +1293,20 @@ class NetworkMonitor:
|
|||||||
elif self.tickets.comments_denied:
|
elif self.tickets.comments_denied:
|
||||||
break
|
break
|
||||||
|
|
||||||
|
def _sync_open_tickets(self) -> None:
|
||||||
|
"""Save which tickets are still open in Tinker, at most every OPEN_TICKETS_SYNC_S.
|
||||||
|
|
||||||
|
On a failed read the previous list stays, with its old checked_at, so the
|
||||||
|
dashboard can tell it is out of date rather than treat every ticket as closed.
|
||||||
|
"""
|
||||||
|
now = time.time()
|
||||||
|
if now - self._open_tickets_synced < OPEN_TICKETS_SYNC_S:
|
||||||
|
return
|
||||||
|
self._open_tickets_synced = now
|
||||||
|
found = self.tickets.open_tickets()
|
||||||
|
if found is not None:
|
||||||
|
db.set_state(OPEN_TICKETS_STATE, {'checked_at': now, 'tickets': found})
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Interface monitoring (Prometheus)
|
# Interface monitoring (Prometheus)
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -1229,7 +1323,10 @@ class NetworkMonitor:
|
|||||||
new_baseline.setdefault(host, {})
|
new_baseline.setdefault(host, {})
|
||||||
host_has_regression = False
|
host_has_regression = False
|
||||||
|
|
||||||
|
unused = self._unused_for(host)
|
||||||
for iface, is_up in ifaces.items():
|
for iface, is_up in ifaces.items():
|
||||||
|
if iface in unused:
|
||||||
|
continue # marked unused: no alerts, and it can't count as a regression
|
||||||
prev = baseline.get(host, {}).get(iface) # 'up', 'initial_down', or None
|
prev = baseline.get(host, {}).get(iface) # 'up', 'initial_down', or None
|
||||||
|
|
||||||
if is_up:
|
if is_up:
|
||||||
@@ -1530,14 +1627,95 @@ class NetworkMonitor:
|
|||||||
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
|
event_type, severity, 'modem', name, detail, f'{description} ({_now_utc()})',
|
||||||
)
|
)
|
||||||
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
|
if not sup and consec >= self.modem_fail_thresh and not ticket_id:
|
||||||
self._open_ticket(
|
self._queue_wan_ticket({
|
||||||
event_id, event_type, name, detail,
|
'event_id': event_id, 'event_type': event_type, 'detail': detail,
|
||||||
title=f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||||
heading='WAN Modem Alert',
|
'name': heading, 'heading': 'WAN Modem Alert',
|
||||||
facts=[('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
|
'facts': [('Modem', f'{name} ({self.modem.base_url})'), ('Detected', _now_utc()),
|
||||||
('Consecutive check failures', str(consec))],
|
('Consecutive check failures', str(consec))],
|
||||||
summary=description, checks=[advice], priority=priority,
|
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': True,
|
||||||
)
|
})
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Modem and WAN tickets: one per incident, held around UniFi updates
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
@contextmanager
|
||||||
|
def _wan_batch(self):
|
||||||
|
"""Collect the cycle's modem and WAN tickets so related ones are opened together."""
|
||||||
|
self._wan_pending = []
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
finally:
|
||||||
|
pending, self._wan_pending = self._wan_pending, None
|
||||||
|
if pending:
|
||||||
|
self._open_wan_tickets(pending)
|
||||||
|
|
||||||
|
def _queue_wan_ticket(self, item: dict) -> None:
|
||||||
|
held = self._update_hold()
|
||||||
|
if held:
|
||||||
|
logger.info(f'Not opening a ticket for {item["event_type"]} yet: {held} just finished')
|
||||||
|
return
|
||||||
|
if getattr(self, '_wan_pending', None) is not None:
|
||||||
|
self._wan_pending.append(item)
|
||||||
|
else:
|
||||||
|
self._open_wan_tickets([item])
|
||||||
|
|
||||||
|
def _update_hold(self) -> Optional[str]:
|
||||||
|
"""The update that modem and WAN tickets are waiting out, if any."""
|
||||||
|
raw = db.get_state(UNIFI_UPDATE_STATE)
|
||||||
|
if not raw:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
update = json.loads(raw)
|
||||||
|
except ValueError:
|
||||||
|
return None
|
||||||
|
if update.get('hold_until', 0) <= time.time():
|
||||||
|
return None
|
||||||
|
return update.get('label') or 'a UniFi update'
|
||||||
|
|
||||||
|
def _open_wan_tickets(self, items: List[dict]) -> None:
|
||||||
|
"""Comment on the incident's ticket if one is recent, else open one ticket for all of *items*."""
|
||||||
|
# Most urgent first; at equal priority an ongoing problem leads over a past event
|
||||||
|
items = sorted(items, key=lambda i: (int(i['priority']), not i['recovers']))
|
||||||
|
incident = db.get_incident_ticket(self.modem_name, WAN_INCIDENT_WINDOW_S)
|
||||||
|
if incident:
|
||||||
|
left = []
|
||||||
|
for item in items:
|
||||||
|
text = ticket_text.incident_comment(item['name'], item['summary'], item['checks'], _now_utc())
|
||||||
|
if self.tickets.comment(incident, text):
|
||||||
|
db.set_ticket_id(item['event_id'], incident)
|
||||||
|
else:
|
||||||
|
left.append(item) # comments failing must not lose the report
|
||||||
|
items = left
|
||||||
|
if not items:
|
||||||
|
return
|
||||||
|
first, others = items[0], items[1:]
|
||||||
|
sections = []
|
||||||
|
if first['recovers']:
|
||||||
|
sections = self._ticket_sections(first['event_id'], first['event_type'], self.modem_name, first['detail'])
|
||||||
|
if others:
|
||||||
|
sections.append(('Also seen in this incident', [f'- **{o["name"]}:** {o["summary"]}' for o in others]))
|
||||||
|
desc = ticket_text.description(first['heading'], first['facts'], first['summary'], first['checks'],
|
||||||
|
sections, recovers=any(i['recovers'] for i in items))
|
||||||
|
tid = self.tickets.create(first['title'], desc, priority=first['priority'])
|
||||||
|
if tid:
|
||||||
|
for item in items:
|
||||||
|
db.set_ticket_id(item['event_id'], tid)
|
||||||
|
|
||||||
|
def _note_unifi_update(self, event: dict) -> None:
|
||||||
|
"""Hold new modem/WAN tickets for a while and explain the ones opened just before."""
|
||||||
|
label = modem_api.update_label(event)
|
||||||
|
when = datetime.fromtimestamp(event['ts'], timezone.utc)
|
||||||
|
logger.info(f'UniFi update finished: {label} at {when:%Y-%m-%d %H:%M:%S} UTC')
|
||||||
|
if event['ts'] + UPDATE_HOLD_S > time.time():
|
||||||
|
db.set_state(UNIFI_UPDATE_STATE, {'label': label, 'at': event['ts'],
|
||||||
|
'hold_until': event['ts'] + UPDATE_HOLD_S})
|
||||||
|
naive = when.replace(tzinfo=None)
|
||||||
|
tickets = db.get_ticket_ids_between(
|
||||||
|
self.modem_name, naive - timedelta(seconds=UPDATE_LOOKBACK_S), naive + timedelta(seconds=UPDATE_HOLD_S))
|
||||||
|
for tid in tickets:
|
||||||
|
self.tickets.comment(tid, ticket_text.update_comment(
|
||||||
|
label, f'{when:%Y-%m-%d %H:%M} UTC', UPDATE_LOOKBACK_S // 60, UPDATE_HOLD_S // 60))
|
||||||
|
|
||||||
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
|
def _process_modem_failure(self, suppressions: list, error: Exception) -> None:
|
||||||
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
|
"""Handle a failed poll. Link and optical events stay as they are: we can't see them."""
|
||||||
@@ -1677,14 +1855,13 @@ class NetworkMonitor:
|
|||||||
name = self.modem_name
|
name = self.modem_name
|
||||||
event_id, _, _, ticket_id = db.upsert_event(event_type, 'warning', 'unifi', name, '', description)
|
event_id, _, _, ticket_id = db.upsert_event(event_type, 'warning', 'unifi', name, '', description)
|
||||||
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
|
if not db.check_suppressed(suppressions, 'host', name) and not ticket_id:
|
||||||
tid = self.tickets.create(
|
self._queue_wan_ticket({
|
||||||
f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
'event_id': event_id, 'event_type': event_type, 'detail': '',
|
||||||
ticket_text.description('WAN Event', [('Modem', name), ('Reported', _now_utc())],
|
'title': f'[{name}][auto][production][issue][network][cluster-wide] {heading}',
|
||||||
description, [advice], recovers=False),
|
'name': heading, 'heading': 'WAN Event',
|
||||||
priority=priority,
|
'facts': [('Modem', name), ('Reported', _now_utc())],
|
||||||
)
|
'summary': description, 'checks': [advice], 'priority': priority, 'recovers': False,
|
||||||
if tid:
|
})
|
||||||
db.set_ticket_id(event_id, tid)
|
|
||||||
db.resolve_event(event_type, name, '')
|
db.resolve_event(event_type, name, '')
|
||||||
|
|
||||||
def _process_wan_log(self, suppressions: list) -> None:
|
def _process_wan_log(self, suppressions: list) -> None:
|
||||||
@@ -1700,12 +1877,21 @@ class NetworkMonitor:
|
|||||||
return
|
return
|
||||||
# After long monitor downtime, don't open tickets for stale events.
|
# After long monitor downtime, don't open tickets for stale events.
|
||||||
since_ms = max(int(raw_cursor) + 1, int((now - 86400) * 1000))
|
since_ms = max(int(raw_cursor) + 1, int((now - 86400) * 1000))
|
||||||
events = self.unifi.get_system_events(since_ms, modem_api.WAN_LOG_KEYS)
|
events = self.unifi.get_system_events(since_ms, modem_api.wanted_log_key)
|
||||||
if not events:
|
if not events:
|
||||||
return # None = log unavailable, so the cursor stays put and it's retried
|
return # None = log unavailable, so the cursor stays put and it's retried
|
||||||
|
|
||||||
|
# Updates first: one in this batch explains (and holds) the WAN events beside it
|
||||||
|
updates = [e for e in events if modem_api.is_update_key(e['key'])]
|
||||||
|
for e in sorted(updates, key=lambda ev: ev['ts']):
|
||||||
|
try:
|
||||||
|
self._note_unifi_update(e)
|
||||||
|
except Exception as ex:
|
||||||
|
logger.error(f'Noting UniFi update failed: {ex}', exc_info=True)
|
||||||
|
|
||||||
history = self._load_wan_history()
|
history = self._load_wan_history()
|
||||||
actionable = modem_api.apply_wan_events(history, events)
|
actionable = modem_api.apply_wan_events(
|
||||||
|
history, [e for e in events if not modem_api.is_update_key(e['key'])])
|
||||||
|
|
||||||
# UniFi repeats "failed multiple times" per failure, so ticket it once a day at most.
|
# UniFi repeats "failed multiple times" per failure, so ticket it once a day at most.
|
||||||
unstable = [e for e in actionable if e['key'] == modem_api.LOG_WAN_UNSTABLE]
|
unstable = [e for e in actionable if e['key'] == modem_api.LOG_WAN_UNSTABLE]
|
||||||
@@ -1799,8 +1985,11 @@ class NetworkMonitor:
|
|||||||
continue # skip Prometheus instances not in config (e.g. LXC app servers)
|
continue # skip Prometheus instances not in config (e.g. LXC app servers)
|
||||||
host = self._hostname(instance)
|
host = self._hostname(instance)
|
||||||
phys = {k: v for k, v in ifaces.items()}
|
phys = {k: v for k, v in ifaces.items()}
|
||||||
up_count = sum(1 for v in phys.values() if v)
|
# Interfaces marked unused don't count toward the host's status
|
||||||
total = len(phys)
|
unused = self._unused_for(host)
|
||||||
|
counted = {k: v for k, v in phys.items() if k not in unused}
|
||||||
|
up_count = sum(1 for v in counted.values() if v)
|
||||||
|
total = len(counted)
|
||||||
if not prom_ok:
|
if not prom_ok:
|
||||||
status = 'unknown'
|
status = 'unknown'
|
||||||
elif total == 0 or up_count == total:
|
elif total == 0 or up_count == total:
|
||||||
@@ -1812,7 +2001,7 @@ class NetworkMonitor:
|
|||||||
|
|
||||||
hosts[host] = {
|
hosts[host] = {
|
||||||
'ip': instance.split(':')[0],
|
'ip': instance.split(':')[0],
|
||||||
'interfaces': {k: ('up' if v else 'down') for k, v in phys.items()},
|
'interfaces': {k: ('unused' if k in unused else ('up' if v else 'down')) for k, v in phys.items()},
|
||||||
'status': status,
|
'status': status,
|
||||||
'source': 'prometheus',
|
'source': 'prometheus',
|
||||||
}
|
}
|
||||||
@@ -1855,6 +2044,7 @@ class NetworkMonitor:
|
|||||||
while True:
|
while True:
|
||||||
try:
|
try:
|
||||||
logger.info('Starting network check cycle')
|
logger.info('Starting network check cycle')
|
||||||
|
self._load_unused()
|
||||||
|
|
||||||
# 1. Fetch interface states once: shared by snapshot and alert processing
|
# 1. Fetch interface states once: shared by snapshot and alert processing
|
||||||
iface_states = self.prom.get_interface_states()
|
iface_states = self.prom.get_interface_states()
|
||||||
@@ -1910,9 +2100,11 @@ class NetworkMonitor:
|
|||||||
# WAN modem: isolated so a failure here can't stop the rest of the cycle
|
# WAN modem: isolated so a failure here can't stop the rest of the cycle
|
||||||
if self.modem:
|
if self.modem:
|
||||||
try:
|
try:
|
||||||
# Log first, so a restart it reports can explain an uptime reset below
|
# Log first, so a restart it reports can explain an uptime reset below.
|
||||||
self._process_wan_log(suppressions)
|
# Tickets open when the batch ends, so one incident gets one ticket.
|
||||||
self._collect_modem(suppressions)
|
with self._wan_batch():
|
||||||
|
self._process_wan_log(suppressions)
|
||||||
|
self._collect_modem(suppressions)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f'Modem collection failed: {e}', exc_info=True)
|
logger.error(f'Modem collection failed: {e}', exc_info=True)
|
||||||
|
|
||||||
@@ -1921,6 +2113,10 @@ class NetworkMonitor:
|
|||||||
self._post_recovery_notes()
|
self._post_recovery_notes()
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f'Posting recovery notes failed: {e}', exc_info=True)
|
logger.error(f'Posting recovery notes failed: {e}', exc_info=True)
|
||||||
|
try:
|
||||||
|
self._sync_open_tickets()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f'Syncing open tickets failed: {e}', exc_info=True)
|
||||||
|
|
||||||
# Housekeeping: deactivate expired suppressions and purge old resolved events
|
# Housekeeping: deactivate expired suppressions and purge old resolved events
|
||||||
db.cleanup_expired_suppressions()
|
db.cleanup_expired_suppressions()
|
||||||
|
|||||||
+108
-32
@@ -48,14 +48,16 @@ async function refreshAll() {
|
|||||||
const net = netResult.value;
|
const net = netResult.value;
|
||||||
updateHostGrid(net.hosts || {});
|
updateHostGrid(net.hosts || {});
|
||||||
updateUnifiTable(net.unifi || []);
|
updateUnifiTable(net.unifi || []);
|
||||||
updateTopology(net.hosts || {});
|
|
||||||
} else {
|
} else {
|
||||||
showToast('Network data unavailable', 'warning');
|
showToast('Network data unavailable', 'warning');
|
||||||
}
|
}
|
||||||
if (statusResult.status === 'fulfilled') {
|
if (statusResult.status === 'fulfilled') {
|
||||||
const status = statusResult.value;
|
const status = statusResult.value;
|
||||||
updateEventsTable(status.events || [], status.total_active);
|
updateEventsTable(status.events || [], status.total_active);
|
||||||
updateStatusBar(status.summary || {}, status.last_check || '', status.daemon_ok);
|
updateFollowUp(status.stale_tickets || []);
|
||||||
|
updateStatusBar(status.summary || {}, status.last_check || '', status.daemon_ok,
|
||||||
|
followUpCount(status.events || [], status.stale_tickets || []),
|
||||||
|
(status.stale_tickets || []).length > 0);
|
||||||
if (typeof window.gandalfNotifUpdate === 'function') window.gandalfNotifUpdate(status.events || []);
|
if (typeof window.gandalfNotifUpdate === 'function') window.gandalfNotifUpdate(status.events || []);
|
||||||
} else {
|
} else {
|
||||||
showToast('Status data unavailable', 'warning');
|
showToast('Status data unavailable', 'warning');
|
||||||
@@ -66,7 +68,14 @@ async function refreshAll() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function updateStatusBar(summary, lastCheck, daemonOk) {
|
// Active alerts flagged stale or with a closed ticket, plus tickets left open after clearing
|
||||||
|
function followUpCount(events, staleTickets) {
|
||||||
|
const flagged = events.filter(e => e.severity !== 'info' &&
|
||||||
|
((e.stale_days !== null && e.stale_days !== undefined) || e.ticket_closed)).length;
|
||||||
|
return flagged + staleTickets.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
function updateStatusBar(summary, lastCheck, daemonOk, followUps = 0, hasStaleTickets = false) {
|
||||||
const bar = document.querySelector('.status-chips');
|
const bar = document.querySelector('.status-chips');
|
||||||
if (!bar) return;
|
if (!bar) return;
|
||||||
const chips = [];
|
const chips = [];
|
||||||
@@ -74,6 +83,10 @@ function updateStatusBar(summary, lastCheck, daemonOk) {
|
|||||||
if (summary.critical) chips.push(`<span class="chip chip-critical">● ${summary.critical} CRITICAL</span>`);
|
if (summary.critical) chips.push(`<span class="chip chip-critical">● ${summary.critical} CRITICAL</span>`);
|
||||||
if (summary.warning) chips.push(`<span class="chip chip-warning">● ${summary.warning} WARNING</span>`);
|
if (summary.warning) chips.push(`<span class="chip chip-warning">● ${summary.warning} WARNING</span>`);
|
||||||
if (!summary.critical && !summary.warning && daemonOk !== false) chips.push('<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>');
|
if (!summary.critical && !summary.warning && daemonOk !== false) chips.push('<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>');
|
||||||
|
if (followUps) {
|
||||||
|
const href = hasStaleTickets ? '#followup' : '#events-table-wrap';
|
||||||
|
chips.push(`<a class="chip chip-warning chip-followup" href="${href}">⏳ ${followUps} TO FOLLOW UP</a>`);
|
||||||
|
}
|
||||||
bar.innerHTML = chips.join('');
|
bar.innerHTML = chips.join('');
|
||||||
|
|
||||||
const lc = document.getElementById('last-check');
|
const lc = document.getElementById('last-check');
|
||||||
@@ -153,26 +166,6 @@ function updateHostGrid(hosts) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function updateTopology(hosts) {
|
|
||||||
document.querySelectorAll('.topo-host').forEach(node => {
|
|
||||||
const name = node.dataset.host;
|
|
||||||
const host = hosts[name];
|
|
||||||
if (!host) return;
|
|
||||||
node.className = node.className.replace(/topo-v2-status-(up|down|degraded|unknown)/g, '');
|
|
||||||
node.classList.add(`topo-v2-status-${host.status}`);
|
|
||||||
const badge = node.querySelector('.topo-badge');
|
|
||||||
if (badge) {
|
|
||||||
badge.className = `topo-badge topo-badge-${host.status}`;
|
|
||||||
badge.textContent = host.status;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Animate the 10G drop-wire red+dashed when host is down
|
|
||||||
document.querySelectorAll(`.topo-v2-wire-10g[data-host="${CSS.escape(name)}"]`).forEach(wire => {
|
|
||||||
wire.classList.toggle('wire-down', host.status === 'down');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
// ── Device search ─────────────────────────────────────────────────────
|
// ── Device search ─────────────────────────────────────────────────────
|
||||||
function deviceUrl(name) {
|
function deviceUrl(name) {
|
||||||
return `/device/${encodeURIComponent(name)}`;
|
return `/device/${encodeURIComponent(name)}`;
|
||||||
@@ -382,13 +375,9 @@ function updateModem(data) {
|
|||||||
detailsLink.hidden = false;
|
detailsLink.hidden = false;
|
||||||
}
|
}
|
||||||
|
|
||||||
const rxLabel = document.getElementById('topo-modem-rx');
|
|
||||||
const topo = document.getElementById('topo-modem');
|
|
||||||
const title = document.getElementById('modem-title');
|
const title = document.getElementById('modem-title');
|
||||||
|
|
||||||
if (!data.reachable) {
|
if (!data.reachable) {
|
||||||
if (rxLabel) rxLabel.textContent = 'unreachable';
|
|
||||||
if (topo) topo.classList.add('topo-v2-status-down');
|
|
||||||
body.innerHTML = `
|
body.innerHTML = `
|
||||||
<div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')}
|
<div class="modem-error">Modem status unavailable: ${lt.escHtml(data.error || 'unknown error')}
|
||||||
<span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '-')}</span></div>
|
<span class="modem-stat-note">Last attempt ${lt.escHtml(data.updated || '-')}</span></div>
|
||||||
@@ -401,10 +390,7 @@ function updateModem(data) {
|
|||||||
const status = data.optics_status || {};
|
const status = data.optics_status || {};
|
||||||
const counters = data.counters || {};
|
const counters = data.counters || {};
|
||||||
const rates = data.rates || {};
|
const rates = data.rates || {};
|
||||||
const rx = optics.rx_power ? optics.rx_power.value : null;
|
|
||||||
|
|
||||||
if (topo) topo.classList.toggle('topo-v2-status-down', problems.length > 0);
|
|
||||||
if (rxLabel) rxLabel.textContent = (rx === null || rx === undefined) ? 'Rx -' : `Rx ${rx} dBm`;
|
|
||||||
if (title) {
|
if (title) {
|
||||||
title.textContent = [data.model || 'AT&T Gateway', data.firmware && `fw ${data.firmware}`]
|
title.textContent = [data.model || 'AT&T Gateway', data.firmware && `fw ${data.firmware}`]
|
||||||
.filter(Boolean).join(' · ');
|
.filter(Boolean).join(' · ');
|
||||||
@@ -469,7 +455,7 @@ function updateEventsTable(events, totalActive) {
|
|||||||
|
|
||||||
const truncated = totalActive !== null && totalActive !== undefined && totalActive > active.length;
|
const truncated = totalActive !== null && totalActive !== undefined && totalActive > active.length;
|
||||||
const countNotice = truncated
|
const countNotice = truncated
|
||||||
? `<div class="pagination-notice">Showing ${active.length} of ${totalActive} active alerts — <a href="/api/events?limit=1000">view all via API</a></div>`
|
? `<div class="pagination-notice">Showing ${active.length} of ${totalActive} active alerts: <a href="/api/events?limit=1000">view all via API</a></div>`
|
||||||
: '';
|
: '';
|
||||||
|
|
||||||
const rows = active.map(e => {
|
const rows = active.map(e => {
|
||||||
@@ -485,9 +471,15 @@ function updateEventsTable(events, totalActive) {
|
|||||||
const supBadge = e.is_suppressed
|
const supBadge = e.is_suppressed
|
||||||
? `<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>`
|
? `<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>`
|
||||||
: '';
|
: '';
|
||||||
|
const staleBadge = (e.stale_days !== null && e.stale_days !== undefined)
|
||||||
|
? `<span class="lt-badge badge-stale" title="Active for ${Number(e.stale_days)} days with no fix yet">⏳ ${Number(e.stale_days)}d</span>`
|
||||||
|
: '';
|
||||||
|
const closedBadge = e.ticket_closed
|
||||||
|
? `<span class="lt-badge badge-ticket-closed" title="Ticket #${lt.escHtml(String(e.ticket_id))} was closed in Tinker, but this alert is still active">ticket closed</span>`
|
||||||
|
: '';
|
||||||
return `
|
return `
|
||||||
<tr class="row-${e.severity}${e.is_suppressed ? ' row-suppressed' : ''}">
|
<tr class="row-${e.severity}${e.is_suppressed ? ' row-suppressed' : ''}">
|
||||||
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}</td>
|
<td><span class="lt-badge badge-${e.severity}">${e.severity}</span>${supBadge}${staleBadge}${closedBadge}</td>
|
||||||
<td>${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
|
<td>${lt.escHtml(e.event_type.replace(/_/g,' '))}</td>
|
||||||
<td><strong>${lt.escHtml(e.target_name)}</strong></td>
|
<td><strong>${lt.escHtml(e.target_name)}</strong></td>
|
||||||
<td>${lt.escHtml(e.target_detail || '-')}</td>
|
<td>${lt.escHtml(e.target_detail || '-')}</td>
|
||||||
@@ -522,6 +514,28 @@ function updateEventsTable(events, totalActive) {
|
|||||||
</div>`;
|
</div>`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Needs follow-up: tickets left open after their alert cleared ──────
|
||||||
|
function updateFollowUp(tickets) {
|
||||||
|
const section = document.getElementById('followup');
|
||||||
|
const tbody = document.querySelector('#followup-table tbody');
|
||||||
|
if (!section || !tbody) return;
|
||||||
|
section.hidden = !tickets.length;
|
||||||
|
const badge = document.getElementById('followup-badge');
|
||||||
|
if (badge) badge.textContent = tickets.length;
|
||||||
|
const ticketBase = (typeof GANDALF_CONFIG !== 'undefined' && GANDALF_CONFIG.ticket_web_url)
|
||||||
|
? GANDALF_CONFIG.ticket_web_url : 'http://t.lotusguild.org/ticket/';
|
||||||
|
tbody.innerHTML = tickets.map(t => `
|
||||||
|
<tr>
|
||||||
|
<td><a href="${lt.escHtml(ticketBase)}${lt.escHtml(String(t.ticket_id))}" target="_blank" rel="noopener"
|
||||||
|
class="ticket-link">#${lt.escHtml(String(t.ticket_id))}</a></td>
|
||||||
|
<td><strong>${lt.escHtml(t.target_name || '')}</strong></td>
|
||||||
|
<td>${lt.escHtml((t.event_type || '').replace(/_/g, ' '))}${t.target_detail
|
||||||
|
? ` <span class="lt-text-muted">${lt.escHtml(t.target_detail)}</span>` : ''}</td>
|
||||||
|
<td class="ts-cell" title="${lt.escHtml(t.cleared_at || '')}">${lt.time.ago(_toIso(t.cleared_at))}</td>
|
||||||
|
<td>${Number(t.events) || 0}</td>
|
||||||
|
</tr>`).join('');
|
||||||
|
}
|
||||||
|
|
||||||
// ── Suppression modal ─────────────────────────────────────────────────
|
// ── Suppression modal ─────────────────────────────────────────────────
|
||||||
function openSuppressModal(type, name, detail) {
|
function openSuppressModal(type, name, detail) {
|
||||||
const modal = document.getElementById('suppress-modal');
|
const modal = document.getElementById('suppress-modal');
|
||||||
@@ -648,3 +662,65 @@ document.addEventListener('click', e => {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ── Maintenance: pause every ticket for a while (a global suppression with an expiry) ──
|
||||||
|
(function () {
|
||||||
|
const until = document.querySelector('.maint-banner .maint-until');
|
||||||
|
if (until) {
|
||||||
|
const t = new Date(until.getAttribute('datetime'));
|
||||||
|
if (!isNaN(t)) {
|
||||||
|
const sameDay = t.toDateString() === new Date().toDateString();
|
||||||
|
until.textContent = sameDay
|
||||||
|
? t.toLocaleTimeString([], { hour: 'numeric', minute: '2-digit' })
|
||||||
|
: t.toLocaleString([], { weekday: 'short', hour: 'numeric', minute: '2-digit' });
|
||||||
|
until.title = t.toLocaleString();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let minutes = 60;
|
||||||
|
document.addEventListener('click', async e => {
|
||||||
|
if (e.target.closest('[data-action="start-maintenance"]')) {
|
||||||
|
lt.modal.open('maint-modal');
|
||||||
|
document.getElementById('maint-reason')?.focus();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const pill = e.target.closest('.maint-pill');
|
||||||
|
if (pill) {
|
||||||
|
minutes = parseInt(pill.dataset.minutes, 10);
|
||||||
|
document.querySelectorAll('.maint-pill').forEach(p => {
|
||||||
|
const on = p === pill;
|
||||||
|
p.classList.toggle('active', on);
|
||||||
|
p.setAttribute('aria-pressed', String(on));
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const end = e.target.closest('[data-action="end-maintenance"]');
|
||||||
|
if (end) {
|
||||||
|
end.disabled = true;
|
||||||
|
try {
|
||||||
|
await lt.api.delete(`/api/suppressions/${encodeURIComponent(end.dataset.supId)}`);
|
||||||
|
window.location.reload();
|
||||||
|
} catch (err) {
|
||||||
|
end.disabled = false;
|
||||||
|
showToast(err.message || 'Could not end maintenance', 'error');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
document.getElementById('maint-form')?.addEventListener('submit', async e => {
|
||||||
|
e.preventDefault();
|
||||||
|
const reason = (document.getElementById('maint-reason').value || '').trim();
|
||||||
|
if (!reason) { showToast('Reason is required', 'error'); return; }
|
||||||
|
const submit = e.target.querySelector('[type="submit"]');
|
||||||
|
submit.disabled = true;
|
||||||
|
try {
|
||||||
|
await lt.api.post('/api/suppressions', {
|
||||||
|
target_type: 'all', target_name: '', reason, expires_minutes: minutes,
|
||||||
|
});
|
||||||
|
window.location.reload();
|
||||||
|
} catch (err) {
|
||||||
|
submit.disabled = false;
|
||||||
|
showToast(err.message || 'Could not start maintenance', 'error');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
})();
|
||||||
+45
-7
@@ -122,9 +122,11 @@ function hostRows(interfaces) {
|
|||||||
return Object.entries(interfaces)
|
return Object.entries(interfaces)
|
||||||
.sort(([a], [b]) => a.localeCompare(b, undefined, { numeric: true }))
|
.sort(([a], [b]) => a.localeCompare(b, undefined, { numeric: true }))
|
||||||
.map(([name, d]) => {
|
.map(([name, d]) => {
|
||||||
const up = d.state ? d.state === 'up' : d.link_detected !== false;
|
const mark = (_device.unused_interfaces || {})[name];
|
||||||
|
const unused = !!mark || d.state === 'unused';
|
||||||
|
const up = !unused && (d.state ? d.state === 'up' : d.link_detected !== false);
|
||||||
return {
|
return {
|
||||||
id: name, label: name, up, data: d, sub: d.port_type || '',
|
id: name, label: name, up, unused, unusedReason: mark ? mark.reason : '', data: d, sub: d.port_type || '',
|
||||||
detail: `
|
detail: `
|
||||||
<div class="link-stats-grid">
|
<div class="link-stats-grid">
|
||||||
${stat('Duplex', esc(d.duplex || '-'), 'Full = both directions at once. Half can cause collisions.')}
|
${stat('Duplex', esc(d.duplex || '-'), 'Full = both directions at once. Half can cause collisions.')}
|
||||||
@@ -154,7 +156,21 @@ function portRows(ports) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function rowHasProblem(r) {
|
function rowHasProblem(r) {
|
||||||
return !!badges(r.data);
|
return !r.unused && !!badges(r.data);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The Health cell: an unused interface says so (with an admin toggle) instead of reporting down
|
||||||
|
function healthCell(r) {
|
||||||
|
if (r.unused) {
|
||||||
|
const tip = r.unusedReason ? ` title="${esc(r.unusedReason)}"` : '';
|
||||||
|
const btn = CAN_EDIT ? ` <button type="button" class="lt-btn lt-btn-ghost lt-btn-xs" data-unused="0" data-iface="${esc(r.id)}">In use again</button>` : '';
|
||||||
|
return `<span class="lt-badge badge-neutral"${tip}>unused</span>${btn}`;
|
||||||
|
}
|
||||||
|
const state = badges(r.data) || (r.up ? '<span class="val-good">OK</span>' : '<span class="device-row-sub">down</span>');
|
||||||
|
const btn = CAN_EDIT && !r.up && r.num === undefined
|
||||||
|
? ` <button type="button" class="lt-btn lt-btn-ghost lt-btn-xs" data-unused="1" data-iface="${esc(r.id)}"
|
||||||
|
title="Not meant to be up: stop alerting and stop counting it against the host">Mark unused</button>` : '';
|
||||||
|
return state + btn;
|
||||||
}
|
}
|
||||||
|
|
||||||
function renderTable(rows) {
|
function renderTable(rows) {
|
||||||
@@ -173,15 +189,15 @@ function renderTable(rows) {
|
|||||||
? (r.neighbor.name ? `<a class="device-link" href="${esc(deviceUrl(r.neighbor.name))}">${esc(r.neighbor.name)}</a>` : esc(r.neighbor.chassis || r.neighbor.port || ''))
|
? (r.neighbor.name ? `<a class="device-link" href="${esc(deviceUrl(r.neighbor.name))}">${esc(r.neighbor.name)}</a>` : esc(r.neighbor.chassis || r.neighbor.port || ''))
|
||||||
: '-';
|
: '-';
|
||||||
return `
|
return `
|
||||||
<tr class="device-row${r.up ? '' : ' device-row--down'}"${isPorts ? ` data-port-idx="${esc(r.num)}"` : ''}>
|
<tr class="device-row${r.unused ? ' device-row--unused' : (r.up ? '' : ' device-row--down')}"${isPorts ? ` data-port-idx="${esc(r.num)}"` : ''}>
|
||||||
<td><button type="button" class="device-row-toggle" data-row="${esc(r.id)}"${isPorts ? '' : ` aria-expanded="${open}"`} aria-label="Details for ${esc(r.label)}">${open ? '▾' : '▸'}</button></td>
|
<td><button type="button" class="device-row-toggle" data-row="${esc(r.id)}"${isPorts ? '' : ` aria-expanded="${open}"`} aria-label="Details for ${esc(r.label)}">${open ? '▾' : '▸'}</button></td>
|
||||||
<td><span class="dot-${r.up ? 'up' : 'down'}"></span></td>
|
<td><span class="dot-${r.unused ? 'unused' : (r.up ? 'up' : 'down')}"></span></td>
|
||||||
<td class="device-row-name">${r.num !== undefined ? `<span class="port-badge port-badge-num">#${esc(r.num)}</span> ` : ''}${esc(r.label)}<div class="device-row-sub">${esc(r.sub)}</div></td>
|
<td class="device-row-name">${r.num !== undefined ? `<span class="port-badge port-badge-num">#${esc(r.num)}</span> ` : ''}${esc(r.label)}<div class="device-row-sub">${esc(r.sub)}</div></td>
|
||||||
<td class="device-num">${r.up ? fmtSpeed(d.speed_mbps) : '-'}</td>
|
<td class="device-num">${r.up ? fmtSpeed(d.speed_mbps) : '-'}</td>
|
||||||
<td class="device-num device-col-wide">${fmtRate(d.rx_bytes_rate)}</td>
|
<td class="device-num device-col-wide">${fmtRate(d.rx_bytes_rate)}</td>
|
||||||
<td class="device-num device-col-wide">${fmtRate(d.tx_bytes_rate)}</td>
|
<td class="device-num device-col-wide">${fmtRate(d.tx_bytes_rate)}</td>
|
||||||
${isPorts ? `<td class="device-col-wide">${neighbor}</td>` : ''}
|
${isPorts ? `<td class="device-col-wide">${neighbor}</td>` : ''}
|
||||||
<td>${badges(d) || (r.up ? '<span class="val-good">OK</span>' : '<span class="device-row-sub">down</span>')}</td>
|
<td>${healthCell(r)}</td>
|
||||||
</tr>
|
</tr>
|
||||||
${open ? `<tr class="device-row-detail"><td colspan="7">${r.detail}</td></tr>` : ''}`;
|
${open ? `<tr class="device-row-detail"><td colspan="7">${r.detail}</td></tr>` : ''}`;
|
||||||
}).join('');
|
}).join('');
|
||||||
@@ -326,8 +342,10 @@ function updateInterfaceSection(rows, title, hasChassis) {
|
|||||||
document.getElementById('device-ports-title').textContent = 'Front panel';
|
document.getElementById('device-ports-title').textContent = 'Front panel';
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
const inUse = rows.filter(r => !r.unused);
|
||||||
|
const unusedCount = rows.length - inUse.length;
|
||||||
document.getElementById('device-ports-title').textContent =
|
document.getElementById('device-ports-title').textContent =
|
||||||
`${title} · ${rows.filter(r => r.up).length} of ${rows.length} up`;
|
`${title} · ${inUse.filter(r => r.up).length} of ${inUse.length} up${unusedCount ? ` · ${unusedCount} unused` : ''}`;
|
||||||
document.getElementById('device-table-search').setAttribute('aria-label', `Filter ${title.toLowerCase()}`);
|
document.getElementById('device-table-search').setAttribute('aria-label', `Filter ${title.toLowerCase()}`);
|
||||||
document.getElementById('device-error-count').textContent = rows.filter(rowHasProblem).length;
|
document.getElementById('device-error-count').textContent = rows.filter(rowHasProblem).length;
|
||||||
}
|
}
|
||||||
@@ -369,7 +387,27 @@ async function saveInventory(e) {
|
|||||||
|
|
||||||
document.addEventListener('click', e => {
|
document.addEventListener('click', e => {
|
||||||
if (e.target.closest('[data-action="edit-inventory"]')) openInventoryEditor();
|
if (e.target.closest('[data-action="edit-inventory"]')) openInventoryEditor();
|
||||||
|
const mark = e.target.closest('[data-unused]');
|
||||||
|
if (mark) toggleUnused(mark.dataset.iface, mark.dataset.unused === '1');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
async function toggleUnused(iface, unused) {
|
||||||
|
const url = `/api/device/${encodeURIComponent(DEVICE_NAME)}/interfaces/${encodeURIComponent(iface)}/unused`;
|
||||||
|
try {
|
||||||
|
if (unused) {
|
||||||
|
const reason = window.prompt(`Why isn't ${iface} in use? (optional, e.g. "spare, never cabled")`, '');
|
||||||
|
if (reason === null) return;
|
||||||
|
await lt.api.put(url, { reason });
|
||||||
|
lt.toast.success(`${iface} marked unused: it won't alert or count against ${DEVICE_NAME}`);
|
||||||
|
} else {
|
||||||
|
await lt.api.delete(url);
|
||||||
|
lt.toast.success(`${iface} is in use again`);
|
||||||
|
}
|
||||||
|
await loadDevice();
|
||||||
|
} catch (err) {
|
||||||
|
lt.toast.error(err.message || 'Save failed');
|
||||||
|
}
|
||||||
|
}
|
||||||
const _inventoryForm = document.getElementById('inventory-form');
|
const _inventoryForm = document.getElementById('inventory-form');
|
||||||
if (_inventoryForm) _inventoryForm.addEventListener('submit', saveInventory);
|
if (_inventoryForm) _inventoryForm.addEventListener('submit', saveInventory);
|
||||||
|
|
||||||
|
|||||||
+12
-1
@@ -1,5 +1,5 @@
|
|||||||
'use strict';
|
'use strict';
|
||||||
/* exported renderFaceplate, faceplateScrollLeft, FACEPLATES */
|
/* exported renderFaceplate, faceplateScrollLeft, faceplatePortAnchor, FACEPLATES */
|
||||||
|
|
||||||
// Realistic front panels, drawn as SVG from each model's real geometry.
|
// Realistic front panels, drawn as SVG from each model's real geometry.
|
||||||
// Coordinates are traced from reference images. Keys match UniFi's model code.
|
// Coordinates are traced from reference images. Keys match UniFi's model code.
|
||||||
@@ -639,6 +639,17 @@ function faceplateSvg(model, ctx, attrs = '') {
|
|||||||
return { svg, marks };
|
return { svg, marks };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Where a port's cable plugs in, in the faceplate's viewBox units: { x, y, face }, face 'top' for a
|
||||||
|
// port the right way up and 'bottom' for a flipped one (the lower row). null when there is no such
|
||||||
|
// port. A model whose one port takes any index (the PDU's management port) answers for every index.
|
||||||
|
function faceplatePortAnchor(spec, idx) {
|
||||||
|
const p = spec.ports.find(q => q.idx === idx) || spec.ports.find(q => q.anyPort);
|
||||||
|
if (!p) return null;
|
||||||
|
const face = p.flip ? 'bottom' : 'top';
|
||||||
|
if (p.type === 'sfp') return { x: p.x + p.w / 2, y: p.y + p.h / 2, face };
|
||||||
|
return { x: p.x, y: p.y, face };
|
||||||
|
}
|
||||||
|
|
||||||
// Where an fp-scroll panel should start: its port area
|
// Where an fp-scroll panel should start: its port area
|
||||||
function faceplateScrollLeft(model, frame) {
|
function faceplateScrollLeft(model, frame) {
|
||||||
const spec = FACEPLATES[model];
|
const spec = FACEPLATES[model];
|
||||||
|
|||||||
+1
-1
@@ -391,7 +391,7 @@ function shelfSvg(item, esc, faceplate) {
|
|||||||
<text class="sh-box-name" x="${x + w / 2}" y="${floor - h / 2 + 5}" text-anchor="middle">${esc(c.name)}</text>`;
|
<text class="sh-box-name" x="${x + w / 2}" y="${floor - h / 2 + 5}" text-anchor="middle">${esc(c.name)}</text>`;
|
||||||
}
|
}
|
||||||
const top = floor - h;
|
const top = floor - h;
|
||||||
const nics = (c.interfaces || []).map(n => `${n.name} ${n.up ? 'up' : 'down'}`).join(', ');
|
const nics = (c.interfaces || []).map(n => `${n.name} ${n.unused ? 'unused' : (n.up ? 'up' : 'down')}`).join(', ');
|
||||||
const label = gear ? gear.label : (desk ? desk.label : c.kind);
|
const label = gear ? gear.label : (desk ? desk.label : c.kind);
|
||||||
const tip = [c.name, label, c.status, nics, gear && gear.specs].filter(Boolean).join(' · ');
|
const tip = [c.name, label, c.status, nics, gear && gear.specs].filter(Boolean).join(' · ');
|
||||||
// A live faceplate's ports are links of their own, so it isn't wrapped in one
|
// A live faceplate's ports are links of their own, so it isn't wrapped in one
|
||||||
|
|||||||
+257
-269
@@ -60,12 +60,6 @@
|
|||||||
--glow-cyan: none;
|
--glow-cyan: none;
|
||||||
--glow-xl: none;
|
--glow-xl: none;
|
||||||
}
|
}
|
||||||
[data-theme="light"] .topology {
|
|
||||||
background-image: radial-gradient(circle, rgba(0,100,160,0.07) 1px, transparent 1px);
|
|
||||||
}
|
|
||||||
[data-theme="light"] .topo-vc-label {
|
|
||||||
background: rgba(235,238,242,.88);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ── Header overlap fix ───────────────────────────────────────────
|
/* ── Header overlap fix ───────────────────────────────────────────
|
||||||
.lt-container's padding shorthand resets padding-top, defeating
|
.lt-container's padding shorthand resets padding-top, defeating
|
||||||
@@ -151,23 +145,6 @@
|
|||||||
/* ── Stats grid spacing variant ──────────────────────────────────── */
|
/* ── Stats grid spacing variant ──────────────────────────────────── */
|
||||||
.lt-stats-grid--mb { margin-bottom: 16px; }
|
.lt-stats-grid--mb { margin-bottom: 16px; }
|
||||||
|
|
||||||
/* ── Topology section collapse toggle ────────────────────────────── */
|
|
||||||
.topo-collapse-btn {
|
|
||||||
margin-left: auto;
|
|
||||||
font-size: .7em;
|
|
||||||
color: var(--text-muted);
|
|
||||||
background: transparent;
|
|
||||||
border: 1px solid var(--border-color);
|
|
||||||
padding: 2px 8px;
|
|
||||||
cursor: pointer;
|
|
||||||
font-family: var(--font);
|
|
||||||
letter-spacing: .04em;
|
|
||||||
transition: border-color .15s, color .15s;
|
|
||||||
}
|
|
||||||
.topo-collapse-btn:hover { border-color: var(--amber); color: var(--amber); }
|
|
||||||
.topo-collapsible { overflow: hidden; transition: max-height .25s ease; }
|
|
||||||
.topo-collapsible.is-collapsed { display: none; }
|
|
||||||
|
|
||||||
/* ── Animations used by custom components ─────────────────────────── */
|
/* ── Animations used by custom components ─────────────────────────── */
|
||||||
@keyframes pulse-red {
|
@keyframes pulse-red {
|
||||||
0%,100% { box-shadow: 0 0 0 0 rgba(255,45,85,.5); }
|
0%,100% { box-shadow: 0 0 0 0 rgba(255,45,85,.5); }
|
||||||
@@ -229,6 +206,12 @@
|
|||||||
.badge-resolved { color: var(--text-muted); border-color: var(--border-color); text-decoration: line-through; }
|
.badge-resolved { color: var(--text-muted); border-color: var(--border-color); text-decoration: line-through; }
|
||||||
.badge-suppressed { font-size: .9em; padding: 0; border: none; color: var(--text-muted); }
|
.badge-suppressed { font-size: .9em; padding: 0; border: none; color: var(--text-muted); }
|
||||||
.badge-purple { color: var(--accent-purple); border-color: var(--accent-purple); }
|
.badge-purple { color: var(--accent-purple); border-color: var(--accent-purple); }
|
||||||
|
.badge-stale { color: var(--amber, var(--orange)); border-color: transparent; margin-left: 4px; white-space: nowrap; }
|
||||||
|
.badge-ticket-closed { color: var(--accent-purple); border-color: var(--accent-purple); margin-left: 4px; white-space: nowrap; }
|
||||||
|
.chip-followup { text-decoration: none; cursor: pointer; }
|
||||||
|
.chip-followup:hover, .chip-followup:focus-visible { background: rgba(255,107,0,.12); }
|
||||||
|
.followup-intro { color: var(--text-muted); font-size: .85em; margin: 0 0 8px; }
|
||||||
|
#followup-table td { white-space: nowrap; }
|
||||||
|
|
||||||
/* ── Table row state colors ───────────────────────────────────────── */
|
/* ── Table row state colors ───────────────────────────────────────── */
|
||||||
.lt-table tr.row-critical td { background: rgba(255,45,85,.04); }
|
.lt-table tr.row-critical td { background: rgba(255,45,85,.04); }
|
||||||
@@ -270,7 +253,7 @@
|
|||||||
border: 1px solid;
|
border: 1px solid;
|
||||||
letter-spacing: .04em;
|
letter-spacing: .04em;
|
||||||
}
|
}
|
||||||
.chip-critical { color: var(--red); border-color: var(--red); text-shadow: var(--glow-red); animation: topo-pulse-down 2s ease-in-out infinite; }
|
.chip-critical { color: var(--red); border-color: var(--red); text-shadow: var(--glow-red); animation: pulse-down 2s ease-in-out infinite; }
|
||||||
.chip-warning { color: var(--orange); border-color: var(--orange); }
|
.chip-warning { color: var(--orange); border-color: var(--orange); }
|
||||||
.chip-ok { color: var(--green); border-color: var(--green-muted); text-shadow: var(--glow); }
|
.chip-ok { color: var(--green); border-color: var(--green-muted); text-shadow: var(--glow); }
|
||||||
.status-meta { display: flex; align-items: center; gap: 10px; white-space: nowrap; }
|
.status-meta { display: flex; align-items: center; gap: 10px; white-space: nowrap; }
|
||||||
@@ -383,6 +366,22 @@
|
|||||||
.pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
|
.pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
|
||||||
.pill.active,
|
.pill.active,
|
||||||
.pill-manual.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
|
.pill-manual.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
|
||||||
|
/* Maintenance modal pills: same look, own class so the suppress modal's handler skips them */
|
||||||
|
.maint-pill { padding: 3px 10px; border: 1px solid var(--border-color); background: transparent;
|
||||||
|
font-family: var(--font); font-size: .72em; font-weight: bold; cursor: pointer;
|
||||||
|
color: var(--text-muted); transition: all .15s; letter-spacing: .04em; }
|
||||||
|
.maint-pill:hover { border-color: var(--green); color: var(--green); background: var(--green-dim); }
|
||||||
|
.maint-pill.active { border-color: var(--amber); color: var(--amber); background: var(--amber-dim); text-shadow: var(--glow-amber); }
|
||||||
|
.maint-intro { margin: 0 0 12px; }
|
||||||
|
|
||||||
|
/* Maintenance banner: every page while a global suppression is active */
|
||||||
|
.maint-banner {
|
||||||
|
display: flex; align-items: center; justify-content: space-between; gap: 12px; flex-wrap: wrap;
|
||||||
|
margin-bottom: 16px; padding: 8px 14px; border: 1px solid var(--amber); background: var(--amber-dim);
|
||||||
|
color: var(--amber); font-size: .82em;
|
||||||
|
}
|
||||||
|
.maint-banner strong { letter-spacing: .06em; text-transform: uppercase; }
|
||||||
|
.maint-reason { color: var(--text-muted); }
|
||||||
|
|
||||||
/* ── Available targets grid (suppressions page) ───────────────────── */
|
/* ── Available targets grid (suppressions page) ───────────────────── */
|
||||||
.targets-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(180px, 1fr)); gap: 10px; }
|
.targets-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(180px, 1fr)); gap: 10px; }
|
||||||
@@ -396,252 +395,12 @@
|
|||||||
.target-ifaces{ display: flex; flex-wrap: wrap; gap: 3px; }
|
.target-ifaces{ display: flex; flex-wrap: wrap; gap: 3px; }
|
||||||
.iface-chip { font-family: var(--font); font-size: .65em; background: var(--bg3); border: 1px solid var(--border-color); padding: 1px 5px; color: var(--text-dim); }
|
.iface-chip { font-family: var(--font); font-size: .65em; background: var(--bg3); border: 1px solid var(--border-color); padding: 1px 5px; color: var(--text-dim); }
|
||||||
|
|
||||||
/* ── Topology diagram ─────────────────────────────────────────────── */
|
/* Pulsing red glow for critical chips and down states */
|
||||||
.topology {
|
@keyframes pulse-down {
|
||||||
background-color: var(--bg2);
|
|
||||||
background-image: radial-gradient(circle, rgba(0,212,255,0.07) 1px, transparent 1px);
|
|
||||||
background-size: 22px 22px;
|
|
||||||
border: 1px solid var(--border-color);
|
|
||||||
padding: 20px 16px 16px;
|
|
||||||
margin-bottom: 16px;
|
|
||||||
text-align: center;
|
|
||||||
overflow-x: auto;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* topo-v2 outer wrapper */
|
|
||||||
.topo-v2 {
|
|
||||||
display: flex;
|
|
||||||
flex-direction: column;
|
|
||||||
align-items: center;
|
|
||||||
gap: 0;
|
|
||||||
min-width: 860px;
|
|
||||||
padding: 20px 24px 24px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.topo-tier {
|
|
||||||
display: flex;
|
|
||||||
justify-content: center;
|
|
||||||
align-items: center;
|
|
||||||
width: 100%;
|
|
||||||
}
|
|
||||||
|
|
||||||
.topo-vc {
|
|
||||||
display: flex;
|
|
||||||
justify-content: center;
|
|
||||||
align-items: flex-start;
|
|
||||||
width: 100%;
|
|
||||||
position: relative;
|
|
||||||
height: 40px;
|
|
||||||
}
|
|
||||||
.topo-vc-wire {
|
|
||||||
position: absolute;
|
|
||||||
left: 50%;
|
|
||||||
top: 0;
|
|
||||||
transform: translateX(-50%);
|
|
||||||
width: 2px;
|
|
||||||
height: 100%;
|
|
||||||
background: linear-gradient(to bottom, var(--cyan), var(--green));
|
|
||||||
opacity: .7;
|
|
||||||
}
|
|
||||||
.topo-vc-wire--wan { background: linear-gradient(to bottom, var(--cyan), rgba(0,212,255,.3)); opacity: .7; }
|
|
||||||
.topo-vc-wire--10g { background: var(--amber); opacity: .6; }
|
|
||||||
.topo-vc-wire--mgmt { background: var(--border-color); opacity: .5; }
|
|
||||||
/* Blurred copy of the wire for a soft glow halo */
|
|
||||||
.topo-vc-wire::before {
|
|
||||||
content: '';
|
|
||||||
position: absolute;
|
|
||||||
inset: 0;
|
|
||||||
background: inherit;
|
|
||||||
filter: blur(5px);
|
|
||||||
opacity: .5;
|
|
||||||
}
|
|
||||||
.topo-vc-label {
|
|
||||||
position: absolute;
|
|
||||||
left: calc(50% + 7px);
|
|
||||||
top: 50%;
|
|
||||||
transform: translateY(-50%);
|
|
||||||
font-size: .58em;
|
|
||||||
color: var(--amber);
|
|
||||||
white-space: nowrap;
|
|
||||||
letter-spacing: .06em;
|
|
||||||
font-family: var(--font);
|
|
||||||
background: rgba(3,5,8,.7);
|
|
||||||
padding: 1px 4px;
|
|
||||||
}
|
|
||||||
|
|
||||||
.topo-v2-node {
|
|
||||||
display: flex;
|
|
||||||
flex-direction: column;
|
|
||||||
align-items: center;
|
|
||||||
gap: 3px;
|
|
||||||
padding: 8px 16px;
|
|
||||||
border: 1px solid var(--border-color);
|
|
||||||
background: var(--bg3);
|
|
||||||
position: relative;
|
|
||||||
font-size: .75em;
|
|
||||||
font-family: var(--font);
|
|
||||||
min-width: 110px;
|
|
||||||
text-align: center;
|
|
||||||
transition: border-color .2s, box-shadow .2s;
|
|
||||||
overflow: hidden;
|
|
||||||
}
|
|
||||||
/* Top highlight strip: color matches node type / status */
|
|
||||||
.topo-v2-node::before {
|
|
||||||
content: '';
|
|
||||||
position: absolute;
|
|
||||||
top: 0; left: 0; right: 0;
|
|
||||||
height: 2px;
|
|
||||||
background: currentColor;
|
|
||||||
opacity: .4;
|
|
||||||
}
|
|
||||||
.topo-v2-status-up.topo-v2-node::before { background: var(--green); opacity: .65; }
|
|
||||||
.topo-v2-status-down.topo-v2-node::before { background: var(--red); opacity: .75; }
|
|
||||||
.topo-v2-status-degraded.topo-v2-node::before { background: var(--orange); opacity: .65; }
|
|
||||||
.topo-v2-status-unknown.topo-v2-node::before { opacity: .15; }
|
|
||||||
.topo-v2-icon { font-size: 1.3em; line-height: 1; }
|
|
||||||
.topo-v2-label { font-weight: bold; letter-spacing: .04em; }
|
|
||||||
.topo-v2-sub { font-size: .58em; color: var(--text-muted); letter-spacing: .02em; }
|
|
||||||
.topo-v2-vlan { font-size: .54em; color: var(--cyan); opacity: .75; }
|
|
||||||
|
|
||||||
.topo-v2-host--bus { min-width: 80px; max-width: 96px; }
|
|
||||||
.topo-v2-internet { border-color: var(--cyan); color: var(--cyan); text-shadow: var(--glow-cyan); box-shadow: 0 0 12px rgba(0,212,255,.12); }
|
|
||||||
.topo-v2-router { border-color: var(--cyan); color: var(--cyan); text-shadow: var(--glow-cyan); box-shadow: 0 0 12px rgba(0,212,255,.14); }
|
|
||||||
.topo-v2-switch { border-color: var(--amber); color: var(--amber); text-shadow: var(--glow-amber); box-shadow: 0 0 12px rgba(255,179,0,.12); }
|
|
||||||
.topo-v2-host { border-color: var(--border-color); color: var(--text); cursor: default; }
|
|
||||||
|
|
||||||
@keyframes topo-pulse-down {
|
|
||||||
0%,100% { box-shadow: 0 0 6px rgba(255,45,85,.3); }
|
0%,100% { box-shadow: 0 0 6px rgba(255,45,85,.3); }
|
||||||
50% { box-shadow: 0 0 18px rgba(255,45,85,.75), 0 0 30px rgba(255,45,85,.2); }
|
50% { box-shadow: 0 0 18px rgba(255,45,85,.75), 0 0 30px rgba(255,45,85,.2); }
|
||||||
}
|
}
|
||||||
|
|
||||||
.topo-v2-status-up { border-color: var(--green); box-shadow: 0 0 8px rgba(0,255,136,.2); }
|
|
||||||
.topo-v2-status-down { border-color: var(--red); animation: topo-pulse-down 2s ease-in-out infinite; }
|
|
||||||
.topo-v2-status-degraded { border-color: var(--orange); box-shadow: 0 0 8px rgba(255,107,0,.2); }
|
|
||||||
.topo-v2-status-unknown { border-color: var(--border-color); }
|
|
||||||
.topo-v2-offrack { border-style: dashed !important; }
|
|
||||||
|
|
||||||
.topo-badge { font-size: .68em; padding: 1px 5px; border: 1px solid; letter-spacing: .03em; }
|
|
||||||
.topo-badge-up { color: var(--green); border-color: var(--green); text-shadow: var(--glow); }
|
|
||||||
.topo-badge-down { color: var(--red); border-color: var(--red); animation: topo-pulse-down 1.5s ease-in-out infinite; }
|
|
||||||
.topo-badge-degraded { color: var(--orange); border-color: var(--orange); }
|
|
||||||
.topo-badge-unknown { color: var(--text-muted); border-color: var(--border-color); }
|
|
||||||
|
|
||||||
/* Host wrap + dual-homing wires */
|
|
||||||
.topo-v2-host-wrap {
|
|
||||||
display: flex;
|
|
||||||
flex-direction: column;
|
|
||||||
align-items: center;
|
|
||||||
}
|
|
||||||
.topo-v2-host-wires {
|
|
||||||
display: flex;
|
|
||||||
gap: 6px;
|
|
||||||
height: 28px;
|
|
||||||
align-items: flex-start;
|
|
||||||
}
|
|
||||||
.topo-v2-wire-10g { width: 2px; height: 100%; background: var(--green); opacity: .55; transition: background .3s, opacity .3s; }
|
|
||||||
.topo-v2-wire-1g { width: 0; height: 100%; border-left: 2px dashed var(--amber); opacity: .45; }
|
|
||||||
|
|
||||||
@keyframes wire-dash-anim { to { background-position: 0 -20px; } }
|
|
||||||
.topo-v2-wire-10g.wire-down {
|
|
||||||
background: repeating-linear-gradient(to bottom, var(--red) 0 6px, transparent 6px 10px) !important;
|
|
||||||
background-size: 2px 10px !important;
|
|
||||||
opacity: .9 !important;
|
|
||||||
transition: none !important;
|
|
||||||
animation: wire-dash-anim .7s linear infinite;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Bus rails */
|
|
||||||
.topo-bus-section {
|
|
||||||
width: 100%;
|
|
||||||
max-width: 860px;
|
|
||||||
display: flex;
|
|
||||||
flex-direction: column;
|
|
||||||
align-items: stretch;
|
|
||||||
gap: 0;
|
|
||||||
}
|
|
||||||
.topo-bus-10g {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
position: relative;
|
|
||||||
height: 20px;
|
|
||||||
}
|
|
||||||
.topo-bus-10g-line {
|
|
||||||
flex: 1;
|
|
||||||
height: 2px;
|
|
||||||
background: var(--green);
|
|
||||||
opacity: .55;
|
|
||||||
margin: 0 4px;
|
|
||||||
box-shadow: 0 0 6px rgba(0,255,136,.4);
|
|
||||||
}
|
|
||||||
.topo-bus-10g-label {
|
|
||||||
font-size: .56em;
|
|
||||||
color: var(--green);
|
|
||||||
white-space: nowrap;
|
|
||||||
letter-spacing: .05em;
|
|
||||||
font-family: var(--font);
|
|
||||||
opacity: .85;
|
|
||||||
padding: 0 8px;
|
|
||||||
}
|
|
||||||
.topo-bus-1g {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
position: relative;
|
|
||||||
height: 18px;
|
|
||||||
}
|
|
||||||
.topo-bus-1g-line {
|
|
||||||
flex: 1;
|
|
||||||
height: 0;
|
|
||||||
border-top: 2px dashed var(--amber);
|
|
||||||
opacity: .35;
|
|
||||||
margin: 0 4px;
|
|
||||||
}
|
|
||||||
.topo-bus-1g-label {
|
|
||||||
font-size: .56em;
|
|
||||||
color: var(--amber);
|
|
||||||
white-space: nowrap;
|
|
||||||
letter-spacing: .05em;
|
|
||||||
font-family: var(--font);
|
|
||||||
opacity: .8;
|
|
||||||
padding: 0 8px;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Hosts row */
|
|
||||||
.topo-v2-hosts {
|
|
||||||
display: flex;
|
|
||||||
justify-content: center;
|
|
||||||
gap: 12px;
|
|
||||||
flex-wrap: wrap;
|
|
||||||
padding-top: 4px;
|
|
||||||
width: 100%;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* Topology legend */
|
|
||||||
.topo-legend {
|
|
||||||
display: flex;
|
|
||||||
gap: 18px;
|
|
||||||
align-items: center;
|
|
||||||
margin-top: 14px;
|
|
||||||
padding-top: 10px;
|
|
||||||
border-top: 1px solid rgba(255,107,0,.12);
|
|
||||||
flex-wrap: wrap;
|
|
||||||
justify-content: center;
|
|
||||||
}
|
|
||||||
.topo-legend-item {
|
|
||||||
display: flex;
|
|
||||||
align-items: center;
|
|
||||||
gap: 5px;
|
|
||||||
font-size: .58em;
|
|
||||||
color: var(--text-muted);
|
|
||||||
font-family: var(--font);
|
|
||||||
}
|
|
||||||
.topo-legend-item--offrack {
|
|
||||||
border: 1px dashed var(--border-color);
|
|
||||||
padding: 1px 5px;
|
|
||||||
}
|
|
||||||
.topo-legend-line-10g { width: 24px; height: 2px; background: var(--green); display: inline-block; box-shadow: 0 0 4px rgba(0,255,136,.5); }
|
|
||||||
.topo-legend-line-1g { width: 24px; height: 0; border-top: 2px dashed var(--amber); display: inline-block; }
|
|
||||||
.topo-legend-line-wan { width: 24px; height: 2px; background: linear-gradient(to right, var(--cyan), var(--green)); display: inline-block; }
|
|
||||||
|
|
||||||
/* ── Link debug page ──────────────────────────────────────────────── */
|
/* ── Link debug page ──────────────────────────────────────────────── */
|
||||||
.link-host-list { display: flex; flex-direction: column; gap: 18px; }
|
.link-host-list { display: flex; flex-direction: column; gap: 18px; }
|
||||||
.link-host-panel {
|
.link-host-panel {
|
||||||
@@ -1045,7 +804,7 @@
|
|||||||
.lt-stat-card--alert {
|
.lt-stat-card--alert {
|
||||||
border-color: var(--red) !important;
|
border-color: var(--red) !important;
|
||||||
box-shadow: 0 0 8px rgba(255,45,85,.25) !important;
|
box-shadow: 0 0 8px rgba(255,45,85,.25) !important;
|
||||||
animation: topo-pulse-down 2s ease-in-out infinite;
|
animation: pulse-down 2s ease-in-out infinite;
|
||||||
}
|
}
|
||||||
.lt-stat-card--alert::before { background: var(--red); box-shadow: var(--glow-red); }
|
.lt-stat-card--alert::before { background: var(--red); box-shadow: var(--glow-red); }
|
||||||
|
|
||||||
@@ -1055,7 +814,6 @@
|
|||||||
/* ── Responsive ───────────────────────────────────────────────────── */
|
/* ── Responsive ───────────────────────────────────────────────────── */
|
||||||
@media (max-width: 768px) {
|
@media (max-width: 768px) {
|
||||||
.host-grid { grid-template-columns: 1fr; }
|
.host-grid { grid-template-columns: 1fr; }
|
||||||
.topology { display: none; }
|
|
||||||
.form-row { flex-direction: column; }
|
.form-row { flex-direction: column; }
|
||||||
.status-bar { flex-direction: column; align-items: flex-start; }
|
.status-bar { flex-direction: column; align-items: flex-start; }
|
||||||
.link-ifaces-grid { grid-template-columns: 1fr; }
|
.link-ifaces-grid { grid-template-columns: 1fr; }
|
||||||
@@ -1128,7 +886,6 @@ a.device-link:hover { color: var(--text-accent); border-bottom-color: current
|
|||||||
.device-row-sub { font-size: .72em; color: var(--text-muted); }
|
.device-row-sub { font-size: .72em; color: var(--text-muted); }
|
||||||
.device-row-toggle { background: none; border: 0; color: inherit; cursor: pointer; font: inherit; padding: 2px 6px; }
|
.device-row-toggle { background: none; border: 0; color: inherit; cursor: pointer; font: inherit; padding: 2px 6px; }
|
||||||
.device-row-detail > td { background: var(--bg-tertiary); padding: 10px 14px; }
|
.device-row-detail > td { background: var(--bg-tertiary); padding: 10px 14px; }
|
||||||
.topo-host[data-host] { cursor: pointer; }
|
|
||||||
|
|
||||||
/* WAN outage list in the modem panel */
|
/* WAN outage list in the modem panel */
|
||||||
.modem-outages { padding: 0 12px 8px; }
|
.modem-outages { padding: 0 12px 8px; }
|
||||||
@@ -1648,3 +1405,234 @@ body.fp-marks-off .fp-svg .fp-mark { display: none; }
|
|||||||
.ub-opening { fill: #0d0e10; }
|
.ub-opening { fill: #0d0e10; }
|
||||||
.ub-seam { fill: #050506; opacity: .8; }
|
.ub-seam { fill: #050506; opacity: .8; }
|
||||||
.ub-strand { fill: #000; }
|
.ub-strand { fill: #000; }
|
||||||
|
|
||||||
|
/* Interfaces marked unused: greyed, never "down" */
|
||||||
|
.dot-unused { display: inline-block; width: 8px; height: 8px; border: 1px dashed var(--text-muted);
|
||||||
|
background: transparent; flex-shrink: 0; vertical-align: middle; }
|
||||||
|
.state-unused { color: var(--text-muted); }
|
||||||
|
.device-row--unused td { opacity: .6; }
|
||||||
|
.device-row--unused td:last-child { opacity: 1; }
|
||||||
|
.lt-btn-xs { padding: 1px 6px; font-size: .68rem; margin-left: 6px; }
|
||||||
|
|
||||||
|
/* ── Topology page ─────────────────────────────────────────────────── */
|
||||||
|
.tp-toolbar { display: flex; flex-wrap: wrap; align-items: center; gap: 10px 18px; margin: 8px 0 10px; }
|
||||||
|
.tp-filters { display: flex; flex-wrap: wrap; gap: 6px 14px; }
|
||||||
|
.tp-filter { display: inline-flex; align-items: center; gap: 6px; font-size: .78em; color: var(--text-dim); cursor: pointer; white-space: nowrap; }
|
||||||
|
.tp-zoom { display: inline-flex; gap: 4px; }
|
||||||
|
.tp-updated { margin-left: auto; font-size: .72em; color: var(--text-muted); }
|
||||||
|
.tp-warnings { border: 1px solid var(--amber); color: var(--amber); background: var(--amber-dim); padding: 8px 12px; margin-bottom: 10px; font-size: .8em; }
|
||||||
|
.tp-canvas {
|
||||||
|
position: relative; overflow: hidden; touch-action: none; cursor: grab;
|
||||||
|
height: clamp(420px, calc(100vh - 300px), 1400px);
|
||||||
|
border: 1px solid var(--border-color);
|
||||||
|
background-color: var(--bg2);
|
||||||
|
background-image: radial-gradient(circle, rgba(0,212,255,.07) 1px, transparent 1px);
|
||||||
|
background-size: 22px 22px;
|
||||||
|
}
|
||||||
|
[data-theme="light"] .tp-canvas { background-image: radial-gradient(circle, rgba(0,100,160,.08) 1px, transparent 1px); }
|
||||||
|
.tp-canvas:focus-visible { outline: 2px solid var(--amber); outline-offset: 2px; }
|
||||||
|
.tp-canvas.tp-dragging { cursor: grabbing; }
|
||||||
|
.tp-svg { display: block; width: 100%; height: 100%; user-select: none; }
|
||||||
|
.tp-loading { position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; }
|
||||||
|
.tp-loading[hidden] { display: none; }
|
||||||
|
|
||||||
|
/* Devices */
|
||||||
|
.tp-node-link { cursor: pointer; outline: none; }
|
||||||
|
.tp-node-hit { fill: transparent; stroke: transparent; stroke-width: 4; }
|
||||||
|
.tp-node-link:hover .tp-node-hit, .tp-node-link:focus-visible .tp-node-hit { stroke: var(--amber); fill: rgba(255,179,0,.04); }
|
||||||
|
.tp-name { font-family: var(--font); font-size: 40px; font-weight: 700; fill: var(--text); letter-spacing: .02em;
|
||||||
|
paint-order: stroke; stroke: var(--bg2); stroke-width: 10px; stroke-linejoin: round; }
|
||||||
|
.tp-sub { font-family: var(--font); font-size: 28px; fill: var(--text-muted);
|
||||||
|
paint-order: stroke; stroke: var(--bg2); stroke-width: 8px; stroke-linejoin: round; }
|
||||||
|
.tp-dot { stroke: rgba(0,0,0,.35); stroke-width: 2; }
|
||||||
|
.tp-dot-up { fill: #2fd36b; }
|
||||||
|
.tp-dot-degraded { fill: #ffb300; }
|
||||||
|
.tp-dot-down { fill: #e0312b; }
|
||||||
|
.tp-dot-unknown { fill: #6b737d; }
|
||||||
|
.tp-ring { fill: none; stroke-width: 7; }
|
||||||
|
.tp-ring-critical { stroke: #e0312b; }
|
||||||
|
.tp-ring-warning { stroke: #ffb300; }
|
||||||
|
.tp-ring-info { stroke: #34a8ff; }
|
||||||
|
.tp-badge circle { stroke: var(--bg2); stroke-width: 5; }
|
||||||
|
.tp-badge text { font-family: var(--font); font-size: 34px; font-weight: 700; fill: #fff; }
|
||||||
|
.tp-badge-critical circle { fill: #e0312b; }
|
||||||
|
.tp-badge-warning circle { fill: #c98a00; }
|
||||||
|
.tp-badge-info circle { fill: #1f7ad6; }
|
||||||
|
.tp-kind-stub .tp-name { font-size: 32px; font-weight: 400; fill: var(--text-dim); }
|
||||||
|
.tp-kind-stub .tp-sub { font-size: 24px; }
|
||||||
|
|
||||||
|
.tp-cloud { fill: #dfe7ef; stroke: #8fa3b5; stroke-width: 4; }
|
||||||
|
.tp-cloud-text { font-family: var(--font); font-size: 44px; font-weight: 700; fill: #34495e; letter-spacing: .12em; }
|
||||||
|
.tp-st-down .tp-cloud { fill: #f3d6d4; stroke: #e0312b; }
|
||||||
|
.tp-modem-body { fill: #16181b; stroke: #050506; stroke-width: 3; }
|
||||||
|
.tp-modem-edge { fill: none; stroke: #2c3035; stroke-width: 3; }
|
||||||
|
.tp-modem-globe { fill: #1f6fd1; }
|
||||||
|
.tp-modem-globe-line { fill: none; stroke: #9cc6ff; stroke-width: 3; }
|
||||||
|
.tp-modem-model { font-family: var(--font); font-size: 22px; fill: #8b929b; }
|
||||||
|
.tp-led { stroke: #000; stroke-width: 2; }
|
||||||
|
.tp-led-on { fill: #2fd36b; }
|
||||||
|
.tp-led-off { fill: #3a3f45; }
|
||||||
|
.tp-led-bad { fill: #e0312b; }
|
||||||
|
.tp-ap-body { fill: #f2f3f4; stroke: #c3c6ca; stroke-width: 4; }
|
||||||
|
.tp-ap-ring { fill: none; stroke-width: 10; stroke: #6b737d; }
|
||||||
|
.tp-ap-up { stroke: #2a8cff; }
|
||||||
|
.tp-ap-down { stroke: #e0312b; }
|
||||||
|
.tp-ap-dot { fill: #d7dadd; }
|
||||||
|
.tp-strip-body { fill: #f0f1f2; stroke: #c3c6ca; stroke-width: 4; }
|
||||||
|
.tp-strip-outlet rect:first-of-type { fill: #fafafa; stroke: #b9bcc0; stroke-width: 3; }
|
||||||
|
.tp-strip-slot { fill: #2b2d31; }
|
||||||
|
.tp-strip-outlet.tp-off rect:first-of-type { fill: #d9dadc; }
|
||||||
|
.tp-box, .tp-host-box { fill: #22272e; stroke: #3a434e; stroke-width: 3; }
|
||||||
|
.tp-box-text { font-family: var(--font); font-size: 34px; fill: #8b97a5; }
|
||||||
|
.tp-host-led { stroke: #000; stroke-width: 2; }
|
||||||
|
.tp-gen-body { fill: #2a2f36; stroke: #12151a; stroke-width: 3; }
|
||||||
|
.tp-gen-model { font-family: var(--font); font-size: 30px; fill: #c9d1d9; }
|
||||||
|
.tp-gen-port { fill: #111418; stroke: #4b5562; stroke-width: 2; }
|
||||||
|
.tp-gen-up { stroke: #2fd36b; }
|
||||||
|
.tp-gen-num { font-family: var(--font); font-size: 16px; fill: #8b97a5; }
|
||||||
|
.tp-stub-body { fill: #3a3f46; stroke: #23272c; stroke-width: 3; }
|
||||||
|
.tp-stub-jack { fill: #15181c; }
|
||||||
|
.tp-stub-q { font-family: var(--font); font-size: 30px; font-weight: 700; fill: #9aa3ad; }
|
||||||
|
.tp-st-down .tp-stub-body { stroke: #e0312b; }
|
||||||
|
.tp-stub-jack.tp-off { opacity: .4; }
|
||||||
|
.tp-group-body { stroke-dasharray: 10 8; stroke: #5d6670; }
|
||||||
|
.tp-group-hint { font-family: var(--font); font-size: 20px; fill: #9aa3ad; }
|
||||||
|
.tp-select { font-size: .95em; padding: 1px 4px; margin-left: 4px; }
|
||||||
|
.tp-tag-bundle rect { stroke: #3b82f6; }
|
||||||
|
|
||||||
|
/* Host NIC sockets */
|
||||||
|
.tp-strip { fill: #1b1f24; stroke: #2f363f; stroke-width: 3; }
|
||||||
|
.tp-nic-rj45, .tp-nic-sfp { fill: #0b0d10; stroke: #59626e; stroke-width: 3; }
|
||||||
|
.tp-nic-sfp-in { fill: #23272d; }
|
||||||
|
.tp-nic-name { font-family: var(--font); font-size: 17px; fill: #aab4bf; }
|
||||||
|
.tp-nic-led { fill: #3a3f45; }
|
||||||
|
.tp-nic-up .tp-nic-led { fill: #2fd36b; }
|
||||||
|
.tp-nic-up .tp-nic-rj45, .tp-nic-up .tp-nic-sfp { stroke: #8f9aa6; }
|
||||||
|
.tp-nic-down .tp-nic-led { fill: #e0312b; }
|
||||||
|
.tp-nic-down .tp-nic-rj45, .tp-nic-down .tp-nic-sfp { stroke: #e0312b; }
|
||||||
|
.tp-nic-unused { opacity: .45; }
|
||||||
|
.tp-nic-unused .tp-nic-rj45, .tp-nic-unused .tp-nic-sfp { stroke-dasharray: 6 5; }
|
||||||
|
.tp-nic-alert-critical .tp-nic-name { fill: #ff6b5e; }
|
||||||
|
.tp-nic-alert-warning .tp-nic-name { fill: #ffb300; }
|
||||||
|
|
||||||
|
/* Cables: colours are the real jackets, the same in both themes */
|
||||||
|
.tp-link { cursor: pointer; outline: none; }
|
||||||
|
.tp-link path { fill: none; stroke-linecap: round; }
|
||||||
|
.tp-casing { stroke: #050607; }
|
||||||
|
.tp-core { stroke: #2f343b; }
|
||||||
|
.tp-hit { stroke: transparent; stroke-width: 40; }
|
||||||
|
.tp-s-10g .tp-casing { stroke-width: 17; } .tp-s-10g .tp-core { stroke-width: 11; }
|
||||||
|
.tp-s-1g .tp-casing { stroke-width: 12; } .tp-s-1g .tp-core { stroke-width: 7; }
|
||||||
|
.tp-s-100m .tp-casing { stroke-width: 8; } .tp-s-100m .tp-core { stroke-width: 4.5; }
|
||||||
|
.tp-m-dac .tp-casing { stroke: #8a929c; } .tp-m-dac .tp-core { stroke: #16181c; }
|
||||||
|
.tp-m-fibre .tp-casing { stroke: #13757c; } .tp-m-fibre .tp-core { stroke: #38c9d3; }
|
||||||
|
.tp-m-wanfibre .tp-casing { stroke: #8a6a00; } .tp-m-wanfibre .tp-core { stroke: #f2c230; }
|
||||||
|
.tp-m-sfp .tp-casing { stroke: #23272c; } .tp-m-sfp .tp-core { stroke: #69717c; }
|
||||||
|
.tp-m-copper .tp-casing { stroke: #173f80; } .tp-m-copper .tp-core { stroke: #3b82f6; }
|
||||||
|
/* A group's cables, bundled: one thick run */
|
||||||
|
.tp-link.tp-bundle .tp-casing { stroke-width: 28; stroke: #0f2b57; } .tp-link.tp-bundle .tp-core { stroke-width: 20; }
|
||||||
|
.tp-down .tp-casing { stroke: transparent; }
|
||||||
|
.tp-down .tp-core { stroke: #e0312b; stroke-dasharray: 28 18; }
|
||||||
|
.tp-ghost { opacity: .8; }
|
||||||
|
.tp-ghost .tp-casing { stroke: transparent; }
|
||||||
|
.tp-ghost .tp-core { stroke: #e0312b; stroke-dasharray: 2 18; stroke-width: 10; }
|
||||||
|
.tp-plug { fill: #c9ccd1; stroke: #5d636b; stroke-width: 2; }
|
||||||
|
.tp-boot { fill: #1e5bbf; stroke: #0d2f66; stroke-width: 2; }
|
||||||
|
.tp-links-over { pointer-events: none; }
|
||||||
|
.tp-links-over .tp-warn { pointer-events: auto; }
|
||||||
|
.tp-flow { stroke: rgba(255,255,255,.4); stroke-width: 3; stroke-dasharray: 3 60; animation: tp-flow linear infinite; pointer-events: none; }
|
||||||
|
.tp-s-100m .tp-flow { stroke-width: 2; }
|
||||||
|
@keyframes tp-flow { to { stroke-dashoffset: -63; } }
|
||||||
|
@media (prefers-reduced-motion: reduce) { .tp-flow { animation: none; display: none; } }
|
||||||
|
.tp-link:hover .tp-casing, .tp-link:focus-visible .tp-casing, .tp-selected .tp-casing { stroke: var(--amber); }
|
||||||
|
.tp-tag rect { fill: var(--bg-primary); stroke: var(--border-color); stroke-width: 2; opacity: .92; }
|
||||||
|
.tp-tag text { font-family: var(--font); font-size: 24px; fill: var(--text); }
|
||||||
|
.tp-tag-ghost rect { stroke: #e0312b; }
|
||||||
|
.tp-tag-ghost text { fill: #ff6b5e; }
|
||||||
|
.tp-rate { display: none; pointer-events: none; }
|
||||||
|
.tp-near .tp-rate, .tp-hot .tp-rate, .tp-selected .tp-rate { display: inline; }
|
||||||
|
.tp-warn path { fill: #ffb300; stroke: #5c4100; stroke-width: 3; }
|
||||||
|
.tp-warn text { font-family: var(--font); font-size: 30px; font-weight: 700; fill: #2b1f00; }
|
||||||
|
|
||||||
|
/* Filters and the traced path */
|
||||||
|
.tp-no-traffic .tp-rate, .tp-no-traffic .tp-flow { display: none !important; }
|
||||||
|
.tp-alerts-only .tp-node:not(.tp-has-alert) { opacity: .25; }
|
||||||
|
.tp-alerts-only .tp-link:not(.tp-touches-alert) { opacity: .2; }
|
||||||
|
.tp-tracing .tp-node:not(.tp-hot), .tp-tracing .tp-link:not(.tp-hot) { opacity: .18; transition: opacity .15s; }
|
||||||
|
.tp-tracing .tp-link.tp-hot .tp-casing { stroke: var(--amber); }
|
||||||
|
.tp-far .tp-sub, .tp-far .tp-nic-name { display: none; }
|
||||||
|
|
||||||
|
/* Cable panel: a side panel, a bottom sheet on phones */
|
||||||
|
.tp-panel {
|
||||||
|
position: absolute; top: 12px; right: 12px; width: min(360px, calc(100% - 24px));
|
||||||
|
max-height: calc(100% - 24px); overflow: auto; z-index: 2;
|
||||||
|
background: var(--bg-primary); border: 1px solid var(--border-color); box-shadow: 0 8px 24px rgba(0,0,0,.35);
|
||||||
|
padding: 10px 14px 14px; cursor: auto; font-size: .8em;
|
||||||
|
}
|
||||||
|
.tp-panel-head { display: flex; align-items: center; justify-content: space-between; margin-bottom: 6px; }
|
||||||
|
.tp-panel-title { font-size: 1em; margin: 0; letter-spacing: .06em; text-transform: uppercase; color: var(--amber); }
|
||||||
|
.tp-panel-table { width: 100%; border-collapse: collapse; }
|
||||||
|
.tp-panel-table th { text-align: left; font-weight: 400; color: var(--text-muted); padding: 4px 10px 4px 0; vertical-align: top; width: 42%; }
|
||||||
|
.tp-panel-table td { padding: 4px 0; overflow-wrap: anywhere; }
|
||||||
|
.tp-panel-warn { border: 1px solid var(--amber); background: var(--amber-dim); color: var(--text); padding: 6px 8px; margin: 0 0 8px; }
|
||||||
|
.tp-bad { color: var(--red); }
|
||||||
|
.tp-warn-text { color: var(--amber); }
|
||||||
|
.tp-dim-text { color: var(--text-muted); }
|
||||||
|
@media (max-width: 640px) {
|
||||||
|
.tp-panel { top: auto; left: 0; right: 0; bottom: 0; width: auto; max-height: 55%; border-width: 1px 0 0; }
|
||||||
|
.tp-updated { margin-left: 0; width: 100%; }
|
||||||
|
.tp-canvas { height: clamp(380px, calc(100vh - 260px), 900px); }
|
||||||
|
}
|
||||||
|
|
||||||
|
.tp-legend { display: flex; flex-wrap: wrap; gap: 6px 16px; margin: 10px 0 4px; font-size: .74em; color: var(--text-dim); }
|
||||||
|
.tp-key { display: inline-flex; align-items: center; gap: 6px; }
|
||||||
|
.tp-key svg { width: 34px; height: 12px; }
|
||||||
|
.tp-key-line { fill: none; stroke-width: 5; stroke-linecap: round; }
|
||||||
|
.tp-key-line.tp-m-dac { stroke: #2b2f35; }
|
||||||
|
.tp-key-line.tp-m-fibre { stroke: #38c9d3; }
|
||||||
|
.tp-key-line.tp-m-wanfibre { stroke: #f2c230; }
|
||||||
|
.tp-key-line.tp-m-sfp { stroke: #69717c; }
|
||||||
|
.tp-key-line.tp-m-copper { stroke: #3b82f6; }
|
||||||
|
.tp-key-down { stroke: #e0312b; stroke-dasharray: 6 4; }
|
||||||
|
.tp-key-ghost { stroke: #e0312b; stroke-dasharray: 1 5; }
|
||||||
|
.tp-key-warn { color: #ffb300; font-weight: 700; }
|
||||||
|
|
||||||
|
/* Topology cards */
|
||||||
|
.tp-card { fill: #121821; stroke: #2a3440; stroke-width: 3; }
|
||||||
|
[data-theme="light"] .tp-card { fill: #ffffff; stroke: #c9d1da; }
|
||||||
|
.tp-band { fill: #0a0e13; stroke: #1c242e; stroke-width: 2; }
|
||||||
|
[data-theme="light"] .tp-band { fill: #eef1f4; stroke: #dde3ea; }
|
||||||
|
.tp-card-name { font-family: var(--font); font-size: 44px; font-weight: 700; fill: var(--text); letter-spacing: .01em; }
|
||||||
|
.tp-card-sub { font-family: var(--font); font-size: 28px; fill: var(--text-muted); }
|
||||||
|
.tp-card-stripe { fill: #6b737d; }
|
||||||
|
.tp-stripe-up { fill: #2fd36b; }
|
||||||
|
.tp-stripe-degraded { fill: #ffb300; }
|
||||||
|
.tp-stripe-down { fill: #e0312b; }
|
||||||
|
.tp-node-link:hover .tp-card, .tp-node-link:focus-visible .tp-card { stroke: var(--amber); }
|
||||||
|
.tp-kind-stub .tp-card, .tp-kind-group .tp-card { stroke-dasharray: 14 10; }
|
||||||
|
.tp-kind-group .tp-node-link { cursor: pointer; }
|
||||||
|
.tp-pigtail { stroke-width: 5 !important; stroke-dasharray: none !important; }
|
||||||
|
[data-theme="light"] .tp-nic-name { fill: #4b5562; }
|
||||||
|
|
||||||
|
/* Healthy cables are quiet; problems are loud */
|
||||||
|
.tp-links .tp-link:not(.tp-problem):not(.tp-hot):not(.tp-selected) { opacity: .6; }
|
||||||
|
.tp-halo { stroke: rgba(224,49,43,.28); stroke-width: 46; fill: none; stroke-linecap: round; }
|
||||||
|
.tp-problem-mismatch .tp-halo, .tp-problem-errors .tp-halo { stroke: rgba(255,179,0,.28); }
|
||||||
|
.tp-node-problem .tp-card { stroke-width: 5; }
|
||||||
|
.tp-node-problem.tp-st-degraded .tp-card { stroke: #ffb300; }
|
||||||
|
.tp-node-problem.tp-st-down .tp-card { stroke: #e0312b; }
|
||||||
|
.tp-summary { display: flex; flex-wrap: wrap; align-items: center; gap: 6px 10px; margin: 0 0 10px; padding: 8px 12px;
|
||||||
|
border: 1px solid var(--border-color); background: var(--bg2); font-size: .8em; }
|
||||||
|
.tp-summary:empty { display: none; }
|
||||||
|
.tp-summary-head { font-weight: 700; letter-spacing: .04em; }
|
||||||
|
.tp-summary-ok { border-color: var(--green-muted); }
|
||||||
|
.tp-summary-ok .tp-summary-head { color: var(--green); }
|
||||||
|
.tp-summary-text { color: var(--text-dim); }
|
||||||
|
.tp-summary-bad { border-color: var(--red); }
|
||||||
|
.tp-summary-bad .tp-summary-head { color: var(--red); }
|
||||||
|
.tp-summary-warn { border-color: var(--amber); }
|
||||||
|
.tp-summary-warn .tp-summary-head { color: var(--amber); }
|
||||||
|
.tp-problem-chip { font: inherit; font-size: .95em; padding: 3px 9px; border: 1px solid; background: transparent; cursor: pointer; text-align: left; }
|
||||||
|
.tp-chip-bad { color: var(--red); border-color: var(--red); background: var(--red-dim); }
|
||||||
|
.tp-chip-warn { color: var(--amber); border-color: var(--amber); background: var(--amber-dim); }
|
||||||
|
.tp-problem-chip:hover, .tp-problem-chip:focus-visible { filter: brightness(1.25); outline: none; }
|
||||||
+1218
File diff suppressed because it is too large.
Load diff
+66
-1
@@ -46,6 +46,9 @@
|
|||||||
<a href="{{ url_for('rack_page') }}"
|
<a href="{{ url_for('rack_page') }}"
|
||||||
class="lt-nav-drawer-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
class="lt-nav-drawer-link{% if request.endpoint == 'rack_page' %} active{% endif %}"
|
||||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>Rack</a>
|
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>Rack</a>
|
||||||
|
<a href="{{ url_for('topology_page') }}"
|
||||||
|
class="lt-nav-drawer-link{% if request.endpoint == 'topology_page' %} active{% endif %}"
|
||||||
|
{% if request.endpoint == 'topology_page' %}aria-current="page"{% endif %}>Topology</a>
|
||||||
{% if user.groups and 'admin' in user.groups %}
|
{% if user.groups and 'admin' in user.groups %}
|
||||||
<a href="{{ url_for('suppressions_page') }}"
|
<a href="{{ url_for('suppressions_page') }}"
|
||||||
class="lt-nav-drawer-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
class="lt-nav-drawer-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
||||||
@@ -103,6 +106,11 @@
|
|||||||
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>
|
{% if request.endpoint == 'rack_page' %}aria-current="page"{% endif %}>
|
||||||
Rack
|
Rack
|
||||||
</a>
|
</a>
|
||||||
|
<a href="{{ url_for('topology_page') }}"
|
||||||
|
class="lt-nav-link{% if request.endpoint == 'topology_page' %} active{% endif %}"
|
||||||
|
{% if request.endpoint == 'topology_page' %}aria-current="page"{% endif %}>
|
||||||
|
Topology
|
||||||
|
</a>
|
||||||
{% if user.groups and 'admin' in user.groups %}
|
{% if user.groups and 'admin' in user.groups %}
|
||||||
<a href="{{ url_for('suppressions_page') }}"
|
<a href="{{ url_for('suppressions_page') }}"
|
||||||
class="lt-nav-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
class="lt-nav-link{% if request.endpoint == 'suppressions_page' %} active{% endif %}"
|
||||||
@@ -125,6 +133,10 @@
|
|||||||
<span class="lt-header-user">{{ _uname }}</span>
|
<span class="lt-header-user">{{ _uname }}</span>
|
||||||
{% if user.groups and 'admin' in user.groups %}
|
{% if user.groups and 'admin' in user.groups %}
|
||||||
<span class="lt-badge lt-badge-admin" aria-label="Administrator">ADMIN</span>
|
<span class="lt-badge lt-badge-admin" aria-label="Administrator">ADMIN</span>
|
||||||
|
{% if not maintenance %}
|
||||||
|
<button type="button" class="lt-btn lt-btn-ghost lt-btn-sm" data-action="start-maintenance"
|
||||||
|
title="Pause all tickets while you work on the network">Maintenance</button>
|
||||||
|
{% endif %}
|
||||||
{% endif %}
|
{% endif %}
|
||||||
|
|
||||||
<!-- Notification bell: shows active monitoring alerts -->
|
<!-- Notification bell: shows active monitoring alerts -->
|
||||||
@@ -184,6 +196,20 @@
|
|||||||
|
|
||||||
<!-- MAIN CONTENT -->
|
<!-- MAIN CONTENT -->
|
||||||
<main class="lt-main lt-container" id="main-content">
|
<main class="lt-main lt-container" id="main-content">
|
||||||
|
{% if maintenance %}
|
||||||
|
<div class="maint-banner" role="status">
|
||||||
|
<span class="maint-banner-text">
|
||||||
|
<strong>Maintenance:</strong> Gandalf is not opening tickets
|
||||||
|
{% if maintenance.expires_at %}until <time class="maint-until" datetime="{{ maintenance.expires_at }}">{{ maintenance.expires_at }}</time>{% else %}until it is ended{% endif %}.
|
||||||
|
Alerts are still recorded, and anything still failing afterwards gets a ticket.
|
||||||
|
<span class="maint-reason">{{ maintenance.reason }}{% if maintenance.suppressed_by %} ({{ maintenance.suppressed_by }}){% endif %}</span>
|
||||||
|
</span>
|
||||||
|
{% if user.groups and 'admin' in user.groups %}
|
||||||
|
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-action="end-maintenance"
|
||||||
|
data-sup-id="{{ maintenance.id }}">End now</button>
|
||||||
|
{% endif %}
|
||||||
|
</div>
|
||||||
|
{% endif %}
|
||||||
{% block content %}{% endblock %}
|
{% block content %}{% endblock %}
|
||||||
</main>
|
</main>
|
||||||
|
|
||||||
@@ -203,9 +229,47 @@
|
|||||||
<span class="lt-footer-sep">|</span>
|
<span class="lt-footer-sep">|</span>
|
||||||
<button type="button" class="lt-footer-hint" data-action="show-keyboard-help"><span class="lt-footer-key">[ ? ]</span> HELP</button>
|
<button type="button" class="lt-footer-hint" data-action="show-keyboard-help"><span class="lt-footer-key">[ ? ]</span> HELP</button>
|
||||||
</nav>
|
</nav>
|
||||||
<span>GANDALF — TDS v1.2</span>
|
<span>GANDALF | TDS v1.2</span>
|
||||||
</footer>
|
</footer>
|
||||||
|
|
||||||
|
{% if user.groups and 'admin' in user.groups and not maintenance %}
|
||||||
|
<!-- MAINTENANCE MODAL: pause every ticket for a while -->
|
||||||
|
<div id="maint-modal" class="lt-modal-overlay"
|
||||||
|
role="dialog" aria-modal="true" aria-labelledby="maint-modal-title" aria-hidden="true">
|
||||||
|
<div class="lt-modal">
|
||||||
|
<div class="lt-modal-header">
|
||||||
|
<h3 class="lt-modal-title" id="maint-modal-title">Start maintenance</h3>
|
||||||
|
<button type="button" class="lt-modal-close" data-modal-close aria-label="Close">✕</button>
|
||||||
|
</div>
|
||||||
|
<form id="maint-form">
|
||||||
|
<div class="lt-modal-body">
|
||||||
|
<p class="lt-field-hint maint-intro">Gandalf keeps checking and recording alerts but opens no tickets
|
||||||
|
until this ends. Anything still failing afterwards gets a ticket as usual.</p>
|
||||||
|
<div class="lt-form-group">
|
||||||
|
<label class="lt-label" for="maint-reason">Reason <span class="required">*</span></label>
|
||||||
|
<input type="text" class="lt-input" id="maint-reason" name="reason" maxlength="500"
|
||||||
|
value="Maintenance" required aria-required="true"
|
||||||
|
placeholder="e.g. UniFi Network update">
|
||||||
|
</div>
|
||||||
|
<div class="lt-form-group lt-form-group--last">
|
||||||
|
<span class="lt-label" id="maint-len-label">How long</span>
|
||||||
|
<div class="duration-pills" role="group" aria-labelledby="maint-len-label">
|
||||||
|
<button type="button" class="maint-pill" data-minutes="30" aria-pressed="false">30 min</button>
|
||||||
|
<button type="button" class="maint-pill active" data-minutes="60" aria-pressed="true">1 hr</button>
|
||||||
|
<button type="button" class="maint-pill" data-minutes="120" aria-pressed="false">2 hr</button>
|
||||||
|
<button type="button" class="maint-pill" data-minutes="240" aria-pressed="false">4 hr</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div class="lt-modal-footer">
|
||||||
|
<button type="button" class="lt-btn lt-btn-secondary" data-modal-close>Cancel</button>
|
||||||
|
<button type="submit" class="lt-btn lt-btn-primary">Start</button>
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
{% endif %}
|
||||||
|
|
||||||
<!-- QUICK-SUPPRESS MODAL: available on all pages via [S] shortcut -->
|
<!-- QUICK-SUPPRESS MODAL: available on all pages via [S] shortcut -->
|
||||||
<div id="suppress-modal" class="lt-modal-overlay"
|
<div id="suppress-modal" class="lt-modal-overlay"
|
||||||
role="dialog" aria-modal="true" aria-labelledby="suppress-modal-title" aria-hidden="true">
|
role="dialog" aria-modal="true" aria-labelledby="suppress-modal-title" aria-hidden="true">
|
||||||
@@ -343,6 +407,7 @@
|
|||||||
{ id: 'nav-history', group: 'Navigate', icon: '⏱', label: 'Alert History', action: function() { window.location.href = '/events'; } },
|
{ id: 'nav-history', group: 'Navigate', icon: '⏱', label: 'Alert History', action: function() { window.location.href = '/events'; } },
|
||||||
{ id: 'nav-cutsheet', group: 'Navigate', icon: '🔌', label: 'Cutsheet', action: function() { window.location.href = '/cutsheet'; } },
|
{ id: 'nav-cutsheet', group: 'Navigate', icon: '🔌', label: 'Cutsheet', action: function() { window.location.href = '/cutsheet'; } },
|
||||||
{ id: 'nav-rack', group: 'Navigate', icon: '🗄', label: 'Rack', action: function() { window.location.href = '/rack'; } },
|
{ id: 'nav-rack', group: 'Navigate', icon: '🗄', label: 'Rack', action: function() { window.location.href = '/rack'; } },
|
||||||
|
{ id: 'nav-topology', group: 'Navigate', icon: '⛓', label: 'Topology', action: function() { window.location.href = '/topology'; } },
|
||||||
{ id: 'nav-suppressions', group: 'Navigate', icon: '🔕', label: 'Suppressions', action: function() { window.location.href = '/suppressions'; } },
|
{ id: 'nav-suppressions', group: 'Navigate', icon: '🔕', label: 'Suppressions', action: function() { window.location.href = '/suppressions'; } },
|
||||||
{ id: 'action-refresh', group: 'Actions', icon: '↻', label: 'Refresh Data', kbd: 'R', action: function() { lt.autoRefresh.now(); } },
|
{ id: 'action-refresh', group: 'Actions', icon: '↻', label: 'Refresh Data', kbd: 'R', action: function() { lt.autoRefresh.now(); } },
|
||||||
{ id: 'action-suppress', group: 'Actions', icon: '🔕', label: 'New Suppression', kbd: 'S', action: function() { if (typeof openSuppressModal === 'function') openSuppressModal('host','',''); else window.location.href='/suppressions'; } },
|
{ id: 'action-suppress', group: 'Actions', icon: '🔕', label: 'New Suppression', kbd: 'S', action: function() { if (typeof openSuppressModal === 'function') openSuppressModal('host','',''); else window.location.href='/suppressions'; } },
|
||||||
|
|||||||
+39
-200
@@ -18,6 +18,10 @@
|
|||||||
{% if daemon_ok and not summary.critical and not summary.warning %}
|
{% if daemon_ok and not summary.critical and not summary.warning %}
|
||||||
<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>
|
<span class="chip chip-ok">✔ ALL SYSTEMS NOMINAL</span>
|
||||||
{% endif %}
|
{% endif %}
|
||||||
|
{% set followups = (stale_tickets | length) + (stale_alerts or 0) %}
|
||||||
|
{% if followups %}
|
||||||
|
<a class="chip chip-warning chip-followup" href="{{ '#followup' if stale_tickets else '#events-table-wrap' }}">⏳ {{ followups }} TO FOLLOW UP</a>
|
||||||
|
{% endif %}
|
||||||
</div>
|
</div>
|
||||||
<div class="status-meta">
|
<div class="status-meta">
|
||||||
<span class="last-check" id="last-check" data-tooltip="Last monitor poll timestamp" data-tooltip-pos="bottom">{{ last_check }}</span>
|
<span class="last-check" id="last-check" data-tooltip="Last monitor poll timestamp" data-tooltip-pos="bottom">{{ last_check }}</span>
|
||||||
@@ -129,6 +133,8 @@
|
|||||||
<td>
|
<td>
|
||||||
<span class="lt-badge badge-{{ e.severity }}">{{ e.severity }}</span>
|
<span class="lt-badge badge-{{ e.severity }}">{{ e.severity }}</span>
|
||||||
{% if e.is_suppressed %}<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>{% endif %}
|
{% if e.is_suppressed %}<span class="lt-badge badge-suppressed" title="Alert suppressed">🔕 sup</span>{% endif %}
|
||||||
|
{% if e.stale_days is not none %}<span class="lt-badge badge-stale" title="Active for {{ e.stale_days }} days with no fix yet">⏳ {{ e.stale_days }}d</span>{% endif %}
|
||||||
|
{% if e.ticket_closed %}<span class="lt-badge badge-ticket-closed" title="Ticket #{{ e.ticket_id }} was closed in Tinker, but this alert is still active">ticket closed</span>{% endif %}
|
||||||
</td>
|
</td>
|
||||||
<td>{{ e.event_type | replace('_', ' ') }}</td>
|
<td>{{ e.event_type | replace('_', ' ') }}</td>
|
||||||
<td><strong>{{ e.target_name }}</strong></td>
|
<td><strong>{{ e.target_name }}</strong></td>
|
||||||
@@ -177,6 +183,39 @@
|
|||||||
</div><!-- /.lt-frame -->
|
</div><!-- /.lt-frame -->
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<!-- ── Needs follow-up: tickets left open after their alert cleared ── -->
|
||||||
|
<section class="g-section" id="followup"{% if not stale_tickets %} hidden{% endif %}>
|
||||||
|
<div class="g-section-header">
|
||||||
|
<h2 class="g-section-title">Needs Follow-up</h2>
|
||||||
|
<span class="g-section-badge" id="followup-badge">{{ stale_tickets | length }}</span>
|
||||||
|
</div>
|
||||||
|
<p class="followup-intro">These tickets are still open in Tinker, but their alert cleared over a day ago. Close them if the problem is fixed.</p>
|
||||||
|
<div class="lt-frame">
|
||||||
|
<span class="lt-frame-bl">╚</span>
|
||||||
|
<span class="lt-frame-br">╝</span>
|
||||||
|
<div class="lt-section-header">Tickets Left Open</div>
|
||||||
|
<div class="lt-table-wrap">
|
||||||
|
<table class="lt-table" id="followup-table">
|
||||||
|
<caption class="lt-sr-only">Tickets still open after their alert cleared</caption>
|
||||||
|
<thead>
|
||||||
|
<tr><th>Ticket</th><th>Target</th><th>Alert</th><th>Cleared</th><th>Alerts on ticket</th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{% for t in stale_tickets %}
|
||||||
|
<tr>
|
||||||
|
<td><a href="{{ config.ticket_api.web_url }}{{ t.ticket_id }}" target="_blank" rel="noopener" class="ticket-link">#{{ t.ticket_id }}</a></td>
|
||||||
|
<td><strong>{{ t.target_name }}</strong></td>
|
||||||
|
<td>{{ t.event_type | replace('_', ' ') }}{% if t.target_detail %} <span class="lt-text-muted">{{ t.target_detail }}</span>{% endif %}</td>
|
||||||
|
<td class="ts-cell" title="{{ t.cleared_at }}"><span class="event-age" data-ts="{{ t.cleared_at }}">{{ t.cleared_at }}</span></td>
|
||||||
|
<td>{{ t.events }}</td>
|
||||||
|
</tr>
|
||||||
|
{% endfor %}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
<!-- ── WAN / fiber ──────────────────────────────────────────────────── -->
|
<!-- ── WAN / fiber ──────────────────────────────────────────────────── -->
|
||||||
{% include "_modem_panel.html" %}
|
{% include "_modem_panel.html" %}
|
||||||
|
|
||||||
@@ -296,176 +335,6 @@
|
|||||||
</section>
|
</section>
|
||||||
{% endif %}
|
{% endif %}
|
||||||
|
|
||||||
<!-- ── Network topology ───────────────────────────────── -->
|
|
||||||
<section class="g-section">
|
|
||||||
<div class="g-section-header">
|
|
||||||
<h2 class="g-section-title">Network Topology</h2>
|
|
||||||
<button type="button" class="topo-collapse-btn" id="topo-toggle-btn"
|
|
||||||
aria-expanded="true" aria-controls="topo-collapsible-wrap">▴ Collapse</button>
|
|
||||||
</div>
|
|
||||||
<div class="topo-collapsible" id="topo-collapsible-wrap">
|
|
||||||
|
|
||||||
<div class="topology" id="topology-diagram">
|
|
||||||
<div class="topo-v2">
|
|
||||||
|
|
||||||
{%- set topo_h = snapshot.hosts if snapshot.hosts else {} -%}
|
|
||||||
|
|
||||||
<!-- ══════════════════════════════════════════════════════════════
|
|
||||||
TIER 1: Internet (WAN edge)
|
|
||||||
══════════════════════════════════════════════════════════ -->
|
|
||||||
<div class="topo-tier">
|
|
||||||
<div class="topo-v2-node topo-v2-internet">
|
|
||||||
<span class="topo-v2-icon">◈</span>
|
|
||||||
<span class="topo-v2-label">INTERNET</span>
|
|
||||||
<span class="topo-v2-sub">WAN uplink</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- Fiber wire: Internet → AT&T gateway -->
|
|
||||||
<div class="topo-vc">
|
|
||||||
<div class="topo-vc-wire topo-vc-wire--wan"></div>
|
|
||||||
<span class="topo-vc-label">AT&T Fiber</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ══════════════════════════════════════════════════════════════
|
|
||||||
TIER 1b: AT&T gateway (Rx filled in by updateModem)
|
|
||||||
══════════════════════════════════════════════════════════ -->
|
|
||||||
<div class="topo-tier">
|
|
||||||
<div class="topo-v2-node topo-v2-router" id="topo-modem">
|
|
||||||
<span class="topo-v2-icon">◈</span>
|
|
||||||
<span class="topo-v2-label">BGW320</span>
|
|
||||||
<span class="topo-v2-sub">AT&T Gateway</span>
|
|
||||||
<span class="topo-v2-sub" id="topo-modem-rx">Rx -</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- WAN wire: gateway → UDM-Pro -->
|
|
||||||
<div class="topo-vc">
|
|
||||||
<div class="topo-vc-wire topo-vc-wire--wan"></div>
|
|
||||||
<span class="topo-vc-label">WAN · Port 9 · 1G</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ══════════════════════════════════════════════════════════════
|
|
||||||
TIER 2: Router, UDM-Pro
|
|
||||||
══════════════════════════════════════════════════════════ -->
|
|
||||||
<div class="topo-tier">
|
|
||||||
<div class="topo-v2-node topo-v2-router">
|
|
||||||
<span class="topo-v2-icon">⬡</span>
|
|
||||||
<span class="topo-v2-label">UDM-Pro</span>
|
|
||||||
<span class="topo-v2-sub">Dream Machine Pro</span>
|
|
||||||
<span class="topo-v2-sub">RU24</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- UDM-Pro → USW-Agg (10G SFP+) -->
|
|
||||||
<div class="topo-vc">
|
|
||||||
<div class="topo-vc-wire topo-vc-wire--10g"></div>
|
|
||||||
<span class="topo-vc-label">10G SFP+</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ══════════════════════════════════════════════════════════════
|
|
||||||
TIER 3: USW-Aggregation
|
|
||||||
══════════════════════════════════════════════════════════ -->
|
|
||||||
<div class="topo-tier">
|
|
||||||
<div class="topo-v2-node topo-v2-switch" id="topo-switch-agg">
|
|
||||||
<span class="topo-v2-icon">⬡</span>
|
|
||||||
<span class="topo-v2-label">USW-Agg</span>
|
|
||||||
<span class="topo-v2-sub">Aggregation · RU22</span>
|
|
||||||
<span class="topo-v2-sub">8 × 10G SFP+</span>
|
|
||||||
<span class="topo-v2-vlan">VLAN90 · 10.10.90.x/24</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- USW-Agg → Pro 24 PoE (10G trunk) -->
|
|
||||||
<div class="topo-vc">
|
|
||||||
<div class="topo-vc-wire topo-vc-wire--10g"></div>
|
|
||||||
<span class="topo-vc-label">10G trunk</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ══════════════════════════════════════════════════════════════
|
|
||||||
TIER 4: Pro 24 PoE
|
|
||||||
══════════════════════════════════════════════════════════ -->
|
|
||||||
<div class="topo-tier">
|
|
||||||
<div class="topo-v2-node topo-v2-switch" id="topo-switch-poe">
|
|
||||||
<span class="topo-v2-icon">⬡</span>
|
|
||||||
<span class="topo-v2-label">Pro 24 PoE</span>
|
|
||||||
<span class="topo-v2-sub">24-Port · RU23</span>
|
|
||||||
<span class="topo-v2-sub">24 × 1G PoE</span>
|
|
||||||
<span class="topo-v2-vlan">DHCP · mgmt</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- Pro 24 PoE → host bus section -->
|
|
||||||
<div class="topo-vc">
|
|
||||||
<div class="topo-vc-wire topo-vc-wire--mgmt"></div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ══════════════════════════════════════════════════════════════
|
|
||||||
TIER 4 connecting bus: two rails (10G green + 1G amber dashed)
|
|
||||||
showing dual-homing for all 6 servers
|
|
||||||
══════════════════════════════════════════════════════════ -->
|
|
||||||
<div class="topo-bus-section">
|
|
||||||
|
|
||||||
<!-- 10G storage bus (Agg → VLAN90) -->
|
|
||||||
<div class="topo-bus-10g">
|
|
||||||
<span class="topo-bus-10g-label">← USW-Agg · 10G SFP+ · VLAN90 →</span>
|
|
||||||
<div class="topo-bus-10g-line"></div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- 1G management bus (PoE → DHCP) -->
|
|
||||||
<div class="topo-bus-1g">
|
|
||||||
<span class="topo-bus-1g-label">← Pro 24 PoE · 1G · DHCP mgmt →</span>
|
|
||||||
<div class="topo-bus-1g-line"></div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- ── Host nodes with drop wires ── -->
|
|
||||||
<div class="topo-v2-hosts">
|
|
||||||
{%- set all_defs = [
|
|
||||||
('compute-storage-gpu-01', 'csg-01', 'RU4-12', 'Ceph · VLAN90', False),
|
|
||||||
('compute-storage-01', 'cs-01', 'RU14-17', 'Ceph · VLAN90', False),
|
|
||||||
('storage-01', 'sto-01', 'rack', 'Ceph · VLAN90', False),
|
|
||||||
('monitor-01', 'mon-01', 'ZimaBoard', 'mgmt', False),
|
|
||||||
('monitor-02', 'mon-02', 'ZimaBoard', 'mgmt', False),
|
|
||||||
('large1', 'large1', 'off-rack', 'table', True),
|
|
||||||
] -%}
|
|
||||||
{%- for hname, hlabel, hsub, hvlan, off_rack in all_defs -%}
|
|
||||||
{%- set st = topo_h[hname].status if hname in topo_h else 'unknown' -%}
|
|
||||||
<div class="topo-v2-host-wrap">
|
|
||||||
<!-- dual-homing wires: 10G solid green + 1G dashed amber -->
|
|
||||||
<div class="topo-v2-host-wires">
|
|
||||||
<div class="topo-v2-wire-10g" data-host="{{ hname }}" title="10G SFP+ → USW-Agg"></div>
|
|
||||||
<div class="topo-v2-wire-1g" data-host="{{ hname }}" title="1G → Pro 24 PoE"></div>
|
|
||||||
</div>
|
|
||||||
<!-- host box -->
|
|
||||||
<div class="topo-v2-node topo-v2-host topo-host topo-v2-status-{{ st }}{{ ' topo-v2-offrack' if off_rack else '' }} topo-v2-host--bus"
|
|
||||||
data-host="{{ hname }}">
|
|
||||||
<span class="topo-v2-icon">▣</span>
|
|
||||||
<span class="topo-v2-label">{{ hlabel }}</span>
|
|
||||||
<span class="topo-v2-sub">{{ hsub }}</span>
|
|
||||||
<span class="topo-v2-vlan">{{ hvlan }}</span>
|
|
||||||
<span class="topo-badge topo-badge-{{ st }}">{{ st if st != 'unknown' else '-' }}</span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
{%- endfor -%}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
</div><!-- /topo-bus-section -->
|
|
||||||
|
|
||||||
<!-- ── Legend ── -->
|
|
||||||
<div class="topo-legend">
|
|
||||||
<div class="topo-legend-item"><span class="topo-legend-line-wan"></span> WAN / uplink</div>
|
|
||||||
<div class="topo-legend-item"><span class="topo-legend-line-10g"></span> 10G SFP+ (Ceph / VLAN90)</div>
|
|
||||||
<div class="topo-legend-item"><span class="topo-legend-line-1g"></span> 1G DHCP (mgmt)</div>
|
|
||||||
<div class="topo-legend-item topo-legend-item--offrack">dashed border = off-rack</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
</div><!-- /topo-v2 -->
|
|
||||||
</div>
|
|
||||||
|
|
||||||
</div><!-- /#topo-collapsible-wrap -->
|
|
||||||
</section>
|
|
||||||
|
|
||||||
|
|
||||||
<!-- ── Recently Resolved (last 24h) ───────────────────────────────── -->
|
<!-- ── Recently Resolved (last 24h) ───────────────────────────────── -->
|
||||||
{% if recent_resolved %}
|
{% if recent_resolved %}
|
||||||
<section class="g-section">
|
<section class="g-section">
|
||||||
@@ -512,30 +381,6 @@
|
|||||||
if (s.refreshInterval > 0) lt.autoRefresh.start(refreshAll, s.refreshInterval * 1000);
|
if (s.refreshInterval > 0) lt.autoRefresh.start(refreshAll, s.refreshInterval * 1000);
|
||||||
};
|
};
|
||||||
|
|
||||||
// ── Topology collapse toggle ───────────────────────────────────
|
|
||||||
(function() {
|
|
||||||
const LS_KEY = 'gandalf_topo_collapsed';
|
|
||||||
const btn = document.getElementById('topo-toggle-btn');
|
|
||||||
const wrap = document.getElementById('topo-collapsible-wrap');
|
|
||||||
if (!btn || !wrap) return;
|
|
||||||
|
|
||||||
function setCollapsed(v) {
|
|
||||||
wrap.classList.toggle('is-collapsed', v);
|
|
||||||
wrap.setAttribute('aria-hidden', v ? 'true' : 'false');
|
|
||||||
btn.setAttribute('aria-expanded', v ? 'false' : 'true');
|
|
||||||
btn.textContent = v ? '▾ Expand' : '▴ Collapse';
|
|
||||||
try { localStorage.setItem(LS_KEY, v ? '1' : '0'); } catch(_) {}
|
|
||||||
}
|
|
||||||
|
|
||||||
let saved = false;
|
|
||||||
try { saved = localStorage.getItem(LS_KEY) === '1'; } catch(_) {}
|
|
||||||
setCollapsed(saved);
|
|
||||||
|
|
||||||
btn.addEventListener('click', function() {
|
|
||||||
setCollapsed(!wrap.classList.contains('is-collapsed'));
|
|
||||||
});
|
|
||||||
})();
|
|
||||||
|
|
||||||
function updateEventAges() {
|
function updateEventAges() {
|
||||||
document.querySelectorAll('.event-age[data-ts]').forEach(el => {
|
document.querySelectorAll('.event-age[data-ts]').forEach(el => {
|
||||||
el.textContent = lt.time.ago(_toIso(el.dataset.ts));
|
el.textContent = lt.time.ago(_toIso(el.dataset.ts));
|
||||||
@@ -596,12 +441,6 @@
|
|||||||
|
|
||||||
initDeviceSearch();
|
initDeviceSearch();
|
||||||
|
|
||||||
// Topology hosts open their device page
|
|
||||||
document.getElementById('topology-diagram')?.addEventListener('click', e => {
|
|
||||||
const node = e.target.closest('.topo-host[data-host]');
|
|
||||||
if (node) window.location.href = deviceUrl(node.dataset.host);
|
|
||||||
});
|
|
||||||
|
|
||||||
// Stat card clicks: filter events table by severity
|
// Stat card clicks: filter events table by severity
|
||||||
document.querySelectorAll('.lt-stat-card[data-stat-filter]').forEach(card => {
|
document.querySelectorAll('.lt-stat-card[data-stat-filter]').forEach(card => {
|
||||||
card.addEventListener('click', () => {
|
card.addEventListener('click', () => {
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
{% extends "base.html" %}
|
||||||
|
{% block title %}Topology{% endblock %}
|
||||||
|
|
||||||
|
{% block content %}
|
||||||
|
<div class="lt-page-header">
|
||||||
|
<div>
|
||||||
|
<h1 class="lt-page-title">Topology</h1>
|
||||||
|
<p class="g-page-sub">Every device and cable as the switches see them right now. Drag to pan, scroll or pinch to zoom. Hover a device to trace its path to the internet; click a cable for its details, or a group of unknown devices to open it.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="tp-toolbar no-print" role="toolbar" aria-label="Topology view">
|
||||||
|
<div class="tp-filters" role="group" aria-label="Show">
|
||||||
|
<label class="tp-filter">Unknown devices
|
||||||
|
<select class="lt-select tp-select" data-filter="stubs" aria-label="Unknown devices">
|
||||||
|
<option value="grouped">grouped per switch</option>
|
||||||
|
<option value="all">one by one</option>
|
||||||
|
<option value="none">hidden</option>
|
||||||
|
</select></label>
|
||||||
|
<label class="tp-filter"><input type="checkbox" data-filter="power"> Power gear</label>
|
||||||
|
<label class="tp-filter"><input type="checkbox" data-filter="traffic"> Traffic</label>
|
||||||
|
<label class="tp-filter" title="Each server's slower second cable. A problem cable is always shown."><input type="checkbox" data-filter="mgmt"> Server 1G links</label>
|
||||||
|
<label class="tp-filter"><input type="checkbox" data-filter="alerts"> Alerts only</label>
|
||||||
|
</div>
|
||||||
|
<div class="tp-zoom" role="group" aria-label="Zoom">
|
||||||
|
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-zoom="out" aria-label="Zoom out" title="Zoom out (-)">−</button>
|
||||||
|
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-zoom="fit" title="Fit everything (0)">Fit</button>
|
||||||
|
<button type="button" class="lt-btn lt-btn-secondary lt-btn-sm" data-zoom="in" aria-label="Zoom in" title="Zoom in (+)">+</button>
|
||||||
|
</div>
|
||||||
|
<span class="tp-updated" id="tp-updated" aria-live="polite"></span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="tp-warnings" id="tp-warnings" role="status" hidden></div>
|
||||||
|
<div class="tp-summary" id="tp-summary" aria-live="polite"></div>
|
||||||
|
|
||||||
|
<div class="tp-canvas" id="tp-canvas" tabindex="0" aria-label="Network topology diagram. Use plus, minus and zero to zoom, and Tab to move between devices and cables.">
|
||||||
|
<svg class="tp-svg" id="tp-svg" xmlns="http://www.w3.org/2000/svg" role="group" aria-label="Network topology">
|
||||||
|
<g id="tp-world"></g>
|
||||||
|
</svg>
|
||||||
|
<div class="link-loading tp-loading" id="tp-loading">Loading topology</div>
|
||||||
|
<aside class="tp-panel" id="tp-panel" hidden aria-live="polite" aria-label="Cable details"></aside>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="tp-legend" aria-label="Legend">
|
||||||
|
<span class="tp-key"><svg viewBox="0 0 40 12" aria-hidden="true"><path class="tp-key-line tp-m-dac" d="M2 6 H38"/></svg>DAC</span>
|
||||||
|
<span class="tp-key"><svg viewBox="0 0 40 12" aria-hidden="true"><path class="tp-key-line tp-m-fibre" d="M2 6 H38"/></svg>Fibre (multimode)</span>
|
||||||
|
<span class="tp-key"><svg viewBox="0 0 40 12" aria-hidden="true"><path class="tp-key-line tp-m-wanfibre" d="M2 6 H38"/></svg>Fibre (provider)</span>
|
||||||
|
<span class="tp-key"><svg viewBox="0 0 40 12" aria-hidden="true"><path class="tp-key-line tp-m-sfp" d="M2 6 H38"/></svg>SFP+ module</span>
|
||||||
|
<span class="tp-key"><svg viewBox="0 0 40 12" aria-hidden="true"><path class="tp-key-line tp-m-copper" d="M2 6 H38"/></svg>Copper</span>
|
||||||
|
<span class="tp-key">Thicker = faster (10G, 1G, 100M)</span>
|
||||||
|
<span class="tp-key"><svg viewBox="0 0 40 12" aria-hidden="true"><path class="tp-key-line tp-key-down" d="M2 6 H38"/></svg>Down</span>
|
||||||
|
<span class="tp-key"><svg viewBox="0 0 40 12" aria-hidden="true"><path class="tp-key-line tp-key-ghost" d="M2 6 H38"/></svg>Last seen</span>
|
||||||
|
<span class="tp-key"><span class="tp-key-warn" aria-hidden="true">⚠</span>Port name points at another device</span>
|
||||||
|
</div>
|
||||||
|
{% endblock %}
|
||||||
|
|
||||||
|
{% block scripts %}
|
||||||
|
<script src="{{ url_for('static', filename='faceplates.js') }}"></script>
|
||||||
|
<script src="{{ url_for('static', filename='fronts.js') }}"></script>
|
||||||
|
<script src="{{ url_for('static', filename='topology.js') }}"></script>
|
||||||
|
{% endblock %}
|
||||||
+163
-3
@@ -1,5 +1,7 @@
|
|||||||
"""Tests for app.py routes: database and LDAP are mocked."""
|
"""Tests for app.py routes: database and LDAP are mocked."""
|
||||||
import json
|
import json
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
@@ -37,6 +39,7 @@ def client():
|
|||||||
db.check_suppressed.return_value = False
|
db.check_suppressed.return_value = False
|
||||||
db.get_device_link.return_value = ''
|
db.get_device_link.return_value = ''
|
||||||
db.cutsheet_racks.return_value = []
|
db.cutsheet_racks.return_value = []
|
||||||
|
db.get_unused_interfaces.return_value = {}
|
||||||
with patch.object(gandalf, '_cfg', CONFIG):
|
with patch.object(gandalf, '_cfg', CONFIG):
|
||||||
yield gandalf.app.test_client()
|
yield gandalf.app.test_client()
|
||||||
|
|
||||||
@@ -93,10 +96,15 @@ class TestPagesAndNav:
|
|||||||
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||||
order = [html.index(m) for m in (
|
order = [html.index(m) for m in (
|
||||||
'id="device-search"', 'Alert Queue', 'WAN / Fiber', 'g-section-title">Hosts<',
|
'id="device-search"', 'Alert Queue', 'WAN / Fiber', 'g-section-title">Hosts<',
|
||||||
'Device Inventory', 'Network Topology',
|
'Device Inventory',
|
||||||
)]
|
)]
|
||||||
assert order == sorted(order)
|
assert order == sorted(order)
|
||||||
|
|
||||||
|
def test_dashboard_topology_moved_to_its_own_page(self, client):
|
||||||
|
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||||
|
assert 'topology-diagram' not in html and 'Network Topology' not in html
|
||||||
|
assert 'href="/topology"' in html
|
||||||
|
|
||||||
def test_old_links_url_redirects_home(self, client):
|
def test_old_links_url_redirects_home(self, client):
|
||||||
resp = client.get('/links', headers=AUTH)
|
resp = client.get('/links', headers=AUTH)
|
||||||
assert resp.status_code == 302 and resp.headers['Location'].endswith('/')
|
assert resp.status_code == 302 and resp.headers['Location'].endswith('/')
|
||||||
@@ -382,7 +390,8 @@ class TestHealth:
|
|||||||
|
|
||||||
class TestPageTitles:
|
class TestPageTitles:
|
||||||
@pytest.mark.parametrize('path,title', [('/', 'Dashboard | GANDALF'), ('/events', 'Alert History | GANDALF'),
|
@pytest.mark.parametrize('path,title', [('/', 'Dashboard | GANDALF'), ('/events', 'Alert History | GANDALF'),
|
||||||
('/cutsheet', 'Cutsheet | GANDALF')])
|
('/cutsheet', 'Cutsheet | GANDALF'),
|
||||||
|
('/topology', 'Topology | GANDALF')])
|
||||||
def test_name_appears_once(self, client, path, title):
|
def test_name_appears_once(self, client, path, title):
|
||||||
gandalf._cutsheet_schema_ready = True
|
gandalf._cutsheet_schema_ready = True
|
||||||
html = client.get(path, headers=AUTH).get_data(as_text=True)
|
html = client.get(path, headers=AUTH).get_data(as_text=True)
|
||||||
@@ -551,7 +560,7 @@ class TestRackFacesAndSettings:
|
|||||||
things = data['items'][0]['contents']
|
things = data['items'][0]['contents']
|
||||||
assert [(c['name'], c['slot'], c['chassis']) for c in things] == [('b-box', 1, 'zimaboard-832'),
|
assert [(c['name'], c['slot'], c['chassis']) for c in things] == [('b-box', 1, 'zimaboard-832'),
|
||||||
('large1', 2, '')]
|
('large1', 2, '')]
|
||||||
assert things[1]['interfaces'] == [{'name': 'eth0', 'up': True}]
|
assert things[1]['interfaces'] == [{'name': 'eth0', 'up': True, 'unused': False}]
|
||||||
assert things[0]['device'] == 'b-box' and things[0]['ports'] == {}
|
assert things[0]['device'] == 'b-box' and things[0]['ports'] == {}
|
||||||
|
|
||||||
def test_saved_racks_replace_config(self, client):
|
def test_saved_racks_replace_config(self, client):
|
||||||
@@ -583,3 +592,154 @@ class TestRackFacesAndSettings:
|
|||||||
assert client.put('/api/cutsheet/racks/1', json={'name': 'x', 'units': 2}, headers=user).status_code == 403
|
assert client.put('/api/cutsheet/racks/1', json={'name': 'x', 'units': 2}, headers=user).status_code == 403
|
||||||
gandalf.db.cutsheet_delete_rack.return_value = False
|
gandalf.db.cutsheet_delete_rack.return_value = False
|
||||||
assert client.delete('/api/cutsheet/racks/9', headers=AUTH).status_code == 404
|
assert client.delete('/api/cutsheet/racks/9', headers=AUTH).status_code == 404
|
||||||
|
|
||||||
|
|
||||||
|
class TestUnusedInterfaces:
|
||||||
|
def _url(self, iface='nic1'):
|
||||||
|
gandalf._cutsheet_schema_ready = True
|
||||||
|
return f'/api/device/compute-storage-01/interfaces/{iface}/unused'
|
||||||
|
|
||||||
|
def test_mark_and_clear(self, client):
|
||||||
|
assert client.put(self._url(), json={'reason': ' never cabled '}, headers=AUTH).status_code == 200
|
||||||
|
gandalf.db.set_unused_interface.assert_called_with('compute-storage-01', 'nic1', 'never cabled', 'jared')
|
||||||
|
gandalf.db.clear_unused_interface.return_value = True
|
||||||
|
resp = client.delete(self._url(), headers=AUTH)
|
||||||
|
assert resp.get_json() == {'success': True, 'changed': True}
|
||||||
|
|
||||||
|
def test_needs_admin_json_and_a_sane_name(self, client):
|
||||||
|
user = dict(AUTH, **{'Remote-Groups': 'staff'})
|
||||||
|
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
|
||||||
|
assert client.put(self._url(), json={}, headers=user).status_code == 403
|
||||||
|
assert client.put(self._url(), data='x', headers=AUTH).status_code == 400
|
||||||
|
assert client.put(self._url('bad name!'), json={}, headers=AUTH).status_code in (400, 404)
|
||||||
|
|
||||||
|
def test_device_api_lists_them(self, client):
|
||||||
|
gandalf._cutsheet_schema_ready = True
|
||||||
|
gandalf.db.get_unused_interfaces.return_value = {'large1': {'enp36s0': {'reason': 'spare'}}}
|
||||||
|
data = client.get('/api/device/large1', headers=AUTH).get_json()
|
||||||
|
assert data['unused_interfaces'] == {'enp36s0': {'reason': 'spare'}}
|
||||||
|
|
||||||
|
|
||||||
|
class TestMaintenance:
|
||||||
|
SUP = {'id': 7, 'target_type': 'all', 'target_name': '', 'reason': 'UniFi Network update',
|
||||||
|
'suppressed_by': 'jared', 'expires_at': '2026-11-12T20:00:00Z'}
|
||||||
|
|
||||||
|
def test_no_banner_and_a_start_button_by_default(self, client):
|
||||||
|
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||||
|
assert 'maint-banner' not in html
|
||||||
|
assert 'data-action="start-maintenance"' in html and 'id="maint-modal"' in html
|
||||||
|
|
||||||
|
def test_banner_on_every_page_while_on(self, client):
|
||||||
|
gandalf.db.get_active_suppressions.return_value = [
|
||||||
|
{'id': 3, 'target_type': 'host', 'target_name': 'large1', 'reason': 'x', 'expires_at': None},
|
||||||
|
self.SUP,
|
||||||
|
]
|
||||||
|
for page in ('/', '/events', '/rack'):
|
||||||
|
html = client.get(page, headers=AUTH).get_data(as_text=True)
|
||||||
|
assert 'class="maint-banner"' in html, page
|
||||||
|
assert 'datetime="2026-11-12T20:00:00Z"' in html and 'UniFi Network update' in html
|
||||||
|
assert 'data-sup-id="7"' in html and 'data-action="start-maintenance"' not in html
|
||||||
|
|
||||||
|
def test_open_ended_maintenance_wins(self, client):
|
||||||
|
gandalf.db.get_active_suppressions.return_value = [
|
||||||
|
self.SUP, dict(self.SUP, id=9, expires_at=None, reason='Rewiring')]
|
||||||
|
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||||
|
assert 'until it is ended' in html and 'data-sup-id="9"' in html
|
||||||
|
|
||||||
|
def test_viewers_see_the_banner_without_buttons(self, client):
|
||||||
|
gandalf.db.get_active_suppressions.return_value = [self.SUP]
|
||||||
|
viewer = dict(AUTH, **{'Remote-Groups': 'staff'})
|
||||||
|
with patch.object(gandalf, '_cfg', dict(CONFIG, auth=dict(CONFIG['auth'], allowed_groups=['admin', 'staff']))):
|
||||||
|
html = client.get('/', headers=viewer).get_data(as_text=True)
|
||||||
|
assert 'class="maint-banner"' in html and 'data-action="end-maintenance"' not in html
|
||||||
|
|
||||||
|
def test_a_db_error_means_no_banner(self, client):
|
||||||
|
gandalf.db.get_active_suppressions.side_effect = RuntimeError('db down')
|
||||||
|
assert gandalf.inject_maintenance() == {'maintenance': None}
|
||||||
|
|
||||||
|
|
||||||
|
class TestFollowUp:
|
||||||
|
def _state(self, tickets, age_s=60):
|
||||||
|
state = dict(STATE, open_tickets=json.dumps({'checked_at': time.time() - age_s, 'tickets': tickets}))
|
||||||
|
gandalf.db.get_state.side_effect = lambda key, default=None: state.get(key, default)
|
||||||
|
|
||||||
|
def _event(self, days_old, ticket_id=None):
|
||||||
|
first = (datetime.now(timezone.utc) - timedelta(days=days_old)).strftime('%Y-%m-%dT%H:%M:%SZ')
|
||||||
|
return {'id': 1, 'event_type': 'interface_down', 'severity': 'critical', 'target_name': 'micro1',
|
||||||
|
'target_detail': 'enp3s0', 'description': 'down', 'first_seen': first, 'last_seen': first,
|
||||||
|
'consecutive_failures': 3, 'ticket_id': ticket_id}
|
||||||
|
|
||||||
|
def test_old_alert_is_stale_and_closed_ticket_is_flagged(self, client):
|
||||||
|
self._state({'9': 'Open'})
|
||||||
|
gandalf.db.get_active_events.return_value = [self._event(5, '8'), self._event(1, '9')]
|
||||||
|
gandalf.db.get_cleared_tickets.return_value = []
|
||||||
|
events = client.get('/api/status', headers=AUTH).get_json()['events']
|
||||||
|
assert events[0]['stale_days'] == 5 and events[0]['ticket_closed'] is True
|
||||||
|
assert events[1]['stale_days'] is None and events[1]['ticket_closed'] is False
|
||||||
|
|
||||||
|
def test_threshold_comes_from_config(self, client):
|
||||||
|
self._state({})
|
||||||
|
gandalf.db.get_active_events.return_value = [self._event(1.5)]
|
||||||
|
gandalf.db.get_cleared_tickets.return_value = []
|
||||||
|
cfg = dict(CONFIG, monitor={'stale_alert_days': 1})
|
||||||
|
with patch.object(gandalf, '_cfg', cfg):
|
||||||
|
events = client.get('/api/status', headers=AUTH).get_json()['events']
|
||||||
|
assert events[0]['stale_days'] == 1
|
||||||
|
|
||||||
|
def test_tickets_left_open_are_listed(self, client):
|
||||||
|
self._state({'281331508': 'Open'})
|
||||||
|
left = [{'ticket_id': '281331508', 'event_type': 'host_unreachable', 'target_name': 'pbs',
|
||||||
|
'target_detail': '', 'severity': 'critical', 'first_seen': '2026-10-14T12:02:05Z',
|
||||||
|
'cleared_at': '2026-11-04T22:12:36Z', 'events': 3}]
|
||||||
|
gandalf.db.get_cleared_tickets.return_value = left
|
||||||
|
assert client.get('/api/status', headers=AUTH).get_json()['stale_tickets'] == left
|
||||||
|
gandalf.db.get_cleared_tickets.assert_called_with(['281331508'], older_than_hours=24)
|
||||||
|
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||||
|
assert '#281331508' in html and 'TO FOLLOW UP' in html
|
||||||
|
assert '<section class="g-section" id="followup">' in html
|
||||||
|
|
||||||
|
def test_out_of_date_list_flags_nothing(self, client):
|
||||||
|
self._state({}, age_s=3 * 3600)
|
||||||
|
gandalf.db.get_active_events.return_value = [self._event(1, '8')]
|
||||||
|
body = client.get('/api/status', headers=AUTH).get_json()
|
||||||
|
assert body['stale_tickets'] == [] and body['events'][0]['ticket_closed'] is False
|
||||||
|
gandalf.db.get_cleared_tickets.assert_not_called()
|
||||||
|
|
||||||
|
def test_no_list_yet_hides_the_panel(self, client):
|
||||||
|
html = client.get('/', headers=AUTH).get_data(as_text=True)
|
||||||
|
assert '<section class="g-section" id="followup" hidden>' in html
|
||||||
|
assert 'TO FOLLOW UP' not in html
|
||||||
|
|
||||||
|
|
||||||
|
class TestTopology:
|
||||||
|
def test_page_needs_auth(self, client):
|
||||||
|
assert client.get('/topology').status_code == 401
|
||||||
|
assert client.get('/api/topology').status_code == 401
|
||||||
|
|
||||||
|
def test_page_loads_the_drawings_first(self, client):
|
||||||
|
html = client.get('/topology', headers=AUTH).get_data(as_text=True)
|
||||||
|
assert 'id="tp-canvas"' in html
|
||||||
|
assert html.index('faceplates.js') < html.index('fronts.js') < html.index('topology.js')
|
||||||
|
|
||||||
|
def test_api_returns_the_graph(self, client):
|
||||||
|
gandalf._cutsheet_schema_ready = True
|
||||||
|
with patch.object(gandalf.db, 'cutsheet_load', return_value={'items': [], 'cables': []}):
|
||||||
|
data = client.get('/api/topology', headers=AUTH).get_json()
|
||||||
|
assert {'nodes', 'links', 'warnings', 'updated'} <= set(data)
|
||||||
|
names = {n['name'] for n in data['nodes']}
|
||||||
|
assert {'large1', 'USW Flex', 'Internet'} <= names
|
||||||
|
|
||||||
|
def test_api_survives_a_cutsheet_failure(self, client):
|
||||||
|
gandalf._cutsheet_schema_ready = True
|
||||||
|
with patch.object(gandalf.db, 'cutsheet_load', side_effect=RuntimeError('db down')):
|
||||||
|
assert client.get('/api/topology', headers=AUTH).status_code == 200
|
||||||
|
|
||||||
|
def test_suppressed_alerts_are_not_drawn(self, client):
|
||||||
|
gandalf._cutsheet_schema_ready = True
|
||||||
|
ev = {'id': 1, 'event_type': 'host_down', 'severity': 'critical', 'target_name': 'large1',
|
||||||
|
'target_detail': '', 'description': ''}
|
||||||
|
with patch.object(gandalf.db, 'cutsheet_load', return_value={'items': [], 'cables': []}), \
|
||||||
|
patch.object(gandalf.db, 'get_active_events', return_value=[ev]), \
|
||||||
|
patch.object(gandalf.db, 'check_suppressed', return_value=True):
|
||||||
|
data = client.get('/api/topology', headers=AUTH).get_json()
|
||||||
|
assert next(n for n in data['nodes'] if n['name'] == 'large1')['alert_count'] == 0
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
"""Tests for db.py query helpers that do work beyond the SQL: the connection is faked."""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
|
||||||
|
|
||||||
|
import db # noqa: E402
|
||||||
|
|
||||||
|
NOW = datetime(2026, 11, 14, 12, 0, 0)
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_conn(rows):
|
||||||
|
cur = MagicMock()
|
||||||
|
cur.fetchall.return_value = rows
|
||||||
|
cur.fetchone.return_value = {'now': NOW}
|
||||||
|
conn = MagicMock()
|
||||||
|
conn.cursor.return_value.__enter__.return_value = cur
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def get_conn():
|
||||||
|
yield conn
|
||||||
|
return patch.object(db, 'get_conn', get_conn), cur
|
||||||
|
|
||||||
|
|
||||||
|
def _event(tid, hours_ago_first, hours_ago_resolved, etype='host_unreachable', target='pbs'):
|
||||||
|
return {
|
||||||
|
'ticket_id': tid, 'event_type': etype, 'target_name': target, 'target_detail': '',
|
||||||
|
'severity': 'critical', 'first_seen': NOW - timedelta(hours=hours_ago_first),
|
||||||
|
'resolved_at': None if hours_ago_resolved is None else NOW - timedelta(hours=hours_ago_resolved),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class TestGetClearedTickets:
|
||||||
|
def test_no_ids_skips_the_query(self):
|
||||||
|
patcher, cur = _fake_conn([])
|
||||||
|
with patcher:
|
||||||
|
assert db.get_cleared_tickets([]) == []
|
||||||
|
cur.execute.assert_not_called()
|
||||||
|
|
||||||
|
def test_only_tickets_whose_every_alert_cleared_long_enough_ago(self):
|
||||||
|
rows = [
|
||||||
|
_event('1', 300, 250), _event('1', 100, 48), # two outages, both long cleared
|
||||||
|
_event('2', 50, 40), _event('2', 5, None), # still active
|
||||||
|
_event('3', 10, 2), # cleared too recently
|
||||||
|
]
|
||||||
|
patcher, cur = _fake_conn(rows)
|
||||||
|
with patcher:
|
||||||
|
out = db.get_cleared_tickets(['3', '1', '2', '1'], older_than_hours=24)
|
||||||
|
assert [r['ticket_id'] for r in out] == ['1']
|
||||||
|
assert out[0]['events'] == 2
|
||||||
|
assert out[0]['cleared_at'] == (NOW - timedelta(hours=48)).isoformat() + 'Z'
|
||||||
|
assert out[0]['first_seen'] == (NOW - timedelta(hours=300)).isoformat() + 'Z'
|
||||||
|
assert cur.execute.call_args_list[0].args[1] == ['1', '2', '3']
|
||||||
|
|
||||||
|
def test_newest_clearance_first(self):
|
||||||
|
rows = [_event('1', 400, 300), _event('2', 100, 30, target='micro1')]
|
||||||
|
patcher, _ = _fake_conn(rows)
|
||||||
|
with patcher:
|
||||||
|
out = db.get_cleared_tickets(['1', '2'])
|
||||||
|
assert [r['ticket_id'] for r in out] == ['2', '1']
|
||||||
|
assert out[0]['target_name'] == 'micro1'
|
||||||
+190
-1
@@ -31,6 +31,8 @@ def db():
|
|||||||
mock_db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
|
mock_db.cutsheet_load.return_value = {'items': [], 'cables': [], 'port_labels': []}
|
||||||
mock_db.get_pending_recovery_notes.return_value = []
|
mock_db.get_pending_recovery_notes.return_value = []
|
||||||
mock_db.get_device_link.return_value = ''
|
mock_db.get_device_link.return_value = ''
|
||||||
|
mock_db.get_incident_ticket.return_value = None
|
||||||
|
mock_db.get_ticket_ids_between.return_value = []
|
||||||
yield mock_db
|
yield mock_db
|
||||||
|
|
||||||
|
|
||||||
@@ -117,6 +119,19 @@ class TestDeadHostDetection:
|
|||||||
mon._process_interfaces({HOST_A: {'eth0': True}}, [])
|
mon._process_interfaces({HOST_A: {'eth0': True}}, [])
|
||||||
assert ('cluster_network_issue', 'test-cluster', '') in upserted(db)
|
assert ('cluster_network_issue', 'test-cluster', '') in upserted(db)
|
||||||
|
|
||||||
|
def test_unused_interface_never_alerts(self, mon, db):
|
||||||
|
mon._unused = {'alpha': {'eth1': {'reason': 'spare'}}}
|
||||||
|
db.get_baseline.return_value = {'alpha': {'eth0': 'up', 'eth1': 'up'}, 'beta': {'eth0': 'up'}}
|
||||||
|
mon._process_interfaces({HOST_A: {'eth0': True, 'eth1': False}, HOST_B: {'eth0': True}}, [])
|
||||||
|
assert ('interface_down', 'alpha', 'eth1') not in upserted(db)
|
||||||
|
assert not mon.tickets.create.called
|
||||||
|
|
||||||
|
def test_unused_interface_does_not_degrade_the_host(self, mon):
|
||||||
|
mon._unused = {'alpha': {'eth1': {}}}
|
||||||
|
snap = mon._collect_snapshot({HOST_A: {'eth0': True, 'eth1': False}, HOST_B: {'eth0': True}})
|
||||||
|
assert snap['hosts']['alpha']['status'] == 'up'
|
||||||
|
assert snap['hosts']['alpha']['interfaces'] == {'eth0': 'up', 'eth1': 'unused'}
|
||||||
|
|
||||||
def test_dead_host_shows_as_down_on_dashboard(self, mon):
|
def test_dead_host_shows_as_down_on_dashboard(self, mon):
|
||||||
snap = mon._collect_snapshot({HOST_A: {'eth0': True}})
|
snap = mon._collect_snapshot({HOST_A: {'eth0': True}})
|
||||||
assert snap['hosts']['alpha']['status'] == 'up'
|
assert snap['hosts']['alpha']['status'] == 'up'
|
||||||
@@ -368,6 +383,47 @@ class TestWanLog:
|
|||||||
mon._process_wan_log([])
|
mon._process_wan_log([])
|
||||||
assert not mon.tickets.create.called
|
assert not mon.tickets.create.called
|
||||||
|
|
||||||
|
def test_unifi_update_is_not_a_wan_event(self, mon, db):
|
||||||
|
now = time.time()
|
||||||
|
self._state(db, unifi_log_cursor=str(int(now * 1000)))
|
||||||
|
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
|
||||||
|
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
|
||||||
|
mon._process_wan_log([])
|
||||||
|
assert not mon.tickets.create.called
|
||||||
|
saved = dict((c.args[0], c.args[1]) for c in db.set_state.call_args_list)
|
||||||
|
assert saved[monitor.UNIFI_UPDATE_STATE]['label'] == 'UniFi Network 11.0.97 to 11.0.99'
|
||||||
|
assert saved['wan_history'] == {'outages': [], 'restarts': []}
|
||||||
|
|
||||||
|
def test_update_comments_on_tickets_opened_just_before(self, mon, db):
|
||||||
|
now = time.time()
|
||||||
|
self._state(db, unifi_log_cursor=str(int(now * 1000)))
|
||||||
|
db.get_ticket_ids_between.return_value = ['207521068']
|
||||||
|
self._mon(mon, [{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '',
|
||||||
|
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}])
|
||||||
|
mon._process_wan_log([])
|
||||||
|
start, end = db.get_ticket_ids_between.call_args.args[1:]
|
||||||
|
assert (end - start).total_seconds() == monitor.UPDATE_LOOKBACK_S + monitor.UPDATE_HOLD_S
|
||||||
|
tid, text = mon.tickets.comment.call_args.args
|
||||||
|
assert tid == '207521068' and 'UniFi Network 11.0.97 to 11.0.99' in text
|
||||||
|
|
||||||
|
def test_update_in_the_same_read_holds_the_power_cycle_ticket(self, mon, db):
|
||||||
|
now = time.time()
|
||||||
|
state = {'unifi_log_cursor': str(int(now * 1000))}
|
||||||
|
db.get_state.side_effect = lambda key, default=None: state.get(key, default)
|
||||||
|
db.set_state.side_effect = lambda key, value: state.__setitem__(
|
||||||
|
key, json.dumps(value) if isinstance(value, dict) else value)
|
||||||
|
self._mon(mon, [
|
||||||
|
{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': now - 600, 'device': 'RPDU', 'wan': ''},
|
||||||
|
{'key': 'NETWORK_UPDATED', 'ts': now, 'device': '', 'wan': '', 'app': 'UniFi Network'},
|
||||||
|
])
|
||||||
|
mon._process_wan_log([])
|
||||||
|
assert not mon.tickets.create.called
|
||||||
|
db.resolve_event.assert_called_with('modem_power_cycled', 'att-modem', '')
|
||||||
|
|
||||||
|
def test_hold_ends(self, mon, db):
|
||||||
|
self._state(db, unifi_update=json.dumps({'label': 'x', 'hold_until': time.time() - 1}))
|
||||||
|
assert mon._update_hold() is None
|
||||||
|
|
||||||
def test_uptime_going_backwards_records_a_restart(self, mon, db):
|
def test_uptime_going_backwards_records_a_restart(self, mon, db):
|
||||||
self._state(db, modem_stats=json.dumps({'uptime_s': 5000}))
|
self._state(db, modem_stats=json.dumps({'uptime_s': 5000}))
|
||||||
mon._note_modem_restart({'uptime_s': 60})
|
mon._note_modem_restart({'uptime_s': 60})
|
||||||
@@ -380,6 +436,55 @@ class TestWanLog:
|
|||||||
assert not db.set_state.called
|
assert not db.set_state.called
|
||||||
|
|
||||||
|
|
||||||
|
class TestWanIncident:
|
||||||
|
"""Last night's sequence: the RPDU power-cycles the modem in the cycle the modem goes unreachable."""
|
||||||
|
|
||||||
|
def _cycle(self, mon, db):
|
||||||
|
db.upsert_event.side_effect = [(30, True, 1, None), (29, False, 2, None)]
|
||||||
|
mon.modem.poll.side_effect = modem_api.ModemUnreachable('Network is unreachable')
|
||||||
|
with mon._wan_batch():
|
||||||
|
mon._one_shot_ticket([], 'modem_power_cycled', 'RPDU power-cycled the modem.',
|
||||||
|
'Modem auto power-cycled after internet loss', '3', 'Check the WAN panel.')
|
||||||
|
mon._collect_modem([])
|
||||||
|
|
||||||
|
def test_one_ticket_led_by_the_most_urgent(self, mon, db):
|
||||||
|
self._cycle(mon, db)
|
||||||
|
assert mon.tickets.create.call_count == 1
|
||||||
|
title, desc = mon.tickets.create.call_args.args
|
||||||
|
assert 'WAN modem unreachable' in title
|
||||||
|
assert mon.tickets.create.call_args.kwargs['priority'] == '2'
|
||||||
|
assert '### Also seen in this incident' in desc and 'power-cycled' in desc
|
||||||
|
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '123'), (30, '123')]
|
||||||
|
|
||||||
|
def test_recent_incident_gets_a_comment_instead(self, mon, db):
|
||||||
|
db.get_incident_ticket.return_value = '555'
|
||||||
|
mon.tickets.comment.return_value = True
|
||||||
|
self._cycle(mon, db)
|
||||||
|
assert not mon.tickets.create.called
|
||||||
|
assert [c.args[0] for c in mon.tickets.comment.call_args_list] == ['555', '555']
|
||||||
|
assert 'Also seen: WAN modem unreachable' in mon.tickets.comment.call_args_list[0].args[1]
|
||||||
|
assert sorted(c.args for c in db.set_ticket_id.call_args_list) == [(29, '555'), (30, '555')]
|
||||||
|
db.get_incident_ticket.assert_called_with('att-modem', monitor.WAN_INCIDENT_WINDOW_S)
|
||||||
|
|
||||||
|
def test_failed_comment_falls_back_to_a_ticket(self, mon, db):
|
||||||
|
db.get_incident_ticket.return_value = '555'
|
||||||
|
mon.tickets.comment.return_value = False
|
||||||
|
self._cycle(mon, db)
|
||||||
|
assert mon.tickets.create.call_count == 1
|
||||||
|
|
||||||
|
def test_outside_a_batch_tickets_open_at_once(self, mon, db):
|
||||||
|
mon._one_shot_ticket([], 'wan_unstable', 'Unstable.', 'WAN connection unstable', '2', 'Look.')
|
||||||
|
assert 'unstable' in mon.tickets.create.call_args.args[0]
|
||||||
|
assert 'no recovery note' in mon.tickets.create.call_args.args[1]
|
||||||
|
|
||||||
|
def test_update_hold_skips_new_tickets(self, mon, db):
|
||||||
|
db.get_state.side_effect = lambda key, default=None: (
|
||||||
|
json.dumps({'label': 'UniFi Network 11.0.99', 'hold_until': time.time() + 60})
|
||||||
|
if key == monitor.UNIFI_UPDATE_STATE else default)
|
||||||
|
self._cycle(mon, db)
|
||||||
|
assert not mon.tickets.create.called and not mon.tickets.comment.called
|
||||||
|
|
||||||
|
|
||||||
class TestUnifiDeviceDetail:
|
class TestUnifiDeviceDetail:
|
||||||
def test_parses_string_stats_and_outlets(self):
|
def test_parses_string_stats_and_outlets(self):
|
||||||
d = monitor.UnifiClient._device_detail({
|
d = monitor.UnifiClient._device_detail({
|
||||||
@@ -408,6 +513,31 @@ class TestUnifiDeviceDetail:
|
|||||||
got = c.get_system_events(1000, modem_api.WAN_LOG_KEYS)
|
got = c.get_system_events(1000, modem_api.WAN_LOG_KEYS)
|
||||||
assert got == [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': 2.0, 'device': 'RPDU', 'wan': ''}]
|
assert got == [{'key': modem_api.LOG_MODEM_RESTARTED, 'ts': 2.0, 'device': 'RPDU', 'wan': ''}]
|
||||||
|
|
||||||
|
def test_system_log_reads_update_versions(self):
|
||||||
|
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
|
||||||
|
page = MagicMock()
|
||||||
|
page.json.side_effect = [
|
||||||
|
{'data': [
|
||||||
|
{'key': 'NETWORK_UPDATED', 'timestamp': 3000, 'parameters': {
|
||||||
|
'APPLICATION': 'UniFi Network', 'PREVIOUS_VERSION': '11.0.97', 'VERSION': '11.0.99'}},
|
||||||
|
{'key': 'FIRMWARE_UPDATE_AVAILABLE_V2', 'timestamp': 3000},
|
||||||
|
{'key': 'MADE_CHANGES_ONE_SETTING_OBJECT', 'timestamp': 3000},
|
||||||
|
]},
|
||||||
|
{'data': []},
|
||||||
|
]
|
||||||
|
c.session.post = MagicMock(return_value=page)
|
||||||
|
got = c.get_system_events(1000, modem_api.wanted_log_key)
|
||||||
|
assert got == [{'key': 'NETWORK_UPDATED', 'ts': 3.0, 'device': '', 'wan': '',
|
||||||
|
'app': 'UniFi Network', 'previous': '11.0.97', 'version': '11.0.99'}]
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('key,expected', [
|
||||||
|
('NETWORK_UPDATED', True), ('FIRMWARE_UPDATED_V2', True), ('PROTECT_UPDATED', True),
|
||||||
|
('FIRMWARE_UPDATE_AVAILABLE_V2', False), ('MADE_CHANGES_ONE_SETTING_OBJECT', False),
|
||||||
|
('NETWORK_WAN_FAILED_2', False),
|
||||||
|
])
|
||||||
|
def test_update_keys(self, key, expected):
|
||||||
|
assert modem_api.is_update_key(key) is expected
|
||||||
|
|
||||||
def test_system_log_failure_is_none_not_empty(self):
|
def test_system_log_failure_is_none_not_empty(self):
|
||||||
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
|
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
|
||||||
c.session.post = MagicMock(side_effect=OSError('boom'))
|
c.session.post = MagicMock(side_effect=OSError('boom'))
|
||||||
@@ -512,9 +642,11 @@ class TestLldpCollection:
|
|||||||
{'metric': {'device': 'vmbr0', 'address': mac, 'instance': '10.0.0.1:9100'}},
|
{'metric': {'device': 'vmbr0', 'address': mac, 'instance': '10.0.0.1:9100'}},
|
||||||
]
|
]
|
||||||
switches = {'sw': {'ports': {'p': {'lldp': {'chassis_id': 'x', 'port_id': mac.upper()}}}}}
|
switches = {'sw': {'ports': {'p': {'lldp': {'chassis_id': 'x', 'port_id': mac.upper()}}}}}
|
||||||
collector._resolve_neighbors(switches, {}, {'10.0.0.1:9100': 'alpha'})
|
host_macs = collector._resolve_neighbors(switches, {}, {'10.0.0.1:9100': 'alpha'})
|
||||||
lldp = switches['sw']['ports']['p']['lldp']
|
lldp = switches['sw']['ports']['p']['lldp']
|
||||||
assert (lldp['system_name'], lldp['port_id']) == ('alpha', 'eth0')
|
assert (lldp['system_name'], lldp['port_id']) == ('alpha', 'eth0')
|
||||||
|
# Kept for the topology page: the physical NIC, not the bond or bridge
|
||||||
|
assert host_macs == {mac: ('alpha', 'eth0')}
|
||||||
|
|
||||||
def test_client_list_failure_is_empty(self):
|
def test_client_list_failure_is_empty(self):
|
||||||
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
|
c = monitor.UnifiClient({'controller': 'https://u.test', 'api_key': 'k'})
|
||||||
@@ -783,3 +915,60 @@ class TestTicketClientCreate:
|
|||||||
with self._post({'success': True, 'ticket_id': '8', 'action': 'none'}):
|
with self._post({'success': True, 'ticket_id': '8', 'action': 'none'}):
|
||||||
assert c.create('t', 'd') == '8'
|
assert c.create('t', 'd') == '8'
|
||||||
assert 't' not in c._closed_matches
|
assert 't' not in c._closed_matches
|
||||||
|
|
||||||
|
|
||||||
|
class TestTicketClientOpenTickets:
|
||||||
|
def _client(self):
|
||||||
|
return monitor.TicketClient({'url': 'http://t.test/create_ticket_api.php', 'api_key': 'k'})
|
||||||
|
|
||||||
|
def _page(self, tickets, pages):
|
||||||
|
resp = MagicMock(status_code=200)
|
||||||
|
resp.json.return_value = {'success': True, 'tickets': tickets, 'page': 1, 'pages': pages}
|
||||||
|
return resp
|
||||||
|
|
||||||
|
def test_read_endpoint_sits_beside_create(self):
|
||||||
|
assert self._client().read_url == 'http://t.test/api/tickets_api.php'
|
||||||
|
|
||||||
|
def test_reads_every_page_of_open_statuses(self):
|
||||||
|
pages = [self._page([{'ticket_id': '1', 'status': 'Open'}], 2),
|
||||||
|
self._page([{'ticket_id': 2, 'status': 'Pending'}], 2)]
|
||||||
|
with patch.object(monitor.requests, 'get', side_effect=pages) as get:
|
||||||
|
assert self._client().open_tickets() == {'1': 'Open', '2': 'Pending'}
|
||||||
|
assert get.call_count == 2
|
||||||
|
params = get.call_args.kwargs['params']
|
||||||
|
assert params['status'] == 'Open,Pending,In Progress' and params['page'] == 2
|
||||||
|
assert get.call_args.kwargs['headers'] == {'Authorization': 'Bearer k'}
|
||||||
|
|
||||||
|
def test_failure_means_unknown_not_empty(self):
|
||||||
|
with patch.object(monitor.requests, 'get', side_effect=OSError('down')):
|
||||||
|
assert self._client().open_tickets() is None
|
||||||
|
bad = MagicMock(status_code=200)
|
||||||
|
bad.json.return_value = {'success': False, 'error': 'nope'}
|
||||||
|
with patch.object(monitor.requests, 'get', return_value=bad):
|
||||||
|
assert self._client().open_tickets() is None
|
||||||
|
|
||||||
|
def test_unconfigured(self):
|
||||||
|
assert monitor.TicketClient({}).open_tickets() is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestSyncOpenTickets:
|
||||||
|
def test_saves_the_list_then_waits(self, mon, db):
|
||||||
|
mon.tickets.open_tickets.return_value = {'7': 'Open'}
|
||||||
|
mon._sync_open_tickets()
|
||||||
|
mon._sync_open_tickets()
|
||||||
|
assert mon.tickets.open_tickets.call_count == 1
|
||||||
|
key, value = db.set_state.call_args.args
|
||||||
|
assert key == monitor.OPEN_TICKETS_STATE and value['tickets'] == {'7': 'Open'}
|
||||||
|
assert value['checked_at'] > 0
|
||||||
|
|
||||||
|
def test_failed_read_keeps_the_previous_list(self, mon, db):
|
||||||
|
mon.tickets.open_tickets.return_value = None
|
||||||
|
mon._sync_open_tickets()
|
||||||
|
db.set_state.assert_not_called()
|
||||||
|
|
||||||
|
def test_refreshes_after_the_interval(self, mon, db):
|
||||||
|
mon.tickets.open_tickets.return_value = {}
|
||||||
|
mon._sync_open_tickets()
|
||||||
|
mon._open_tickets_synced -= monitor.OPEN_TICKETS_SYNC_S + 1
|
||||||
|
mon._sync_open_tickets()
|
||||||
|
assert mon.tickets.open_tickets.call_count == 2
|
||||||
@@ -0,0 +1,338 @@
|
|||||||
|
"""Tests for topology.build_graph, on a small network shaped like the real one."""
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
sys.path.insert(0, os.path.dirname(os.path.dirname(__file__)))
|
||||||
|
|
||||||
|
import topology # noqa: E402
|
||||||
|
|
||||||
|
|
||||||
|
def port(idx, name=None, up=True, speed=1000, neighbour='', port_id='', chassis='', uplink=False, sfp=None,
|
||||||
|
tx=0.0, rx=0.0, source=None):
|
||||||
|
lldp = {}
|
||||||
|
if neighbour or chassis:
|
||||||
|
lldp = {'system_name': neighbour, 'port_id': port_id, 'chassis_id': chassis, 'port_desc': '', 'mgmt_ips': []}
|
||||||
|
if source:
|
||||||
|
lldp['source'] = source
|
||||||
|
p = {'port_idx': idx, 'up': up, 'speed_mbps': speed if up else 0, 'is_uplink': uplink, 'media': 'GE',
|
||||||
|
'lldp': lldp, 'tx_bytes_rate': tx, 'rx_bytes_rate': rx, 'rx_errs_rate': 0.0, 'tx_errs_rate': 0.0,
|
||||||
|
'rx_drops_rate': 0.0, 'tx_drops_rate': 0.0}
|
||||||
|
if sfp:
|
||||||
|
p['sfp'] = dict({'found': True, 'vendor': 'OEM', 'serial': 'SN1'}, **sfp)
|
||||||
|
p['media'] = 'SFP+'
|
||||||
|
return (name or f'Port {idx}'), p
|
||||||
|
|
||||||
|
|
||||||
|
def ports(*items):
|
||||||
|
return dict(items)
|
||||||
|
|
||||||
|
|
||||||
|
def dac(peer=None, peer_port=None):
|
||||||
|
s = {'kind': 'dac', 'part': 'SFP-H10GB-CU1M'}
|
||||||
|
if peer:
|
||||||
|
s['cable_peer'] = {'switch': peer, 'port_idx': peer_port}
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def network(poe_port_11='Port 11', pbs_up=True):
|
||||||
|
"""Gateway -> aggregation -> PoE switch, two dual-homed hosts, an AP, a PDU and unknown devices."""
|
||||||
|
switches = {
|
||||||
|
'Dream Machine Pro': {'model': 'UDMPRO', 'ports': ports(
|
||||||
|
port(1, neighbour='host-a', port_id='eno2', source='unifi-clients'),
|
||||||
|
port(8), # up, nothing reported: a cable to something Gandalf can't see
|
||||||
|
port(9, uplink=True, chassis='02:00:5e:00:00:09', port_id='02:00:5e:00:00:0a'),
|
||||||
|
port(11, 'Downlink to USW-Aggregation', speed=10000, neighbour='8SFP-AGG', port_id='local Port 7',
|
||||||
|
sfp=dac('8SFP-AGG', 7), tx=2000.0, rx=1000.0),
|
||||||
|
)},
|
||||||
|
'8SFP-AGG': {'model': 'USL8A', 'ports': ports(
|
||||||
|
port(1, 'SFP+ 1', speed=10000, neighbour='host-a', port_id='eno1', sfp={'kind': 'dac', 'part': 'X'}),
|
||||||
|
port(2, 'SFP+ 2', speed=10000, neighbour='host-b', port_id='enp1s0', sfp={'kind': '', 'part': 'L56'}),
|
||||||
|
port(5, 'Downlink to 24RJ45-POE', speed=10000, neighbour='24RJ45-POE', port_id='Port 26',
|
||||||
|
sfp=dac('24RJ45-POE', 26)),
|
||||||
|
port(7, 'SFP+ 7', speed=10000, neighbour='Dream Machine Pro', port_id='02:00:5e:00:00:11',
|
||||||
|
uplink=True, sfp=dac('Dream Machine Pro', 11), tx=1000.0, rx=2000.0),
|
||||||
|
port(8, 'SFP+ 8', speed=10000, neighbour='host-c', port_id='nic4', sfp={'kind': 'optic', 'part': 'SR'}),
|
||||||
|
)},
|
||||||
|
'24RJ45-POE': {'model': 'US24PRO', 'ports': ports(
|
||||||
|
port(1, 'Micro 2 Eth2', chassis='02:00:5e:00:01:01', port_id='02:00:5E:00:01:01'),
|
||||||
|
port(3, 'Living Room Downlink', neighbour='USW Flex', port_id='02:00:5E:00:02:01'),
|
||||||
|
port(4, neighbour='host-b', port_id='enp2s0', source='unifi-clients'),
|
||||||
|
port(8, neighbour='192.0.2.44', port_id='eth0', chassis='02:00:5e:00:01:08', source='unifi-clients'),
|
||||||
|
port(11, poe_port_11, neighbour='AP', port_id='02:00:5E:00:03:01'),
|
||||||
|
port(12, up=False),
|
||||||
|
port(15, 'PBS', up=pbs_up, neighbour='pbs' if pbs_up else '', port_id='enp0s25' if pbs_up else ''),
|
||||||
|
port(19, neighbour='RPDU', port_id='02:00:5E:00:04:01', speed=100),
|
||||||
|
port(26, 'SFP+ 2', speed=10000, neighbour='8SFP-AGG', port_id='Port 5', uplink=True,
|
||||||
|
sfp=dac('8SFP-AGG', 5)),
|
||||||
|
)},
|
||||||
|
'USW Flex': {'model': 'USF5P', 'ports': ports(
|
||||||
|
port(1, uplink=True, neighbour='24RJ45-POE', port_id='Port 3'),
|
||||||
|
port(5, neighbour='UPS 2U Pro', port_id='Port 1', speed=100),
|
||||||
|
)},
|
||||||
|
'RPDU': {'model': 'USPPDUP', 'ports': ports(port(1, uplink=True, speed=100))},
|
||||||
|
}
|
||||||
|
unifi = [
|
||||||
|
{'name': 'Dream Machine Pro', 'type': 'udm', 'model': 'UDMPRO', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
{'name': '8SFP-AGG', 'type': 'usw', 'model': 'USL8A', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
{'name': '24RJ45-POE', 'type': 'usw', 'model': 'US24PRO', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
{'name': 'USW Flex', 'type': 'usw', 'model': 'USF5P', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
{'name': 'AP', 'type': 'uap', 'model': 'UAL6', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
{'name': 'RPDU', 'type': 'usw', 'model': 'USPPDUP', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
{'name': 'UPS 2U Pro', 'type': 'usp', 'model': 'USPDA2B', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
{'name': 'USP Strip', 'type': 'uap', 'model': 'UP6', 'connected': True, 'mac': '', 'ip': ''},
|
||||||
|
]
|
||||||
|
hosts = {
|
||||||
|
'host-a': {'status': 'up', 'interfaces': {'eno1': 'up', 'eno2': 'up', 'eno3': 'unused', 'vmbr0': 'up'}},
|
||||||
|
'host-b': {'status': 'degraded',
|
||||||
|
'interfaces': {'bond0': 'up', 'enp1s0': 'up', 'enp2s0': 'up', 'enp3s0': 'down'}},
|
||||||
|
'host-c': {'status': 'up', 'interfaces': {'nic4': 'up'}},
|
||||||
|
'pbs': {'status': 'up' if pbs_up else 'down', 'interfaces': {'enp0s25': 'up' if pbs_up else 'down'}},
|
||||||
|
}
|
||||||
|
snapshot = {'hosts': hosts, 'unifi': unifi}
|
||||||
|
link_stats = {
|
||||||
|
'hosts': {'192.0.2.44': {'eth0': {}}}, # an IP-only Prometheus target
|
||||||
|
'unifi_switches': switches,
|
||||||
|
'unifi_devices': {'Dream Machine Pro': {'wan': {'wan1': {'up': True}}},
|
||||||
|
'RPDU': {'power_w': 120.5, 'outlets': [{'index': 1, 'name': 'AT&T Modem', 'on': True}]}},
|
||||||
|
'updated': '2026-11-14 12:00:00 UTC',
|
||||||
|
}
|
||||||
|
modem = {'name': 'att-modem', 'model': 'BGW320-505', 'reachable': True, 'broadband_state': 'Up',
|
||||||
|
'source': 'FIBER', 'line_speed_mbps': 10000, 'rates': {'rx_bytes': 5000.0, 'tx_bytes': 700.0},
|
||||||
|
'wan_problems': []}
|
||||||
|
return snapshot, link_stats, modem
|
||||||
|
|
||||||
|
|
||||||
|
def build(events=(), items=(), cables=(), known=None, **kw):
|
||||||
|
snapshot, link_stats, modem = network(**kw)
|
||||||
|
return topology.build_graph(snapshot, link_stats, modem, list(events), list(items), list(cables), known)
|
||||||
|
|
||||||
|
|
||||||
|
def find(graph, a, b):
|
||||||
|
"""The links between two nodes, either way round."""
|
||||||
|
return [lk for lk in graph['links'] if {lk['a']['node'], lk['b']['node']} == {a, b}]
|
||||||
|
|
||||||
|
|
||||||
|
def node(graph, name):
|
||||||
|
return next(n for n in graph['nodes'] if n['id'] == name)
|
||||||
|
|
||||||
|
|
||||||
|
class TestSwitchCables:
|
||||||
|
def test_cable_seen_from_both_ends_is_drawn_once(self):
|
||||||
|
g = build()
|
||||||
|
assert len(find(g, 'Dream Machine Pro', '8SFP-AGG')) == 1
|
||||||
|
assert len(find(g, '8SFP-AGG', '24RJ45-POE')) == 1
|
||||||
|
assert len(find(g, '24RJ45-POE', 'USW Flex')) == 1
|
||||||
|
|
||||||
|
def test_both_ports_of_a_switch_cable_are_known(self):
|
||||||
|
g = build()
|
||||||
|
(link,) = find(g, 'Dream Machine Pro', '8SFP-AGG')
|
||||||
|
ends = {link['a']['node']: link['a']['port_idx'], link['b']['node']: link['b']['port_idx']}
|
||||||
|
assert ends == {'Dream Machine Pro': 11, '8SFP-AGG': 7}
|
||||||
|
(flex,) = find(g, '24RJ45-POE', 'USW Flex')
|
||||||
|
assert {flex['a']['port_idx'], flex['b']['port_idx']} == {3, 1}
|
||||||
|
|
||||||
|
def test_gear_without_lldp_lands_on_its_uplink(self):
|
||||||
|
(pdu,) = find(build(), '24RJ45-POE', 'RPDU')
|
||||||
|
assert pdu['b'] == {'node': 'RPDU', 'port_idx': 1, 'port_name': 'Port 1', 'iface': ''}
|
||||||
|
|
||||||
|
def test_cables_point_away_from_the_internet(self):
|
||||||
|
g = build()
|
||||||
|
(link,) = find(g, 'Dream Machine Pro', '8SFP-AGG')
|
||||||
|
assert link['a']['node'] == 'Dream Machine Pro'
|
||||||
|
# The gateway sends 2000 B/s down the cable and gets 1000 back, whichever end measured it
|
||||||
|
assert (link['a_to_b_bps'], link['b_to_a_bps']) == (2000.0, 1000.0)
|
||||||
|
(down,) = find(g, '8SFP-AGG', '24RJ45-POE')
|
||||||
|
assert down['a']['node'] == '8SFP-AGG'
|
||||||
|
|
||||||
|
def test_medium_and_module(self):
|
||||||
|
g = build()
|
||||||
|
assert find(g, 'Dream Machine Pro', '8SFP-AGG')[0]['medium'] == 'dac'
|
||||||
|
assert find(g, '8SFP-AGG', 'host-c')[0]['medium'] == 'fibre'
|
||||||
|
assert find(g, '8SFP-AGG', 'host-b')[0]['medium'] == 'sfp'
|
||||||
|
assert find(g, '24RJ45-POE', 'host-b')[0]['medium'] == 'copper'
|
||||||
|
assert find(g, 'Dream Machine Pro', '8SFP-AGG')[0]['sfp_part'] == 'SFP-H10GB-CU1M'
|
||||||
|
|
||||||
|
|
||||||
|
class TestWanEdge:
|
||||||
|
def test_gateway_to_modem_to_internet(self):
|
||||||
|
g = build()
|
||||||
|
(wan,) = find(g, 'Dream Machine Pro', 'att-modem')
|
||||||
|
assert wan['wan'] and wan['b']['node'] == 'Dream Machine Pro' and wan['a']['node'] == 'att-modem'
|
||||||
|
assert wan['b']['port_idx'] == 9
|
||||||
|
(fibre,) = find(g, 'att-modem', topology.INTERNET)
|
||||||
|
assert fibre['medium'] == 'fibre' and fibre['speed_mbps'] == 10000 and fibre['up']
|
||||||
|
# Down from the internet is what the modem received
|
||||||
|
assert fibre['a_to_b_bps'] == 5000.0
|
||||||
|
|
||||||
|
def test_the_wan_port_is_not_a_stub(self):
|
||||||
|
assert 'stub:Dream Machine Pro:9' not in {n['id'] for n in build()['nodes']}
|
||||||
|
|
||||||
|
def test_without_a_modem_the_gateway_goes_straight_out(self):
|
||||||
|
snapshot, link_stats, _ = network()
|
||||||
|
g = topology.build_graph(snapshot, link_stats, None, [], [], [], None)
|
||||||
|
assert len(find(g, 'Dream Machine Pro', topology.INTERNET)) == 1
|
||||||
|
|
||||||
|
def test_internet_status_follows_the_gateway_wan(self):
|
||||||
|
snapshot, link_stats, modem = network()
|
||||||
|
link_stats['unifi_devices']['Dream Machine Pro']['wan']['wan1']['up'] = False
|
||||||
|
g = topology.build_graph(snapshot, link_stats, modem, [], [], [], None)
|
||||||
|
assert node(g, topology.INTERNET)['status'] == 'down'
|
||||||
|
|
||||||
|
|
||||||
|
class TestHosts:
|
||||||
|
def test_dual_homed_host_has_two_cables(self):
|
||||||
|
g = build()
|
||||||
|
a = [lk for lk in g['links'] if 'host-a' in (lk['a']['node'], lk['b']['node'])]
|
||||||
|
assert sorted((lk['speed_mbps'], lk['b']['iface']) for lk in a) == [(1000, 'eno2'), (10000, 'eno1')]
|
||||||
|
|
||||||
|
def test_physical_nics_only_with_the_cables_on_them(self):
|
||||||
|
nics = {n['name']: n for n in node(build(), 'host-a')['interfaces']}
|
||||||
|
assert set(nics) == {'eno1', 'eno2', 'eno3'}
|
||||||
|
assert nics['eno3']['unused'] and 'links' not in nics['eno3']
|
||||||
|
assert len(nics['eno1']['links']) == 1
|
||||||
|
|
||||||
|
def test_ip_only_target_is_a_stub_not_a_host(self):
|
||||||
|
g = build()
|
||||||
|
assert '192.0.2.44' not in {n['id'] for n in g['nodes'] if n['kind'] != 'stub'}
|
||||||
|
stub = node(g, 'stub:24RJ45-POE:8')
|
||||||
|
assert stub['name'] == '192.0.2.44' and stub['ip'] == '192.0.2.44' and stub['iface'] == 'eth0'
|
||||||
|
|
||||||
|
def test_chassis_comes_from_the_cutsheet_item(self):
|
||||||
|
items = [{'name': 'Server A', 'device': 'host-a', 'chassis': 'sliger-cx4712-red'},
|
||||||
|
{'name': 'host-b', 'device': '', 'chassis': 'zimaboard-832-cx3'}]
|
||||||
|
g = build(items=items)
|
||||||
|
assert node(g, 'host-a')['chassis'] == 'sliger-cx4712-red'
|
||||||
|
assert node(g, 'host-b')['chassis'] == 'zimaboard-832-cx3'
|
||||||
|
assert node(g, 'host-c')['chassis'] == ''
|
||||||
|
|
||||||
|
|
||||||
|
class TestStubs:
|
||||||
|
def test_unnamed_neighbour_is_a_stub_named_by_its_port(self):
|
||||||
|
stub = node(build(), 'stub:24RJ45-POE:1')
|
||||||
|
assert stub['kind'] == 'stub' and stub['name'] == 'Micro 2 Eth2'
|
||||||
|
assert stub['mac'] == '02:00:5e:00:01:01' and not stub['anonymous']
|
||||||
|
|
||||||
|
def test_link_with_no_neighbour_is_an_anonymous_stub(self):
|
||||||
|
stub = node(build(), 'stub:Dream Machine Pro:8')
|
||||||
|
assert stub['anonymous'] and stub['name'] == 'unknown device'
|
||||||
|
|
||||||
|
def test_ports_with_no_link_are_left_out(self):
|
||||||
|
assert 'stub:24RJ45-POE:12' not in {n['id'] for n in build()['nodes']}
|
||||||
|
|
||||||
|
|
||||||
|
class TestNameMismatch:
|
||||||
|
def test_port_named_after_another_device(self):
|
||||||
|
(link,) = find(build(poe_port_11='Large1 Management'), '24RJ45-POE', 'AP')
|
||||||
|
assert link['name_mismatch'] is None # no device called large1 in this network
|
||||||
|
|
||||||
|
def test_flags_when_the_named_device_exists(self):
|
||||||
|
(link,) = find(build(poe_port_11='host-b management'), '24RJ45-POE', 'AP')
|
||||||
|
assert link['name_mismatch'] == {'named': 'host-b', 'actual': 'AP', 'end': 'a'}
|
||||||
|
|
||||||
|
def test_downlink_and_generic_names_are_fine(self):
|
||||||
|
g = build()
|
||||||
|
assert all(lk['name_mismatch'] is None for lk in g['links'])
|
||||||
|
|
||||||
|
@pytest.mark.parametrize('name', ['Port 4', 'SFP+ 2', 'Uplink', 'Downlink to 24RJ45-POE'])
|
||||||
|
def test_names_that_never_flag(self, name):
|
||||||
|
known = {topology._norm(n): n for n in ('24RJ45-POE', 'host-a')}
|
||||||
|
assert topology.name_mismatch(name, '24RJ45-POE', 'x', known) is None
|
||||||
|
|
||||||
|
def test_matches_ignore_case_spacing_and_dashes(self):
|
||||||
|
known = {topology._norm('compute-storage-01'): 'compute-storage-01'}
|
||||||
|
assert topology.name_mismatch('Compute Storage 01 mgmt', 'AP', 'sw', known)['named'] == 'compute-storage-01'
|
||||||
|
|
||||||
|
def test_a_stub_is_checked_against_what_was_seen(self):
|
||||||
|
snapshot, link_stats, modem = network()
|
||||||
|
link_stats['unifi_switches']['24RJ45-POE']['ports']['host-c'] = port(
|
||||||
|
20, 'host-c', chassis='02:00:5e:00:09:09')[1]
|
||||||
|
g = topology.build_graph(snapshot, link_stats, modem, [], [], [], None)
|
||||||
|
(link,) = find(g, '24RJ45-POE', 'stub:24RJ45-POE:20')
|
||||||
|
assert link['name_mismatch']['named'] == 'host-c'
|
||||||
|
|
||||||
|
|
||||||
|
class TestGhosts:
|
||||||
|
KNOWN = {'pbs': {'seen': '2026-11-14 20:49:08 UTC', 'uplinks': [
|
||||||
|
{'switch': '24RJ45-POE', 'port': 'PBS', 'port_idx': 15, 'host_interface': 'enp0s25'}]}}
|
||||||
|
|
||||||
|
def test_missing_host_is_drawn_where_last_seen(self):
|
||||||
|
g = build(known=self.KNOWN, pbs_up=False)
|
||||||
|
(ghost,) = find(g, '24RJ45-POE', 'pbs')
|
||||||
|
assert ghost['ghost'] and not ghost['up'] and ghost['seen'] == '2026-11-14 20:49:08 UTC'
|
||||||
|
assert ghost['b'] == {'node': 'pbs', 'port_idx': None, 'port_name': '', 'iface': 'enp0s25'}
|
||||||
|
|
||||||
|
def test_no_ghost_while_the_host_is_linked(self):
|
||||||
|
g = build(known=self.KNOWN)
|
||||||
|
assert [lk['ghost'] for lk in find(g, '24RJ45-POE', 'pbs')] == [False]
|
||||||
|
|
||||||
|
def test_no_ghost_onto_a_port_now_used_by_something_else(self):
|
||||||
|
known = {'pbs': {'seen': 'x', 'uplinks': [{'switch': '24RJ45-POE', 'port_idx': 4, 'host_interface': 'e'}]}}
|
||||||
|
assert find(build(known=known, pbs_up=False), '24RJ45-POE', 'pbs') == []
|
||||||
|
|
||||||
|
|
||||||
|
class TestDepth:
|
||||||
|
def test_hops_from_the_internet(self):
|
||||||
|
g = build()
|
||||||
|
depths = {n['id']: n['depth'] for n in g['nodes']}
|
||||||
|
assert depths[topology.INTERNET] == 0
|
||||||
|
assert depths['att-modem'] == 1
|
||||||
|
assert depths['Dream Machine Pro'] == 2
|
||||||
|
assert depths['8SFP-AGG'] == 3
|
||||||
|
assert depths['24RJ45-POE'] == 4
|
||||||
|
assert depths['USW Flex'] == 5 and depths['AP'] == 5
|
||||||
|
assert depths['UPS 2U Pro'] == 6
|
||||||
|
# A host's depth is its nearest switch's plus one; hosts never pass the path on
|
||||||
|
assert depths['host-a'] == 3
|
||||||
|
|
||||||
|
def test_unlinked_gear_has_no_depth(self):
|
||||||
|
assert node(build(), 'USP Strip')['depth'] is None
|
||||||
|
|
||||||
|
def test_power_gear_is_marked(self):
|
||||||
|
g = build()
|
||||||
|
assert {n['id']: n['kind'] for n in g['nodes'] if n['power']} == {
|
||||||
|
'RPDU': 'pdu', 'UPS 2U Pro': 'ups', 'USP Strip': 'strip'}
|
||||||
|
assert node(g, 'RPDU')['power_w'] == 120.5
|
||||||
|
|
||||||
|
|
||||||
|
class TestAlerts:
|
||||||
|
def test_active_alerts_count_on_their_device(self):
|
||||||
|
events = [
|
||||||
|
{'id': 1, 'event_type': 'interface_down', 'severity': 'critical', 'target_name': 'host-b',
|
||||||
|
'target_detail': 'enp3s0', 'description': ''},
|
||||||
|
{'id': 2, 'event_type': 'unifi_port_errors', 'severity': 'warning', 'target_name': '24RJ45-POE',
|
||||||
|
'target_detail': 'Port 9', 'description': ''},
|
||||||
|
{'id': 3, 'event_type': 'host_down', 'severity': 'critical', 'target_name': 'host-a',
|
||||||
|
'target_detail': '', 'description': '', 'is_suppressed': True},
|
||||||
|
]
|
||||||
|
g = build(events=events)
|
||||||
|
b = node(g, 'host-b')
|
||||||
|
assert (b['alert_count'], b['severity']) == (1, 'critical')
|
||||||
|
assert next(n for n in b['interfaces'] if n['name'] == 'enp3s0')['alert'] == 'critical'
|
||||||
|
assert node(g, '24RJ45-POE')['severity'] == 'warning'
|
||||||
|
assert node(g, 'host-a')['alert_count'] == 0
|
||||||
|
|
||||||
|
def test_status_comes_from_the_snapshot(self):
|
||||||
|
assert node(build(), 'host-b')['status'] == 'degraded'
|
||||||
|
|
||||||
|
|
||||||
|
class TestCutsheetLabels:
|
||||||
|
def test_cable_label_on_either_end(self):
|
||||||
|
items = [{'name': 'Patch', 'device': '', 'chassis': ''}, {'name': 'UDM-Pro', 'device': 'Dream Machine Pro'}]
|
||||||
|
cables = [{'id': 7, 'label': 'C07', 'a_device': 'UDM-Pro', 'a_port': '8', 'b_device': 'Patch',
|
||||||
|
'b_port': '1', 'status': 'installed', 'cable_type': 'Cat6', 'color': 'Black'},
|
||||||
|
{'id': 8, 'label': 'C08', 'a_device': 'host-a', 'a_port': 'eno2', 'b_device': '',
|
||||||
|
'b_port': '', 'status': 'retired'}]
|
||||||
|
g = build(items=items, cables=cables)
|
||||||
|
(stub,) = find(g, 'Dream Machine Pro', 'stub:Dream Machine Pro:8')
|
||||||
|
assert stub['cable'] == {'id': 7, 'label': 'C07', 'type': 'Cat6', 'color': 'Black'}
|
||||||
|
# Retired cables are not on the network any more
|
||||||
|
assert find(g, 'Dream Machine Pro', 'host-a')[0]['cable'] is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestEmpty:
|
||||||
|
def test_no_data_yet(self):
|
||||||
|
g = topology.build_graph({}, {}, None, [], [], [], None)
|
||||||
|
assert [n['id'] for n in g['nodes']] == [topology.INTERNET]
|
||||||
|
assert g['links'] == [] and g['warnings']
|
||||||
@@ -147,3 +147,25 @@ def recovery_comment(event: dict, now: datetime, url: str = '') -> str:
|
|||||||
lines += ['', '_Gandalf leaves this ticket open. Close it once you know why it happened, '
|
lines += ['', '_Gandalf leaves this ticket open. Close it once you know why it happened, '
|
||||||
'or leave it open if it needs follow-up._']
|
'or leave it open if it needs follow-up._']
|
||||||
return '\n'.join(lines)
|
return '\n'.join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def incident_comment(name: str, summary: str, checks: Sequence[str], when: str) -> str:
|
||||||
|
"""A further report that belongs to the incident this ticket is already about."""
|
||||||
|
lines = [f'**Also seen: {name}** ({when})', '', summary.strip()]
|
||||||
|
if checks:
|
||||||
|
lines += ['', '**What to check**'] + [f'{n}. {check}' for n, check in enumerate(checks, 1)]
|
||||||
|
lines += ['', '_Gandalf added this here instead of opening another ticket, since it is part of '
|
||||||
|
'the same incident._']
|
||||||
|
return '\n'.join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def update_comment(label: str, when: str, lookback_min: int, hold_min: int) -> str:
|
||||||
|
"""Note on a ticket that a UniFi update finished around the time it was opened."""
|
||||||
|
return '\n'.join([
|
||||||
|
f'**UniFi update:** {label} finished at {when}.', '',
|
||||||
|
f'This ticket was opened within {lookback_min} minutes of that update, which UniFi logs once '
|
||||||
|
f'it is done. Updating the Network app or the gateway firmware can drop the WAN, and Smart '
|
||||||
|
f'Power may then power-cycle the modem, so the update is the likely cause.', '',
|
||||||
|
f'_Gandalf holds new modem and WAN tickets for {hold_min} minutes after an update; anything '
|
||||||
|
f'still wrong after that gets its own ticket._',
|
||||||
|
])
|
||||||
+411
@@ -0,0 +1,411 @@
|
|||||||
|
"""The live network topology: every device and every physical cable between them.
|
||||||
|
|
||||||
|
Pure functions with no I/O. app.py passes in the monitor's cached state and the
|
||||||
|
cutsheet, and the /topology page draws what build_graph returns.
|
||||||
|
|
||||||
|
The graph comes from the switches' point of view: each UniFi port with a neighbour
|
||||||
|
is one end of a cable. The same switch-to-switch cable is seen from both ends, so
|
||||||
|
those are merged into one link.
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
from collections import deque
|
||||||
|
from typing import Dict, List, Optional, Tuple
|
||||||
|
|
||||||
|
import devices
|
||||||
|
|
||||||
|
INTERNET = 'internet'
|
||||||
|
|
||||||
|
# How UniFi devices are drawn and grouped. Power gear goes in its own rail.
|
||||||
|
_POWER_MODELS = {'USPPDUP': 'pdu', 'USPDA2B': 'ups', 'UP6': 'strip'}
|
||||||
|
POWER_KINDS = ('pdu', 'ups', 'strip')
|
||||||
|
|
||||||
|
_SEVERITY_RANK = {'critical': 3, 'warning': 2, 'info': 1}
|
||||||
|
_PORT_REF_RE = re.compile(r'(?:^|\s)port\s*(\d+)$', re.IGNORECASE)
|
||||||
|
_TOKEN_RE = re.compile(r'[a-z0-9]+')
|
||||||
|
|
||||||
|
|
||||||
|
def _norm(name: str) -> str:
|
||||||
|
"""A name with case, spaces, dashes and underscores ignored: "Large1" == "large-1"."""
|
||||||
|
return ''.join(_TOKEN_RE.findall((name or '').lower()))
|
||||||
|
|
||||||
|
|
||||||
|
def _node_kind(entry: dict) -> str:
|
||||||
|
if entry['kind'] == 'pdu':
|
||||||
|
return _POWER_MODELS.get(entry.get('model', ''), 'pdu')
|
||||||
|
return entry['kind']
|
||||||
|
|
||||||
|
|
||||||
|
def _medium(port: dict) -> str:
|
||||||
|
"""dac, fibre, sfp (a module that doesn't say which) or copper."""
|
||||||
|
sfp = port.get('sfp') or {}
|
||||||
|
if sfp.get('found') or sfp.get('part'):
|
||||||
|
kind = sfp.get('kind') or ''
|
||||||
|
if kind == 'dac':
|
||||||
|
return 'dac'
|
||||||
|
if kind == 'optic':
|
||||||
|
return 'fibre'
|
||||||
|
return 'sfp'
|
||||||
|
if str(port.get('media') or '').upper().startswith('SFP'):
|
||||||
|
return 'sfp' if port.get('up') else 'copper'
|
||||||
|
return 'copper'
|
||||||
|
|
||||||
|
|
||||||
|
def _rate(value) -> float:
|
||||||
|
try:
|
||||||
|
return max(float(value or 0), 0.0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def _utilisation(a_to_b: float, b_to_a: float, speed_mbps) -> Optional[float]:
|
||||||
|
try:
|
||||||
|
speed = float(speed_mbps or 0)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
speed = 0.0
|
||||||
|
if speed <= 0:
|
||||||
|
return None
|
||||||
|
return round(min(max(a_to_b, b_to_a) * 8 / (speed * 1e6), 1.0), 4)
|
||||||
|
|
||||||
|
|
||||||
|
def _port_stats(port: dict) -> dict:
|
||||||
|
sfp = port.get('sfp') or {}
|
||||||
|
return {
|
||||||
|
'speed_mbps': port.get('speed_mbps') or 0,
|
||||||
|
'up': bool(port.get('up')),
|
||||||
|
'medium': _medium(port),
|
||||||
|
'sfp_part': sfp.get('part') or '',
|
||||||
|
'sfp_vendor': sfp.get('vendor') or '',
|
||||||
|
'sfp_serial': sfp.get('serial') or '',
|
||||||
|
'errors': round(_rate(port.get('rx_errs_rate')) + _rate(port.get('tx_errs_rate')), 3),
|
||||||
|
'drops': round(_rate(port.get('rx_drops_rate')) + _rate(port.get('tx_drops_rate')), 3),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _end(node: str, port_name: str = '', port_idx=None, iface: str = '') -> dict:
|
||||||
|
return {'node': node, 'port_idx': port_idx, 'port_name': port_name, 'iface': iface}
|
||||||
|
|
||||||
|
|
||||||
|
def _far_port(switches: dict, here: str, port: dict, neighbour: str) -> Optional[int]:
|
||||||
|
"""Which port on a neighbouring switch the cable from `port` lands on, if it can be told."""
|
||||||
|
peer = (port.get('sfp') or {}).get('cable_peer') or {}
|
||||||
|
if peer.get('switch') == neighbour and peer.get('port_idx') is not None:
|
||||||
|
return peer['port_idx']
|
||||||
|
far = switches.get(neighbour)
|
||||||
|
if not far:
|
||||||
|
return None
|
||||||
|
ports = [p for p in (far.get('ports') or {}).values() if p.get('port_idx') is not None]
|
||||||
|
# LLDP's port ID names the neighbour's own port ("Port 3", "local Port 7")
|
||||||
|
m = _PORT_REF_RE.search(str((port.get('lldp') or {}).get('port_id') or '').strip())
|
||||||
|
if m and any(p['port_idx'] == int(m.group(1)) for p in ports):
|
||||||
|
return int(m.group(1))
|
||||||
|
back = [p['port_idx'] for p in ports if (p.get('lldp') or {}).get('system_name') == here]
|
||||||
|
if len(back) == 1:
|
||||||
|
return back[0]
|
||||||
|
# Gear that sends no LLDP of its own (a PDU): its single uplink
|
||||||
|
uplinks = [p['port_idx'] for p in ports if p.get('is_uplink')]
|
||||||
|
return uplinks[0] if len(uplinks) == 1 else None
|
||||||
|
|
||||||
|
|
||||||
|
def _is_generic_port_name(name: str) -> bool:
|
||||||
|
return bool(re.fullmatch(r'(port|sfp\+?|uplink|downlink)\s*\d*', (name or '').strip(), re.IGNORECASE))
|
||||||
|
|
||||||
|
|
||||||
|
def name_mismatch(port_name: str, neighbour: str, own: str, known: Dict[str, str]) -> Optional[dict]:
|
||||||
|
"""A port named after a different device than the one plugged into it.
|
||||||
|
|
||||||
|
known: {normalised name: display name} of every device. Runs of up to four words of
|
||||||
|
the port name are compared with them, so "Large1 Management" names large1 and
|
||||||
|
"Downlink to 24RJ45-POE" names 24RJ45-POE. Ports named only "Port 4" never match.
|
||||||
|
"""
|
||||||
|
if not port_name or _is_generic_port_name(port_name):
|
||||||
|
return None
|
||||||
|
words = _TOKEN_RE.findall(port_name.lower())
|
||||||
|
skip = {_norm(neighbour), _norm(own)}
|
||||||
|
for size in range(min(4, len(words)), 0, -1):
|
||||||
|
for i in range(len(words) - size + 1):
|
||||||
|
run = ''.join(words[i:i + size])
|
||||||
|
if run in known and run not in skip:
|
||||||
|
return {'named': known[run], 'actual': neighbour}
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _cable_labels(items: List[dict], cables: List[dict]) -> Dict[Tuple[str, str], dict]:
|
||||||
|
"""{(device, port): cable} for cutsheet cables still in place. A cable can name the
|
||||||
|
device itself or a cutsheet item linked to it."""
|
||||||
|
device_of = {i['name']: (i.get('device') or i['name']) for i in items or []}
|
||||||
|
out: Dict[Tuple[str, str], dict] = {}
|
||||||
|
for c in cables or []:
|
||||||
|
if c.get('status') == 'retired':
|
||||||
|
continue
|
||||||
|
for end in ('a', 'b'):
|
||||||
|
dev = c.get(f'{end}_device') or ''
|
||||||
|
if not dev:
|
||||||
|
continue
|
||||||
|
ref = {'id': c.get('id'), 'label': c.get('label') or '', 'type': c.get('cable_type') or '',
|
||||||
|
'color': c.get('color') or ''}
|
||||||
|
for name in {dev, device_of.get(dev, dev)}:
|
||||||
|
out[(name, str(c.get(f'{end}_port') or ''))] = ref
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _alerts_by_target(events: List[dict]) -> Dict[str, List[dict]]:
|
||||||
|
out: Dict[str, List[dict]] = {}
|
||||||
|
for ev in events or []:
|
||||||
|
if ev.get('is_suppressed'):
|
||||||
|
continue
|
||||||
|
out.setdefault(ev.get('target_name') or '', []).append({
|
||||||
|
'id': ev.get('id'), 'severity': ev.get('severity') or 'info',
|
||||||
|
'type': ev.get('event_type') or '', 'detail': ev.get('target_detail') or '',
|
||||||
|
'description': ev.get('description') or '',
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _worst(alerts: List[dict]) -> str:
|
||||||
|
return max((a['severity'] for a in alerts), key=lambda s: _SEVERITY_RANK.get(s, 0), default='')
|
||||||
|
|
||||||
|
|
||||||
|
def build_graph(snapshot: dict, link_stats: dict, modem: Optional[dict], events: List[dict],
|
||||||
|
cutsheet_items: List[dict], cutsheet_cables: List[dict],
|
||||||
|
known_uplinks: Optional[dict] = None) -> dict:
|
||||||
|
"""Every device and cable, as {nodes, links, warnings, updated}.
|
||||||
|
|
||||||
|
snapshot, link_stats, modem: the monitor's saved state.
|
||||||
|
events: active events, with is_suppressed set (suppressed ones are left out).
|
||||||
|
known_uplinks: the monitor's record of where each device was last plugged in,
|
||||||
|
drawn as "last seen" links for a device with no live link.
|
||||||
|
"""
|
||||||
|
snapshot = snapshot or {}
|
||||||
|
link_stats = link_stats or {}
|
||||||
|
switches = link_stats.get('unifi_switches') or {}
|
||||||
|
unifi_info = link_stats.get('unifi_devices') or {}
|
||||||
|
alerts = _alerts_by_target(events)
|
||||||
|
labels = _cable_labels(cutsheet_items, cutsheet_cables)
|
||||||
|
chassis = {}
|
||||||
|
for item in cutsheet_items or []:
|
||||||
|
chassis.setdefault(item.get('device') or item['name'], item.get('chassis') or '')
|
||||||
|
warnings: List[str] = []
|
||||||
|
|
||||||
|
# ── Nodes ────────────────────────────────────────────────────────────
|
||||||
|
nodes: Dict[str, dict] = {}
|
||||||
|
configured = set(snapshot.get('hosts') or {})
|
||||||
|
for entry in devices.build_index(snapshot, link_stats, modem):
|
||||||
|
name = entry['name']
|
||||||
|
if entry['kind'] == 'host' and name not in configured:
|
||||||
|
continue # IP-only Prometheus targets: shown only as a stub where a port sees them
|
||||||
|
kind = _node_kind(entry)
|
||||||
|
node = {
|
||||||
|
'id': name, 'name': name, 'kind': kind, 'status': entry['status'], 'model': entry.get('model', ''),
|
||||||
|
'ip': entry.get('ip', ''), 'url': f'/device/{name}', 'power': kind in POWER_KINDS,
|
||||||
|
}
|
||||||
|
if kind == 'host':
|
||||||
|
node['chassis'] = chassis.get(name, '')
|
||||||
|
node['interfaces'] = devices.physical_interfaces(name, snapshot)
|
||||||
|
for nic in node['interfaces']:
|
||||||
|
nic_alerts = [a for a in alerts.get(name, []) if a['detail'] == nic['name']]
|
||||||
|
nic['alert'] = _worst(nic_alerts)
|
||||||
|
elif name in switches or kind in ('router', 'switch'):
|
||||||
|
node['ports'] = {
|
||||||
|
str(p['port_idx']): {'name': pname, 'up': bool(p.get('up')), 'speed_mbps': p.get('speed_mbps') or 0,
|
||||||
|
'is_uplink': bool(p.get('is_uplink')), 'media': p.get('media') or '',
|
||||||
|
'poe_power': p.get('poe_power'), 'sfp': p.get('sfp') or None,
|
||||||
|
'tx_bytes_rate': _rate(p.get('tx_bytes_rate')),
|
||||||
|
'rx_bytes_rate': _rate(p.get('rx_bytes_rate')),
|
||||||
|
'neighbour': (p.get('lldp') or {}).get('system_name') or ''}
|
||||||
|
for pname, p in ((switches.get(name) or {}).get('ports') or {}).items()
|
||||||
|
if p.get('port_idx') is not None
|
||||||
|
}
|
||||||
|
if kind in POWER_KINDS:
|
||||||
|
info = unifi_info.get(name) or {}
|
||||||
|
node['power_w'] = info.get('power_w')
|
||||||
|
node['outlets'] = info.get('outlets') or []
|
||||||
|
node_alerts = alerts.get(name, [])
|
||||||
|
node['alerts'] = node_alerts[:10]
|
||||||
|
node['alert_count'] = len(node_alerts)
|
||||||
|
node['severity'] = _worst(node_alerts)
|
||||||
|
nodes[name] = node
|
||||||
|
|
||||||
|
router = next((n for n in nodes.values() if n['kind'] == 'router'), None)
|
||||||
|
modem_name = (modem or {}).get('name') if (modem or {}).get('name') in nodes else None
|
||||||
|
wan = ((unifi_info.get(router['name']) or {}).get('wan') or {}) if router else {}
|
||||||
|
wan_up = any(w.get('up') for w in wan.values()) if wan else None
|
||||||
|
if modem_name:
|
||||||
|
wan_up = (modem or {}).get('broadband_state', '').lower() == 'up' if wan_up is None else wan_up
|
||||||
|
nodes[INTERNET] = {
|
||||||
|
'id': INTERNET, 'name': 'Internet', 'kind': INTERNET, 'model': '', 'ip': '', 'url': '',
|
||||||
|
'status': 'unknown' if wan_up is None else ('up' if wan_up else 'down'),
|
||||||
|
'alerts': alerts.get('WAN', [])[:10], 'alert_count': len(alerts.get('WAN', [])),
|
||||||
|
'severity': _worst(alerts.get('WAN', [])), 'power': False,
|
||||||
|
}
|
||||||
|
known_names = {_norm(n['name']): n['name'] for n in nodes.values() if n['id'] != INTERNET}
|
||||||
|
|
||||||
|
# ── Links ────────────────────────────────────────────────────────────
|
||||||
|
links: List[dict] = []
|
||||||
|
claimed: Dict[Tuple[str, int], dict] = {}
|
||||||
|
|
||||||
|
def add_link(a: dict, b: dict, port: dict, **extra) -> dict:
|
||||||
|
link = {'id': '', 'a': a, 'b': b, 'ghost': False, 'name_mismatch': None,
|
||||||
|
'cable': labels.get((a['node'], str(a['port_idx'] if a['port_idx'] is not None else a['iface'])))
|
||||||
|
or labels.get((b['node'], str(b['port_idx'] if b['port_idx'] is not None else b['iface']))),
|
||||||
|
**_port_stats(port),
|
||||||
|
'_tx': _rate(port.get('tx_bytes_rate')), '_rx': _rate(port.get('rx_bytes_rate'))}
|
||||||
|
link.update(extra)
|
||||||
|
links.append(link)
|
||||||
|
for end in (a, b):
|
||||||
|
if end['port_idx'] is not None and end['node'] in switches:
|
||||||
|
claimed[(end['node'], end['port_idx'])] = link
|
||||||
|
return link
|
||||||
|
|
||||||
|
def ordered_ports(sw: dict):
|
||||||
|
return sorted(((pn, p) for pn, p in (sw.get('ports') or {}).items() if p.get('port_idx') is not None),
|
||||||
|
key=lambda kv: kv[1]['port_idx'])
|
||||||
|
|
||||||
|
# The router's internet port goes to the modem, then the modem to the internet
|
||||||
|
wan_port = None
|
||||||
|
if router and router['name'] in switches:
|
||||||
|
wan_port = next(((pn, p) for pn, p in ordered_ports(switches[router['name']]) if p.get('is_uplink')), None)
|
||||||
|
upstream = modem_name or INTERNET
|
||||||
|
if wan_port:
|
||||||
|
pn, p = wan_port
|
||||||
|
add_link(_end(router['name'], pn, p['port_idx']), _end(upstream, 'WAN' if modem_name else ''), p, wan=True)
|
||||||
|
elif router:
|
||||||
|
warnings.append(f'No internet port found on {router["name"]}')
|
||||||
|
if modem_name:
|
||||||
|
m = modem or {}
|
||||||
|
rates = m.get('rates') or {}
|
||||||
|
up = str(m.get('broadband_state') or '').lower() == 'up'
|
||||||
|
links.append({
|
||||||
|
'id': '', 'a': _end(INTERNET), 'b': _end(modem_name, str(m.get('source') or 'WAN').title()),
|
||||||
|
'ghost': False, 'name_mismatch': None, 'cable': None, 'wan': True,
|
||||||
|
'speed_mbps': m.get('line_speed_mbps') or 0, 'up': up,
|
||||||
|
'medium': 'fibre' if str(m.get('source') or '').upper() == 'FIBER' else 'copper',
|
||||||
|
'sfp_part': m.get('optics_part_no') or '', 'sfp_vendor': '', 'sfp_serial': '',
|
||||||
|
'errors': round(_rate(rates.get('rx_errors')) + _rate(rates.get('tx_errors')), 3),
|
||||||
|
'drops': round(_rate(rates.get('rx_drops')) + _rate(rates.get('tx_drops')), 3),
|
||||||
|
# Seen from the modem's broadband side: rx comes down from the internet
|
||||||
|
'_tx': _rate(rates.get('rx_bytes')), '_rx': _rate(rates.get('tx_bytes')),
|
||||||
|
})
|
||||||
|
elif not router:
|
||||||
|
warnings.append('No UniFi gateway reported yet, so the internet edge is not drawn')
|
||||||
|
|
||||||
|
# Device-to-device and device-to-host cables, from every switch port with a neighbour
|
||||||
|
for sw_name in sorted(switches):
|
||||||
|
for pname, port in ordered_ports(switches[sw_name]):
|
||||||
|
idx = port['port_idx']
|
||||||
|
if (sw_name, idx) in claimed:
|
||||||
|
continue
|
||||||
|
lldp = port.get('lldp') or {}
|
||||||
|
neighbour = lldp.get('system_name') or ''
|
||||||
|
here = _end(sw_name, pname, idx)
|
||||||
|
if neighbour in nodes and neighbour != sw_name:
|
||||||
|
nb = nodes[neighbour]
|
||||||
|
if nb['kind'] == 'host':
|
||||||
|
far = _end(neighbour, iface=lldp.get('port_id') or '')
|
||||||
|
else:
|
||||||
|
far_idx = _far_port(switches, sw_name, port, neighbour)
|
||||||
|
if far_idx is not None and (neighbour, far_idx) in claimed:
|
||||||
|
continue # the other end already drew this cable
|
||||||
|
far_port = next((pn for pn, p in ((switches.get(neighbour) or {}).get('ports') or {}).items()
|
||||||
|
if p.get('port_idx') == far_idx), '') if far_idx is not None else ''
|
||||||
|
far = _end(neighbour, far_port, far_idx)
|
||||||
|
link = add_link(here, far, port)
|
||||||
|
link['name_mismatch'] = name_mismatch(pname, neighbour, sw_name, known_names)
|
||||||
|
if link['name_mismatch'] is None and far['port_name']:
|
||||||
|
link['name_mismatch'] = name_mismatch(far['port_name'], sw_name, neighbour, known_names)
|
||||||
|
if link['name_mismatch']:
|
||||||
|
link['name_mismatch']['end'] = 'b'
|
||||||
|
elif link['name_mismatch']:
|
||||||
|
link['name_mismatch']['end'] = 'a'
|
||||||
|
continue
|
||||||
|
if port.get('is_uplink') and not neighbour:
|
||||||
|
continue # leads upstream to gear already drawn (a PDU's management port)
|
||||||
|
if not port.get('up') and not lldp:
|
||||||
|
continue
|
||||||
|
# Something is plugged in that Gandalf can't name: a grey stub
|
||||||
|
stub_id = f'stub:{sw_name}:{idx}'
|
||||||
|
mac = lldp.get('chassis_id') or ''
|
||||||
|
label = neighbour or ('' if _is_generic_port_name(pname) else pname) or mac or 'unknown device'
|
||||||
|
nodes[stub_id] = {
|
||||||
|
'id': stub_id, 'name': label, 'kind': 'stub', 'status': 'up' if port.get('up') else 'down',
|
||||||
|
'model': '', 'ip': neighbour if re.fullmatch(r'[\d.]+', neighbour) else '', 'mac': mac,
|
||||||
|
'url': f'/device/{sw_name}#port-{idx}', 'switch': sw_name, 'port_idx': idx, 'port_name': pname,
|
||||||
|
'iface': (lldp.get('port_id') or '') if neighbour else '', 'anonymous': not lldp,
|
||||||
|
'alerts': [], 'alert_count': 0, 'severity': '', 'power': False,
|
||||||
|
}
|
||||||
|
link = add_link(here, _end(stub_id, iface=nodes[stub_id]['iface']), port)
|
||||||
|
# A stub labelled by its port name would always "match" it, so compare with what was seen
|
||||||
|
link['name_mismatch'] = name_mismatch(pname, neighbour or mac or 'an unnamed device', sw_name, known_names)
|
||||||
|
if link['name_mismatch']:
|
||||||
|
link['name_mismatch']['end'] = 'a'
|
||||||
|
|
||||||
|
# Gear with no live cable: draw where it was last plugged in
|
||||||
|
linked = {e['node'] for link in links for e in (link['a'], link['b'])}
|
||||||
|
for name, entry in sorted((known_uplinks or {}).items()):
|
||||||
|
node = nodes.get(name)
|
||||||
|
if not node or name in linked or node['kind'] == 'stub':
|
||||||
|
continue
|
||||||
|
for up in entry.get('uplinks') or []:
|
||||||
|
sw, idx = up.get('switch'), up.get('port_idx')
|
||||||
|
if sw not in nodes or idx is None or (sw, idx) in claimed:
|
||||||
|
continue
|
||||||
|
port = next((p for p in ((switches.get(sw) or {}).get('ports') or {}).values()
|
||||||
|
if p.get('port_idx') == idx), {})
|
||||||
|
far = (_end(name, iface=up.get('host_interface') or '') if node['kind'] == 'host'
|
||||||
|
else _end(name, '', None))
|
||||||
|
link = add_link(_end(sw, up.get('port') or '', idx), far, port,
|
||||||
|
ghost=True, seen=entry.get('seen') or '')
|
||||||
|
link['up'] = False
|
||||||
|
|
||||||
|
# ── Depth: hops from the internet over device-to-device cables ───────
|
||||||
|
adjacency: Dict[str, set] = {}
|
||||||
|
for link in links:
|
||||||
|
a, b = link['a']['node'], link['b']['node']
|
||||||
|
adjacency.setdefault(a, set()).add(b)
|
||||||
|
adjacency.setdefault(b, set()).add(a)
|
||||||
|
depth: Dict[str, int] = {INTERNET: 0}
|
||||||
|
queue = deque([INTERNET])
|
||||||
|
while queue:
|
||||||
|
cur = queue.popleft()
|
||||||
|
for nxt in sorted(adjacency.get(cur, ())):
|
||||||
|
if nxt in depth:
|
||||||
|
continue
|
||||||
|
depth[nxt] = depth[cur] + 1
|
||||||
|
# Hosts and stubs are leaves: they never carry the path on to another device
|
||||||
|
if nodes[nxt]['kind'] not in ('host', 'stub'):
|
||||||
|
queue.append(nxt)
|
||||||
|
for node in nodes.values():
|
||||||
|
node['depth'] = depth.get(node['id'])
|
||||||
|
|
||||||
|
# Orient each cable from the internet side (a) to the far side (b), and count traffic that way
|
||||||
|
for i, link in enumerate(links):
|
||||||
|
da = depth.get(link['a']['node'], 1 << 20)
|
||||||
|
db_ = depth.get(link['b']['node'], 1 << 20)
|
||||||
|
# Rates were measured at end a as added: the reporting switch port, or the internet side
|
||||||
|
tx, rx = link.pop('_tx'), link.pop('_rx')
|
||||||
|
if db_ < da:
|
||||||
|
link['a'], link['b'] = link['b'], link['a']
|
||||||
|
if link.get('name_mismatch') and link['name_mismatch'].get('end'):
|
||||||
|
link['name_mismatch']['end'] = 'a' if link['name_mismatch']['end'] == 'b' else 'b'
|
||||||
|
tx, rx = rx, tx
|
||||||
|
link['a_to_b_bps'] = round(tx, 1)
|
||||||
|
link['b_to_a_bps'] = round(rx, 1)
|
||||||
|
link['utilisation'] = _utilisation(tx, rx, link['speed_mbps'])
|
||||||
|
link['id'] = f'l{i}'
|
||||||
|
|
||||||
|
# Which host NIC each cable lands on, so the drawing can light the right socket
|
||||||
|
for link in links:
|
||||||
|
for end in (link['a'], link['b']):
|
||||||
|
node = nodes[end['node']]
|
||||||
|
if node['kind'] == 'host' and end['iface']:
|
||||||
|
for nic in node.get('interfaces', []):
|
||||||
|
if nic['name'] == end['iface']:
|
||||||
|
nic.setdefault('links', []).append(link['id'])
|
||||||
|
|
||||||
|
if not switches:
|
||||||
|
warnings.append('No UniFi switch data yet: the monitor has not reported any ports')
|
||||||
|
return {
|
||||||
|
'nodes': sorted(nodes.values(), key=lambda n: (99 if n['depth'] is None else n['depth'], n['name'].lower())),
|
||||||
|
'links': links,
|
||||||
|
'warnings': warnings,
|
||||||
|
'updated': link_stats.get('updated'),
|
||||||
|
}
|
||||||
Reference in new issue
Block a user